2024-08-06 06:41:41,490 INFO [trainer.py:870] (1/8) Training started
2024-08-06 06:41:41,491 INFO [trainer.py:889] (1/8) Device: cuda:1
2024-08-06 06:41:41,491 INFO [trainer.py:890] (1/8) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 100, 'reset_interval': 200, 'valid_interval': 2000, 'env_info': {'k2-version': '1.24.3', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '279b0c87015a615b81b147251814d737a548f397', 'k2-git-date': 'Wed May 24 22:24:09 2023', 'lhotse-version': '1.26.0', 'torch-version': '2.0.1+cu118', 'torch-cuda-available': True, 'torch-cuda-version': '11.8', 'python-version': '3.10', 'icefall-git-branch': 'main', 'icefall-git-sha1': '3e4fbb6-dirty', 'icefall-git-date': 'Tue Aug 6 06:30:45 2024', 'icefall-path': '/workspace/icefall_llm', 'k2-path': '/usr/local/lib/python3.10/dist-packages/k2/__init__.py', 'lhotse-path': '/usr/local/lib/python3.10/dist-packages/lhotse/__init__.py', 'hostname': '6865771', 'IP address': '0.104.195.107'}, 'world_size': 8, 'master_port': 12354, 'tensorboard': True, 'num_epochs': 40, 'start_epoch': 100, 'start_batch': 0, 'exp_dir': PosixPath('exp/valle'), 'optimizer_name': 'ScaledAdam', 'scheduler_name': 'Eden', 'base_lr': 0.03, 'warmup_steps': 200, 'seed': 42, 'inf_check': False, 'save_every_n': 1000, 'keep_last_k': 20, 'average_period': 0, 'accumulate_grad_steps': 2, 'dtype': 'float32', 'filter_min_duration': 0.5, 'filter_max_duration': 14.0, 'train_stage': 2, 'visualize': False, 'oom_check': False, 'model_name': 'valle', 'decoder_dim': 1024, 'nhead': 16, 'num_decoder_layers': 12, 'scale_factor': 1.0, 'norm_first': True, 'add_prenet': False, 'prefix_mode': 1, 'share_embedding': True, 'prepend_bos': False, 'num_quantizers': 8, 'scaling_xformers': False, 'manifest_dir': PosixPath('data/tokenized'), 'max_duration': 160, 'bucketing_sampler': True, 'num_buckets': 6, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 0.1, 'on_the_fly_feats': False, 'shuffle': True, 'buffer_size': 40000, 'shuffle_buffer_size': 100000, 'drop_last': False, 'return_cuts': True, 'num_workers': 8, 'enable_spec_aug': False, 'spec_aug_time_warp_factor': 80, 'input_strategy': 'PrecomputedFeatures', 'dataset': 'libritts', 'text_tokens': 'data/tokenized/unique_text_tokens.k2symbols', 'sampling_rate': 24000}
2024-08-06 06:41:41,491 INFO [trainer.py:892] (1/8) About to create model
2024-08-06 06:41:42,394 INFO [trainer.py:899] (1/8) Number of model parameters: 367386628
2024-08-06 06:41:42,395 INFO [checkpoint.py:112] (1/8) Loading checkpoint from exp/valle/epoch-99.pt
2024-08-06 06:41:44,848 INFO [trainer.py:914] (1/8) Using DDP
2024-08-06 06:41:46,902 INFO [datamodule.py:427] (1/8) About to get train cuts
2024-08-06 06:41:46,905 INFO [datamodule.py:434] (1/8) About to get dev cuts
2024-08-06 06:41:46,907 INFO [datamodule.py:292] (1/8) Disable SpecAugment
2024-08-06 06:41:46,907 INFO [datamodule.py:294] (1/8) About to create train dataset
2024-08-06 06:41:46,908 INFO [datamodule.py:323] (1/8) Using DynamicBucketingSampler
2024-08-06 06:41:47,544 INFO [datamodule.py:344] (1/8) About to create train dataloader
2024-08-06 06:41:47,544 INFO [datamodule.py:367] (1/8) About to create dev dataset
2024-08-06 06:41:47,885 INFO [datamodule.py:388] (1/8) About to create dev dataloader
2024-08-06 06:42:36,136 INFO [trainer.py:765] (1/8) Epoch 1, batch 100, train_loss[loss=95.48, NarTop10Accuracy=0.01071, over 7338.00 frames. ], tot_loss[loss=80.61, NarTop10Accuracy=0.0526, over 2353.95 frames. ], batch size: 31, lr: 2.25e-02
2024-08-06 06:43:05,818 INFO [trainer.py:765] (1/8) Epoch 1, batch 200, train_loss[loss=123.2, NarTop10Accuracy=0.01814, over 7100.00 frames. ], tot_loss[loss=99.44, NarTop10Accuracy=0.04509, over 3862.73 frames. ], batch size: 18, lr: 3.00e-02
2024-08-06 06:43:33,849 INFO [trainer.py:765] (1/8) Epoch 1, batch 300, train_loss[loss=74.02, NarTop10Accuracy=0.02318, over 7021.00 frames. ], tot_loss[loss=87.04, NarTop10Accuracy=0.04609, over 4655.62 frames. ], batch size: 22, lr: 3.00e-02
2024-08-06 06:44:05,251 INFO [trainer.py:765] (1/8) Epoch 1, batch 400, train_loss[loss=32.65, NarTop10Accuracy=0.05181, over 5146.00 frames. ], tot_loss[loss=68.01, NarTop10Accuracy=0.05057, over 5100.67 frames. ], batch size: 7, lr: 3.00e-02
2024-08-06 06:44:33,445 INFO [trainer.py:765] (1/8) Epoch 1, batch 500, train_loss[loss=16.62, NarTop10Accuracy=0.02294, over 6211.00 frames. ], tot_loss[loss=48.55, NarTop10Accuracy=0.05707, over 5406.64 frames. ], batch size: 11, lr: 2.99e-02
2024-08-06 06:45:02,924 INFO [trainer.py:765] (1/8) Epoch 1, batch 600, train_loss[loss=5.973, NarTop10Accuracy=0.2194, over 5789.00 frames. ], tot_loss[loss=33.26, NarTop10Accuracy=0.06419, over 5671.07 frames. ], batch size: 9, lr: 2.99e-02
2024-08-06 06:45:40,481 INFO [trainer.py:765] (1/8) Epoch 1, batch 700, train_loss[loss=7.159, NarTop10Accuracy=0.1122, over 5112.00 frames. ], tot_loss[loss=23.51, NarTop10Accuracy=0.07151, over 5732.78 frames. ], batch size: 6, lr: 2.99e-02
2024-08-06 06:46:09,663 INFO [trainer.py:765] (1/8) Epoch 1, batch 800, train_loss[loss=6.634, NarTop10Accuracy=0.1097, over 4287.00 frames. ], tot_loss[loss=17.49, NarTop10Accuracy=0.08213, over 5799.88 frames. ], batch size: 5, lr: 2.98e-02
2024-08-06 06:46:37,734 INFO [trainer.py:765] (1/8) Epoch 1, batch 900, train_loss[loss=5.655, NarTop10Accuracy=0.2295, over 6122.00 frames. ], tot_loss[loss=13, NarTop10Accuracy=0.1115, over 5824.88 frames. ], batch size: 13, lr: 2.98e-02
2024-08-06 06:47:13,909 INFO [trainer.py:765] (1/8) Epoch 1, batch 1000, train_loss[loss=5.909, NarTop10Accuracy=0.1985, over 6163.00 frames. ], tot_loss[loss=10.16, NarTop10Accuracy=0.1368, over 5918.80 frames. ], batch size: 13, lr: 2.97e-02
2024-08-06 06:47:47,141 INFO [trainer.py:765] (1/8) Epoch 1, batch 1100, train_loss[loss=5.399, NarTop10Accuracy=0.2207, over 6811.00 frames. ], tot_loss[loss=8.407, NarTop10Accuracy=0.1577, over 5955.83 frames. ], batch size: 17, lr: 2.96e-02
2024-08-06 06:48:15,710 INFO [trainer.py:765] (1/8) Epoch 1, batch 1200, train_loss[loss=6.288, NarTop10Accuracy=0.1443, over 7167.00 frames. ], tot_loss[loss=7.301, NarTop10Accuracy=0.1748, over 5945.82 frames. ], batch size: 30, lr: 2.96e-02
2024-08-06 06:48:47,236 INFO [trainer.py:765] (1/8) Epoch 1, batch 1300, train_loss[loss=5.522, NarTop10Accuracy=0.1902, over 5214.00 frames. ], tot_loss[loss=6.605, NarTop10Accuracy=0.1865, over 6016.95 frames. ], batch size: 6, lr: 2.95e-02
2024-08-06 06:49:23,566 INFO [trainer.py:765] (1/8) Epoch 1, batch 1400, train_loss[loss=5.7, NarTop10Accuracy=0.1619, over 6036.00 frames. ], tot_loss[loss=6.19, NarTop10Accuracy=0.1922, over 6048.58 frames. ], batch size: 11, lr: 2.94e-02
2024-08-06 06:49:51,506 INFO [trainer.py:765] (1/8) Epoch 1, batch 1500, train_loss[loss=5.577, NarTop10Accuracy=0.2024, over 6291.00 frames. ], tot_loss[loss=5.927, NarTop10Accuracy=0.1985, over 5982.25 frames. ], batch size: 51, lr: 2.94e-02
2024-08-06 06:50:19,162 INFO [trainer.py:765] (1/8) Epoch 1, batch 1600, train_loss[loss=5.27, NarTop10Accuracy=0.2494, over 7050.00 frames. ], tot_loss[loss=5.75, NarTop10Accuracy=0.205, over 5967.19 frames. ], batch size: 22, lr: 2.93e-02
2024-08-06 06:50:45,596 INFO [trainer.py:765] (1/8) Epoch 1, batch 1700, train_loss[loss=5.405, NarTop10Accuracy=0.2146, over 6639.00 frames. ], tot_loss[loss=5.635, NarTop10Accuracy=0.2099, over 5936.66 frames. ], batch size: 14, lr: 2.92e-02
2024-08-06 06:51:11,956 INFO [trainer.py:765] (1/8) Epoch 1, batch 1800, train_loss[loss=5.472, NarTop10Accuracy=0.2135, over 7368.00 frames. ], tot_loss[loss=5.55, NarTop10Accuracy=0.2162, over 6003.93 frames. ], batch size: 22, lr: 2.91e-02
2024-08-06 06:51:38,225 INFO [trainer.py:765] (1/8) Epoch 1, batch 1900, train_loss[loss=5.583, NarTop10Accuracy=0.2018, over 6101.00 frames. ], tot_loss[loss=5.497, NarTop10Accuracy=0.2208, over 6047.91 frames. ], batch size: 48, lr: 2.90e-02
2024-08-06 06:52:03,653 INFO [trainer.py:765] (1/8) Epoch 1, batch 2000, train_loss[loss=5.398, NarTop10Accuracy=0.2356, over 6059.00 frames. ], tot_loss[loss=5.442, NarTop10Accuracy=0.2279, over 6013.41 frames. ], batch size: 49, lr: 2.89e-02
2024-08-06 06:52:03,654 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 06:52:13,994 INFO [trainer.py:811] (1/8) Epoch 1, validation: loss=5.351, NarTop10Accuracy=0.2423, over 1907754.00 frames. 
2024-08-06 06:52:13,995 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 26462MB
2024-08-06 06:52:14,534 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 4.341e+01 2.262e+02 7.241e+02 2.074e+04 7.259e+05, threshold=1.448e+03, percent-clipped=0.0
2024-08-06 06:52:39,585 INFO [trainer.py:765] (1/8) Epoch 1, batch 2100, train_loss[loss=5.6, NarTop10Accuracy=0.1976, over 3937.00 frames. ], tot_loss[loss=5.382, NarTop10Accuracy=0.2379, over 6003.57 frames. ], batch size: 4, lr: 2.88e-02
2024-08-06 06:53:05,354 INFO [trainer.py:765] (1/8) Epoch 1, batch 2200, train_loss[loss=5.363, NarTop10Accuracy=0.2375, over 7206.00 frames. ], tot_loss[loss=5.353, NarTop10Accuracy=0.2419, over 6042.58 frames. ], batch size: 31, lr: 2.87e-02
2024-08-06 06:53:30,701 INFO [trainer.py:765] (1/8) Epoch 1, batch 2300, train_loss[loss=5.369, NarTop10Accuracy=0.2308, over 5687.00 frames. ], tot_loss[loss=5.338, NarTop10Accuracy=0.245, over 6080.34 frames. ], batch size: 9, lr: 2.86e-02
2024-08-06 06:53:55,359 INFO [trainer.py:765] (1/8) Epoch 1, batch 2400, train_loss[loss=5.363, NarTop10Accuracy=0.2375, over 5222.00 frames. ], tot_loss[loss=5.313, NarTop10Accuracy=0.2502, over 5887.24 frames. ], batch size: 7, lr: 2.85e-02
2024-08-06 06:54:18,659 INFO [trainer.py:765] (1/8) Epoch 1, batch 2500, train_loss[loss=5.221, NarTop10Accuracy=0.2556, over 4233.00 frames. ], tot_loss[loss=5.263, NarTop10Accuracy=0.2595, over 5533.74 frames. ], batch size: 5, lr: 2.84e-02
2024-08-06 06:54:39,787 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 06:55:37,937 INFO [trainer.py:765] (1/8) Epoch 2, batch 100, train_loss[loss=5.307, NarTop10Accuracy=0.2642, over 7298.00 frames. ], tot_loss[loss=5.173, NarTop10Accuracy=0.2809, over 2356.32 frames. ], batch size: 31, lr: 2.77e-02
2024-08-06 06:56:16,405 INFO [trainer.py:765] (1/8) Epoch 2, batch 200, train_loss[loss=4.921, NarTop10Accuracy=0.3477, over 6958.00 frames. ], tot_loss[loss=5.16, NarTop10Accuracy=0.2826, over 3861.05 frames. ], batch size: 17, lr: 2.76e-02
2024-08-06 06:56:44,973 INFO [trainer.py:765] (1/8) Epoch 2, batch 300, train_loss[loss=5.264, NarTop10Accuracy=0.2665, over 7129.00 frames. ], tot_loss[loss=5.151, NarTop10Accuracy=0.285, over 4667.31 frames. ], batch size: 22, lr: 2.75e-02
2024-08-06 06:57:13,939 INFO [trainer.py:765] (1/8) Epoch 2, batch 400, train_loss[loss=5.347, NarTop10Accuracy=0.2348, over 5249.00 frames. ], tot_loss[loss=5.14, NarTop10Accuracy=0.2873, over 5114.37 frames. ], batch size: 7, lr: 2.74e-02
2024-08-06 06:57:56,209 INFO [trainer.py:765] (1/8) Epoch 2, batch 500, train_loss[loss=4.807, NarTop10Accuracy=0.3438, over 6044.00 frames. ], tot_loss[loss=5.1, NarTop10Accuracy=0.2946, over 5403.90 frames. ], batch size: 11, lr: 2.73e-02
2024-08-06 06:58:25,426 INFO [trainer.py:765] (1/8) Epoch 2, batch 600, train_loss[loss=4.977, NarTop10Accuracy=0.3255, over 5672.00 frames. ], tot_loss[loss=5.089, NarTop10Accuracy=0.2963, over 5672.19 frames. ], batch size: 9, lr: 2.71e-02
2024-08-06 06:58:55,282 INFO [trainer.py:765] (1/8) Epoch 2, batch 700, train_loss[loss=4.863, NarTop10Accuracy=0.3315, over 5063.00 frames. ], tot_loss[loss=5.08, NarTop10Accuracy=0.2984, over 5758.54 frames. ], batch size: 6, lr: 2.70e-02
2024-08-06 06:59:31,889 INFO [trainer.py:765] (1/8) Epoch 2, batch 800, train_loss[loss=4.998, NarTop10Accuracy=0.3177, over 5039.00 frames. ], tot_loss[loss=5.081, NarTop10Accuracy=0.2974, over 5815.87 frames. ], batch size: 6, lr: 2.69e-02
2024-08-06 07:00:03,184 INFO [trainer.py:765] (1/8) Epoch 2, batch 900, train_loss[loss=5.461, NarTop10Accuracy=0.209, over 6287.00 frames. ], tot_loss[loss=5.043, NarTop10Accuracy=0.305, over 5828.99 frames. ], batch size: 13, lr: 2.68e-02
2024-08-06 07:00:33,143 INFO [trainer.py:765] (1/8) Epoch 2, batch 1000, train_loss[loss=4.857, NarTop10Accuracy=0.3409, over 6749.00 frames. ], tot_loss[loss=5.014, NarTop10Accuracy=0.311, over 5920.84 frames. ], batch size: 14, lr: 2.66e-02
2024-08-06 07:01:05,573 INFO [trainer.py:765] (1/8) Epoch 2, batch 1100, train_loss[loss=4.928, NarTop10Accuracy=0.3316, over 7044.00 frames. ], tot_loss[loss=5, NarTop10Accuracy=0.3138, over 5938.50 frames. ], batch size: 17, lr: 2.65e-02
2024-08-06 07:01:46,285 INFO [trainer.py:765] (1/8) Epoch 2, batch 1200, train_loss[loss=4.887, NarTop10Accuracy=0.3364, over 7103.00 frames. ], tot_loss[loss=5.002, NarTop10Accuracy=0.3138, over 5954.53 frames. ], batch size: 30, lr: 2.64e-02
2024-08-06 07:02:15,645 INFO [trainer.py:765] (1/8) Epoch 2, batch 1300, train_loss[loss=5.306, NarTop10Accuracy=0.2512, over 5060.00 frames. ], tot_loss[loss=4.963, NarTop10Accuracy=0.3214, over 6029.01 frames. ], batch size: 6, lr: 2.63e-02
2024-08-06 07:02:45,252 INFO [trainer.py:765] (1/8) Epoch 2, batch 1400, train_loss[loss=4.619, NarTop10Accuracy=0.3899, over 6023.00 frames. ], tot_loss[loss=4.942, NarTop10Accuracy=0.3254, over 6043.89 frames. ], batch size: 11, lr: 2.61e-02
2024-08-06 07:02:50,268 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 07:03:02,094 INFO [trainer.py:811] (1/8) Epoch 2, validation: loss=4.943, NarTop10Accuracy=0.3266, over 1907754.00 frames. 
2024-08-06 07:03:02,095 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 29668MB
2024-08-06 07:03:02,638 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 5.429e+01 1.166e+02 1.425e+02 1.750e+02 6.435e+02, threshold=2.851e+02, percent-clipped=0.0
2024-08-06 07:03:25,471 INFO [trainer.py:765] (1/8) Epoch 2, batch 1500, train_loss[loss=5.084, NarTop10Accuracy=0.3025, over 6170.00 frames. ], tot_loss[loss=4.937, NarTop10Accuracy=0.3259, over 5965.99 frames. ], batch size: 49, lr: 2.60e-02
2024-08-06 07:03:53,554 INFO [trainer.py:765] (1/8) Epoch 2, batch 1600, train_loss[loss=4.74, NarTop10Accuracy=0.3589, over 7196.00 frames. ], tot_loss[loss=4.918, NarTop10Accuracy=0.3297, over 5950.39 frames. ], batch size: 22, lr: 2.59e-02
2024-08-06 07:04:20,313 INFO [trainer.py:765] (1/8) Epoch 2, batch 1700, train_loss[loss=4.651, NarTop10Accuracy=0.3798, over 6384.00 frames. ], tot_loss[loss=4.911, NarTop10Accuracy=0.3315, over 5935.16 frames. ], batch size: 13, lr: 2.58e-02
2024-08-06 07:04:46,888 INFO [trainer.py:765] (1/8) Epoch 2, batch 1800, train_loss[loss=5.009, NarTop10Accuracy=0.3254, over 7093.00 frames. ], tot_loss[loss=4.896, NarTop10Accuracy=0.3341, over 5995.91 frames. ], batch size: 22, lr: 2.56e-02
2024-08-06 07:05:13,586 INFO [trainer.py:765] (1/8) Epoch 2, batch 1900, train_loss[loss=4.9, NarTop10Accuracy=0.3313, over 5847.00 frames. ], tot_loss[loss=4.875, NarTop10Accuracy=0.3385, over 6033.26 frames. ], batch size: 50, lr: 2.55e-02
2024-08-06 07:05:39,285 INFO [trainer.py:765] (1/8) Epoch 2, batch 2000, train_loss[loss=4.785, NarTop10Accuracy=0.3602, over 6537.00 frames. ], tot_loss[loss=4.853, NarTop10Accuracy=0.3429, over 6010.76 frames. ], batch size: 49, lr: 2.54e-02
2024-08-06 07:06:04,829 INFO [trainer.py:765] (1/8) Epoch 2, batch 2100, train_loss[loss=4.389, NarTop10Accuracy=0.4335, over 4784.00 frames. ], tot_loss[loss=4.857, NarTop10Accuracy=0.342, over 5997.84 frames. ], batch size: 5, lr: 2.52e-02
2024-08-06 07:06:30,372 INFO [trainer.py:765] (1/8) Epoch 2, batch 2200, train_loss[loss=4.8, NarTop10Accuracy=0.3612, over 7333.00 frames. ], tot_loss[loss=4.816, NarTop10Accuracy=0.3507, over 6034.34 frames. ], batch size: 31, lr: 2.51e-02
2024-08-06 07:06:55,874 INFO [trainer.py:765] (1/8) Epoch 2, batch 2300, train_loss[loss=4.501, NarTop10Accuracy=0.409, over 5741.00 frames. ], tot_loss[loss=4.804, NarTop10Accuracy=0.3533, over 6060.88 frames. ], batch size: 9, lr: 2.50e-02
2024-08-06 07:07:20,577 INFO [trainer.py:765] (1/8) Epoch 2, batch 2400, train_loss[loss=4.445, NarTop10Accuracy=0.4323, over 5054.00 frames. ], tot_loss[loss=4.768, NarTop10Accuracy=0.3605, over 5874.61 frames. ], batch size: 7, lr: 2.49e-02
2024-08-06 07:07:47,112 INFO [trainer.py:765] (1/8) Epoch 2, batch 2500, train_loss[loss=4.437, NarTop10Accuracy=0.4204, over 4970.00 frames. ], tot_loss[loss=4.741, NarTop10Accuracy=0.3658, over 5544.05 frames. ], batch size: 6, lr: 2.47e-02
2024-08-06 07:08:08,316 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 07:09:08,538 INFO [trainer.py:765] (1/8) Epoch 3, batch 100, train_loss[loss=4.768, NarTop10Accuracy=0.3476, over 7389.00 frames. ], tot_loss[loss=4.663, NarTop10Accuracy=0.3831, over 2356.81 frames. ], batch size: 30, lr: 2.35e-02
2024-08-06 07:09:41,500 INFO [trainer.py:765] (1/8) Epoch 3, batch 200, train_loss[loss=4.356, NarTop10Accuracy=0.4441, over 6962.00 frames. ], tot_loss[loss=4.622, NarTop10Accuracy=0.3913, over 3865.61 frames. ], batch size: 17, lr: 2.34e-02
2024-08-06 07:10:16,976 INFO [trainer.py:765] (1/8) Epoch 3, batch 300, train_loss[loss=4.33, NarTop10Accuracy=0.4402, over 7218.00 frames. ], tot_loss[loss=4.61, NarTop10Accuracy=0.3931, over 4670.78 frames. ], batch size: 22, lr: 2.33e-02
2024-08-06 07:10:49,792 INFO [trainer.py:765] (1/8) Epoch 3, batch 400, train_loss[loss=4.424, NarTop10Accuracy=0.4357, over 5298.00 frames. ], tot_loss[loss=4.584, NarTop10Accuracy=0.3979, over 5125.51 frames. ], batch size: 7, lr: 2.32e-02
2024-08-06 07:11:18,179 INFO [trainer.py:765] (1/8) Epoch 3, batch 500, train_loss[loss=4.853, NarTop10Accuracy=0.3459, over 6024.00 frames. ], tot_loss[loss=4.586, NarTop10Accuracy=0.398, over 5405.77 frames. ], batch size: 11, lr: 2.31e-02
2024-08-06 07:11:51,262 INFO [trainer.py:765] (1/8) Epoch 3, batch 600, train_loss[loss=4.493, NarTop10Accuracy=0.4242, over 5752.00 frames. ], tot_loss[loss=4.568, NarTop10Accuracy=0.4012, over 5674.47 frames. ], batch size: 9, lr: 2.30e-02
2024-08-06 07:12:32,101 INFO [trainer.py:765] (1/8) Epoch 3, batch 700, train_loss[loss=4.423, NarTop10Accuracy=0.4249, over 5080.00 frames. ], tot_loss[loss=4.548, NarTop10Accuracy=0.4045, over 5737.26 frames. ], batch size: 6, lr: 2.29e-02
2024-08-06 07:13:01,919 INFO [trainer.py:765] (1/8) Epoch 3, batch 800, train_loss[loss=4.211, NarTop10Accuracy=0.4682, over 5190.00 frames. ], tot_loss[loss=4.539, NarTop10Accuracy=0.4063, over 5802.64 frames. ], batch size: 6, lr: 2.27e-02
2024-08-06 07:13:12,668 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 07:13:22,883 INFO [trainer.py:811] (1/8) Epoch 3, validation: loss=4.43, NarTop10Accuracy=0.4285, over 1907754.00 frames. 
2024-08-06 07:13:22,884 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 29668MB
2024-08-06 07:13:23,430 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 6.823e+01 1.318e+02 1.583e+02 1.978e+02 8.364e+02, threshold=3.166e+02, percent-clipped=5.2
2024-08-06 07:13:42,435 INFO [trainer.py:765] (1/8) Epoch 3, batch 900, train_loss[loss=4.478, NarTop10Accuracy=0.421, over 6378.00 frames. ], tot_loss[loss=4.513, NarTop10Accuracy=0.4107, over 5816.20 frames. ], batch size: 13, lr: 2.26e-02
2024-08-06 07:14:25,627 INFO [trainer.py:765] (1/8) Epoch 3, batch 1000, train_loss[loss=4.208, NarTop10Accuracy=0.4673, over 6324.00 frames. ], tot_loss[loss=4.496, NarTop10Accuracy=0.4137, over 5915.77 frames. ], batch size: 13, lr: 2.25e-02
2024-08-06 07:14:56,325 INFO [trainer.py:765] (1/8) Epoch 3, batch 1100, train_loss[loss=4.494, NarTop10Accuracy=0.4153, over 6930.00 frames. ], tot_loss[loss=4.486, NarTop10Accuracy=0.4155, over 5931.41 frames. ], batch size: 17, lr: 2.24e-02
2024-08-06 07:15:29,867 INFO [trainer.py:765] (1/8) Epoch 3, batch 1200, train_loss[loss=4.326, NarTop10Accuracy=0.4405, over 7107.00 frames. ], tot_loss[loss=4.476, NarTop10Accuracy=0.4174, over 5938.41 frames. ], batch size: 31, lr: 2.23e-02
2024-08-06 07:16:12,665 INFO [trainer.py:765] (1/8) Epoch 3, batch 1300, train_loss[loss=4.668, NarTop10Accuracy=0.3851, over 4970.00 frames. ], tot_loss[loss=4.462, NarTop10Accuracy=0.4202, over 6012.13 frames. ], batch size: 6, lr: 2.22e-02
2024-08-06 07:16:42,204 INFO [trainer.py:765] (1/8) Epoch 3, batch 1400, train_loss[loss=4.089, NarTop10Accuracy=0.486, over 6067.00 frames. ], tot_loss[loss=4.459, NarTop10Accuracy=0.4205, over 6036.41 frames. ], batch size: 11, lr: 2.21e-02
2024-08-06 07:17:10,663 INFO [trainer.py:765] (1/8) Epoch 3, batch 1500, train_loss[loss=4.647, NarTop10Accuracy=0.3802, over 5656.00 frames. ], tot_loss[loss=4.447, NarTop10Accuracy=0.4225, over 5976.02 frames. ], batch size: 50, lr: 2.20e-02
2024-08-06 07:17:38,769 INFO [trainer.py:765] (1/8) Epoch 3, batch 1600, train_loss[loss=4.173, NarTop10Accuracy=0.482, over 7150.00 frames. ], tot_loss[loss=4.43, NarTop10Accuracy=0.4261, over 5956.80 frames. ], batch size: 22, lr: 2.19e-02
2024-08-06 07:18:05,503 INFO [trainer.py:765] (1/8) Epoch 3, batch 1700, train_loss[loss=4.336, NarTop10Accuracy=0.4374, over 6395.00 frames. ], tot_loss[loss=4.402, NarTop10Accuracy=0.4314, over 5937.46 frames. ], batch size: 13, lr: 2.18e-02
2024-08-06 07:18:32,161 INFO [trainer.py:765] (1/8) Epoch 3, batch 1800, train_loss[loss=4.269, NarTop10Accuracy=0.458, over 7195.00 frames. ], tot_loss[loss=4.384, NarTop10Accuracy=0.4353, over 6006.76 frames. ], batch size: 22, lr: 2.17e-02
2024-08-06 07:19:01,958 INFO [trainer.py:765] (1/8) Epoch 3, batch 1900, train_loss[loss=4.692, NarTop10Accuracy=0.3803, over 5947.00 frames. ], tot_loss[loss=4.369, NarTop10Accuracy=0.4384, over 6036.35 frames. ], batch size: 48, lr: 2.16e-02
2024-08-06 07:19:27,622 INFO [trainer.py:765] (1/8) Epoch 3, batch 2000, train_loss[loss=4.584, NarTop10Accuracy=0.3964, over 5955.00 frames. ], tot_loss[loss=4.348, NarTop10Accuracy=0.4422, over 5998.48 frames. ], batch size: 48, lr: 2.15e-02
2024-08-06 07:19:53,071 INFO [trainer.py:765] (1/8) Epoch 3, batch 2100, train_loss[loss=4.166, NarTop10Accuracy=0.4669, over 4008.00 frames. ], tot_loss[loss=4.313, NarTop10Accuracy=0.449, over 5981.14 frames. ], batch size: 4, lr: 2.14e-02
2024-08-06 07:20:18,554 INFO [trainer.py:765] (1/8) Epoch 3, batch 2200, train_loss[loss=4.49, NarTop10Accuracy=0.4119, over 7193.00 frames. ], tot_loss[loss=4.302, NarTop10Accuracy=0.4514, over 6033.05 frames. ], batch size: 30, lr: 2.13e-02
2024-08-06 07:20:44,051 INFO [trainer.py:765] (1/8) Epoch 3, batch 2300, train_loss[loss=4.179, NarTop10Accuracy=0.4805, over 5865.00 frames. ], tot_loss[loss=4.315, NarTop10Accuracy=0.4489, over 6059.83 frames. ], batch size: 9, lr: 2.12e-02
2024-08-06 07:21:08,677 INFO [trainer.py:765] (1/8) Epoch 3, batch 2400, train_loss[loss=4.301, NarTop10Accuracy=0.4559, over 5668.00 frames. ], tot_loss[loss=4.298, NarTop10Accuracy=0.4519, over 5882.51 frames. ], batch size: 50, lr: 2.11e-02
2024-08-06 07:21:32,172 INFO [trainer.py:765] (1/8) Epoch 3, batch 2500, train_loss[loss=3.955, NarTop10Accuracy=0.5074, over 4933.00 frames. ], tot_loss[loss=4.253, NarTop10Accuracy=0.4603, over 5534.39 frames. ], batch size: 6, lr: 2.10e-02
2024-08-06 07:21:53,202 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 07:23:00,977 INFO [trainer.py:765] (1/8) Epoch 4, batch 100, train_loss[loss=4.148, NarTop10Accuracy=0.4642, over 7455.00 frames. ], tot_loss[loss=4.181, NarTop10Accuracy=0.4776, over 2377.19 frames. ], batch size: 31, lr: 1.97e-02
2024-08-06 07:23:33,304 INFO [trainer.py:765] (1/8) Epoch 4, batch 200, train_loss[loss=4.065, NarTop10Accuracy=0.4959, over 6905.00 frames. ], tot_loss[loss=4.177, NarTop10Accuracy=0.4784, over 3867.89 frames. ], batch size: 17, lr: 1.96e-02
2024-08-06 07:23:51,466 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 07:24:01,517 INFO [trainer.py:811] (1/8) Epoch 4, validation: loss=4.035, NarTop10Accuracy=0.5085, over 1907754.00 frames. 
2024-08-06 07:24:01,517 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 29668MB
2024-08-06 07:24:02,097 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 9.910e+01 1.530e+02 1.750e+02 2.064e+02 5.317e+02, threshold=3.500e+02, percent-clipped=3.3
2024-08-06 07:24:14,362 INFO [trainer.py:765] (1/8) Epoch 4, batch 300, train_loss[loss=4.084, NarTop10Accuracy=0.4976, over 7166.00 frames. ], tot_loss[loss=4.166, NarTop10Accuracy=0.4805, over 4677.58 frames. ], batch size: 22, lr: 1.95e-02
2024-08-06 07:24:53,597 INFO [trainer.py:765] (1/8) Epoch 4, batch 400, train_loss[loss=3.584, NarTop10Accuracy=0.5779, over 5169.00 frames. ], tot_loss[loss=4.172, NarTop10Accuracy=0.4791, over 5128.39 frames. ], batch size: 7, lr: 1.94e-02
2024-08-06 07:25:25,295 INFO [trainer.py:765] (1/8) Epoch 4, batch 500, train_loss[loss=4.333, NarTop10Accuracy=0.4542, over 6093.00 frames. ], tot_loss[loss=4.16, NarTop10Accuracy=0.481, over 5404.85 frames. ], batch size: 11, lr: 1.93e-02
2024-08-06 07:25:56,976 INFO [trainer.py:765] (1/8) Epoch 4, batch 600, train_loss[loss=4.132, NarTop10Accuracy=0.4953, over 5805.00 frames. ], tot_loss[loss=4.149, NarTop10Accuracy=0.4828, over 5674.00 frames. ], batch size: 9, lr: 1.92e-02
2024-08-06 07:26:37,607 INFO [trainer.py:765] (1/8) Epoch 4, batch 700, train_loss[loss=4.212, NarTop10Accuracy=0.47, over 5161.00 frames. ], tot_loss[loss=4.152, NarTop10Accuracy=0.4821, over 5745.21 frames. ], batch size: 6, lr: 1.92e-02
2024-08-06 07:27:07,433 INFO [trainer.py:765] (1/8) Epoch 4, batch 800, train_loss[loss=4.235, NarTop10Accuracy=0.4745, over 5097.00 frames. ], tot_loss[loss=4.146, NarTop10Accuracy=0.4836, over 5788.50 frames. ], batch size: 6, lr: 1.91e-02
2024-08-06 07:27:42,042 INFO [trainer.py:765] (1/8) Epoch 4, batch 900, train_loss[loss=4.148, NarTop10Accuracy=0.4868, over 6306.00 frames. ], tot_loss[loss=4.11, NarTop10Accuracy=0.4907, over 5815.76 frames. ], batch size: 13, lr: 1.90e-02
2024-08-06 07:28:20,670 INFO [trainer.py:765] (1/8) Epoch 4, batch 1000, train_loss[loss=4.064, NarTop10Accuracy=0.4922, over 6723.00 frames. ], tot_loss[loss=4.107, NarTop10Accuracy=0.4913, over 5918.26 frames. ], batch size: 14, lr: 1.89e-02
2024-08-06 07:28:54,071 INFO [trainer.py:765] (1/8) Epoch 4, batch 1100, train_loss[loss=4.12, NarTop10Accuracy=0.5018, over 6834.00 frames. ], tot_loss[loss=4.112, NarTop10Accuracy=0.4909, over 5959.60 frames. ], batch size: 17, lr: 1.88e-02
2024-08-06 07:29:29,599 INFO [trainer.py:765] (1/8) Epoch 4, batch 1200, train_loss[loss=4.137, NarTop10Accuracy=0.4836, over 7527.00 frames. ], tot_loss[loss=4.105, NarTop10Accuracy=0.4913, over 5954.86 frames. ], batch size: 31, lr: 1.87e-02
2024-08-06 07:30:04,991 INFO [trainer.py:765] (1/8) Epoch 4, batch 1300, train_loss[loss=4.116, NarTop10Accuracy=0.4939, over 4986.00 frames. ], tot_loss[loss=4.076, NarTop10Accuracy=0.497, over 6016.67 frames. ], batch size: 6, lr: 1.87e-02
2024-08-06 07:30:43,380 INFO [trainer.py:765] (1/8) Epoch 4, batch 1400, train_loss[loss=4.036, NarTop10Accuracy=0.5135, over 6224.00 frames. ], tot_loss[loss=4.081, NarTop10Accuracy=0.4968, over 6032.02 frames. ], batch size: 11, lr: 1.86e-02
2024-08-06 07:31:11,832 INFO [trainer.py:765] (1/8) Epoch 4, batch 1500, train_loss[loss=4.045, NarTop10Accuracy=0.5068, over 6153.00 frames. ], tot_loss[loss=4.074, NarTop10Accuracy=0.498, over 5968.61 frames. ], batch size: 48, lr: 1.85e-02
2024-08-06 07:31:39,961 INFO [trainer.py:765] (1/8) Epoch 4, batch 1600, train_loss[loss=4.196, NarTop10Accuracy=0.4814, over 7182.00 frames. ], tot_loss[loss=4.067, NarTop10Accuracy=0.4995, over 5955.53 frames. ], batch size: 22, lr: 1.84e-02
2024-08-06 07:32:06,854 INFO [trainer.py:765] (1/8) Epoch 4, batch 1700, train_loss[loss=4.483, NarTop10Accuracy=0.414, over 6288.00 frames. ], tot_loss[loss=4.042, NarTop10Accuracy=0.5042, over 5927.69 frames. ], batch size: 13, lr: 1.84e-02
2024-08-06 07:32:33,483 INFO [trainer.py:765] (1/8) Epoch 4, batch 1800, train_loss[loss=4.269, NarTop10Accuracy=0.4583, over 7169.00 frames. ], tot_loss[loss=4.045, NarTop10Accuracy=0.5037, over 6002.74 frames. ], batch size: 22, lr: 1.83e-02
2024-08-06 07:33:00,193 INFO [trainer.py:765] (1/8) Epoch 4, batch 1900, train_loss[loss=4.119, NarTop10Accuracy=0.4905, over 6124.00 frames. ], tot_loss[loss=4.068, NarTop10Accuracy=0.4992, over 6042.02 frames. ], batch size: 49, lr: 1.82e-02
2024-08-06 07:33:25,990 INFO [trainer.py:765] (1/8) Epoch 4, batch 2000, train_loss[loss=4.484, NarTop10Accuracy=0.424, over 5916.00 frames. ], tot_loss[loss=4.049, NarTop10Accuracy=0.5038, over 6036.04 frames. ], batch size: 50, lr: 1.81e-02
2024-08-06 07:33:51,512 INFO [trainer.py:765] (1/8) Epoch 4, batch 2100, train_loss[loss=3.77, NarTop10Accuracy=0.5771, over 4031.00 frames. ], tot_loss[loss=4.034, NarTop10Accuracy=0.5067, over 6000.97 frames. ], batch size: 4, lr: 1.81e-02
2024-08-06 07:34:16,906 INFO [trainer.py:765] (1/8) Epoch 4, batch 2200, train_loss[loss=4.135, NarTop10Accuracy=0.4873, over 7353.00 frames. ], tot_loss[loss=4.038, NarTop10Accuracy=0.506, over 6058.37 frames. ], batch size: 31, lr: 1.80e-02
2024-08-06 07:34:31,431 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 07:34:41,462 INFO [trainer.py:811] (1/8) Epoch 4, validation: loss=3.858, NarTop10Accuracy=0.5445, over 1907754.00 frames. 
2024-08-06 07:34:41,463 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 29668MB
2024-08-06 07:34:41,980 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.230e+02 1.721e+02 1.919e+02 2.225e+02 9.682e+02, threshold=3.839e+02, percent-clipped=2.3
2024-08-06 07:34:52,442 INFO [trainer.py:765] (1/8) Epoch 4, batch 2300, train_loss[loss=3.938, NarTop10Accuracy=0.5234, over 5739.00 frames. ], tot_loss[loss=4.037, NarTop10Accuracy=0.5065, over 6080.01 frames. ], batch size: 9, lr: 1.79e-02
2024-08-06 07:35:17,166 INFO [trainer.py:765] (1/8) Epoch 4, batch 2400, train_loss[loss=3.689, NarTop10Accuracy=0.574, over 5091.00 frames. ], tot_loss[loss=4.025, NarTop10Accuracy=0.5089, over 5874.56 frames. ], batch size: 7, lr: 1.78e-02
2024-08-06 07:35:40,622 INFO [trainer.py:765] (1/8) Epoch 4, batch 2500, train_loss[loss=4.23, NarTop10Accuracy=0.4682, over 5041.00 frames. ], tot_loss[loss=4.006, NarTop10Accuracy=0.5123, over 5528.81 frames. ], batch size: 6, lr: 1.78e-02
2024-08-06 07:36:01,614 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 07:37:02,524 INFO [trainer.py:765] (1/8) Epoch 5, batch 100, train_loss[loss=3.892, NarTop10Accuracy=0.5303, over 7378.00 frames. ], tot_loss[loss=3.966, NarTop10Accuracy=0.5207, over 2364.36 frames. ], batch size: 30, lr: 1.66e-02
2024-08-06 07:37:39,815 INFO [trainer.py:765] (1/8) Epoch 5, batch 200, train_loss[loss=4.005, NarTop10Accuracy=0.5137, over 6947.00 frames. ], tot_loss[loss=3.937, NarTop10Accuracy=0.5267, over 3860.82 frames. ], batch size: 17, lr: 1.65e-02
2024-08-06 07:38:13,471 INFO [trainer.py:765] (1/8) Epoch 5, batch 300, train_loss[loss=4.162, NarTop10Accuracy=0.4848, over 7034.00 frames. ], tot_loss[loss=3.922, NarTop10Accuracy=0.5301, over 4670.22 frames. ], batch size: 22, lr: 1.65e-02
2024-08-06 07:38:42,429 INFO [trainer.py:765] (1/8) Epoch 5, batch 400, train_loss[loss=3.982, NarTop10Accuracy=0.5282, over 5074.00 frames. ], tot_loss[loss=3.92, NarTop10Accuracy=0.5301, over 5121.30 frames. ], batch size: 7, lr: 1.64e-02
2024-08-06 07:39:17,020 INFO [trainer.py:765] (1/8) Epoch 5, batch 500, train_loss[loss=3.852, NarTop10Accuracy=0.54, over 6190.00 frames. ], tot_loss[loss=3.923, NarTop10Accuracy=0.5293, over 5407.91 frames. ], batch size: 11, lr: 1.63e-02
2024-08-06 07:39:51,943 INFO [trainer.py:765] (1/8) Epoch 5, batch 600, train_loss[loss=3.896, NarTop10Accuracy=0.5335, over 5894.00 frames. ], tot_loss[loss=3.909, NarTop10Accuracy=0.5325, over 5663.70 frames. ], batch size: 9, lr: 1.63e-02
2024-08-06 07:40:28,626 INFO [trainer.py:765] (1/8) Epoch 5, batch 700, train_loss[loss=3.624, NarTop10Accuracy=0.5871, over 4988.00 frames. ], tot_loss[loss=3.914, NarTop10Accuracy=0.5319, over 5733.77 frames. ], batch size: 6, lr: 1.62e-02
2024-08-06 07:41:02,367 INFO [trainer.py:765] (1/8) Epoch 5, batch 800, train_loss[loss=3.873, NarTop10Accuracy=0.5286, over 4965.00 frames. ], tot_loss[loss=3.919, NarTop10Accuracy=0.5309, over 5781.46 frames. ], batch size: 6, lr: 1.62e-02
2024-08-06 07:41:37,937 INFO [trainer.py:765] (1/8) Epoch 5, batch 900, train_loss[loss=4.266, NarTop10Accuracy=0.4645, over 6251.00 frames. ], tot_loss[loss=3.907, NarTop10Accuracy=0.5333, over 5816.88 frames. ], batch size: 13, lr: 1.61e-02
2024-08-06 07:42:13,845 INFO [trainer.py:765] (1/8) Epoch 5, batch 1000, train_loss[loss=4.004, NarTop10Accuracy=0.5216, over 6226.00 frames. ], tot_loss[loss=3.891, NarTop10Accuracy=0.5359, over 5911.96 frames. ], batch size: 13, lr: 1.60e-02
2024-08-06 07:42:46,468 INFO [trainer.py:765] (1/8) Epoch 5, batch 1100, train_loss[loss=3.866, NarTop10Accuracy=0.5318, over 7034.00 frames. ], tot_loss[loss=3.909, NarTop10Accuracy=0.5325, over 5951.20 frames. ], batch size: 17, lr: 1.60e-02
2024-08-06 07:43:25,226 INFO [trainer.py:765] (1/8) Epoch 5, batch 1200, train_loss[loss=4.069, NarTop10Accuracy=0.4905, over 7235.00 frames. ], tot_loss[loss=3.91, NarTop10Accuracy=0.5324, over 5950.62 frames. ], batch size: 30, lr: 1.59e-02
2024-08-06 07:44:00,557 INFO [trainer.py:765] (1/8) Epoch 5, batch 1300, train_loss[loss=3.835, NarTop10Accuracy=0.5477, over 5049.00 frames. ], tot_loss[loss=3.905, NarTop10Accuracy=0.5334, over 6025.44 frames. ], batch size: 6, lr: 1.59e-02
2024-08-06 07:44:30,238 INFO [trainer.py:765] (1/8) Epoch 5, batch 1400, train_loss[loss=3.899, NarTop10Accuracy=0.5402, over 6070.00 frames. ], tot_loss[loss=3.906, NarTop10Accuracy=0.5333, over 6057.45 frames. ], batch size: 11, lr: 1.58e-02
2024-08-06 07:45:02,845 INFO [trainer.py:765] (1/8) Epoch 5, batch 1500, train_loss[loss=3.837, NarTop10Accuracy=0.5511, over 6763.00 frames. ], tot_loss[loss=3.899, NarTop10Accuracy=0.5344, over 6002.51 frames. ], batch size: 49, lr: 1.57e-02
2024-08-06 07:45:31,008 INFO [trainer.py:765] (1/8) Epoch 5, batch 1600, train_loss[loss=4.231, NarTop10Accuracy=0.4763, over 7097.00 frames. ], tot_loss[loss=3.913, NarTop10Accuracy=0.5319, over 5974.89 frames. ], batch size: 22, lr: 1.57e-02
2024-08-06 07:45:51,058 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 07:46:01,621 INFO [trainer.py:811] (1/8) Epoch 5, validation: loss=3.749, NarTop10Accuracy=0.5672, over 1907754.00 frames. 
2024-08-06 07:46:01,622 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 29668MB
2024-08-06 07:46:02,123 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.160e+02 1.669e+02 1.884e+02 2.190e+02 6.243e+02, threshold=3.768e+02, percent-clipped=1.8
2024-08-06 07:46:08,362 INFO [trainer.py:765] (1/8) Epoch 5, batch 1700, train_loss[loss=3.777, NarTop10Accuracy=0.5604, over 6333.00 frames. ], tot_loss[loss=3.899, NarTop10Accuracy=0.5346, over 5947.41 frames. ], batch size: 13, lr: 1.56e-02
2024-08-06 07:46:34,966 INFO [trainer.py:765] (1/8) Epoch 5, batch 1800, train_loss[loss=3.868, NarTop10Accuracy=0.5416, over 7114.00 frames. ], tot_loss[loss=3.897, NarTop10Accuracy=0.5352, over 6000.24 frames. ], batch size: 22, lr: 1.56e-02
2024-08-06 07:47:01,489 INFO [trainer.py:765] (1/8) Epoch 5, batch 1900, train_loss[loss=3.801, NarTop10Accuracy=0.5581, over 5799.00 frames. ], tot_loss[loss=3.897, NarTop10Accuracy=0.5354, over 6032.44 frames. ], batch size: 49, lr: 1.55e-02
2024-08-06 07:47:27,146 INFO [trainer.py:765] (1/8) Epoch 5, batch 2000, train_loss[loss=3.816, NarTop10Accuracy=0.5637, over 6037.00 frames. ], tot_loss[loss=3.895, NarTop10Accuracy=0.5358, over 5997.48 frames. ], batch size: 48, lr: 1.55e-02
2024-08-06 07:47:52,618 INFO [trainer.py:765] (1/8) Epoch 5, batch 2100, train_loss[loss=3.861, NarTop10Accuracy=0.5442, over 3998.00 frames. ], tot_loss[loss=3.905, NarTop10Accuracy=0.5337, over 5990.23 frames. ], batch size: 4, lr: 1.54e-02
2024-08-06 07:48:17,992 INFO [trainer.py:765] (1/8) Epoch 5, batch 2200, train_loss[loss=3.94, NarTop10Accuracy=0.522, over 7377.00 frames. ], tot_loss[loss=3.893, NarTop10Accuracy=0.5362, over 6028.27 frames. ], batch size: 30, lr: 1.54e-02
2024-08-06 07:48:43,421 INFO [trainer.py:765] (1/8) Epoch 5, batch 2300, train_loss[loss=4.03, NarTop10Accuracy=0.509, over 5783.00 frames. ], tot_loss[loss=3.901, NarTop10Accuracy=0.5345, over 6058.71 frames. ], batch size: 9, lr: 1.53e-02
2024-08-06 07:49:08,169 INFO [trainer.py:765] (1/8) Epoch 5, batch 2400, train_loss[loss=4.023, NarTop10Accuracy=0.5078, over 6738.00 frames. ], tot_loss[loss=3.894, NarTop10Accuracy=0.5361, over 5884.97 frames. ], batch size: 49, lr: 1.53e-02
2024-08-06 07:49:31,644 INFO [trainer.py:765] (1/8) Epoch 5, batch 2500, train_loss[loss=3.757, NarTop10Accuracy=0.5655, over 5007.00 frames. ], tot_loss[loss=3.857, NarTop10Accuracy=0.5431, over 5523.92 frames. ], batch size: 6, lr: 1.52e-02
2024-08-06 07:49:52,738 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 07:50:58,969 INFO [trainer.py:765] (1/8) Epoch 6, batch 100, train_loss[loss=3.73, NarTop10Accuracy=0.5753, over 7156.00 frames. ], tot_loss[loss=3.804, NarTop10Accuracy=0.555, over 2372.57 frames. ], batch size: 30, lr: 1.42e-02
2024-08-06 07:51:31,788 INFO [trainer.py:765] (1/8) Epoch 6, batch 200, train_loss[loss=3.722, NarTop10Accuracy=0.5804, over 6900.00 frames. ], tot_loss[loss=3.796, NarTop10Accuracy=0.5564, over 3873.32 frames. ], batch size: 17, lr: 1.42e-02
2024-08-06 07:52:04,696 INFO [trainer.py:765] (1/8) Epoch 6, batch 300, train_loss[loss=3.538, NarTop10Accuracy=0.6178, over 7047.00 frames. ], tot_loss[loss=3.784, NarTop10Accuracy=0.5593, over 4677.92 frames. ], batch size: 22, lr: 1.41e-02
2024-08-06 07:52:36,200 INFO [trainer.py:765] (1/8) Epoch 6, batch 400, train_loss[loss=3.773, NarTop10Accuracy=0.5595, over 5095.00 frames. ], tot_loss[loss=3.781, NarTop10Accuracy=0.5598, over 5118.65 frames. ], batch size: 7, lr: 1.41e-02
2024-08-06 07:53:06,102 INFO [trainer.py:765] (1/8) Epoch 6, batch 500, train_loss[loss=3.929, NarTop10Accuracy=0.5299, over 6083.00 frames. ], tot_loss[loss=3.767, NarTop10Accuracy=0.5624, over 5406.40 frames. ], batch size: 11, lr: 1.40e-02
2024-08-06 07:53:43,285 INFO [trainer.py:765] (1/8) Epoch 6, batch 600, train_loss[loss=3.849, NarTop10Accuracy=0.5458, over 5849.00 frames. ], tot_loss[loss=3.784, NarTop10Accuracy=0.5592, over 5669.10 frames. ], batch size: 9, lr: 1.40e-02
2024-08-06 07:54:15,438 INFO [trainer.py:765] (1/8) Epoch 6, batch 700, train_loss[loss=4.058, NarTop10Accuracy=0.4975, over 4930.00 frames. ], tot_loss[loss=3.795, NarTop10Accuracy=0.5572, over 5728.99 frames. ], batch size: 6, lr: 1.39e-02
2024-08-06 07:54:49,526 INFO [trainer.py:765] (1/8) Epoch 6, batch 800, train_loss[loss=3.631, NarTop10Accuracy=0.5903, over 5060.00 frames. ], tot_loss[loss=3.792, NarTop10Accuracy=0.5572, over 5797.67 frames. ], batch size: 6, lr: 1.39e-02
2024-08-06 07:55:21,984 INFO [trainer.py:765] (1/8) Epoch 6, batch 900, train_loss[loss=3.43, NarTop10Accuracy=0.6379, over 6292.00 frames. ], tot_loss[loss=3.787, NarTop10Accuracy=0.5584, over 5820.39 frames. ], batch size: 13, lr: 1.38e-02
2024-08-06 07:56:00,804 INFO [trainer.py:765] (1/8) Epoch 6, batch 1000, train_loss[loss=3.608, NarTop10Accuracy=0.6, over 6371.00 frames. ], tot_loss[loss=3.799, NarTop10Accuracy=0.5561, over 5929.63 frames. ], batch size: 13, lr: 1.38e-02
2024-08-06 07:56:34,171 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 07:56:44,742 INFO [trainer.py:811] (1/8) Epoch 6, validation: loss=3.634, NarTop10Accuracy=0.5919, over 1907754.00 frames. 
2024-08-06 07:56:44,743 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30083MB
2024-08-06 07:56:45,277 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.300e+02 1.714e+02 1.918e+02 2.211e+02 6.360e+02, threshold=3.836e+02, percent-clipped=1.6
2024-08-06 07:56:46,639 INFO [trainer.py:765] (1/8) Epoch 6, batch 1100, train_loss[loss=3.741, NarTop10Accuracy=0.578, over 6746.00 frames. ], tot_loss[loss=3.799, NarTop10Accuracy=0.5558, over 5957.60 frames. ], batch size: 17, lr: 1.37e-02
2024-08-06 07:57:24,888 INFO [trainer.py:765] (1/8) Epoch 6, batch 1200, train_loss[loss=4.056, NarTop10Accuracy=0.504, over 7329.00 frames. ], tot_loss[loss=3.797, NarTop10Accuracy=0.5566, over 5961.50 frames. ], batch size: 31, lr: 1.37e-02
2024-08-06 07:57:56,612 INFO [trainer.py:765] (1/8) Epoch 6, batch 1300, train_loss[loss=3.523, NarTop10Accuracy=0.6008, over 5095.00 frames. ], tot_loss[loss=3.795, NarTop10Accuracy=0.5566, over 6025.39 frames. ], batch size: 6, lr: 1.37e-02
2024-08-06 07:58:30,735 INFO [trainer.py:765] (1/8) Epoch 6, batch 1400, train_loss[loss=3.98, NarTop10Accuracy=0.5149, over 6103.00 frames. ], tot_loss[loss=3.8, NarTop10Accuracy=0.5556, over 6043.35 frames. ], batch size: 11, lr: 1.36e-02
2024-08-06 07:59:00,999 INFO [trainer.py:765] (1/8) Epoch 6, batch 1500, train_loss[loss=4.076, NarTop10Accuracy=0.4978, over 5884.00 frames. ], tot_loss[loss=3.803, NarTop10Accuracy=0.5545, over 5971.24 frames. ], batch size: 48, lr: 1.36e-02
2024-08-06 07:59:28,933 INFO [trainer.py:765] (1/8) Epoch 6, batch 1600, train_loss[loss=3.71, NarTop10Accuracy=0.5652, over 7043.00 frames. ], tot_loss[loss=3.799, NarTop10Accuracy=0.5555, over 5953.64 frames. ], batch size: 22, lr: 1.35e-02
2024-08-06 07:59:55,618 INFO [trainer.py:765] (1/8) Epoch 6, batch 1700, train_loss[loss=3.708, NarTop10Accuracy=0.5658, over 6367.00 frames. ], tot_loss[loss=3.791, NarTop10Accuracy=0.5577, over 5943.57 frames. ], batch size: 13, lr: 1.35e-02
2024-08-06 08:00:22,188 INFO [trainer.py:765] (1/8) Epoch 6, batch 1800, train_loss[loss=3.932, NarTop10Accuracy=0.5292, over 6694.00 frames. ], tot_loss[loss=3.786, NarTop10Accuracy=0.5579, over 5993.59 frames. ], batch size: 21, lr: 1.35e-02
2024-08-06 08:00:48,795 INFO [trainer.py:765] (1/8) Epoch 6, batch 1900, train_loss[loss=4.129, NarTop10Accuracy=0.4894, over 6106.00 frames. ], tot_loss[loss=3.817, NarTop10Accuracy=0.552, over 6047.51 frames. ], batch size: 50, lr: 1.34e-02
2024-08-06 08:01:14,461 INFO [trainer.py:765] (1/8) Epoch 6, batch 2000, train_loss[loss=3.971, NarTop10Accuracy=0.5225, over 6129.00 frames. ], tot_loss[loss=3.796, NarTop10Accuracy=0.5558, over 6027.10 frames. ], batch size: 49, lr: 1.34e-02
2024-08-06 08:01:43,133 INFO [trainer.py:765] (1/8) Epoch 6, batch 2100, train_loss[loss=3.274, NarTop10Accuracy=0.6612, over 4838.00 frames. ], tot_loss[loss=3.793, NarTop10Accuracy=0.5566, over 6009.84 frames. ], batch size: 5, lr: 1.33e-02
2024-08-06 08:02:08,518 INFO [trainer.py:765] (1/8) Epoch 6, batch 2200, train_loss[loss=3.683, NarTop10Accuracy=0.5801, over 7373.00 frames. ], tot_loss[loss=3.798, NarTop10Accuracy=0.5559, over 6047.91 frames. ], batch size: 31, lr: 1.33e-02
2024-08-06 08:02:33,916 INFO [trainer.py:765] (1/8) Epoch 6, batch 2300, train_loss[loss=3.509, NarTop10Accuracy=0.6173, over 5801.00 frames. ], tot_loss[loss=3.801, NarTop10Accuracy=0.555, over 6065.73 frames. ], batch size: 9, lr: 1.33e-02
2024-08-06 08:02:58,616 INFO [trainer.py:765] (1/8) Epoch 6, batch 2400, train_loss[loss=3.541, NarTop10Accuracy=0.597, over 5151.00 frames. ], tot_loss[loss=3.794, NarTop10Accuracy=0.5563, over 5875.63 frames. ], batch size: 7, lr: 1.32e-02
2024-08-06 08:03:21,939 INFO [trainer.py:765] (1/8) Epoch 6, batch 2500, train_loss[loss=3.792, NarTop10Accuracy=0.5611, over 5073.00 frames. ], tot_loss[loss=3.77, NarTop10Accuracy=0.561, over 5522.47 frames. ], batch size: 6, lr: 1.32e-02
2024-08-06 08:03:42,842 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 08:04:42,817 INFO [trainer.py:765] (1/8) Epoch 7, batch 100, train_loss[loss=4.007, NarTop10Accuracy=0.521, over 7487.00 frames. ], tot_loss[loss=3.694, NarTop10Accuracy=0.5785, over 2378.93 frames. ], batch size: 31, lr: 1.23e-02
2024-08-06 08:05:18,347 INFO [trainer.py:765] (1/8) Epoch 7, batch 200, train_loss[loss=3.816, NarTop10Accuracy=0.5591, over 6833.00 frames. ], tot_loss[loss=3.717, NarTop10Accuracy=0.5737, over 3882.01 frames. ], batch size: 17, lr: 1.23e-02
2024-08-06 08:05:46,773 INFO [trainer.py:765] (1/8) Epoch 7, batch 300, train_loss[loss=3.556, NarTop10Accuracy=0.6168, over 7149.00 frames. ], tot_loss[loss=3.724, NarTop10Accuracy=0.5722, over 4680.97 frames. ], batch size: 22, lr: 1.23e-02
2024-08-06 08:06:22,091 INFO [trainer.py:765] (1/8) Epoch 7, batch 400, train_loss[loss=3.789, NarTop10Accuracy=0.5515, over 5221.00 frames. ], tot_loss[loss=3.716, NarTop10Accuracy=0.5733, over 5134.30 frames. ], batch size: 7, lr: 1.22e-02
2024-08-06 08:06:52,316 INFO [trainer.py:765] (1/8) Epoch 7, batch 500, train_loss[loss=3.843, NarTop10Accuracy=0.5546, over 6200.00 frames. ], tot_loss[loss=3.717, NarTop10Accuracy=0.5729, over 5407.12 frames. ], batch size: 11, lr: 1.22e-02
2024-08-06 08:06:56,086 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 08:07:06,251 INFO [trainer.py:811] (1/8) Epoch 7, validation: loss=3.56, NarTop10Accuracy=0.6069, over 1907754.00 frames. 
2024-08-06 08:07:06,252 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30083MB
2024-08-06 08:07:06,837 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.316e+02 1.760e+02 1.958e+02 2.227e+02 5.399e+02, threshold=3.916e+02, percent-clipped=0.8
2024-08-06 08:07:33,151 INFO [trainer.py:765] (1/8) Epoch 7, batch 600, train_loss[loss=3.527, NarTop10Accuracy=0.6106, over 5751.00 frames. ], tot_loss[loss=3.713, NarTop10Accuracy=0.5738, over 5669.33 frames. ], batch size: 9, lr: 1.22e-02
2024-08-06 08:08:11,333 INFO [trainer.py:765] (1/8) Epoch 7, batch 700, train_loss[loss=3.518, NarTop10Accuracy=0.6119, over 4276.00 frames. ], tot_loss[loss=3.714, NarTop10Accuracy=0.5738, over 5727.57 frames. ], batch size: 5, lr: 1.21e-02
2024-08-06 08:08:45,557 INFO [trainer.py:765] (1/8) Epoch 7, batch 800, train_loss[loss=3.558, NarTop10Accuracy=0.6016, over 5075.00 frames. ], tot_loss[loss=3.694, NarTop10Accuracy=0.5778, over 5773.85 frames. ], batch size: 6, lr: 1.21e-02
2024-08-06 08:09:17,739 INFO [trainer.py:765] (1/8) Epoch 7, batch 900, train_loss[loss=3.764, NarTop10Accuracy=0.5656, over 6203.00 frames. ], tot_loss[loss=3.701, NarTop10Accuracy=0.5762, over 5801.86 frames. ], batch size: 13, lr: 1.21e-02
2024-08-06 08:09:54,192 INFO [trainer.py:765] (1/8) Epoch 7, batch 1000, train_loss[loss=3.821, NarTop10Accuracy=0.5413, over 6206.00 frames. ], tot_loss[loss=3.703, NarTop10Accuracy=0.5754, over 5893.67 frames. ], batch size: 13, lr: 1.20e-02
2024-08-06 08:10:29,570 INFO [trainer.py:765] (1/8) Epoch 7, batch 1100, train_loss[loss=3.741, NarTop10Accuracy=0.567, over 6804.00 frames. ], tot_loss[loss=3.717, NarTop10Accuracy=0.5726, over 5941.20 frames. ], batch size: 17, lr: 1.20e-02
2024-08-06 08:11:02,491 INFO [trainer.py:765] (1/8) Epoch 7, batch 1200, train_loss[loss=3.975, NarTop10Accuracy=0.5139, over 7483.00 frames. ], tot_loss[loss=3.716, NarTop10Accuracy=0.5727, over 5952.66 frames. ], batch size: 31, lr: 1.20e-02
2024-08-06 08:11:33,447 INFO [trainer.py:765] (1/8) Epoch 7, batch 1300, train_loss[loss=3.629, NarTop10Accuracy=0.5992, over 5073.00 frames. ], tot_loss[loss=3.707, NarTop10Accuracy=0.5744, over 6015.32 frames. ], batch size: 6, lr: 1.19e-02
2024-08-06 08:12:10,912 INFO [trainer.py:765] (1/8) Epoch 7, batch 1400, train_loss[loss=3.75, NarTop10Accuracy=0.5638, over 6174.00 frames. ], tot_loss[loss=3.713, NarTop10Accuracy=0.5731, over 6016.51 frames. ], batch size: 11, lr: 1.19e-02
2024-08-06 08:12:42,109 INFO [trainer.py:765] (1/8) Epoch 7, batch 1500, train_loss[loss=3.802, NarTop10Accuracy=0.5595, over 5407.00 frames. ], tot_loss[loss=3.701, NarTop10Accuracy=0.5748, over 5949.80 frames. ], batch size: 48, lr: 1.19e-02
2024-08-06 08:13:13,238 INFO [trainer.py:765] (1/8) Epoch 7, batch 1600, train_loss[loss=3.53, NarTop10Accuracy=0.6128, over 7004.00 frames. ], tot_loss[loss=3.711, NarTop10Accuracy=0.5734, over 5940.20 frames. ], batch size: 22, lr: 1.18e-02
2024-08-06 08:13:40,016 INFO [trainer.py:765] (1/8) Epoch 7, batch 1700, train_loss[loss=3.624, NarTop10Accuracy=0.5796, over 6647.00 frames. ], tot_loss[loss=3.717, NarTop10Accuracy=0.5719, over 5944.71 frames. ], batch size: 14, lr: 1.18e-02
2024-08-06 08:14:06,584 INFO [trainer.py:765] (1/8) Epoch 7, batch 1800, train_loss[loss=3.777, NarTop10Accuracy=0.5653, over 7097.00 frames. ], tot_loss[loss=3.726, NarTop10Accuracy=0.5703, over 6006.58 frames. ], batch size: 22, lr: 1.18e-02
2024-08-06 08:14:33,223 INFO [trainer.py:765] (1/8) Epoch 7, batch 1900, train_loss[loss=4.137, NarTop10Accuracy=0.4908, over 6266.00 frames. ], tot_loss[loss=3.727, NarTop10Accuracy=0.57, over 6045.73 frames. ], batch size: 49, lr: 1.17e-02
2024-08-06 08:14:58,995 INFO [trainer.py:765] (1/8) Epoch 7, batch 2000, train_loss[loss=3.754, NarTop10Accuracy=0.57, over 6201.00 frames. ], tot_loss[loss=3.73, NarTop10Accuracy=0.5696, over 6023.54 frames. ], batch size: 49, lr: 1.17e-02
2024-08-06 08:15:24,423 INFO [trainer.py:765] (1/8) Epoch 7, batch 2100, train_loss[loss=3.766, NarTop10Accuracy=0.5711, over 4613.00 frames. ], tot_loss[loss=3.724, NarTop10Accuracy=0.5707, over 5996.30 frames. ], batch size: 5, lr: 1.17e-02
2024-08-06 08:15:49,960 INFO [trainer.py:765] (1/8) Epoch 7, batch 2200, train_loss[loss=4.132, NarTop10Accuracy=0.4787, over 7303.00 frames. ], tot_loss[loss=3.735, NarTop10Accuracy=0.5682, over 6024.28 frames. ], batch size: 31, lr: 1.17e-02
2024-08-06 08:16:15,490 INFO [trainer.py:765] (1/8) Epoch 7, batch 2300, train_loss[loss=3.977, NarTop10Accuracy=0.5185, over 5853.00 frames. ], tot_loss[loss=3.738, NarTop10Accuracy=0.5673, over 6069.22 frames. ], batch size: 9, lr: 1.16e-02
2024-08-06 08:16:40,319 INFO [trainer.py:765] (1/8) Epoch 7, batch 2400, train_loss[loss=3.892, NarTop10Accuracy=0.5341, over 6117.00 frames. ], tot_loss[loss=3.741, NarTop10Accuracy=0.5675, over 5901.63 frames. ], batch size: 49, lr: 1.16e-02
2024-08-06 08:17:03,739 INFO [trainer.py:765] (1/8) Epoch 7, batch 2500, train_loss[loss=3.732, NarTop10Accuracy=0.5647, over 5108.00 frames. ], tot_loss[loss=3.721, NarTop10Accuracy=0.571, over 5546.44 frames. ], batch size: 6, lr: 1.16e-02
2024-08-06 08:17:06,843 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 08:17:17,433 INFO [trainer.py:811] (1/8) Epoch 7, validation: loss=3.591, NarTop10Accuracy=0.6002, over 1907754.00 frames. 
2024-08-06 08:17:17,433 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30083MB
2024-08-06 08:17:17,902 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.356e+02 1.794e+02 1.981e+02 2.246e+02 4.644e+02, threshold=3.962e+02, percent-clipped=1.0
2024-08-06 08:17:35,358 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 08:18:36,193 INFO [trainer.py:765] (1/8) Epoch 8, batch 100, train_loss[loss=3.732, NarTop10Accuracy=0.5625, over 7398.00 frames. ], tot_loss[loss=3.664, NarTop10Accuracy=0.5843, over 2376.18 frames. ], batch size: 30, lr: 1.09e-02
2024-08-06 08:19:15,019 INFO [trainer.py:765] (1/8) Epoch 8, batch 200, train_loss[loss=3.569, NarTop10Accuracy=0.6045, over 6806.00 frames. ], tot_loss[loss=3.663, NarTop10Accuracy=0.5844, over 3873.47 frames. ], batch size: 17, lr: 1.09e-02
2024-08-06 08:19:43,561 INFO [trainer.py:765] (1/8) Epoch 8, batch 300, train_loss[loss=3.574, NarTop10Accuracy=0.6002, over 7223.00 frames. ], tot_loss[loss=3.667, NarTop10Accuracy=0.584, over 4673.96 frames. ], batch size: 22, lr: 1.08e-02
2024-08-06 08:20:16,268 INFO [trainer.py:765] (1/8) Epoch 8, batch 400, train_loss[loss=3.573, NarTop10Accuracy=0.5974, over 5127.00 frames. ], tot_loss[loss=3.666, NarTop10Accuracy=0.5841, over 5115.23 frames. ], batch size: 7, lr: 1.08e-02
2024-08-06 08:20:48,421 INFO [trainer.py:765] (1/8) Epoch 8, batch 500, train_loss[loss=3.458, NarTop10Accuracy=0.6359, over 6514.00 frames. ], tot_loss[loss=3.655, NarTop10Accuracy=0.586, over 5396.80 frames. ], batch size: 12, lr: 1.08e-02
2024-08-06 08:21:23,737 INFO [trainer.py:765] (1/8) Epoch 8, batch 600, train_loss[loss=3.902, NarTop10Accuracy=0.5278, over 5619.00 frames. ], tot_loss[loss=3.661, NarTop10Accuracy=0.5842, over 5678.32 frames. ], batch size: 9, lr: 1.07e-02
2024-08-06 08:21:57,607 INFO [trainer.py:765] (1/8) Epoch 8, batch 700, train_loss[loss=3.803, NarTop10Accuracy=0.5494, over 5129.00 frames. ], tot_loss[loss=3.665, NarTop10Accuracy=0.5838, over 5753.78 frames. ], batch size: 6, lr: 1.07e-02
2024-08-06 08:22:27,341 INFO [trainer.py:765] (1/8) Epoch 8, batch 800, train_loss[loss=3.178, NarTop10Accuracy=0.6707, over 5100.00 frames. ], tot_loss[loss=3.673, NarTop10Accuracy=0.582, over 5806.52 frames. ], batch size: 6, lr: 1.07e-02
2024-08-06 08:23:06,892 INFO [trainer.py:765] (1/8) Epoch 8, batch 900, train_loss[loss=3.357, NarTop10Accuracy=0.6456, over 6656.00 frames. ], tot_loss[loss=3.662, NarTop10Accuracy=0.5836, over 5822.93 frames. ], batch size: 14, lr: 1.07e-02
2024-08-06 08:23:42,943 INFO [trainer.py:765] (1/8) Epoch 8, batch 1000, train_loss[loss=3.552, NarTop10Accuracy=0.6147, over 6367.00 frames. ], tot_loss[loss=3.665, NarTop10Accuracy=0.5828, over 5924.84 frames. ], batch size: 13, lr: 1.06e-02
2024-08-06 08:24:15,105 INFO [trainer.py:765] (1/8) Epoch 8, batch 1100, train_loss[loss=3.897, NarTop10Accuracy=0.5405, over 6896.00 frames. ], tot_loss[loss=3.673, NarTop10Accuracy=0.5806, over 5955.57 frames. ], batch size: 17, lr: 1.06e-02
2024-08-06 08:24:57,339 INFO [trainer.py:765] (1/8) Epoch 8, batch 1200, train_loss[loss=3.563, NarTop10Accuracy=0.6049, over 7311.00 frames. ], tot_loss[loss=3.681, NarTop10Accuracy=0.5788, over 5940.48 frames. ], batch size: 31, lr: 1.06e-02
2024-08-06 08:25:26,604 INFO [trainer.py:765] (1/8) Epoch 8, batch 1300, train_loss[loss=3.694, NarTop10Accuracy=0.5863, over 5179.00 frames. ], tot_loss[loss=3.66, NarTop10Accuracy=0.5831, over 6016.84 frames. ], batch size: 6, lr: 1.06e-02
2024-08-06 08:26:00,605 INFO [trainer.py:765] (1/8) Epoch 8, batch 1400, train_loss[loss=3.626, NarTop10Accuracy=0.5866, over 6072.00 frames. ], tot_loss[loss=3.677, NarTop10Accuracy=0.5802, over 6017.50 frames. ], batch size: 11, lr: 1.05e-02
2024-08-06 08:26:28,987 INFO [trainer.py:765] (1/8) Epoch 8, batch 1500, train_loss[loss=3.763, NarTop10Accuracy=0.5724, over 6488.00 frames. ], tot_loss[loss=3.666, NarTop10Accuracy=0.5827, over 5979.72 frames. ], batch size: 50, lr: 1.05e-02
2024-08-06 08:26:56,933 INFO [trainer.py:765] (1/8) Epoch 8, batch 1600, train_loss[loss=3.858, NarTop10Accuracy=0.5433, over 7214.00 frames. ], tot_loss[loss=3.668, NarTop10Accuracy=0.5822, over 5958.10 frames. ], batch size: 22, lr: 1.05e-02
2024-08-06 08:27:23,763 INFO [trainer.py:765] (1/8) Epoch 8, batch 1700, train_loss[loss=3.554, NarTop10Accuracy=0.6035, over 6232.00 frames. ], tot_loss[loss=3.668, NarTop10Accuracy=0.5824, over 5936.60 frames. ], batch size: 13, lr: 1.05e-02
2024-08-06 08:27:50,462 INFO [trainer.py:765] (1/8) Epoch 8, batch 1800, train_loss[loss=3.613, NarTop10Accuracy=0.5994, over 7107.00 frames. ], tot_loss[loss=3.665, NarTop10Accuracy=0.5836, over 6008.18 frames. ], batch size: 22, lr: 1.04e-02
2024-08-06 08:28:17,180 INFO [trainer.py:765] (1/8) Epoch 8, batch 1900, train_loss[loss=4.129, NarTop10Accuracy=0.4908, over 6105.00 frames. ], tot_loss[loss=3.672, NarTop10Accuracy=0.5816, over 6045.34 frames. ], batch size: 49, lr: 1.04e-02
2024-08-06 08:28:25,164 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 08:28:35,290 INFO [trainer.py:811] (1/8) Epoch 8, validation: loss=3.507, NarTop10Accuracy=0.6181, over 1907754.00 frames. 
2024-08-06 08:28:35,291 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30083MB
2024-08-06 08:28:35,796 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.304e+02 1.789e+02 1.988e+02 2.230e+02 4.452e+02, threshold=3.975e+02, percent-clipped=0.5
2024-08-06 08:28:52,983 INFO [trainer.py:765] (1/8) Epoch 8, batch 2000, train_loss[loss=3.751, NarTop10Accuracy=0.5651, over 5654.00 frames. ], tot_loss[loss=3.666, NarTop10Accuracy=0.5828, over 6014.65 frames. ], batch size: 48, lr: 1.04e-02
2024-08-06 08:29:18,485 INFO [trainer.py:765] (1/8) Epoch 8, batch 2100, train_loss[loss=3.523, NarTop10Accuracy=0.6154, over 3942.00 frames. ], tot_loss[loss=3.663, NarTop10Accuracy=0.5832, over 5998.49 frames. ], batch size: 4, lr: 1.04e-02
2024-08-06 08:29:43,790 INFO [trainer.py:765] (1/8) Epoch 8, batch 2200, train_loss[loss=4.028, NarTop10Accuracy=0.5113, over 7234.00 frames. ], tot_loss[loss=3.675, NarTop10Accuracy=0.5806, over 6030.68 frames. ], batch size: 30, lr: 1.03e-02
2024-08-06 08:30:09,134 INFO [trainer.py:765] (1/8) Epoch 8, batch 2300, train_loss[loss=3.534, NarTop10Accuracy=0.5986, over 5797.00 frames. ], tot_loss[loss=3.688, NarTop10Accuracy=0.5787, over 6049.10 frames. ], batch size: 9, lr: 1.03e-02
2024-08-06 08:30:33,791 INFO [trainer.py:765] (1/8) Epoch 8, batch 2400, train_loss[loss=3.76, NarTop10Accuracy=0.5615, over 5810.00 frames. ], tot_loss[loss=3.696, NarTop10Accuracy=0.5774, over 5851.14 frames. ], batch size: 48, lr: 1.03e-02
2024-08-06 08:30:57,139 INFO [trainer.py:765] (1/8) Epoch 8, batch 2500, train_loss[loss=3.452, NarTop10Accuracy=0.6305, over 5112.00 frames. ], tot_loss[loss=3.675, NarTop10Accuracy=0.5808, over 5518.30 frames. ], batch size: 6, lr: 1.03e-02
2024-08-06 08:31:18,348 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 08:32:19,099 INFO [trainer.py:765] (1/8) Epoch 9, batch 100, train_loss[loss=3.783, NarTop10Accuracy=0.5637, over 7263.00 frames. ], tot_loss[loss=3.611, NarTop10Accuracy=0.596, over 2374.54 frames. ], batch size: 30, lr: 9.71e-03
2024-08-06 08:32:51,461 INFO [trainer.py:765] (1/8) Epoch 9, batch 200, train_loss[loss=3.501, NarTop10Accuracy=0.6108, over 6791.00 frames. ], tot_loss[loss=3.595, NarTop10Accuracy=0.5989, over 3879.07 frames. ], batch size: 17, lr: 9.69e-03
2024-08-06 08:33:27,115 INFO [trainer.py:765] (1/8) Epoch 9, batch 300, train_loss[loss=3.821, NarTop10Accuracy=0.5558, over 7093.00 frames. ], tot_loss[loss=3.596, NarTop10Accuracy=0.5989, over 4683.25 frames. ], batch size: 22, lr: 9.67e-03
2024-08-06 08:34:00,964 INFO [trainer.py:765] (1/8) Epoch 9, batch 400, train_loss[loss=3.482, NarTop10Accuracy=0.6196, over 5113.00 frames. ], tot_loss[loss=3.59, NarTop10Accuracy=0.5995, over 5133.50 frames. ], batch size: 7, lr: 9.64e-03
2024-08-06 08:34:32,880 INFO [trainer.py:765] (1/8) Epoch 9, batch 500, train_loss[loss=3.76, NarTop10Accuracy=0.5704, over 6224.00 frames. ], tot_loss[loss=3.577, NarTop10Accuracy=0.6017, over 5400.48 frames. ], batch size: 11, lr: 9.62e-03
2024-08-06 08:35:07,497 INFO [trainer.py:765] (1/8) Epoch 9, batch 600, train_loss[loss=3.398, NarTop10Accuracy=0.6427, over 5809.00 frames. ], tot_loss[loss=3.582, NarTop10Accuracy=0.601, over 5667.94 frames. ], batch size: 9, lr: 9.60e-03
2024-08-06 08:35:42,824 INFO [trainer.py:765] (1/8) Epoch 9, batch 700, train_loss[loss=3.776, NarTop10Accuracy=0.5656, over 4973.00 frames. ], tot_loss[loss=3.59, NarTop10Accuracy=0.5995, over 5743.66 frames. ], batch size: 6, lr: 9.58e-03
2024-08-06 08:36:14,821 INFO [trainer.py:765] (1/8) Epoch 9, batch 800, train_loss[loss=3.249, NarTop10Accuracy=0.6659, over 4996.00 frames. ], tot_loss[loss=3.607, NarTop10Accuracy=0.5959, over 5803.23 frames. ], batch size: 6, lr: 9.56e-03
2024-08-06 08:36:46,454 INFO [trainer.py:765] (1/8) Epoch 9, batch 900, train_loss[loss=3.259, NarTop10Accuracy=0.6664, over 6682.00 frames. ], tot_loss[loss=3.612, NarTop10Accuracy=0.5952, over 5827.55 frames. ], batch size: 14, lr: 9.54e-03
2024-08-06 08:37:26,564 INFO [trainer.py:765] (1/8) Epoch 9, batch 1000, train_loss[loss=3.552, NarTop10Accuracy=0.6071, over 6354.00 frames. ], tot_loss[loss=3.617, NarTop10Accuracy=0.5935, over 5928.68 frames. ], batch size: 13, lr: 9.52e-03
2024-08-06 08:37:59,421 INFO [trainer.py:765] (1/8) Epoch 9, batch 1100, train_loss[loss=3.94, NarTop10Accuracy=0.5256, over 6941.00 frames. ], tot_loss[loss=3.629, NarTop10Accuracy=0.5907, over 5951.85 frames. ], batch size: 17, lr: 9.50e-03
2024-08-06 08:38:31,995 INFO [trainer.py:765] (1/8) Epoch 9, batch 1200, train_loss[loss=3.747, NarTop10Accuracy=0.5729, over 6999.00 frames. ], tot_loss[loss=3.63, NarTop10Accuracy=0.5905, over 5945.08 frames. ], batch size: 30, lr: 9.48e-03
2024-08-06 08:39:11,840 INFO [trainer.py:765] (1/8) Epoch 9, batch 1300, train_loss[loss=3.574, NarTop10Accuracy=0.6046, over 4992.00 frames. ], tot_loss[loss=3.63, NarTop10Accuracy=0.5904, over 6020.09 frames. ], batch size: 6, lr: 9.46e-03
2024-08-06 08:39:27,117 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 08:39:38,197 INFO [trainer.py:811] (1/8) Epoch 9, validation: loss=3.495, NarTop10Accuracy=0.6214, over 1907754.00 frames. 
2024-08-06 08:39:38,197 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30083MB
2024-08-06 08:39:38,758 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.320e+02 1.781e+02 1.970e+02 2.189e+02 6.315e+02, threshold=3.940e+02, percent-clipped=0.6
2024-08-06 08:39:52,278 INFO [trainer.py:765] (1/8) Epoch 9, batch 1400, train_loss[loss=3.532, NarTop10Accuracy=0.6133, over 6147.00 frames. ], tot_loss[loss=3.625, NarTop10Accuracy=0.592, over 6036.07 frames. ], batch size: 11, lr: 9.43e-03
2024-08-06 08:40:22,331 INFO [trainer.py:765] (1/8) Epoch 9, batch 1500, train_loss[loss=3.963, NarTop10Accuracy=0.5218, over 6355.00 frames. ], tot_loss[loss=3.634, NarTop10Accuracy=0.5899, over 5982.08 frames. ], batch size: 51, lr: 9.41e-03
2024-08-06 08:40:50,367 INFO [trainer.py:765] (1/8) Epoch 9, batch 1600, train_loss[loss=3.814, NarTop10Accuracy=0.5483, over 7034.00 frames. ], tot_loss[loss=3.636, NarTop10Accuracy=0.5897, over 5964.21 frames. ], batch size: 22, lr: 9.39e-03
2024-08-06 08:41:17,151 INFO [trainer.py:765] (1/8) Epoch 9, batch 1700, train_loss[loss=3.78, NarTop10Accuracy=0.5632, over 6251.00 frames. ], tot_loss[loss=3.642, NarTop10Accuracy=0.5881, over 5951.50 frames. ], batch size: 13, lr: 9.37e-03
2024-08-06 08:41:43,812 INFO [trainer.py:765] (1/8) Epoch 9, batch 1800, train_loss[loss=3.92, NarTop10Accuracy=0.5377, over 7211.00 frames. ], tot_loss[loss=3.628, NarTop10Accuracy=0.5908, over 6026.49 frames. ], batch size: 22, lr: 9.35e-03
2024-08-06 08:42:10,495 INFO [trainer.py:765] (1/8) Epoch 9, batch 1900, train_loss[loss=3.655, NarTop10Accuracy=0.5955, over 6257.00 frames. ], tot_loss[loss=3.633, NarTop10Accuracy=0.59, over 6045.55 frames. ], batch size: 49, lr: 9.33e-03
2024-08-06 08:42:36,203 INFO [trainer.py:765] (1/8) Epoch 9, batch 2000, train_loss[loss=3.801, NarTop10Accuracy=0.5613, over 6116.00 frames. ], tot_loss[loss=3.639, NarTop10Accuracy=0.5886, over 6020.27 frames. ], batch size: 49, lr: 9.31e-03
2024-08-06 08:43:01,667 INFO [trainer.py:765] (1/8) Epoch 9, batch 2100, train_loss[loss=3.607, NarTop10Accuracy=0.591, over 3951.00 frames. ], tot_loss[loss=3.637, NarTop10Accuracy=0.5893, over 6001.08 frames. ], batch size: 4, lr: 9.30e-03
2024-08-06 08:43:27,178 INFO [trainer.py:765] (1/8) Epoch 9, batch 2200, train_loss[loss=3.626, NarTop10Accuracy=0.5965, over 7371.00 frames. ], tot_loss[loss=3.633, NarTop10Accuracy=0.5894, over 6041.21 frames. ], batch size: 31, lr: 9.28e-03
2024-08-06 08:43:52,671 INFO [trainer.py:765] (1/8) Epoch 9, batch 2300, train_loss[loss=3.548, NarTop10Accuracy=0.5889, over 5864.00 frames. ], tot_loss[loss=3.654, NarTop10Accuracy=0.5851, over 6058.76 frames. ], batch size: 9, lr: 9.26e-03
2024-08-06 08:44:20,549 INFO [trainer.py:765] (1/8) Epoch 9, batch 2400, train_loss[loss=3.735, NarTop10Accuracy=0.5749, over 5879.00 frames. ], tot_loss[loss=3.647, NarTop10Accuracy=0.5867, over 5903.57 frames. ], batch size: 48, lr: 9.24e-03
2024-08-06 08:44:44,001 INFO [trainer.py:765] (1/8) Epoch 9, batch 2500, train_loss[loss=3.708, NarTop10Accuracy=0.5791, over 5016.00 frames. ], tot_loss[loss=3.63, NarTop10Accuracy=0.5905, over 5563.30 frames. ], batch size: 6, lr: 9.22e-03
2024-08-06 08:45:04,912 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 08:46:09,063 INFO [trainer.py:765] (1/8) Epoch 10, batch 100, train_loss[loss=3.433, NarTop10Accuracy=0.6349, over 7337.00 frames. ], tot_loss[loss=3.599, NarTop10Accuracy=0.5982, over 2375.26 frames. ], batch size: 30, lr: 8.75e-03
2024-08-06 08:46:44,073 INFO [trainer.py:765] (1/8) Epoch 10, batch 200, train_loss[loss=3.431, NarTop10Accuracy=0.6365, over 6837.00 frames. ], tot_loss[loss=3.569, NarTop10Accuracy=0.604, over 3862.32 frames. ], batch size: 17, lr: 8.73e-03
2024-08-06 08:47:14,442 INFO [trainer.py:765] (1/8) Epoch 10, batch 300, train_loss[loss=3.531, NarTop10Accuracy=0.5996, over 7228.00 frames. ], tot_loss[loss=3.562, NarTop10Accuracy=0.605, over 4675.91 frames. ], batch size: 22, lr: 8.72e-03
2024-08-06 08:47:46,120 INFO [trainer.py:765] (1/8) Epoch 10, batch 400, train_loss[loss=3.677, NarTop10Accuracy=0.5788, over 5160.00 frames. ], tot_loss[loss=3.573, NarTop10Accuracy=0.6029, over 5121.23 frames. ], batch size: 7, lr: 8.70e-03
2024-08-06 08:48:22,369 INFO [trainer.py:765] (1/8) Epoch 10, batch 500, train_loss[loss=3.355, NarTop10Accuracy=0.6475, over 6086.00 frames. ], tot_loss[loss=3.569, NarTop10Accuracy=0.6038, over 5397.45 frames. ], batch size: 11, lr: 8.68e-03
2024-08-06 08:48:53,459 INFO [trainer.py:765] (1/8) Epoch 10, batch 600, train_loss[loss=3.405, NarTop10Accuracy=0.6368, over 5822.00 frames. ], tot_loss[loss=3.572, NarTop10Accuracy=0.6028, over 5664.96 frames. ], batch size: 9, lr: 8.66e-03
2024-08-06 08:49:26,707 INFO [trainer.py:765] (1/8) Epoch 10, batch 700, train_loss[loss=3.202, NarTop10Accuracy=0.6861, over 4934.00 frames. ], tot_loss[loss=3.582, NarTop10Accuracy=0.6005, over 5727.86 frames. ], batch size: 6, lr: 8.65e-03
2024-08-06 08:49:49,165 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 08:50:00,983 INFO [trainer.py:811] (1/8) Epoch 10, validation: loss=3.46, NarTop10Accuracy=0.6279, over 1907754.00 frames. 
2024-08-06 08:50:00,984 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30083MB
2024-08-06 08:50:01,724 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.353e+02 1.818e+02 1.985e+02 2.213e+02 4.843e+02, threshold=3.970e+02, percent-clipped=0.2
2024-08-06 08:50:09,801 INFO [trainer.py:765] (1/8) Epoch 10, batch 800, train_loss[loss=3.656, NarTop10Accuracy=0.593, over 5129.00 frames. ], tot_loss[loss=3.581, NarTop10Accuracy=0.6006, over 5795.32 frames. ], batch size: 6, lr: 8.63e-03
2024-08-06 08:50:42,890 INFO [trainer.py:765] (1/8) Epoch 10, batch 900, train_loss[loss=3.315, NarTop10Accuracy=0.6606, over 6197.00 frames. ], tot_loss[loss=3.574, NarTop10Accuracy=0.6024, over 5826.76 frames. ], batch size: 13, lr: 8.61e-03
2024-08-06 08:51:18,460 INFO [trainer.py:765] (1/8) Epoch 10, batch 1000, train_loss[loss=3.88, NarTop10Accuracy=0.5443, over 6227.00 frames. ], tot_loss[loss=3.588, NarTop10Accuracy=0.5994, over 5919.76 frames. ], batch size: 13, lr: 8.59e-03
2024-08-06 08:51:57,362 INFO [trainer.py:765] (1/8) Epoch 10, batch 1100, train_loss[loss=3.483, NarTop10Accuracy=0.6311, over 7021.00 frames. ], tot_loss[loss=3.602, NarTop10Accuracy=0.5964, over 5946.18 frames. ], batch size: 17, lr: 8.58e-03
2024-08-06 08:52:32,048 INFO [trainer.py:765] (1/8) Epoch 10, batch 1200, train_loss[loss=3.555, NarTop10Accuracy=0.6193, over 7389.00 frames. ], tot_loss[loss=3.6, NarTop10Accuracy=0.5967, over 5951.03 frames. ], batch size: 30, lr: 8.56e-03
2024-08-06 08:53:06,607 INFO [trainer.py:765] (1/8) Epoch 10, batch 1300, train_loss[loss=3.621, NarTop10Accuracy=0.596, over 5147.00 frames. ], tot_loss[loss=3.592, NarTop10Accuracy=0.5978, over 6011.47 frames. ], batch size: 6, lr: 8.54e-03
2024-08-06 08:53:46,880 INFO [trainer.py:765] (1/8) Epoch 10, batch 1400, train_loss[loss=3.631, NarTop10Accuracy=0.5968, over 6185.00 frames. ], tot_loss[loss=3.603, NarTop10Accuracy=0.596, over 6034.53 frames. ], batch size: 11, lr: 8.53e-03
2024-08-06 08:54:17,501 INFO [trainer.py:765] (1/8) Epoch 10, batch 1500, train_loss[loss=3.618, NarTop10Accuracy=0.5999, over 5995.00 frames. ], tot_loss[loss=3.584, NarTop10Accuracy=0.5999, over 5989.93 frames. ], batch size: 49, lr: 8.51e-03
2024-08-06 08:54:45,525 INFO [trainer.py:765] (1/8) Epoch 10, batch 1600, train_loss[loss=3.506, NarTop10Accuracy=0.6191, over 7185.00 frames. ], tot_loss[loss=3.6, NarTop10Accuracy=0.5966, over 5962.96 frames. ], batch size: 22, lr: 8.49e-03
2024-08-06 08:55:12,299 INFO [trainer.py:765] (1/8) Epoch 10, batch 1700, train_loss[loss=3.344, NarTop10Accuracy=0.6426, over 6343.00 frames. ], tot_loss[loss=3.602, NarTop10Accuracy=0.5962, over 5948.11 frames. ], batch size: 13, lr: 8.48e-03
2024-08-06 08:55:41,989 INFO [trainer.py:765] (1/8) Epoch 10, batch 1800, train_loss[loss=3.556, NarTop10Accuracy=0.61, over 7239.00 frames. ], tot_loss[loss=3.602, NarTop10Accuracy=0.5961, over 6019.11 frames. ], batch size: 22, lr: 8.46e-03
2024-08-06 08:56:08,572 INFO [trainer.py:765] (1/8) Epoch 10, batch 1900, train_loss[loss=3.993, NarTop10Accuracy=0.5155, over 6288.00 frames. ], tot_loss[loss=3.603, NarTop10Accuracy=0.5958, over 6055.65 frames. ], batch size: 48, lr: 8.45e-03
2024-08-06 08:56:34,287 INFO [trainer.py:765] (1/8) Epoch 10, batch 2000, train_loss[loss=3.685, NarTop10Accuracy=0.5739, over 6309.00 frames. ], tot_loss[loss=3.607, NarTop10Accuracy=0.5954, over 6025.70 frames. ], batch size: 49, lr: 8.43e-03
2024-08-06 08:56:59,751 INFO [trainer.py:765] (1/8) Epoch 10, batch 2100, train_loss[loss=3.635, NarTop10Accuracy=0.593, over 4766.00 frames. ], tot_loss[loss=3.604, NarTop10Accuracy=0.5957, over 6013.90 frames. ], batch size: 5, lr: 8.41e-03
2024-08-06 08:57:25,280 INFO [trainer.py:765] (1/8) Epoch 10, batch 2200, train_loss[loss=3.776, NarTop10Accuracy=0.5636, over 7148.00 frames. ], tot_loss[loss=3.605, NarTop10Accuracy=0.5955, over 6042.29 frames. ], batch size: 30, lr: 8.40e-03
2024-08-06 08:57:50,682 INFO [trainer.py:765] (1/8) Epoch 10, batch 2300, train_loss[loss=3.454, NarTop10Accuracy=0.6384, over 5881.00 frames. ], tot_loss[loss=3.618, NarTop10Accuracy=0.5929, over 6056.82 frames. ], batch size: 9, lr: 8.38e-03
2024-08-06 08:58:15,344 INFO [trainer.py:765] (1/8) Epoch 10, batch 2400, train_loss[loss=3.497, NarTop10Accuracy=0.6096, over 5258.00 frames. ], tot_loss[loss=3.616, NarTop10Accuracy=0.5937, over 5874.37 frames. ], batch size: 7, lr: 8.37e-03
2024-08-06 08:58:38,809 INFO [trainer.py:765] (1/8) Epoch 10, batch 2500, train_loss[loss=3.249, NarTop10Accuracy=0.6497, over 5130.00 frames. ], tot_loss[loss=3.592, NarTop10Accuracy=0.5982, over 5533.91 frames. ], batch size: 6, lr: 8.35e-03
2024-08-06 08:58:59,809 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 09:00:03,681 INFO [trainer.py:765] (1/8) Epoch 11, batch 100, train_loss[loss=3.457, NarTop10Accuracy=0.6259, over 7301.00 frames. ], tot_loss[loss=3.533, NarTop10Accuracy=0.6123, over 2359.10 frames. ], batch size: 30, lr: 7.96e-03
2024-08-06 09:00:30,915 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 09:00:41,217 INFO [trainer.py:811] (1/8) Epoch 11, validation: loss=3.404, NarTop10Accuracy=0.6396, over 1907754.00 frames. 
2024-08-06 09:00:41,218 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30083MB
2024-08-06 09:00:41,774 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.377e+02 1.800e+02 1.980e+02 2.200e+02 4.491e+02, threshold=3.959e+02, percent-clipped=0.2
2024-08-06 09:00:46,859 INFO [trainer.py:765] (1/8) Epoch 11, batch 200, train_loss[loss=3.825, NarTop10Accuracy=0.5553, over 6919.00 frames. ], tot_loss[loss=3.531, NarTop10Accuracy=0.6118, over 3861.58 frames. ], batch size: 17, lr: 7.94e-03
2024-08-06 09:01:17,853 INFO [trainer.py:765] (1/8) Epoch 11, batch 300, train_loss[loss=3.591, NarTop10Accuracy=0.5953, over 7217.00 frames. ], tot_loss[loss=3.537, NarTop10Accuracy=0.6105, over 4677.83 frames. ], batch size: 22, lr: 7.93e-03
2024-08-06 09:01:50,533 INFO [trainer.py:765] (1/8) Epoch 11, batch 400, train_loss[loss=3.295, NarTop10Accuracy=0.6555, over 5125.00 frames. ], tot_loss[loss=3.537, NarTop10Accuracy=0.611, over 5120.94 frames. ], batch size: 7, lr: 7.91e-03
2024-08-06 09:02:21,238 INFO [trainer.py:765] (1/8) Epoch 11, batch 500, train_loss[loss=3.482, NarTop10Accuracy=0.622, over 6011.00 frames. ], tot_loss[loss=3.538, NarTop10Accuracy=0.6105, over 5410.24 frames. ], batch size: 11, lr: 7.90e-03
2024-08-06 09:03:01,741 INFO [trainer.py:765] (1/8) Epoch 11, batch 600, train_loss[loss=3.373, NarTop10Accuracy=0.6372, over 5787.00 frames. ], tot_loss[loss=3.543, NarTop10Accuracy=0.6095, over 5688.62 frames. ], batch size: 9, lr: 7.88e-03
2024-08-06 09:03:38,236 INFO [trainer.py:765] (1/8) Epoch 11, batch 700, train_loss[loss=3.414, NarTop10Accuracy=0.6345, over 5074.00 frames. ], tot_loss[loss=3.544, NarTop10Accuracy=0.6087, over 5746.82 frames. ], batch size: 6, lr: 7.87e-03
2024-08-06 09:04:10,755 INFO [trainer.py:765] (1/8) Epoch 11, batch 800, train_loss[loss=3.679, NarTop10Accuracy=0.5939, over 4992.00 frames. ], tot_loss[loss=3.559, NarTop10Accuracy=0.6051, over 5806.18 frames. ], batch size: 6, lr: 7.86e-03
2024-08-06 09:04:50,082 INFO [trainer.py:765] (1/8) Epoch 11, batch 900, train_loss[loss=3.44, NarTop10Accuracy=0.6319, over 6232.00 frames. ], tot_loss[loss=3.544, NarTop10Accuracy=0.6082, over 5830.76 frames. ], batch size: 13, lr: 7.84e-03
2024-08-06 09:05:27,011 INFO [trainer.py:765] (1/8) Epoch 11, batch 1000, train_loss[loss=3.389, NarTop10Accuracy=0.6354, over 6310.00 frames. ], tot_loss[loss=3.541, NarTop10Accuracy=0.6087, over 5936.03 frames. ], batch size: 13, lr: 7.83e-03
2024-08-06 09:06:00,349 INFO [trainer.py:765] (1/8) Epoch 11, batch 1100, train_loss[loss=3.604, NarTop10Accuracy=0.6068, over 6816.00 frames. ], tot_loss[loss=3.557, NarTop10Accuracy=0.605, over 5953.47 frames. ], batch size: 17, lr: 7.81e-03
2024-08-06 09:06:40,945 INFO [trainer.py:765] (1/8) Epoch 11, batch 1200, train_loss[loss=3.652, NarTop10Accuracy=0.5893, over 7503.00 frames. ], tot_loss[loss=3.564, NarTop10Accuracy=0.6038, over 5957.13 frames. ], batch size: 30, lr: 7.80e-03
2024-08-06 09:07:15,493 INFO [trainer.py:765] (1/8) Epoch 11, batch 1300, train_loss[loss=3.476, NarTop10Accuracy=0.6312, over 5133.00 frames. ], tot_loss[loss=3.574, NarTop10Accuracy=0.6016, over 6025.16 frames. ], batch size: 6, lr: 7.79e-03
2024-08-06 09:07:47,628 INFO [trainer.py:765] (1/8) Epoch 11, batch 1400, train_loss[loss=3.448, NarTop10Accuracy=0.6225, over 6134.00 frames. ], tot_loss[loss=3.584, NarTop10Accuracy=0.5998, over 6036.20 frames. ], batch size: 11, lr: 7.77e-03
2024-08-06 09:08:18,986 INFO [trainer.py:765] (1/8) Epoch 11, batch 1500, train_loss[loss=3.604, NarTop10Accuracy=0.6005, over 5806.00 frames. ], tot_loss[loss=3.588, NarTop10Accuracy=0.5989, over 5973.86 frames. ], batch size: 49, lr: 7.76e-03
2024-08-06 09:08:47,148 INFO [trainer.py:765] (1/8) Epoch 11, batch 1600, train_loss[loss=3.419, NarTop10Accuracy=0.6405, over 7031.00 frames. ], tot_loss[loss=3.588, NarTop10Accuracy=0.599, over 5965.66 frames. ], batch size: 22, lr: 7.74e-03
2024-08-06 09:09:13,949 INFO [trainer.py:765] (1/8) Epoch 11, batch 1700, train_loss[loss=3.455, NarTop10Accuracy=0.6191, over 6304.00 frames. ], tot_loss[loss=3.581, NarTop10Accuracy=0.6002, over 5939.79 frames. ], batch size: 13, lr: 7.73e-03
2024-08-06 09:09:40,731 INFO [trainer.py:765] (1/8) Epoch 11, batch 1800, train_loss[loss=3.497, NarTop10Accuracy=0.6055, over 6902.00 frames. ], tot_loss[loss=3.593, NarTop10Accuracy=0.5982, over 5999.80 frames. ], batch size: 22, lr: 7.72e-03
2024-08-06 09:10:07,341 INFO [trainer.py:765] (1/8) Epoch 11, batch 1900, train_loss[loss=3.759, NarTop10Accuracy=0.5719, over 6534.00 frames. ], tot_loss[loss=3.601, NarTop10Accuracy=0.5966, over 6028.41 frames. ], batch size: 48, lr: 7.70e-03
2024-08-06 09:10:33,038 INFO [trainer.py:765] (1/8) Epoch 11, batch 2000, train_loss[loss=3.633, NarTop10Accuracy=0.5896, over 5739.00 frames. ], tot_loss[loss=3.597, NarTop10Accuracy=0.597, over 6019.53 frames. ], batch size: 48, lr: 7.69e-03
2024-08-06 09:10:58,440 INFO [trainer.py:765] (1/8) Epoch 11, batch 2100, train_loss[loss=3.145, NarTop10Accuracy=0.675, over 3977.00 frames. ], tot_loss[loss=3.579, NarTop10Accuracy=0.6007, over 5986.24 frames. ], batch size: 4, lr: 7.68e-03
2024-08-06 09:11:20,709 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 09:11:31,457 INFO [trainer.py:811] (1/8) Epoch 11, validation: loss=3.372, NarTop10Accuracy=0.6462, over 1907754.00 frames. 
2024-08-06 09:11:31,458 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30083MB
2024-08-06 09:11:31,930 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.411e+02 1.800e+02 1.966e+02 2.160e+02 4.000e+02, threshold=3.933e+02, percent-clipped=0.1
2024-08-06 09:11:34,519 INFO [trainer.py:765] (1/8) Epoch 11, batch 2200, train_loss[loss=3.545, NarTop10Accuracy=0.6097, over 7112.00 frames. ], tot_loss[loss=3.584, NarTop10Accuracy=0.6, over 6019.35 frames. ], batch size: 30, lr: 7.66e-03
2024-08-06 09:11:59,940 INFO [trainer.py:765] (1/8) Epoch 11, batch 2300, train_loss[loss=3.587, NarTop10Accuracy=0.5965, over 5891.00 frames. ], tot_loss[loss=3.592, NarTop10Accuracy=0.5985, over 6061.59 frames. ], batch size: 9, lr: 7.65e-03
2024-08-06 09:12:24,696 INFO [trainer.py:765] (1/8) Epoch 11, batch 2400, train_loss[loss=3.913, NarTop10Accuracy=0.5394, over 6056.00 frames. ], tot_loss[loss=3.607, NarTop10Accuracy=0.5954, over 5889.79 frames. ], batch size: 51, lr: 7.64e-03
2024-08-06 09:12:47,879 INFO [trainer.py:765] (1/8) Epoch 11, batch 2500, train_loss[loss=3.569, NarTop10Accuracy=0.596, over 5115.00 frames. ], tot_loss[loss=3.584, NarTop10Accuracy=0.6005, over 5530.31 frames. ], batch size: 6, lr: 7.62e-03
2024-08-06 09:13:09,242 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 09:14:12,279 INFO [trainer.py:765] (1/8) Epoch 12, batch 100, train_loss[loss=3.4, NarTop10Accuracy=0.6417, over 7147.00 frames. ], tot_loss[loss=3.536, NarTop10Accuracy=0.6112, over 2364.65 frames. ], batch size: 30, lr: 7.29e-03
2024-08-06 09:14:48,096 INFO [trainer.py:765] (1/8) Epoch 12, batch 200, train_loss[loss=3.461, NarTop10Accuracy=0.6259, over 6858.00 frames. ], tot_loss[loss=3.515, NarTop10Accuracy=0.6155, over 3853.84 frames. ], batch size: 17, lr: 7.28e-03
2024-08-06 09:15:20,021 INFO [trainer.py:765] (1/8) Epoch 12, batch 300, train_loss[loss=3.38, NarTop10Accuracy=0.6343, over 7112.00 frames. ], tot_loss[loss=3.5, NarTop10Accuracy=0.6179, over 4651.25 frames. ], batch size: 22, lr: 7.27e-03
2024-08-06 09:15:52,634 INFO [trainer.py:765] (1/8) Epoch 12, batch 400, train_loss[loss=3.473, NarTop10Accuracy=0.6313, over 5123.00 frames. ], tot_loss[loss=3.522, NarTop10Accuracy=0.6138, over 5116.28 frames. ], batch size: 7, lr: 7.25e-03
2024-08-06 09:16:26,433 INFO [trainer.py:765] (1/8) Epoch 12, batch 500, train_loss[loss=3.634, NarTop10Accuracy=0.5974, over 6078.00 frames. ], tot_loss[loss=3.522, NarTop10Accuracy=0.6134, over 5384.23 frames. ], batch size: 11, lr: 7.24e-03
2024-08-06 09:16:59,239 INFO [trainer.py:765] (1/8) Epoch 12, batch 600, train_loss[loss=3.267, NarTop10Accuracy=0.6511, over 5765.00 frames. ], tot_loss[loss=3.533, NarTop10Accuracy=0.6108, over 5655.65 frames. ], batch size: 9, lr: 7.23e-03
2024-08-06 09:17:36,318 INFO [trainer.py:765] (1/8) Epoch 12, batch 700, train_loss[loss=3.581, NarTop10Accuracy=0.598, over 5134.00 frames. ], tot_loss[loss=3.525, NarTop10Accuracy=0.6124, over 5729.81 frames. ], batch size: 6, lr: 7.22e-03
2024-08-06 09:18:07,753 INFO [trainer.py:765] (1/8) Epoch 12, batch 800, train_loss[loss=3.543, NarTop10Accuracy=0.597, over 4913.00 frames. ], tot_loss[loss=3.521, NarTop10Accuracy=0.613, over 5792.62 frames. ], batch size: 6, lr: 7.21e-03
2024-08-06 09:18:43,779 INFO [trainer.py:765] (1/8) Epoch 12, batch 900, train_loss[loss=3.745, NarTop10Accuracy=0.5549, over 6241.00 frames. ], tot_loss[loss=3.535, NarTop10Accuracy=0.6099, over 5803.74 frames. ], batch size: 13, lr: 7.19e-03
2024-08-06 09:19:17,689 INFO [trainer.py:765] (1/8) Epoch 12, batch 1000, train_loss[loss=3.693, NarTop10Accuracy=0.5841, over 6246.00 frames. ], tot_loss[loss=3.537, NarTop10Accuracy=0.6097, over 5922.97 frames. ], batch size: 13, lr: 7.18e-03
2024-08-06 09:19:52,427 INFO [trainer.py:765] (1/8) Epoch 12, batch 1100, train_loss[loss=3.501, NarTop10Accuracy=0.6205, over 6883.00 frames. ], tot_loss[loss=3.537, NarTop10Accuracy=0.6095, over 5962.38 frames. ], batch size: 17, lr: 7.17e-03
2024-08-06 09:20:29,443 INFO [trainer.py:765] (1/8) Epoch 12, batch 1200, train_loss[loss=3.432, NarTop10Accuracy=0.6273, over 7494.00 frames. ], tot_loss[loss=3.547, NarTop10Accuracy=0.6076, over 5955.79 frames. ], batch size: 31, lr: 7.16e-03
2024-08-06 09:21:02,826 INFO [trainer.py:765] (1/8) Epoch 12, batch 1300, train_loss[loss=3.683, NarTop10Accuracy=0.591, over 5132.00 frames. ], tot_loss[loss=3.547, NarTop10Accuracy=0.6076, over 6021.71 frames. ], batch size: 6, lr: 7.15e-03
2024-08-06 09:21:36,981 INFO [trainer.py:765] (1/8) Epoch 12, batch 1400, train_loss[loss=3.335, NarTop10Accuracy=0.6382, over 5965.00 frames. ], tot_loss[loss=3.553, NarTop10Accuracy=0.6061, over 6034.25 frames. ], batch size: 11, lr: 7.13e-03
2024-08-06 09:22:09,920 INFO [trainer.py:765] (1/8) Epoch 12, batch 1500, train_loss[loss=3.619, NarTop10Accuracy=0.5926, over 5899.00 frames. ], tot_loss[loss=3.555, NarTop10Accuracy=0.6059, over 5965.54 frames. ], batch size: 49, lr: 7.12e-03
2024-08-06 09:22:38,026 INFO [trainer.py:765] (1/8) Epoch 12, batch 1600, train_loss[loss=3.656, NarTop10Accuracy=0.5866, over 7213.00 frames. ], tot_loss[loss=3.562, NarTop10Accuracy=0.6045, over 5956.03 frames. ], batch size: 22, lr: 7.11e-03
2024-08-06 09:22:39,859 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 09:22:49,889 INFO [trainer.py:811] (1/8) Epoch 12, validation: loss=3.364, NarTop10Accuracy=0.6481, over 1907754.00 frames. 
2024-08-06 09:22:49,889 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30083MB
2024-08-06 09:22:50,413 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.453e+02 1.796e+02 1.978e+02 2.176e+02 4.603e+02, threshold=3.957e+02, percent-clipped=0.2
2024-08-06 09:23:14,786 INFO [trainer.py:765] (1/8) Epoch 12, batch 1700, train_loss[loss=3.503, NarTop10Accuracy=0.6158, over 6808.00 frames. ], tot_loss[loss=3.562, NarTop10Accuracy=0.6042, over 5942.23 frames. ], batch size: 14, lr: 7.10e-03
2024-08-06 09:23:41,386 INFO [trainer.py:765] (1/8) Epoch 12, batch 1800, train_loss[loss=3.354, NarTop10Accuracy=0.6622, over 7136.00 frames. ], tot_loss[loss=3.554, NarTop10Accuracy=0.6059, over 6011.58 frames. ], batch size: 22, lr: 7.09e-03
2024-08-06 09:24:07,957 INFO [trainer.py:765] (1/8) Epoch 12, batch 1900, train_loss[loss=3.558, NarTop10Accuracy=0.6031, over 5980.00 frames. ], tot_loss[loss=3.56, NarTop10Accuracy=0.6047, over 6054.02 frames. ], batch size: 49, lr: 7.08e-03
2024-08-06 09:24:33,619 INFO [trainer.py:765] (1/8) Epoch 12, batch 2000, train_loss[loss=3.61, NarTop10Accuracy=0.5958, over 6079.00 frames. ], tot_loss[loss=3.565, NarTop10Accuracy=0.6035, over 6027.07 frames. ], batch size: 50, lr: 7.07e-03
2024-08-06 09:24:59,038 INFO [trainer.py:765] (1/8) Epoch 12, batch 2100, train_loss[loss=3.773, NarTop10Accuracy=0.5631, over 4803.00 frames. ], tot_loss[loss=3.569, NarTop10Accuracy=0.6033, over 6021.49 frames. ], batch size: 5, lr: 7.05e-03
2024-08-06 09:25:24,509 INFO [trainer.py:765] (1/8) Epoch 12, batch 2200, train_loss[loss=3.422, NarTop10Accuracy=0.6288, over 7329.00 frames. ], tot_loss[loss=3.57, NarTop10Accuracy=0.6027, over 6056.22 frames. ], batch size: 30, lr: 7.04e-03
2024-08-06 09:25:49,927 INFO [trainer.py:765] (1/8) Epoch 12, batch 2300, train_loss[loss=3.713, NarTop10Accuracy=0.5691, over 5601.00 frames. ], tot_loss[loss=3.578, NarTop10Accuracy=0.6012, over 6071.65 frames. ], batch size: 9, lr: 7.03e-03
2024-08-06 09:26:14,656 INFO [trainer.py:765] (1/8) Epoch 12, batch 2400, train_loss[loss=3.454, NarTop10Accuracy=0.6311, over 5248.00 frames. ], tot_loss[loss=3.58, NarTop10Accuracy=0.6014, over 5890.08 frames. ], batch size: 7, lr: 7.02e-03
2024-08-06 09:26:38,155 INFO [trainer.py:765] (1/8) Epoch 12, batch 2500, train_loss[loss=3.55, NarTop10Accuracy=0.5992, over 4968.00 frames. ], tot_loss[loss=3.56, NarTop10Accuracy=0.6054, over 5547.67 frames. ], batch size: 6, lr: 7.01e-03
2024-08-06 09:26:59,493 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 09:28:03,611 INFO [trainer.py:765] (1/8) Epoch 13, batch 100, train_loss[loss=3.518, NarTop10Accuracy=0.6167, over 7129.00 frames. ], tot_loss[loss=3.523, NarTop10Accuracy=0.6144, over 2364.61 frames. ], batch size: 30, lr: 6.72e-03
2024-08-06 09:28:36,906 INFO [trainer.py:765] (1/8) Epoch 13, batch 200, train_loss[loss=3.287, NarTop10Accuracy=0.6615, over 6979.00 frames. ], tot_loss[loss=3.512, NarTop10Accuracy=0.6159, over 3862.11 frames. ], batch size: 17, lr: 6.71e-03
2024-08-06 09:29:07,170 INFO [trainer.py:765] (1/8) Epoch 13, batch 300, train_loss[loss=3.313, NarTop10Accuracy=0.6539, over 7028.00 frames. ], tot_loss[loss=3.5, NarTop10Accuracy=0.6181, over 4661.41 frames. ], batch size: 22, lr: 6.70e-03
2024-08-06 09:29:41,039 INFO [trainer.py:765] (1/8) Epoch 13, batch 400, train_loss[loss=3.391, NarTop10Accuracy=0.6409, over 5704.00 frames. ], tot_loss[loss=3.484, NarTop10Accuracy=0.6209, over 5110.53 frames. ], batch size: 8, lr: 6.69e-03
2024-08-06 09:30:13,730 INFO [trainer.py:765] (1/8) Epoch 13, batch 500, train_loss[loss=3.801, NarTop10Accuracy=0.5469, over 6153.00 frames. ], tot_loss[loss=3.482, NarTop10Accuracy=0.6214, over 5383.58 frames. ], batch size: 11, lr: 6.68e-03
2024-08-06 09:30:47,198 INFO [trainer.py:765] (1/8) Epoch 13, batch 600, train_loss[loss=3.417, NarTop10Accuracy=0.6498, over 5824.00 frames. ], tot_loss[loss=3.489, NarTop10Accuracy=0.6203, over 5662.41 frames. ], batch size: 9, lr: 6.67e-03
2024-08-06 09:31:23,821 INFO [trainer.py:765] (1/8) Epoch 13, batch 700, train_loss[loss=3.496, NarTop10Accuracy=0.6243, over 4275.00 frames. ], tot_loss[loss=3.504, NarTop10Accuracy=0.6167, over 5740.82 frames. ], batch size: 5, lr: 6.66e-03
2024-08-06 09:31:58,208 INFO [trainer.py:765] (1/8) Epoch 13, batch 800, train_loss[loss=3.689, NarTop10Accuracy=0.5734, over 5051.00 frames. ], tot_loss[loss=3.509, NarTop10Accuracy=0.6157, over 5807.44 frames. ], batch size: 6, lr: 6.65e-03
2024-08-06 09:32:29,193 INFO [trainer.py:765] (1/8) Epoch 13, batch 900, train_loss[loss=3.406, NarTop10Accuracy=0.6374, over 6417.00 frames. ], tot_loss[loss=3.504, NarTop10Accuracy=0.6166, over 5817.60 frames. ], batch size: 13, lr: 6.64e-03
2024-08-06 09:33:03,133 INFO [trainer.py:765] (1/8) Epoch 13, batch 1000, train_loss[loss=3.497, NarTop10Accuracy=0.6168, over 6829.00 frames. ], tot_loss[loss=3.5, NarTop10Accuracy=0.6168, over 5921.44 frames. ], batch size: 14, lr: 6.63e-03
2024-08-06 09:33:14,219 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 09:33:24,525 INFO [trainer.py:811] (1/8) Epoch 13, validation: loss=3.389, NarTop10Accuracy=0.6428, over 1907754.00 frames. 
2024-08-06 09:33:24,525 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30083MB
2024-08-06 09:33:25,132 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.457e+02 1.794e+02 1.964e+02 2.145e+02 3.608e+02, threshold=3.929e+02, percent-clipped=0.0
2024-08-06 09:33:51,714 INFO [trainer.py:765] (1/8) Epoch 13, batch 1100, train_loss[loss=3.597, NarTop10Accuracy=0.604, over 6848.00 frames. ], tot_loss[loss=3.526, NarTop10Accuracy=0.6117, over 5947.74 frames. ], batch size: 17, lr: 6.62e-03
2024-08-06 09:34:25,485 INFO [trainer.py:765] (1/8) Epoch 13, batch 1200, train_loss[loss=3.663, NarTop10Accuracy=0.5839, over 7203.00 frames. ], tot_loss[loss=3.52, NarTop10Accuracy=0.613, over 5958.10 frames. ], batch size: 30, lr: 6.61e-03
2024-08-06 09:35:05,085 INFO [trainer.py:765] (1/8) Epoch 13, batch 1300, train_loss[loss=3.49, NarTop10Accuracy=0.6167, over 5039.00 frames. ], tot_loss[loss=3.518, NarTop10Accuracy=0.6132, over 6012.41 frames. ], batch size: 6, lr: 6.60e-03
2024-08-06 09:35:36,404 INFO [trainer.py:765] (1/8) Epoch 13, batch 1400, train_loss[loss=3.442, NarTop10Accuracy=0.627, over 6242.00 frames. ], tot_loss[loss=3.535, NarTop10Accuracy=0.6099, over 6038.05 frames. ], batch size: 11, lr: 6.59e-03
2024-08-06 09:36:07,320 INFO [trainer.py:765] (1/8) Epoch 13, batch 1500, train_loss[loss=3.668, NarTop10Accuracy=0.5843, over 6087.00 frames. ], tot_loss[loss=3.539, NarTop10Accuracy=0.6088, over 5971.40 frames. ], batch size: 49, lr: 6.58e-03
2024-08-06 09:36:35,389 INFO [trainer.py:765] (1/8) Epoch 13, batch 1600, train_loss[loss=3.663, NarTop10Accuracy=0.5846, over 7288.00 frames. ], tot_loss[loss=3.541, NarTop10Accuracy=0.6088, over 5966.93 frames. ], batch size: 22, lr: 6.57e-03
2024-08-06 09:37:02,143 INFO [trainer.py:765] (1/8) Epoch 13, batch 1700, train_loss[loss=3.531, NarTop10Accuracy=0.6144, over 6324.00 frames. ], tot_loss[loss=3.538, NarTop10Accuracy=0.6096, over 5938.59 frames. ], batch size: 13, lr: 6.56e-03
2024-08-06 09:37:28,778 INFO [trainer.py:765] (1/8) Epoch 13, batch 1800, train_loss[loss=3.466, NarTop10Accuracy=0.6239, over 7300.00 frames. ], tot_loss[loss=3.53, NarTop10Accuracy=0.6114, over 5999.74 frames. ], batch size: 22, lr: 6.55e-03
2024-08-06 09:37:55,386 INFO [trainer.py:765] (1/8) Epoch 13, batch 1900, train_loss[loss=3.515, NarTop10Accuracy=0.614, over 5859.00 frames. ], tot_loss[loss=3.547, NarTop10Accuracy=0.6083, over 6031.79 frames. ], batch size: 50, lr: 6.54e-03
2024-08-06 09:38:21,123 INFO [trainer.py:765] (1/8) Epoch 13, batch 2000, train_loss[loss=3.54, NarTop10Accuracy=0.6021, over 5429.00 frames. ], tot_loss[loss=3.546, NarTop10Accuracy=0.6085, over 5998.24 frames. ], batch size: 49, lr: 6.53e-03
2024-08-06 09:38:49,691 INFO [trainer.py:765] (1/8) Epoch 13, batch 2100, train_loss[loss=3.38, NarTop10Accuracy=0.65, over 3874.00 frames. ], tot_loss[loss=3.546, NarTop10Accuracy=0.6086, over 5981.51 frames. ], batch size: 4, lr: 6.52e-03
2024-08-06 09:39:15,107 INFO [trainer.py:765] (1/8) Epoch 13, batch 2200, train_loss[loss=3.619, NarTop10Accuracy=0.5941, over 6903.00 frames. ], tot_loss[loss=3.549, NarTop10Accuracy=0.6075, over 6009.30 frames. ], batch size: 30, lr: 6.51e-03
2024-08-06 09:39:40,618 INFO [trainer.py:765] (1/8) Epoch 13, batch 2300, train_loss[loss=3.482, NarTop10Accuracy=0.6158, over 5696.00 frames. ], tot_loss[loss=3.546, NarTop10Accuracy=0.6079, over 6051.56 frames. ], batch size: 9, lr: 6.50e-03
2024-08-06 09:40:05,343 INFO [trainer.py:765] (1/8) Epoch 13, batch 2400, train_loss[loss=3.311, NarTop10Accuracy=0.6553, over 5292.00 frames. ], tot_loss[loss=3.555, NarTop10Accuracy=0.606, over 5864.33 frames. ], batch size: 7, lr: 6.49e-03
2024-08-06 09:40:28,767 INFO [trainer.py:765] (1/8) Epoch 13, batch 2500, train_loss[loss=3.33, NarTop10Accuracy=0.6604, over 4220.00 frames. ], tot_loss[loss=3.538, NarTop10Accuracy=0.6091, over 5536.44 frames. ], batch size: 5, lr: 6.48e-03
2024-08-06 09:40:50,382 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 09:41:48,980 INFO [trainer.py:765] (1/8) Epoch 14, batch 100, train_loss[loss=3.42, NarTop10Accuracy=0.6413, over 6956.00 frames. ], tot_loss[loss=3.464, NarTop10Accuracy=0.6254, over 2376.19 frames. ], batch size: 30, lr: 6.24e-03
2024-08-06 09:42:22,938 INFO [trainer.py:765] (1/8) Epoch 14, batch 200, train_loss[loss=3.758, NarTop10Accuracy=0.5647, over 6979.00 frames. ], tot_loss[loss=3.46, NarTop10Accuracy=0.6261, over 3874.03 frames. ], batch size: 17, lr: 6.23e-03
2024-08-06 09:42:58,414 INFO [trainer.py:765] (1/8) Epoch 14, batch 300, train_loss[loss=3.64, NarTop10Accuracy=0.582, over 7098.00 frames. ], tot_loss[loss=3.487, NarTop10Accuracy=0.6202, over 4690.39 frames. ], batch size: 22, lr: 6.22e-03
2024-08-06 09:43:30,439 INFO [trainer.py:765] (1/8) Epoch 14, batch 400, train_loss[loss=3.335, NarTop10Accuracy=0.6444, over 5121.00 frames. ], tot_loss[loss=3.483, NarTop10Accuracy=0.6205, over 5133.91 frames. ], batch size: 7, lr: 6.21e-03
2024-08-06 09:43:42,487 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 09:43:53,651 INFO [trainer.py:811] (1/8) Epoch 14, validation: loss=3.321, NarTop10Accuracy=0.6566, over 1907754.00 frames. 
2024-08-06 09:43:53,652 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30083MB
2024-08-06 09:43:54,212 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.427e+02 1.805e+02 1.968e+02 2.158e+02 4.264e+02, threshold=3.936e+02, percent-clipped=0.2
2024-08-06 09:44:11,700 INFO [trainer.py:765] (1/8) Epoch 14, batch 500, train_loss[loss=3.356, NarTop10Accuracy=0.6567, over 6256.00 frames. ], tot_loss[loss=3.481, NarTop10Accuracy=0.6211, over 5430.65 frames. ], batch size: 11, lr: 6.20e-03
2024-08-06 09:44:47,166 INFO [trainer.py:765] (1/8) Epoch 14, batch 600, train_loss[loss=3.876, NarTop10Accuracy=0.5351, over 5897.00 frames. ], tot_loss[loss=3.471, NarTop10Accuracy=0.623, over 5680.10 frames. ], batch size: 9, lr: 6.19e-03
2024-08-06 09:45:19,803 INFO [trainer.py:765] (1/8) Epoch 14, batch 700, train_loss[loss=3.741, NarTop10Accuracy=0.5736, over 5161.00 frames. ], tot_loss[loss=3.475, NarTop10Accuracy=0.623, over 5755.15 frames. ], batch size: 6, lr: 6.18e-03
2024-08-06 09:45:58,435 INFO [trainer.py:765] (1/8) Epoch 14, batch 800, train_loss[loss=3.282, NarTop10Accuracy=0.6561, over 4985.00 frames. ], tot_loss[loss=3.49, NarTop10Accuracy=0.6199, over 5791.76 frames. ], batch size: 6, lr: 6.17e-03
2024-08-06 09:46:35,420 INFO [trainer.py:765] (1/8) Epoch 14, batch 900, train_loss[loss=3.691, NarTop10Accuracy=0.5768, over 6665.00 frames. ], tot_loss[loss=3.491, NarTop10Accuracy=0.6193, over 5838.34 frames. ], batch size: 14, lr: 6.17e-03
2024-08-06 09:47:08,400 INFO [trainer.py:765] (1/8) Epoch 14, batch 1000, train_loss[loss=3.474, NarTop10Accuracy=0.618, over 6192.00 frames. ], tot_loss[loss=3.492, NarTop10Accuracy=0.6195, over 5936.96 frames. ], batch size: 13, lr: 6.16e-03
2024-08-06 09:47:47,663 INFO [trainer.py:765] (1/8) Epoch 14, batch 1100, train_loss[loss=3.499, NarTop10Accuracy=0.6206, over 6964.00 frames. ], tot_loss[loss=3.497, NarTop10Accuracy=0.6183, over 5969.26 frames. ], batch size: 17, lr: 6.15e-03
2024-08-06 09:48:23,500 INFO [trainer.py:765] (1/8) Epoch 14, batch 1200, train_loss[loss=3.365, NarTop10Accuracy=0.6419, over 7419.00 frames. ], tot_loss[loss=3.492, NarTop10Accuracy=0.6183, over 5966.05 frames. ], batch size: 31, lr: 6.14e-03
2024-08-06 09:48:57,971 INFO [trainer.py:765] (1/8) Epoch 14, batch 1300, train_loss[loss=3.384, NarTop10Accuracy=0.6521, over 4917.00 frames. ], tot_loss[loss=3.496, NarTop10Accuracy=0.6181, over 6017.36 frames. ], batch size: 6, lr: 6.13e-03
2024-08-06 09:49:30,234 INFO [trainer.py:765] (1/8) Epoch 14, batch 1400, train_loss[loss=3.346, NarTop10Accuracy=0.6378, over 6236.00 frames. ], tot_loss[loss=3.515, NarTop10Accuracy=0.6139, over 6041.65 frames. ], batch size: 11, lr: 6.12e-03
2024-08-06 09:50:07,531 INFO [trainer.py:765] (1/8) Epoch 14, batch 1500, train_loss[loss=3.593, NarTop10Accuracy=0.5987, over 6034.00 frames. ], tot_loss[loss=3.522, NarTop10Accuracy=0.6122, over 5978.59 frames. ], batch size: 49, lr: 6.11e-03
2024-08-06 09:50:35,637 INFO [trainer.py:765] (1/8) Epoch 14, batch 1600, train_loss[loss=3.408, NarTop10Accuracy=0.633, over 7194.00 frames. ], tot_loss[loss=3.519, NarTop10Accuracy=0.6131, over 5966.05 frames. ], batch size: 22, lr: 6.10e-03
2024-08-06 09:51:02,378 INFO [trainer.py:765] (1/8) Epoch 14, batch 1700, train_loss[loss=3.455, NarTop10Accuracy=0.6339, over 6193.00 frames. ], tot_loss[loss=3.517, NarTop10Accuracy=0.6138, over 5955.07 frames. ], batch size: 13, lr: 6.10e-03
2024-08-06 09:51:28,994 INFO [trainer.py:765] (1/8) Epoch 14, batch 1800, train_loss[loss=3.541, NarTop10Accuracy=0.6101, over 7131.00 frames. ], tot_loss[loss=3.502, NarTop10Accuracy=0.6168, over 6026.09 frames. ], batch size: 22, lr: 6.09e-03
2024-08-06 09:51:55,729 INFO [trainer.py:765] (1/8) Epoch 14, batch 1900, train_loss[loss=3.809, NarTop10Accuracy=0.559, over 6496.00 frames. ], tot_loss[loss=3.514, NarTop10Accuracy=0.6143, over 6061.54 frames. ], batch size: 48, lr: 6.08e-03
2024-08-06 09:52:21,503 INFO [trainer.py:765] (1/8) Epoch 14, batch 2000, train_loss[loss=3.499, NarTop10Accuracy=0.62, over 5931.00 frames. ], tot_loss[loss=3.514, NarTop10Accuracy=0.6142, over 6031.81 frames. ], batch size: 48, lr: 6.07e-03
2024-08-06 09:52:47,011 INFO [trainer.py:765] (1/8) Epoch 14, batch 2100, train_loss[loss=3.324, NarTop10Accuracy=0.6572, over 3943.00 frames. ], tot_loss[loss=3.509, NarTop10Accuracy=0.6151, over 6021.13 frames. ], batch size: 4, lr: 6.06e-03
2024-08-06 09:53:12,480 INFO [trainer.py:765] (1/8) Epoch 14, batch 2200, train_loss[loss=3.496, NarTop10Accuracy=0.6236, over 7201.00 frames. ], tot_loss[loss=3.513, NarTop10Accuracy=0.614, over 6040.77 frames. ], batch size: 31, lr: 6.05e-03
2024-08-06 09:53:37,976 INFO [trainer.py:765] (1/8) Epoch 14, batch 2300, train_loss[loss=3.522, NarTop10Accuracy=0.6112, over 5700.00 frames. ], tot_loss[loss=3.527, NarTop10Accuracy=0.6113, over 6061.08 frames. ], batch size: 9, lr: 6.05e-03
2024-08-06 09:54:02,717 INFO [trainer.py:765] (1/8) Epoch 14, batch 2400, train_loss[loss=3.773, NarTop10Accuracy=0.5647, over 5122.00 frames. ], tot_loss[loss=3.526, NarTop10Accuracy=0.6118, over 5893.70 frames. ], batch size: 7, lr: 6.04e-03
2024-08-06 09:54:12,820 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 09:54:24,304 INFO [trainer.py:811] (1/8) Epoch 14, validation: loss=3.364, NarTop10Accuracy=0.6477, over 1907754.00 frames. 
2024-08-06 09:54:24,304 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30083MB
2024-08-06 09:54:24,752 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.448e+02 1.815e+02 1.970e+02 2.165e+02 3.684e+02, threshold=3.939e+02, percent-clipped=0.0
2024-08-06 09:54:37,619 INFO [trainer.py:765] (1/8) Epoch 14, batch 2500, train_loss[loss=3.495, NarTop10Accuracy=0.6064, over 4965.00 frames. ], tot_loss[loss=3.502, NarTop10Accuracy=0.6161, over 5538.24 frames. ], batch size: 6, lr: 6.03e-03
2024-08-06 09:54:58,950 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 09:56:03,097 INFO [trainer.py:765] (1/8) Epoch 15, batch 100, train_loss[loss=3.404, NarTop10Accuracy=0.6402, over 7049.00 frames. ], tot_loss[loss=3.46, NarTop10Accuracy=0.6272, over 2360.32 frames. ], batch size: 31, lr: 5.81e-03
2024-08-06 09:56:35,980 INFO [trainer.py:765] (1/8) Epoch 15, batch 200, train_loss[loss=3.197, NarTop10Accuracy=0.6824, over 6944.00 frames. ], tot_loss[loss=3.455, NarTop10Accuracy=0.6276, over 3855.32 frames. ], batch size: 17, lr: 5.81e-03
2024-08-06 09:57:07,653 INFO [trainer.py:765] (1/8) Epoch 15, batch 300, train_loss[loss=3.226, NarTop10Accuracy=0.6768, over 7218.00 frames. ], tot_loss[loss=3.458, NarTop10Accuracy=0.6273, over 4666.13 frames. ], batch size: 22, lr: 5.80e-03
2024-08-06 09:57:38,464 INFO [trainer.py:765] (1/8) Epoch 15, batch 400, train_loss[loss=3.645, NarTop10Accuracy=0.5928, over 5089.00 frames. ], tot_loss[loss=3.456, NarTop10Accuracy=0.6271, over 5111.58 frames. ], batch size: 7, lr: 5.79e-03
2024-08-06 09:58:12,235 INFO [trainer.py:765] (1/8) Epoch 15, batch 500, train_loss[loss=3.359, NarTop10Accuracy=0.648, over 6094.00 frames. ], tot_loss[loss=3.461, NarTop10Accuracy=0.6259, over 5386.84 frames. ], batch size: 11, lr: 5.78e-03
2024-08-06 09:58:47,543 INFO [trainer.py:765] (1/8) Epoch 15, batch 600, train_loss[loss=3.489, NarTop10Accuracy=0.6183, over 5805.00 frames. ], tot_loss[loss=3.471, NarTop10Accuracy=0.6236, over 5667.01 frames. ], batch size: 9, lr: 5.77e-03
2024-08-06 09:59:17,062 INFO [trainer.py:765] (1/8) Epoch 15, batch 700, train_loss[loss=3.169, NarTop10Accuracy=0.6914, over 5167.00 frames. ], tot_loss[loss=3.482, NarTop10Accuracy=0.621, over 5726.29 frames. ], batch size: 6, lr: 5.77e-03
2024-08-06 09:59:55,588 INFO [trainer.py:765] (1/8) Epoch 15, batch 800, train_loss[loss=3.718, NarTop10Accuracy=0.5739, over 4981.00 frames. ], tot_loss[loss=3.481, NarTop10Accuracy=0.6209, over 5804.10 frames. ], batch size: 6, lr: 5.76e-03
2024-08-06 10:00:32,024 INFO [trainer.py:765] (1/8) Epoch 15, batch 900, train_loss[loss=3.628, NarTop10Accuracy=0.5956, over 6731.00 frames. ], tot_loss[loss=3.475, NarTop10Accuracy=0.6217, over 5815.22 frames. ], batch size: 14, lr: 5.75e-03
2024-08-06 10:01:05,538 INFO [trainer.py:765] (1/8) Epoch 15, batch 1000, train_loss[loss=3.274, NarTop10Accuracy=0.6627, over 6593.00 frames. ], tot_loss[loss=3.471, NarTop10Accuracy=0.6228, over 5932.81 frames. ], batch size: 14, lr: 5.74e-03
2024-08-06 10:01:45,154 INFO [trainer.py:765] (1/8) Epoch 15, batch 1100, train_loss[loss=3.284, NarTop10Accuracy=0.65, over 6799.00 frames. ], tot_loss[loss=3.491, NarTop10Accuracy=0.6186, over 5954.46 frames. ], batch size: 17, lr: 5.74e-03
2024-08-06 10:02:18,756 INFO [trainer.py:765] (1/8) Epoch 15, batch 1200, train_loss[loss=3.822, NarTop10Accuracy=0.5506, over 7269.00 frames. ], tot_loss[loss=3.485, NarTop10Accuracy=0.6201, over 5949.61 frames. ], batch size: 30, lr: 5.73e-03
2024-08-06 10:02:51,921 INFO [trainer.py:765] (1/8) Epoch 15, batch 1300, train_loss[loss=3.193, NarTop10Accuracy=0.6569, over 4351.00 frames. ], tot_loss[loss=3.488, NarTop10Accuracy=0.6196, over 6020.30 frames. ], batch size: 5, lr: 5.72e-03
2024-08-06 10:03:25,435 INFO [trainer.py:765] (1/8) Epoch 15, batch 1400, train_loss[loss=3.598, NarTop10Accuracy=0.5952, over 6186.00 frames. ], tot_loss[loss=3.505, NarTop10Accuracy=0.6162, over 6051.83 frames. ], batch size: 11, lr: 5.71e-03
2024-08-06 10:03:59,041 INFO [trainer.py:765] (1/8) Epoch 15, batch 1500, train_loss[loss=3.45, NarTop10Accuracy=0.6203, over 6175.00 frames. ], tot_loss[loss=3.494, NarTop10Accuracy=0.6183, over 5971.58 frames. ], batch size: 49, lr: 5.71e-03
2024-08-06 10:04:27,106 INFO [trainer.py:765] (1/8) Epoch 15, batch 1600, train_loss[loss=3.795, NarTop10Accuracy=0.5601, over 7394.00 frames. ], tot_loss[loss=3.483, NarTop10Accuracy=0.6206, over 5964.28 frames. ], batch size: 22, lr: 5.70e-03
2024-08-06 10:04:53,907 INFO [trainer.py:765] (1/8) Epoch 15, batch 1700, train_loss[loss=3.782, NarTop10Accuracy=0.5549, over 6321.00 frames. ], tot_loss[loss=3.493, NarTop10Accuracy=0.6187, over 5947.46 frames. ], batch size: 13, lr: 5.69e-03
2024-08-06 10:05:20,729 INFO [trainer.py:765] (1/8) Epoch 15, batch 1800, train_loss[loss=3.792, NarTop10Accuracy=0.559, over 7027.00 frames. ], tot_loss[loss=3.507, NarTop10Accuracy=0.6161, over 6014.90 frames. ], batch size: 22, lr: 5.68e-03
2024-08-06 10:05:37,266 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 10:05:47,411 INFO [trainer.py:811] (1/8) Epoch 15, validation: loss=3.325, NarTop10Accuracy=0.6551, over 1907754.00 frames. 
2024-08-06 10:05:47,412 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30083MB
2024-08-06 10:05:47,919 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.405e+02 1.835e+02 1.986e+02 2.156e+02 4.531e+02, threshold=3.972e+02, percent-clipped=0.1
2024-08-06 10:05:57,569 INFO [trainer.py:765] (1/8) Epoch 15, batch 1900, train_loss[loss=3.834, NarTop10Accuracy=0.5619, over 6350.00 frames. ], tot_loss[loss=3.515, NarTop10Accuracy=0.6142, over 6048.00 frames. ], batch size: 50, lr: 5.68e-03
2024-08-06 10:06:23,372 INFO [trainer.py:765] (1/8) Epoch 15, batch 2000, train_loss[loss=3.544, NarTop10Accuracy=0.6188, over 5895.00 frames. ], tot_loss[loss=3.501, NarTop10Accuracy=0.6176, over 6011.31 frames. ], batch size: 49, lr: 5.67e-03
2024-08-06 10:06:48,758 INFO [trainer.py:765] (1/8) Epoch 15, batch 2100, train_loss[loss=3.291, NarTop10Accuracy=0.6667, over 3894.00 frames. ], tot_loss[loss=3.504, NarTop10Accuracy=0.6165, over 5995.84 frames. ], batch size: 4, lr: 5.66e-03
2024-08-06 10:07:14,171 INFO [trainer.py:765] (1/8) Epoch 15, batch 2200, train_loss[loss=3.281, NarTop10Accuracy=0.661, over 7284.00 frames. ], tot_loss[loss=3.498, NarTop10Accuracy=0.6178, over 6042.90 frames. ], batch size: 30, lr: 5.65e-03
2024-08-06 10:07:39,629 INFO [trainer.py:765] (1/8) Epoch 15, batch 2300, train_loss[loss=3.376, NarTop10Accuracy=0.6329, over 5787.00 frames. ], tot_loss[loss=3.511, NarTop10Accuracy=0.615, over 6069.21 frames. ], batch size: 9, lr: 5.65e-03
2024-08-06 10:08:04,361 INFO [trainer.py:765] (1/8) Epoch 15, batch 2400, train_loss[loss=3.737, NarTop10Accuracy=0.5682, over 5239.00 frames. ], tot_loss[loss=3.51, NarTop10Accuracy=0.615, over 5882.56 frames. ], batch size: 7, lr: 5.64e-03
2024-08-06 10:08:27,713 INFO [trainer.py:765] (1/8) Epoch 15, batch 2500, train_loss[loss=3.62, NarTop10Accuracy=0.5948, over 4958.00 frames. ], tot_loss[loss=3.509, NarTop10Accuracy=0.615, over 5522.27 frames. ], batch size: 6, lr: 5.63e-03
2024-08-06 10:08:49,095 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 10:09:44,183 INFO [trainer.py:765] (1/8) Epoch 16, batch 100, train_loss[loss=3.548, NarTop10Accuracy=0.6103, over 6856.00 frames. ], tot_loss[loss=3.457, NarTop10Accuracy=0.6274, over 2365.63 frames. ], batch size: 30, lr: 5.44e-03
2024-08-06 10:10:23,207 INFO [trainer.py:765] (1/8) Epoch 16, batch 200, train_loss[loss=3.391, NarTop10Accuracy=0.6396, over 6771.00 frames. ], tot_loss[loss=3.446, NarTop10Accuracy=0.6294, over 3854.58 frames. ], batch size: 17, lr: 5.44e-03
2024-08-06 10:10:58,841 INFO [trainer.py:765] (1/8) Epoch 16, batch 300, train_loss[loss=3.293, NarTop10Accuracy=0.6652, over 7255.00 frames. ], tot_loss[loss=3.447, NarTop10Accuracy=0.6287, over 4665.14 frames. ], batch size: 22, lr: 5.43e-03
2024-08-06 10:11:29,593 INFO [trainer.py:765] (1/8) Epoch 16, batch 400, train_loss[loss=3.379, NarTop10Accuracy=0.6461, over 5192.00 frames. ], tot_loss[loss=3.454, NarTop10Accuracy=0.6272, over 5117.03 frames. ], batch size: 7, lr: 5.42e-03
2024-08-06 10:12:02,297 INFO [trainer.py:765] (1/8) Epoch 16, batch 500, train_loss[loss=3.511, NarTop10Accuracy=0.6241, over 6110.00 frames. ], tot_loss[loss=3.45, NarTop10Accuracy=0.6275, over 5411.05 frames. ], batch size: 11, lr: 5.42e-03
2024-08-06 10:12:42,340 INFO [trainer.py:765] (1/8) Epoch 16, batch 600, train_loss[loss=3.366, NarTop10Accuracy=0.6422, over 5795.00 frames. ], tot_loss[loss=3.448, NarTop10Accuracy=0.6285, over 5696.32 frames. ], batch size: 9, lr: 5.41e-03
2024-08-06 10:13:13,948 INFO [trainer.py:765] (1/8) Epoch 16, batch 700, train_loss[loss=3.135, NarTop10Accuracy=0.6862, over 4996.00 frames. ], tot_loss[loss=3.462, NarTop10Accuracy=0.6259, over 5754.62 frames. ], batch size: 6, lr: 5.40e-03
2024-08-06 10:13:46,283 INFO [trainer.py:765] (1/8) Epoch 16, batch 800, train_loss[loss=3.869, NarTop10Accuracy=0.5505, over 5279.00 frames. ], tot_loss[loss=3.457, NarTop10Accuracy=0.6268, over 5808.71 frames. ], batch size: 6, lr: 5.40e-03
2024-08-06 10:14:23,293 INFO [trainer.py:765] (1/8) Epoch 16, batch 900, train_loss[loss=3.292, NarTop10Accuracy=0.6439, over 6234.00 frames. ], tot_loss[loss=3.446, NarTop10Accuracy=0.6286, over 5822.89 frames. ], batch size: 13, lr: 5.39e-03
2024-08-06 10:15:00,057 INFO [trainer.py:765] (1/8) Epoch 16, batch 1000, train_loss[loss=3.591, NarTop10Accuracy=0.6019, over 6387.00 frames. ], tot_loss[loss=3.468, NarTop10Accuracy=0.6242, over 5919.20 frames. ], batch size: 13, lr: 5.38e-03
2024-08-06 10:15:30,507 INFO [trainer.py:765] (1/8) Epoch 16, batch 1100, train_loss[loss=3.328, NarTop10Accuracy=0.6468, over 7029.00 frames. ], tot_loss[loss=3.48, NarTop10Accuracy=0.6213, over 5933.42 frames. ], batch size: 17, lr: 5.38e-03
2024-08-06 10:16:11,382 INFO [trainer.py:765] (1/8) Epoch 16, batch 1200, train_loss[loss=3.558, NarTop10Accuracy=0.6168, over 6906.00 frames. ], tot_loss[loss=3.481, NarTop10Accuracy=0.6214, over 5933.14 frames. ], batch size: 30, lr: 5.37e-03
2024-08-06 10:16:39,395 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 10:16:49,676 INFO [trainer.py:811] (1/8) Epoch 16, validation: loss=3.375, NarTop10Accuracy=0.6455, over 1907754.00 frames. 
2024-08-06 10:16:49,676 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30083MB
2024-08-06 10:16:52,482 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.406e+02 1.814e+02 1.975e+02 2.151e+02 4.776e+02, threshold=3.950e+02, percent-clipped=0.2
2024-08-06 10:16:58,042 INFO [trainer.py:765] (1/8) Epoch 16, batch 1300, train_loss[loss=3.879, NarTop10Accuracy=0.541, over 5176.00 frames. ], tot_loss[loss=3.479, NarTop10Accuracy=0.6219, over 6002.22 frames. ], batch size: 6, lr: 5.36e-03
2024-08-06 10:17:29,376 INFO [trainer.py:765] (1/8) Epoch 16, batch 1400, train_loss[loss=3.312, NarTop10Accuracy=0.6487, over 6154.00 frames. ], tot_loss[loss=3.476, NarTop10Accuracy=0.6227, over 6011.92 frames. ], batch size: 11, lr: 5.36e-03
2024-08-06 10:18:02,354 INFO [trainer.py:765] (1/8) Epoch 16, batch 1500, train_loss[loss=3.508, NarTop10Accuracy=0.6186, over 6364.00 frames. ], tot_loss[loss=3.477, NarTop10Accuracy=0.6226, over 5971.58 frames. ], batch size: 49, lr: 5.35e-03
2024-08-06 10:18:30,469 INFO [trainer.py:765] (1/8) Epoch 16, batch 1600, train_loss[loss=3.537, NarTop10Accuracy=0.6006, over 6965.00 frames. ], tot_loss[loss=3.49, NarTop10Accuracy=0.62, over 5959.83 frames. ], batch size: 22, lr: 5.34e-03
2024-08-06 10:18:57,272 INFO [trainer.py:765] (1/8) Epoch 16, batch 1700, train_loss[loss=3.697, NarTop10Accuracy=0.5839, over 6264.00 frames. ], tot_loss[loss=3.475, NarTop10Accuracy=0.6227, over 5948.56 frames. ], batch size: 13, lr: 5.34e-03
2024-08-06 10:19:23,978 INFO [trainer.py:765] (1/8) Epoch 16, batch 1800, train_loss[loss=3.766, NarTop10Accuracy=0.5714, over 7109.00 frames. ], tot_loss[loss=3.487, NarTop10Accuracy=0.6207, over 6012.80 frames. ], batch size: 22, lr: 5.33e-03
2024-08-06 10:19:50,772 INFO [trainer.py:765] (1/8) Epoch 16, batch 1900, train_loss[loss=3.779, NarTop10Accuracy=0.5682, over 6048.00 frames. ], tot_loss[loss=3.481, NarTop10Accuracy=0.6215, over 6045.10 frames. ], batch size: 49, lr: 5.32e-03
2024-08-06 10:20:16,602 INFO [trainer.py:765] (1/8) Epoch 16, batch 2000, train_loss[loss=3.55, NarTop10Accuracy=0.6061, over 5340.00 frames. ], tot_loss[loss=3.487, NarTop10Accuracy=0.6201, over 6008.99 frames. ], batch size: 48, lr: 5.32e-03
2024-08-06 10:20:42,160 INFO [trainer.py:765] (1/8) Epoch 16, batch 2100, train_loss[loss=3.806, NarTop10Accuracy=0.5579, over 3958.00 frames. ], tot_loss[loss=3.496, NarTop10Accuracy=0.6183, over 5993.76 frames. ], batch size: 4, lr: 5.31e-03
2024-08-06 10:21:07,651 INFO [trainer.py:765] (1/8) Epoch 16, batch 2200, train_loss[loss=3.289, NarTop10Accuracy=0.6532, over 7207.00 frames. ], tot_loss[loss=3.491, NarTop10Accuracy=0.6193, over 6032.20 frames. ], batch size: 30, lr: 5.30e-03
2024-08-06 10:21:36,082 INFO [trainer.py:765] (1/8) Epoch 16, batch 2300, train_loss[loss=3.692, NarTop10Accuracy=0.5732, over 5769.00 frames. ], tot_loss[loss=3.501, NarTop10Accuracy=0.617, over 6056.37 frames. ], batch size: 9, lr: 5.30e-03
2024-08-06 10:22:00,907 INFO [trainer.py:765] (1/8) Epoch 16, batch 2400, train_loss[loss=3.211, NarTop10Accuracy=0.6788, over 5077.00 frames. ], tot_loss[loss=3.499, NarTop10Accuracy=0.6177, over 5862.50 frames. ], batch size: 7, lr: 5.29e-03
2024-08-06 10:22:24,290 INFO [trainer.py:765] (1/8) Epoch 16, batch 2500, train_loss[loss=3.412, NarTop10Accuracy=0.6333, over 5070.00 frames. ], tot_loss[loss=3.47, NarTop10Accuracy=0.6236, over 5523.19 frames. ], batch size: 6, lr: 5.28e-03
2024-08-06 10:22:45,790 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 10:23:45,727 INFO [trainer.py:765] (1/8) Epoch 17, batch 100, train_loss[loss=3.515, NarTop10Accuracy=0.6085, over 7110.00 frames. ], tot_loss[loss=3.439, NarTop10Accuracy=0.6319, over 2379.03 frames. ], batch size: 30, lr: 5.12e-03
2024-08-06 10:24:19,033 INFO [trainer.py:765] (1/8) Epoch 17, batch 200, train_loss[loss=3.296, NarTop10Accuracy=0.6637, over 6907.00 frames. ], tot_loss[loss=3.42, NarTop10Accuracy=0.6341, over 3877.26 frames. ], batch size: 17, lr: 5.11e-03
2024-08-06 10:24:53,441 INFO [trainer.py:765] (1/8) Epoch 17, batch 300, train_loss[loss=3.542, NarTop10Accuracy=0.6062, over 6976.00 frames. ], tot_loss[loss=3.43, NarTop10Accuracy=0.6324, over 4676.53 frames. ], batch size: 22, lr: 5.10e-03
2024-08-06 10:25:28,013 INFO [trainer.py:765] (1/8) Epoch 17, batch 400, train_loss[loss=3.47, NarTop10Accuracy=0.6229, over 5196.00 frames. ], tot_loss[loss=3.434, NarTop10Accuracy=0.6313, over 5107.71 frames. ], batch size: 7, lr: 5.10e-03
2024-08-06 10:25:58,606 INFO [trainer.py:765] (1/8) Epoch 17, batch 500, train_loss[loss=3.393, NarTop10Accuracy=0.6296, over 6133.00 frames. ], tot_loss[loss=3.429, NarTop10Accuracy=0.6319, over 5386.09 frames. ], batch size: 11, lr: 5.09e-03
2024-08-06 10:26:29,756 INFO [trainer.py:765] (1/8) Epoch 17, batch 600, train_loss[loss=3.59, NarTop10Accuracy=0.6097, over 5786.00 frames. ], tot_loss[loss=3.43, NarTop10Accuracy=0.6318, over 5653.92 frames. ], batch size: 9, lr: 5.09e-03
2024-08-06 10:27:07,498 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 10:27:17,547 INFO [trainer.py:811] (1/8) Epoch 17, validation: loss=3.327, NarTop10Accuracy=0.6554, over 1907754.00 frames. 
2024-08-06 10:27:17,548 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30083MB
2024-08-06 10:27:18,066 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.474e+02 1.825e+02 1.985e+02 2.150e+02 4.169e+02, threshold=3.970e+02, percent-clipped=0.2
2024-08-06 10:27:18,071 INFO [trainer.py:765] (1/8) Epoch 17, batch 700, train_loss[loss=3.451, NarTop10Accuracy=0.6289, over 4985.00 frames. ], tot_loss[loss=3.454, NarTop10Accuracy=0.627, over 5733.05 frames. ], batch size: 6, lr: 5.08e-03
2024-08-06 10:27:49,841 INFO [trainer.py:765] (1/8) Epoch 17, batch 800, train_loss[loss=3.427, NarTop10Accuracy=0.6429, over 5260.00 frames. ], tot_loss[loss=3.449, NarTop10Accuracy=0.6283, over 5774.77 frames. ], batch size: 6, lr: 5.07e-03
2024-08-06 10:28:24,838 INFO [trainer.py:765] (1/8) Epoch 17, batch 900, train_loss[loss=3.295, NarTop10Accuracy=0.6632, over 6214.00 frames. ], tot_loss[loss=3.453, NarTop10Accuracy=0.627, over 5812.01 frames. ], batch size: 13, lr: 5.07e-03
2024-08-06 10:28:59,683 INFO [trainer.py:765] (1/8) Epoch 17, batch 1000, train_loss[loss=3.36, NarTop10Accuracy=0.6571, over 6617.00 frames. ], tot_loss[loss=3.452, NarTop10Accuracy=0.6272, over 5922.64 frames. ], batch size: 14, lr: 5.06e-03
2024-08-06 10:29:36,659 INFO [trainer.py:765] (1/8) Epoch 17, batch 1100, train_loss[loss=3.414, NarTop10Accuracy=0.637, over 6934.00 frames. ], tot_loss[loss=3.462, NarTop10Accuracy=0.625, over 5967.25 frames. ], batch size: 17, lr: 5.06e-03
2024-08-06 10:30:08,241 INFO [trainer.py:765] (1/8) Epoch 17, batch 1200, train_loss[loss=3.57, NarTop10Accuracy=0.6065, over 7246.00 frames. ], tot_loss[loss=3.462, NarTop10Accuracy=0.6248, over 5962.68 frames. ], batch size: 30, lr: 5.05e-03
2024-08-06 10:30:47,102 INFO [trainer.py:765] (1/8) Epoch 17, batch 1300, train_loss[loss=3.116, NarTop10Accuracy=0.6787, over 5042.00 frames. ], tot_loss[loss=3.464, NarTop10Accuracy=0.6243, over 6032.45 frames. ], batch size: 6, lr: 5.04e-03
2024-08-06 10:31:20,893 INFO [trainer.py:765] (1/8) Epoch 17, batch 1400, train_loss[loss=3.388, NarTop10Accuracy=0.6285, over 6066.00 frames. ], tot_loss[loss=3.471, NarTop10Accuracy=0.6231, over 6046.59 frames. ], batch size: 11, lr: 5.04e-03
2024-08-06 10:31:51,401 INFO [trainer.py:765] (1/8) Epoch 17, batch 1500, train_loss[loss=3.426, NarTop10Accuracy=0.6354, over 5838.00 frames. ], tot_loss[loss=3.461, NarTop10Accuracy=0.6254, over 5960.74 frames. ], batch size: 49, lr: 5.03e-03
2024-08-06 10:32:19,400 INFO [trainer.py:765] (1/8) Epoch 17, batch 1600, train_loss[loss=3.482, NarTop10Accuracy=0.6153, over 7054.00 frames. ], tot_loss[loss=3.466, NarTop10Accuracy=0.6242, over 5936.66 frames. ], batch size: 22, lr: 5.03e-03
2024-08-06 10:32:50,394 INFO [trainer.py:765] (1/8) Epoch 17, batch 1700, train_loss[loss=3.82, NarTop10Accuracy=0.5485, over 6252.00 frames. ], tot_loss[loss=3.487, NarTop10Accuracy=0.6203, over 5915.83 frames. ], batch size: 13, lr: 5.02e-03
2024-08-06 10:33:17,036 INFO [trainer.py:765] (1/8) Epoch 17, batch 1800, train_loss[loss=3.716, NarTop10Accuracy=0.57, over 7041.00 frames. ], tot_loss[loss=3.489, NarTop10Accuracy=0.6199, over 5984.11 frames. ], batch size: 22, lr: 5.02e-03
2024-08-06 10:33:43,596 INFO [trainer.py:765] (1/8) Epoch 17, batch 1900, train_loss[loss=3.724, NarTop10Accuracy=0.5606, over 5470.00 frames. ], tot_loss[loss=3.49, NarTop10Accuracy=0.6196, over 6017.72 frames. ], batch size: 51, lr: 5.01e-03
2024-08-06 10:34:09,287 INFO [trainer.py:765] (1/8) Epoch 17, batch 2000, train_loss[loss=3.893, NarTop10Accuracy=0.5358, over 6183.00 frames. ], tot_loss[loss=3.487, NarTop10Accuracy=0.6205, over 6006.14 frames. ], batch size: 52, lr: 5.00e-03
2024-08-06 10:34:34,801 INFO [trainer.py:765] (1/8) Epoch 17, batch 2100, train_loss[loss=3.209, NarTop10Accuracy=0.6614, over 3942.00 frames. ], tot_loss[loss=3.493, NarTop10Accuracy=0.6189, over 5985.91 frames. ], batch size: 4, lr: 5.00e-03
2024-08-06 10:35:00,245 INFO [trainer.py:765] (1/8) Epoch 17, batch 2200, train_loss[loss=3.226, NarTop10Accuracy=0.6668, over 7200.00 frames. ], tot_loss[loss=3.474, NarTop10Accuracy=0.6231, over 6038.85 frames. ], batch size: 31, lr: 4.99e-03
2024-08-06 10:35:25,733 INFO [trainer.py:765] (1/8) Epoch 17, batch 2300, train_loss[loss=3.377, NarTop10Accuracy=0.6413, over 5791.00 frames. ], tot_loss[loss=3.488, NarTop10Accuracy=0.6206, over 6068.41 frames. ], batch size: 9, lr: 4.99e-03
2024-08-06 10:35:50,526 INFO [trainer.py:765] (1/8) Epoch 17, batch 2400, train_loss[loss=3.735, NarTop10Accuracy=0.5546, over 5302.00 frames. ], tot_loss[loss=3.491, NarTop10Accuracy=0.6198, over 5881.53 frames. ], batch size: 7, lr: 4.98e-03
2024-08-06 10:36:14,105 INFO [trainer.py:765] (1/8) Epoch 17, batch 2500, train_loss[loss=3.8, NarTop10Accuracy=0.5577, over 5105.00 frames. ], tot_loss[loss=3.475, NarTop10Accuracy=0.6225, over 5537.33 frames. ], batch size: 6, lr: 4.98e-03
2024-08-06 10:36:35,793 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 10:37:32,051 INFO [trainer.py:765] (1/8) Epoch 18, batch 100, train_loss[loss=3.237, NarTop10Accuracy=0.6708, over 7162.00 frames. ], tot_loss[loss=3.417, NarTop10Accuracy=0.6353, over 2362.58 frames. ], batch size: 30, lr: 4.83e-03
2024-08-06 10:37:39,163 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 10:37:49,085 INFO [trainer.py:811] (1/8) Epoch 18, validation: loss=3.339, NarTop10Accuracy=0.6526, over 1907754.00 frames. 
2024-08-06 10:37:49,085 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30083MB
2024-08-06 10:37:49,685 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.466e+02 1.841e+02 1.993e+02 2.161e+02 3.871e+02, threshold=3.985e+02, percent-clipped=0.0
2024-08-06 10:38:18,144 INFO [trainer.py:765] (1/8) Epoch 18, batch 200, train_loss[loss=3.486, NarTop10Accuracy=0.6233, over 6986.00 frames. ], tot_loss[loss=3.425, NarTop10Accuracy=0.6332, over 3856.60 frames. ], batch size: 17, lr: 4.82e-03
2024-08-06 10:38:50,198 INFO [trainer.py:765] (1/8) Epoch 18, batch 300, train_loss[loss=3.582, NarTop10Accuracy=0.6083, over 7126.00 frames. ], tot_loss[loss=3.424, NarTop10Accuracy=0.6338, over 4669.24 frames. ], batch size: 22, lr: 4.81e-03
2024-08-06 10:39:23,744 INFO [trainer.py:765] (1/8) Epoch 18, batch 400, train_loss[loss=3.543, NarTop10Accuracy=0.6088, over 5206.00 frames. ], tot_loss[loss=3.427, NarTop10Accuracy=0.6334, over 5125.29 frames. ], batch size: 7, lr: 4.81e-03
2024-08-06 10:39:54,103 INFO [trainer.py:765] (1/8) Epoch 18, batch 500, train_loss[loss=3.286, NarTop10Accuracy=0.6529, over 6228.00 frames. ], tot_loss[loss=3.419, NarTop10Accuracy=0.6346, over 5413.21 frames. ], batch size: 11, lr: 4.80e-03
2024-08-06 10:40:28,526 INFO [trainer.py:765] (1/8) Epoch 18, batch 600, train_loss[loss=3.407, NarTop10Accuracy=0.6441, over 5772.00 frames. ], tot_loss[loss=3.419, NarTop10Accuracy=0.6342, over 5676.60 frames. ], batch size: 9, lr: 4.80e-03
2024-08-06 10:41:02,143 INFO [trainer.py:765] (1/8) Epoch 18, batch 700, train_loss[loss=3.335, NarTop10Accuracy=0.6526, over 5256.00 frames. ], tot_loss[loss=3.435, NarTop10Accuracy=0.6308, over 5761.17 frames. ], batch size: 6, lr: 4.79e-03
2024-08-06 10:41:38,519 INFO [trainer.py:765] (1/8) Epoch 18, batch 800, train_loss[loss=3.602, NarTop10Accuracy=0.6104, over 5246.00 frames. ], tot_loss[loss=3.436, NarTop10Accuracy=0.6307, over 5804.95 frames. ], batch size: 6, lr: 4.79e-03
2024-08-06 10:42:12,611 INFO [trainer.py:765] (1/8) Epoch 18, batch 900, train_loss[loss=3.54, NarTop10Accuracy=0.6112, over 6341.00 frames. ], tot_loss[loss=3.44, NarTop10Accuracy=0.6301, over 5818.26 frames. ], batch size: 13, lr: 4.78e-03
2024-08-06 10:42:46,702 INFO [trainer.py:765] (1/8) Epoch 18, batch 1000, train_loss[loss=3.354, NarTop10Accuracy=0.6489, over 6203.00 frames. ], tot_loss[loss=3.442, NarTop10Accuracy=0.6296, over 5921.04 frames. ], batch size: 13, lr: 4.78e-03
2024-08-06 10:43:24,183 INFO [trainer.py:765] (1/8) Epoch 18, batch 1100, train_loss[loss=3.538, NarTop10Accuracy=0.5988, over 6436.00 frames. ], tot_loss[loss=3.464, NarTop10Accuracy=0.625, over 5939.10 frames. ], batch size: 16, lr: 4.77e-03
2024-08-06 10:44:02,363 INFO [trainer.py:765] (1/8) Epoch 18, batch 1200, train_loss[loss=3.45, NarTop10Accuracy=0.632, over 7186.00 frames. ], tot_loss[loss=3.46, NarTop10Accuracy=0.626, over 5942.50 frames. ], batch size: 30, lr: 4.77e-03
2024-08-06 10:44:35,920 INFO [trainer.py:765] (1/8) Epoch 18, batch 1300, train_loss[loss=3.399, NarTop10Accuracy=0.6539, over 5063.00 frames. ], tot_loss[loss=3.448, NarTop10Accuracy=0.6277, over 6003.77 frames. ], batch size: 6, lr: 4.76e-03
2024-08-06 10:45:10,238 INFO [trainer.py:765] (1/8) Epoch 18, batch 1400, train_loss[loss=3.42, NarTop10Accuracy=0.6276, over 6241.00 frames. ], tot_loss[loss=3.455, NarTop10Accuracy=0.6267, over 6033.64 frames. ], batch size: 11, lr: 4.76e-03
2024-08-06 10:45:40,976 INFO [trainer.py:765] (1/8) Epoch 18, batch 1500, train_loss[loss=3.816, NarTop10Accuracy=0.5579, over 5828.00 frames. ], tot_loss[loss=3.458, NarTop10Accuracy=0.6259, over 5966.95 frames. ], batch size: 48, lr: 4.75e-03
2024-08-06 10:46:09,056 INFO [trainer.py:765] (1/8) Epoch 18, batch 1600, train_loss[loss=3.292, NarTop10Accuracy=0.6622, over 7038.00 frames. ], tot_loss[loss=3.468, NarTop10Accuracy=0.6238, over 5962.28 frames. ], batch size: 22, lr: 4.75e-03
2024-08-06 10:46:35,859 INFO [trainer.py:765] (1/8) Epoch 18, batch 1700, train_loss[loss=3.737, NarTop10Accuracy=0.561, over 6244.00 frames. ], tot_loss[loss=3.459, NarTop10Accuracy=0.6253, over 5957.36 frames. ], batch size: 13, lr: 4.74e-03
2024-08-06 10:47:02,438 INFO [trainer.py:765] (1/8) Epoch 18, batch 1800, train_loss[loss=3.397, NarTop10Accuracy=0.6365, over 7361.00 frames. ], tot_loss[loss=3.464, NarTop10Accuracy=0.6246, over 6020.01 frames. ], batch size: 22, lr: 4.74e-03
2024-08-06 10:47:29,093 INFO [trainer.py:765] (1/8) Epoch 18, batch 1900, train_loss[loss=3.73, NarTop10Accuracy=0.5701, over 6250.00 frames. ], tot_loss[loss=3.474, NarTop10Accuracy=0.6228, over 6049.11 frames. ], batch size: 49, lr: 4.73e-03
2024-08-06 10:47:54,884 INFO [trainer.py:765] (1/8) Epoch 18, batch 2000, train_loss[loss=3.435, NarTop10Accuracy=0.627, over 6031.00 frames. ], tot_loss[loss=3.478, NarTop10Accuracy=0.6222, over 6010.43 frames. ], batch size: 48, lr: 4.73e-03
2024-08-06 10:48:20,370 INFO [trainer.py:765] (1/8) Epoch 18, batch 2100, train_loss[loss=3.259, NarTop10Accuracy=0.6642, over 3967.00 frames. ], tot_loss[loss=3.475, NarTop10Accuracy=0.6228, over 5998.74 frames. ], batch size: 4, lr: 4.72e-03
2024-08-06 10:48:24,747 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 10:48:35,039 INFO [trainer.py:811] (1/8) Epoch 18, validation: loss=3.307, NarTop10Accuracy=0.6593, over 1907754.00 frames. 
2024-08-06 10:48:35,040 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30083MB
2024-08-06 10:48:35,534 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.484e+02 1.855e+02 2.003e+02 2.193e+02 3.481e+02, threshold=4.005e+02, percent-clipped=0.0
2024-08-06 10:48:56,096 INFO [trainer.py:765] (1/8) Epoch 18, batch 2200, train_loss[loss=3.382, NarTop10Accuracy=0.6486, over 7136.00 frames. ], tot_loss[loss=3.47, NarTop10Accuracy=0.6233, over 6027.10 frames. ], batch size: 30, lr: 4.72e-03
2024-08-06 10:49:21,521 INFO [trainer.py:765] (1/8) Epoch 18, batch 2300, train_loss[loss=3.323, NarTop10Accuracy=0.6545, over 5877.00 frames. ], tot_loss[loss=3.47, NarTop10Accuracy=0.6233, over 6055.61 frames. ], batch size: 9, lr: 4.71e-03
2024-08-06 10:49:46,256 INFO [trainer.py:765] (1/8) Epoch 18, batch 2400, train_loss[loss=3.189, NarTop10Accuracy=0.6671, over 5300.00 frames. ], tot_loss[loss=3.473, NarTop10Accuracy=0.6227, over 5877.44 frames. ], batch size: 7, lr: 4.71e-03
2024-08-06 10:50:09,708 INFO [trainer.py:765] (1/8) Epoch 18, batch 2500, train_loss[loss=3.301, NarTop10Accuracy=0.6661, over 5109.00 frames. ], tot_loss[loss=3.458, NarTop10Accuracy=0.6254, over 5541.51 frames. ], batch size: 6, lr: 4.70e-03
2024-08-06 10:50:30,782 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 10:51:33,564 INFO [trainer.py:765] (1/8) Epoch 19, batch 100, train_loss[loss=3.337, NarTop10Accuracy=0.6588, over 7440.00 frames. ], tot_loss[loss=3.41, NarTop10Accuracy=0.6371, over 2353.29 frames. ], batch size: 31, lr: 4.57e-03
2024-08-06 10:52:06,164 INFO [trainer.py:765] (1/8) Epoch 19, batch 200, train_loss[loss=3.647, NarTop10Accuracy=0.5901, over 6646.00 frames. ], tot_loss[loss=3.406, NarTop10Accuracy=0.6375, over 3861.58 frames. ], batch size: 17, lr: 4.56e-03
2024-08-06 10:52:40,031 INFO [trainer.py:765] (1/8) Epoch 19, batch 300, train_loss[loss=3.551, NarTop10Accuracy=0.599, over 7018.00 frames. ], tot_loss[loss=3.405, NarTop10Accuracy=0.6377, over 4676.36 frames. ], batch size: 22, lr: 4.56e-03
2024-08-06 10:53:12,830 INFO [trainer.py:765] (1/8) Epoch 19, batch 400, train_loss[loss=3.013, NarTop10Accuracy=0.7106, over 5222.00 frames. ], tot_loss[loss=3.417, NarTop10Accuracy=0.6348, over 5125.02 frames. ], batch size: 7, lr: 4.55e-03
2024-08-06 10:53:45,020 INFO [trainer.py:765] (1/8) Epoch 19, batch 500, train_loss[loss=3.278, NarTop10Accuracy=0.6469, over 6055.00 frames. ], tot_loss[loss=3.406, NarTop10Accuracy=0.637, over 5408.91 frames. ], batch size: 11, lr: 4.55e-03
2024-08-06 10:54:18,601 INFO [trainer.py:765] (1/8) Epoch 19, batch 600, train_loss[loss=3.122, NarTop10Accuracy=0.6904, over 5797.00 frames. ], tot_loss[loss=3.407, NarTop10Accuracy=0.637, over 5688.97 frames. ], batch size: 9, lr: 4.54e-03
2024-08-06 10:54:54,112 INFO [trainer.py:765] (1/8) Epoch 19, batch 700, train_loss[loss=3.433, NarTop10Accuracy=0.6184, over 5210.00 frames. ], tot_loss[loss=3.419, NarTop10Accuracy=0.6345, over 5750.36 frames. ], batch size: 6, lr: 4.54e-03
2024-08-06 10:55:29,925 INFO [trainer.py:765] (1/8) Epoch 19, batch 800, train_loss[loss=3.539, NarTop10Accuracy=0.6179, over 4294.00 frames. ], tot_loss[loss=3.424, NarTop10Accuracy=0.6329, over 5799.41 frames. ], batch size: 5, lr: 4.53e-03
2024-08-06 10:56:02,238 INFO [trainer.py:765] (1/8) Epoch 19, batch 900, train_loss[loss=3.562, NarTop10Accuracy=0.6176, over 6137.00 frames. ], tot_loss[loss=3.426, NarTop10Accuracy=0.6326, over 5812.35 frames. ], batch size: 13, lr: 4.53e-03
2024-08-06 10:56:38,300 INFO [trainer.py:765] (1/8) Epoch 19, batch 1000, train_loss[loss=3.297, NarTop10Accuracy=0.6538, over 6213.00 frames. ], tot_loss[loss=3.437, NarTop10Accuracy=0.6304, over 5926.42 frames. ], batch size: 13, lr: 4.52e-03
2024-08-06 10:57:15,187 INFO [trainer.py:765] (1/8) Epoch 19, batch 1100, train_loss[loss=3.268, NarTop10Accuracy=0.6657, over 6916.00 frames. ], tot_loss[loss=3.445, NarTop10Accuracy=0.6284, over 5954.30 frames. ], batch size: 17, lr: 4.52e-03
2024-08-06 10:57:46,665 INFO [trainer.py:765] (1/8) Epoch 19, batch 1200, train_loss[loss=3.346, NarTop10Accuracy=0.6566, over 6997.00 frames. ], tot_loss[loss=3.442, NarTop10Accuracy=0.6287, over 5934.09 frames. ], batch size: 30, lr: 4.51e-03
2024-08-06 10:58:23,901 INFO [trainer.py:765] (1/8) Epoch 19, batch 1300, train_loss[loss=3.464, NarTop10Accuracy=0.6305, over 4966.00 frames. ], tot_loss[loss=3.442, NarTop10Accuracy=0.6288, over 6004.29 frames. ], batch size: 6, lr: 4.51e-03
2024-08-06 10:58:58,028 INFO [trainer.py:765] (1/8) Epoch 19, batch 1400, train_loss[loss=3.352, NarTop10Accuracy=0.6449, over 6117.00 frames. ], tot_loss[loss=3.449, NarTop10Accuracy=0.6277, over 6017.98 frames. ], batch size: 11, lr: 4.50e-03
2024-08-06 10:59:30,770 INFO [trainer.py:765] (1/8) Epoch 19, batch 1500, train_loss[loss=3.691, NarTop10Accuracy=0.5762, over 5718.00 frames. ], tot_loss[loss=3.442, NarTop10Accuracy=0.6291, over 5977.04 frames. ], batch size: 49, lr: 4.50e-03
2024-08-06 10:59:40,831 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 10:59:50,899 INFO [trainer.py:811] (1/8) Epoch 19, validation: loss=3.276, NarTop10Accuracy=0.6653, over 1907754.00 frames. 
2024-08-06 10:59:50,899 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30472MB
2024-08-06 10:59:51,426 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.437e+02 1.829e+02 1.984e+02 2.176e+02 3.542e+02, threshold=3.967e+02, percent-clipped=0.0
2024-08-06 11:00:08,816 INFO [trainer.py:765] (1/8) Epoch 19, batch 1600, train_loss[loss=3.632, NarTop10Accuracy=0.5814, over 7086.00 frames. ], tot_loss[loss=3.442, NarTop10Accuracy=0.6287, over 5953.21 frames. ], batch size: 22, lr: 4.49e-03
2024-08-06 11:00:35,588 INFO [trainer.py:765] (1/8) Epoch 19, batch 1700, train_loss[loss=3.645, NarTop10Accuracy=0.5867, over 6256.00 frames. ], tot_loss[loss=3.46, NarTop10Accuracy=0.625, over 5949.83 frames. ], batch size: 13, lr: 4.49e-03
2024-08-06 11:01:02,257 INFO [trainer.py:765] (1/8) Epoch 19, batch 1800, train_loss[loss=3.391, NarTop10Accuracy=0.6437, over 7107.00 frames. ], tot_loss[loss=3.454, NarTop10Accuracy=0.627, over 6009.03 frames. ], batch size: 22, lr: 4.49e-03
2024-08-06 11:01:28,930 INFO [trainer.py:765] (1/8) Epoch 19, batch 1900, train_loss[loss=3.587, NarTop10Accuracy=0.5935, over 5837.00 frames. ], tot_loss[loss=3.475, NarTop10Accuracy=0.6224, over 6058.36 frames. ], batch size: 50, lr: 4.48e-03
2024-08-06 11:01:54,633 INFO [trainer.py:765] (1/8) Epoch 19, batch 2000, train_loss[loss=3.559, NarTop10Accuracy=0.6018, over 5758.00 frames. ], tot_loss[loss=3.461, NarTop10Accuracy=0.6249, over 6037.36 frames. ], batch size: 48, lr: 4.48e-03
2024-08-06 11:02:20,186 INFO [trainer.py:765] (1/8) Epoch 19, batch 2100, train_loss[loss=3.333, NarTop10Accuracy=0.6576, over 4031.00 frames. ], tot_loss[loss=3.463, NarTop10Accuracy=0.6247, over 6010.37 frames. ], batch size: 4, lr: 4.47e-03
2024-08-06 11:02:45,695 INFO [trainer.py:765] (1/8) Epoch 19, batch 2200, train_loss[loss=3.492, NarTop10Accuracy=0.6159, over 7127.00 frames. ], tot_loss[loss=3.467, NarTop10Accuracy=0.6241, over 6033.86 frames. ], batch size: 31, lr: 4.47e-03
2024-08-06 11:03:11,131 INFO [trainer.py:765] (1/8) Epoch 19, batch 2300, train_loss[loss=3.175, NarTop10Accuracy=0.6875, over 5878.00 frames. ], tot_loss[loss=3.464, NarTop10Accuracy=0.6245, over 6056.55 frames. ], batch size: 9, lr: 4.46e-03
2024-08-06 11:03:35,951 INFO [trainer.py:765] (1/8) Epoch 19, batch 2400, train_loss[loss=3.664, NarTop10Accuracy=0.5882, over 6723.00 frames. ], tot_loss[loss=3.475, NarTop10Accuracy=0.6224, over 5878.19 frames. ], batch size: 49, lr: 4.46e-03
2024-08-06 11:03:59,406 INFO [trainer.py:765] (1/8) Epoch 19, batch 2500, train_loss[loss=3.651, NarTop10Accuracy=0.5893, over 4915.00 frames. ], tot_loss[loss=3.441, NarTop10Accuracy=0.6292, over 5535.31 frames. ], batch size: 6, lr: 4.45e-03
2024-08-06 11:04:23,889 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 11:05:26,561 INFO [trainer.py:765] (1/8) Epoch 20, batch 100, train_loss[loss=3.402, NarTop10Accuracy=0.6343, over 6996.00 frames. ], tot_loss[loss=3.39, NarTop10Accuracy=0.6412, over 2361.34 frames. ], batch size: 30, lr: 4.33e-03
2024-08-06 11:05:57,409 INFO [trainer.py:765] (1/8) Epoch 20, batch 200, train_loss[loss=3.182, NarTop10Accuracy=0.6765, over 6873.00 frames. ], tot_loss[loss=3.38, NarTop10Accuracy=0.643, over 3868.54 frames. ], batch size: 17, lr: 4.33e-03
2024-08-06 11:06:30,634 INFO [trainer.py:765] (1/8) Epoch 20, batch 300, train_loss[loss=3.252, NarTop10Accuracy=0.672, over 6973.00 frames. ], tot_loss[loss=3.392, NarTop10Accuracy=0.6402, over 4656.76 frames. ], batch size: 22, lr: 4.32e-03
2024-08-06 11:07:06,396 INFO [trainer.py:765] (1/8) Epoch 20, batch 400, train_loss[loss=3.135, NarTop10Accuracy=0.6985, over 5197.00 frames. ], tot_loss[loss=3.387, NarTop10Accuracy=0.6407, over 5115.69 frames. ], batch size: 7, lr: 4.32e-03
2024-08-06 11:07:38,166 INFO [trainer.py:765] (1/8) Epoch 20, batch 500, train_loss[loss=3.353, NarTop10Accuracy=0.6451, over 6127.00 frames. ], tot_loss[loss=3.401, NarTop10Accuracy=0.6382, over 5398.09 frames. ], batch size: 11, lr: 4.31e-03
2024-08-06 11:08:11,568 INFO [trainer.py:765] (1/8) Epoch 20, batch 600, train_loss[loss=3.258, NarTop10Accuracy=0.6668, over 5767.00 frames. ], tot_loss[loss=3.399, NarTop10Accuracy=0.6384, over 5670.13 frames. ], batch size: 9, lr: 4.31e-03
2024-08-06 11:08:46,274 INFO [trainer.py:765] (1/8) Epoch 20, batch 700, train_loss[loss=3.427, NarTop10Accuracy=0.6432, over 5092.00 frames. ], tot_loss[loss=3.419, NarTop10Accuracy=0.6345, over 5731.27 frames. ], batch size: 6, lr: 4.31e-03
2024-08-06 11:09:23,425 INFO [trainer.py:765] (1/8) Epoch 20, batch 800, train_loss[loss=3.318, NarTop10Accuracy=0.6545, over 5092.00 frames. ], tot_loss[loss=3.43, NarTop10Accuracy=0.6324, over 5806.26 frames. ], batch size: 6, lr: 4.30e-03
2024-08-06 11:09:53,513 INFO [trainer.py:765] (1/8) Epoch 20, batch 900, train_loss[loss=3.448, NarTop10Accuracy=0.641, over 6179.00 frames. ], tot_loss[loss=3.427, NarTop10Accuracy=0.6326, over 5825.85 frames. ], batch size: 13, lr: 4.30e-03
2024-08-06 11:10:12,199 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 11:10:23,738 INFO [trainer.py:811] (1/8) Epoch 20, validation: loss=3.279, NarTop10Accuracy=0.6658, over 1907754.00 frames. 
2024-08-06 11:10:23,739 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30472MB
2024-08-06 11:10:24,298 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.491e+02 1.847e+02 2.007e+02 2.180e+02 4.417e+02, threshold=4.013e+02, percent-clipped=0.1
2024-08-06 11:10:42,964 INFO [trainer.py:765] (1/8) Epoch 20, batch 1000, train_loss[loss=3.238, NarTop10Accuracy=0.6643, over 6211.00 frames. ], tot_loss[loss=3.419, NarTop10Accuracy=0.6336, over 5919.13 frames. ], batch size: 13, lr: 4.29e-03
2024-08-06 11:11:21,021 INFO [trainer.py:765] (1/8) Epoch 20, batch 1100, train_loss[loss=3.397, NarTop10Accuracy=0.6422, over 6841.00 frames. ], tot_loss[loss=3.428, NarTop10Accuracy=0.6318, over 5961.94 frames. ], batch size: 17, lr: 4.29e-03
2024-08-06 11:11:55,393 INFO [trainer.py:765] (1/8) Epoch 20, batch 1200, train_loss[loss=3.429, NarTop10Accuracy=0.6278, over 7460.00 frames. ], tot_loss[loss=3.439, NarTop10Accuracy=0.6295, over 5963.17 frames. ], batch size: 31, lr: 4.28e-03
2024-08-06 11:12:30,751 INFO [trainer.py:765] (1/8) Epoch 20, batch 1300, train_loss[loss=3.906, NarTop10Accuracy=0.5406, over 5140.00 frames. ], tot_loss[loss=3.425, NarTop10Accuracy=0.6326, over 6034.74 frames. ], batch size: 6, lr: 4.28e-03
2024-08-06 11:13:10,291 INFO [trainer.py:765] (1/8) Epoch 20, batch 1400, train_loss[loss=3.381, NarTop10Accuracy=0.6262, over 6158.00 frames. ], tot_loss[loss=3.434, NarTop10Accuracy=0.6308, over 6063.59 frames. ], batch size: 11, lr: 4.28e-03
2024-08-06 11:13:38,988 INFO [trainer.py:765] (1/8) Epoch 20, batch 1500, train_loss[loss=3.555, NarTop10Accuracy=0.6135, over 6184.00 frames. ], tot_loss[loss=3.434, NarTop10Accuracy=0.6309, over 6000.87 frames. ], batch size: 49, lr: 4.27e-03
2024-08-06 11:14:07,051 INFO [trainer.py:765] (1/8) Epoch 20, batch 1600, train_loss[loss=3.446, NarTop10Accuracy=0.616, over 7215.00 frames. ], tot_loss[loss=3.44, NarTop10Accuracy=0.6293, over 5963.72 frames. ], batch size: 22, lr: 4.27e-03
2024-08-06 11:14:33,909 INFO [trainer.py:765] (1/8) Epoch 20, batch 1700, train_loss[loss=3.586, NarTop10Accuracy=0.6045, over 6582.00 frames. ], tot_loss[loss=3.436, NarTop10Accuracy=0.6303, over 5933.87 frames. ], batch size: 14, lr: 4.26e-03
2024-08-06 11:15:00,589 INFO [trainer.py:765] (1/8) Epoch 20, batch 1800, train_loss[loss=3.196, NarTop10Accuracy=0.6712, over 7195.00 frames. ], tot_loss[loss=3.444, NarTop10Accuracy=0.6284, over 5996.30 frames. ], batch size: 22, lr: 4.26e-03
2024-08-06 11:15:27,276 INFO [trainer.py:765] (1/8) Epoch 20, batch 1900, train_loss[loss=3.456, NarTop10Accuracy=0.6334, over 6379.00 frames. ], tot_loss[loss=3.454, NarTop10Accuracy=0.6261, over 6036.52 frames. ], batch size: 49, lr: 4.26e-03
2024-08-06 11:15:56,437 INFO [trainer.py:765] (1/8) Epoch 20, batch 2000, train_loss[loss=3.568, NarTop10Accuracy=0.6081, over 6796.00 frames. ], tot_loss[loss=3.458, NarTop10Accuracy=0.6258, over 6027.32 frames. ], batch size: 49, lr: 4.25e-03
2024-08-06 11:16:21,960 INFO [trainer.py:765] (1/8) Epoch 20, batch 2100, train_loss[loss=3.402, NarTop10Accuracy=0.6304, over 3998.00 frames. ], tot_loss[loss=3.46, NarTop10Accuracy=0.6255, over 5997.31 frames. ], batch size: 4, lr: 4.25e-03
2024-08-06 11:16:47,405 INFO [trainer.py:765] (1/8) Epoch 20, batch 2200, train_loss[loss=3.441, NarTop10Accuracy=0.6316, over 7340.00 frames. ], tot_loss[loss=3.456, NarTop10Accuracy=0.6259, over 6047.74 frames. ], batch size: 33, lr: 4.24e-03
2024-08-06 11:17:12,907 INFO [trainer.py:765] (1/8) Epoch 20, batch 2300, train_loss[loss=3.698, NarTop10Accuracy=0.5714, over 5813.00 frames. ], tot_loss[loss=3.464, NarTop10Accuracy=0.6249, over 6067.35 frames. ], batch size: 9, lr: 4.24e-03
2024-08-06 11:17:37,714 INFO [trainer.py:765] (1/8) Epoch 20, batch 2400, train_loss[loss=3.217, NarTop10Accuracy=0.6664, over 5121.00 frames. ], tot_loss[loss=3.462, NarTop10Accuracy=0.6254, over 5883.61 frames. ], batch size: 7, lr: 4.24e-03
2024-08-06 11:18:01,247 INFO [trainer.py:765] (1/8) Epoch 20, batch 2500, train_loss[loss=3.831, NarTop10Accuracy=0.5435, over 5006.00 frames. ], tot_loss[loss=3.437, NarTop10Accuracy=0.63, over 5558.47 frames. ], batch size: 6, lr: 4.23e-03
2024-08-06 11:18:22,207 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 11:19:21,459 INFO [trainer.py:765] (1/8) Epoch 21, batch 100, train_loss[loss=3.249, NarTop10Accuracy=0.6837, over 7330.00 frames. ], tot_loss[loss=3.396, NarTop10Accuracy=0.6392, over 2370.29 frames. ], batch size: 30, lr: 4.12e-03
2024-08-06 11:19:56,522 INFO [trainer.py:765] (1/8) Epoch 21, batch 200, train_loss[loss=3.537, NarTop10Accuracy=0.6134, over 6826.00 frames. ], tot_loss[loss=3.392, NarTop10Accuracy=0.6393, over 3867.20 frames. ], batch size: 17, lr: 4.12e-03
2024-08-06 11:20:26,597 INFO [trainer.py:765] (1/8) Epoch 21, batch 300, train_loss[loss=3.561, NarTop10Accuracy=0.6121, over 7224.00 frames. ], tot_loss[loss=3.376, NarTop10Accuracy=0.6433, over 4670.17 frames. ], batch size: 22, lr: 4.11e-03
2024-08-06 11:20:54,240 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 11:21:04,970 INFO [trainer.py:811] (1/8) Epoch 21, validation: loss=3.291, NarTop10Accuracy=0.6625, over 1907754.00 frames. 
2024-08-06 11:21:04,970 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30472MB
2024-08-06 11:21:05,486 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.509e+02 1.858e+02 2.007e+02 2.193e+02 3.729e+02, threshold=4.015e+02, percent-clipped=0.0
2024-08-06 11:21:12,221 INFO [trainer.py:765] (1/8) Epoch 21, batch 400, train_loss[loss=3.783, NarTop10Accuracy=0.5666, over 5103.00 frames. ], tot_loss[loss=3.386, NarTop10Accuracy=0.6406, over 5135.01 frames. ], batch size: 7, lr: 4.11e-03
2024-08-06 11:21:47,569 INFO [trainer.py:765] (1/8) Epoch 21, batch 500, train_loss[loss=3.216, NarTop10Accuracy=0.6753, over 6179.00 frames. ], tot_loss[loss=3.379, NarTop10Accuracy=0.6423, over 5413.78 frames. ], batch size: 11, lr: 4.11e-03
2024-08-06 11:22:18,237 INFO [trainer.py:765] (1/8) Epoch 21, batch 600, train_loss[loss=3.528, NarTop10Accuracy=0.611, over 5768.00 frames. ], tot_loss[loss=3.393, NarTop10Accuracy=0.6397, over 5670.72 frames. ], batch size: 9, lr: 4.10e-03
2024-08-06 11:22:56,842 INFO [trainer.py:765] (1/8) Epoch 21, batch 700, train_loss[loss=3.33, NarTop10Accuracy=0.6516, over 5078.00 frames. ], tot_loss[loss=3.408, NarTop10Accuracy=0.6367, over 5745.60 frames. ], batch size: 6, lr: 4.10e-03
2024-08-06 11:23:33,075 INFO [trainer.py:765] (1/8) Epoch 21, batch 800, train_loss[loss=3.232, NarTop10Accuracy=0.6647, over 5221.00 frames. ], tot_loss[loss=3.409, NarTop10Accuracy=0.6364, over 5807.84 frames. ], batch size: 6, lr: 4.09e-03
2024-08-06 11:24:03,021 INFO [trainer.py:765] (1/8) Epoch 21, batch 900, train_loss[loss=3.623, NarTop10Accuracy=0.5858, over 6295.00 frames. ], tot_loss[loss=3.411, NarTop10Accuracy=0.6357, over 5827.06 frames. ], batch size: 13, lr: 4.09e-03
2024-08-06 11:24:37,089 INFO [trainer.py:765] (1/8) Epoch 21, batch 1000, train_loss[loss=3.411, NarTop10Accuracy=0.6355, over 6774.00 frames. ], tot_loss[loss=3.422, NarTop10Accuracy=0.6333, over 5932.00 frames. ], batch size: 14, lr: 4.09e-03
2024-08-06 11:25:16,427 INFO [trainer.py:765] (1/8) Epoch 21, batch 1100, train_loss[loss=3.494, NarTop10Accuracy=0.6166, over 6897.00 frames. ], tot_loss[loss=3.429, NarTop10Accuracy=0.6318, over 5969.49 frames. ], batch size: 17, lr: 4.08e-03
2024-08-06 11:25:47,739 INFO [trainer.py:765] (1/8) Epoch 21, batch 1200, train_loss[loss=3.395, NarTop10Accuracy=0.6369, over 7292.00 frames. ], tot_loss[loss=3.413, NarTop10Accuracy=0.635, over 5953.63 frames. ], batch size: 30, lr: 4.08e-03
2024-08-06 11:26:23,056 INFO [trainer.py:765] (1/8) Epoch 21, batch 1300, train_loss[loss=3.482, NarTop10Accuracy=0.6206, over 4998.00 frames. ], tot_loss[loss=3.41, NarTop10Accuracy=0.6358, over 6033.40 frames. ], batch size: 6, lr: 4.07e-03
2024-08-06 11:27:00,081 INFO [trainer.py:765] (1/8) Epoch 21, batch 1400, train_loss[loss=3.324, NarTop10Accuracy=0.6605, over 6053.00 frames. ], tot_loss[loss=3.423, NarTop10Accuracy=0.6327, over 6034.94 frames. ], batch size: 11, lr: 4.07e-03
2024-08-06 11:27:35,326 INFO [trainer.py:765] (1/8) Epoch 21, batch 1500, train_loss[loss=3.718, NarTop10Accuracy=0.5747, over 6220.00 frames. ], tot_loss[loss=3.426, NarTop10Accuracy=0.6317, over 5975.49 frames. ], batch size: 49, lr: 4.07e-03
2024-08-06 11:28:03,315 INFO [trainer.py:765] (1/8) Epoch 21, batch 1600, train_loss[loss=3.358, NarTop10Accuracy=0.6457, over 7105.00 frames. ], tot_loss[loss=3.429, NarTop10Accuracy=0.6318, over 5953.66 frames. ], batch size: 22, lr: 4.06e-03
2024-08-06 11:28:30,105 INFO [trainer.py:765] (1/8) Epoch 21, batch 1700, train_loss[loss=3.456, NarTop10Accuracy=0.6325, over 6822.00 frames. ], tot_loss[loss=3.435, NarTop10Accuracy=0.6307, over 5924.98 frames. ], batch size: 14, lr: 4.06e-03
2024-08-06 11:28:56,641 INFO [trainer.py:765] (1/8) Epoch 21, batch 1800, train_loss[loss=3.574, NarTop10Accuracy=0.5965, over 6963.00 frames. ], tot_loss[loss=3.438, NarTop10Accuracy=0.63, over 6002.41 frames. ], batch size: 22, lr: 4.06e-03
2024-08-06 11:29:23,198 INFO [trainer.py:765] (1/8) Epoch 21, batch 1900, train_loss[loss=3.522, NarTop10Accuracy=0.6126, over 6517.00 frames. ], tot_loss[loss=3.45, NarTop10Accuracy=0.6277, over 6052.94 frames. ], batch size: 49, lr: 4.05e-03
2024-08-06 11:29:49,029 INFO [trainer.py:765] (1/8) Epoch 21, batch 2000, train_loss[loss=3.458, NarTop10Accuracy=0.6322, over 6307.00 frames. ], tot_loss[loss=3.446, NarTop10Accuracy=0.6291, over 6025.12 frames. ], batch size: 50, lr: 4.05e-03
2024-08-06 11:30:14,528 INFO [trainer.py:765] (1/8) Epoch 21, batch 2100, train_loss[loss=3.211, NarTop10Accuracy=0.6747, over 3857.00 frames. ], tot_loss[loss=3.444, NarTop10Accuracy=0.6289, over 6004.18 frames. ], batch size: 4, lr: 4.04e-03
2024-08-06 11:30:39,870 INFO [trainer.py:765] (1/8) Epoch 21, batch 2200, train_loss[loss=3.548, NarTop10Accuracy=0.6088, over 7465.00 frames. ], tot_loss[loss=3.449, NarTop10Accuracy=0.6277, over 6031.48 frames. ], batch size: 31, lr: 4.04e-03
2024-08-06 11:31:05,472 INFO [trainer.py:765] (1/8) Epoch 21, batch 2300, train_loss[loss=3.468, NarTop10Accuracy=0.6227, over 5773.00 frames. ], tot_loss[loss=3.456, NarTop10Accuracy=0.6267, over 6054.90 frames. ], batch size: 9, lr: 4.04e-03
2024-08-06 11:31:23,874 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 11:31:34,439 INFO [trainer.py:811] (1/8) Epoch 21, validation: loss=3.272, NarTop10Accuracy=0.6665, over 1907754.00 frames. 
2024-08-06 11:31:34,439 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30472MB
2024-08-06 11:31:34,937 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.496e+02 1.892e+02 2.038e+02 2.210e+02 4.910e+02, threshold=4.076e+02, percent-clipped=0.1
2024-08-06 11:31:40,751 INFO [trainer.py:765] (1/8) Epoch 21, batch 2400, train_loss[loss=3.349, NarTop10Accuracy=0.655, over 5243.00 frames. ], tot_loss[loss=3.454, NarTop10Accuracy=0.6269, over 5878.80 frames. ], batch size: 7, lr: 4.03e-03
2024-08-06 11:32:04,055 INFO [trainer.py:765] (1/8) Epoch 21, batch 2500, train_loss[loss=3.148, NarTop10Accuracy=0.6935, over 5053.00 frames. ], tot_loss[loss=3.418, NarTop10Accuracy=0.6335, over 5527.38 frames. ], batch size: 6, lr: 4.03e-03
2024-08-06 11:32:25,462 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 11:33:29,682 INFO [trainer.py:765] (1/8) Epoch 22, batch 100, train_loss[loss=3.566, NarTop10Accuracy=0.6113, over 7093.00 frames. ], tot_loss[loss=3.36, NarTop10Accuracy=0.6462, over 2362.74 frames. ], batch size: 30, lr: 3.93e-03
2024-08-06 11:34:05,036 INFO [trainer.py:765] (1/8) Epoch 22, batch 200, train_loss[loss=3.27, NarTop10Accuracy=0.6639, over 7019.00 frames. ], tot_loss[loss=3.369, NarTop10Accuracy=0.6448, over 3865.59 frames. ], batch size: 17, lr: 3.93e-03
2024-08-06 11:34:37,619 INFO [trainer.py:765] (1/8) Epoch 22, batch 300, train_loss[loss=3.253, NarTop10Accuracy=0.6711, over 6907.00 frames. ], tot_loss[loss=3.369, NarTop10Accuracy=0.6456, over 4661.57 frames. ], batch size: 22, lr: 3.92e-03
2024-08-06 11:35:09,969 INFO [trainer.py:765] (1/8) Epoch 22, batch 400, train_loss[loss=3.276, NarTop10Accuracy=0.6566, over 5291.00 frames. ], tot_loss[loss=3.386, NarTop10Accuracy=0.6418, over 5120.77 frames. ], batch size: 7, lr: 3.92e-03
2024-08-06 11:35:42,508 INFO [trainer.py:765] (1/8) Epoch 22, batch 500, train_loss[loss=3.444, NarTop10Accuracy=0.6324, over 6210.00 frames. ], tot_loss[loss=3.386, NarTop10Accuracy=0.6409, over 5405.26 frames. ], batch size: 11, lr: 3.91e-03
2024-08-06 11:36:16,059 INFO [trainer.py:765] (1/8) Epoch 22, batch 600, train_loss[loss=3.271, NarTop10Accuracy=0.6719, over 5698.00 frames. ], tot_loss[loss=3.383, NarTop10Accuracy=0.6409, over 5681.56 frames. ], batch size: 9, lr: 3.91e-03
2024-08-06 11:36:53,858 INFO [trainer.py:765] (1/8) Epoch 22, batch 700, train_loss[loss=3.283, NarTop10Accuracy=0.6577, over 5051.00 frames. ], tot_loss[loss=3.391, NarTop10Accuracy=0.6394, over 5763.01 frames. ], batch size: 6, lr: 3.91e-03
2024-08-06 11:37:28,480 INFO [trainer.py:765] (1/8) Epoch 22, batch 800, train_loss[loss=3.149, NarTop10Accuracy=0.6889, over 5252.00 frames. ], tot_loss[loss=3.401, NarTop10Accuracy=0.6375, over 5831.08 frames. ], batch size: 6, lr: 3.90e-03
2024-08-06 11:38:03,950 INFO [trainer.py:765] (1/8) Epoch 22, batch 900, train_loss[loss=3.254, NarTop10Accuracy=0.6709, over 6150.00 frames. ], tot_loss[loss=3.396, NarTop10Accuracy=0.6384, over 5845.46 frames. ], batch size: 13, lr: 3.90e-03
2024-08-06 11:38:38,329 INFO [trainer.py:765] (1/8) Epoch 22, batch 1000, train_loss[loss=3.22, NarTop10Accuracy=0.6762, over 6424.00 frames. ], tot_loss[loss=3.402, NarTop10Accuracy=0.6375, over 5934.81 frames. ], batch size: 13, lr: 3.90e-03
2024-08-06 11:39:14,789 INFO [trainer.py:765] (1/8) Epoch 22, batch 1100, train_loss[loss=3.531, NarTop10Accuracy=0.6139, over 6785.00 frames. ], tot_loss[loss=3.405, NarTop10Accuracy=0.6366, over 5958.45 frames. ], batch size: 17, lr: 3.89e-03
2024-08-06 11:39:48,523 INFO [trainer.py:765] (1/8) Epoch 22, batch 1200, train_loss[loss=3.384, NarTop10Accuracy=0.6428, over 7369.00 frames. ], tot_loss[loss=3.402, NarTop10Accuracy=0.637, over 5949.78 frames. ], batch size: 31, lr: 3.89e-03
2024-08-06 11:40:25,246 INFO [trainer.py:765] (1/8) Epoch 22, batch 1300, train_loss[loss=3.454, NarTop10Accuracy=0.6256, over 5017.00 frames. ], tot_loss[loss=3.407, NarTop10Accuracy=0.6359, over 6026.76 frames. ], batch size: 6, lr: 3.89e-03
2024-08-06 11:41:00,609 INFO [trainer.py:765] (1/8) Epoch 22, batch 1400, train_loss[loss=3.657, NarTop10Accuracy=0.593, over 6063.00 frames. ], tot_loss[loss=3.409, NarTop10Accuracy=0.6358, over 6052.39 frames. ], batch size: 11, lr: 3.88e-03
2024-08-06 11:41:31,584 INFO [trainer.py:765] (1/8) Epoch 22, batch 1500, train_loss[loss=3.599, NarTop10Accuracy=0.5979, over 6009.00 frames. ], tot_loss[loss=3.417, NarTop10Accuracy=0.634, over 5985.80 frames. ], batch size: 48, lr: 3.88e-03
2024-08-06 11:41:59,677 INFO [trainer.py:765] (1/8) Epoch 22, batch 1600, train_loss[loss=3.377, NarTop10Accuracy=0.6544, over 7158.00 frames. ], tot_loss[loss=3.432, NarTop10Accuracy=0.6313, over 5957.28 frames. ], batch size: 22, lr: 3.88e-03
2024-08-06 11:42:26,463 INFO [trainer.py:765] (1/8) Epoch 22, batch 1700, train_loss[loss=3.535, NarTop10Accuracy=0.6186, over 6271.00 frames. ], tot_loss[loss=3.436, NarTop10Accuracy=0.6303, over 5946.76 frames. ], batch size: 13, lr: 3.87e-03
2024-08-06 11:42:50,723 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 11:43:00,818 INFO [trainer.py:811] (1/8) Epoch 22, validation: loss=3.305, NarTop10Accuracy=0.6597, over 1907754.00 frames. 
2024-08-06 11:43:00,819 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30472MB
2024-08-06 11:43:01,327 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.500e+02 1.900e+02 2.042e+02 2.234e+02 3.494e+02, threshold=4.085e+02, percent-clipped=0.0
2024-08-06 11:43:03,219 INFO [trainer.py:765] (1/8) Epoch 22, batch 1800, train_loss[loss=3.419, NarTop10Accuracy=0.6381, over 7085.00 frames. ], tot_loss[loss=3.433, NarTop10Accuracy=0.6313, over 6003.92 frames. ], batch size: 22, lr: 3.87e-03
2024-08-06 11:43:29,752 INFO [trainer.py:765] (1/8) Epoch 22, batch 1900, train_loss[loss=3.712, NarTop10Accuracy=0.5697, over 6063.00 frames. ], tot_loss[loss=3.432, NarTop10Accuracy=0.6312, over 6039.88 frames. ], batch size: 50, lr: 3.87e-03
2024-08-06 11:43:55,485 INFO [trainer.py:765] (1/8) Epoch 22, batch 2000, train_loss[loss=3.762, NarTop10Accuracy=0.5607, over 5734.00 frames. ], tot_loss[loss=3.431, NarTop10Accuracy=0.6311, over 6015.74 frames. ], batch size: 51, lr: 3.86e-03
2024-08-06 11:44:20,932 INFO [trainer.py:765] (1/8) Epoch 22, batch 2100, train_loss[loss=3.189, NarTop10Accuracy=0.6845, over 4817.00 frames. ], tot_loss[loss=3.434, NarTop10Accuracy=0.631, over 5988.07 frames. ], batch size: 5, lr: 3.86e-03
2024-08-06 11:44:46,456 INFO [trainer.py:765] (1/8) Epoch 22, batch 2200, train_loss[loss=3.807, NarTop10Accuracy=0.5577, over 7113.00 frames. ], tot_loss[loss=3.432, NarTop10Accuracy=0.6314, over 6039.79 frames. ], batch size: 30, lr: 3.86e-03
2024-08-06 11:45:11,882 INFO [trainer.py:765] (1/8) Epoch 22, batch 2300, train_loss[loss=3.296, NarTop10Accuracy=0.6568, over 5752.00 frames. ], tot_loss[loss=3.44, NarTop10Accuracy=0.6294, over 6053.62 frames. ], batch size: 9, lr: 3.85e-03
2024-08-06 11:45:36,583 INFO [trainer.py:765] (1/8) Epoch 22, batch 2400, train_loss[loss=3.66, NarTop10Accuracy=0.5804, over 5537.00 frames. ], tot_loss[loss=3.451, NarTop10Accuracy=0.627, over 5872.42 frames. ], batch size: 48, lr: 3.85e-03
2024-08-06 11:46:00,081 INFO [trainer.py:765] (1/8) Epoch 22, batch 2500, train_loss[loss=3.262, NarTop10Accuracy=0.6472, over 5069.00 frames. ], tot_loss[loss=3.423, NarTop10Accuracy=0.6331, over 5538.06 frames. ], batch size: 6, lr: 3.85e-03
2024-08-06 11:46:21,605 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 11:47:20,476 INFO [trainer.py:765] (1/8) Epoch 23, batch 100, train_loss[loss=3.27, NarTop10Accuracy=0.6723, over 7310.00 frames. ], tot_loss[loss=3.359, NarTop10Accuracy=0.647, over 2368.66 frames. ], batch size: 31, lr: 3.75e-03
2024-08-06 11:47:52,035 INFO [trainer.py:765] (1/8) Epoch 23, batch 200, train_loss[loss=3.452, NarTop10Accuracy=0.6302, over 6889.00 frames. ], tot_loss[loss=3.357, NarTop10Accuracy=0.6474, over 3854.34 frames. ], batch size: 17, lr: 3.75e-03
2024-08-06 11:48:33,921 INFO [trainer.py:765] (1/8) Epoch 23, batch 300, train_loss[loss=3.441, NarTop10Accuracy=0.6297, over 7108.00 frames. ], tot_loss[loss=3.374, NarTop10Accuracy=0.644, over 4659.21 frames. ], batch size: 22, lr: 3.75e-03
2024-08-06 11:49:06,655 INFO [trainer.py:765] (1/8) Epoch 23, batch 400, train_loss[loss=3.263, NarTop10Accuracy=0.6775, over 5203.00 frames. ], tot_loss[loss=3.383, NarTop10Accuracy=0.6421, over 5109.34 frames. ], batch size: 7, lr: 3.74e-03
2024-08-06 11:49:37,618 INFO [trainer.py:765] (1/8) Epoch 23, batch 500, train_loss[loss=3.462, NarTop10Accuracy=0.6395, over 6198.00 frames. ], tot_loss[loss=3.384, NarTop10Accuracy=0.6418, over 5383.77 frames. ], batch size: 11, lr: 3.74e-03
2024-08-06 11:50:06,740 INFO [trainer.py:765] (1/8) Epoch 23, batch 600, train_loss[loss=3.496, NarTop10Accuracy=0.6142, over 5776.00 frames. ], tot_loss[loss=3.386, NarTop10Accuracy=0.6419, over 5650.96 frames. ], batch size: 9, lr: 3.74e-03
2024-08-06 11:50:47,600 INFO [trainer.py:765] (1/8) Epoch 23, batch 700, train_loss[loss=3.328, NarTop10Accuracy=0.656, over 5030.00 frames. ], tot_loss[loss=3.396, NarTop10Accuracy=0.6395, over 5738.24 frames. ], batch size: 6, lr: 3.73e-03
2024-08-06 11:51:21,344 INFO [trainer.py:765] (1/8) Epoch 23, batch 800, train_loss[loss=3.401, NarTop10Accuracy=0.647, over 5084.00 frames. ], tot_loss[loss=3.389, NarTop10Accuracy=0.6409, over 5796.86 frames. ], batch size: 6, lr: 3.73e-03
2024-08-06 11:51:52,397 INFO [trainer.py:765] (1/8) Epoch 23, batch 900, train_loss[loss=3.261, NarTop10Accuracy=0.6633, over 6212.00 frames. ], tot_loss[loss=3.381, NarTop10Accuracy=0.6422, over 5817.48 frames. ], batch size: 13, lr: 3.73e-03
2024-08-06 11:52:33,918 INFO [trainer.py:765] (1/8) Epoch 23, batch 1000, train_loss[loss=3.266, NarTop10Accuracy=0.6641, over 6690.00 frames. ], tot_loss[loss=3.393, NarTop10Accuracy=0.6396, over 5919.19 frames. ], batch size: 14, lr: 3.73e-03
2024-08-06 11:53:08,607 INFO [trainer.py:765] (1/8) Epoch 23, batch 1100, train_loss[loss=3.444, NarTop10Accuracy=0.6165, over 6990.00 frames. ], tot_loss[loss=3.41, NarTop10Accuracy=0.6362, over 5947.87 frames. ], batch size: 17, lr: 3.72e-03
2024-08-06 11:53:40,339 INFO [trainer.py:765] (1/8) Epoch 23, batch 1200, train_loss[loss=3.366, NarTop10Accuracy=0.6406, over 7323.00 frames. ], tot_loss[loss=3.413, NarTop10Accuracy=0.6355, over 5943.55 frames. ], batch size: 30, lr: 3.72e-03
2024-08-06 11:53:42,824 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 11:53:53,935 INFO [trainer.py:811] (1/8) Epoch 23, validation: loss=3.236, NarTop10Accuracy=0.6739, over 1907754.00 frames. 
2024-08-06 11:53:53,935 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30472MB
2024-08-06 11:53:54,457 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.540e+02 1.901e+02 2.047e+02 2.234e+02 4.368e+02, threshold=4.093e+02, percent-clipped=0.1
2024-08-06 11:54:30,447 INFO [trainer.py:765] (1/8) Epoch 23, batch 1300, train_loss[loss=3.338, NarTop10Accuracy=0.6556, over 5139.00 frames. ], tot_loss[loss=3.405, NarTop10Accuracy=0.6366, over 6032.55 frames. ], batch size: 6, lr: 3.72e-03
2024-08-06 11:55:04,197 INFO [trainer.py:765] (1/8) Epoch 23, batch 1400, train_loss[loss=3.396, NarTop10Accuracy=0.6466, over 6055.00 frames. ], tot_loss[loss=3.408, NarTop10Accuracy=0.6362, over 6027.93 frames. ], batch size: 11, lr: 3.71e-03
2024-08-06 11:55:35,398 INFO [trainer.py:765] (1/8) Epoch 23, batch 1500, train_loss[loss=3.658, NarTop10Accuracy=0.587, over 6041.00 frames. ], tot_loss[loss=3.433, NarTop10Accuracy=0.6316, over 5978.07 frames. ], batch size: 49, lr: 3.71e-03
2024-08-06 11:56:03,428 INFO [trainer.py:765] (1/8) Epoch 23, batch 1600, train_loss[loss=3.428, NarTop10Accuracy=0.6374, over 7043.00 frames. ], tot_loss[loss=3.423, NarTop10Accuracy=0.6331, over 5959.04 frames. ], batch size: 22, lr: 3.71e-03
2024-08-06 11:56:30,202 INFO [trainer.py:765] (1/8) Epoch 23, batch 1700, train_loss[loss=3.498, NarTop10Accuracy=0.6159, over 6235.00 frames. ], tot_loss[loss=3.433, NarTop10Accuracy=0.6308, over 5941.15 frames. ], batch size: 13, lr: 3.70e-03
2024-08-06 11:56:56,969 INFO [trainer.py:765] (1/8) Epoch 23, batch 1800, train_loss[loss=3.34, NarTop10Accuracy=0.66, over 7312.00 frames. ], tot_loss[loss=3.429, NarTop10Accuracy=0.6318, over 5993.39 frames. ], batch size: 22, lr: 3.70e-03
2024-08-06 11:57:23,597 INFO [trainer.py:765] (1/8) Epoch 23, batch 1900, train_loss[loss=3.452, NarTop10Accuracy=0.6212, over 6218.00 frames. ], tot_loss[loss=3.44, NarTop10Accuracy=0.6301, over 6033.93 frames. ], batch size: 49, lr: 3.70e-03
2024-08-06 11:57:49,251 INFO [trainer.py:765] (1/8) Epoch 23, batch 2000, train_loss[loss=3.791, NarTop10Accuracy=0.5622, over 6382.00 frames. ], tot_loss[loss=3.442, NarTop10Accuracy=0.6296, over 6022.15 frames. ], batch size: 52, lr: 3.69e-03
2024-08-06 11:58:14,770 INFO [trainer.py:765] (1/8) Epoch 23, batch 2100, train_loss[loss=3.459, NarTop10Accuracy=0.6255, over 4032.00 frames. ], tot_loss[loss=3.45, NarTop10Accuracy=0.6279, over 6002.98 frames. ], batch size: 4, lr: 3.69e-03
2024-08-06 11:58:40,238 INFO [trainer.py:765] (1/8) Epoch 23, batch 2200, train_loss[loss=3.537, NarTop10Accuracy=0.6163, over 7334.00 frames. ], tot_loss[loss=3.436, NarTop10Accuracy=0.6311, over 6046.69 frames. ], batch size: 32, lr: 3.69e-03
2024-08-06 11:59:08,916 INFO [trainer.py:765] (1/8) Epoch 23, batch 2300, train_loss[loss=3.347, NarTop10Accuracy=0.6582, over 5802.00 frames. ], tot_loss[loss=3.447, NarTop10Accuracy=0.6289, over 6068.30 frames. ], batch size: 9, lr: 3.68e-03
2024-08-06 11:59:33,602 INFO [trainer.py:765] (1/8) Epoch 23, batch 2400, train_loss[loss=3.323, NarTop10Accuracy=0.661, over 5097.00 frames. ], tot_loss[loss=3.454, NarTop10Accuracy=0.6273, over 5898.51 frames. ], batch size: 7, lr: 3.68e-03
2024-08-06 11:59:57,011 INFO [trainer.py:765] (1/8) Epoch 23, batch 2500, train_loss[loss=3.403, NarTop10Accuracy=0.643, over 5073.00 frames. ], tot_loss[loss=3.419, NarTop10Accuracy=0.634, over 5555.16 frames. ], batch size: 6, lr: 3.68e-03
2024-08-06 12:00:18,571 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 12:01:22,111 INFO [trainer.py:765] (1/8) Epoch 24, batch 100, train_loss[loss=3.482, NarTop10Accuracy=0.6203, over 7231.00 frames. ], tot_loss[loss=3.377, NarTop10Accuracy=0.6433, over 2356.13 frames. ], batch size: 30, lr: 3.59e-03
2024-08-06 12:01:51,342 INFO [trainer.py:765] (1/8) Epoch 24, batch 200, train_loss[loss=3.563, NarTop10Accuracy=0.6001, over 6991.00 frames. ], tot_loss[loss=3.368, NarTop10Accuracy=0.6454, over 3868.03 frames. ], batch size: 17, lr: 3.59e-03
2024-08-06 12:02:23,513 INFO [trainer.py:765] (1/8) Epoch 24, batch 300, train_loss[loss=3.291, NarTop10Accuracy=0.6687, over 7060.00 frames. ], tot_loss[loss=3.368, NarTop10Accuracy=0.645, over 4680.90 frames. ], batch size: 22, lr: 3.59e-03
2024-08-06 12:03:02,847 INFO [trainer.py:765] (1/8) Epoch 24, batch 400, train_loss[loss=3.141, NarTop10Accuracy=0.6856, over 5265.00 frames. ], tot_loss[loss=3.37, NarTop10Accuracy=0.645, over 5121.27 frames. ], batch size: 7, lr: 3.59e-03
2024-08-06 12:03:31,256 INFO [trainer.py:765] (1/8) Epoch 24, batch 500, train_loss[loss=3.246, NarTop10Accuracy=0.6734, over 6128.00 frames. ], tot_loss[loss=3.363, NarTop10Accuracy=0.6461, over 5412.30 frames. ], batch size: 11, lr: 3.58e-03
2024-08-06 12:04:00,173 INFO [trainer.py:765] (1/8) Epoch 24, batch 600, train_loss[loss=3.465, NarTop10Accuracy=0.6321, over 5754.00 frames. ], tot_loss[loss=3.361, NarTop10Accuracy=0.6457, over 5685.28 frames. ], batch size: 9, lr: 3.58e-03
2024-08-06 12:04:12,531 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 12:04:22,775 INFO [trainer.py:811] (1/8) Epoch 24, validation: loss=3.282, NarTop10Accuracy=0.6644, over 1907754.00 frames. 
2024-08-06 12:04:22,776 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30472MB
2024-08-06 12:04:23,310 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.497e+02 1.905e+02 2.071e+02 2.258e+02 3.709e+02, threshold=4.142e+02, percent-clipped=0.0
2024-08-06 12:04:51,733 INFO [trainer.py:765] (1/8) Epoch 24, batch 700, train_loss[loss=3.077, NarTop10Accuracy=0.6929, over 5023.00 frames. ], tot_loss[loss=3.364, NarTop10Accuracy=0.6451, over 5748.66 frames. ], batch size: 6, lr: 3.58e-03
2024-08-06 12:05:21,274 INFO [trainer.py:765] (1/8) Epoch 24, batch 800, train_loss[loss=3.361, NarTop10Accuracy=0.6558, over 4990.00 frames. ], tot_loss[loss=3.37, NarTop10Accuracy=0.644, over 5785.57 frames. ], batch size: 6, lr: 3.57e-03
2024-08-06 12:05:51,754 INFO [trainer.py:765] (1/8) Epoch 24, batch 900, train_loss[loss=3.746, NarTop10Accuracy=0.5671, over 6172.00 frames. ], tot_loss[loss=3.375, NarTop10Accuracy=0.6434, over 5807.06 frames. ], batch size: 13, lr: 3.57e-03
2024-08-06 12:06:32,812 INFO [trainer.py:765] (1/8) Epoch 24, batch 1000, train_loss[loss=3.03, NarTop10Accuracy=0.7097, over 6153.00 frames. ], tot_loss[loss=3.381, NarTop10Accuracy=0.6421, over 5911.45 frames. ], batch size: 13, lr: 3.57e-03
2024-08-06 12:07:09,040 INFO [trainer.py:765] (1/8) Epoch 24, batch 1100, train_loss[loss=3.394, NarTop10Accuracy=0.6335, over 6861.00 frames. ], tot_loss[loss=3.382, NarTop10Accuracy=0.6413, over 5952.95 frames. ], batch size: 17, lr: 3.56e-03
2024-08-06 12:07:38,135 INFO [trainer.py:765] (1/8) Epoch 24, batch 1200, train_loss[loss=3.515, NarTop10Accuracy=0.6231, over 7158.00 frames. ], tot_loss[loss=3.395, NarTop10Accuracy=0.6387, over 5944.31 frames. ], batch size: 31, lr: 3.56e-03
2024-08-06 12:08:20,731 INFO [trainer.py:765] (1/8) Epoch 24, batch 1300, train_loss[loss=3.297, NarTop10Accuracy=0.6555, over 4999.00 frames. ], tot_loss[loss=3.402, NarTop10Accuracy=0.6369, over 6013.76 frames. ], batch size: 6, lr: 3.56e-03
2024-08-06 12:08:56,066 INFO [trainer.py:765] (1/8) Epoch 24, batch 1400, train_loss[loss=3.207, NarTop10Accuracy=0.6678, over 6057.00 frames. ], tot_loss[loss=3.409, NarTop10Accuracy=0.6352, over 6025.32 frames. ], batch size: 11, lr: 3.56e-03
2024-08-06 12:09:24,338 INFO [trainer.py:765] (1/8) Epoch 24, batch 1500, train_loss[loss=3.493, NarTop10Accuracy=0.6269, over 6195.00 frames. ], tot_loss[loss=3.417, NarTop10Accuracy=0.6343, over 5972.17 frames. ], batch size: 49, lr: 3.55e-03
2024-08-06 12:09:52,525 INFO [trainer.py:765] (1/8) Epoch 24, batch 1600, train_loss[loss=3.327, NarTop10Accuracy=0.6562, over 7254.00 frames. ], tot_loss[loss=3.407, NarTop10Accuracy=0.6361, over 5961.07 frames. ], batch size: 22, lr: 3.55e-03
2024-08-06 12:10:22,546 INFO [trainer.py:765] (1/8) Epoch 24, batch 1700, train_loss[loss=3.375, NarTop10Accuracy=0.6341, over 6310.00 frames. ], tot_loss[loss=3.418, NarTop10Accuracy=0.6338, over 5947.18 frames. ], batch size: 13, lr: 3.55e-03
2024-08-06 12:10:49,273 INFO [trainer.py:765] (1/8) Epoch 24, batch 1800, train_loss[loss=3.349, NarTop10Accuracy=0.6527, over 7150.00 frames. ], tot_loss[loss=3.411, NarTop10Accuracy=0.6349, over 6006.35 frames. ], batch size: 22, lr: 3.54e-03
2024-08-06 12:11:15,847 INFO [trainer.py:765] (1/8) Epoch 24, batch 1900, train_loss[loss=3.457, NarTop10Accuracy=0.6271, over 6107.00 frames. ], tot_loss[loss=3.419, NarTop10Accuracy=0.6335, over 6037.15 frames. ], batch size: 49, lr: 3.54e-03
2024-08-06 12:11:41,667 INFO [trainer.py:765] (1/8) Epoch 24, batch 2000, train_loss[loss=3.381, NarTop10Accuracy=0.6378, over 5864.00 frames. ], tot_loss[loss=3.423, NarTop10Accuracy=0.6333, over 6028.86 frames. ], batch size: 50, lr: 3.54e-03
2024-08-06 12:12:07,104 INFO [trainer.py:765] (1/8) Epoch 24, batch 2100, train_loss[loss=3.473, NarTop10Accuracy=0.6274, over 4829.00 frames. ], tot_loss[loss=3.411, NarTop10Accuracy=0.6357, over 6002.37 frames. ], batch size: 5, lr: 3.54e-03
2024-08-06 12:12:33,373 INFO [trainer.py:765] (1/8) Epoch 24, batch 2200, train_loss[loss=3.456, NarTop10Accuracy=0.6273, over 7710.00 frames. ], tot_loss[loss=3.419, NarTop10Accuracy=0.6343, over 6045.45 frames. ], batch size: 31, lr: 3.53e-03
2024-08-06 12:12:58,772 INFO [trainer.py:765] (1/8) Epoch 24, batch 2300, train_loss[loss=3.291, NarTop10Accuracy=0.6545, over 5657.00 frames. ], tot_loss[loss=3.436, NarTop10Accuracy=0.6312, over 6064.07 frames. ], batch size: 9, lr: 3.53e-03
2024-08-06 12:13:23,487 INFO [trainer.py:765] (1/8) Epoch 24, batch 2400, train_loss[loss=3.581, NarTop10Accuracy=0.6097, over 6204.00 frames. ], tot_loss[loss=3.44, NarTop10Accuracy=0.6299, over 5899.04 frames. ], batch size: 49, lr: 3.53e-03
2024-08-06 12:13:47,006 INFO [trainer.py:765] (1/8) Epoch 24, batch 2500, train_loss[loss=3.146, NarTop10Accuracy=0.6945, over 4997.00 frames. ], tot_loss[loss=3.413, NarTop10Accuracy=0.6354, over 5556.84 frames. ], batch size: 6, lr: 3.52e-03
2024-08-06 12:14:08,242 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 12:14:50,196 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 12:15:00,657 INFO [trainer.py:811] (1/8) Epoch 25, validation: loss=3.279, NarTop10Accuracy=0.6656, over 1907754.00 frames. 
2024-08-06 12:15:00,658 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 12:15:01,363 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.547e+02 1.921e+02 2.068e+02 2.276e+02 6.228e+02, threshold=4.136e+02, percent-clipped=0.3
2024-08-06 12:15:17,917 INFO [trainer.py:765] (1/8) Epoch 25, batch 100, train_loss[loss=3.284, NarTop10Accuracy=0.6642, over 7236.00 frames. ], tot_loss[loss=3.354, NarTop10Accuracy=0.6488, over 2360.09 frames. ], batch size: 30, lr: 3.45e-03
2024-08-06 12:15:53,499 INFO [trainer.py:765] (1/8) Epoch 25, batch 200, train_loss[loss=3.222, NarTop10Accuracy=0.6725, over 6807.00 frames. ], tot_loss[loss=3.349, NarTop10Accuracy=0.6493, over 3857.83 frames. ], batch size: 17, lr: 3.44e-03
2024-08-06 12:16:23,595 INFO [trainer.py:765] (1/8) Epoch 25, batch 300, train_loss[loss=3.381, NarTop10Accuracy=0.6468, over 7178.00 frames. ], tot_loss[loss=3.364, NarTop10Accuracy=0.6464, over 4669.30 frames. ], batch size: 22, lr: 3.44e-03
2024-08-06 12:16:59,163 INFO [trainer.py:765] (1/8) Epoch 25, batch 400, train_loss[loss=3.492, NarTop10Accuracy=0.6248, over 5104.00 frames. ], tot_loss[loss=3.36, NarTop10Accuracy=0.6467, over 5113.24 frames. ], batch size: 7, lr: 3.44e-03
2024-08-06 12:17:32,096 INFO [trainer.py:765] (1/8) Epoch 25, batch 500, train_loss[loss=3.352, NarTop10Accuracy=0.6499, over 6285.00 frames. ], tot_loss[loss=3.355, NarTop10Accuracy=0.6482, over 5406.04 frames. ], batch size: 11, lr: 3.44e-03
2024-08-06 12:18:05,181 INFO [trainer.py:765] (1/8) Epoch 25, batch 600, train_loss[loss=3.395, NarTop10Accuracy=0.6406, over 5788.00 frames. ], tot_loss[loss=3.355, NarTop10Accuracy=0.6479, over 5671.93 frames. ], batch size: 9, lr: 3.43e-03
2024-08-06 12:18:39,597 INFO [trainer.py:765] (1/8) Epoch 25, batch 700, train_loss[loss=3.329, NarTop10Accuracy=0.6508, over 4990.00 frames. ], tot_loss[loss=3.358, NarTop10Accuracy=0.647, over 5746.37 frames. ], batch size: 6, lr: 3.43e-03
2024-08-06 12:19:16,014 INFO [trainer.py:765] (1/8) Epoch 25, batch 800, train_loss[loss=3.273, NarTop10Accuracy=0.6754, over 5022.00 frames. ], tot_loss[loss=3.377, NarTop10Accuracy=0.6433, over 5804.69 frames. ], batch size: 6, lr: 3.43e-03
2024-08-06 12:19:49,558 INFO [trainer.py:765] (1/8) Epoch 25, batch 900, train_loss[loss=3.259, NarTop10Accuracy=0.6679, over 6343.00 frames. ], tot_loss[loss=3.38, NarTop10Accuracy=0.6423, over 5826.59 frames. ], batch size: 13, lr: 3.43e-03
2024-08-06 12:20:23,876 INFO [trainer.py:765] (1/8) Epoch 25, batch 1000, train_loss[loss=3.212, NarTop10Accuracy=0.6763, over 6285.00 frames. ], tot_loss[loss=3.375, NarTop10Accuracy=0.6428, over 5919.46 frames. ], batch size: 13, lr: 3.42e-03
2024-08-06 12:21:01,915 INFO [trainer.py:765] (1/8) Epoch 25, batch 1100, train_loss[loss=3.33, NarTop10Accuracy=0.6485, over 6837.00 frames. ], tot_loss[loss=3.393, NarTop10Accuracy=0.639, over 5952.75 frames. ], batch size: 17, lr: 3.42e-03
2024-08-06 12:21:40,638 INFO [trainer.py:765] (1/8) Epoch 25, batch 1200, train_loss[loss=3.437, NarTop10Accuracy=0.6258, over 7271.00 frames. ], tot_loss[loss=3.386, NarTop10Accuracy=0.6402, over 5965.41 frames. ], batch size: 30, lr: 3.42e-03
2024-08-06 12:22:11,837 INFO [trainer.py:765] (1/8) Epoch 25, batch 1300, train_loss[loss=3.544, NarTop10Accuracy=0.6109, over 5179.00 frames. ], tot_loss[loss=3.387, NarTop10Accuracy=0.6403, over 6021.88 frames. ], batch size: 6, lr: 3.41e-03
2024-08-06 12:22:48,550 INFO [trainer.py:765] (1/8) Epoch 25, batch 1400, train_loss[loss=3.632, NarTop10Accuracy=0.6039, over 6156.00 frames. ], tot_loss[loss=3.386, NarTop10Accuracy=0.6403, over 6054.48 frames. ], batch size: 11, lr: 3.41e-03
2024-08-06 12:23:21,655 INFO [trainer.py:765] (1/8) Epoch 25, batch 1500, train_loss[loss=3.853, NarTop10Accuracy=0.5432, over 5949.00 frames. ], tot_loss[loss=3.396, NarTop10Accuracy=0.6385, over 5970.10 frames. ], batch size: 50, lr: 3.41e-03
2024-08-06 12:23:49,717 INFO [trainer.py:765] (1/8) Epoch 25, batch 1600, train_loss[loss=3.39, NarTop10Accuracy=0.6385, over 7170.00 frames. ], tot_loss[loss=3.401, NarTop10Accuracy=0.6375, over 5947.62 frames. ], batch size: 22, lr: 3.41e-03
2024-08-06 12:24:16,373 INFO [trainer.py:765] (1/8) Epoch 25, batch 1700, train_loss[loss=3.459, NarTop10Accuracy=0.6177, over 6759.00 frames. ], tot_loss[loss=3.405, NarTop10Accuracy=0.6369, over 5941.97 frames. ], batch size: 14, lr: 3.40e-03
2024-08-06 12:24:43,092 INFO [trainer.py:765] (1/8) Epoch 25, batch 1800, train_loss[loss=3.164, NarTop10Accuracy=0.6757, over 7002.00 frames. ], tot_loss[loss=3.404, NarTop10Accuracy=0.6369, over 6011.40 frames. ], batch size: 22, lr: 3.40e-03
2024-08-06 12:25:09,776 INFO [trainer.py:765] (1/8) Epoch 25, batch 1900, train_loss[loss=3.694, NarTop10Accuracy=0.5792, over 6385.00 frames. ], tot_loss[loss=3.429, NarTop10Accuracy=0.6317, over 6052.73 frames. ], batch size: 50, lr: 3.40e-03
2024-08-06 12:25:35,710 INFO [trainer.py:765] (1/8) Epoch 25, batch 2000, train_loss[loss=3.634, NarTop10Accuracy=0.588, over 5507.00 frames. ], tot_loss[loss=3.409, NarTop10Accuracy=0.6355, over 6013.86 frames. ], batch size: 49, lr: 3.40e-03
2024-08-06 12:25:47,854 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 12:25:58,846 INFO [trainer.py:811] (1/8) Epoch 25, validation: loss=3.265, NarTop10Accuracy=0.667, over 1907754.00 frames. 
2024-08-06 12:25:58,847 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 12:25:59,344 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.566e+02 1.947e+02 2.092e+02 2.280e+02 8.190e+02, threshold=4.185e+02, percent-clipped=0.2
2024-08-06 12:26:12,224 INFO [trainer.py:765] (1/8) Epoch 25, batch 2100, train_loss[loss=3.277, NarTop10Accuracy=0.6495, over 4805.00 frames. ], tot_loss[loss=3.419, NarTop10Accuracy=0.6334, over 6003.74 frames. ], batch size: 5, lr: 3.39e-03
2024-08-06 12:26:37,833 INFO [trainer.py:765] (1/8) Epoch 25, batch 2200, train_loss[loss=3.455, NarTop10Accuracy=0.6307, over 7329.00 frames. ], tot_loss[loss=3.419, NarTop10Accuracy=0.634, over 6038.36 frames. ], batch size: 31, lr: 3.39e-03
2024-08-06 12:27:03,344 INFO [trainer.py:765] (1/8) Epoch 25, batch 2300, train_loss[loss=3.6, NarTop10Accuracy=0.6079, over 5917.00 frames. ], tot_loss[loss=3.427, NarTop10Accuracy=0.6328, over 6077.23 frames. ], batch size: 9, lr: 3.39e-03
2024-08-06 12:27:28,151 INFO [trainer.py:765] (1/8) Epoch 25, batch 2400, train_loss[loss=3.796, NarTop10Accuracy=0.5625, over 6097.00 frames. ], tot_loss[loss=3.428, NarTop10Accuracy=0.6323, over 5886.77 frames. ], batch size: 52, lr: 3.39e-03
2024-08-06 12:27:51,732 INFO [trainer.py:765] (1/8) Epoch 25, batch 2500, train_loss[loss=3.344, NarTop10Accuracy=0.6537, over 5095.00 frames. ], tot_loss[loss=3.402, NarTop10Accuracy=0.6375, over 5534.07 frames. ], batch size: 6, lr: 3.38e-03
2024-08-06 12:28:13,195 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 12:29:08,881 INFO [trainer.py:765] (1/8) Epoch 26, batch 100, train_loss[loss=3.73, NarTop10Accuracy=0.5614, over 7150.00 frames. ], tot_loss[loss=3.349, NarTop10Accuracy=0.6489, over 2375.37 frames. ], batch size: 30, lr: 3.31e-03
2024-08-06 12:29:44,318 INFO [trainer.py:765] (1/8) Epoch 26, batch 200, train_loss[loss=3.269, NarTop10Accuracy=0.6641, over 6556.00 frames. ], tot_loss[loss=3.358, NarTop10Accuracy=0.6472, over 3878.89 frames. ], batch size: 16, lr: 3.31e-03
2024-08-06 12:30:19,754 INFO [trainer.py:765] (1/8) Epoch 26, batch 300, train_loss[loss=3.284, NarTop10Accuracy=0.6541, over 7197.00 frames. ], tot_loss[loss=3.372, NarTop10Accuracy=0.6446, over 4691.17 frames. ], batch size: 22, lr: 3.31e-03
2024-08-06 12:30:52,509 INFO [trainer.py:765] (1/8) Epoch 26, batch 400, train_loss[loss=3.302, NarTop10Accuracy=0.6528, over 5065.00 frames. ], tot_loss[loss=3.364, NarTop10Accuracy=0.646, over 5144.37 frames. ], batch size: 7, lr: 3.30e-03
2024-08-06 12:31:26,531 INFO [trainer.py:765] (1/8) Epoch 26, batch 500, train_loss[loss=3.456, NarTop10Accuracy=0.6294, over 6066.00 frames. ], tot_loss[loss=3.355, NarTop10Accuracy=0.6474, over 5425.37 frames. ], batch size: 11, lr: 3.30e-03
2024-08-06 12:31:59,782 INFO [trainer.py:765] (1/8) Epoch 26, batch 600, train_loss[loss=3.381, NarTop10Accuracy=0.6485, over 5927.00 frames. ], tot_loss[loss=3.364, NarTop10Accuracy=0.6454, over 5685.35 frames. ], batch size: 9, lr: 3.30e-03
2024-08-06 12:32:36,966 INFO [trainer.py:765] (1/8) Epoch 26, batch 700, train_loss[loss=3.41, NarTop10Accuracy=0.6433, over 4980.00 frames. ], tot_loss[loss=3.359, NarTop10Accuracy=0.6472, over 5753.14 frames. ], batch size: 6, lr: 3.30e-03
2024-08-06 12:33:10,809 INFO [trainer.py:765] (1/8) Epoch 26, batch 800, train_loss[loss=3.288, NarTop10Accuracy=0.6565, over 5041.00 frames. ], tot_loss[loss=3.361, NarTop10Accuracy=0.6461, over 5798.49 frames. ], batch size: 6, lr: 3.29e-03
2024-08-06 12:33:46,257 INFO [trainer.py:765] (1/8) Epoch 26, batch 900, train_loss[loss=3.587, NarTop10Accuracy=0.5918, over 6227.00 frames. ], tot_loss[loss=3.372, NarTop10Accuracy=0.6432, over 5806.82 frames. ], batch size: 13, lr: 3.29e-03
2024-08-06 12:34:22,903 INFO [trainer.py:765] (1/8) Epoch 26, batch 1000, train_loss[loss=3.133, NarTop10Accuracy=0.7004, over 6355.00 frames. ], tot_loss[loss=3.381, NarTop10Accuracy=0.6414, over 5916.11 frames. ], batch size: 13, lr: 3.29e-03
2024-08-06 12:34:57,798 INFO [trainer.py:765] (1/8) Epoch 26, batch 1100, train_loss[loss=3.346, NarTop10Accuracy=0.6655, over 6933.00 frames. ], tot_loss[loss=3.384, NarTop10Accuracy=0.6411, over 5971.29 frames. ], batch size: 17, lr: 3.29e-03
2024-08-06 12:35:31,893 INFO [trainer.py:765] (1/8) Epoch 26, batch 1200, train_loss[loss=3.303, NarTop10Accuracy=0.6647, over 7208.00 frames. ], tot_loss[loss=3.384, NarTop10Accuracy=0.641, over 5960.14 frames. ], batch size: 30, lr: 3.28e-03
2024-08-06 12:36:10,658 INFO [trainer.py:765] (1/8) Epoch 26, batch 1300, train_loss[loss=3.661, NarTop10Accuracy=0.5827, over 5012.00 frames. ], tot_loss[loss=3.381, NarTop10Accuracy=0.6409, over 6024.23 frames. ], batch size: 6, lr: 3.28e-03
2024-08-06 12:36:44,564 INFO [trainer.py:765] (1/8) Epoch 26, batch 1400, train_loss[loss=3.201, NarTop10Accuracy=0.684, over 5969.00 frames. ], tot_loss[loss=3.397, NarTop10Accuracy=0.6377, over 6032.14 frames. ], batch size: 11, lr: 3.28e-03
2024-08-06 12:37:03,593 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 12:37:13,567 INFO [trainer.py:811] (1/8) Epoch 26, validation: loss=3.231, NarTop10Accuracy=0.6753, over 1907754.00 frames. 
2024-08-06 12:37:13,568 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 12:37:14,077 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.547e+02 1.928e+02 2.102e+02 2.299e+02 4.602e+02, threshold=4.203e+02, percent-clipped=0.2
2024-08-06 12:37:23,027 INFO [trainer.py:765] (1/8) Epoch 26, batch 1500, train_loss[loss=3.751, NarTop10Accuracy=0.5732, over 6276.00 frames. ], tot_loss[loss=3.395, NarTop10Accuracy=0.6387, over 5961.31 frames. ], batch size: 49, lr: 3.28e-03
2024-08-06 12:37:51,061 INFO [trainer.py:765] (1/8) Epoch 26, batch 1600, train_loss[loss=3.481, NarTop10Accuracy=0.616, over 7049.00 frames. ], tot_loss[loss=3.394, NarTop10Accuracy=0.6388, over 5948.74 frames. ], batch size: 22, lr: 3.27e-03
2024-08-06 12:38:17,853 INFO [trainer.py:765] (1/8) Epoch 26, batch 1700, train_loss[loss=3.562, NarTop10Accuracy=0.6063, over 6656.00 frames. ], tot_loss[loss=3.411, NarTop10Accuracy=0.6358, over 5944.23 frames. ], batch size: 14, lr: 3.27e-03
2024-08-06 12:38:44,383 INFO [trainer.py:765] (1/8) Epoch 26, batch 1800, train_loss[loss=3.149, NarTop10Accuracy=0.6865, over 6986.00 frames. ], tot_loss[loss=3.407, NarTop10Accuracy=0.6365, over 5999.27 frames. ], batch size: 22, lr: 3.27e-03
2024-08-06 12:39:10,952 INFO [trainer.py:765] (1/8) Epoch 26, batch 1900, train_loss[loss=3.584, NarTop10Accuracy=0.6003, over 5926.00 frames. ], tot_loss[loss=3.415, NarTop10Accuracy=0.6344, over 6040.86 frames. ], batch size: 51, lr: 3.27e-03
2024-08-06 12:39:36,610 INFO [trainer.py:765] (1/8) Epoch 26, batch 2000, train_loss[loss=3.492, NarTop10Accuracy=0.6175, over 5953.00 frames. ], tot_loss[loss=3.407, NarTop10Accuracy=0.6359, over 6016.74 frames. ], batch size: 48, lr: 3.26e-03
2024-08-06 12:40:02,147 INFO [trainer.py:765] (1/8) Epoch 26, batch 2100, train_loss[loss=3.319, NarTop10Accuracy=0.6452, over 4885.00 frames. ], tot_loss[loss=3.393, NarTop10Accuracy=0.639, over 6013.44 frames. ], batch size: 5, lr: 3.26e-03
2024-08-06 12:40:27,758 INFO [trainer.py:765] (1/8) Epoch 26, batch 2200, train_loss[loss=3.319, NarTop10Accuracy=0.6472, over 7058.00 frames. ], tot_loss[loss=3.398, NarTop10Accuracy=0.6383, over 6059.59 frames. ], batch size: 30, lr: 3.26e-03
2024-08-06 12:40:53,232 INFO [trainer.py:765] (1/8) Epoch 26, batch 2300, train_loss[loss=3.286, NarTop10Accuracy=0.6723, over 5635.00 frames. ], tot_loss[loss=3.414, NarTop10Accuracy=0.6351, over 6104.21 frames. ], batch size: 9, lr: 3.26e-03
2024-08-06 12:41:17,930 INFO [trainer.py:765] (1/8) Epoch 26, batch 2400, train_loss[loss=3.21, NarTop10Accuracy=0.6678, over 5265.00 frames. ], tot_loss[loss=3.414, NarTop10Accuracy=0.6355, over 5900.80 frames. ], batch size: 7, lr: 3.25e-03
2024-08-06 12:41:44,477 INFO [trainer.py:765] (1/8) Epoch 26, batch 2500, train_loss[loss=2.852, NarTop10Accuracy=0.7464, over 5143.00 frames. ], tot_loss[loss=3.389, NarTop10Accuracy=0.6406, over 5553.22 frames. ], batch size: 6, lr: 3.25e-03
2024-08-06 12:42:05,684 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 12:43:12,533 INFO [trainer.py:765] (1/8) Epoch 27, batch 100, train_loss[loss=3.58, NarTop10Accuracy=0.6076, over 7302.00 frames. ], tot_loss[loss=3.363, NarTop10Accuracy=0.6466, over 2375.60 frames. ], batch size: 31, lr: 3.19e-03
2024-08-06 12:43:43,576 INFO [trainer.py:765] (1/8) Epoch 27, batch 200, train_loss[loss=3.626, NarTop10Accuracy=0.5877, over 6773.00 frames. ], tot_loss[loss=3.344, NarTop10Accuracy=0.6504, over 3864.41 frames. ], batch size: 17, lr: 3.18e-03
2024-08-06 12:44:13,786 INFO [trainer.py:765] (1/8) Epoch 27, batch 300, train_loss[loss=3.178, NarTop10Accuracy=0.6838, over 7099.00 frames. ], tot_loss[loss=3.333, NarTop10Accuracy=0.6526, over 4684.12 frames. ], batch size: 22, lr: 3.18e-03
2024-08-06 12:44:50,461 INFO [trainer.py:765] (1/8) Epoch 27, batch 400, train_loss[loss=3.069, NarTop10Accuracy=0.7087, over 5223.00 frames. ], tot_loss[loss=3.339, NarTop10Accuracy=0.6513, over 5140.72 frames. ], batch size: 7, lr: 3.18e-03
2024-08-06 12:45:20,670 INFO [trainer.py:765] (1/8) Epoch 27, batch 500, train_loss[loss=3.183, NarTop10Accuracy=0.677, over 6091.00 frames. ], tot_loss[loss=3.335, NarTop10Accuracy=0.6512, over 5423.78 frames. ], batch size: 11, lr: 3.18e-03
2024-08-06 12:45:55,261 INFO [trainer.py:765] (1/8) Epoch 27, batch 600, train_loss[loss=3.312, NarTop10Accuracy=0.654, over 5775.00 frames. ], tot_loss[loss=3.348, NarTop10Accuracy=0.6491, over 5686.82 frames. ], batch size: 9, lr: 3.17e-03
2024-08-06 12:46:26,747 INFO [trainer.py:765] (1/8) Epoch 27, batch 700, train_loss[loss=3.617, NarTop10Accuracy=0.6008, over 5052.00 frames. ], tot_loss[loss=3.345, NarTop10Accuracy=0.6496, over 5752.25 frames. ], batch size: 6, lr: 3.17e-03
2024-08-06 12:47:05,016 INFO [trainer.py:765] (1/8) Epoch 27, batch 800, train_loss[loss=3.331, NarTop10Accuracy=0.6561, over 5046.00 frames. ], tot_loss[loss=3.366, NarTop10Accuracy=0.6455, over 5797.62 frames. ], batch size: 6, lr: 3.17e-03
2024-08-06 12:47:32,742 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 12:47:42,765 INFO [trainer.py:811] (1/8) Epoch 27, validation: loss=3.258, NarTop10Accuracy=0.6695, over 1907754.00 frames. 
2024-08-06 12:47:42,766 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 12:47:43,335 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.554e+02 1.939e+02 2.100e+02 2.298e+02 4.859e+02, threshold=4.201e+02, percent-clipped=0.2
2024-08-06 12:47:47,259 INFO [trainer.py:765] (1/8) Epoch 27, batch 900, train_loss[loss=3.29, NarTop10Accuracy=0.653, over 6360.00 frames. ], tot_loss[loss=3.367, NarTop10Accuracy=0.6449, over 5820.17 frames. ], batch size: 13, lr: 3.17e-03
2024-08-06 12:48:22,861 INFO [trainer.py:765] (1/8) Epoch 27, batch 1000, train_loss[loss=3.389, NarTop10Accuracy=0.6375, over 6084.00 frames. ], tot_loss[loss=3.366, NarTop10Accuracy=0.645, over 5911.67 frames. ], batch size: 13, lr: 3.17e-03
2024-08-06 12:48:58,083 INFO [trainer.py:765] (1/8) Epoch 27, batch 1100, train_loss[loss=3.635, NarTop10Accuracy=0.585, over 6830.00 frames. ], tot_loss[loss=3.37, NarTop10Accuracy=0.6439, over 5947.76 frames. ], batch size: 17, lr: 3.16e-03
2024-08-06 12:49:34,895 INFO [trainer.py:765] (1/8) Epoch 27, batch 1200, train_loss[loss=3.109, NarTop10Accuracy=0.6833, over 7068.00 frames. ], tot_loss[loss=3.37, NarTop10Accuracy=0.6435, over 5936.68 frames. ], batch size: 30, lr: 3.16e-03
2024-08-06 12:50:06,240 INFO [trainer.py:765] (1/8) Epoch 27, batch 1300, train_loss[loss=3.261, NarTop10Accuracy=0.6581, over 5103.00 frames. ], tot_loss[loss=3.376, NarTop10Accuracy=0.6424, over 6007.23 frames. ], batch size: 6, lr: 3.16e-03
2024-08-06 12:50:42,949 INFO [trainer.py:765] (1/8) Epoch 27, batch 1400, train_loss[loss=3.105, NarTop10Accuracy=0.6987, over 6125.00 frames. ], tot_loss[loss=3.387, NarTop10Accuracy=0.6408, over 6021.67 frames. ], batch size: 11, lr: 3.16e-03
2024-08-06 12:51:11,276 INFO [trainer.py:765] (1/8) Epoch 27, batch 1500, train_loss[loss=3.525, NarTop10Accuracy=0.6102, over 5586.00 frames. ], tot_loss[loss=3.386, NarTop10Accuracy=0.6409, over 5968.43 frames. ], batch size: 49, lr: 3.15e-03
2024-08-06 12:51:39,351 INFO [trainer.py:765] (1/8) Epoch 27, batch 1600, train_loss[loss=3.433, NarTop10Accuracy=0.6344, over 7292.00 frames. ], tot_loss[loss=3.396, NarTop10Accuracy=0.6388, over 5944.53 frames. ], batch size: 22, lr: 3.15e-03
2024-08-06 12:52:06,060 INFO [trainer.py:765] (1/8) Epoch 27, batch 1700, train_loss[loss=3.452, NarTop10Accuracy=0.6289, over 6327.00 frames. ], tot_loss[loss=3.39, NarTop10Accuracy=0.6399, over 5934.65 frames. ], batch size: 13, lr: 3.15e-03
2024-08-06 12:52:32,667 INFO [trainer.py:765] (1/8) Epoch 27, batch 1800, train_loss[loss=3.398, NarTop10Accuracy=0.6444, over 7142.00 frames. ], tot_loss[loss=3.384, NarTop10Accuracy=0.6414, over 5999.12 frames. ], batch size: 22, lr: 3.15e-03
2024-08-06 12:53:02,287 INFO [trainer.py:765] (1/8) Epoch 27, batch 1900, train_loss[loss=3.737, NarTop10Accuracy=0.5684, over 5856.00 frames. ], tot_loss[loss=3.401, NarTop10Accuracy=0.6382, over 6034.75 frames. ], batch size: 50, lr: 3.14e-03
2024-08-06 12:53:27,997 INFO [trainer.py:765] (1/8) Epoch 27, batch 2000, train_loss[loss=3.517, NarTop10Accuracy=0.6152, over 6059.00 frames. ], tot_loss[loss=3.396, NarTop10Accuracy=0.6389, over 6016.10 frames. ], batch size: 49, lr: 3.14e-03
2024-08-06 12:53:53,537 INFO [trainer.py:765] (1/8) Epoch 27, batch 2100, train_loss[loss=3.467, NarTop10Accuracy=0.6165, over 4836.00 frames. ], tot_loss[loss=3.4, NarTop10Accuracy=0.6381, over 5999.58 frames. ], batch size: 5, lr: 3.14e-03
2024-08-06 12:54:18,996 INFO [trainer.py:765] (1/8) Epoch 27, batch 2200, train_loss[loss=3.329, NarTop10Accuracy=0.655, over 7415.00 frames. ], tot_loss[loss=3.396, NarTop10Accuracy=0.6388, over 6046.88 frames. ], batch size: 31, lr: 3.14e-03
2024-08-06 12:54:44,479 INFO [trainer.py:765] (1/8) Epoch 27, batch 2300, train_loss[loss=3.085, NarTop10Accuracy=0.7001, over 5786.00 frames. ], tot_loss[loss=3.411, NarTop10Accuracy=0.6356, over 6076.49 frames. ], batch size: 9, lr: 3.14e-03
2024-08-06 12:55:09,217 INFO [trainer.py:765] (1/8) Epoch 27, batch 2400, train_loss[loss=3.623, NarTop10Accuracy=0.5887, over 5912.00 frames. ], tot_loss[loss=3.432, NarTop10Accuracy=0.632, over 5890.64 frames. ], batch size: 50, lr: 3.13e-03
2024-08-06 12:55:32,725 INFO [trainer.py:765] (1/8) Epoch 27, batch 2500, train_loss[loss=3.398, NarTop10Accuracy=0.6418, over 5063.00 frames. ], tot_loss[loss=3.394, NarTop10Accuracy=0.6385, over 5547.22 frames. ], batch size: 6, lr: 3.13e-03
2024-08-06 12:55:54,037 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 12:56:46,803 INFO [trainer.py:765] (1/8) Epoch 28, batch 100, train_loss[loss=3.235, NarTop10Accuracy=0.6791, over 7374.00 frames. ], tot_loss[loss=3.32, NarTop10Accuracy=0.6542, over 2359.89 frames. ], batch size: 31, lr: 3.07e-03
2024-08-06 12:57:23,204 INFO [trainer.py:765] (1/8) Epoch 28, batch 200, train_loss[loss=3.351, NarTop10Accuracy=0.6491, over 6996.00 frames. ], tot_loss[loss=3.345, NarTop10Accuracy=0.6498, over 3863.39 frames. ], batch size: 17, lr: 3.07e-03
2024-08-06 12:57:55,704 INFO [trainer.py:765] (1/8) Epoch 28, batch 300, train_loss[loss=3.433, NarTop10Accuracy=0.634, over 7063.00 frames. ], tot_loss[loss=3.354, NarTop10Accuracy=0.6483, over 4674.01 frames. ], batch size: 22, lr: 3.07e-03
2024-08-06 12:57:56,457 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 12:58:06,828 INFO [trainer.py:811] (1/8) Epoch 28, validation: loss=3.275, NarTop10Accuracy=0.6665, over 1907754.00 frames. 
2024-08-06 12:58:06,828 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 12:58:07,334 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.570e+02 1.944e+02 2.106e+02 2.298e+02 4.786e+02, threshold=4.211e+02, percent-clipped=0.1
2024-08-06 12:58:34,932 INFO [trainer.py:765] (1/8) Epoch 28, batch 400, train_loss[loss=3.528, NarTop10Accuracy=0.6163, over 5234.00 frames. ], tot_loss[loss=3.362, NarTop10Accuracy=0.6467, over 5124.11 frames. ], batch size: 7, lr: 3.06e-03
2024-08-06 12:59:11,437 INFO [trainer.py:765] (1/8) Epoch 28, batch 500, train_loss[loss=3.196, NarTop10Accuracy=0.6929, over 6289.00 frames. ], tot_loss[loss=3.347, NarTop10Accuracy=0.649, over 5392.89 frames. ], batch size: 11, lr: 3.06e-03
2024-08-06 12:59:44,487 INFO [trainer.py:765] (1/8) Epoch 28, batch 600, train_loss[loss=3.382, NarTop10Accuracy=0.6443, over 5799.00 frames. ], tot_loss[loss=3.346, NarTop10Accuracy=0.6489, over 5679.19 frames. ], batch size: 9, lr: 3.06e-03
2024-08-06 13:00:20,012 INFO [trainer.py:765] (1/8) Epoch 28, batch 700, train_loss[loss=3.513, NarTop10Accuracy=0.6145, over 5080.00 frames. ], tot_loss[loss=3.356, NarTop10Accuracy=0.6471, over 5759.72 frames. ], batch size: 6, lr: 3.06e-03
2024-08-06 13:00:56,433 INFO [trainer.py:765] (1/8) Epoch 28, batch 800, train_loss[loss=3.292, NarTop10Accuracy=0.6483, over 5021.00 frames. ], tot_loss[loss=3.353, NarTop10Accuracy=0.6477, over 5807.41 frames. ], batch size: 6, lr: 3.05e-03
2024-08-06 13:01:31,042 INFO [trainer.py:765] (1/8) Epoch 28, batch 900, train_loss[loss=3.08, NarTop10Accuracy=0.7002, over 6297.00 frames. ], tot_loss[loss=3.36, NarTop10Accuracy=0.646, over 5816.83 frames. ], batch size: 13, lr: 3.05e-03
2024-08-06 13:02:06,494 INFO [trainer.py:765] (1/8) Epoch 28, batch 1000, train_loss[loss=3.444, NarTop10Accuracy=0.6202, over 6228.00 frames. ], tot_loss[loss=3.375, NarTop10Accuracy=0.643, over 5909.26 frames. ], batch size: 13, lr: 3.05e-03
2024-08-06 13:02:41,229 INFO [trainer.py:765] (1/8) Epoch 28, batch 1100, train_loss[loss=3.325, NarTop10Accuracy=0.6478, over 6914.00 frames. ], tot_loss[loss=3.38, NarTop10Accuracy=0.6422, over 5943.80 frames. ], batch size: 17, lr: 3.05e-03
2024-08-06 13:03:16,895 INFO [trainer.py:765] (1/8) Epoch 28, batch 1200, train_loss[loss=3.48, NarTop10Accuracy=0.6158, over 7265.00 frames. ], tot_loss[loss=3.386, NarTop10Accuracy=0.6407, over 5936.90 frames. ], batch size: 30, lr: 3.05e-03
2024-08-06 13:03:54,153 INFO [trainer.py:765] (1/8) Epoch 28, batch 1300, train_loss[loss=3.25, NarTop10Accuracy=0.6625, over 5089.00 frames. ], tot_loss[loss=3.382, NarTop10Accuracy=0.6417, over 6026.19 frames. ], batch size: 6, lr: 3.04e-03
2024-08-06 13:04:28,712 INFO [trainer.py:765] (1/8) Epoch 28, batch 1400, train_loss[loss=3.321, NarTop10Accuracy=0.6411, over 6187.00 frames. ], tot_loss[loss=3.391, NarTop10Accuracy=0.64, over 6054.31 frames. ], batch size: 11, lr: 3.04e-03
2024-08-06 13:05:02,348 INFO [trainer.py:765] (1/8) Epoch 28, batch 1500, train_loss[loss=3.465, NarTop10Accuracy=0.6316, over 6316.00 frames. ], tot_loss[loss=3.379, NarTop10Accuracy=0.642, over 5981.94 frames. ], batch size: 49, lr: 3.04e-03
2024-08-06 13:05:30,370 INFO [trainer.py:765] (1/8) Epoch 28, batch 1600, train_loss[loss=3.645, NarTop10Accuracy=0.5873, over 6960.00 frames. ], tot_loss[loss=3.389, NarTop10Accuracy=0.6402, over 5966.66 frames. ], batch size: 22, lr: 3.04e-03
2024-08-06 13:05:57,130 INFO [trainer.py:765] (1/8) Epoch 28, batch 1700, train_loss[loss=3.712, NarTop10Accuracy=0.5735, over 6211.00 frames. ], tot_loss[loss=3.38, NarTop10Accuracy=0.6418, over 5953.09 frames. ], batch size: 13, lr: 3.04e-03
2024-08-06 13:06:23,732 INFO [trainer.py:765] (1/8) Epoch 28, batch 1800, train_loss[loss=3.564, NarTop10Accuracy=0.6116, over 7220.00 frames. ], tot_loss[loss=3.371, NarTop10Accuracy=0.6435, over 6023.31 frames. ], batch size: 22, lr: 3.03e-03
2024-08-06 13:06:50,372 INFO [trainer.py:765] (1/8) Epoch 28, batch 1900, train_loss[loss=3.497, NarTop10Accuracy=0.615, over 5875.00 frames. ], tot_loss[loss=3.386, NarTop10Accuracy=0.6406, over 6058.11 frames. ], batch size: 49, lr: 3.03e-03
2024-08-06 13:07:16,115 INFO [trainer.py:765] (1/8) Epoch 28, batch 2000, train_loss[loss=3.503, NarTop10Accuracy=0.6189, over 6270.00 frames. ], tot_loss[loss=3.371, NarTop10Accuracy=0.6434, over 6032.42 frames. ], batch size: 49, lr: 3.03e-03
2024-08-06 13:07:41,546 INFO [trainer.py:765] (1/8) Epoch 28, batch 2100, train_loss[loss=3.852, NarTop10Accuracy=0.5468, over 3921.00 frames. ], tot_loss[loss=3.387, NarTop10Accuracy=0.64, over 6011.45 frames. ], batch size: 4, lr: 3.03e-03
2024-08-06 13:08:06,931 INFO [trainer.py:765] (1/8) Epoch 28, batch 2200, train_loss[loss=3.427, NarTop10Accuracy=0.6332, over 7298.00 frames. ], tot_loss[loss=3.396, NarTop10Accuracy=0.6389, over 6054.75 frames. ], batch size: 31, lr: 3.02e-03
2024-08-06 13:08:32,387 INFO [trainer.py:765] (1/8) Epoch 28, batch 2300, train_loss[loss=3.39, NarTop10Accuracy=0.6474, over 5824.00 frames. ], tot_loss[loss=3.401, NarTop10Accuracy=0.6378, over 6077.81 frames. ], batch size: 9, lr: 3.02e-03
2024-08-06 13:08:33,134 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 13:08:43,385 INFO [trainer.py:811] (1/8) Epoch 28, validation: loss=3.224, NarTop10Accuracy=0.676, over 1907754.00 frames. 
2024-08-06 13:08:43,386 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 13:08:43,890 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.605e+02 1.997e+02 2.131e+02 2.314e+02 6.875e+02, threshold=4.261e+02, percent-clipped=0.5
2024-08-06 13:09:07,389 INFO [trainer.py:765] (1/8) Epoch 28, batch 2400, train_loss[loss=3.863, NarTop10Accuracy=0.5462, over 6486.00 frames. ], tot_loss[loss=3.409, NarTop10Accuracy=0.6362, over 5896.61 frames. ], batch size: 48, lr: 3.02e-03
2024-08-06 13:09:30,781 INFO [trainer.py:765] (1/8) Epoch 28, batch 2500, train_loss[loss=3.478, NarTop10Accuracy=0.623, over 4981.00 frames. ], tot_loss[loss=3.383, NarTop10Accuracy=0.6407, over 5551.94 frames. ], batch size: 6, lr: 3.02e-03
2024-08-06 13:09:51,898 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 13:10:48,193 INFO [trainer.py:765] (1/8) Epoch 29, batch 100, train_loss[loss=3.586, NarTop10Accuracy=0.5999, over 6997.00 frames. ], tot_loss[loss=3.312, NarTop10Accuracy=0.6568, over 2361.53 frames. ], batch size: 31, lr: 2.96e-03
2024-08-06 13:11:20,841 INFO [trainer.py:765] (1/8) Epoch 29, batch 200, train_loss[loss=3.567, NarTop10Accuracy=0.6086, over 6883.00 frames. ], tot_loss[loss=3.325, NarTop10Accuracy=0.6541, over 3865.29 frames. ], batch size: 17, lr: 2.96e-03
2024-08-06 13:11:56,950 INFO [trainer.py:765] (1/8) Epoch 29, batch 300, train_loss[loss=3.226, NarTop10Accuracy=0.6838, over 7061.00 frames. ], tot_loss[loss=3.332, NarTop10Accuracy=0.6527, over 4669.10 frames. ], batch size: 22, lr: 2.96e-03
2024-08-06 13:12:29,716 INFO [trainer.py:765] (1/8) Epoch 29, batch 400, train_loss[loss=3.43, NarTop10Accuracy=0.6285, over 5166.00 frames. ], tot_loss[loss=3.331, NarTop10Accuracy=0.6528, over 5129.23 frames. ], batch size: 7, lr: 2.96e-03
2024-08-06 13:12:59,921 INFO [trainer.py:765] (1/8) Epoch 29, batch 500, train_loss[loss=3.449, NarTop10Accuracy=0.6258, over 6160.00 frames. ], tot_loss[loss=3.336, NarTop10Accuracy=0.6518, over 5397.90 frames. ], batch size: 11, lr: 2.95e-03
2024-08-06 13:13:33,547 INFO [trainer.py:765] (1/8) Epoch 29, batch 600, train_loss[loss=3.733, NarTop10Accuracy=0.5699, over 5792.00 frames. ], tot_loss[loss=3.346, NarTop10Accuracy=0.6493, over 5672.27 frames. ], batch size: 9, lr: 2.95e-03
2024-08-06 13:14:09,937 INFO [trainer.py:765] (1/8) Epoch 29, batch 700, train_loss[loss=3.65, NarTop10Accuracy=0.5889, over 5057.00 frames. ], tot_loss[loss=3.352, NarTop10Accuracy=0.648, over 5747.72 frames. ], batch size: 6, lr: 2.95e-03
2024-08-06 13:14:46,676 INFO [trainer.py:765] (1/8) Epoch 29, batch 800, train_loss[loss=3.564, NarTop10Accuracy=0.6027, over 4996.00 frames. ], tot_loss[loss=3.383, NarTop10Accuracy=0.6415, over 5806.95 frames. ], batch size: 6, lr: 2.95e-03
2024-08-06 13:15:17,114 INFO [trainer.py:765] (1/8) Epoch 29, batch 900, train_loss[loss=3.178, NarTop10Accuracy=0.6823, over 6329.00 frames. ], tot_loss[loss=3.375, NarTop10Accuracy=0.6432, over 5816.56 frames. ], batch size: 13, lr: 2.95e-03
2024-08-06 13:15:59,363 INFO [trainer.py:765] (1/8) Epoch 29, batch 1000, train_loss[loss=3.564, NarTop10Accuracy=0.6025, over 6184.00 frames. ], tot_loss[loss=3.367, NarTop10Accuracy=0.6448, over 5918.73 frames. ], batch size: 13, lr: 2.94e-03
2024-08-06 13:16:31,713 INFO [trainer.py:765] (1/8) Epoch 29, batch 1100, train_loss[loss=3.468, NarTop10Accuracy=0.6217, over 6479.00 frames. ], tot_loss[loss=3.378, NarTop10Accuracy=0.6423, over 5956.13 frames. ], batch size: 16, lr: 2.94e-03
2024-08-06 13:17:04,933 INFO [trainer.py:765] (1/8) Epoch 29, batch 1200, train_loss[loss=3.507, NarTop10Accuracy=0.6226, over 7130.00 frames. ], tot_loss[loss=3.377, NarTop10Accuracy=0.6425, over 5939.86 frames. ], batch size: 30, lr: 2.94e-03
2024-08-06 13:17:43,957 INFO [trainer.py:765] (1/8) Epoch 29, batch 1300, train_loss[loss=3.2, NarTop10Accuracy=0.6839, over 5165.00 frames. ], tot_loss[loss=3.371, NarTop10Accuracy=0.6436, over 6014.87 frames. ], batch size: 6, lr: 2.94e-03
2024-08-06 13:18:17,924 INFO [trainer.py:765] (1/8) Epoch 29, batch 1400, train_loss[loss=3.702, NarTop10Accuracy=0.5711, over 6182.00 frames. ], tot_loss[loss=3.389, NarTop10Accuracy=0.64, over 6034.77 frames. ], batch size: 11, lr: 2.94e-03
2024-08-06 13:18:48,306 INFO [trainer.py:765] (1/8) Epoch 29, batch 1500, train_loss[loss=3.812, NarTop10Accuracy=0.5517, over 6208.00 frames. ], tot_loss[loss=3.384, NarTop10Accuracy=0.6412, over 5979.98 frames. ], batch size: 50, lr: 2.93e-03
2024-08-06 13:19:16,409 INFO [trainer.py:765] (1/8) Epoch 29, batch 1600, train_loss[loss=3.303, NarTop10Accuracy=0.666, over 7183.00 frames. ], tot_loss[loss=3.394, NarTop10Accuracy=0.6394, over 5954.54 frames. ], batch size: 22, lr: 2.93e-03
2024-08-06 13:19:43,242 INFO [trainer.py:765] (1/8) Epoch 29, batch 1700, train_loss[loss=3.269, NarTop10Accuracy=0.6688, over 6264.00 frames. ], tot_loss[loss=3.392, NarTop10Accuracy=0.6397, over 5929.50 frames. ], batch size: 13, lr: 2.93e-03
2024-08-06 13:19:49,091 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 13:19:59,386 INFO [trainer.py:811] (1/8) Epoch 29, validation: loss=3.233, NarTop10Accuracy=0.6754, over 1907754.00 frames. 
2024-08-06 13:19:59,387 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 13:19:59,903 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.560e+02 1.964e+02 2.123e+02 2.299e+02 5.520e+02, threshold=4.246e+02, percent-clipped=0.2
2024-08-06 13:20:20,108 INFO [trainer.py:765] (1/8) Epoch 29, batch 1800, train_loss[loss=3.416, NarTop10Accuracy=0.6369, over 7086.00 frames. ], tot_loss[loss=3.386, NarTop10Accuracy=0.6407, over 6005.15 frames. ], batch size: 22, lr: 2.93e-03
2024-08-06 13:20:46,844 INFO [trainer.py:765] (1/8) Epoch 29, batch 1900, train_loss[loss=3.466, NarTop10Accuracy=0.6209, over 5938.00 frames. ], tot_loss[loss=3.402, NarTop10Accuracy=0.6377, over 6048.21 frames. ], batch size: 49, lr: 2.93e-03
2024-08-06 13:21:12,479 INFO [trainer.py:765] (1/8) Epoch 29, batch 2000, train_loss[loss=3.647, NarTop10Accuracy=0.5935, over 6578.00 frames. ], tot_loss[loss=3.392, NarTop10Accuracy=0.6391, over 6003.50 frames. ], batch size: 49, lr: 2.92e-03
2024-08-06 13:21:37,983 INFO [trainer.py:765] (1/8) Epoch 29, batch 2100, train_loss[loss=3.447, NarTop10Accuracy=0.6258, over 3880.00 frames. ], tot_loss[loss=3.396, NarTop10Accuracy=0.6386, over 5978.66 frames. ], batch size: 4, lr: 2.92e-03
2024-08-06 13:22:03,360 INFO [trainer.py:765] (1/8) Epoch 29, batch 2200, train_loss[loss=3.252, NarTop10Accuracy=0.6714, over 7259.00 frames. ], tot_loss[loss=3.394, NarTop10Accuracy=0.6392, over 6012.46 frames. ], batch size: 31, lr: 2.92e-03
2024-08-06 13:22:28,831 INFO [trainer.py:765] (1/8) Epoch 29, batch 2300, train_loss[loss=3.428, NarTop10Accuracy=0.6312, over 5837.00 frames. ], tot_loss[loss=3.409, NarTop10Accuracy=0.6367, over 6060.26 frames. ], batch size: 9, lr: 2.92e-03
2024-08-06 13:22:53,621 INFO [trainer.py:765] (1/8) Epoch 29, batch 2400, train_loss[loss=3.421, NarTop10Accuracy=0.6291, over 5771.00 frames. ], tot_loss[loss=3.409, NarTop10Accuracy=0.6367, over 5866.90 frames. ], batch size: 8, lr: 2.92e-03
2024-08-06 13:23:16,979 INFO [trainer.py:765] (1/8) Epoch 29, batch 2500, train_loss[loss=3.313, NarTop10Accuracy=0.6467, over 5163.00 frames. ], tot_loss[loss=3.394, NarTop10Accuracy=0.6394, over 5528.01 frames. ], batch size: 6, lr: 2.91e-03
2024-08-06 13:23:38,351 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 13:24:38,391 INFO [trainer.py:765] (1/8) Epoch 30, batch 100, train_loss[loss=3.272, NarTop10Accuracy=0.6639, over 7385.00 frames. ], tot_loss[loss=3.292, NarTop10Accuracy=0.6612, over 2374.95 frames. ], batch size: 33, lr: 2.86e-03
2024-08-06 13:25:14,782 INFO [trainer.py:765] (1/8) Epoch 30, batch 200, train_loss[loss=3.397, NarTop10Accuracy=0.637, over 6601.00 frames. ], tot_loss[loss=3.303, NarTop10Accuracy=0.6588, over 3866.22 frames. ], batch size: 17, lr: 2.86e-03
2024-08-06 13:25:46,846 INFO [trainer.py:765] (1/8) Epoch 30, batch 300, train_loss[loss=3.189, NarTop10Accuracy=0.6807, over 6996.00 frames. ], tot_loss[loss=3.322, NarTop10Accuracy=0.655, over 4668.30 frames. ], batch size: 22, lr: 2.86e-03
2024-08-06 13:26:17,538 INFO [trainer.py:765] (1/8) Epoch 30, batch 400, train_loss[loss=3.371, NarTop10Accuracy=0.6531, over 5296.00 frames. ], tot_loss[loss=3.334, NarTop10Accuracy=0.652, over 5135.72 frames. ], batch size: 7, lr: 2.86e-03
2024-08-06 13:26:53,919 INFO [trainer.py:765] (1/8) Epoch 30, batch 500, train_loss[loss=3.299, NarTop10Accuracy=0.6705, over 6135.00 frames. ], tot_loss[loss=3.317, NarTop10Accuracy=0.6556, over 5409.30 frames. ], batch size: 11, lr: 2.85e-03
2024-08-06 13:27:25,422 INFO [trainer.py:765] (1/8) Epoch 30, batch 600, train_loss[loss=3.253, NarTop10Accuracy=0.678, over 5808.00 frames. ], tot_loss[loss=3.326, NarTop10Accuracy=0.6534, over 5672.13 frames. ], batch size: 9, lr: 2.85e-03
2024-08-06 13:28:00,307 INFO [trainer.py:765] (1/8) Epoch 30, batch 700, train_loss[loss=3.877, NarTop10Accuracy=0.5425, over 4362.00 frames. ], tot_loss[loss=3.33, NarTop10Accuracy=0.6525, over 5731.46 frames. ], batch size: 5, lr: 2.85e-03
2024-08-06 13:28:37,477 INFO [trainer.py:765] (1/8) Epoch 30, batch 800, train_loss[loss=3.453, NarTop10Accuracy=0.6212, over 5004.00 frames. ], tot_loss[loss=3.353, NarTop10Accuracy=0.648, over 5797.63 frames. ], batch size: 6, lr: 2.85e-03
2024-08-06 13:29:10,425 INFO [trainer.py:765] (1/8) Epoch 30, batch 900, train_loss[loss=3.323, NarTop10Accuracy=0.6497, over 6625.00 frames. ], tot_loss[loss=3.364, NarTop10Accuracy=0.6454, over 5811.98 frames. ], batch size: 14, lr: 2.85e-03
2024-08-06 13:29:45,913 INFO [trainer.py:765] (1/8) Epoch 30, batch 1000, train_loss[loss=3.3, NarTop10Accuracy=0.6584, over 6198.00 frames. ], tot_loss[loss=3.371, NarTop10Accuracy=0.6439, over 5917.32 frames. ], batch size: 13, lr: 2.84e-03
2024-08-06 13:30:24,171 INFO [trainer.py:765] (1/8) Epoch 30, batch 1100, train_loss[loss=3.423, NarTop10Accuracy=0.6376, over 6842.00 frames. ], tot_loss[loss=3.374, NarTop10Accuracy=0.6432, over 5954.52 frames. ], batch size: 17, lr: 2.84e-03
2024-08-06 13:30:38,001 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 13:30:48,195 INFO [trainer.py:811] (1/8) Epoch 30, validation: loss=3.239, NarTop10Accuracy=0.6729, over 1907754.00 frames. 
2024-08-06 13:30:48,196 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 13:30:48,916 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.612e+02 1.985e+02 2.139e+02 2.326e+02 4.628e+02, threshold=4.279e+02, percent-clipped=0.1
2024-08-06 13:31:05,665 INFO [trainer.py:765] (1/8) Epoch 30, batch 1200, train_loss[loss=3.348, NarTop10Accuracy=0.6451, over 7234.00 frames. ], tot_loss[loss=3.383, NarTop10Accuracy=0.641, over 5954.47 frames. ], batch size: 31, lr: 2.84e-03
2024-08-06 13:31:43,020 INFO [trainer.py:765] (1/8) Epoch 30, batch 1300, train_loss[loss=3.342, NarTop10Accuracy=0.6526, over 5104.00 frames. ], tot_loss[loss=3.372, NarTop10Accuracy=0.6431, over 6033.30 frames. ], batch size: 6, lr: 2.84e-03
2024-08-06 13:32:19,325 INFO [trainer.py:765] (1/8) Epoch 30, batch 1400, train_loss[loss=3.62, NarTop10Accuracy=0.5948, over 6132.00 frames. ], tot_loss[loss=3.383, NarTop10Accuracy=0.641, over 6034.86 frames. ], batch size: 11, lr: 2.84e-03
2024-08-06 13:32:52,335 INFO [trainer.py:765] (1/8) Epoch 30, batch 1500, train_loss[loss=3.599, NarTop10Accuracy=0.6073, over 6150.00 frames. ], tot_loss[loss=3.384, NarTop10Accuracy=0.6409, over 5981.39 frames. ], batch size: 48, lr: 2.83e-03
2024-08-06 13:33:20,408 INFO [trainer.py:765] (1/8) Epoch 30, batch 1600, train_loss[loss=3.472, NarTop10Accuracy=0.6266, over 6972.00 frames. ], tot_loss[loss=3.388, NarTop10Accuracy=0.6399, over 5955.81 frames. ], batch size: 22, lr: 2.83e-03
2024-08-06 13:33:47,200 INFO [trainer.py:765] (1/8) Epoch 30, batch 1700, train_loss[loss=3.68, NarTop10Accuracy=0.5838, over 6674.00 frames. ], tot_loss[loss=3.392, NarTop10Accuracy=0.6395, over 5931.46 frames. ], batch size: 14, lr: 2.83e-03
2024-08-06 13:34:13,887 INFO [trainer.py:765] (1/8) Epoch 30, batch 1800, train_loss[loss=3.562, NarTop10Accuracy=0.598, over 7276.00 frames. ], tot_loss[loss=3.381, NarTop10Accuracy=0.6417, over 5999.50 frames. ], batch size: 22, lr: 2.83e-03
2024-08-06 13:34:40,548 INFO [trainer.py:765] (1/8) Epoch 30, batch 1900, train_loss[loss=3.69, NarTop10Accuracy=0.5816, over 6319.00 frames. ], tot_loss[loss=3.395, NarTop10Accuracy=0.639, over 6032.48 frames. ], batch size: 49, lr: 2.83e-03
2024-08-06 13:35:06,315 INFO [trainer.py:765] (1/8) Epoch 30, batch 2000, train_loss[loss=3.669, NarTop10Accuracy=0.583, over 5821.00 frames. ], tot_loss[loss=3.387, NarTop10Accuracy=0.6407, over 6008.59 frames. ], batch size: 49, lr: 2.83e-03
2024-08-06 13:35:31,872 INFO [trainer.py:765] (1/8) Epoch 30, batch 2100, train_loss[loss=3.508, NarTop10Accuracy=0.6013, over 3862.00 frames. ], tot_loss[loss=3.389, NarTop10Accuracy=0.64, over 5991.95 frames. ], batch size: 4, lr: 2.82e-03
2024-08-06 13:36:00,553 INFO [trainer.py:765] (1/8) Epoch 30, batch 2200, train_loss[loss=3.433, NarTop10Accuracy=0.6334, over 7211.00 frames. ], tot_loss[loss=3.384, NarTop10Accuracy=0.6411, over 6024.00 frames. ], batch size: 31, lr: 2.82e-03
2024-08-06 13:36:26,030 INFO [trainer.py:765] (1/8) Epoch 30, batch 2300, train_loss[loss=3.426, NarTop10Accuracy=0.6232, over 5732.00 frames. ], tot_loss[loss=3.389, NarTop10Accuracy=0.6404, over 6065.31 frames. ], batch size: 9, lr: 2.82e-03
2024-08-06 13:36:50,824 INFO [trainer.py:765] (1/8) Epoch 30, batch 2400, train_loss[loss=3.061, NarTop10Accuracy=0.7064, over 5128.00 frames. ], tot_loss[loss=3.399, NarTop10Accuracy=0.6383, over 5872.67 frames. ], batch size: 7, lr: 2.82e-03
2024-08-06 13:37:14,388 INFO [trainer.py:765] (1/8) Epoch 30, batch 2500, train_loss[loss=3.074, NarTop10Accuracy=0.7039, over 4812.00 frames. ], tot_loss[loss=3.373, NarTop10Accuracy=0.6428, over 5517.86 frames. ], batch size: 6, lr: 2.82e-03
2024-08-06 13:37:35,887 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 13:38:28,438 INFO [trainer.py:765] (1/8) Epoch 31, batch 100, train_loss[loss=3.22, NarTop10Accuracy=0.6711, over 7327.00 frames. ], tot_loss[loss=3.321, NarTop10Accuracy=0.6554, over 2371.51 frames. ], batch size: 30, lr: 2.77e-03
2024-08-06 13:39:02,651 INFO [trainer.py:765] (1/8) Epoch 31, batch 200, train_loss[loss=3.315, NarTop10Accuracy=0.6599, over 6818.00 frames. ], tot_loss[loss=3.326, NarTop10Accuracy=0.6546, over 3871.34 frames. ], batch size: 17, lr: 2.76e-03
2024-08-06 13:39:34,676 INFO [trainer.py:765] (1/8) Epoch 31, batch 300, train_loss[loss=3.101, NarTop10Accuracy=0.6904, over 6853.00 frames. ], tot_loss[loss=3.323, NarTop10Accuracy=0.6557, over 4672.38 frames. ], batch size: 21, lr: 2.76e-03
2024-08-06 13:40:07,363 INFO [trainer.py:765] (1/8) Epoch 31, batch 400, train_loss[loss=3.605, NarTop10Accuracy=0.5952, over 5120.00 frames. ], tot_loss[loss=3.337, NarTop10Accuracy=0.652, over 5102.97 frames. ], batch size: 7, lr: 2.76e-03
2024-08-06 13:40:37,813 INFO [trainer.py:765] (1/8) Epoch 31, batch 500, train_loss[loss=3.19, NarTop10Accuracy=0.6747, over 6072.00 frames. ], tot_loss[loss=3.328, NarTop10Accuracy=0.6533, over 5378.04 frames. ], batch size: 11, lr: 2.76e-03
2024-08-06 13:40:58,298 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 13:41:08,777 INFO [trainer.py:811] (1/8) Epoch 31, validation: loss=3.268, NarTop10Accuracy=0.6673, over 1907754.00 frames. 
2024-08-06 13:41:08,778 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 13:41:09,338 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.599e+02 1.987e+02 2.143e+02 2.328e+02 4.341e+02, threshold=4.287e+02, percent-clipped=0.1
2024-08-06 13:41:20,863 INFO [trainer.py:765] (1/8) Epoch 31, batch 600, train_loss[loss=3.387, NarTop10Accuracy=0.6357, over 5835.00 frames. ], tot_loss[loss=3.328, NarTop10Accuracy=0.6528, over 5647.48 frames. ], batch size: 9, lr: 2.76e-03
2024-08-06 13:41:54,260 INFO [trainer.py:765] (1/8) Epoch 31, batch 700, train_loss[loss=3.203, NarTop10Accuracy=0.67, over 5008.00 frames. ], tot_loss[loss=3.334, NarTop10Accuracy=0.6518, over 5726.51 frames. ], batch size: 6, lr: 2.76e-03
2024-08-06 13:42:32,158 INFO [trainer.py:765] (1/8) Epoch 31, batch 800, train_loss[loss=3.205, NarTop10Accuracy=0.6767, over 5099.00 frames. ], tot_loss[loss=3.34, NarTop10Accuracy=0.6505, over 5774.02 frames. ], batch size: 6, lr: 2.75e-03
2024-08-06 13:43:06,274 INFO [trainer.py:765] (1/8) Epoch 31, batch 900, train_loss[loss=3.224, NarTop10Accuracy=0.6601, over 6764.00 frames. ], tot_loss[loss=3.32, NarTop10Accuracy=0.6541, over 5789.48 frames. ], batch size: 14, lr: 2.75e-03
2024-08-06 13:43:38,009 INFO [trainer.py:765] (1/8) Epoch 31, batch 1000, train_loss[loss=3.185, NarTop10Accuracy=0.6806, over 6244.00 frames. ], tot_loss[loss=3.333, NarTop10Accuracy=0.6517, over 5908.98 frames. ], batch size: 13, lr: 2.75e-03
2024-08-06 13:44:14,513 INFO [trainer.py:765] (1/8) Epoch 31, batch 1100, train_loss[loss=3.407, NarTop10Accuracy=0.6432, over 6897.00 frames. ], tot_loss[loss=3.345, NarTop10Accuracy=0.6489, over 5944.52 frames. ], batch size: 17, lr: 2.75e-03
2024-08-06 13:44:53,786 INFO [trainer.py:765] (1/8) Epoch 31, batch 1200, train_loss[loss=3.356, NarTop10Accuracy=0.6461, over 7317.00 frames. ], tot_loss[loss=3.36, NarTop10Accuracy=0.6456, over 5938.94 frames. ], batch size: 30, lr: 2.75e-03
2024-08-06 13:45:25,076 INFO [trainer.py:765] (1/8) Epoch 31, batch 1300, train_loss[loss=3.243, NarTop10Accuracy=0.6621, over 5040.00 frames. ], tot_loss[loss=3.351, NarTop10Accuracy=0.6481, over 6015.97 frames. ], batch size: 6, lr: 2.75e-03
2024-08-06 13:45:58,741 INFO [trainer.py:765] (1/8) Epoch 31, batch 1400, train_loss[loss=3.061, NarTop10Accuracy=0.6916, over 6102.00 frames. ], tot_loss[loss=3.365, NarTop10Accuracy=0.6453, over 6031.31 frames. ], batch size: 11, lr: 2.74e-03
2024-08-06 13:46:33,490 INFO [trainer.py:765] (1/8) Epoch 31, batch 1500, train_loss[loss=3.542, NarTop10Accuracy=0.6132, over 6011.00 frames. ], tot_loss[loss=3.369, NarTop10Accuracy=0.6444, over 5986.41 frames. ], batch size: 49, lr: 2.74e-03
2024-08-06 13:47:04,658 INFO [trainer.py:765] (1/8) Epoch 31, batch 1600, train_loss[loss=3.214, NarTop10Accuracy=0.6782, over 6978.00 frames. ], tot_loss[loss=3.37, NarTop10Accuracy=0.6446, over 5966.84 frames. ], batch size: 22, lr: 2.74e-03
2024-08-06 13:47:31,424 INFO [trainer.py:765] (1/8) Epoch 31, batch 1700, train_loss[loss=3.585, NarTop10Accuracy=0.5887, over 6614.00 frames. ], tot_loss[loss=3.366, NarTop10Accuracy=0.6449, over 5944.11 frames. ], batch size: 14, lr: 2.74e-03
2024-08-06 13:47:58,016 INFO [trainer.py:765] (1/8) Epoch 31, batch 1800, train_loss[loss=3.461, NarTop10Accuracy=0.6244, over 7077.00 frames. ], tot_loss[loss=3.373, NarTop10Accuracy=0.6434, over 5997.31 frames. ], batch size: 22, lr: 2.74e-03
2024-08-06 13:48:24,576 INFO [trainer.py:765] (1/8) Epoch 31, batch 1900, train_loss[loss=3.432, NarTop10Accuracy=0.6372, over 5681.00 frames. ], tot_loss[loss=3.38, NarTop10Accuracy=0.6419, over 6047.62 frames. ], batch size: 48, lr: 2.74e-03
2024-08-06 13:48:50,258 INFO [trainer.py:765] (1/8) Epoch 31, batch 2000, train_loss[loss=3.685, NarTop10Accuracy=0.5871, over 5972.00 frames. ], tot_loss[loss=3.372, NarTop10Accuracy=0.6437, over 6025.79 frames. ], batch size: 49, lr: 2.73e-03
2024-08-06 13:49:15,764 INFO [trainer.py:765] (1/8) Epoch 31, batch 2100, train_loss[loss=3.424, NarTop10Accuracy=0.6367, over 4060.00 frames. ], tot_loss[loss=3.372, NarTop10Accuracy=0.6433, over 6006.92 frames. ], batch size: 4, lr: 2.73e-03
2024-08-06 13:49:41,278 INFO [trainer.py:765] (1/8) Epoch 31, batch 2200, train_loss[loss=3.401, NarTop10Accuracy=0.6487, over 7435.00 frames. ], tot_loss[loss=3.373, NarTop10Accuracy=0.6436, over 6048.24 frames. ], batch size: 31, lr: 2.73e-03
2024-08-06 13:50:06,708 INFO [trainer.py:765] (1/8) Epoch 31, batch 2300, train_loss[loss=3.356, NarTop10Accuracy=0.6403, over 5869.00 frames. ], tot_loss[loss=3.395, NarTop10Accuracy=0.6387, over 6066.26 frames. ], batch size: 9, lr: 2.73e-03
2024-08-06 13:50:31,393 INFO [trainer.py:765] (1/8) Epoch 31, batch 2400, train_loss[loss=3.373, NarTop10Accuracy=0.6309, over 5024.00 frames. ], tot_loss[loss=3.405, NarTop10Accuracy=0.6368, over 5872.62 frames. ], batch size: 7, lr: 2.73e-03
2024-08-06 13:50:54,892 INFO [trainer.py:765] (1/8) Epoch 31, batch 2500, train_loss[loss=3.339, NarTop10Accuracy=0.6253, over 4982.00 frames. ], tot_loss[loss=3.378, NarTop10Accuracy=0.6414, over 5533.52 frames. ], batch size: 6, lr: 2.72e-03
2024-08-06 13:51:08,995 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 13:51:19,070 INFO [trainer.py:811] (1/8) Epoch 31, validation: loss=3.234, NarTop10Accuracy=0.6746, over 1907754.00 frames. 
2024-08-06 13:51:19,070 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 13:51:19,540 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.591e+02 2.007e+02 2.182e+02 2.368e+02 4.565e+02, threshold=4.363e+02, percent-clipped=0.1
2024-08-06 13:51:25,934 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 13:52:19,910 INFO [trainer.py:765] (1/8) Epoch 32, batch 100, train_loss[loss=3.238, NarTop10Accuracy=0.6668, over 7097.00 frames. ], tot_loss[loss=3.305, NarTop10Accuracy=0.6586, over 2373.25 frames. ], batch size: 30, lr: 2.68e-03
2024-08-06 13:52:52,538 INFO [trainer.py:765] (1/8) Epoch 32, batch 200, train_loss[loss=3.664, NarTop10Accuracy=0.5909, over 6856.00 frames. ], tot_loss[loss=3.323, NarTop10Accuracy=0.6549, over 3873.88 frames. ], batch size: 17, lr: 2.68e-03
2024-08-06 13:53:28,093 INFO [trainer.py:765] (1/8) Epoch 32, batch 300, train_loss[loss=3.221, NarTop10Accuracy=0.6766, over 7193.00 frames. ], tot_loss[loss=3.322, NarTop10Accuracy=0.6545, over 4668.02 frames. ], batch size: 22, lr: 2.68e-03
2024-08-06 13:54:00,887 INFO [trainer.py:765] (1/8) Epoch 32, batch 400, train_loss[loss=3.556, NarTop10Accuracy=0.6092, over 5147.00 frames. ], tot_loss[loss=3.33, NarTop10Accuracy=0.6533, over 5118.16 frames. ], batch size: 7, lr: 2.67e-03
2024-08-06 13:54:32,822 INFO [trainer.py:765] (1/8) Epoch 32, batch 500, train_loss[loss=2.873, NarTop10Accuracy=0.7344, over 6099.00 frames. ], tot_loss[loss=3.323, NarTop10Accuracy=0.6547, over 5405.13 frames. ], batch size: 11, lr: 2.67e-03
2024-08-06 13:55:01,773 INFO [trainer.py:765] (1/8) Epoch 32, batch 600, train_loss[loss=3.455, NarTop10Accuracy=0.6234, over 5742.00 frames. ], tot_loss[loss=3.314, NarTop10Accuracy=0.6563, over 5678.19 frames. ], batch size: 9, lr: 2.67e-03
2024-08-06 13:55:41,512 INFO [trainer.py:765] (1/8) Epoch 32, batch 700, train_loss[loss=3.211, NarTop10Accuracy=0.6779, over 5082.00 frames. ], tot_loss[loss=3.327, NarTop10Accuracy=0.6535, over 5728.50 frames. ], batch size: 6, lr: 2.67e-03
2024-08-06 13:56:13,172 INFO [trainer.py:765] (1/8) Epoch 32, batch 800, train_loss[loss=2.945, NarTop10Accuracy=0.7265, over 4933.00 frames. ], tot_loss[loss=3.331, NarTop10Accuracy=0.6524, over 5777.56 frames. ], batch size: 6, lr: 2.67e-03
2024-08-06 13:56:43,166 INFO [trainer.py:765] (1/8) Epoch 32, batch 900, train_loss[loss=3.432, NarTop10Accuracy=0.6239, over 6302.00 frames. ], tot_loss[loss=3.334, NarTop10Accuracy=0.652, over 5811.33 frames. ], batch size: 13, lr: 2.67e-03
2024-08-06 13:57:24,521 INFO [trainer.py:765] (1/8) Epoch 32, batch 1000, train_loss[loss=3.583, NarTop10Accuracy=0.5969, over 6134.00 frames. ], tot_loss[loss=3.343, NarTop10Accuracy=0.6498, over 5911.94 frames. ], batch size: 13, lr: 2.66e-03
2024-08-06 13:57:57,452 INFO [trainer.py:765] (1/8) Epoch 32, batch 1100, train_loss[loss=3.128, NarTop10Accuracy=0.6927, over 6694.00 frames. ], tot_loss[loss=3.355, NarTop10Accuracy=0.647, over 5944.09 frames. ], batch size: 17, lr: 2.66e-03
2024-08-06 13:58:30,541 INFO [trainer.py:765] (1/8) Epoch 32, batch 1200, train_loss[loss=3.129, NarTop10Accuracy=0.6963, over 7187.00 frames. ], tot_loss[loss=3.357, NarTop10Accuracy=0.6468, over 5938.51 frames. ], batch size: 30, lr: 2.66e-03
2024-08-06 13:59:08,260 INFO [trainer.py:765] (1/8) Epoch 32, batch 1300, train_loss[loss=3.08, NarTop10Accuracy=0.6893, over 5135.00 frames. ], tot_loss[loss=3.355, NarTop10Accuracy=0.6464, over 6006.60 frames. ], batch size: 6, lr: 2.66e-03
2024-08-06 13:59:42,266 INFO [trainer.py:765] (1/8) Epoch 32, batch 1400, train_loss[loss=3.407, NarTop10Accuracy=0.638, over 6034.00 frames. ], tot_loss[loss=3.354, NarTop10Accuracy=0.6471, over 6013.48 frames. ], batch size: 11, lr: 2.66e-03
2024-08-06 14:00:12,976 INFO [trainer.py:765] (1/8) Epoch 32, batch 1500, train_loss[loss=3.773, NarTop10Accuracy=0.5569, over 6297.00 frames. ], tot_loss[loss=3.355, NarTop10Accuracy=0.6468, over 5971.49 frames. ], batch size: 49, lr: 2.66e-03
2024-08-06 14:00:40,824 INFO [trainer.py:765] (1/8) Epoch 32, batch 1600, train_loss[loss=3.178, NarTop10Accuracy=0.6822, over 7250.00 frames. ], tot_loss[loss=3.357, NarTop10Accuracy=0.6464, over 5943.52 frames. ], batch size: 22, lr: 2.65e-03
2024-08-06 14:01:07,534 INFO [trainer.py:765] (1/8) Epoch 32, batch 1700, train_loss[loss=3.368, NarTop10Accuracy=0.6426, over 6692.00 frames. ], tot_loss[loss=3.369, NarTop10Accuracy=0.6443, over 5941.33 frames. ], batch size: 14, lr: 2.65e-03
2024-08-06 14:01:34,089 INFO [trainer.py:765] (1/8) Epoch 32, batch 1800, train_loss[loss=3.175, NarTop10Accuracy=0.6853, over 7136.00 frames. ], tot_loss[loss=3.374, NarTop10Accuracy=0.6437, over 6002.78 frames. ], batch size: 22, lr: 2.65e-03
2024-08-06 14:02:00,636 INFO [trainer.py:765] (1/8) Epoch 32, batch 1900, train_loss[loss=3.472, NarTop10Accuracy=0.6249, over 6571.00 frames. ], tot_loss[loss=3.382, NarTop10Accuracy=0.642, over 6021.80 frames. ], batch size: 52, lr: 2.65e-03
2024-08-06 14:02:20,591 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 14:02:30,653 INFO [trainer.py:811] (1/8) Epoch 32, validation: loss=3.204, NarTop10Accuracy=0.6812, over 1907754.00 frames. 
2024-08-06 14:02:30,653 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 14:02:31,152 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.595e+02 2.032e+02 2.200e+02 2.392e+02 6.182e+02, threshold=4.401e+02, percent-clipped=0.1
2024-08-06 14:02:36,384 INFO [trainer.py:765] (1/8) Epoch 32, batch 2000, train_loss[loss=3.525, NarTop10Accuracy=0.6103, over 6206.00 frames. ], tot_loss[loss=3.38, NarTop10Accuracy=0.6424, over 6004.76 frames. ], batch size: 48, lr: 2.65e-03
2024-08-06 14:03:01,698 INFO [trainer.py:765] (1/8) Epoch 32, batch 2100, train_loss[loss=3.415, NarTop10Accuracy=0.635, over 3919.00 frames. ], tot_loss[loss=3.38, NarTop10Accuracy=0.6419, over 5991.67 frames. ], batch size: 4, lr: 2.65e-03
2024-08-06 14:03:27,177 INFO [trainer.py:765] (1/8) Epoch 32, batch 2200, train_loss[loss=3.493, NarTop10Accuracy=0.6157, over 7111.00 frames. ], tot_loss[loss=3.381, NarTop10Accuracy=0.6417, over 6032.43 frames. ], batch size: 31, lr: 2.64e-03
2024-08-06 14:03:52,586 INFO [trainer.py:765] (1/8) Epoch 32, batch 2300, train_loss[loss=3.378, NarTop10Accuracy=0.6346, over 5817.00 frames. ], tot_loss[loss=3.387, NarTop10Accuracy=0.6414, over 6053.92 frames. ], batch size: 9, lr: 2.64e-03
2024-08-06 14:04:17,274 INFO [trainer.py:765] (1/8) Epoch 32, batch 2400, train_loss[loss=3.48, NarTop10Accuracy=0.6293, over 6374.00 frames. ], tot_loss[loss=3.401, NarTop10Accuracy=0.6389, over 5892.42 frames. ], batch size: 49, lr: 2.64e-03
2024-08-06 14:04:40,635 INFO [trainer.py:765] (1/8) Epoch 32, batch 2500, train_loss[loss=2.934, NarTop10Accuracy=0.722, over 5048.00 frames. ], tot_loss[loss=3.36, NarTop10Accuracy=0.6466, over 5532.52 frames. ], batch size: 6, lr: 2.64e-03
2024-08-06 14:05:02,275 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 14:06:02,906 INFO [trainer.py:765] (1/8) Epoch 33, batch 100, train_loss[loss=3.526, NarTop10Accuracy=0.6119, over 7370.00 frames. ], tot_loss[loss=3.32, NarTop10Accuracy=0.6555, over 2371.68 frames. ], batch size: 30, lr: 2.60e-03
2024-08-06 14:06:36,079 INFO [trainer.py:765] (1/8) Epoch 33, batch 200, train_loss[loss=3.356, NarTop10Accuracy=0.6468, over 6836.00 frames. ], tot_loss[loss=3.32, NarTop10Accuracy=0.6563, over 3868.08 frames. ], batch size: 17, lr: 2.59e-03
2024-08-06 14:07:12,146 INFO [trainer.py:765] (1/8) Epoch 33, batch 300, train_loss[loss=3.243, NarTop10Accuracy=0.6791, over 7174.00 frames. ], tot_loss[loss=3.314, NarTop10Accuracy=0.6568, over 4679.88 frames. ], batch size: 22, lr: 2.59e-03
2024-08-06 14:07:48,256 INFO [trainer.py:765] (1/8) Epoch 33, batch 400, train_loss[loss=3.523, NarTop10Accuracy=0.6133, over 5224.00 frames. ], tot_loss[loss=3.314, NarTop10Accuracy=0.6562, over 5136.38 frames. ], batch size: 7, lr: 2.59e-03
2024-08-06 14:08:18,547 INFO [trainer.py:765] (1/8) Epoch 33, batch 500, train_loss[loss=3.124, NarTop10Accuracy=0.6994, over 6112.00 frames. ], tot_loss[loss=3.324, NarTop10Accuracy=0.6542, over 5409.92 frames. ], batch size: 11, lr: 2.59e-03
2024-08-06 14:08:49,792 INFO [trainer.py:765] (1/8) Epoch 33, batch 600, train_loss[loss=3.308, NarTop10Accuracy=0.6668, over 6020.00 frames. ], tot_loss[loss=3.335, NarTop10Accuracy=0.6517, over 5683.35 frames. ], batch size: 9, lr: 2.59e-03
2024-08-06 14:09:32,925 INFO [trainer.py:765] (1/8) Epoch 33, batch 700, train_loss[loss=3.204, NarTop10Accuracy=0.6699, over 5146.00 frames. ], tot_loss[loss=3.33, NarTop10Accuracy=0.6525, over 5758.46 frames. ], batch size: 6, lr: 2.59e-03
2024-08-06 14:10:04,596 INFO [trainer.py:765] (1/8) Epoch 33, batch 800, train_loss[loss=3.117, NarTop10Accuracy=0.6998, over 5088.00 frames. ], tot_loss[loss=3.35, NarTop10Accuracy=0.6485, over 5789.85 frames. ], batch size: 6, lr: 2.58e-03
2024-08-06 14:10:35,386 INFO [trainer.py:765] (1/8) Epoch 33, batch 900, train_loss[loss=3.146, NarTop10Accuracy=0.6887, over 6232.00 frames. ], tot_loss[loss=3.348, NarTop10Accuracy=0.6488, over 5826.25 frames. ], batch size: 13, lr: 2.58e-03
2024-08-06 14:11:15,068 INFO [trainer.py:765] (1/8) Epoch 33, batch 1000, train_loss[loss=3.161, NarTop10Accuracy=0.6775, over 6282.00 frames. ], tot_loss[loss=3.35, NarTop10Accuracy=0.6481, over 5936.48 frames. ], batch size: 13, lr: 2.58e-03
2024-08-06 14:11:47,302 INFO [trainer.py:765] (1/8) Epoch 33, batch 1100, train_loss[loss=3.536, NarTop10Accuracy=0.6063, over 6844.00 frames. ], tot_loss[loss=3.359, NarTop10Accuracy=0.6462, over 5961.58 frames. ], batch size: 17, lr: 2.58e-03
2024-08-06 14:12:20,928 INFO [trainer.py:765] (1/8) Epoch 33, batch 1200, train_loss[loss=3.535, NarTop10Accuracy=0.6102, over 6957.00 frames. ], tot_loss[loss=3.356, NarTop10Accuracy=0.6462, over 5952.83 frames. ], batch size: 30, lr: 2.58e-03
2024-08-06 14:12:57,629 INFO [trainer.py:765] (1/8) Epoch 33, batch 1300, train_loss[loss=3.69, NarTop10Accuracy=0.5848, over 4977.00 frames. ], tot_loss[loss=3.347, NarTop10Accuracy=0.6485, over 6020.00 frames. ], batch size: 6, lr: 2.58e-03
2024-08-06 14:13:30,666 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 14:13:41,686 INFO [trainer.py:811] (1/8) Epoch 33, validation: loss=3.242, NarTop10Accuracy=0.6732, over 1907754.00 frames. 
2024-08-06 14:13:41,687 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 14:13:42,264 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.623e+02 2.031e+02 2.174e+02 2.363e+02 4.871e+02, threshold=4.347e+02, percent-clipped=0.1
2024-08-06 14:13:42,802 INFO [trainer.py:765] (1/8) Epoch 33, batch 1400, train_loss[loss=3.177, NarTop10Accuracy=0.6793, over 6140.00 frames. ], tot_loss[loss=3.339, NarTop10Accuracy=0.6504, over 6049.21 frames. ], batch size: 11, lr: 2.58e-03
2024-08-06 14:14:11,245 INFO [trainer.py:765] (1/8) Epoch 33, batch 1500, train_loss[loss=3.4, NarTop10Accuracy=0.6426, over 6434.00 frames. ], tot_loss[loss=3.353, NarTop10Accuracy=0.6472, over 5983.48 frames. ], batch size: 48, lr: 2.57e-03
2024-08-06 14:14:39,191 INFO [trainer.py:765] (1/8) Epoch 33, batch 1600, train_loss[loss=3.406, NarTop10Accuracy=0.6417, over 7082.00 frames. ], tot_loss[loss=3.367, NarTop10Accuracy=0.6447, over 5953.98 frames. ], batch size: 22, lr: 2.57e-03
2024-08-06 14:15:05,857 INFO [trainer.py:765] (1/8) Epoch 33, batch 1700, train_loss[loss=3.589, NarTop10Accuracy=0.606, over 6121.00 frames. ], tot_loss[loss=3.363, NarTop10Accuracy=0.6451, over 5942.71 frames. ], batch size: 13, lr: 2.57e-03
2024-08-06 14:15:32,589 INFO [trainer.py:765] (1/8) Epoch 33, batch 1800, train_loss[loss=3.406, NarTop10Accuracy=0.6398, over 7181.00 frames. ], tot_loss[loss=3.356, NarTop10Accuracy=0.6468, over 6016.29 frames. ], batch size: 22, lr: 2.57e-03
2024-08-06 14:15:59,214 INFO [trainer.py:765] (1/8) Epoch 33, batch 1900, train_loss[loss=3.362, NarTop10Accuracy=0.6476, over 6341.00 frames. ], tot_loss[loss=3.361, NarTop10Accuracy=0.6459, over 6050.68 frames. ], batch size: 49, lr: 2.57e-03
2024-08-06 14:16:24,894 INFO [trainer.py:765] (1/8) Epoch 33, batch 2000, train_loss[loss=3.543, NarTop10Accuracy=0.6062, over 5424.00 frames. ], tot_loss[loss=3.357, NarTop10Accuracy=0.6468, over 6021.92 frames. ], batch size: 48, lr: 2.57e-03
2024-08-06 14:16:50,350 INFO [trainer.py:765] (1/8) Epoch 33, batch 2100, train_loss[loss=3.459, NarTop10Accuracy=0.6206, over 4910.00 frames. ], tot_loss[loss=3.368, NarTop10Accuracy=0.6445, over 6001.08 frames. ], batch size: 5, lr: 2.56e-03
2024-08-06 14:17:15,825 INFO [trainer.py:765] (1/8) Epoch 33, batch 2200, train_loss[loss=3.42, NarTop10Accuracy=0.642, over 7212.00 frames. ], tot_loss[loss=3.361, NarTop10Accuracy=0.646, over 6025.88 frames. ], batch size: 30, lr: 2.56e-03
2024-08-06 14:17:41,309 INFO [trainer.py:765] (1/8) Epoch 33, batch 2300, train_loss[loss=3.27, NarTop10Accuracy=0.6634, over 5762.00 frames. ], tot_loss[loss=3.372, NarTop10Accuracy=0.6434, over 6047.74 frames. ], batch size: 9, lr: 2.56e-03
2024-08-06 14:18:10,143 INFO [trainer.py:765] (1/8) Epoch 33, batch 2400, train_loss[loss=3.87, NarTop10Accuracy=0.5433, over 5082.00 frames. ], tot_loss[loss=3.388, NarTop10Accuracy=0.6404, over 5853.97 frames. ], batch size: 7, lr: 2.56e-03
2024-08-06 14:18:33,706 INFO [trainer.py:765] (1/8) Epoch 33, batch 2500, train_loss[loss=3.283, NarTop10Accuracy=0.6408, over 5055.00 frames. ], tot_loss[loss=3.358, NarTop10Accuracy=0.6456, over 5527.58 frames. ], batch size: 6, lr: 2.56e-03
2024-08-06 14:18:54,729 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 14:19:51,932 INFO [trainer.py:765] (1/8) Epoch 34, batch 100, train_loss[loss=3.192, NarTop10Accuracy=0.6875, over 7301.00 frames. ], tot_loss[loss=3.318, NarTop10Accuracy=0.6568, over 2386.91 frames. ], batch size: 31, lr: 2.52e-03
2024-08-06 14:20:24,372 INFO [trainer.py:765] (1/8) Epoch 34, batch 200, train_loss[loss=3.418, NarTop10Accuracy=0.6307, over 6771.00 frames. ], tot_loss[loss=3.292, NarTop10Accuracy=0.6607, over 3884.20 frames. ], batch size: 17, lr: 2.52e-03
2024-08-06 14:21:00,841 INFO [trainer.py:765] (1/8) Epoch 34, batch 300, train_loss[loss=3.471, NarTop10Accuracy=0.6299, over 7153.00 frames. ], tot_loss[loss=3.303, NarTop10Accuracy=0.6588, over 4670.16 frames. ], batch size: 22, lr: 2.51e-03
2024-08-06 14:21:31,449 INFO [trainer.py:765] (1/8) Epoch 34, batch 400, train_loss[loss=3.257, NarTop10Accuracy=0.6683, over 5154.00 frames. ], tot_loss[loss=3.304, NarTop10Accuracy=0.6588, over 5123.97 frames. ], batch size: 7, lr: 2.51e-03
2024-08-06 14:22:01,875 INFO [trainer.py:765] (1/8) Epoch 34, batch 500, train_loss[loss=3.163, NarTop10Accuracy=0.6794, over 6087.00 frames. ], tot_loss[loss=3.315, NarTop10Accuracy=0.6565, over 5401.38 frames. ], batch size: 11, lr: 2.51e-03
2024-08-06 14:22:36,826 INFO [trainer.py:765] (1/8) Epoch 34, batch 600, train_loss[loss=3.387, NarTop10Accuracy=0.6376, over 5703.00 frames. ], tot_loss[loss=3.315, NarTop10Accuracy=0.6562, over 5682.04 frames. ], batch size: 9, lr: 2.51e-03
2024-08-06 14:23:14,605 INFO [trainer.py:765] (1/8) Epoch 34, batch 700, train_loss[loss=3.085, NarTop10Accuracy=0.6988, over 5112.00 frames. ], tot_loss[loss=3.328, NarTop10Accuracy=0.6538, over 5746.31 frames. ], batch size: 6, lr: 2.51e-03
2024-08-06 14:23:46,606 INFO [trainer.py:765] (1/8) Epoch 34, batch 800, train_loss[loss=3.263, NarTop10Accuracy=0.6595, over 4220.00 frames. ], tot_loss[loss=3.334, NarTop10Accuracy=0.652, over 5799.84 frames. ], batch size: 5, lr: 2.51e-03
2024-08-06 14:23:50,718 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 14:24:00,855 INFO [trainer.py:811] (1/8) Epoch 34, validation: loss=3.226, NarTop10Accuracy=0.6758, over 1907754.00 frames. 
2024-08-06 14:24:00,856 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 14:24:01,413 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.652e+02 2.033e+02 2.200e+02 2.391e+02 5.918e+02, threshold=4.399e+02, percent-clipped=0.1
2024-08-06 14:24:28,899 INFO [trainer.py:765] (1/8) Epoch 34, batch 900, train_loss[loss=3.163, NarTop10Accuracy=0.6836, over 6297.00 frames. ], tot_loss[loss=3.329, NarTop10Accuracy=0.6522, over 5828.03 frames. ], batch size: 13, lr: 2.51e-03
2024-08-06 14:25:05,287 INFO [trainer.py:765] (1/8) Epoch 34, batch 1000, train_loss[loss=3.316, NarTop10Accuracy=0.6508, over 6329.00 frames. ], tot_loss[loss=3.343, NarTop10Accuracy=0.6497, over 5921.40 frames. ], batch size: 13, lr: 2.50e-03
2024-08-06 14:25:37,996 INFO [trainer.py:765] (1/8) Epoch 34, batch 1100, train_loss[loss=3.453, NarTop10Accuracy=0.6302, over 6826.00 frames. ], tot_loss[loss=3.355, NarTop10Accuracy=0.6475, over 5966.55 frames. ], batch size: 17, lr: 2.50e-03
2024-08-06 14:26:13,975 INFO [trainer.py:765] (1/8) Epoch 34, batch 1200, train_loss[loss=3.587, NarTop10Accuracy=0.5984, over 7113.00 frames. ], tot_loss[loss=3.348, NarTop10Accuracy=0.6487, over 5949.71 frames. ], batch size: 31, lr: 2.50e-03
2024-08-06 14:26:52,652 INFO [trainer.py:765] (1/8) Epoch 34, batch 1300, train_loss[loss=3.509, NarTop10Accuracy=0.6092, over 5178.00 frames. ], tot_loss[loss=3.347, NarTop10Accuracy=0.6487, over 6022.10 frames. ], batch size: 6, lr: 2.50e-03
2024-08-06 14:27:24,383 INFO [trainer.py:765] (1/8) Epoch 34, batch 1400, train_loss[loss=3.045, NarTop10Accuracy=0.7052, over 6208.00 frames. ], tot_loss[loss=3.346, NarTop10Accuracy=0.6485, over 6042.95 frames. ], batch size: 11, lr: 2.50e-03
2024-08-06 14:27:52,726 INFO [trainer.py:765] (1/8) Epoch 34, batch 1500, train_loss[loss=3.749, NarTop10Accuracy=0.5733, over 6113.00 frames. ], tot_loss[loss=3.337, NarTop10Accuracy=0.6504, over 5965.83 frames. ], batch size: 48, lr: 2.50e-03
2024-08-06 14:28:20,672 INFO [trainer.py:765] (1/8) Epoch 34, batch 1600, train_loss[loss=3.291, NarTop10Accuracy=0.6685, over 7042.00 frames. ], tot_loss[loss=3.348, NarTop10Accuracy=0.6483, over 5969.40 frames. ], batch size: 22, lr: 2.50e-03
2024-08-06 14:28:47,384 INFO [trainer.py:765] (1/8) Epoch 34, batch 1700, train_loss[loss=3.51, NarTop10Accuracy=0.6165, over 6687.00 frames. ], tot_loss[loss=3.362, NarTop10Accuracy=0.6454, over 5950.42 frames. ], batch size: 14, lr: 2.49e-03
2024-08-06 14:29:14,010 INFO [trainer.py:765] (1/8) Epoch 34, batch 1800, train_loss[loss=3.54, NarTop10Accuracy=0.6109, over 7134.00 frames. ], tot_loss[loss=3.355, NarTop10Accuracy=0.6469, over 6009.48 frames. ], batch size: 22, lr: 2.49e-03
2024-08-06 14:29:43,752 INFO [trainer.py:765] (1/8) Epoch 34, batch 1900, train_loss[loss=3.678, NarTop10Accuracy=0.5816, over 5922.00 frames. ], tot_loss[loss=3.368, NarTop10Accuracy=0.6441, over 6055.05 frames. ], batch size: 48, lr: 2.49e-03
2024-08-06 14:30:09,515 INFO [trainer.py:765] (1/8) Epoch 34, batch 2000, train_loss[loss=3.495, NarTop10Accuracy=0.6238, over 6397.00 frames. ], tot_loss[loss=3.359, NarTop10Accuracy=0.6458, over 6039.46 frames. ], batch size: 50, lr: 2.49e-03
2024-08-06 14:30:35,016 INFO [trainer.py:765] (1/8) Epoch 34, batch 2100, train_loss[loss=3.162, NarTop10Accuracy=0.6804, over 4032.00 frames. ], tot_loss[loss=3.354, NarTop10Accuracy=0.6471, over 6011.82 frames. ], batch size: 4, lr: 2.49e-03
2024-08-06 14:31:00,511 INFO [trainer.py:765] (1/8) Epoch 34, batch 2200, train_loss[loss=3.426, NarTop10Accuracy=0.6413, over 7116.00 frames. ], tot_loss[loss=3.364, NarTop10Accuracy=0.6451, over 6043.34 frames. ], batch size: 30, lr: 2.49e-03
2024-08-06 14:31:25,979 INFO [trainer.py:765] (1/8) Epoch 34, batch 2300, train_loss[loss=3.066, NarTop10Accuracy=0.7098, over 5725.00 frames. ], tot_loss[loss=3.367, NarTop10Accuracy=0.6444, over 6068.78 frames. ], batch size: 9, lr: 2.49e-03
2024-08-06 14:31:50,751 INFO [trainer.py:765] (1/8) Epoch 34, batch 2400, train_loss[loss=3.714, NarTop10Accuracy=0.586, over 5203.00 frames. ], tot_loss[loss=3.374, NarTop10Accuracy=0.6431, over 5899.17 frames. ], batch size: 7, lr: 2.48e-03
2024-08-06 14:32:14,249 INFO [trainer.py:765] (1/8) Epoch 34, batch 2500, train_loss[loss=3.327, NarTop10Accuracy=0.6556, over 5078.00 frames. ], tot_loss[loss=3.354, NarTop10Accuracy=0.6466, over 5551.93 frames. ], batch size: 6, lr: 2.48e-03
2024-08-06 14:32:35,332 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 14:33:26,337 INFO [trainer.py:765] (1/8) Epoch 35, batch 100, train_loss[loss=3.292, NarTop10Accuracy=0.6661, over 7611.00 frames. ], tot_loss[loss=3.297, NarTop10Accuracy=0.6605, over 2396.00 frames. ], batch size: 31, lr: 2.44e-03
2024-08-06 14:34:03,581 INFO [trainer.py:765] (1/8) Epoch 35, batch 200, train_loss[loss=3.513, NarTop10Accuracy=0.6113, over 6818.00 frames. ], tot_loss[loss=3.306, NarTop10Accuracy=0.6588, over 3897.24 frames. ], batch size: 17, lr: 2.44e-03
2024-08-06 14:34:13,185 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 14:34:23,574 INFO [trainer.py:811] (1/8) Epoch 35, validation: loss=3.163, NarTop10Accuracy=0.689, over 1907754.00 frames. 
2024-08-06 14:34:23,575 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 14:34:24,110 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.644e+02 2.042e+02 2.203e+02 2.360e+02 4.181e+02, threshold=4.406e+02, percent-clipped=0.0
2024-08-06 14:34:44,665 INFO [trainer.py:765] (1/8) Epoch 35, batch 300, train_loss[loss=3.599, NarTop10Accuracy=0.5993, over 7143.00 frames. ], tot_loss[loss=3.311, NarTop10Accuracy=0.6575, over 4691.85 frames. ], batch size: 22, lr: 2.44e-03
2024-08-06 14:35:13,543 INFO [trainer.py:765] (1/8) Epoch 35, batch 400, train_loss[loss=3.303, NarTop10Accuracy=0.659, over 5156.00 frames. ], tot_loss[loss=3.309, NarTop10Accuracy=0.6569, over 5132.49 frames. ], batch size: 7, lr: 2.44e-03
2024-08-06 14:35:48,188 INFO [trainer.py:765] (1/8) Epoch 35, batch 500, train_loss[loss=3.426, NarTop10Accuracy=0.6362, over 6175.00 frames. ], tot_loss[loss=3.316, NarTop10Accuracy=0.6554, over 5407.47 frames. ], batch size: 11, lr: 2.44e-03
2024-08-06 14:36:22,747 INFO [trainer.py:765] (1/8) Epoch 35, batch 600, train_loss[loss=3.151, NarTop10Accuracy=0.6903, over 5777.00 frames. ], tot_loss[loss=3.311, NarTop10Accuracy=0.6565, over 5678.22 frames. ], batch size: 9, lr: 2.44e-03
2024-08-06 14:36:57,827 INFO [trainer.py:765] (1/8) Epoch 35, batch 700, train_loss[loss=3.26, NarTop10Accuracy=0.655, over 5089.00 frames. ], tot_loss[loss=3.314, NarTop10Accuracy=0.6562, over 5743.83 frames. ], batch size: 6, lr: 2.44e-03
2024-08-06 14:37:29,769 INFO [trainer.py:765] (1/8) Epoch 35, batch 800, train_loss[loss=3.325, NarTop10Accuracy=0.6573, over 4989.00 frames. ], tot_loss[loss=3.322, NarTop10Accuracy=0.6544, over 5807.21 frames. ], batch size: 6, lr: 2.43e-03
2024-08-06 14:38:03,304 INFO [trainer.py:765] (1/8) Epoch 35, batch 900, train_loss[loss=2.903, NarTop10Accuracy=0.736, over 6264.00 frames. ], tot_loss[loss=3.321, NarTop10Accuracy=0.6545, over 5822.59 frames. ], batch size: 13, lr: 2.43e-03
2024-08-06 14:38:43,709 INFO [trainer.py:765] (1/8) Epoch 35, batch 1000, train_loss[loss=3.251, NarTop10Accuracy=0.6663, over 6274.00 frames. ], tot_loss[loss=3.324, NarTop10Accuracy=0.6544, over 5920.98 frames. ], batch size: 13, lr: 2.43e-03
2024-08-06 14:39:16,568 INFO [trainer.py:765] (1/8) Epoch 35, batch 1100, train_loss[loss=3.51, NarTop10Accuracy=0.6223, over 7006.00 frames. ], tot_loss[loss=3.326, NarTop10Accuracy=0.6535, over 5962.70 frames. ], batch size: 17, lr: 2.43e-03
2024-08-06 14:39:50,838 INFO [trainer.py:765] (1/8) Epoch 35, batch 1200, train_loss[loss=3.397, NarTop10Accuracy=0.644, over 7516.00 frames. ], tot_loss[loss=3.336, NarTop10Accuracy=0.6512, over 5956.23 frames. ], batch size: 32, lr: 2.43e-03
2024-08-06 14:40:33,953 INFO [trainer.py:765] (1/8) Epoch 35, batch 1300, train_loss[loss=3.045, NarTop10Accuracy=0.7102, over 4966.00 frames. ], tot_loss[loss=3.332, NarTop10Accuracy=0.652, over 6025.80 frames. ], batch size: 6, lr: 2.43e-03
2024-08-06 14:41:03,184 INFO [trainer.py:765] (1/8) Epoch 35, batch 1400, train_loss[loss=3.374, NarTop10Accuracy=0.6383, over 5952.00 frames. ], tot_loss[loss=3.351, NarTop10Accuracy=0.6481, over 6024.58 frames. ], batch size: 11, lr: 2.43e-03
2024-08-06 14:41:33,824 INFO [trainer.py:765] (1/8) Epoch 35, batch 1500, train_loss[loss=3.386, NarTop10Accuracy=0.6394, over 6112.00 frames. ], tot_loss[loss=3.354, NarTop10Accuracy=0.6471, over 5966.34 frames. ], batch size: 48, lr: 2.43e-03
2024-08-06 14:42:01,777 INFO [trainer.py:765] (1/8) Epoch 35, batch 1600, train_loss[loss=3.679, NarTop10Accuracy=0.577, over 7387.00 frames. ], tot_loss[loss=3.346, NarTop10Accuracy=0.6485, over 5952.44 frames. ], batch size: 23, lr: 2.42e-03
2024-08-06 14:42:28,467 INFO [trainer.py:765] (1/8) Epoch 35, batch 1700, train_loss[loss=3.405, NarTop10Accuracy=0.6354, over 6260.00 frames. ], tot_loss[loss=3.351, NarTop10Accuracy=0.6479, over 5938.85 frames. ], batch size: 13, lr: 2.42e-03
2024-08-06 14:42:55,040 INFO [trainer.py:765] (1/8) Epoch 35, batch 1800, train_loss[loss=3.262, NarTop10Accuracy=0.6713, over 7061.00 frames. ], tot_loss[loss=3.357, NarTop10Accuracy=0.6464, over 5997.17 frames. ], batch size: 22, lr: 2.42e-03
2024-08-06 14:43:21,646 INFO [trainer.py:765] (1/8) Epoch 35, batch 1900, train_loss[loss=3.39, NarTop10Accuracy=0.6411, over 6181.00 frames. ], tot_loss[loss=3.362, NarTop10Accuracy=0.6455, over 6027.10 frames. ], batch size: 50, lr: 2.42e-03
2024-08-06 14:43:47,367 INFO [trainer.py:765] (1/8) Epoch 35, batch 2000, train_loss[loss=3.517, NarTop10Accuracy=0.6194, over 6646.00 frames. ], tot_loss[loss=3.361, NarTop10Accuracy=0.6461, over 6005.25 frames. ], batch size: 49, lr: 2.42e-03
2024-08-06 14:44:12,857 INFO [trainer.py:765] (1/8) Epoch 35, batch 2100, train_loss[loss=3.13, NarTop10Accuracy=0.6835, over 4901.00 frames. ], tot_loss[loss=3.359, NarTop10Accuracy=0.6466, over 5994.90 frames. ], batch size: 5, lr: 2.42e-03
2024-08-06 14:44:38,389 INFO [trainer.py:765] (1/8) Epoch 35, batch 2200, train_loss[loss=3.622, NarTop10Accuracy=0.5904, over 7391.00 frames. ], tot_loss[loss=3.373, NarTop10Accuracy=0.6437, over 6039.16 frames. ], batch size: 30, lr: 2.42e-03
2024-08-06 14:44:47,200 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 14:44:57,441 INFO [trainer.py:811] (1/8) Epoch 35, validation: loss=3.219, NarTop10Accuracy=0.6773, over 1907754.00 frames. 
2024-08-06 14:44:57,441 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 14:44:57,973 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.690e+02 2.083e+02 2.237e+02 2.412e+02 3.944e+02, threshold=4.474e+02, percent-clipped=0.0
2024-08-06 14:45:14,100 INFO [trainer.py:765] (1/8) Epoch 35, batch 2300, train_loss[loss=3.44, NarTop10Accuracy=0.6362, over 5692.00 frames. ], tot_loss[loss=3.373, NarTop10Accuracy=0.644, over 6073.75 frames. ], batch size: 9, lr: 2.41e-03
2024-08-06 14:45:38,820 INFO [trainer.py:765] (1/8) Epoch 35, batch 2400, train_loss[loss=3.453, NarTop10Accuracy=0.6315, over 6255.00 frames. ], tot_loss[loss=3.378, NarTop10Accuracy=0.6423, over 5885.73 frames. ], batch size: 49, lr: 2.41e-03
2024-08-06 14:46:02,146 INFO [trainer.py:765] (1/8) Epoch 35, batch 2500, train_loss[loss=3.557, NarTop10Accuracy=0.6246, over 5159.00 frames. ], tot_loss[loss=3.345, NarTop10Accuracy=0.6488, over 5549.96 frames. ], batch size: 6, lr: 2.41e-03
2024-08-06 14:46:23,256 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 14:47:25,441 INFO [trainer.py:765] (1/8) Epoch 36, batch 100, train_loss[loss=3.201, NarTop10Accuracy=0.6794, over 7448.00 frames. ], tot_loss[loss=3.315, NarTop10Accuracy=0.6571, over 2362.15 frames. ], batch size: 30, lr: 2.38e-03
2024-08-06 14:47:58,358 INFO [trainer.py:765] (1/8) Epoch 36, batch 200, train_loss[loss=3.227, NarTop10Accuracy=0.6749, over 6836.00 frames. ], tot_loss[loss=3.287, NarTop10Accuracy=0.6615, over 3863.30 frames. ], batch size: 17, lr: 2.37e-03
2024-08-06 14:48:30,724 INFO [trainer.py:765] (1/8) Epoch 36, batch 300, train_loss[loss=3.246, NarTop10Accuracy=0.6728, over 7043.00 frames. ], tot_loss[loss=3.285, NarTop10Accuracy=0.6621, over 4671.39 frames. ], batch size: 22, lr: 2.37e-03
2024-08-06 14:49:04,814 INFO [trainer.py:765] (1/8) Epoch 36, batch 400, train_loss[loss=3.322, NarTop10Accuracy=0.6651, over 5075.00 frames. ], tot_loss[loss=3.292, NarTop10Accuracy=0.6613, over 5137.36 frames. ], batch size: 7, lr: 2.37e-03
2024-08-06 14:49:36,588 INFO [trainer.py:765] (1/8) Epoch 36, batch 500, train_loss[loss=3.507, NarTop10Accuracy=0.6177, over 6160.00 frames. ], tot_loss[loss=3.285, NarTop10Accuracy=0.6616, over 5402.38 frames. ], batch size: 11, lr: 2.37e-03
2024-08-06 14:50:09,654 INFO [trainer.py:765] (1/8) Epoch 36, batch 600, train_loss[loss=3.357, NarTop10Accuracy=0.6506, over 5591.00 frames. ], tot_loss[loss=3.3, NarTop10Accuracy=0.6587, over 5666.76 frames. ], batch size: 9, lr: 2.37e-03
2024-08-06 14:50:46,513 INFO [trainer.py:765] (1/8) Epoch 36, batch 700, train_loss[loss=3.189, NarTop10Accuracy=0.6839, over 5047.00 frames. ], tot_loss[loss=3.315, NarTop10Accuracy=0.6557, over 5746.99 frames. ], batch size: 6, lr: 2.37e-03
2024-08-06 14:51:23,703 INFO [trainer.py:765] (1/8) Epoch 36, batch 800, train_loss[loss=3.678, NarTop10Accuracy=0.5799, over 5154.00 frames. ], tot_loss[loss=3.317, NarTop10Accuracy=0.6554, over 5799.87 frames. ], batch size: 6, lr: 2.37e-03
2024-08-06 14:51:54,346 INFO [trainer.py:765] (1/8) Epoch 36, batch 900, train_loss[loss=3.132, NarTop10Accuracy=0.6934, over 6353.00 frames. ], tot_loss[loss=3.316, NarTop10Accuracy=0.6553, over 5808.67 frames. ], batch size: 13, lr: 2.36e-03
2024-08-06 14:52:30,324 INFO [trainer.py:765] (1/8) Epoch 36, batch 1000, train_loss[loss=3.292, NarTop10Accuracy=0.6621, over 6758.00 frames. ], tot_loss[loss=3.325, NarTop10Accuracy=0.6534, over 5921.35 frames. ], batch size: 14, lr: 2.36e-03
2024-08-06 14:53:06,863 INFO [trainer.py:765] (1/8) Epoch 36, batch 1100, train_loss[loss=3.254, NarTop10Accuracy=0.6749, over 6830.00 frames. ], tot_loss[loss=3.343, NarTop10Accuracy=0.6499, over 5959.89 frames. ], batch size: 17, lr: 2.36e-03
2024-08-06 14:53:40,248 INFO [trainer.py:765] (1/8) Epoch 36, batch 1200, train_loss[loss=3.498, NarTop10Accuracy=0.6148, over 7203.00 frames. ], tot_loss[loss=3.327, NarTop10Accuracy=0.653, over 5966.55 frames. ], batch size: 30, lr: 2.36e-03
2024-08-06 14:54:15,855 INFO [trainer.py:765] (1/8) Epoch 36, batch 1300, train_loss[loss=3.307, NarTop10Accuracy=0.655, over 5005.00 frames. ], tot_loss[loss=3.335, NarTop10Accuracy=0.6507, over 6033.47 frames. ], batch size: 6, lr: 2.36e-03
2024-08-06 14:54:51,541 INFO [trainer.py:765] (1/8) Epoch 36, batch 1400, train_loss[loss=3.166, NarTop10Accuracy=0.6848, over 6136.00 frames. ], tot_loss[loss=3.332, NarTop10Accuracy=0.6513, over 6041.82 frames. ], batch size: 11, lr: 2.36e-03
2024-08-06 14:55:21,802 INFO [trainer.py:765] (1/8) Epoch 36, batch 1500, train_loss[loss=3.545, NarTop10Accuracy=0.6025, over 6223.00 frames. ], tot_loss[loss=3.336, NarTop10Accuracy=0.6502, over 5978.01 frames. ], batch size: 51, lr: 2.36e-03
2024-08-06 14:55:49,902 INFO [trainer.py:765] (1/8) Epoch 36, batch 1600, train_loss[loss=3.171, NarTop10Accuracy=0.6762, over 7215.00 frames. ], tot_loss[loss=3.337, NarTop10Accuracy=0.65, over 5961.82 frames. ], batch size: 22, lr: 2.36e-03
2024-08-06 14:56:04,132 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 14:56:14,600 INFO [trainer.py:811] (1/8) Epoch 36, validation: loss=3.22, NarTop10Accuracy=0.6784, over 1907754.00 frames. 
2024-08-06 14:56:14,601 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 14:56:15,103 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.690e+02 2.063e+02 2.224e+02 2.398e+02 5.290e+02, threshold=4.447e+02, percent-clipped=0.1
2024-08-06 14:56:27,177 INFO [trainer.py:765] (1/8) Epoch 36, batch 1700, train_loss[loss=2.997, NarTop10Accuracy=0.7186, over 6646.00 frames. ], tot_loss[loss=3.334, NarTop10Accuracy=0.6509, over 5939.43 frames. ], batch size: 14, lr: 2.35e-03
2024-08-06 14:56:53,759 INFO [trainer.py:765] (1/8) Epoch 36, batch 1800, train_loss[loss=3.457, NarTop10Accuracy=0.6322, over 7121.00 frames. ], tot_loss[loss=3.341, NarTop10Accuracy=0.6499, over 6008.96 frames. ], batch size: 22, lr: 2.35e-03
2024-08-06 14:57:20,335 INFO [trainer.py:765] (1/8) Epoch 36, batch 1900, train_loss[loss=3.499, NarTop10Accuracy=0.6165, over 5359.00 frames. ], tot_loss[loss=3.342, NarTop10Accuracy=0.6497, over 6043.38 frames. ], batch size: 49, lr: 2.35e-03
2024-08-06 14:57:46,057 INFO [trainer.py:765] (1/8) Epoch 36, batch 2000, train_loss[loss=3.769, NarTop10Accuracy=0.5624, over 5941.00 frames. ], tot_loss[loss=3.357, NarTop10Accuracy=0.6464, over 6009.34 frames. ], batch size: 49, lr: 2.35e-03
2024-08-06 14:58:11,404 INFO [trainer.py:765] (1/8) Epoch 36, batch 2100, train_loss[loss=3.372, NarTop10Accuracy=0.6479, over 3939.00 frames. ], tot_loss[loss=3.361, NarTop10Accuracy=0.6458, over 6002.15 frames. ], batch size: 4, lr: 2.35e-03
2024-08-06 14:58:36,832 INFO [trainer.py:765] (1/8) Epoch 36, batch 2200, train_loss[loss=3.653, NarTop10Accuracy=0.5835, over 7029.00 frames. ], tot_loss[loss=3.367, NarTop10Accuracy=0.6443, over 6035.28 frames. ], batch size: 30, lr: 2.35e-03
2024-08-06 14:59:02,344 INFO [trainer.py:765] (1/8) Epoch 36, batch 2300, train_loss[loss=3.369, NarTop10Accuracy=0.6387, over 5692.00 frames. ], tot_loss[loss=3.381, NarTop10Accuracy=0.6422, over 6064.35 frames. ], batch size: 9, lr: 2.35e-03
2024-08-06 14:59:27,094 INFO [trainer.py:765] (1/8) Epoch 36, batch 2400, train_loss[loss=3.467, NarTop10Accuracy=0.6189, over 5125.00 frames. ], tot_loss[loss=3.381, NarTop10Accuracy=0.642, over 5891.61 frames. ], batch size: 7, lr: 2.35e-03
2024-08-06 14:59:50,503 INFO [trainer.py:765] (1/8) Epoch 36, batch 2500, train_loss[loss=3.558, NarTop10Accuracy=0.602, over 5109.00 frames. ], tot_loss[loss=3.354, NarTop10Accuracy=0.6468, over 5544.90 frames. ], batch size: 6, lr: 2.34e-03
2024-08-06 15:00:11,845 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 15:01:14,218 INFO [trainer.py:765] (1/8) Epoch 37, batch 100, train_loss[loss=3.316, NarTop10Accuracy=0.6594, over 7573.00 frames. ], tot_loss[loss=3.295, NarTop10Accuracy=0.6622, over 2367.16 frames. ], batch size: 31, lr: 2.31e-03
2024-08-06 15:01:44,097 INFO [trainer.py:765] (1/8) Epoch 37, batch 200, train_loss[loss=3.287, NarTop10Accuracy=0.6634, over 6729.00 frames. ], tot_loss[loss=3.287, NarTop10Accuracy=0.6632, over 3863.04 frames. ], batch size: 17, lr: 2.31e-03
2024-08-06 15:02:17,382 INFO [trainer.py:765] (1/8) Epoch 37, batch 300, train_loss[loss=3.206, NarTop10Accuracy=0.6773, over 7121.00 frames. ], tot_loss[loss=3.279, NarTop10Accuracy=0.664, over 4671.18 frames. ], batch size: 22, lr: 2.31e-03
2024-08-06 15:02:48,346 INFO [trainer.py:765] (1/8) Epoch 37, batch 400, train_loss[loss=3.277, NarTop10Accuracy=0.6647, over 5073.00 frames. ], tot_loss[loss=3.286, NarTop10Accuracy=0.6622, over 5125.42 frames. ], batch size: 7, lr: 2.31e-03
2024-08-06 15:03:26,570 INFO [trainer.py:765] (1/8) Epoch 37, batch 500, train_loss[loss=3.277, NarTop10Accuracy=0.6654, over 6158.00 frames. ], tot_loss[loss=3.294, NarTop10Accuracy=0.6606, over 5406.50 frames. ], batch size: 11, lr: 2.30e-03
2024-08-06 15:03:58,032 INFO [trainer.py:765] (1/8) Epoch 37, batch 600, train_loss[loss=3.438, NarTop10Accuracy=0.6335, over 5736.00 frames. ], tot_loss[loss=3.308, NarTop10Accuracy=0.6578, over 5685.63 frames. ], batch size: 9, lr: 2.30e-03
2024-08-06 15:04:30,247 INFO [trainer.py:765] (1/8) Epoch 37, batch 700, train_loss[loss=3.088, NarTop10Accuracy=0.7039, over 5095.00 frames. ], tot_loss[loss=3.319, NarTop10Accuracy=0.655, over 5760.33 frames. ], batch size: 6, lr: 2.30e-03
2024-08-06 15:05:12,163 INFO [trainer.py:765] (1/8) Epoch 37, batch 800, train_loss[loss=3.503, NarTop10Accuracy=0.6274, over 5081.00 frames. ], tot_loss[loss=3.328, NarTop10Accuracy=0.6528, over 5805.30 frames. ], batch size: 6, lr: 2.30e-03
2024-08-06 15:05:40,606 INFO [trainer.py:765] (1/8) Epoch 37, batch 900, train_loss[loss=3.357, NarTop10Accuracy=0.6558, over 6334.00 frames. ], tot_loss[loss=3.321, NarTop10Accuracy=0.654, over 5812.42 frames. ], batch size: 13, lr: 2.30e-03
2024-08-06 15:06:15,608 INFO [trainer.py:765] (1/8) Epoch 37, batch 1000, train_loss[loss=3.23, NarTop10Accuracy=0.6839, over 6264.00 frames. ], tot_loss[loss=3.334, NarTop10Accuracy=0.6514, over 5914.65 frames. ], batch size: 13, lr: 2.30e-03
2024-08-06 15:06:42,491 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 15:06:53,169 INFO [trainer.py:811] (1/8) Epoch 37, validation: loss=3.234, NarTop10Accuracy=0.6744, over 1907754.00 frames. 
2024-08-06 15:06:53,169 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 15:06:53,809 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.659e+02 2.068e+02 2.238e+02 2.409e+02 6.392e+02, threshold=4.475e+02, percent-clipped=0.1
2024-08-06 15:07:01,306 INFO [trainer.py:765] (1/8) Epoch 37, batch 1100, train_loss[loss=3.476, NarTop10Accuracy=0.6234, over 6994.00 frames. ], tot_loss[loss=3.331, NarTop10Accuracy=0.6518, over 5936.65 frames. ], batch size: 17, lr: 2.30e-03
2024-08-06 15:07:32,718 INFO [trainer.py:765] (1/8) Epoch 37, batch 1200, train_loss[loss=3.322, NarTop10Accuracy=0.6505, over 7184.00 frames. ], tot_loss[loss=3.326, NarTop10Accuracy=0.6527, over 5945.77 frames. ], batch size: 31, lr: 2.30e-03
2024-08-06 15:08:04,777 INFO [trainer.py:765] (1/8) Epoch 37, batch 1300, train_loss[loss=3.308, NarTop10Accuracy=0.6682, over 5000.00 frames. ], tot_loss[loss=3.335, NarTop10Accuracy=0.6506, over 6015.00 frames. ], batch size: 6, lr: 2.29e-03
2024-08-06 15:08:47,879 INFO [trainer.py:765] (1/8) Epoch 37, batch 1400, train_loss[loss=3.189, NarTop10Accuracy=0.6817, over 6144.00 frames. ], tot_loss[loss=3.342, NarTop10Accuracy=0.649, over 6024.28 frames. ], batch size: 11, lr: 2.29e-03
2024-08-06 15:09:16,180 INFO [trainer.py:765] (1/8) Epoch 37, batch 1500, train_loss[loss=3.691, NarTop10Accuracy=0.5892, over 6139.00 frames. ], tot_loss[loss=3.344, NarTop10Accuracy=0.6496, over 5971.53 frames. ], batch size: 49, lr: 2.29e-03
2024-08-06 15:09:44,190 INFO [trainer.py:765] (1/8) Epoch 37, batch 1600, train_loss[loss=3.399, NarTop10Accuracy=0.6349, over 7090.00 frames. ], tot_loss[loss=3.35, NarTop10Accuracy=0.6485, over 5952.76 frames. ], batch size: 22, lr: 2.29e-03
2024-08-06 15:10:11,082 INFO [trainer.py:765] (1/8) Epoch 37, batch 1700, train_loss[loss=3.431, NarTop10Accuracy=0.6361, over 6350.00 frames. ], tot_loss[loss=3.342, NarTop10Accuracy=0.6497, over 5946.30 frames. ], batch size: 13, lr: 2.29e-03
2024-08-06 15:10:37,752 INFO [trainer.py:765] (1/8) Epoch 37, batch 1800, train_loss[loss=3.325, NarTop10Accuracy=0.6566, over 7247.00 frames. ], tot_loss[loss=3.336, NarTop10Accuracy=0.6513, over 6008.13 frames. ], batch size: 22, lr: 2.29e-03
2024-08-06 15:11:04,270 INFO [trainer.py:765] (1/8) Epoch 37, batch 1900, train_loss[loss=3.655, NarTop10Accuracy=0.5855, over 6546.00 frames. ], tot_loss[loss=3.348, NarTop10Accuracy=0.6491, over 6044.17 frames. ], batch size: 49, lr: 2.29e-03
2024-08-06 15:11:29,941 INFO [trainer.py:765] (1/8) Epoch 37, batch 2000, train_loss[loss=3.582, NarTop10Accuracy=0.5958, over 6548.00 frames. ], tot_loss[loss=3.358, NarTop10Accuracy=0.6468, over 6024.99 frames. ], batch size: 49, lr: 2.29e-03
2024-08-06 15:11:58,797 INFO [trainer.py:765] (1/8) Epoch 37, batch 2100, train_loss[loss=3.157, NarTop10Accuracy=0.6828, over 3841.00 frames. ], tot_loss[loss=3.352, NarTop10Accuracy=0.6477, over 6000.82 frames. ], batch size: 4, lr: 2.29e-03
2024-08-06 15:12:24,312 INFO [trainer.py:765] (1/8) Epoch 37, batch 2200, train_loss[loss=3.248, NarTop10Accuracy=0.6624, over 7143.00 frames. ], tot_loss[loss=3.347, NarTop10Accuracy=0.6491, over 6052.86 frames. ], batch size: 30, lr: 2.28e-03
2024-08-06 15:12:49,787 INFO [trainer.py:765] (1/8) Epoch 37, batch 2300, train_loss[loss=3.138, NarTop10Accuracy=0.6913, over 5757.00 frames. ], tot_loss[loss=3.345, NarTop10Accuracy=0.6495, over 6072.49 frames. ], batch size: 9, lr: 2.28e-03
2024-08-06 15:13:14,526 INFO [trainer.py:765] (1/8) Epoch 37, batch 2400, train_loss[loss=3.318, NarTop10Accuracy=0.6586, over 5064.00 frames. ], tot_loss[loss=3.355, NarTop10Accuracy=0.6476, over 5876.88 frames. ], batch size: 7, lr: 2.28e-03
2024-08-06 15:13:37,942 INFO [trainer.py:765] (1/8) Epoch 37, batch 2500, train_loss[loss=3.452, NarTop10Accuracy=0.6257, over 5229.00 frames. ], tot_loss[loss=3.334, NarTop10Accuracy=0.6512, over 5516.36 frames. ], batch size: 6, lr: 2.28e-03
2024-08-06 15:13:59,183 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 15:14:50,846 INFO [trainer.py:765] (1/8) Epoch 38, batch 100, train_loss[loss=3.403, NarTop10Accuracy=0.6296, over 7201.00 frames. ], tot_loss[loss=3.311, NarTop10Accuracy=0.6559, over 2378.73 frames. ], batch size: 30, lr: 2.25e-03
2024-08-06 15:15:27,289 INFO [trainer.py:765] (1/8) Epoch 38, batch 200, train_loss[loss=3.138, NarTop10Accuracy=0.6907, over 6783.00 frames. ], tot_loss[loss=3.282, NarTop10Accuracy=0.6624, over 3865.44 frames. ], batch size: 17, lr: 2.25e-03
2024-08-06 15:16:01,281 INFO [trainer.py:765] (1/8) Epoch 38, batch 300, train_loss[loss=3.281, NarTop10Accuracy=0.654, over 6972.00 frames. ], tot_loss[loss=3.276, NarTop10Accuracy=0.6642, over 4680.42 frames. ], batch size: 22, lr: 2.25e-03
2024-08-06 15:16:32,595 INFO [trainer.py:765] (1/8) Epoch 38, batch 400, train_loss[loss=3.444, NarTop10Accuracy=0.6271, over 5209.00 frames. ], tot_loss[loss=3.281, NarTop10Accuracy=0.6625, over 5121.01 frames. ], batch size: 7, lr: 2.24e-03
2024-08-06 15:17:04,258 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 15:17:14,104 INFO [trainer.py:811] (1/8) Epoch 38, validation: loss=3.229, NarTop10Accuracy=0.6755, over 1907754.00 frames. 
2024-08-06 15:17:14,105 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 15:17:14,630 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.687e+02 2.062e+02 2.214e+02 2.396e+02 3.845e+02, threshold=4.429e+02, percent-clipped=0.0
2024-08-06 15:17:16,480 INFO [trainer.py:765] (1/8) Epoch 38, batch 500, train_loss[loss=3.297, NarTop10Accuracy=0.6548, over 6281.00 frames. ], tot_loss[loss=3.277, NarTop10Accuracy=0.6634, over 5409.71 frames. ], batch size: 11, lr: 2.24e-03
2024-08-06 15:17:53,875 INFO [trainer.py:765] (1/8) Epoch 38, batch 600, train_loss[loss=3.067, NarTop10Accuracy=0.6974, over 5779.00 frames. ], tot_loss[loss=3.281, NarTop10Accuracy=0.6627, over 5673.66 frames. ], batch size: 9, lr: 2.24e-03
2024-08-06 15:18:26,466 INFO [trainer.py:765] (1/8) Epoch 38, batch 700, train_loss[loss=3.22, NarTop10Accuracy=0.6703, over 4881.00 frames. ], tot_loss[loss=3.294, NarTop10Accuracy=0.6596, over 5748.83 frames. ], batch size: 6, lr: 2.24e-03
2024-08-06 15:19:01,129 INFO [trainer.py:765] (1/8) Epoch 38, batch 800, train_loss[loss=3.378, NarTop10Accuracy=0.6554, over 5143.00 frames. ], tot_loss[loss=3.31, NarTop10Accuracy=0.6568, over 5797.67 frames. ], batch size: 6, lr: 2.24e-03
2024-08-06 15:19:36,540 INFO [trainer.py:765] (1/8) Epoch 38, batch 900, train_loss[loss=3.436, NarTop10Accuracy=0.6208, over 6277.00 frames. ], tot_loss[loss=3.314, NarTop10Accuracy=0.6553, over 5809.31 frames. ], batch size: 13, lr: 2.24e-03
2024-08-06 15:20:09,134 INFO [trainer.py:765] (1/8) Epoch 38, batch 1000, train_loss[loss=3.444, NarTop10Accuracy=0.6326, over 6300.00 frames. ], tot_loss[loss=3.315, NarTop10Accuracy=0.6549, over 5904.44 frames. ], batch size: 13, lr: 2.24e-03
2024-08-06 15:20:47,346 INFO [trainer.py:765] (1/8) Epoch 38, batch 1100, train_loss[loss=3.693, NarTop10Accuracy=0.5832, over 6838.00 frames. ], tot_loss[loss=3.332, NarTop10Accuracy=0.6518, over 5943.90 frames. ], batch size: 17, lr: 2.24e-03
2024-08-06 15:21:25,594 INFO [trainer.py:765] (1/8) Epoch 38, batch 1200, train_loss[loss=3.339, NarTop10Accuracy=0.6563, over 7024.00 frames. ], tot_loss[loss=3.339, NarTop10Accuracy=0.6504, over 5936.99 frames. ], batch size: 30, lr: 2.23e-03
2024-08-06 15:21:57,556 INFO [trainer.py:765] (1/8) Epoch 38, batch 1300, train_loss[loss=3.294, NarTop10Accuracy=0.6588, over 5052.00 frames. ], tot_loss[loss=3.327, NarTop10Accuracy=0.6522, over 6009.82 frames. ], batch size: 6, lr: 2.23e-03
2024-08-06 15:22:29,468 INFO [trainer.py:765] (1/8) Epoch 38, batch 1400, train_loss[loss=3.059, NarTop10Accuracy=0.7113, over 6311.00 frames. ], tot_loss[loss=3.34, NarTop10Accuracy=0.65, over 6047.09 frames. ], batch size: 11, lr: 2.23e-03
2024-08-06 15:23:06,615 INFO [trainer.py:765] (1/8) Epoch 38, batch 1500, train_loss[loss=3.47, NarTop10Accuracy=0.6242, over 6071.00 frames. ], tot_loss[loss=3.347, NarTop10Accuracy=0.6487, over 5970.23 frames. ], batch size: 48, lr: 2.23e-03
2024-08-06 15:23:34,640 INFO [trainer.py:765] (1/8) Epoch 38, batch 1600, train_loss[loss=3.481, NarTop10Accuracy=0.6156, over 7340.00 frames. ], tot_loss[loss=3.355, NarTop10Accuracy=0.6472, over 5945.12 frames. ], batch size: 22, lr: 2.23e-03
2024-08-06 15:24:01,433 INFO [trainer.py:765] (1/8) Epoch 38, batch 1700, train_loss[loss=3.284, NarTop10Accuracy=0.6634, over 6137.00 frames. ], tot_loss[loss=3.36, NarTop10Accuracy=0.6465, over 5923.28 frames. ], batch size: 13, lr: 2.23e-03
2024-08-06 15:24:28,065 INFO [trainer.py:765] (1/8) Epoch 38, batch 1800, train_loss[loss=3.23, NarTop10Accuracy=0.6628, over 7134.00 frames. ], tot_loss[loss=3.359, NarTop10Accuracy=0.6462, over 6001.84 frames. ], batch size: 22, lr: 2.23e-03
2024-08-06 15:24:54,673 INFO [trainer.py:765] (1/8) Epoch 38, batch 1900, train_loss[loss=3.314, NarTop10Accuracy=0.653, over 6676.00 frames. ], tot_loss[loss=3.358, NarTop10Accuracy=0.6468, over 6054.68 frames. ], batch size: 51, lr: 2.23e-03
2024-08-06 15:25:20,411 INFO [trainer.py:765] (1/8) Epoch 38, batch 2000, train_loss[loss=3.618, NarTop10Accuracy=0.594, over 6205.00 frames. ], tot_loss[loss=3.351, NarTop10Accuracy=0.6481, over 6026.47 frames. ], batch size: 49, lr: 2.23e-03
2024-08-06 15:25:45,857 INFO [trainer.py:765] (1/8) Epoch 38, batch 2100, train_loss[loss=3.445, NarTop10Accuracy=0.6254, over 4884.00 frames. ], tot_loss[loss=3.351, NarTop10Accuracy=0.6483, over 6015.31 frames. ], batch size: 5, lr: 2.22e-03
2024-08-06 15:26:11,317 INFO [trainer.py:765] (1/8) Epoch 38, batch 2200, train_loss[loss=3.483, NarTop10Accuracy=0.6215, over 7361.00 frames. ], tot_loss[loss=3.341, NarTop10Accuracy=0.6501, over 6061.71 frames. ], batch size: 30, lr: 2.22e-03
2024-08-06 15:26:36,709 INFO [trainer.py:765] (1/8) Epoch 38, batch 2300, train_loss[loss=3.19, NarTop10Accuracy=0.6797, over 5696.00 frames. ], tot_loss[loss=3.359, NarTop10Accuracy=0.6468, over 6084.42 frames. ], batch size: 9, lr: 2.22e-03
2024-08-06 15:27:01,479 INFO [trainer.py:765] (1/8) Epoch 38, batch 2400, train_loss[loss=3.353, NarTop10Accuracy=0.6468, over 5258.00 frames. ], tot_loss[loss=3.367, NarTop10Accuracy=0.6446, over 5893.56 frames. ], batch size: 7, lr: 2.22e-03
2024-08-06 15:27:23,144 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 15:27:33,589 INFO [trainer.py:811] (1/8) Epoch 38, validation: loss=3.213, NarTop10Accuracy=0.6782, over 1907754.00 frames. 
2024-08-06 15:27:33,590 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 15:27:34,076 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.659e+02 2.098e+02 2.247e+02 2.437e+02 3.550e+02, threshold=4.494e+02, percent-clipped=0.0
2024-08-06 15:27:35,515 INFO [trainer.py:765] (1/8) Epoch 38, batch 2500, train_loss[loss=3.266, NarTop10Accuracy=0.6674, over 5006.00 frames. ], tot_loss[loss=3.334, NarTop10Accuracy=0.6512, over 5547.01 frames. ], batch size: 6, lr: 2.22e-03
2024-08-06 15:27:56,734 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 15:28:51,227 INFO [trainer.py:765] (1/8) Epoch 39, batch 100, train_loss[loss=3.247, NarTop10Accuracy=0.6654, over 7191.00 frames. ], tot_loss[loss=3.254, NarTop10Accuracy=0.6687, over 2355.94 frames. ], batch size: 30, lr: 2.19e-03
2024-08-06 15:29:28,052 INFO [trainer.py:765] (1/8) Epoch 39, batch 200, train_loss[loss=3.534, NarTop10Accuracy=0.6142, over 6947.00 frames. ], tot_loss[loss=3.266, NarTop10Accuracy=0.6658, over 3853.37 frames. ], batch size: 17, lr: 2.19e-03
2024-08-06 15:30:02,018 INFO [trainer.py:765] (1/8) Epoch 39, batch 300, train_loss[loss=3.141, NarTop10Accuracy=0.6926, over 7060.00 frames. ], tot_loss[loss=3.271, NarTop10Accuracy=0.6643, over 4677.27 frames. ], batch size: 22, lr: 2.19e-03
2024-08-06 15:30:32,992 INFO [trainer.py:765] (1/8) Epoch 39, batch 400, train_loss[loss=3.128, NarTop10Accuracy=0.7016, over 5044.00 frames. ], tot_loss[loss=3.286, NarTop10Accuracy=0.6622, over 5125.73 frames. ], batch size: 7, lr: 2.19e-03
2024-08-06 15:31:03,569 INFO [trainer.py:765] (1/8) Epoch 39, batch 500, train_loss[loss=3.033, NarTop10Accuracy=0.7111, over 6133.00 frames. ], tot_loss[loss=3.294, NarTop10Accuracy=0.6602, over 5411.98 frames. ], batch size: 11, lr: 2.18e-03
2024-08-06 15:31:40,850 INFO [trainer.py:765] (1/8) Epoch 39, batch 600, train_loss[loss=3.508, NarTop10Accuracy=0.6326, over 5781.00 frames. ], tot_loss[loss=3.294, NarTop10Accuracy=0.6605, over 5687.01 frames. ], batch size: 9, lr: 2.18e-03
2024-08-06 15:32:14,451 INFO [trainer.py:765] (1/8) Epoch 39, batch 700, train_loss[loss=3.291, NarTop10Accuracy=0.6662, over 5271.00 frames. ], tot_loss[loss=3.302, NarTop10Accuracy=0.6587, over 5751.46 frames. ], batch size: 6, lr: 2.18e-03
2024-08-06 15:32:44,165 INFO [trainer.py:765] (1/8) Epoch 39, batch 800, train_loss[loss=3.351, NarTop10Accuracy=0.6441, over 4896.00 frames. ], tot_loss[loss=3.309, NarTop10Accuracy=0.6573, over 5800.40 frames. ], batch size: 6, lr: 2.18e-03
2024-08-06 15:33:21,117 INFO [trainer.py:765] (1/8) Epoch 39, batch 900, train_loss[loss=3.117, NarTop10Accuracy=0.6908, over 6183.00 frames. ], tot_loss[loss=3.308, NarTop10Accuracy=0.6574, over 5825.34 frames. ], batch size: 13, lr: 2.18e-03
2024-08-06 15:34:02,655 INFO [trainer.py:765] (1/8) Epoch 39, batch 1000, train_loss[loss=3.08, NarTop10Accuracy=0.698, over 6666.00 frames. ], tot_loss[loss=3.31, NarTop10Accuracy=0.6566, over 5921.79 frames. ], batch size: 14, lr: 2.18e-03
2024-08-06 15:34:33,094 INFO [trainer.py:765] (1/8) Epoch 39, batch 1100, train_loss[loss=3.006, NarTop10Accuracy=0.7196, over 6944.00 frames. ], tot_loss[loss=3.314, NarTop10Accuracy=0.6555, over 5956.66 frames. ], batch size: 17, lr: 2.18e-03
2024-08-06 15:35:09,244 INFO [trainer.py:765] (1/8) Epoch 39, batch 1200, train_loss[loss=3.258, NarTop10Accuracy=0.6652, over 7362.00 frames. ], tot_loss[loss=3.31, NarTop10Accuracy=0.6555, over 5964.64 frames. ], batch size: 31, lr: 2.18e-03
2024-08-06 15:35:46,813 INFO [trainer.py:765] (1/8) Epoch 39, batch 1300, train_loss[loss=3.233, NarTop10Accuracy=0.6533, over 4448.00 frames. ], tot_loss[loss=3.314, NarTop10Accuracy=0.655, over 6021.21 frames. ], batch size: 5, lr: 2.18e-03
2024-08-06 15:36:18,850 INFO [trainer.py:765] (1/8) Epoch 39, batch 1400, train_loss[loss=3.165, NarTop10Accuracy=0.6801, over 6159.00 frames. ], tot_loss[loss=3.324, NarTop10Accuracy=0.653, over 6050.37 frames. ], batch size: 11, lr: 2.17e-03
2024-08-06 15:36:47,214 INFO [trainer.py:765] (1/8) Epoch 39, batch 1500, train_loss[loss=3.406, NarTop10Accuracy=0.635, over 5796.00 frames. ], tot_loss[loss=3.338, NarTop10Accuracy=0.6501, over 5967.59 frames. ], batch size: 48, lr: 2.17e-03
2024-08-06 15:37:15,216 INFO [trainer.py:765] (1/8) Epoch 39, batch 1600, train_loss[loss=3.304, NarTop10Accuracy=0.6591, over 7109.00 frames. ], tot_loss[loss=3.343, NarTop10Accuracy=0.6492, over 5951.14 frames. ], batch size: 22, lr: 2.17e-03
2024-08-06 15:37:41,883 INFO [trainer.py:765] (1/8) Epoch 39, batch 1700, train_loss[loss=3.303, NarTop10Accuracy=0.6517, over 6311.00 frames. ], tot_loss[loss=3.353, NarTop10Accuracy=0.6472, over 5952.45 frames. ], batch size: 13, lr: 2.17e-03
2024-08-06 15:38:08,509 INFO [trainer.py:765] (1/8) Epoch 39, batch 1800, train_loss[loss=3.225, NarTop10Accuracy=0.6711, over 7291.00 frames. ], tot_loss[loss=3.347, NarTop10Accuracy=0.6487, over 6011.11 frames. ], batch size: 22, lr: 2.17e-03
2024-08-06 15:38:35,253 INFO [trainer.py:765] (1/8) Epoch 39, batch 1900, train_loss[loss=3.37, NarTop10Accuracy=0.6473, over 5851.00 frames. ], tot_loss[loss=3.358, NarTop10Accuracy=0.6468, over 6028.05 frames. ], batch size: 49, lr: 2.17e-03
2024-08-06 15:38:37,990 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 15:38:48,262 INFO [trainer.py:811] (1/8) Epoch 39, validation: loss=3.177, NarTop10Accuracy=0.6866, over 1907754.00 frames. 
2024-08-06 15:38:48,263 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 15:38:48,768 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.714e+02 2.106e+02 2.266e+02 2.462e+02 4.274e+02, threshold=4.532e+02, percent-clipped=0.0
2024-08-06 15:39:11,227 INFO [trainer.py:765] (1/8) Epoch 39, batch 2000, train_loss[loss=3.485, NarTop10Accuracy=0.6175, over 6541.00 frames. ], tot_loss[loss=3.339, NarTop10Accuracy=0.6502, over 6002.94 frames. ], batch size: 48, lr: 2.17e-03
2024-08-06 15:39:36,691 INFO [trainer.py:765] (1/8) Epoch 39, batch 2100, train_loss[loss=3.591, NarTop10Accuracy=0.608, over 4837.00 frames. ], tot_loss[loss=3.347, NarTop10Accuracy=0.649, over 6008.66 frames. ], batch size: 5, lr: 2.17e-03
2024-08-06 15:40:02,086 INFO [trainer.py:765] (1/8) Epoch 39, batch 2200, train_loss[loss=3.46, NarTop10Accuracy=0.6383, over 7402.00 frames. ], tot_loss[loss=3.335, NarTop10Accuracy=0.6514, over 6045.21 frames. ], batch size: 31, lr: 2.17e-03
2024-08-06 15:40:27,496 INFO [trainer.py:765] (1/8) Epoch 39, batch 2300, train_loss[loss=3.397, NarTop10Accuracy=0.6445, over 5887.00 frames. ], tot_loss[loss=3.347, NarTop10Accuracy=0.649, over 6065.47 frames. ], batch size: 9, lr: 2.16e-03
2024-08-06 15:40:52,331 INFO [trainer.py:765] (1/8) Epoch 39, batch 2400, train_loss[loss=3.361, NarTop10Accuracy=0.6488, over 5752.00 frames. ], tot_loss[loss=3.346, NarTop10Accuracy=0.6489, over 5886.09 frames. ], batch size: 8, lr: 2.16e-03
2024-08-06 15:41:15,695 INFO [trainer.py:765] (1/8) Epoch 39, batch 2500, train_loss[loss=3.352, NarTop10Accuracy=0.6386, over 5177.00 frames. ], tot_loss[loss=3.317, NarTop10Accuracy=0.6541, over 5518.84 frames. ], batch size: 6, lr: 2.16e-03
2024-08-06 15:41:36,914 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 15:42:35,254 INFO [trainer.py:765] (1/8) Epoch 40, batch 100, train_loss[loss=3.534, NarTop10Accuracy=0.6157, over 7341.00 frames. ], tot_loss[loss=3.321, NarTop10Accuracy=0.6565, over 2366.31 frames. ], batch size: 31, lr: 2.13e-03
2024-08-06 15:43:09,645 INFO [trainer.py:765] (1/8) Epoch 40, batch 200, train_loss[loss=3.477, NarTop10Accuracy=0.6175, over 6961.00 frames. ], tot_loss[loss=3.293, NarTop10Accuracy=0.6605, over 3864.00 frames. ], batch size: 17, lr: 2.13e-03
2024-08-06 15:43:43,738 INFO [trainer.py:765] (1/8) Epoch 40, batch 300, train_loss[loss=3.319, NarTop10Accuracy=0.6386, over 7066.00 frames. ], tot_loss[loss=3.291, NarTop10Accuracy=0.661, over 4660.91 frames. ], batch size: 22, lr: 2.13e-03
2024-08-06 15:44:18,202 INFO [trainer.py:765] (1/8) Epoch 40, batch 400, train_loss[loss=3.267, NarTop10Accuracy=0.6743, over 5133.00 frames. ], tot_loss[loss=3.283, NarTop10Accuracy=0.6626, over 5121.87 frames. ], batch size: 7, lr: 2.13e-03
2024-08-06 15:44:50,257 INFO [trainer.py:765] (1/8) Epoch 40, batch 500, train_loss[loss=3.14, NarTop10Accuracy=0.7009, over 6055.00 frames. ], tot_loss[loss=3.276, NarTop10Accuracy=0.6641, over 5396.37 frames. ], batch size: 11, lr: 2.13e-03
2024-08-06 15:45:25,431 INFO [trainer.py:765] (1/8) Epoch 40, batch 600, train_loss[loss=3.201, NarTop10Accuracy=0.6754, over 5710.00 frames. ], tot_loss[loss=3.288, NarTop10Accuracy=0.6614, over 5674.73 frames. ], batch size: 9, lr: 2.13e-03
2024-08-06 15:45:58,647 INFO [trainer.py:765] (1/8) Epoch 40, batch 700, train_loss[loss=3.107, NarTop10Accuracy=0.6822, over 5047.00 frames. ], tot_loss[loss=3.302, NarTop10Accuracy=0.6583, over 5739.09 frames. ], batch size: 6, lr: 2.13e-03
2024-08-06 15:46:34,887 INFO [trainer.py:765] (1/8) Epoch 40, batch 800, train_loss[loss=3.288, NarTop10Accuracy=0.6612, over 5088.00 frames. ], tot_loss[loss=3.308, NarTop10Accuracy=0.6572, over 5792.27 frames. ], batch size: 6, lr: 2.13e-03
2024-08-06 15:47:07,290 INFO [trainer.py:765] (1/8) Epoch 40, batch 900, train_loss[loss=3.066, NarTop10Accuracy=0.6979, over 6761.00 frames. ], tot_loss[loss=3.303, NarTop10Accuracy=0.6578, over 5826.22 frames. ], batch size: 14, lr: 2.12e-03
2024-08-06 15:47:43,510 INFO [trainer.py:765] (1/8) Epoch 40, batch 1000, train_loss[loss=3.46, NarTop10Accuracy=0.6279, over 6236.00 frames. ], tot_loss[loss=3.311, NarTop10Accuracy=0.6561, over 5923.09 frames. ], batch size: 13, lr: 2.12e-03
2024-08-06 15:48:18,710 INFO [trainer.py:765] (1/8) Epoch 40, batch 1100, train_loss[loss=3.374, NarTop10Accuracy=0.6416, over 6851.00 frames. ], tot_loss[loss=3.317, NarTop10Accuracy=0.6548, over 5925.14 frames. ], batch size: 17, lr: 2.12e-03
2024-08-06 15:48:52,094 INFO [trainer.py:765] (1/8) Epoch 40, batch 1200, train_loss[loss=3.308, NarTop10Accuracy=0.6559, over 7025.00 frames. ], tot_loss[loss=3.319, NarTop10Accuracy=0.6544, over 5940.96 frames. ], batch size: 30, lr: 2.12e-03
2024-08-06 15:49:29,782 INFO [trainer.py:765] (1/8) Epoch 40, batch 1300, train_loss[loss=3.566, NarTop10Accuracy=0.6041, over 5061.00 frames. ], tot_loss[loss=3.313, NarTop10Accuracy=0.6553, over 6007.32 frames. ], batch size: 6, lr: 2.12e-03
2024-08-06 15:49:38,246 INFO [trainer.py:803] (1/8) Computing validation loss
2024-08-06 15:49:48,934 INFO [trainer.py:811] (1/8) Epoch 40, validation: loss=3.171, NarTop10Accuracy=0.6871, over 1907754.00 frames. 
2024-08-06 15:49:48,935 INFO [trainer.py:814] (1/8) Maximum memory allocated so far is 30619MB
2024-08-06 15:49:49,615 INFO [optim.py:386] (1/8) Clipping_scale=2.0, grad-norm quartiles 1.708e+02 2.095e+02 2.264e+02 2.441e+02 4.960e+02, threshold=4.528e+02, percent-clipped=0.1
2024-08-06 15:50:12,460 INFO [trainer.py:765] (1/8) Epoch 40, batch 1400, train_loss[loss=3.18, NarTop10Accuracy=0.6834, over 6247.00 frames. ], tot_loss[loss=3.315, NarTop10Accuracy=0.655, over 6024.68 frames. ], batch size: 11, lr: 2.12e-03
2024-08-06 15:50:45,930 INFO [trainer.py:765] (1/8) Epoch 40, batch 1500, train_loss[loss=3.479, NarTop10Accuracy=0.6162, over 6594.00 frames. ], tot_loss[loss=3.322, NarTop10Accuracy=0.6535, over 5968.11 frames. ], batch size: 49, lr: 2.12e-03
2024-08-06 15:51:13,821 INFO [trainer.py:765] (1/8) Epoch 40, batch 1600, train_loss[loss=3.1, NarTop10Accuracy=0.7007, over 7156.00 frames. ], tot_loss[loss=3.311, NarTop10Accuracy=0.6554, over 5951.95 frames. ], batch size: 22, lr: 2.12e-03
2024-08-06 15:51:40,571 INFO [trainer.py:765] (1/8) Epoch 40, batch 1700, train_loss[loss=3.281, NarTop10Accuracy=0.6446, over 6217.00 frames. ], tot_loss[loss=3.312, NarTop10Accuracy=0.6551, over 5930.64 frames. ], batch size: 13, lr: 2.12e-03
2024-08-06 15:52:07,236 INFO [trainer.py:765] (1/8) Epoch 40, batch 1800, train_loss[loss=3.636, NarTop10Accuracy=0.5927, over 6940.00 frames. ], tot_loss[loss=3.319, NarTop10Accuracy=0.654, over 5995.04 frames. ], batch size: 22, lr: 2.12e-03
2024-08-06 15:52:33,820 INFO [trainer.py:765] (1/8) Epoch 40, batch 1900, train_loss[loss=3.505, NarTop10Accuracy=0.6156, over 6048.00 frames. ], tot_loss[loss=3.334, NarTop10Accuracy=0.6513, over 6043.57 frames. ], batch size: 50, lr: 2.11e-03
2024-08-06 15:52:59,511 INFO [trainer.py:765] (1/8) Epoch 40, batch 2000, train_loss[loss=3.39, NarTop10Accuracy=0.6366, over 6364.00 frames. ], tot_loss[loss=3.333, NarTop10Accuracy=0.6516, over 6019.08 frames. ], batch size: 49, lr: 2.11e-03
2024-08-06 15:53:24,913 INFO [trainer.py:765] (1/8) Epoch 40, batch 2100, train_loss[loss=2.98, NarTop10Accuracy=0.7174, over 3996.00 frames. ], tot_loss[loss=3.326, NarTop10Accuracy=0.6526, over 5991.72 frames. ], batch size: 4, lr: 2.11e-03
2024-08-06 15:53:50,419 INFO [trainer.py:765] (1/8) Epoch 40, batch 2200, train_loss[loss=3.338, NarTop10Accuracy=0.6509, over 7446.00 frames. ], tot_loss[loss=3.326, NarTop10Accuracy=0.6525, over 6045.66 frames. ], batch size: 31, lr: 2.11e-03
2024-08-06 15:54:15,886 INFO [trainer.py:765] (1/8) Epoch 40, batch 2300, train_loss[loss=3.246, NarTop10Accuracy=0.6696, over 5677.00 frames. ], tot_loss[loss=3.332, NarTop10Accuracy=0.652, over 6081.15 frames. ], batch size: 9, lr: 2.11e-03
2024-08-06 15:54:43,787 INFO [trainer.py:765] (1/8) Epoch 40, batch 2400, train_loss[loss=3.578, NarTop10Accuracy=0.6072, over 5134.00 frames. ], tot_loss[loss=3.341, NarTop10Accuracy=0.6501, over 5890.04 frames. ], batch size: 7, lr: 2.11e-03
2024-08-06 15:55:07,364 INFO [trainer.py:765] (1/8) Epoch 40, batch 2500, train_loss[loss=3.109, NarTop10Accuracy=0.712, over 5095.00 frames. ], tot_loss[loss=3.308, NarTop10Accuracy=0.656, over 5549.56 frames. ], batch size: 6, lr: 2.11e-03
2024-08-06 15:55:28,694 INFO [trainer.py:650] (1/8) Reaches end of dataloader.
2024-08-06 15:55:28,697 INFO [trainer.py:1069] (1/8) Done!