2024-08-06 06:41:41,476 INFO [trainer.py:870] (0/8) Training started
2024-08-06 06:41:41,481 INFO [trainer.py:889] (0/8) Device: cuda:0
2024-08-06 06:41:41,481 INFO [trainer.py:890] (0/8) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 100, 'reset_interval': 200, 'valid_interval': 2000, 'env_info': {'k2-version': '1.24.3', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '279b0c87015a615b81b147251814d737a548f397', 'k2-git-date': 'Wed May 24 22:24:09 2023', 'lhotse-version': '1.26.0', 'torch-version': '2.0.1+cu118', 'torch-cuda-available': True, 'torch-cuda-version': '11.8', 'python-version': '3.10', 'icefall-git-branch': 'main', 'icefall-git-sha1': '3e4fbb6-dirty', 'icefall-git-date': 'Tue Aug 6 06:30:45 2024', 'icefall-path': '/workspace/icefall_llm', 'k2-path': '/usr/local/lib/python3.10/dist-packages/k2/__init__.py', 'lhotse-path': '/usr/local/lib/python3.10/dist-packages/lhotse/__init__.py', 'hostname': '6865771', 'IP address': '0.104.195.107'}, 'world_size': 8, 'master_port': 12354, 'tensorboard': True, 'num_epochs': 40, 'start_epoch': 100, 'start_batch': 0, 'exp_dir': PosixPath('exp/valle'), 'optimizer_name': 'ScaledAdam', 'scheduler_name': 'Eden', 'base_lr': 0.03, 'warmup_steps': 200, 'seed': 42, 'inf_check': False, 'save_every_n': 1000, 'keep_last_k': 20, 'average_period': 0, 'accumulate_grad_steps': 2, 'dtype': 'float32', 'filter_min_duration': 0.5, 'filter_max_duration': 14.0, 'train_stage': 2, 'visualize': False, 'oom_check': False, 'model_name': 'valle', 'decoder_dim': 1024, 'nhead': 16, 'num_decoder_layers': 12, 'scale_factor': 1.0, 'norm_first': True, 'add_prenet': False, 'prefix_mode': 1, 'share_embedding': True, 'prepend_bos': False, 'num_quantizers': 8, 'scaling_xformers': False, 'manifest_dir': PosixPath('data/tokenized'), 'max_duration': 160, 'bucketing_sampler': True, 'num_buckets': 6, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 0.1, 'on_the_fly_feats': False, 'shuffle': True, 'buffer_size': 40000, 'shuffle_buffer_size': 100000, 'drop_last': False, 'return_cuts': True, 'num_workers': 8, 'enable_spec_aug': False, 'spec_aug_time_warp_factor': 80, 'input_strategy': 'PrecomputedFeatures', 'dataset': 'libritts', 'text_tokens': 'data/tokenized/unique_text_tokens.k2symbols', 'sampling_rate': 24000}
2024-08-06 06:41:41,481 INFO [trainer.py:892] (0/8) About to create model
2024-08-06 06:41:42,388 INFO [trainer.py:899] (0/8) Number of model parameters: 367386628
2024-08-06 06:41:42,388 INFO [checkpoint.py:112] (0/8) Loading checkpoint from exp/valle/epoch-99.pt
2024-08-06 06:41:44,859 INFO [trainer.py:914] (0/8) Using DDP
2024-08-06 06:41:46,902 INFO [datamodule.py:427] (0/8) About to get train cuts
2024-08-06 06:41:46,905 INFO [datamodule.py:434] (0/8) About to get dev cuts
2024-08-06 06:41:46,906 INFO [datamodule.py:292] (0/8) Disable SpecAugment
2024-08-06 06:41:46,906 INFO [datamodule.py:294] (0/8) About to create train dataset
2024-08-06 06:41:46,906 INFO [datamodule.py:323] (0/8) Using DynamicBucketingSampler
2024-08-06 06:41:47,527 INFO [datamodule.py:344] (0/8) About to create train dataloader
2024-08-06 06:41:47,527 INFO [datamodule.py:367] (0/8) About to create dev dataset
2024-08-06 06:41:47,861 INFO [datamodule.py:388] (0/8) About to create dev dataloader
2024-08-06 06:42:36,135 INFO [trainer.py:765] (0/8) Epoch 1, batch 100, train_loss[loss=93.96, NarTop10Accuracy=0.02016, over 7247.00 frames. ], tot_loss[loss=80.09, NarTop10Accuracy=0.05241, over 2370.46 frames. ], batch size: 31, lr: 2.25e-02
2024-08-06 06:43:05,818 INFO [trainer.py:765] (0/8) Epoch 1, batch 200, train_loss[loss=126.1, NarTop10Accuracy=0.02241, over 6956.00 frames. ], tot_loss[loss=98.98, NarTop10Accuracy=0.04469, over 3855.17 frames. ], batch size: 17, lr: 3.00e-02
2024-08-06 06:43:33,848 INFO [trainer.py:765] (0/8) Epoch 1, batch 300, train_loss[loss=74.09, NarTop10Accuracy=0.02428, over 7160.00 frames. ], tot_loss[loss=86.75, NarTop10Accuracy=0.04646, over 4649.32 frames. ], batch size: 22, lr: 3.00e-02
2024-08-06 06:44:05,251 INFO [trainer.py:765] (0/8) Epoch 1, batch 400, train_loss[loss=32.01, NarTop10Accuracy=0.05699, over 5162.00 frames. ], tot_loss[loss=67.92, NarTop10Accuracy=0.05099, over 5120.94 frames. ], batch size: 7, lr: 3.00e-02
2024-08-06 06:44:33,445 INFO [trainer.py:765] (0/8) Epoch 1, batch 500, train_loss[loss=16.29, NarTop10Accuracy=0.02792, over 6165.00 frames. ], tot_loss[loss=48.69, NarTop10Accuracy=0.05609, over 5399.63 frames. ], batch size: 11, lr: 2.99e-02
2024-08-06 06:45:02,924 INFO [trainer.py:765] (0/8) Epoch 1, batch 600, train_loss[loss=5.921, NarTop10Accuracy=0.1809, over 5785.00 frames. ], tot_loss[loss=33.34, NarTop10Accuracy=0.06239, over 5667.07 frames. ], batch size: 9, lr: 2.99e-02
2024-08-06 06:45:40,481 INFO [trainer.py:765] (0/8) Epoch 1, batch 700, train_loss[loss=7.066, NarTop10Accuracy=0.1038, over 5001.00 frames. ], tot_loss[loss=23.5, NarTop10Accuracy=0.07028, over 5746.27 frames. ], batch size: 6, lr: 2.99e-02
2024-08-06 06:46:09,663 INFO [trainer.py:765] (0/8) Epoch 1, batch 800, train_loss[loss=6.654, NarTop10Accuracy=0.09871, over 4448.00 frames. ], tot_loss[loss=17.52, NarTop10Accuracy=0.08054, over 5789.51 frames. ], batch size: 5, lr: 2.98e-02
2024-08-06 06:46:37,733 INFO [trainer.py:765] (0/8) Epoch 1, batch 900, train_loss[loss=6.064, NarTop10Accuracy=0.1631, over 6318.00 frames. ], tot_loss[loss=13.02, NarTop10Accuracy=0.1115, over 5813.80 frames. ], batch size: 13, lr: 2.98e-02
2024-08-06 06:47:13,908 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-1000.pt
2024-08-06 06:47:17,131 INFO [trainer.py:765] (0/8) Epoch 1, batch 1000, train_loss[loss=5.936, NarTop10Accuracy=0.1929, over 6259.00 frames. ], tot_loss[loss=10.17, NarTop10Accuracy=0.1374, over 5906.22 frames. ], batch size: 13, lr: 2.97e-02
2024-08-06 06:47:47,141 INFO [trainer.py:765] (0/8) Epoch 1, batch 1100, train_loss[loss=5.61, NarTop10Accuracy=0.1818, over 6846.00 frames. ], tot_loss[loss=8.416, NarTop10Accuracy=0.1573, over 5941.35 frames. ], batch size: 17, lr: 2.96e-02
2024-08-06 06:48:15,709 INFO [trainer.py:765] (0/8) Epoch 1, batch 1200, train_loss[loss=6.284, NarTop10Accuracy=0.142, over 7266.00 frames. ], tot_loss[loss=7.309, NarTop10Accuracy=0.174, over 5955.98 frames. ], batch size: 30, lr: 2.96e-02
2024-08-06 06:48:47,235 INFO [trainer.py:765] (0/8) Epoch 1, batch 1300, train_loss[loss=5.459, NarTop10Accuracy=0.2113, over 4197.00 frames. ], tot_loss[loss=6.614, NarTop10Accuracy=0.1851, over 6014.77 frames. ], batch size: 5, lr: 2.95e-02
2024-08-06 06:49:23,567 INFO [trainer.py:765] (0/8) Epoch 1, batch 1400, train_loss[loss=5.373, NarTop10Accuracy=0.2092, over 6088.00 frames. ], tot_loss[loss=6.191, NarTop10Accuracy=0.1918, over 6033.51 frames. ], batch size: 11, lr: 2.94e-02
2024-08-06 06:49:51,506 INFO [trainer.py:765] (0/8) Epoch 1, batch 1500, train_loss[loss=5.58, NarTop10Accuracy=0.1949, over 6435.00 frames. ], tot_loss[loss=5.925, NarTop10Accuracy=0.1986, over 5983.61 frames. ], batch size: 49, lr: 2.94e-02
2024-08-06 06:50:19,162 INFO [trainer.py:765] (0/8) Epoch 1, batch 1600, train_loss[loss=5.545, NarTop10Accuracy=0.2081, over 7181.00 frames. ], tot_loss[loss=5.748, NarTop10Accuracy=0.2047, over 5961.91 frames. ], batch size: 22, lr: 2.93e-02
2024-08-06 06:50:45,597 INFO [trainer.py:765] (0/8) Epoch 1, batch 1700, train_loss[loss=5.247, NarTop10Accuracy=0.2516, over 6695.00 frames. ], tot_loss[loss=5.628, NarTop10Accuracy=0.2103, over 5947.33 frames. ], batch size: 14, lr: 2.92e-02
2024-08-06 06:51:11,954 INFO [trainer.py:765] (0/8) Epoch 1, batch 1800, train_loss[loss=5.415, NarTop10Accuracy=0.2226, over 7117.00 frames. ], tot_loss[loss=5.548, NarTop10Accuracy=0.2161, over 6025.23 frames. ], batch size: 22, lr: 2.91e-02
2024-08-06 06:51:38,222 INFO [trainer.py:765] (0/8) Epoch 1, batch 1900, train_loss[loss=5.482, NarTop10Accuracy=0.2136, over 6163.00 frames. ], tot_loss[loss=5.492, NarTop10Accuracy=0.2209, over 6050.10 frames. ], batch size: 51, lr: 2.90e-02
2024-08-06 06:52:03,652 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-2000.pt
2024-08-06 06:52:07,190 INFO [trainer.py:765] (0/8) Epoch 1, batch 2000, train_loss[loss=5.5, NarTop10Accuracy=0.2182, over 6349.00 frames. ], tot_loss[loss=5.438, NarTop10Accuracy=0.2287, over 6013.65 frames. ], batch size: 50, lr: 2.89e-02
2024-08-06 06:52:07,192 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 06:52:13,994 INFO [trainer.py:811] (0/8) Epoch 1, validation: loss=5.351, NarTop10Accuracy=0.2423, over 1907754.00 frames. 
2024-08-06 06:52:13,994 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 29794MB
2024-08-06 06:52:14,534 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 4.341e+01 2.262e+02 7.241e+02 2.074e+04 7.259e+05, threshold=1.448e+03, percent-clipped=0.0
2024-08-06 06:52:39,585 INFO [trainer.py:765] (0/8) Epoch 1, batch 2100, train_loss[loss=5.466, NarTop10Accuracy=0.224, over 4745.00 frames. ], tot_loss[loss=5.393, NarTop10Accuracy=0.2367, over 6010.12 frames. ], batch size: 5, lr: 2.88e-02
2024-08-06 06:53:05,354 INFO [trainer.py:765] (0/8) Epoch 1, batch 2200, train_loss[loss=5.164, NarTop10Accuracy=0.285, over 7317.00 frames. ], tot_loss[loss=5.36, NarTop10Accuracy=0.2409, over 6049.38 frames. ], batch size: 31, lr: 2.87e-02
2024-08-06 06:53:30,700 INFO [trainer.py:765] (0/8) Epoch 1, batch 2300, train_loss[loss=5.41, NarTop10Accuracy=0.2237, over 5848.00 frames. ], tot_loss[loss=5.345, NarTop10Accuracy=0.2433, over 6072.25 frames. ], batch size: 9, lr: 2.86e-02
2024-08-06 06:53:55,358 INFO [trainer.py:765] (0/8) Epoch 1, batch 2400, train_loss[loss=5.3, NarTop10Accuracy=0.2495, over 6106.00 frames. ], tot_loss[loss=5.317, NarTop10Accuracy=0.2493, over 5902.86 frames. ], batch size: 48, lr: 2.85e-02
2024-08-06 06:54:18,659 INFO [trainer.py:765] (0/8) Epoch 1, batch 2500, train_loss[loss=5.107, NarTop10Accuracy=0.3016, over 5209.00 frames. ], tot_loss[loss=5.258, NarTop10Accuracy=0.2607, over 5555.34 frames. ], batch size: 6, lr: 2.84e-02
2024-08-06 06:54:39,603 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 06:54:39,606 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-1.pt
2024-08-06 06:55:37,936 INFO [trainer.py:765] (0/8) Epoch 2, batch 100, train_loss[loss=5.107, NarTop10Accuracy=0.2974, over 7423.00 frames. ], tot_loss[loss=5.185, NarTop10Accuracy=0.2793, over 2373.96 frames. ], batch size: 31, lr: 2.77e-02
2024-08-06 06:56:16,405 INFO [trainer.py:765] (0/8) Epoch 2, batch 200, train_loss[loss=5.004, NarTop10Accuracy=0.3274, over 6486.00 frames. ], tot_loss[loss=5.164, NarTop10Accuracy=0.2823, over 3861.00 frames. ], batch size: 16, lr: 2.76e-02
2024-08-06 06:56:44,972 INFO [trainer.py:765] (0/8) Epoch 2, batch 300, train_loss[loss=5.149, NarTop10Accuracy=0.2906, over 7045.00 frames. ], tot_loss[loss=5.157, NarTop10Accuracy=0.2838, over 4668.94 frames. ], batch size: 22, lr: 2.75e-02
2024-08-06 06:57:13,938 INFO [trainer.py:765] (0/8) Epoch 2, batch 400, train_loss[loss=5.292, NarTop10Accuracy=0.24, over 5163.00 frames. ], tot_loss[loss=5.148, NarTop10Accuracy=0.2853, over 5112.78 frames. ], batch size: 7, lr: 2.74e-02
2024-08-06 06:57:18,919 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-3000.pt
2024-08-06 06:57:56,208 INFO [trainer.py:765] (0/8) Epoch 2, batch 500, train_loss[loss=4.984, NarTop10Accuracy=0.3199, over 6097.00 frames. ], tot_loss[loss=5.111, NarTop10Accuracy=0.2922, over 5392.69 frames. ], batch size: 11, lr: 2.73e-02
2024-08-06 06:58:25,425 INFO [trainer.py:765] (0/8) Epoch 2, batch 600, train_loss[loss=4.903, NarTop10Accuracy=0.3415, over 5838.00 frames. ], tot_loss[loss=5.091, NarTop10Accuracy=0.2964, over 5666.11 frames. ], batch size: 9, lr: 2.71e-02
2024-08-06 06:58:55,282 INFO [trainer.py:765] (0/8) Epoch 2, batch 700, train_loss[loss=4.706, NarTop10Accuracy=0.3613, over 4987.00 frames. ], tot_loss[loss=5.08, NarTop10Accuracy=0.2984, over 5734.87 frames. ], batch size: 6, lr: 2.70e-02
2024-08-06 06:59:31,889 INFO [trainer.py:765] (0/8) Epoch 2, batch 800, train_loss[loss=5.114, NarTop10Accuracy=0.273, over 5006.00 frames. ], tot_loss[loss=5.082, NarTop10Accuracy=0.2977, over 5789.50 frames. ], batch size: 6, lr: 2.69e-02
2024-08-06 07:00:03,183 INFO [trainer.py:765] (0/8) Epoch 2, batch 900, train_loss[loss=5.388, NarTop10Accuracy=0.2271, over 6248.00 frames. ], tot_loss[loss=5.05, NarTop10Accuracy=0.3043, over 5820.64 frames. ], batch size: 13, lr: 2.68e-02
2024-08-06 07:00:33,141 INFO [trainer.py:765] (0/8) Epoch 2, batch 1000, train_loss[loss=4.861, NarTop10Accuracy=0.34, over 6331.00 frames. ], tot_loss[loss=5.019, NarTop10Accuracy=0.3106, over 5931.97 frames. ], batch size: 13, lr: 2.66e-02
2024-08-06 07:01:05,572 INFO [trainer.py:765] (0/8) Epoch 2, batch 1100, train_loss[loss=4.974, NarTop10Accuracy=0.317, over 6833.00 frames. ], tot_loss[loss=5.001, NarTop10Accuracy=0.3133, over 5970.00 frames. ], batch size: 17, lr: 2.65e-02
2024-08-06 07:01:46,285 INFO [trainer.py:765] (0/8) Epoch 2, batch 1200, train_loss[loss=4.804, NarTop10Accuracy=0.3518, over 7399.00 frames. ], tot_loss[loss=4.994, NarTop10Accuracy=0.3142, over 5966.24 frames. ], batch size: 31, lr: 2.64e-02
2024-08-06 07:02:15,644 INFO [trainer.py:765] (0/8) Epoch 2, batch 1300, train_loss[loss=5.246, NarTop10Accuracy=0.245, over 5093.00 frames. ], tot_loss[loss=4.956, NarTop10Accuracy=0.3217, over 6021.52 frames. ], batch size: 6, lr: 2.63e-02
2024-08-06 07:02:45,252 INFO [trainer.py:765] (0/8) Epoch 2, batch 1400, train_loss[loss=4.802, NarTop10Accuracy=0.3457, over 6083.00 frames. ], tot_loss[loss=4.941, NarTop10Accuracy=0.3247, over 6036.88 frames. ], batch size: 11, lr: 2.61e-02
2024-08-06 07:02:50,267 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-4000.pt
2024-08-06 07:02:54,010 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 07:03:02,094 INFO [trainer.py:811] (0/8) Epoch 2, validation: loss=4.943, NarTop10Accuracy=0.3266, over 1907754.00 frames. 
2024-08-06 07:03:02,095 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 29794MB
2024-08-06 07:03:02,638 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 5.429e+01 1.166e+02 1.425e+02 1.750e+02 6.435e+02, threshold=2.851e+02, percent-clipped=0.0
2024-08-06 07:03:25,472 INFO [trainer.py:765] (0/8) Epoch 2, batch 1500, train_loss[loss=5.067, NarTop10Accuracy=0.3091, over 6205.00 frames. ], tot_loss[loss=4.936, NarTop10Accuracy=0.3252, over 5965.12 frames. ], batch size: 49, lr: 2.60e-02
2024-08-06 07:03:53,553 INFO [trainer.py:765] (0/8) Epoch 2, batch 1600, train_loss[loss=4.736, NarTop10Accuracy=0.3577, over 7148.00 frames. ], tot_loss[loss=4.919, NarTop10Accuracy=0.3286, over 5952.23 frames. ], batch size: 22, lr: 2.59e-02
2024-08-06 07:04:20,313 INFO [trainer.py:765] (0/8) Epoch 2, batch 1700, train_loss[loss=4.841, NarTop10Accuracy=0.3504, over 6386.00 frames. ], tot_loss[loss=4.914, NarTop10Accuracy=0.3306, over 5937.54 frames. ], batch size: 13, lr: 2.58e-02
2024-08-06 07:04:46,888 INFO [trainer.py:765] (0/8) Epoch 2, batch 1800, train_loss[loss=4.922, NarTop10Accuracy=0.3277, over 7229.00 frames. ], tot_loss[loss=4.896, NarTop10Accuracy=0.3339, over 6006.51 frames. ], batch size: 22, lr: 2.56e-02
2024-08-06 07:05:13,587 INFO [trainer.py:765] (0/8) Epoch 2, batch 1900, train_loss[loss=4.81, NarTop10Accuracy=0.3594, over 5208.00 frames. ], tot_loss[loss=4.875, NarTop10Accuracy=0.3384, over 6037.71 frames. ], batch size: 49, lr: 2.55e-02
2024-08-06 07:05:39,285 INFO [trainer.py:765] (0/8) Epoch 2, batch 2000, train_loss[loss=4.661, NarTop10Accuracy=0.3837, over 6449.00 frames. ], tot_loss[loss=4.851, NarTop10Accuracy=0.3429, over 6038.78 frames. ], batch size: 49, lr: 2.54e-02
2024-08-06 07:06:04,829 INFO [trainer.py:765] (0/8) Epoch 2, batch 2100, train_loss[loss=4.561, NarTop10Accuracy=0.4018, over 4970.00 frames. ], tot_loss[loss=4.856, NarTop10Accuracy=0.3421, over 6016.72 frames. ], batch size: 5, lr: 2.52e-02
2024-08-06 07:06:30,372 INFO [trainer.py:765] (0/8) Epoch 2, batch 2200, train_loss[loss=4.835, NarTop10Accuracy=0.3473, over 7292.00 frames. ], tot_loss[loss=4.816, NarTop10Accuracy=0.3505, over 6055.14 frames. ], batch size: 31, lr: 2.51e-02
2024-08-06 07:06:55,874 INFO [trainer.py:765] (0/8) Epoch 2, batch 2300, train_loss[loss=4.2, NarTop10Accuracy=0.4668, over 5777.00 frames. ], tot_loss[loss=4.802, NarTop10Accuracy=0.3537, over 6090.25 frames. ], batch size: 9, lr: 2.50e-02
2024-08-06 07:07:20,576 INFO [trainer.py:765] (0/8) Epoch 2, batch 2400, train_loss[loss=4.942, NarTop10Accuracy=0.3168, over 5457.00 frames. ], tot_loss[loss=4.777, NarTop10Accuracy=0.359, over 5889.56 frames. ], batch size: 49, lr: 2.49e-02
2024-08-06 07:07:24,788 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-5000.pt
2024-08-06 07:07:47,111 INFO [trainer.py:765] (0/8) Epoch 2, batch 2500, train_loss[loss=4.602, NarTop10Accuracy=0.3809, over 5104.00 frames. ], tot_loss[loss=4.745, NarTop10Accuracy=0.3655, over 5555.13 frames. ], batch size: 6, lr: 2.47e-02
2024-08-06 07:08:08,378 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 07:08:08,382 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-2.pt
2024-08-06 07:09:08,538 INFO [trainer.py:765] (0/8) Epoch 3, batch 100, train_loss[loss=5.007, NarTop10Accuracy=0.3196, over 7128.00 frames. ], tot_loss[loss=4.639, NarTop10Accuracy=0.3864, over 2395.52 frames. ], batch size: 30, lr: 2.35e-02
2024-08-06 07:09:41,499 INFO [trainer.py:765] (0/8) Epoch 3, batch 200, train_loss[loss=4.336, NarTop10Accuracy=0.4421, over 6873.00 frames. ], tot_loss[loss=4.614, NarTop10Accuracy=0.3917, over 3879.35 frames. ], batch size: 17, lr: 2.34e-02
2024-08-06 07:10:16,976 INFO [trainer.py:765] (0/8) Epoch 3, batch 300, train_loss[loss=4.443, NarTop10Accuracy=0.4248, over 7191.00 frames. ], tot_loss[loss=4.606, NarTop10Accuracy=0.3923, over 4667.32 frames. ], batch size: 22, lr: 2.33e-02
2024-08-06 07:10:49,792 INFO [trainer.py:765] (0/8) Epoch 3, batch 400, train_loss[loss=4.52, NarTop10Accuracy=0.4115, over 5232.00 frames. ], tot_loss[loss=4.576, NarTop10Accuracy=0.3981, over 5119.60 frames. ], batch size: 7, lr: 2.32e-02
2024-08-06 07:11:18,179 INFO [trainer.py:765] (0/8) Epoch 3, batch 500, train_loss[loss=4.636, NarTop10Accuracy=0.387, over 6175.00 frames. ], tot_loss[loss=4.585, NarTop10Accuracy=0.3967, over 5400.55 frames. ], batch size: 11, lr: 2.31e-02
2024-08-06 07:11:51,263 INFO [trainer.py:765] (0/8) Epoch 3, batch 600, train_loss[loss=4.497, NarTop10Accuracy=0.4153, over 5869.00 frames. ], tot_loss[loss=4.564, NarTop10Accuracy=0.4006, over 5667.42 frames. ], batch size: 9, lr: 2.30e-02
2024-08-06 07:12:32,101 INFO [trainer.py:765] (0/8) Epoch 3, batch 700, train_loss[loss=4.554, NarTop10Accuracy=0.4046, over 5124.00 frames. ], tot_loss[loss=4.552, NarTop10Accuracy=0.4031, over 5751.37 frames. ], batch size: 6, lr: 2.29e-02
2024-08-06 07:13:01,919 INFO [trainer.py:765] (0/8) Epoch 3, batch 800, train_loss[loss=4.45, NarTop10Accuracy=0.4264, over 4969.00 frames. ], tot_loss[loss=4.534, NarTop10Accuracy=0.4065, over 5803.93 frames. ], batch size: 6, lr: 2.27e-02
2024-08-06 07:13:12,668 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-6000.pt
2024-08-06 07:13:16,175 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 07:13:22,883 INFO [trainer.py:811] (0/8) Epoch 3, validation: loss=4.43, NarTop10Accuracy=0.4285, over 1907754.00 frames. 
2024-08-06 07:13:22,884 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 29794MB
2024-08-06 07:13:23,429 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 6.823e+01 1.318e+02 1.583e+02 1.978e+02 8.364e+02, threshold=3.166e+02, percent-clipped=5.2
2024-08-06 07:13:42,435 INFO [trainer.py:765] (0/8) Epoch 3, batch 900, train_loss[loss=4.246, NarTop10Accuracy=0.4538, over 6706.00 frames. ], tot_loss[loss=4.517, NarTop10Accuracy=0.4102, over 5818.43 frames. ], batch size: 14, lr: 2.26e-02
2024-08-06 07:14:25,627 INFO [trainer.py:765] (0/8) Epoch 3, batch 1000, train_loss[loss=4.282, NarTop10Accuracy=0.4584, over 6615.00 frames. ], tot_loss[loss=4.498, NarTop10Accuracy=0.4135, over 5924.68 frames. ], batch size: 14, lr: 2.25e-02
2024-08-06 07:14:56,325 INFO [trainer.py:765] (0/8) Epoch 3, batch 1100, train_loss[loss=4.468, NarTop10Accuracy=0.4242, over 6930.00 frames. ], tot_loss[loss=4.492, NarTop10Accuracy=0.4142, over 5983.32 frames. ], batch size: 17, lr: 2.24e-02
2024-08-06 07:15:29,866 INFO [trainer.py:765] (0/8) Epoch 3, batch 1200, train_loss[loss=4.324, NarTop10Accuracy=0.4345, over 7263.00 frames. ], tot_loss[loss=4.481, NarTop10Accuracy=0.4166, over 5973.90 frames. ], batch size: 31, lr: 2.23e-02
2024-08-06 07:16:12,665 INFO [trainer.py:765] (0/8) Epoch 3, batch 1300, train_loss[loss=4.505, NarTop10Accuracy=0.4198, over 4982.00 frames. ], tot_loss[loss=4.458, NarTop10Accuracy=0.4211, over 6023.75 frames. ], batch size: 6, lr: 2.22e-02
2024-08-06 07:16:42,204 INFO [trainer.py:765] (0/8) Epoch 3, batch 1400, train_loss[loss=4.225, NarTop10Accuracy=0.4661, over 6138.00 frames. ], tot_loss[loss=4.45, NarTop10Accuracy=0.4225, over 6034.71 frames. ], batch size: 11, lr: 2.21e-02
2024-08-06 07:17:10,663 INFO [trainer.py:765] (0/8) Epoch 3, batch 1500, train_loss[loss=4.618, NarTop10Accuracy=0.3886, over 6121.00 frames. ], tot_loss[loss=4.445, NarTop10Accuracy=0.4233, over 5970.62 frames. ], batch size: 49, lr: 2.20e-02
2024-08-06 07:17:38,769 INFO [trainer.py:765] (0/8) Epoch 3, batch 1600, train_loss[loss=4.131, NarTop10Accuracy=0.4947, over 7237.00 frames. ], tot_loss[loss=4.42, NarTop10Accuracy=0.4283, over 5945.24 frames. ], batch size: 22, lr: 2.19e-02
2024-08-06 07:18:05,503 INFO [trainer.py:765] (0/8) Epoch 3, batch 1700, train_loss[loss=4.357, NarTop10Accuracy=0.4364, over 6318.00 frames. ], tot_loss[loss=4.404, NarTop10Accuracy=0.4316, over 5933.14 frames. ], batch size: 13, lr: 2.18e-02
2024-08-06 07:18:32,160 INFO [trainer.py:765] (0/8) Epoch 3, batch 1800, train_loss[loss=4.321, NarTop10Accuracy=0.4576, over 7219.00 frames. ], tot_loss[loss=4.386, NarTop10Accuracy=0.4348, over 5994.89 frames. ], batch size: 22, lr: 2.17e-02
2024-08-06 07:18:41,965 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-7000.pt
2024-08-06 07:19:01,958 INFO [trainer.py:765] (0/8) Epoch 3, batch 1900, train_loss[loss=4.678, NarTop10Accuracy=0.3898, over 5797.00 frames. ], tot_loss[loss=4.373, NarTop10Accuracy=0.4368, over 6034.36 frames. ], batch size: 49, lr: 2.16e-02
2024-08-06 07:19:27,621 INFO [trainer.py:765] (0/8) Epoch 3, batch 2000, train_loss[loss=4.507, NarTop10Accuracy=0.4059, over 7021.00 frames. ], tot_loss[loss=4.354, NarTop10Accuracy=0.4406, over 6014.79 frames. ], batch size: 49, lr: 2.15e-02
2024-08-06 07:19:53,070 INFO [trainer.py:765] (0/8) Epoch 3, batch 2100, train_loss[loss=4.051, NarTop10Accuracy=0.5002, over 3993.00 frames. ], tot_loss[loss=4.332, NarTop10Accuracy=0.4447, over 5994.85 frames. ], batch size: 4, lr: 2.14e-02
2024-08-06 07:20:18,553 INFO [trainer.py:765] (0/8) Epoch 3, batch 2200, train_loss[loss=4.518, NarTop10Accuracy=0.4067, over 7138.00 frames. ], tot_loss[loss=4.32, NarTop10Accuracy=0.4478, over 6027.39 frames. ], batch size: 30, lr: 2.13e-02
2024-08-06 07:20:44,051 INFO [trainer.py:765] (0/8) Epoch 3, batch 2300, train_loss[loss=3.863, NarTop10Accuracy=0.539, over 5800.00 frames. ], tot_loss[loss=4.315, NarTop10Accuracy=0.449, over 6060.69 frames. ], batch size: 9, lr: 2.12e-02
2024-08-06 07:21:08,677 INFO [trainer.py:765] (0/8) Epoch 3, batch 2400, train_loss[loss=4.235, NarTop10Accuracy=0.4759, over 6444.00 frames. ], tot_loss[loss=4.303, NarTop10Accuracy=0.4513, over 5891.07 frames. ], batch size: 50, lr: 2.11e-02
2024-08-06 07:21:32,172 INFO [trainer.py:765] (0/8) Epoch 3, batch 2500, train_loss[loss=4.417, NarTop10Accuracy=0.4266, over 5040.00 frames. ], tot_loss[loss=4.256, NarTop10Accuracy=0.4605, over 5543.46 frames. ], batch size: 6, lr: 2.10e-02
2024-08-06 07:21:53,322 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 07:21:53,325 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-3.pt
2024-08-06 07:23:00,978 INFO [trainer.py:765] (0/8) Epoch 4, batch 100, train_loss[loss=4.225, NarTop10Accuracy=0.4607, over 7485.00 frames. ], tot_loss[loss=4.209, NarTop10Accuracy=0.4717, over 2378.70 frames. ], batch size: 31, lr: 1.97e-02
2024-08-06 07:23:33,304 INFO [trainer.py:765] (0/8) Epoch 4, batch 200, train_loss[loss=4.236, NarTop10Accuracy=0.4627, over 6966.00 frames. ], tot_loss[loss=4.189, NarTop10Accuracy=0.4754, over 3867.02 frames. ], batch size: 17, lr: 1.96e-02
2024-08-06 07:23:51,466 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-8000.pt
2024-08-06 07:23:55,118 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 07:24:01,516 INFO [trainer.py:811] (0/8) Epoch 4, validation: loss=4.035, NarTop10Accuracy=0.5085, over 1907754.00 frames. 
2024-08-06 07:24:01,517 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 29794MB
2024-08-06 07:24:02,097 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 9.910e+01 1.530e+02 1.750e+02 2.064e+02 5.317e+02, threshold=3.500e+02, percent-clipped=3.3
2024-08-06 07:24:14,361 INFO [trainer.py:765] (0/8) Epoch 4, batch 300, train_loss[loss=4.035, NarTop10Accuracy=0.5032, over 7167.00 frames. ], tot_loss[loss=4.19, NarTop10Accuracy=0.4757, over 4676.63 frames. ], batch size: 22, lr: 1.95e-02
2024-08-06 07:24:53,596 INFO [trainer.py:765] (0/8) Epoch 4, batch 400, train_loss[loss=3.847, NarTop10Accuracy=0.5487, over 5217.00 frames. ], tot_loss[loss=4.184, NarTop10Accuracy=0.4764, over 5131.72 frames. ], batch size: 7, lr: 1.94e-02
2024-08-06 07:25:25,295 INFO [trainer.py:765] (0/8) Epoch 4, batch 500, train_loss[loss=4.022, NarTop10Accuracy=0.5057, over 6180.00 frames. ], tot_loss[loss=4.164, NarTop10Accuracy=0.48, over 5422.50 frames. ], batch size: 11, lr: 1.93e-02
2024-08-06 07:25:56,975 INFO [trainer.py:765] (0/8) Epoch 4, batch 600, train_loss[loss=3.766, NarTop10Accuracy=0.5386, over 5705.00 frames. ], tot_loss[loss=4.151, NarTop10Accuracy=0.482, over 5686.48 frames. ], batch size: 9, lr: 1.92e-02
2024-08-06 07:26:37,607 INFO [trainer.py:765] (0/8) Epoch 4, batch 700, train_loss[loss=4.5, NarTop10Accuracy=0.4179, over 5033.00 frames. ], tot_loss[loss=4.154, NarTop10Accuracy=0.4814, over 5746.74 frames. ], batch size: 6, lr: 1.92e-02
2024-08-06 07:27:07,433 INFO [trainer.py:765] (0/8) Epoch 4, batch 800, train_loss[loss=3.995, NarTop10Accuracy=0.5081, over 5076.00 frames. ], tot_loss[loss=4.15, NarTop10Accuracy=0.4826, over 5798.88 frames. ], batch size: 6, lr: 1.91e-02
2024-08-06 07:27:42,042 INFO [trainer.py:765] (0/8) Epoch 4, batch 900, train_loss[loss=4.08, NarTop10Accuracy=0.4962, over 6214.00 frames. ], tot_loss[loss=4.12, NarTop10Accuracy=0.489, over 5824.43 frames. ], batch size: 13, lr: 1.90e-02
2024-08-06 07:28:20,670 INFO [trainer.py:765] (0/8) Epoch 4, batch 1000, train_loss[loss=3.86, NarTop10Accuracy=0.5365, over 6180.00 frames. ], tot_loss[loss=4.116, NarTop10Accuracy=0.49, over 5923.54 frames. ], batch size: 13, lr: 1.89e-02
2024-08-06 07:28:54,071 INFO [trainer.py:765] (0/8) Epoch 4, batch 1100, train_loss[loss=3.716, NarTop10Accuracy=0.5732, over 6826.00 frames. ], tot_loss[loss=4.112, NarTop10Accuracy=0.4907, over 5955.32 frames. ], batch size: 17, lr: 1.88e-02
2024-08-06 07:29:29,599 INFO [trainer.py:765] (0/8) Epoch 4, batch 1200, train_loss[loss=4.136, NarTop10Accuracy=0.489, over 7319.00 frames. ], tot_loss[loss=4.109, NarTop10Accuracy=0.491, over 5931.23 frames. ], batch size: 31, lr: 1.87e-02
2024-08-06 07:29:45,948 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-9000.pt
2024-08-06 07:30:04,991 INFO [trainer.py:765] (0/8) Epoch 4, batch 1300, train_loss[loss=3.691, NarTop10Accuracy=0.5897, over 5150.00 frames. ], tot_loss[loss=4.075, NarTop10Accuracy=0.498, over 6010.56 frames. ], batch size: 6, lr: 1.87e-02
2024-08-06 07:30:43,380 INFO [trainer.py:765] (0/8) Epoch 4, batch 1400, train_loss[loss=3.82, NarTop10Accuracy=0.5404, over 6030.00 frames. ], tot_loss[loss=4.069, NarTop10Accuracy=0.4988, over 6018.05 frames. ], batch size: 11, lr: 1.86e-02
2024-08-06 07:31:11,831 INFO [trainer.py:765] (0/8) Epoch 4, batch 1500, train_loss[loss=4.153, NarTop10Accuracy=0.4855, over 5703.00 frames. ], tot_loss[loss=4.06, NarTop10Accuracy=0.5003, over 5955.79 frames. ], batch size: 50, lr: 1.85e-02
2024-08-06 07:31:39,961 INFO [trainer.py:765] (0/8) Epoch 4, batch 1600, train_loss[loss=4.31, NarTop10Accuracy=0.4543, over 7170.00 frames. ], tot_loss[loss=4.07, NarTop10Accuracy=0.4984, over 5953.83 frames. ], batch size: 22, lr: 1.84e-02
2024-08-06 07:32:06,855 INFO [trainer.py:765] (0/8) Epoch 4, batch 1700, train_loss[loss=4.332, NarTop10Accuracy=0.4537, over 6176.00 frames. ], tot_loss[loss=4.048, NarTop10Accuracy=0.5027, over 5940.33 frames. ], batch size: 13, lr: 1.84e-02
2024-08-06 07:32:33,483 INFO [trainer.py:765] (0/8) Epoch 4, batch 1800, train_loss[loss=4.251, NarTop10Accuracy=0.4669, over 7138.00 frames. ], tot_loss[loss=4.043, NarTop10Accuracy=0.5035, over 6000.80 frames. ], batch size: 22, lr: 1.83e-02
2024-08-06 07:33:00,194 INFO [trainer.py:765] (0/8) Epoch 4, batch 1900, train_loss[loss=4.237, NarTop10Accuracy=0.4618, over 5634.00 frames. ], tot_loss[loss=4.054, NarTop10Accuracy=0.5013, over 6026.72 frames. ], batch size: 49, lr: 1.82e-02
2024-08-06 07:33:25,990 INFO [trainer.py:765] (0/8) Epoch 4, batch 2000, train_loss[loss=4.368, NarTop10Accuracy=0.4382, over 6237.00 frames. ], tot_loss[loss=4.032, NarTop10Accuracy=0.5057, over 6011.95 frames. ], batch size: 51, lr: 1.81e-02
2024-08-06 07:33:51,512 INFO [trainer.py:765] (0/8) Epoch 4, batch 2100, train_loss[loss=4.003, NarTop10Accuracy=0.5062, over 4663.00 frames. ], tot_loss[loss=4.022, NarTop10Accuracy=0.5082, over 5984.70 frames. ], batch size: 5, lr: 1.81e-02
2024-08-06 07:34:16,906 INFO [trainer.py:765] (0/8) Epoch 4, batch 2200, train_loss[loss=4.317, NarTop10Accuracy=0.453, over 7218.00 frames. ], tot_loss[loss=4.032, NarTop10Accuracy=0.5061, over 6035.10 frames. ], batch size: 30, lr: 1.80e-02
2024-08-06 07:34:31,430 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-10000.pt
2024-08-06 07:34:34,952 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 07:34:41,462 INFO [trainer.py:811] (0/8) Epoch 4, validation: loss=3.858, NarTop10Accuracy=0.5445, over 1907754.00 frames. 
2024-08-06 07:34:41,463 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 29794MB
2024-08-06 07:34:41,980 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.230e+02 1.721e+02 1.919e+02 2.225e+02 9.682e+02, threshold=3.839e+02, percent-clipped=2.3
2024-08-06 07:34:52,442 INFO [trainer.py:765] (0/8) Epoch 4, batch 2300, train_loss[loss=3.907, NarTop10Accuracy=0.5275, over 5759.00 frames. ], tot_loss[loss=4.036, NarTop10Accuracy=0.5062, over 6060.62 frames. ], batch size: 9, lr: 1.79e-02
2024-08-06 07:35:17,166 INFO [trainer.py:765] (0/8) Epoch 4, batch 2400, train_loss[loss=3.878, NarTop10Accuracy=0.5344, over 5245.00 frames. ], tot_loss[loss=4.025, NarTop10Accuracy=0.5084, over 5877.31 frames. ], batch size: 7, lr: 1.78e-02
2024-08-06 07:35:40,623 INFO [trainer.py:765] (0/8) Epoch 4, batch 2500, train_loss[loss=4.214, NarTop10Accuracy=0.475, over 5089.00 frames. ], tot_loss[loss=4.008, NarTop10Accuracy=0.5124, over 5534.14 frames. ], batch size: 6, lr: 1.78e-02
2024-08-06 07:36:01,729 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 07:36:01,732 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-4.pt
2024-08-06 07:37:02,524 INFO [trainer.py:765] (0/8) Epoch 5, batch 100, train_loss[loss=4.023, NarTop10Accuracy=0.5128, over 7185.00 frames. ], tot_loss[loss=3.96, NarTop10Accuracy=0.5236, over 2380.81 frames. ], batch size: 30, lr: 1.66e-02
2024-08-06 07:37:39,815 INFO [trainer.py:765] (0/8) Epoch 5, batch 200, train_loss[loss=4.217, NarTop10Accuracy=0.4781, over 6633.00 frames. ], tot_loss[loss=3.951, NarTop10Accuracy=0.524, over 3870.44 frames. ], batch size: 17, lr: 1.65e-02
2024-08-06 07:38:13,471 INFO [trainer.py:765] (0/8) Epoch 5, batch 300, train_loss[loss=4.154, NarTop10Accuracy=0.4822, over 7207.00 frames. ], tot_loss[loss=3.926, NarTop10Accuracy=0.5293, over 4686.12 frames. ], batch size: 22, lr: 1.65e-02
2024-08-06 07:38:42,429 INFO [trainer.py:765] (0/8) Epoch 5, batch 400, train_loss[loss=3.81, NarTop10Accuracy=0.549, over 5116.00 frames. ], tot_loss[loss=3.924, NarTop10Accuracy=0.5296, over 5130.33 frames. ], batch size: 7, lr: 1.64e-02
2024-08-06 07:39:17,020 INFO [trainer.py:765] (0/8) Epoch 5, batch 500, train_loss[loss=3.741, NarTop10Accuracy=0.5683, over 6048.00 frames. ], tot_loss[loss=3.93, NarTop10Accuracy=0.5286, over 5414.03 frames. ], batch size: 11, lr: 1.63e-02
2024-08-06 07:39:51,944 INFO [trainer.py:765] (0/8) Epoch 5, batch 600, train_loss[loss=3.898, NarTop10Accuracy=0.5393, over 5757.00 frames. ], tot_loss[loss=3.916, NarTop10Accuracy=0.5314, over 5674.01 frames. ], batch size: 9, lr: 1.63e-02
2024-08-06 07:40:18,576 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-11000.pt
2024-08-06 07:40:28,626 INFO [trainer.py:765] (0/8) Epoch 5, batch 700, train_loss[loss=3.796, NarTop10Accuracy=0.5604, over 5058.00 frames. ], tot_loss[loss=3.911, NarTop10Accuracy=0.5322, over 5738.73 frames. ], batch size: 6, lr: 1.62e-02
2024-08-06 07:41:02,367 INFO [trainer.py:765] (0/8) Epoch 5, batch 800, train_loss[loss=4.414, NarTop10Accuracy=0.43, over 4948.00 frames. ], tot_loss[loss=3.912, NarTop10Accuracy=0.5317, over 5800.87 frames. ], batch size: 6, lr: 1.62e-02
2024-08-06 07:41:37,938 INFO [trainer.py:765] (0/8) Epoch 5, batch 900, train_loss[loss=4.103, NarTop10Accuracy=0.4947, over 6739.00 frames. ], tot_loss[loss=3.9, NarTop10Accuracy=0.5343, over 5821.68 frames. ], batch size: 14, lr: 1.61e-02
2024-08-06 07:42:13,846 INFO [trainer.py:765] (0/8) Epoch 5, batch 1000, train_loss[loss=3.994, NarTop10Accuracy=0.5065, over 6303.00 frames. ], tot_loss[loss=3.892, NarTop10Accuracy=0.5355, over 5936.81 frames. ], batch size: 13, lr: 1.60e-02
2024-08-06 07:42:46,468 INFO [trainer.py:765] (0/8) Epoch 5, batch 1100, train_loss[loss=3.841, NarTop10Accuracy=0.5297, over 6847.00 frames. ], tot_loss[loss=3.898, NarTop10Accuracy=0.5343, over 5953.53 frames. ], batch size: 17, lr: 1.60e-02
2024-08-06 07:43:25,226 INFO [trainer.py:765] (0/8) Epoch 5, batch 1200, train_loss[loss=4.162, NarTop10Accuracy=0.4769, over 7097.00 frames. ], tot_loss[loss=3.914, NarTop10Accuracy=0.5312, over 5946.32 frames. ], batch size: 30, lr: 1.59e-02
2024-08-06 07:44:00,557 INFO [trainer.py:765] (0/8) Epoch 5, batch 1300, train_loss[loss=3.749, NarTop10Accuracy=0.5476, over 5044.00 frames. ], tot_loss[loss=3.904, NarTop10Accuracy=0.5333, over 6014.80 frames. ], batch size: 6, lr: 1.59e-02
2024-08-06 07:44:30,238 INFO [trainer.py:765] (0/8) Epoch 5, batch 1400, train_loss[loss=4.027, NarTop10Accuracy=0.5139, over 6042.00 frames. ], tot_loss[loss=3.899, NarTop10Accuracy=0.5344, over 6043.64 frames. ], batch size: 11, lr: 1.58e-02
2024-08-06 07:45:02,845 INFO [trainer.py:765] (0/8) Epoch 5, batch 1500, train_loss[loss=3.874, NarTop10Accuracy=0.5432, over 6027.00 frames. ], tot_loss[loss=3.9, NarTop10Accuracy=0.5341, over 5977.47 frames. ], batch size: 49, lr: 1.57e-02
2024-08-06 07:45:31,008 INFO [trainer.py:765] (0/8) Epoch 5, batch 1600, train_loss[loss=4.071, NarTop10Accuracy=0.5072, over 7257.00 frames. ], tot_loss[loss=3.906, NarTop10Accuracy=0.5335, over 5965.25 frames. ], batch size: 22, lr: 1.57e-02
2024-08-06 07:45:51,057 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-12000.pt
2024-08-06 07:45:54,929 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 07:46:01,621 INFO [trainer.py:811] (0/8) Epoch 5, validation: loss=3.749, NarTop10Accuracy=0.5672, over 1907754.00 frames. 
2024-08-06 07:46:01,622 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 29794MB
2024-08-06 07:46:02,122 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.160e+02 1.669e+02 1.884e+02 2.190e+02 6.243e+02, threshold=3.768e+02, percent-clipped=1.8
2024-08-06 07:46:08,362 INFO [trainer.py:765] (0/8) Epoch 5, batch 1700, train_loss[loss=3.812, NarTop10Accuracy=0.5525, over 6171.00 frames. ], tot_loss[loss=3.91, NarTop10Accuracy=0.5327, over 5935.53 frames. ], batch size: 13, lr: 1.56e-02
2024-08-06 07:46:34,966 INFO [trainer.py:765] (0/8) Epoch 5, batch 1800, train_loss[loss=3.871, NarTop10Accuracy=0.5512, over 6856.00 frames. ], tot_loss[loss=3.892, NarTop10Accuracy=0.5361, over 6007.50 frames. ], batch size: 21, lr: 1.56e-02
2024-08-06 07:47:01,489 INFO [trainer.py:765] (0/8) Epoch 5, batch 1900, train_loss[loss=3.793, NarTop10Accuracy=0.5572, over 6095.00 frames. ], tot_loss[loss=3.903, NarTop10Accuracy=0.534, over 6052.75 frames. ], batch size: 49, lr: 1.55e-02
2024-08-06 07:47:27,146 INFO [trainer.py:765] (0/8) Epoch 5, batch 2000, train_loss[loss=3.932, NarTop10Accuracy=0.5216, over 5586.00 frames. ], tot_loss[loss=3.897, NarTop10Accuracy=0.5349, over 6022.66 frames. ], batch size: 49, lr: 1.55e-02
2024-08-06 07:47:52,618 INFO [trainer.py:765] (0/8) Epoch 5, batch 2100, train_loss[loss=3.834, NarTop10Accuracy=0.5505, over 4008.00 frames. ], tot_loss[loss=3.897, NarTop10Accuracy=0.5349, over 6000.46 frames. ], batch size: 4, lr: 1.54e-02
2024-08-06 07:48:17,992 INFO [trainer.py:765] (0/8) Epoch 5, batch 2200, train_loss[loss=3.933, NarTop10Accuracy=0.5267, over 7032.00 frames. ], tot_loss[loss=3.888, NarTop10Accuracy=0.537, over 6049.62 frames. ], batch size: 30, lr: 1.54e-02
2024-08-06 07:48:43,421 INFO [trainer.py:765] (0/8) Epoch 5, batch 2300, train_loss[loss=3.987, NarTop10Accuracy=0.5127, over 5813.00 frames. ], tot_loss[loss=3.892, NarTop10Accuracy=0.5363, over 6070.35 frames. ], batch size: 9, lr: 1.53e-02
2024-08-06 07:49:08,169 INFO [trainer.py:765] (0/8) Epoch 5, batch 2400, train_loss[loss=3.91, NarTop10Accuracy=0.5391, over 6175.00 frames. ], tot_loss[loss=3.887, NarTop10Accuracy=0.5375, over 5894.74 frames. ], batch size: 49, lr: 1.53e-02
2024-08-06 07:49:31,644 INFO [trainer.py:765] (0/8) Epoch 5, batch 2500, train_loss[loss=3.758, NarTop10Accuracy=0.5574, over 5077.00 frames. ], tot_loss[loss=3.851, NarTop10Accuracy=0.5444, over 5534.22 frames. ], batch size: 6, lr: 1.52e-02
2024-08-06 07:49:53,739 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 07:49:53,743 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-5.pt
2024-08-06 07:50:52,546 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-13000.pt
2024-08-06 07:50:58,969 INFO [trainer.py:765] (0/8) Epoch 6, batch 100, train_loss[loss=3.766, NarTop10Accuracy=0.5628, over 7183.00 frames. ], tot_loss[loss=3.796, NarTop10Accuracy=0.5584, over 2387.39 frames. ], batch size: 30, lr: 1.42e-02
2024-08-06 07:51:31,789 INFO [trainer.py:765] (0/8) Epoch 6, batch 200, train_loss[loss=3.693, NarTop10Accuracy=0.5765, over 6813.00 frames. ], tot_loss[loss=3.798, NarTop10Accuracy=0.5565, over 3882.51 frames. ], batch size: 17, lr: 1.42e-02
2024-08-06 07:52:04,696 INFO [trainer.py:765] (0/8) Epoch 6, batch 300, train_loss[loss=3.572, NarTop10Accuracy=0.601, over 7170.00 frames. ], tot_loss[loss=3.789, NarTop10Accuracy=0.5577, over 4672.39 frames. ], batch size: 22, lr: 1.41e-02
2024-08-06 07:52:36,200 INFO [trainer.py:765] (0/8) Epoch 6, batch 400, train_loss[loss=3.608, NarTop10Accuracy=0.5833, over 5201.00 frames. ], tot_loss[loss=3.788, NarTop10Accuracy=0.5575, over 5135.26 frames. ], batch size: 7, lr: 1.41e-02
2024-08-06 07:53:06,103 INFO [trainer.py:765] (0/8) Epoch 6, batch 500, train_loss[loss=3.798, NarTop10Accuracy=0.5516, over 6246.00 frames. ], tot_loss[loss=3.777, NarTop10Accuracy=0.5601, over 5398.67 frames. ], batch size: 11, lr: 1.40e-02
2024-08-06 07:53:43,286 INFO [trainer.py:765] (0/8) Epoch 6, batch 600, train_loss[loss=3.62, NarTop10Accuracy=0.5927, over 5724.00 frames. ], tot_loss[loss=3.78, NarTop10Accuracy=0.5599, over 5663.40 frames. ], batch size: 9, lr: 1.40e-02
2024-08-06 07:54:15,439 INFO [trainer.py:765] (0/8) Epoch 6, batch 700, train_loss[loss=4.058, NarTop10Accuracy=0.5069, over 4982.00 frames. ], tot_loss[loss=3.783, NarTop10Accuracy=0.5591, over 5743.36 frames. ], batch size: 6, lr: 1.39e-02
2024-08-06 07:54:49,526 INFO [trainer.py:765] (0/8) Epoch 6, batch 800, train_loss[loss=3.829, NarTop10Accuracy=0.5426, over 5141.00 frames. ], tot_loss[loss=3.793, NarTop10Accuracy=0.5565, over 5804.15 frames. ], batch size: 6, lr: 1.39e-02
2024-08-06 07:55:21,984 INFO [trainer.py:765] (0/8) Epoch 6, batch 900, train_loss[loss=3.552, NarTop10Accuracy=0.6199, over 6370.00 frames. ], tot_loss[loss=3.782, NarTop10Accuracy=0.5581, over 5822.09 frames. ], batch size: 13, lr: 1.38e-02
2024-08-06 07:56:00,804 INFO [trainer.py:765] (0/8) Epoch 6, batch 1000, train_loss[loss=3.55, NarTop10Accuracy=0.6039, over 6241.00 frames. ], tot_loss[loss=3.803, NarTop10Accuracy=0.5542, over 5924.05 frames. ], batch size: 13, lr: 1.38e-02
2024-08-06 07:56:34,171 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-14000.pt
2024-08-06 07:56:38,061 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 07:56:44,742 INFO [trainer.py:811] (0/8) Epoch 6, validation: loss=3.634, NarTop10Accuracy=0.5919, over 1907754.00 frames. 
2024-08-06 07:56:44,743 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 29794MB
2024-08-06 07:56:45,277 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.300e+02 1.714e+02 1.918e+02 2.211e+02 6.360e+02, threshold=3.836e+02, percent-clipped=1.6
2024-08-06 07:56:46,639 INFO [trainer.py:765] (0/8) Epoch 6, batch 1100, train_loss[loss=3.639, NarTop10Accuracy=0.5988, over 7013.00 frames. ], tot_loss[loss=3.803, NarTop10Accuracy=0.5545, over 5947.35 frames. ], batch size: 17, lr: 1.37e-02
2024-08-06 07:57:24,888 INFO [trainer.py:765] (0/8) Epoch 6, batch 1200, train_loss[loss=4.046, NarTop10Accuracy=0.5026, over 7308.00 frames. ], tot_loss[loss=3.801, NarTop10Accuracy=0.555, over 5935.69 frames. ], batch size: 30, lr: 1.37e-02
2024-08-06 07:57:56,612 INFO [trainer.py:765] (0/8) Epoch 6, batch 1300, train_loss[loss=3.539, NarTop10Accuracy=0.599, over 4821.00 frames. ], tot_loss[loss=3.796, NarTop10Accuracy=0.5559, over 5989.63 frames. ], batch size: 6, lr: 1.37e-02
2024-08-06 07:58:30,736 INFO [trainer.py:765] (0/8) Epoch 6, batch 1400, train_loss[loss=4.025, NarTop10Accuracy=0.4994, over 6203.00 frames. ], tot_loss[loss=3.799, NarTop10Accuracy=0.5552, over 6022.36 frames. ], batch size: 11, lr: 1.36e-02
2024-08-06 07:59:00,998 INFO [trainer.py:765] (0/8) Epoch 6, batch 1500, train_loss[loss=4.182, NarTop10Accuracy=0.4805, over 6020.00 frames. ], tot_loss[loss=3.808, NarTop10Accuracy=0.5535, over 5957.77 frames. ], batch size: 48, lr: 1.36e-02
2024-08-06 07:59:28,933 INFO [trainer.py:765] (0/8) Epoch 6, batch 1600, train_loss[loss=3.648, NarTop10Accuracy=0.5844, over 7129.00 frames. ], tot_loss[loss=3.795, NarTop10Accuracy=0.556, over 5949.86 frames. ], batch size: 22, lr: 1.35e-02
2024-08-06 07:59:55,617 INFO [trainer.py:765] (0/8) Epoch 6, batch 1700, train_loss[loss=3.841, NarTop10Accuracy=0.5529, over 6668.00 frames. ], tot_loss[loss=3.787, NarTop10Accuracy=0.558, over 5938.53 frames. ], batch size: 14, lr: 1.35e-02
2024-08-06 08:00:22,187 INFO [trainer.py:765] (0/8) Epoch 6, batch 1800, train_loss[loss=3.951, NarTop10Accuracy=0.5256, over 7145.00 frames. ], tot_loss[loss=3.781, NarTop10Accuracy=0.5594, over 6004.87 frames. ], batch size: 22, lr: 1.35e-02
2024-08-06 08:00:48,794 INFO [trainer.py:765] (0/8) Epoch 6, batch 1900, train_loss[loss=4.029, NarTop10Accuracy=0.505, over 6344.00 frames. ], tot_loss[loss=3.82, NarTop10Accuracy=0.5518, over 6054.42 frames. ], batch size: 49, lr: 1.34e-02
2024-08-06 08:01:14,461 INFO [trainer.py:765] (0/8) Epoch 6, batch 2000, train_loss[loss=3.903, NarTop10Accuracy=0.5459, over 6469.00 frames. ], tot_loss[loss=3.801, NarTop10Accuracy=0.5551, over 6020.45 frames. ], batch size: 51, lr: 1.34e-02
2024-08-06 08:01:38,209 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-15000.pt
2024-08-06 08:01:43,133 INFO [trainer.py:765] (0/8) Epoch 6, batch 2100, train_loss[loss=3.615, NarTop10Accuracy=0.5819, over 4832.00 frames. ], tot_loss[loss=3.797, NarTop10Accuracy=0.5559, over 5996.85 frames. ], batch size: 5, lr: 1.33e-02
2024-08-06 08:02:08,518 INFO [trainer.py:765] (0/8) Epoch 6, batch 2200, train_loss[loss=3.706, NarTop10Accuracy=0.5771, over 7176.00 frames. ], tot_loss[loss=3.801, NarTop10Accuracy=0.555, over 6039.11 frames. ], batch size: 30, lr: 1.33e-02
2024-08-06 08:02:33,916 INFO [trainer.py:765] (0/8) Epoch 6, batch 2300, train_loss[loss=3.911, NarTop10Accuracy=0.538, over 5772.00 frames. ], tot_loss[loss=3.8, NarTop10Accuracy=0.5552, over 6081.85 frames. ], batch size: 9, lr: 1.33e-02
2024-08-06 08:02:58,616 INFO [trainer.py:765] (0/8) Epoch 6, batch 2400, train_loss[loss=3.485, NarTop10Accuracy=0.5952, over 5157.00 frames. ], tot_loss[loss=3.798, NarTop10Accuracy=0.5556, over 5906.70 frames. ], batch size: 7, lr: 1.32e-02
2024-08-06 08:03:21,939 INFO [trainer.py:765] (0/8) Epoch 6, batch 2500, train_loss[loss=3.944, NarTop10Accuracy=0.5241, over 5126.00 frames. ], tot_loss[loss=3.782, NarTop10Accuracy=0.5587, over 5553.33 frames. ], batch size: 6, lr: 1.32e-02
2024-08-06 08:03:43,317 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 08:03:43,319 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-6.pt
2024-08-06 08:04:42,817 INFO [trainer.py:765] (0/8) Epoch 7, batch 100, train_loss[loss=3.71, NarTop10Accuracy=0.5733, over 7101.00 frames. ], tot_loss[loss=3.68, NarTop10Accuracy=0.5808, over 2384.29 frames. ], batch size: 31, lr: 1.23e-02
2024-08-06 08:05:18,347 INFO [trainer.py:765] (0/8) Epoch 7, batch 200, train_loss[loss=4.046, NarTop10Accuracy=0.5141, over 6905.00 frames. ], tot_loss[loss=3.693, NarTop10Accuracy=0.5778, over 3882.60 frames. ], batch size: 17, lr: 1.23e-02
2024-08-06 08:05:46,773 INFO [trainer.py:765] (0/8) Epoch 7, batch 300, train_loss[loss=3.413, NarTop10Accuracy=0.637, over 7017.00 frames. ], tot_loss[loss=3.708, NarTop10Accuracy=0.5749, over 4679.21 frames. ], batch size: 22, lr: 1.23e-02
2024-08-06 08:06:22,091 INFO [trainer.py:765] (0/8) Epoch 7, batch 400, train_loss[loss=3.999, NarTop10Accuracy=0.5076, over 5019.00 frames. ], tot_loss[loss=3.714, NarTop10Accuracy=0.5732, over 5132.09 frames. ], batch size: 7, lr: 1.22e-02
2024-08-06 08:06:52,315 INFO [trainer.py:765] (0/8) Epoch 7, batch 500, train_loss[loss=3.674, NarTop10Accuracy=0.5838, over 6192.00 frames. ], tot_loss[loss=3.712, NarTop10Accuracy=0.5736, over 5418.25 frames. ], batch size: 11, lr: 1.22e-02
2024-08-06 08:06:56,085 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-16000.pt
2024-08-06 08:06:59,575 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 08:07:06,251 INFO [trainer.py:811] (0/8) Epoch 7, validation: loss=3.56, NarTop10Accuracy=0.6069, over 1907754.00 frames. 
2024-08-06 08:07:06,252 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 29794MB
2024-08-06 08:07:06,837 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.316e+02 1.760e+02 1.958e+02 2.227e+02 5.399e+02, threshold=3.916e+02, percent-clipped=0.8
2024-08-06 08:07:33,151 INFO [trainer.py:765] (0/8) Epoch 7, batch 600, train_loss[loss=3.501, NarTop10Accuracy=0.6139, over 5778.00 frames. ], tot_loss[loss=3.714, NarTop10Accuracy=0.5733, over 5680.18 frames. ], batch size: 9, lr: 1.22e-02
2024-08-06 08:08:11,333 INFO [trainer.py:765] (0/8) Epoch 7, batch 700, train_loss[loss=3.457, NarTop10Accuracy=0.6335, over 5219.00 frames. ], tot_loss[loss=3.719, NarTop10Accuracy=0.5722, over 5742.75 frames. ], batch size: 6, lr: 1.21e-02
2024-08-06 08:08:45,557 INFO [trainer.py:765] (0/8) Epoch 7, batch 800, train_loss[loss=3.584, NarTop10Accuracy=0.6009, over 5006.00 frames. ], tot_loss[loss=3.707, NarTop10Accuracy=0.5745, over 5817.44 frames. ], batch size: 6, lr: 1.21e-02
2024-08-06 08:09:17,739 INFO [trainer.py:765] (0/8) Epoch 7, batch 900, train_loss[loss=3.603, NarTop10Accuracy=0.5988, over 6320.00 frames. ], tot_loss[loss=3.713, NarTop10Accuracy=0.5729, over 5828.88 frames. ], batch size: 13, lr: 1.21e-02
2024-08-06 08:09:54,191 INFO [trainer.py:765] (0/8) Epoch 7, batch 1000, train_loss[loss=3.798, NarTop10Accuracy=0.553, over 6254.00 frames. ], tot_loss[loss=3.715, NarTop10Accuracy=0.573, over 5931.26 frames. ], batch size: 13, lr: 1.20e-02
2024-08-06 08:10:29,570 INFO [trainer.py:765] (0/8) Epoch 7, batch 1100, train_loss[loss=3.548, NarTop10Accuracy=0.6054, over 6857.00 frames. ], tot_loss[loss=3.715, NarTop10Accuracy=0.5727, over 5942.19 frames. ], batch size: 17, lr: 1.20e-02
2024-08-06 08:11:02,491 INFO [trainer.py:765] (0/8) Epoch 7, batch 1200, train_loss[loss=3.97, NarTop10Accuracy=0.5151, over 7255.00 frames. ], tot_loss[loss=3.712, NarTop10Accuracy=0.5729, over 5938.27 frames. ], batch size: 32, lr: 1.20e-02
2024-08-06 08:11:33,447 INFO [trainer.py:765] (0/8) Epoch 7, batch 1300, train_loss[loss=3.198, NarTop10Accuracy=0.6754, over 4914.00 frames. ], tot_loss[loss=3.715, NarTop10Accuracy=0.5722, over 6008.26 frames. ], batch size: 6, lr: 1.19e-02
2024-08-06 08:12:10,912 INFO [trainer.py:765] (0/8) Epoch 7, batch 1400, train_loss[loss=3.559, NarTop10Accuracy=0.6001, over 6241.00 frames. ], tot_loss[loss=3.717, NarTop10Accuracy=0.572, over 6036.16 frames. ], batch size: 11, lr: 1.19e-02
2024-08-06 08:12:42,109 INFO [trainer.py:765] (0/8) Epoch 7, batch 1500, train_loss[loss=3.78, NarTop10Accuracy=0.5628, over 5622.00 frames. ], tot_loss[loss=3.718, NarTop10Accuracy=0.5721, over 5979.06 frames. ], batch size: 48, lr: 1.19e-02
2024-08-06 08:12:45,668 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-17000.pt
2024-08-06 08:13:13,237 INFO [trainer.py:765] (0/8) Epoch 7, batch 1600, train_loss[loss=3.596, NarTop10Accuracy=0.5943, over 6905.00 frames. ], tot_loss[loss=3.715, NarTop10Accuracy=0.5726, over 5954.22 frames. ], batch size: 22, lr: 1.18e-02
2024-08-06 08:13:40,016 INFO [trainer.py:765] (0/8) Epoch 7, batch 1700, train_loss[loss=3.771, NarTop10Accuracy=0.5594, over 6304.00 frames. ], tot_loss[loss=3.722, NarTop10Accuracy=0.5711, over 5943.68 frames. ], batch size: 13, lr: 1.18e-02
2024-08-06 08:14:06,584 INFO [trainer.py:765] (0/8) Epoch 7, batch 1800, train_loss[loss=3.773, NarTop10Accuracy=0.5594, over 7108.00 frames. ], tot_loss[loss=3.726, NarTop10Accuracy=0.5702, over 6021.24 frames. ], batch size: 22, lr: 1.18e-02
2024-08-06 08:14:33,223 INFO [trainer.py:765] (0/8) Epoch 7, batch 1900, train_loss[loss=4.128, NarTop10Accuracy=0.4834, over 6151.00 frames. ], tot_loss[loss=3.732, NarTop10Accuracy=0.5693, over 6051.65 frames. ], batch size: 49, lr: 1.17e-02
2024-08-06 08:14:58,994 INFO [trainer.py:765] (0/8) Epoch 7, batch 2000, train_loss[loss=3.703, NarTop10Accuracy=0.5799, over 6365.00 frames. ], tot_loss[loss=3.723, NarTop10Accuracy=0.5713, over 6027.13 frames. ], batch size: 49, lr: 1.17e-02
2024-08-06 08:15:24,423 INFO [trainer.py:765] (0/8) Epoch 7, batch 2100, train_loss[loss=3.872, NarTop10Accuracy=0.5542, over 3951.00 frames. ], tot_loss[loss=3.726, NarTop10Accuracy=0.5706, over 6003.19 frames. ], batch size: 4, lr: 1.17e-02
2024-08-06 08:15:49,960 INFO [trainer.py:765] (0/8) Epoch 7, batch 2200, train_loss[loss=3.922, NarTop10Accuracy=0.5381, over 7523.00 frames. ], tot_loss[loss=3.746, NarTop10Accuracy=0.5667, over 6046.09 frames. ], batch size: 31, lr: 1.17e-02
2024-08-06 08:16:15,490 INFO [trainer.py:765] (0/8) Epoch 7, batch 2300, train_loss[loss=4.085, NarTop10Accuracy=0.5026, over 5875.00 frames. ], tot_loss[loss=3.742, NarTop10Accuracy=0.5673, over 6073.98 frames. ], batch size: 9, lr: 1.16e-02
2024-08-06 08:16:40,319 INFO [trainer.py:765] (0/8) Epoch 7, batch 2400, train_loss[loss=4.011, NarTop10Accuracy=0.521, over 6125.00 frames. ], tot_loss[loss=3.739, NarTop10Accuracy=0.5678, over 5886.93 frames. ], batch size: 49, lr: 1.16e-02
2024-08-06 08:17:03,739 INFO [trainer.py:765] (0/8) Epoch 7, batch 2500, train_loss[loss=3.48, NarTop10Accuracy=0.6075, over 5034.00 frames. ], tot_loss[loss=3.717, NarTop10Accuracy=0.5717, over 5548.56 frames. ], batch size: 6, lr: 1.16e-02
2024-08-06 08:17:06,843 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-18000.pt
2024-08-06 08:17:10,507 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 08:17:17,432 INFO [trainer.py:811] (0/8) Epoch 7, validation: loss=3.591, NarTop10Accuracy=0.6002, over 1907754.00 frames. 
2024-08-06 08:17:17,433 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 29794MB
2024-08-06 08:17:17,901 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.356e+02 1.794e+02 1.981e+02 2.246e+02 4.644e+02, threshold=3.962e+02, percent-clipped=1.0
2024-08-06 08:17:35,456 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 08:17:35,459 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-7.pt
2024-08-06 08:18:36,194 INFO [trainer.py:765] (0/8) Epoch 8, batch 100, train_loss[loss=3.696, NarTop10Accuracy=0.5774, over 7227.00 frames. ], tot_loss[loss=3.662, NarTop10Accuracy=0.5843, over 2373.95 frames. ], batch size: 31, lr: 1.09e-02
2024-08-06 08:19:15,020 INFO [trainer.py:765] (0/8) Epoch 8, batch 200, train_loss[loss=3.474, NarTop10Accuracy=0.616, over 6771.00 frames. ], tot_loss[loss=3.67, NarTop10Accuracy=0.5828, over 3857.37 frames. ], batch size: 17, lr: 1.09e-02
2024-08-06 08:19:43,561 INFO [trainer.py:765] (0/8) Epoch 8, batch 300, train_loss[loss=3.674, NarTop10Accuracy=0.5813, over 7306.00 frames. ], tot_loss[loss=3.659, NarTop10Accuracy=0.5855, over 4678.62 frames. ], batch size: 22, lr: 1.08e-02
2024-08-06 08:20:16,269 INFO [trainer.py:765] (0/8) Epoch 8, batch 400, train_loss[loss=3.518, NarTop10Accuracy=0.6075, over 5025.00 frames. ], tot_loss[loss=3.659, NarTop10Accuracy=0.5852, over 5137.89 frames. ], batch size: 7, lr: 1.08e-02
2024-08-06 08:20:48,421 INFO [trainer.py:765] (0/8) Epoch 8, batch 500, train_loss[loss=3.312, NarTop10Accuracy=0.6541, over 6240.00 frames. ], tot_loss[loss=3.653, NarTop10Accuracy=0.5862, over 5401.92 frames. ], batch size: 11, lr: 1.08e-02
2024-08-06 08:21:23,737 INFO [trainer.py:765] (0/8) Epoch 8, batch 600, train_loss[loss=3.518, NarTop10Accuracy=0.6197, over 5738.00 frames. ], tot_loss[loss=3.663, NarTop10Accuracy=0.5839, over 5667.19 frames. ], batch size: 9, lr: 1.07e-02
2024-08-06 08:21:57,607 INFO [trainer.py:765] (0/8) Epoch 8, batch 700, train_loss[loss=3.987, NarTop10Accuracy=0.5082, over 4878.00 frames. ], tot_loss[loss=3.673, NarTop10Accuracy=0.5817, over 5750.33 frames. ], batch size: 6, lr: 1.07e-02
2024-08-06 08:22:27,342 INFO [trainer.py:765] (0/8) Epoch 8, batch 800, train_loss[loss=3.516, NarTop10Accuracy=0.6262, over 4999.00 frames. ], tot_loss[loss=3.672, NarTop10Accuracy=0.5819, over 5795.84 frames. ], batch size: 6, lr: 1.07e-02
2024-08-06 08:23:06,892 INFO [trainer.py:765] (0/8) Epoch 8, batch 900, train_loss[loss=3.176, NarTop10Accuracy=0.6618, over 6355.00 frames. ], tot_loss[loss=3.664, NarTop10Accuracy=0.5835, over 5817.36 frames. ], batch size: 13, lr: 1.07e-02
2024-08-06 08:23:16,216 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-19000.pt
2024-08-06 08:23:42,943 INFO [trainer.py:765] (0/8) Epoch 8, batch 1000, train_loss[loss=3.636, NarTop10Accuracy=0.5943, over 6188.00 frames. ], tot_loss[loss=3.664, NarTop10Accuracy=0.5835, over 5912.06 frames. ], batch size: 13, lr: 1.06e-02
2024-08-06 08:24:15,105 INFO [trainer.py:765] (0/8) Epoch 8, batch 1100, train_loss[loss=3.734, NarTop10Accuracy=0.5818, over 6773.00 frames. ], tot_loss[loss=3.665, NarTop10Accuracy=0.5836, over 5936.16 frames. ], batch size: 17, lr: 1.06e-02
2024-08-06 08:24:57,340 INFO [trainer.py:765] (0/8) Epoch 8, batch 1200, train_loss[loss=3.464, NarTop10Accuracy=0.6215, over 7169.00 frames. ], tot_loss[loss=3.669, NarTop10Accuracy=0.5825, over 5944.29 frames. ], batch size: 30, lr: 1.06e-02
2024-08-06 08:25:26,605 INFO [trainer.py:765] (0/8) Epoch 8, batch 1300, train_loss[loss=3.823, NarTop10Accuracy=0.5542, over 5229.00 frames. ], tot_loss[loss=3.664, NarTop10Accuracy=0.5833, over 6001.06 frames. ], batch size: 6, lr: 1.06e-02
2024-08-06 08:26:00,605 INFO [trainer.py:765] (0/8) Epoch 8, batch 1400, train_loss[loss=3.892, NarTop10Accuracy=0.551, over 6098.00 frames. ], tot_loss[loss=3.682, NarTop10Accuracy=0.5794, over 6028.09 frames. ], batch size: 11, lr: 1.05e-02
2024-08-06 08:26:28,987 INFO [trainer.py:765] (0/8) Epoch 8, batch 1500, train_loss[loss=3.722, NarTop10Accuracy=0.5765, over 6464.00 frames. ], tot_loss[loss=3.677, NarTop10Accuracy=0.5802, over 5958.48 frames. ], batch size: 49, lr: 1.05e-02
2024-08-06 08:26:56,933 INFO [trainer.py:765] (0/8) Epoch 8, batch 1600, train_loss[loss=3.731, NarTop10Accuracy=0.5692, over 7239.00 frames. ], tot_loss[loss=3.676, NarTop10Accuracy=0.5804, over 5946.21 frames. ], batch size: 22, lr: 1.05e-02
2024-08-06 08:27:23,763 INFO [trainer.py:765] (0/8) Epoch 8, batch 1700, train_loss[loss=3.467, NarTop10Accuracy=0.6268, over 6320.00 frames. ], tot_loss[loss=3.671, NarTop10Accuracy=0.5816, over 5937.85 frames. ], batch size: 13, lr: 1.05e-02
2024-08-06 08:27:50,462 INFO [trainer.py:765] (0/8) Epoch 8, batch 1800, train_loss[loss=3.611, NarTop10Accuracy=0.5911, over 7052.00 frames. ], tot_loss[loss=3.663, NarTop10Accuracy=0.5834, over 5994.76 frames. ], batch size: 22, lr: 1.04e-02
2024-08-06 08:28:17,181 INFO [trainer.py:765] (0/8) Epoch 8, batch 1900, train_loss[loss=4.195, NarTop10Accuracy=0.4752, over 6141.00 frames. ], tot_loss[loss=3.66, NarTop10Accuracy=0.5843, over 6023.73 frames. ], batch size: 49, lr: 1.04e-02
2024-08-06 08:28:25,164 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-20000.pt
2024-08-06 08:28:28,748 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 08:28:35,290 INFO [trainer.py:811] (0/8) Epoch 8, validation: loss=3.507, NarTop10Accuracy=0.6181, over 1907754.00 frames. 
2024-08-06 08:28:35,291 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30501MB
2024-08-06 08:28:35,795 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.304e+02 1.789e+02 1.988e+02 2.230e+02 4.452e+02, threshold=3.975e+02, percent-clipped=0.5
2024-08-06 08:28:52,983 INFO [trainer.py:765] (0/8) Epoch 8, batch 2000, train_loss[loss=3.849, NarTop10Accuracy=0.5466, over 6268.00 frames. ], tot_loss[loss=3.662, NarTop10Accuracy=0.5838, over 6010.19 frames. ], batch size: 48, lr: 1.04e-02
2024-08-06 08:29:18,485 INFO [trainer.py:765] (0/8) Epoch 8, batch 2100, train_loss[loss=3.569, NarTop10Accuracy=0.5995, over 4763.00 frames. ], tot_loss[loss=3.667, NarTop10Accuracy=0.583, over 6022.36 frames. ], batch size: 5, lr: 1.04e-02
2024-08-06 08:29:43,790 INFO [trainer.py:765] (0/8) Epoch 8, batch 2200, train_loss[loss=3.836, NarTop10Accuracy=0.5446, over 7488.00 frames. ], tot_loss[loss=3.664, NarTop10Accuracy=0.5833, over 6063.11 frames. ], batch size: 30, lr: 1.03e-02
2024-08-06 08:30:09,134 INFO [trainer.py:765] (0/8) Epoch 8, batch 2300, train_loss[loss=3.471, NarTop10Accuracy=0.6154, over 5858.00 frames. ], tot_loss[loss=3.669, NarTop10Accuracy=0.5824, over 6080.13 frames. ], batch size: 9, lr: 1.03e-02
2024-08-06 08:30:33,791 INFO [trainer.py:765] (0/8) Epoch 8, batch 2400, train_loss[loss=3.797, NarTop10Accuracy=0.565, over 6106.00 frames. ], tot_loss[loss=3.684, NarTop10Accuracy=0.5793, over 5879.89 frames. ], batch size: 49, lr: 1.03e-02
2024-08-06 08:30:57,139 INFO [trainer.py:765] (0/8) Epoch 8, batch 2500, train_loss[loss=3.598, NarTop10Accuracy=0.5972, over 4948.00 frames. ], tot_loss[loss=3.666, NarTop10Accuracy=0.5824, over 5547.01 frames. ], batch size: 6, lr: 1.03e-02
2024-08-06 08:31:18,653 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 08:31:18,660 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-8.pt
2024-08-06 08:32:19,098 INFO [trainer.py:765] (0/8) Epoch 9, batch 100, train_loss[loss=3.86, NarTop10Accuracy=0.5517, over 7244.00 frames. ], tot_loss[loss=3.607, NarTop10Accuracy=0.5953, over 2383.37 frames. ], batch size: 31, lr: 9.71e-03
2024-08-06 08:32:51,461 INFO [trainer.py:765] (0/8) Epoch 9, batch 200, train_loss[loss=3.394, NarTop10Accuracy=0.6332, over 6838.00 frames. ], tot_loss[loss=3.584, NarTop10Accuracy=0.6004, over 3857.41 frames. ], batch size: 17, lr: 9.69e-03
2024-08-06 08:33:27,115 INFO [trainer.py:765] (0/8) Epoch 9, batch 300, train_loss[loss=3.586, NarTop10Accuracy=0.5872, over 7265.00 frames. ], tot_loss[loss=3.586, NarTop10Accuracy=0.6002, over 4673.13 frames. ], batch size: 23, lr: 9.67e-03
2024-08-06 08:33:44,054 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-21000.pt
2024-08-06 08:34:00,964 INFO [trainer.py:765] (0/8) Epoch 9, batch 400, train_loss[loss=3.459, NarTop10Accuracy=0.6349, over 5188.00 frames. ], tot_loss[loss=3.576, NarTop10Accuracy=0.6025, over 5123.22 frames. ], batch size: 7, lr: 9.64e-03
2024-08-06 08:34:32,880 INFO [trainer.py:765] (0/8) Epoch 9, batch 500, train_loss[loss=3.84, NarTop10Accuracy=0.5522, over 6004.00 frames. ], tot_loss[loss=3.571, NarTop10Accuracy=0.6031, over 5425.79 frames. ], batch size: 11, lr: 9.62e-03
2024-08-06 08:35:07,498 INFO [trainer.py:765] (0/8) Epoch 9, batch 600, train_loss[loss=3.64, NarTop10Accuracy=0.5954, over 5720.00 frames. ], tot_loss[loss=3.578, NarTop10Accuracy=0.6017, over 5681.69 frames. ], batch size: 9, lr: 9.60e-03
2024-08-06 08:35:42,824 INFO [trainer.py:765] (0/8) Epoch 9, batch 700, train_loss[loss=3.922, NarTop10Accuracy=0.5215, over 4245.00 frames. ], tot_loss[loss=3.593, NarTop10Accuracy=0.5986, over 5741.95 frames. ], batch size: 5, lr: 9.58e-03
2024-08-06 08:36:14,822 INFO [trainer.py:765] (0/8) Epoch 9, batch 800, train_loss[loss=3.575, NarTop10Accuracy=0.6104, over 4300.00 frames. ], tot_loss[loss=3.613, NarTop10Accuracy=0.5941, over 5816.00 frames. ], batch size: 5, lr: 9.56e-03
2024-08-06 08:36:46,455 INFO [trainer.py:765] (0/8) Epoch 9, batch 900, train_loss[loss=3.455, NarTop10Accuracy=0.6297, over 6230.00 frames. ], tot_loss[loss=3.619, NarTop10Accuracy=0.5931, over 5833.15 frames. ], batch size: 13, lr: 9.54e-03
2024-08-06 08:37:26,564 INFO [trainer.py:765] (0/8) Epoch 9, batch 1000, train_loss[loss=3.451, NarTop10Accuracy=0.6105, over 6207.00 frames. ], tot_loss[loss=3.623, NarTop10Accuracy=0.5917, over 5924.30 frames. ], batch size: 13, lr: 9.52e-03
2024-08-06 08:37:59,421 INFO [trainer.py:765] (0/8) Epoch 9, batch 1100, train_loss[loss=3.868, NarTop10Accuracy=0.5473, over 6856.00 frames. ], tot_loss[loss=3.64, NarTop10Accuracy=0.5886, over 5971.38 frames. ], batch size: 17, lr: 9.50e-03
2024-08-06 08:38:31,995 INFO [trainer.py:765] (0/8) Epoch 9, batch 1200, train_loss[loss=3.725, NarTop10Accuracy=0.5711, over 7134.00 frames. ], tot_loss[loss=3.644, NarTop10Accuracy=0.5878, over 5945.82 frames. ], batch size: 30, lr: 9.48e-03
2024-08-06 08:39:11,841 INFO [trainer.py:765] (0/8) Epoch 9, batch 1300, train_loss[loss=3.837, NarTop10Accuracy=0.5482, over 5159.00 frames. ], tot_loss[loss=3.64, NarTop10Accuracy=0.5879, over 6022.17 frames. ], batch size: 6, lr: 9.46e-03
2024-08-06 08:39:27,116 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-22000.pt
2024-08-06 08:39:30,597 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 08:39:38,196 INFO [trainer.py:811] (0/8) Epoch 9, validation: loss=3.495, NarTop10Accuracy=0.6214, over 1907754.00 frames. 
2024-08-06 08:39:38,197 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30501MB
2024-08-06 08:39:38,758 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.320e+02 1.781e+02 1.970e+02 2.189e+02 6.315e+02, threshold=3.940e+02, percent-clipped=0.6
2024-08-06 08:39:52,278 INFO [trainer.py:765] (0/8) Epoch 9, batch 1400, train_loss[loss=3.58, NarTop10Accuracy=0.6019, over 6107.00 frames. ], tot_loss[loss=3.631, NarTop10Accuracy=0.5901, over 6032.03 frames. ], batch size: 11, lr: 9.43e-03
2024-08-06 08:40:22,332 INFO [trainer.py:765] (0/8) Epoch 9, batch 1500, train_loss[loss=3.918, NarTop10Accuracy=0.5263, over 6152.00 frames. ], tot_loss[loss=3.634, NarTop10Accuracy=0.5898, over 5972.55 frames. ], batch size: 50, lr: 9.41e-03
2024-08-06 08:40:50,368 INFO [trainer.py:765] (0/8) Epoch 9, batch 1600, train_loss[loss=3.872, NarTop10Accuracy=0.5455, over 7338.00 frames. ], tot_loss[loss=3.634, NarTop10Accuracy=0.5898, over 5959.86 frames. ], batch size: 22, lr: 9.39e-03
2024-08-06 08:41:17,152 INFO [trainer.py:765] (0/8) Epoch 9, batch 1700, train_loss[loss=3.459, NarTop10Accuracy=0.6202, over 6248.00 frames. ], tot_loss[loss=3.642, NarTop10Accuracy=0.5884, over 5939.35 frames. ], batch size: 13, lr: 9.37e-03
2024-08-06 08:41:43,812 INFO [trainer.py:765] (0/8) Epoch 9, batch 1800, train_loss[loss=4.052, NarTop10Accuracy=0.4983, over 7041.00 frames. ], tot_loss[loss=3.633, NarTop10Accuracy=0.5898, over 5998.82 frames. ], batch size: 22, lr: 9.35e-03
2024-08-06 08:42:10,496 INFO [trainer.py:765] (0/8) Epoch 9, batch 1900, train_loss[loss=3.715, NarTop10Accuracy=0.5691, over 5946.00 frames. ], tot_loss[loss=3.637, NarTop10Accuracy=0.5893, over 6036.97 frames. ], batch size: 49, lr: 9.33e-03
2024-08-06 08:42:36,203 INFO [trainer.py:765] (0/8) Epoch 9, batch 2000, train_loss[loss=3.916, NarTop10Accuracy=0.5382, over 6131.00 frames. ], tot_loss[loss=3.642, NarTop10Accuracy=0.5883, over 6009.19 frames. ], batch size: 50, lr: 9.31e-03
2024-08-06 08:43:01,668 INFO [trainer.py:765] (0/8) Epoch 9, batch 2100, train_loss[loss=3.36, NarTop10Accuracy=0.6465, over 3969.00 frames. ], tot_loss[loss=3.633, NarTop10Accuracy=0.5903, over 5987.78 frames. ], batch size: 4, lr: 9.30e-03
2024-08-06 08:43:27,179 INFO [trainer.py:765] (0/8) Epoch 9, batch 2200, train_loss[loss=3.549, NarTop10Accuracy=0.6043, over 7247.00 frames. ], tot_loss[loss=3.638, NarTop10Accuracy=0.589, over 6034.43 frames. ], batch size: 30, lr: 9.28e-03
2024-08-06 08:43:52,671 INFO [trainer.py:765] (0/8) Epoch 9, batch 2300, train_loss[loss=3.575, NarTop10Accuracy=0.5945, over 5779.00 frames. ], tot_loss[loss=3.655, NarTop10Accuracy=0.5856, over 6074.03 frames. ], batch size: 9, lr: 9.26e-03
2024-08-06 08:44:05,503 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-23000.pt
2024-08-06 08:44:20,550 INFO [trainer.py:765] (0/8) Epoch 9, batch 2400, train_loss[loss=3.688, NarTop10Accuracy=0.5842, over 6015.00 frames. ], tot_loss[loss=3.659, NarTop10Accuracy=0.5847, over 5903.28 frames. ], batch size: 49, lr: 9.24e-03
2024-08-06 08:44:44,002 INFO [trainer.py:765] (0/8) Epoch 9, batch 2500, train_loss[loss=3.972, NarTop10Accuracy=0.5312, over 4333.00 frames. ], tot_loss[loss=3.624, NarTop10Accuracy=0.5912, over 5553.37 frames. ], batch size: 5, lr: 9.22e-03
2024-08-06 08:45:05,295 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 08:45:05,298 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-9.pt
2024-08-06 08:46:09,064 INFO [trainer.py:765] (0/8) Epoch 10, batch 100, train_loss[loss=3.477, NarTop10Accuracy=0.6214, over 7408.00 frames. ], tot_loss[loss=3.588, NarTop10Accuracy=0.5995, over 2351.73 frames. ], batch size: 31, lr: 8.75e-03
2024-08-06 08:46:44,075 INFO [trainer.py:765] (0/8) Epoch 10, batch 200, train_loss[loss=3.558, NarTop10Accuracy=0.6102, over 6961.00 frames. ], tot_loss[loss=3.573, NarTop10Accuracy=0.6026, over 3849.94 frames. ], batch size: 17, lr: 8.73e-03
2024-08-06 08:47:14,444 INFO [trainer.py:765] (0/8) Epoch 10, batch 300, train_loss[loss=3.661, NarTop10Accuracy=0.5858, over 7158.00 frames. ], tot_loss[loss=3.578, NarTop10Accuracy=0.6019, over 4654.40 frames. ], batch size: 22, lr: 8.72e-03
2024-08-06 08:47:46,120 INFO [trainer.py:765] (0/8) Epoch 10, batch 400, train_loss[loss=3.983, NarTop10Accuracy=0.5173, over 5216.00 frames. ], tot_loss[loss=3.586, NarTop10Accuracy=0.6, over 5117.58 frames. ], batch size: 7, lr: 8.70e-03
2024-08-06 08:48:22,371 INFO [trainer.py:765] (0/8) Epoch 10, batch 500, train_loss[loss=3.296, NarTop10Accuracy=0.6624, over 5996.00 frames. ], tot_loss[loss=3.579, NarTop10Accuracy=0.6016, over 5391.67 frames. ], batch size: 11, lr: 8.68e-03
2024-08-06 08:48:53,460 INFO [trainer.py:765] (0/8) Epoch 10, batch 600, train_loss[loss=3.585, NarTop10Accuracy=0.5987, over 5835.00 frames. ], tot_loss[loss=3.587, NarTop10Accuracy=0.5997, over 5650.31 frames. ], batch size: 9, lr: 8.66e-03
2024-08-06 08:49:26,707 INFO [trainer.py:765] (0/8) Epoch 10, batch 700, train_loss[loss=3.334, NarTop10Accuracy=0.6538, over 5053.00 frames. ], tot_loss[loss=3.595, NarTop10Accuracy=0.5979, over 5741.09 frames. ], batch size: 6, lr: 8.65e-03
2024-08-06 08:49:49,164 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-24000.pt
2024-08-06 08:49:53,523 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 08:50:00,983 INFO [trainer.py:811] (0/8) Epoch 10, validation: loss=3.46, NarTop10Accuracy=0.6279, over 1907754.00 frames. 
2024-08-06 08:50:00,984 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30501MB
2024-08-06 08:50:01,725 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.353e+02 1.818e+02 1.985e+02 2.213e+02 4.843e+02, threshold=3.970e+02, percent-clipped=0.2
2024-08-06 08:50:09,801 INFO [trainer.py:765] (0/8) Epoch 10, batch 800, train_loss[loss=3.514, NarTop10Accuracy=0.623, over 4311.00 frames. ], tot_loss[loss=3.595, NarTop10Accuracy=0.5978, over 5785.74 frames. ], batch size: 5, lr: 8.63e-03
2024-08-06 08:50:42,891 INFO [trainer.py:765] (0/8) Epoch 10, batch 900, train_loss[loss=3.509, NarTop10Accuracy=0.6133, over 6149.00 frames. ], tot_loss[loss=3.571, NarTop10Accuracy=0.6025, over 5802.10 frames. ], batch size: 13, lr: 8.61e-03
2024-08-06 08:51:18,460 INFO [trainer.py:765] (0/8) Epoch 10, batch 1000, train_loss[loss=3.806, NarTop10Accuracy=0.5503, over 6758.00 frames. ], tot_loss[loss=3.586, NarTop10Accuracy=0.5998, over 5903.04 frames. ], batch size: 14, lr: 8.59e-03
2024-08-06 08:51:57,363 INFO [trainer.py:765] (0/8) Epoch 10, batch 1100, train_loss[loss=3.428, NarTop10Accuracy=0.6365, over 6821.00 frames. ], tot_loss[loss=3.603, NarTop10Accuracy=0.5958, over 5935.00 frames. ], batch size: 17, lr: 8.58e-03
2024-08-06 08:52:32,048 INFO [trainer.py:765] (0/8) Epoch 10, batch 1200, train_loss[loss=3.485, NarTop10Accuracy=0.6121, over 7264.00 frames. ], tot_loss[loss=3.6, NarTop10Accuracy=0.5959, over 5941.85 frames. ], batch size: 31, lr: 8.56e-03
2024-08-06 08:53:06,607 INFO [trainer.py:765] (0/8) Epoch 10, batch 1300, train_loss[loss=3.453, NarTop10Accuracy=0.6196, over 5114.00 frames. ], tot_loss[loss=3.599, NarTop10Accuracy=0.5964, over 6019.00 frames. ], batch size: 6, lr: 8.54e-03
2024-08-06 08:53:46,881 INFO [trainer.py:765] (0/8) Epoch 10, batch 1400, train_loss[loss=3.609, NarTop10Accuracy=0.5969, over 6199.00 frames. ], tot_loss[loss=3.616, NarTop10Accuracy=0.5928, over 6033.05 frames. ], batch size: 11, lr: 8.53e-03
2024-08-06 08:54:17,501 INFO [trainer.py:765] (0/8) Epoch 10, batch 1500, train_loss[loss=3.629, NarTop10Accuracy=0.5957, over 6407.00 frames. ], tot_loss[loss=3.609, NarTop10Accuracy=0.5947, over 5977.42 frames. ], batch size: 48, lr: 8.51e-03
2024-08-06 08:54:45,526 INFO [trainer.py:765] (0/8) Epoch 10, batch 1600, train_loss[loss=3.567, NarTop10Accuracy=0.6043, over 7212.00 frames. ], tot_loss[loss=3.61, NarTop10Accuracy=0.5942, over 5955.29 frames. ], batch size: 22, lr: 8.49e-03
2024-08-06 08:55:12,300 INFO [trainer.py:765] (0/8) Epoch 10, batch 1700, train_loss[loss=3.432, NarTop10Accuracy=0.625, over 6419.00 frames. ], tot_loss[loss=3.612, NarTop10Accuracy=0.5942, over 5947.06 frames. ], batch size: 13, lr: 8.48e-03
2024-08-06 08:55:30,798 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-25000.pt
2024-08-06 08:55:41,989 INFO [trainer.py:765] (0/8) Epoch 10, batch 1800, train_loss[loss=3.685, NarTop10Accuracy=0.5767, over 7196.00 frames. ], tot_loss[loss=3.606, NarTop10Accuracy=0.5954, over 5993.31 frames. ], batch size: 22, lr: 8.46e-03
2024-08-06 08:56:08,572 INFO [trainer.py:765] (0/8) Epoch 10, batch 1900, train_loss[loss=4.071, NarTop10Accuracy=0.5032, over 5872.00 frames. ], tot_loss[loss=3.603, NarTop10Accuracy=0.5957, over 6022.31 frames. ], batch size: 48, lr: 8.45e-03
2024-08-06 08:56:34,287 INFO [trainer.py:765] (0/8) Epoch 10, batch 2000, train_loss[loss=3.722, NarTop10Accuracy=0.5667, over 6291.00 frames. ], tot_loss[loss=3.606, NarTop10Accuracy=0.5953, over 6005.84 frames. ], batch size: 48, lr: 8.43e-03
2024-08-06 08:56:59,751 INFO [trainer.py:765] (0/8) Epoch 10, batch 2100, train_loss[loss=3.376, NarTop10Accuracy=0.6549, over 3931.00 frames. ], tot_loss[loss=3.609, NarTop10Accuracy=0.5946, over 5973.46 frames. ], batch size: 4, lr: 8.41e-03
2024-08-06 08:57:25,279 INFO [trainer.py:765] (0/8) Epoch 10, batch 2200, train_loss[loss=3.725, NarTop10Accuracy=0.5691, over 7294.00 frames. ], tot_loss[loss=3.606, NarTop10Accuracy=0.5949, over 6029.47 frames. ], batch size: 31, lr: 8.40e-03
2024-08-06 08:57:50,682 INFO [trainer.py:765] (0/8) Epoch 10, batch 2300, train_loss[loss=3.448, NarTop10Accuracy=0.6167, over 5719.00 frames. ], tot_loss[loss=3.617, NarTop10Accuracy=0.5929, over 6063.88 frames. ], batch size: 9, lr: 8.38e-03
2024-08-06 08:58:15,344 INFO [trainer.py:765] (0/8) Epoch 10, batch 2400, train_loss[loss=3.615, NarTop10Accuracy=0.6009, over 6453.00 frames. ], tot_loss[loss=3.622, NarTop10Accuracy=0.5921, over 5895.18 frames. ], batch size: 51, lr: 8.37e-03
2024-08-06 08:58:38,808 INFO [trainer.py:765] (0/8) Epoch 10, batch 2500, train_loss[loss=3.404, NarTop10Accuracy=0.6459, over 5151.00 frames. ], tot_loss[loss=3.613, NarTop10Accuracy=0.5937, over 5535.09 frames. ], batch size: 6, lr: 8.35e-03
2024-08-06 08:59:00,223 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 08:59:00,226 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-10.pt
2024-08-06 09:00:03,681 INFO [trainer.py:765] (0/8) Epoch 11, batch 100, train_loss[loss=3.437, NarTop10Accuracy=0.6268, over 7166.00 frames. ], tot_loss[loss=3.537, NarTop10Accuracy=0.611, over 2362.90 frames. ], batch size: 30, lr: 7.96e-03
2024-08-06 09:00:30,916 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-26000.pt
2024-08-06 09:00:34,445 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 09:00:41,217 INFO [trainer.py:811] (0/8) Epoch 11, validation: loss=3.404, NarTop10Accuracy=0.6396, over 1907754.00 frames. 
2024-08-06 09:00:41,218 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30501MB
2024-08-06 09:00:41,774 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.377e+02 1.800e+02 1.980e+02 2.200e+02 4.491e+02, threshold=3.959e+02, percent-clipped=0.2
2024-08-06 09:00:46,860 INFO [trainer.py:765] (0/8) Epoch 11, batch 200, train_loss[loss=3.882, NarTop10Accuracy=0.5397, over 6881.00 frames. ], tot_loss[loss=3.529, NarTop10Accuracy=0.6121, over 3860.45 frames. ], batch size: 17, lr: 7.94e-03
2024-08-06 09:01:17,854 INFO [trainer.py:765] (0/8) Epoch 11, batch 300, train_loss[loss=3.279, NarTop10Accuracy=0.664, over 7183.00 frames. ], tot_loss[loss=3.536, NarTop10Accuracy=0.6106, over 4669.68 frames. ], batch size: 22, lr: 7.93e-03
2024-08-06 09:01:50,534 INFO [trainer.py:765] (0/8) Epoch 11, batch 400, train_loss[loss=3.23, NarTop10Accuracy=0.6779, over 5099.00 frames. ], tot_loss[loss=3.533, NarTop10Accuracy=0.6111, over 5109.41 frames. ], batch size: 7, lr: 7.91e-03
2024-08-06 09:02:21,239 INFO [trainer.py:765] (0/8) Epoch 11, batch 500, train_loss[loss=3.391, NarTop10Accuracy=0.6398, over 6167.00 frames. ], tot_loss[loss=3.535, NarTop10Accuracy=0.6111, over 5394.60 frames. ], batch size: 11, lr: 7.90e-03
2024-08-06 09:03:01,743 INFO [trainer.py:765] (0/8) Epoch 11, batch 600, train_loss[loss=3.516, NarTop10Accuracy=0.6248, over 5799.00 frames. ], tot_loss[loss=3.54, NarTop10Accuracy=0.6097, over 5672.92 frames. ], batch size: 9, lr: 7.88e-03
2024-08-06 09:03:38,237 INFO [trainer.py:765] (0/8) Epoch 11, batch 700, train_loss[loss=3.289, NarTop10Accuracy=0.6565, over 5103.00 frames. ], tot_loss[loss=3.54, NarTop10Accuracy=0.6098, over 5746.87 frames. ], batch size: 6, lr: 7.87e-03
2024-08-06 09:04:10,756 INFO [trainer.py:765] (0/8) Epoch 11, batch 800, train_loss[loss=3.281, NarTop10Accuracy=0.6708, over 5065.00 frames. ], tot_loss[loss=3.562, NarTop10Accuracy=0.6057, over 5799.96 frames. ], batch size: 6, lr: 7.86e-03
2024-08-06 09:04:50,084 INFO [trainer.py:765] (0/8) Epoch 11, batch 900, train_loss[loss=3.419, NarTop10Accuracy=0.6239, over 6649.00 frames. ], tot_loss[loss=3.548, NarTop10Accuracy=0.6077, over 5818.17 frames. ], batch size: 14, lr: 7.84e-03
2024-08-06 09:05:27,013 INFO [trainer.py:765] (0/8) Epoch 11, batch 1000, train_loss[loss=3.426, NarTop10Accuracy=0.6302, over 6308.00 frames. ], tot_loss[loss=3.551, NarTop10Accuracy=0.6069, over 5931.02 frames. ], batch size: 13, lr: 7.83e-03
2024-08-06 09:06:00,351 INFO [trainer.py:765] (0/8) Epoch 11, batch 1100, train_loss[loss=3.558, NarTop10Accuracy=0.6073, over 6791.00 frames. ], tot_loss[loss=3.564, NarTop10Accuracy=0.6042, over 5964.91 frames. ], batch size: 17, lr: 7.81e-03
2024-08-06 09:06:31,393 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-27000.pt
2024-08-06 09:06:40,946 INFO [trainer.py:765] (0/8) Epoch 11, batch 1200, train_loss[loss=3.694, NarTop10Accuracy=0.5734, over 7219.00 frames. ], tot_loss[loss=3.573, NarTop10Accuracy=0.6019, over 5962.45 frames. ], batch size: 30, lr: 7.80e-03
2024-08-06 09:07:15,495 INFO [trainer.py:765] (0/8) Epoch 11, batch 1300, train_loss[loss=3.566, NarTop10Accuracy=0.6176, over 5192.00 frames. ], tot_loss[loss=3.57, NarTop10Accuracy=0.6023, over 6022.95 frames. ], batch size: 6, lr: 7.79e-03
2024-08-06 09:07:47,629 INFO [trainer.py:765] (0/8) Epoch 11, batch 1400, train_loss[loss=3.413, NarTop10Accuracy=0.6282, over 6160.00 frames. ], tot_loss[loss=3.577, NarTop10Accuracy=0.6006, over 6029.30 frames. ], batch size: 11, lr: 7.77e-03
2024-08-06 09:08:18,988 INFO [trainer.py:765] (0/8) Epoch 11, batch 1500, train_loss[loss=3.576, NarTop10Accuracy=0.6082, over 6282.00 frames. ], tot_loss[loss=3.583, NarTop10Accuracy=0.5993, over 5978.15 frames. ], batch size: 48, lr: 7.76e-03
2024-08-06 09:08:47,149 INFO [trainer.py:765] (0/8) Epoch 11, batch 1600, train_loss[loss=3.532, NarTop10Accuracy=0.6103, over 7195.00 frames. ], tot_loss[loss=3.575, NarTop10Accuracy=0.6013, over 5976.97 frames. ], batch size: 22, lr: 7.74e-03
2024-08-06 09:09:13,951 INFO [trainer.py:765] (0/8) Epoch 11, batch 1700, train_loss[loss=3.502, NarTop10Accuracy=0.6276, over 6447.00 frames. ], tot_loss[loss=3.575, NarTop10Accuracy=0.6016, over 5951.23 frames. ], batch size: 14, lr: 7.73e-03
2024-08-06 09:09:40,733 INFO [trainer.py:765] (0/8) Epoch 11, batch 1800, train_loss[loss=3.691, NarTop10Accuracy=0.5781, over 7050.00 frames. ], tot_loss[loss=3.575, NarTop10Accuracy=0.6016, over 6003.46 frames. ], batch size: 22, lr: 7.72e-03
2024-08-06 09:10:07,343 INFO [trainer.py:765] (0/8) Epoch 11, batch 1900, train_loss[loss=3.761, NarTop10Accuracy=0.5645, over 6362.00 frames. ], tot_loss[loss=3.592, NarTop10Accuracy=0.598, over 6041.23 frames. ], batch size: 52, lr: 7.70e-03
2024-08-06 09:10:33,040 INFO [trainer.py:765] (0/8) Epoch 11, batch 2000, train_loss[loss=3.601, NarTop10Accuracy=0.6046, over 5913.00 frames. ], tot_loss[loss=3.589, NarTop10Accuracy=0.5988, over 6010.07 frames. ], batch size: 49, lr: 7.69e-03
2024-08-06 09:10:58,442 INFO [trainer.py:765] (0/8) Epoch 11, batch 2100, train_loss[loss=3.493, NarTop10Accuracy=0.619, over 4923.00 frames. ], tot_loss[loss=3.574, NarTop10Accuracy=0.6017, over 6000.09 frames. ], batch size: 5, lr: 7.68e-03
2024-08-06 09:11:20,708 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-28000.pt
2024-08-06 09:11:24,493 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 09:11:31,457 INFO [trainer.py:811] (0/8) Epoch 11, validation: loss=3.372, NarTop10Accuracy=0.6462, over 1907754.00 frames. 
2024-08-06 09:11:31,458 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30501MB
2024-08-06 09:11:31,930 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.411e+02 1.800e+02 1.966e+02 2.160e+02 4.000e+02, threshold=3.933e+02, percent-clipped=0.1
2024-08-06 09:11:34,519 INFO [trainer.py:765] (0/8) Epoch 11, batch 2200, train_loss[loss=3.458, NarTop10Accuracy=0.6221, over 7333.00 frames. ], tot_loss[loss=3.573, NarTop10Accuracy=0.6021, over 6035.96 frames. ], batch size: 32, lr: 7.66e-03
2024-08-06 09:11:59,940 INFO [trainer.py:765] (0/8) Epoch 11, batch 2300, train_loss[loss=3.41, NarTop10Accuracy=0.6401, over 5751.00 frames. ], tot_loss[loss=3.582, NarTop10Accuracy=0.6001, over 6074.01 frames. ], batch size: 9, lr: 7.65e-03
2024-08-06 09:12:24,696 INFO [trainer.py:765] (0/8) Epoch 11, batch 2400, train_loss[loss=3.899, NarTop10Accuracy=0.5409, over 6411.00 frames. ], tot_loss[loss=3.601, NarTop10Accuracy=0.5966, over 5891.73 frames. ], batch size: 50, lr: 7.64e-03
2024-08-06 09:12:47,879 INFO [trainer.py:765] (0/8) Epoch 11, batch 2500, train_loss[loss=3.826, NarTop10Accuracy=0.5491, over 5114.00 frames. ], tot_loss[loss=3.582, NarTop10Accuracy=0.5997, over 5537.57 frames. ], batch size: 6, lr: 7.62e-03
2024-08-06 09:13:09,236 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 09:13:09,239 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-11.pt
2024-08-06 09:14:12,278 INFO [trainer.py:765] (0/8) Epoch 12, batch 100, train_loss[loss=3.424, NarTop10Accuracy=0.6485, over 7247.00 frames. ], tot_loss[loss=3.536, NarTop10Accuracy=0.6108, over 2365.90 frames. ], batch size: 31, lr: 7.29e-03
2024-08-06 09:14:48,096 INFO [trainer.py:765] (0/8) Epoch 12, batch 200, train_loss[loss=3.416, NarTop10Accuracy=0.6393, over 6817.00 frames. ], tot_loss[loss=3.509, NarTop10Accuracy=0.6165, over 3865.92 frames. ], batch size: 17, lr: 7.28e-03
2024-08-06 09:15:20,021 INFO [trainer.py:765] (0/8) Epoch 12, batch 300, train_loss[loss=3.388, NarTop10Accuracy=0.6405, over 7241.00 frames. ], tot_loss[loss=3.504, NarTop10Accuracy=0.6178, over 4656.48 frames. ], batch size: 22, lr: 7.27e-03
2024-08-06 09:15:52,633 INFO [trainer.py:765] (0/8) Epoch 12, batch 400, train_loss[loss=3.352, NarTop10Accuracy=0.6288, over 5085.00 frames. ], tot_loss[loss=3.519, NarTop10Accuracy=0.6146, over 5122.30 frames. ], batch size: 7, lr: 7.25e-03
2024-08-06 09:16:26,433 INFO [trainer.py:765] (0/8) Epoch 12, batch 500, train_loss[loss=3.82, NarTop10Accuracy=0.5551, over 6150.00 frames. ], tot_loss[loss=3.529, NarTop10Accuracy=0.6121, over 5403.60 frames. ], batch size: 11, lr: 7.24e-03
2024-08-06 09:16:59,239 INFO [trainer.py:765] (0/8) Epoch 12, batch 600, train_loss[loss=3.467, NarTop10Accuracy=0.6259, over 5804.00 frames. ], tot_loss[loss=3.537, NarTop10Accuracy=0.6101, over 5673.31 frames. ], batch size: 9, lr: 7.23e-03
2024-08-06 09:17:01,178 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-29000.pt
2024-08-06 09:17:36,318 INFO [trainer.py:765] (0/8) Epoch 12, batch 700, train_loss[loss=3.659, NarTop10Accuracy=0.582, over 5130.00 frames. ], tot_loss[loss=3.535, NarTop10Accuracy=0.61, over 5744.73 frames. ], batch size: 6, lr: 7.22e-03
2024-08-06 09:18:07,753 INFO [trainer.py:765] (0/8) Epoch 12, batch 800, train_loss[loss=3.423, NarTop10Accuracy=0.6315, over 5081.00 frames. ], tot_loss[loss=3.534, NarTop10Accuracy=0.6104, over 5802.17 frames. ], batch size: 6, lr: 7.21e-03
2024-08-06 09:18:43,779 INFO [trainer.py:765] (0/8) Epoch 12, batch 900, train_loss[loss=3.847, NarTop10Accuracy=0.5466, over 6669.00 frames. ], tot_loss[loss=3.538, NarTop10Accuracy=0.6093, over 5834.93 frames. ], batch size: 14, lr: 7.19e-03
2024-08-06 09:19:17,689 INFO [trainer.py:765] (0/8) Epoch 12, batch 1000, train_loss[loss=3.26, NarTop10Accuracy=0.6513, over 6312.00 frames. ], tot_loss[loss=3.538, NarTop10Accuracy=0.6089, over 5931.65 frames. ], batch size: 13, lr: 7.18e-03
2024-08-06 09:19:52,427 INFO [trainer.py:765] (0/8) Epoch 12, batch 1100, train_loss[loss=3.762, NarTop10Accuracy=0.5639, over 6854.00 frames. ], tot_loss[loss=3.546, NarTop10Accuracy=0.6076, over 5964.26 frames. ], batch size: 17, lr: 7.17e-03
2024-08-06 09:20:29,443 INFO [trainer.py:765] (0/8) Epoch 12, batch 1200, train_loss[loss=3.51, NarTop10Accuracy=0.6229, over 7286.00 frames. ], tot_loss[loss=3.554, NarTop10Accuracy=0.6063, over 5951.88 frames. ], batch size: 30, lr: 7.16e-03
2024-08-06 09:21:02,826 INFO [trainer.py:765] (0/8) Epoch 12, batch 1300, train_loss[loss=3.67, NarTop10Accuracy=0.569, over 4907.00 frames. ], tot_loss[loss=3.557, NarTop10Accuracy=0.605, over 6020.02 frames. ], batch size: 6, lr: 7.15e-03
2024-08-06 09:21:36,981 INFO [trainer.py:765] (0/8) Epoch 12, batch 1400, train_loss[loss=3.229, NarTop10Accuracy=0.677, over 6231.00 frames. ], tot_loss[loss=3.564, NarTop10Accuracy=0.6035, over 6041.58 frames. ], batch size: 11, lr: 7.13e-03
2024-08-06 09:22:09,920 INFO [trainer.py:765] (0/8) Epoch 12, batch 1500, train_loss[loss=3.607, NarTop10Accuracy=0.6043, over 5908.00 frames. ], tot_loss[loss=3.557, NarTop10Accuracy=0.6051, over 5968.91 frames. ], batch size: 49, lr: 7.12e-03
2024-08-06 09:22:38,027 INFO [trainer.py:765] (0/8) Epoch 12, batch 1600, train_loss[loss=3.786, NarTop10Accuracy=0.5645, over 7100.00 frames. ], tot_loss[loss=3.561, NarTop10Accuracy=0.6041, over 5944.31 frames. ], batch size: 22, lr: 7.11e-03
2024-08-06 09:22:39,859 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-30000.pt
2024-08-06 09:22:43,418 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 09:22:49,889 INFO [trainer.py:811] (0/8) Epoch 12, validation: loss=3.364, NarTop10Accuracy=0.6481, over 1907754.00 frames. 
2024-08-06 09:22:49,889 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30501MB
2024-08-06 09:22:50,413 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.453e+02 1.796e+02 1.978e+02 2.176e+02 4.603e+02, threshold=3.957e+02, percent-clipped=0.2
2024-08-06 09:23:14,785 INFO [trainer.py:765] (0/8) Epoch 12, batch 1700, train_loss[loss=3.303, NarTop10Accuracy=0.655, over 6302.00 frames. ], tot_loss[loss=3.56, NarTop10Accuracy=0.6044, over 5943.91 frames. ], batch size: 13, lr: 7.10e-03
2024-08-06 09:23:41,386 INFO [trainer.py:765] (0/8) Epoch 12, batch 1800, train_loss[loss=3.33, NarTop10Accuracy=0.6538, over 7085.00 frames. ], tot_loss[loss=3.548, NarTop10Accuracy=0.6069, over 6008.60 frames. ], batch size: 22, lr: 7.09e-03
2024-08-06 09:24:07,957 INFO [trainer.py:765] (0/8) Epoch 12, batch 1900, train_loss[loss=3.598, NarTop10Accuracy=0.6011, over 5885.00 frames. ], tot_loss[loss=3.568, NarTop10Accuracy=0.6033, over 6053.45 frames. ], batch size: 49, lr: 7.08e-03
2024-08-06 09:24:33,618 INFO [trainer.py:765] (0/8) Epoch 12, batch 2000, train_loss[loss=3.523, NarTop10Accuracy=0.6195, over 5745.00 frames. ], tot_loss[loss=3.568, NarTop10Accuracy=0.6035, over 6024.57 frames. ], batch size: 49, lr: 7.07e-03
2024-08-06 09:24:59,038 INFO [trainer.py:765] (0/8) Epoch 12, batch 2100, train_loss[loss=3.637, NarTop10Accuracy=0.5865, over 4709.00 frames. ], tot_loss[loss=3.567, NarTop10Accuracy=0.6033, over 5985.60 frames. ], batch size: 5, lr: 7.05e-03
2024-08-06 09:25:24,509 INFO [trainer.py:765] (0/8) Epoch 12, batch 2200, train_loss[loss=3.461, NarTop10Accuracy=0.6182, over 7141.00 frames. ], tot_loss[loss=3.562, NarTop10Accuracy=0.6041, over 6020.57 frames. ], batch size: 30, lr: 7.04e-03
2024-08-06 09:25:49,926 INFO [trainer.py:765] (0/8) Epoch 12, batch 2300, train_loss[loss=3.606, NarTop10Accuracy=0.5777, over 5856.00 frames. ], tot_loss[loss=3.575, NarTop10Accuracy=0.6016, over 6061.03 frames. ], batch size: 9, lr: 7.03e-03
2024-08-06 09:26:14,656 INFO [trainer.py:765] (0/8) Epoch 12, batch 2400, train_loss[loss=3.55, NarTop10Accuracy=0.6073, over 6343.00 frames. ], tot_loss[loss=3.581, NarTop10Accuracy=0.6006, over 5880.95 frames. ], batch size: 49, lr: 7.02e-03
2024-08-06 09:26:38,154 INFO [trainer.py:765] (0/8) Epoch 12, batch 2500, train_loss[loss=3.535, NarTop10Accuracy=0.5954, over 5159.00 frames. ], tot_loss[loss=3.559, NarTop10Accuracy=0.6044, over 5528.06 frames. ], batch size: 6, lr: 7.01e-03
2024-08-06 09:26:59,637 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 09:26:59,639 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-12.pt
2024-08-06 09:27:36,183 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-31000.pt
2024-08-06 09:28:03,610 INFO [trainer.py:765] (0/8) Epoch 13, batch 100, train_loss[loss=3.532, NarTop10Accuracy=0.6127, over 7151.00 frames. ], tot_loss[loss=3.521, NarTop10Accuracy=0.6134, over 2355.84 frames. ], batch size: 30, lr: 6.72e-03
2024-08-06 09:28:36,905 INFO [trainer.py:765] (0/8) Epoch 13, batch 200, train_loss[loss=3.352, NarTop10Accuracy=0.6436, over 6918.00 frames. ], tot_loss[loss=3.507, NarTop10Accuracy=0.6164, over 3857.99 frames. ], batch size: 17, lr: 6.71e-03
2024-08-06 09:29:07,170 INFO [trainer.py:765] (0/8) Epoch 13, batch 300, train_loss[loss=3.473, NarTop10Accuracy=0.6271, over 7220.00 frames. ], tot_loss[loss=3.508, NarTop10Accuracy=0.6167, over 4671.60 frames. ], batch size: 22, lr: 6.70e-03
2024-08-06 09:29:41,038 INFO [trainer.py:765] (0/8) Epoch 13, batch 400, train_loss[loss=3.232, NarTop10Accuracy=0.6843, over 5151.00 frames. ], tot_loss[loss=3.503, NarTop10Accuracy=0.6179, over 5126.29 frames. ], batch size: 7, lr: 6.69e-03
2024-08-06 09:30:13,729 INFO [trainer.py:765] (0/8) Epoch 13, batch 500, train_loss[loss=3.784, NarTop10Accuracy=0.56, over 6088.00 frames. ], tot_loss[loss=3.501, NarTop10Accuracy=0.6179, over 5403.82 frames. ], batch size: 11, lr: 6.68e-03
2024-08-06 09:30:47,198 INFO [trainer.py:765] (0/8) Epoch 13, batch 600, train_loss[loss=3.664, NarTop10Accuracy=0.5874, over 5846.00 frames. ], tot_loss[loss=3.498, NarTop10Accuracy=0.6182, over 5670.21 frames. ], batch size: 9, lr: 6.67e-03
2024-08-06 09:31:23,820 INFO [trainer.py:765] (0/8) Epoch 13, batch 700, train_loss[loss=3.431, NarTop10Accuracy=0.6261, over 5128.00 frames. ], tot_loss[loss=3.508, NarTop10Accuracy=0.6163, over 5741.58 frames. ], batch size: 6, lr: 6.66e-03
2024-08-06 09:31:58,208 INFO [trainer.py:765] (0/8) Epoch 13, batch 800, train_loss[loss=3.359, NarTop10Accuracy=0.6345, over 4989.00 frames. ], tot_loss[loss=3.518, NarTop10Accuracy=0.6146, over 5793.13 frames. ], batch size: 6, lr: 6.65e-03
2024-08-06 09:32:29,193 INFO [trainer.py:765] (0/8) Epoch 13, batch 900, train_loss[loss=3.302, NarTop10Accuracy=0.649, over 6194.00 frames. ], tot_loss[loss=3.504, NarTop10Accuracy=0.6167, over 5819.52 frames. ], batch size: 13, lr: 6.64e-03
2024-08-06 09:33:03,133 INFO [trainer.py:765] (0/8) Epoch 13, batch 1000, train_loss[loss=3.743, NarTop10Accuracy=0.5751, over 6200.00 frames. ], tot_loss[loss=3.517, NarTop10Accuracy=0.6139, over 5914.57 frames. ], batch size: 13, lr: 6.63e-03
2024-08-06 09:33:14,217 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-32000.pt
2024-08-06 09:33:17,889 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 09:33:24,525 INFO [trainer.py:811] (0/8) Epoch 13, validation: loss=3.389, NarTop10Accuracy=0.6428, over 1907754.00 frames. 
2024-08-06 09:33:24,525 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30501MB
2024-08-06 09:33:25,132 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.457e+02 1.794e+02 1.964e+02 2.145e+02 3.608e+02, threshold=3.929e+02, percent-clipped=0.0
2024-08-06 09:33:51,714 INFO [trainer.py:765] (0/8) Epoch 13, batch 1100, train_loss[loss=3.632, NarTop10Accuracy=0.5871, over 6833.00 frames. ], tot_loss[loss=3.54, NarTop10Accuracy=0.6091, over 5951.24 frames. ], batch size: 17, lr: 6.62e-03
2024-08-06 09:34:25,485 INFO [trainer.py:765] (0/8) Epoch 13, batch 1200, train_loss[loss=3.627, NarTop10Accuracy=0.5903, over 7226.00 frames. ], tot_loss[loss=3.534, NarTop10Accuracy=0.6099, over 5960.27 frames. ], batch size: 30, lr: 6.61e-03
2024-08-06 09:35:05,084 INFO [trainer.py:765] (0/8) Epoch 13, batch 1300, train_loss[loss=3.339, NarTop10Accuracy=0.6303, over 5088.00 frames. ], tot_loss[loss=3.529, NarTop10Accuracy=0.6109, over 6027.02 frames. ], batch size: 6, lr: 6.60e-03
2024-08-06 09:35:36,404 INFO [trainer.py:765] (0/8) Epoch 13, batch 1400, train_loss[loss=3.392, NarTop10Accuracy=0.6494, over 6225.00 frames. ], tot_loss[loss=3.533, NarTop10Accuracy=0.6097, over 6047.47 frames. ], batch size: 11, lr: 6.59e-03
2024-08-06 09:36:07,319 INFO [trainer.py:765] (0/8) Epoch 13, batch 1500, train_loss[loss=3.804, NarTop10Accuracy=0.5581, over 6127.00 frames. ], tot_loss[loss=3.532, NarTop10Accuracy=0.6098, over 5965.76 frames. ], batch size: 48, lr: 6.58e-03
2024-08-06 09:36:35,388 INFO [trainer.py:765] (0/8) Epoch 13, batch 1600, train_loss[loss=3.794, NarTop10Accuracy=0.5643, over 6989.00 frames. ], tot_loss[loss=3.535, NarTop10Accuracy=0.6091, over 5954.69 frames. ], batch size: 22, lr: 6.57e-03
2024-08-06 09:37:02,143 INFO [trainer.py:765] (0/8) Epoch 13, batch 1700, train_loss[loss=3.568, NarTop10Accuracy=0.6032, over 6309.00 frames. ], tot_loss[loss=3.529, NarTop10Accuracy=0.6101, over 5931.50 frames. ], batch size: 13, lr: 6.56e-03
2024-08-06 09:37:28,778 INFO [trainer.py:765] (0/8) Epoch 13, batch 1800, train_loss[loss=3.481, NarTop10Accuracy=0.6306, over 6972.00 frames. ], tot_loss[loss=3.532, NarTop10Accuracy=0.6105, over 5990.53 frames. ], batch size: 22, lr: 6.55e-03
2024-08-06 09:37:55,386 INFO [trainer.py:765] (0/8) Epoch 13, batch 1900, train_loss[loss=3.555, NarTop10Accuracy=0.6074, over 6292.00 frames. ], tot_loss[loss=3.533, NarTop10Accuracy=0.6104, over 6037.28 frames. ], batch size: 48, lr: 6.54e-03
2024-08-06 09:38:21,122 INFO [trainer.py:765] (0/8) Epoch 13, batch 2000, train_loss[loss=3.646, NarTop10Accuracy=0.5893, over 5966.00 frames. ], tot_loss[loss=3.532, NarTop10Accuracy=0.6106, over 6003.19 frames. ], batch size: 49, lr: 6.53e-03
2024-08-06 09:38:27,582 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-33000.pt
2024-08-06 09:38:49,691 INFO [trainer.py:765] (0/8) Epoch 13, batch 2100, train_loss[loss=3.109, NarTop10Accuracy=0.6985, over 3971.00 frames. ], tot_loss[loss=3.524, NarTop10Accuracy=0.6121, over 5977.62 frames. ], batch size: 4, lr: 6.52e-03
2024-08-06 09:39:15,107 INFO [trainer.py:765] (0/8) Epoch 13, batch 2200, train_loss[loss=3.507, NarTop10Accuracy=0.6141, over 7361.00 frames. ], tot_loss[loss=3.537, NarTop10Accuracy=0.6097, over 6031.36 frames. ], batch size: 31, lr: 6.51e-03
2024-08-06 09:39:40,617 INFO [trainer.py:765] (0/8) Epoch 13, batch 2300, train_loss[loss=3.516, NarTop10Accuracy=0.6162, over 5709.00 frames. ], tot_loss[loss=3.544, NarTop10Accuracy=0.6083, over 6056.74 frames. ], batch size: 9, lr: 6.50e-03
2024-08-06 09:40:05,343 INFO [trainer.py:765] (0/8) Epoch 13, batch 2400, train_loss[loss=3.733, NarTop10Accuracy=0.5769, over 6415.00 frames. ], tot_loss[loss=3.553, NarTop10Accuracy=0.6064, over 5858.37 frames. ], batch size: 49, lr: 6.49e-03
2024-08-06 09:40:28,767 INFO [trainer.py:765] (0/8) Epoch 13, batch 2500, train_loss[loss=3.395, NarTop10Accuracy=0.6333, over 5067.00 frames. ], tot_loss[loss=3.527, NarTop10Accuracy=0.6111, over 5530.29 frames. ], batch size: 6, lr: 6.48e-03
2024-08-06 09:40:49,792 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 09:40:49,794 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-13.pt
2024-08-06 09:41:48,979 INFO [trainer.py:765] (0/8) Epoch 14, batch 100, train_loss[loss=3.407, NarTop10Accuracy=0.6368, over 7189.00 frames. ], tot_loss[loss=3.47, NarTop10Accuracy=0.6234, over 2374.56 frames. ], batch size: 30, lr: 6.24e-03
2024-08-06 09:42:22,937 INFO [trainer.py:765] (0/8) Epoch 14, batch 200, train_loss[loss=3.355, NarTop10Accuracy=0.6338, over 6898.00 frames. ], tot_loss[loss=3.463, NarTop10Accuracy=0.6252, over 3869.57 frames. ], batch size: 17, lr: 6.23e-03
2024-08-06 09:42:58,414 INFO [trainer.py:765] (0/8) Epoch 14, batch 300, train_loss[loss=3.553, NarTop10Accuracy=0.5984, over 7180.00 frames. ], tot_loss[loss=3.486, NarTop10Accuracy=0.6211, over 4678.93 frames. ], batch size: 22, lr: 6.22e-03
2024-08-06 09:43:30,439 INFO [trainer.py:765] (0/8) Epoch 14, batch 400, train_loss[loss=3.366, NarTop10Accuracy=0.6524, over 5231.00 frames. ], tot_loss[loss=3.486, NarTop10Accuracy=0.6211, over 5132.60 frames. ], batch size: 7, lr: 6.21e-03
2024-08-06 09:43:42,485 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-34000.pt
2024-08-06 09:43:46,094 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 09:43:53,651 INFO [trainer.py:811] (0/8) Epoch 14, validation: loss=3.321, NarTop10Accuracy=0.6566, over 1907754.00 frames. 
2024-08-06 09:43:53,652 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30501MB
2024-08-06 09:43:54,211 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.427e+02 1.805e+02 1.968e+02 2.158e+02 4.264e+02, threshold=3.936e+02, percent-clipped=0.2
2024-08-06 09:44:11,700 INFO [trainer.py:765] (0/8) Epoch 14, batch 500, train_loss[loss=3.68, NarTop10Accuracy=0.5938, over 6072.00 frames. ], tot_loss[loss=3.486, NarTop10Accuracy=0.6213, over 5406.98 frames. ], batch size: 11, lr: 6.20e-03
2024-08-06 09:44:47,166 INFO [trainer.py:765] (0/8) Epoch 14, batch 600, train_loss[loss=3.892, NarTop10Accuracy=0.5449, over 5947.00 frames. ], tot_loss[loss=3.485, NarTop10Accuracy=0.6211, over 5686.73 frames. ], batch size: 9, lr: 6.19e-03
2024-08-06 09:45:19,804 INFO [trainer.py:765] (0/8) Epoch 14, batch 700, train_loss[loss=3.498, NarTop10Accuracy=0.5997, over 5182.00 frames. ], tot_loss[loss=3.476, NarTop10Accuracy=0.6226, over 5744.37 frames. ], batch size: 6, lr: 6.18e-03
2024-08-06 09:45:58,435 INFO [trainer.py:765] (0/8) Epoch 14, batch 800, train_loss[loss=3.444, NarTop10Accuracy=0.6424, over 5044.00 frames. ], tot_loss[loss=3.489, NarTop10Accuracy=0.62, over 5784.84 frames. ], batch size: 6, lr: 6.17e-03
2024-08-06 09:46:35,420 INFO [trainer.py:765] (0/8) Epoch 14, batch 900, train_loss[loss=3.55, NarTop10Accuracy=0.6, over 6431.00 frames. ], tot_loss[loss=3.492, NarTop10Accuracy=0.6191, over 5819.81 frames. ], batch size: 13, lr: 6.17e-03
2024-08-06 09:47:08,399 INFO [trainer.py:765] (0/8) Epoch 14, batch 1000, train_loss[loss=3.646, NarTop10Accuracy=0.5753, over 6331.00 frames. ], tot_loss[loss=3.497, NarTop10Accuracy=0.6181, over 5920.45 frames. ], batch size: 13, lr: 6.16e-03
2024-08-06 09:47:47,663 INFO [trainer.py:765] (0/8) Epoch 14, batch 1100, train_loss[loss=3.444, NarTop10Accuracy=0.6291, over 6765.00 frames. ], tot_loss[loss=3.503, NarTop10Accuracy=0.6167, over 5960.46 frames. ], batch size: 17, lr: 6.15e-03
2024-08-06 09:48:23,500 INFO [trainer.py:765] (0/8) Epoch 14, batch 1200, train_loss[loss=3.414, NarTop10Accuracy=0.6362, over 7068.00 frames. ], tot_loss[loss=3.497, NarTop10Accuracy=0.6174, over 5963.53 frames. ], batch size: 30, lr: 6.14e-03
2024-08-06 09:48:57,972 INFO [trainer.py:765] (0/8) Epoch 14, batch 1300, train_loss[loss=3.465, NarTop10Accuracy=0.6307, over 5115.00 frames. ], tot_loss[loss=3.503, NarTop10Accuracy=0.6166, over 6021.94 frames. ], batch size: 6, lr: 6.13e-03
2024-08-06 09:49:30,234 INFO [trainer.py:765] (0/8) Epoch 14, batch 1400, train_loss[loss=3.328, NarTop10Accuracy=0.6395, over 6029.00 frames. ], tot_loss[loss=3.518, NarTop10Accuracy=0.613, over 6034.30 frames. ], batch size: 11, lr: 6.12e-03
2024-08-06 09:49:48,789 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-35000.pt
2024-08-06 09:50:07,531 INFO [trainer.py:765] (0/8) Epoch 14, batch 1500, train_loss[loss=3.598, NarTop10Accuracy=0.599, over 6214.00 frames. ], tot_loss[loss=3.516, NarTop10Accuracy=0.6131, over 5986.71 frames. ], batch size: 49, lr: 6.11e-03
2024-08-06 09:50:35,637 INFO [trainer.py:765] (0/8) Epoch 14, batch 1600, train_loss[loss=3.416, NarTop10Accuracy=0.6316, over 7212.00 frames. ], tot_loss[loss=3.51, NarTop10Accuracy=0.6149, over 5960.23 frames. ], batch size: 22, lr: 6.10e-03
2024-08-06 09:51:02,377 INFO [trainer.py:765] (0/8) Epoch 14, batch 1700, train_loss[loss=3.451, NarTop10Accuracy=0.6321, over 6309.00 frames. ], tot_loss[loss=3.515, NarTop10Accuracy=0.6138, over 5936.01 frames. ], batch size: 13, lr: 6.10e-03
2024-08-06 09:51:28,994 INFO [trainer.py:765] (0/8) Epoch 14, batch 1800, train_loss[loss=3.549, NarTop10Accuracy=0.6126, over 7063.00 frames. ], tot_loss[loss=3.508, NarTop10Accuracy=0.6161, over 5991.62 frames. ], batch size: 22, lr: 6.09e-03
2024-08-06 09:51:55,729 INFO [trainer.py:765] (0/8) Epoch 14, batch 1900, train_loss[loss=3.946, NarTop10Accuracy=0.5258, over 6080.00 frames. ], tot_loss[loss=3.52, NarTop10Accuracy=0.6135, over 6031.51 frames. ], batch size: 50, lr: 6.08e-03
2024-08-06 09:52:21,503 INFO [trainer.py:765] (0/8) Epoch 14, batch 2000, train_loss[loss=3.46, NarTop10Accuracy=0.6221, over 6257.00 frames. ], tot_loss[loss=3.533, NarTop10Accuracy=0.6108, over 6017.36 frames. ], batch size: 49, lr: 6.07e-03
2024-08-06 09:52:47,011 INFO [trainer.py:765] (0/8) Epoch 14, batch 2100, train_loss[loss=3.792, NarTop10Accuracy=0.5572, over 4802.00 frames. ], tot_loss[loss=3.52, NarTop10Accuracy=0.6135, over 6000.70 frames. ], batch size: 5, lr: 6.06e-03
2024-08-06 09:53:12,480 INFO [trainer.py:765] (0/8) Epoch 14, batch 2200, train_loss[loss=3.351, NarTop10Accuracy=0.6504, over 7229.00 frames. ], tot_loss[loss=3.524, NarTop10Accuracy=0.6127, over 6033.36 frames. ], batch size: 30, lr: 6.05e-03
2024-08-06 09:53:37,975 INFO [trainer.py:765] (0/8) Epoch 14, batch 2300, train_loss[loss=3.421, NarTop10Accuracy=0.636, over 5805.00 frames. ], tot_loss[loss=3.543, NarTop10Accuracy=0.6087, over 6069.19 frames. ], batch size: 9, lr: 6.05e-03
2024-08-06 09:54:02,717 INFO [trainer.py:765] (0/8) Epoch 14, batch 2400, train_loss[loss=3.645, NarTop10Accuracy=0.5886, over 5615.00 frames. ], tot_loss[loss=3.541, NarTop10Accuracy=0.6094, over 5882.68 frames. ], batch size: 48, lr: 6.04e-03
2024-08-06 09:54:12,820 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-36000.pt
2024-08-06 09:54:17,718 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 09:54:24,304 INFO [trainer.py:811] (0/8) Epoch 14, validation: loss=3.364, NarTop10Accuracy=0.6477, over 1907754.00 frames. 
2024-08-06 09:54:24,304 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30501MB
2024-08-06 09:54:24,752 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.448e+02 1.815e+02 1.970e+02 2.165e+02 3.684e+02, threshold=3.939e+02, percent-clipped=0.0
2024-08-06 09:54:37,619 INFO [trainer.py:765] (0/8) Epoch 14, batch 2500, train_loss[loss=3.98, NarTop10Accuracy=0.5167, over 4996.00 frames. ], tot_loss[loss=3.523, NarTop10Accuracy=0.6132, over 5547.93 frames. ], batch size: 6, lr: 6.03e-03
2024-08-06 09:54:58,850 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 09:54:58,855 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-14.pt
2024-08-06 09:56:03,097 INFO [trainer.py:765] (0/8) Epoch 15, batch 100, train_loss[loss=3.529, NarTop10Accuracy=0.6146, over 7427.00 frames. ], tot_loss[loss=3.466, NarTop10Accuracy=0.626, over 2376.08 frames. ], batch size: 31, lr: 5.81e-03
2024-08-06 09:56:35,980 INFO [trainer.py:765] (0/8) Epoch 15, batch 200, train_loss[loss=3.428, NarTop10Accuracy=0.6274, over 6801.00 frames. ], tot_loss[loss=3.459, NarTop10Accuracy=0.6266, over 3857.21 frames. ], batch size: 17, lr: 5.81e-03
2024-08-06 09:57:07,653 INFO [trainer.py:765] (0/8) Epoch 15, batch 300, train_loss[loss=3.309, NarTop10Accuracy=0.6577, over 7082.00 frames. ], tot_loss[loss=3.451, NarTop10Accuracy=0.6281, over 4673.92 frames. ], batch size: 22, lr: 5.80e-03
2024-08-06 09:57:38,463 INFO [trainer.py:765] (0/8) Epoch 15, batch 400, train_loss[loss=3.487, NarTop10Accuracy=0.627, over 5033.00 frames. ], tot_loss[loss=3.459, NarTop10Accuracy=0.6262, over 5111.59 frames. ], batch size: 7, lr: 5.79e-03
2024-08-06 09:58:12,235 INFO [trainer.py:765] (0/8) Epoch 15, batch 500, train_loss[loss=3.379, NarTop10Accuracy=0.6392, over 6220.00 frames. ], tot_loss[loss=3.483, NarTop10Accuracy=0.6211, over 5383.14 frames. ], batch size: 11, lr: 5.78e-03
2024-08-06 09:58:47,543 INFO [trainer.py:765] (0/8) Epoch 15, batch 600, train_loss[loss=3.812, NarTop10Accuracy=0.557, over 5637.00 frames. ], tot_loss[loss=3.484, NarTop10Accuracy=0.6211, over 5665.34 frames. ], batch size: 9, lr: 5.77e-03
2024-08-06 09:59:17,062 INFO [trainer.py:765] (0/8) Epoch 15, batch 700, train_loss[loss=3.323, NarTop10Accuracy=0.6453, over 5091.00 frames. ], tot_loss[loss=3.485, NarTop10Accuracy=0.6211, over 5725.67 frames. ], batch size: 6, lr: 5.77e-03
2024-08-06 09:59:55,588 INFO [trainer.py:765] (0/8) Epoch 15, batch 800, train_loss[loss=3.748, NarTop10Accuracy=0.5799, over 4994.00 frames. ], tot_loss[loss=3.485, NarTop10Accuracy=0.6214, over 5807.32 frames. ], batch size: 6, lr: 5.76e-03
2024-08-06 10:00:15,459 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-37000.pt
2024-08-06 10:00:32,024 INFO [trainer.py:765] (0/8) Epoch 15, batch 900, train_loss[loss=3.362, NarTop10Accuracy=0.6503, over 6519.00 frames. ], tot_loss[loss=3.478, NarTop10Accuracy=0.6225, over 5835.53 frames. ], batch size: 14, lr: 5.75e-03
2024-08-06 10:01:05,538 INFO [trainer.py:765] (0/8) Epoch 15, batch 1000, train_loss[loss=3.266, NarTop10Accuracy=0.6784, over 6267.00 frames. ], tot_loss[loss=3.479, NarTop10Accuracy=0.622, over 5934.89 frames. ], batch size: 13, lr: 5.74e-03
2024-08-06 10:01:45,154 INFO [trainer.py:765] (0/8) Epoch 15, batch 1100, train_loss[loss=3.423, NarTop10Accuracy=0.6252, over 6800.00 frames. ], tot_loss[loss=3.498, NarTop10Accuracy=0.6179, over 5963.03 frames. ], batch size: 17, lr: 5.74e-03
2024-08-06 10:02:18,756 INFO [trainer.py:765] (0/8) Epoch 15, batch 1200, train_loss[loss=3.7, NarTop10Accuracy=0.5779, over 7017.00 frames. ], tot_loss[loss=3.482, NarTop10Accuracy=0.6207, over 5945.27 frames. ], batch size: 30, lr: 5.73e-03
2024-08-06 10:02:51,921 INFO [trainer.py:765] (0/8) Epoch 15, batch 1300, train_loss[loss=3.459, NarTop10Accuracy=0.6122, over 4240.00 frames. ], tot_loss[loss=3.482, NarTop10Accuracy=0.6207, over 6019.87 frames. ], batch size: 5, lr: 5.72e-03
2024-08-06 10:03:25,436 INFO [trainer.py:765] (0/8) Epoch 15, batch 1400, train_loss[loss=3.576, NarTop10Accuracy=0.5972, over 6134.00 frames. ], tot_loss[loss=3.495, NarTop10Accuracy=0.6181, over 6037.85 frames. ], batch size: 11, lr: 5.71e-03
2024-08-06 10:03:59,042 INFO [trainer.py:765] (0/8) Epoch 15, batch 1500, train_loss[loss=3.553, NarTop10Accuracy=0.6034, over 6214.00 frames. ], tot_loss[loss=3.49, NarTop10Accuracy=0.619, over 5966.51 frames. ], batch size: 48, lr: 5.71e-03
2024-08-06 10:04:27,106 INFO [trainer.py:765] (0/8) Epoch 15, batch 1600, train_loss[loss=3.741, NarTop10Accuracy=0.5745, over 7154.00 frames. ], tot_loss[loss=3.483, NarTop10Accuracy=0.6212, over 5952.21 frames. ], batch size: 22, lr: 5.70e-03
2024-08-06 10:04:53,907 INFO [trainer.py:765] (0/8) Epoch 15, batch 1700, train_loss[loss=3.691, NarTop10Accuracy=0.5815, over 6239.00 frames. ], tot_loss[loss=3.488, NarTop10Accuracy=0.6204, over 5942.69 frames. ], batch size: 13, lr: 5.69e-03
2024-08-06 10:05:20,728 INFO [trainer.py:765] (0/8) Epoch 15, batch 1800, train_loss[loss=3.692, NarTop10Accuracy=0.5816, over 7202.00 frames. ], tot_loss[loss=3.507, NarTop10Accuracy=0.6163, over 6005.64 frames. ], batch size: 22, lr: 5.68e-03
2024-08-06 10:05:37,265 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-38000.pt
2024-08-06 10:05:40,780 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 10:05:47,411 INFO [trainer.py:811] (0/8) Epoch 15, validation: loss=3.325, NarTop10Accuracy=0.6551, over 1907754.00 frames. 
2024-08-06 10:05:47,412 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30501MB
2024-08-06 10:05:47,920 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.405e+02 1.835e+02 1.986e+02 2.156e+02 4.531e+02, threshold=3.972e+02, percent-clipped=0.1
2024-08-06 10:05:57,568 INFO [trainer.py:765] (0/8) Epoch 15, batch 1900, train_loss[loss=3.715, NarTop10Accuracy=0.5751, over 6884.00 frames. ], tot_loss[loss=3.516, NarTop10Accuracy=0.6146, over 6047.41 frames. ], batch size: 49, lr: 5.68e-03
2024-08-06 10:06:23,372 INFO [trainer.py:765] (0/8) Epoch 15, batch 2000, train_loss[loss=3.55, NarTop10Accuracy=0.604, over 6537.00 frames. ], tot_loss[loss=3.51, NarTop10Accuracy=0.6157, over 6018.00 frames. ], batch size: 49, lr: 5.67e-03
2024-08-06 10:06:48,759 INFO [trainer.py:765] (0/8) Epoch 15, batch 2100, train_loss[loss=3.072, NarTop10Accuracy=0.6891, over 3840.00 frames. ], tot_loss[loss=3.503, NarTop10Accuracy=0.6166, over 5990.47 frames. ], batch size: 4, lr: 5.66e-03
2024-08-06 10:07:14,170 INFO [trainer.py:765] (0/8) Epoch 15, batch 2200, train_loss[loss=3.347, NarTop10Accuracy=0.6483, over 7171.00 frames. ], tot_loss[loss=3.497, NarTop10Accuracy=0.6182, over 6033.65 frames. ], batch size: 30, lr: 5.65e-03
2024-08-06 10:07:39,629 INFO [trainer.py:765] (0/8) Epoch 15, batch 2300, train_loss[loss=3.223, NarTop10Accuracy=0.665, over 5819.00 frames. ], tot_loss[loss=3.508, NarTop10Accuracy=0.616, over 6054.65 frames. ], batch size: 9, lr: 5.65e-03
2024-08-06 10:08:04,361 INFO [trainer.py:765] (0/8) Epoch 15, batch 2400, train_loss[loss=3.893, NarTop10Accuracy=0.529, over 6052.00 frames. ], tot_loss[loss=3.514, NarTop10Accuracy=0.6148, over 5874.77 frames. ], batch size: 49, lr: 5.64e-03
2024-08-06 10:08:27,713 INFO [trainer.py:765] (0/8) Epoch 15, batch 2500, train_loss[loss=3.534, NarTop10Accuracy=0.6112, over 4969.00 frames. ], tot_loss[loss=3.494, NarTop10Accuracy=0.6184, over 5530.44 frames. ], batch size: 6, lr: 5.63e-03
2024-08-06 10:08:49,186 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 10:08:49,192 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-15.pt
2024-08-06 10:09:44,183 INFO [trainer.py:765] (0/8) Epoch 16, batch 100, train_loss[loss=3.668, NarTop10Accuracy=0.5889, over 7489.00 frames. ], tot_loss[loss=3.463, NarTop10Accuracy=0.6261, over 2375.49 frames. ], batch size: 31, lr: 5.44e-03
2024-08-06 10:10:23,207 INFO [trainer.py:765] (0/8) Epoch 16, batch 200, train_loss[loss=3.358, NarTop10Accuracy=0.6581, over 6865.00 frames. ], tot_loss[loss=3.44, NarTop10Accuracy=0.6305, over 3882.41 frames. ], batch size: 17, lr: 5.44e-03
2024-08-06 10:10:50,262 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-39000.pt
2024-08-06 10:10:58,841 INFO [trainer.py:765] (0/8) Epoch 16, batch 300, train_loss[loss=3.245, NarTop10Accuracy=0.6709, over 7048.00 frames. ], tot_loss[loss=3.446, NarTop10Accuracy=0.6287, over 4696.14 frames. ], batch size: 22, lr: 5.43e-03
2024-08-06 10:11:29,595 INFO [trainer.py:765] (0/8) Epoch 16, batch 400, train_loss[loss=3.4, NarTop10Accuracy=0.6404, over 5705.00 frames. ], tot_loss[loss=3.447, NarTop10Accuracy=0.6284, over 5133.20 frames. ], batch size: 8, lr: 5.42e-03
2024-08-06 10:12:02,298 INFO [trainer.py:765] (0/8) Epoch 16, batch 500, train_loss[loss=3.76, NarTop10Accuracy=0.5652, over 6117.00 frames. ], tot_loss[loss=3.445, NarTop10Accuracy=0.6291, over 5431.10 frames. ], batch size: 11, lr: 5.42e-03
2024-08-06 10:12:42,340 INFO [trainer.py:765] (0/8) Epoch 16, batch 600, train_loss[loss=3.571, NarTop10Accuracy=0.6137, over 5864.00 frames. ], tot_loss[loss=3.439, NarTop10Accuracy=0.63, over 5698.16 frames. ], batch size: 9, lr: 5.41e-03
2024-08-06 10:13:13,952 INFO [trainer.py:765] (0/8) Epoch 16, batch 700, train_loss[loss=3.169, NarTop10Accuracy=0.6738, over 4974.00 frames. ], tot_loss[loss=3.449, NarTop10Accuracy=0.6279, over 5778.31 frames. ], batch size: 6, lr: 5.40e-03
2024-08-06 10:13:46,285 INFO [trainer.py:765] (0/8) Epoch 16, batch 800, train_loss[loss=3.663, NarTop10Accuracy=0.5856, over 4967.00 frames. ], tot_loss[loss=3.444, NarTop10Accuracy=0.6286, over 5816.67 frames. ], batch size: 6, lr: 5.40e-03
2024-08-06 10:14:23,296 INFO [trainer.py:765] (0/8) Epoch 16, batch 900, train_loss[loss=3.329, NarTop10Accuracy=0.6512, over 6170.00 frames. ], tot_loss[loss=3.445, NarTop10Accuracy=0.6286, over 5833.40 frames. ], batch size: 13, lr: 5.39e-03
2024-08-06 10:15:00,059 INFO [trainer.py:765] (0/8) Epoch 16, batch 1000, train_loss[loss=3.688, NarTop10Accuracy=0.5727, over 6362.00 frames. ], tot_loss[loss=3.467, NarTop10Accuracy=0.6239, over 5947.70 frames. ], batch size: 13, lr: 5.38e-03
2024-08-06 10:15:30,509 INFO [trainer.py:765] (0/8) Epoch 16, batch 1100, train_loss[loss=3.456, NarTop10Accuracy=0.6299, over 6922.00 frames. ], tot_loss[loss=3.479, NarTop10Accuracy=0.6212, over 5977.35 frames. ], batch size: 17, lr: 5.38e-03
2024-08-06 10:16:11,384 INFO [trainer.py:765] (0/8) Epoch 16, batch 1200, train_loss[loss=3.496, NarTop10Accuracy=0.6262, over 7283.00 frames. ], tot_loss[loss=3.474, NarTop10Accuracy=0.6221, over 5962.36 frames. ], batch size: 31, lr: 5.37e-03
2024-08-06 10:16:39,396 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-40000.pt
2024-08-06 10:16:42,816 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 10:16:49,676 INFO [trainer.py:811] (0/8) Epoch 16, validation: loss=3.375, NarTop10Accuracy=0.6455, over 1907754.00 frames. 
2024-08-06 10:16:49,676 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30501MB
2024-08-06 10:16:52,482 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.406e+02 1.814e+02 1.975e+02 2.151e+02 4.776e+02, threshold=3.950e+02, percent-clipped=0.2
2024-08-06 10:16:58,042 INFO [trainer.py:765] (0/8) Epoch 16, batch 1300, train_loss[loss=3.847, NarTop10Accuracy=0.5535, over 4978.00 frames. ], tot_loss[loss=3.466, NarTop10Accuracy=0.6233, over 6018.96 frames. ], batch size: 6, lr: 5.36e-03
2024-08-06 10:17:29,376 INFO [trainer.py:765] (0/8) Epoch 16, batch 1400, train_loss[loss=3.443, NarTop10Accuracy=0.6378, over 6184.00 frames. ], tot_loss[loss=3.473, NarTop10Accuracy=0.6223, over 6028.72 frames. ], batch size: 11, lr: 5.36e-03
2024-08-06 10:18:02,354 INFO [trainer.py:765] (0/8) Epoch 16, batch 1500, train_loss[loss=3.562, NarTop10Accuracy=0.6162, over 6475.00 frames. ], tot_loss[loss=3.475, NarTop10Accuracy=0.622, over 5961.91 frames. ], batch size: 48, lr: 5.35e-03
2024-08-06 10:18:30,469 INFO [trainer.py:765] (0/8) Epoch 16, batch 1600, train_loss[loss=3.638, NarTop10Accuracy=0.5969, over 7153.00 frames. ], tot_loss[loss=3.488, NarTop10Accuracy=0.6194, over 5946.03 frames. ], batch size: 22, lr: 5.34e-03
2024-08-06 10:18:57,273 INFO [trainer.py:765] (0/8) Epoch 16, batch 1700, train_loss[loss=3.756, NarTop10Accuracy=0.5646, over 6343.00 frames. ], tot_loss[loss=3.476, NarTop10Accuracy=0.6219, over 5940.11 frames. ], batch size: 13, lr: 5.34e-03
2024-08-06 10:19:23,979 INFO [trainer.py:765] (0/8) Epoch 16, batch 1800, train_loss[loss=3.677, NarTop10Accuracy=0.5763, over 7114.00 frames. ], tot_loss[loss=3.491, NarTop10Accuracy=0.6192, over 5995.30 frames. ], batch size: 22, lr: 5.33e-03
2024-08-06 10:19:50,773 INFO [trainer.py:765] (0/8) Epoch 16, batch 1900, train_loss[loss=3.752, NarTop10Accuracy=0.5707, over 5887.00 frames. ], tot_loss[loss=3.494, NarTop10Accuracy=0.6191, over 6022.56 frames. ], batch size: 49, lr: 5.32e-03
2024-08-06 10:20:16,602 INFO [trainer.py:765] (0/8) Epoch 16, batch 2000, train_loss[loss=3.525, NarTop10Accuracy=0.6033, over 6009.00 frames. ], tot_loss[loss=3.504, NarTop10Accuracy=0.6166, over 6022.59 frames. ], batch size: 49, lr: 5.32e-03
2024-08-06 10:20:42,160 INFO [trainer.py:765] (0/8) Epoch 16, batch 2100, train_loss[loss=3.462, NarTop10Accuracy=0.6013, over 3890.00 frames. ], tot_loss[loss=3.515, NarTop10Accuracy=0.6139, over 6007.81 frames. ], batch size: 4, lr: 5.31e-03
2024-08-06 10:21:07,651 INFO [trainer.py:765] (0/8) Epoch 16, batch 2200, train_loss[loss=3.365, NarTop10Accuracy=0.6478, over 7039.00 frames. ], tot_loss[loss=3.503, NarTop10Accuracy=0.6166, over 6047.01 frames. ], batch size: 30, lr: 5.30e-03
2024-08-06 10:21:28,124 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-41000.pt
2024-08-06 10:21:36,082 INFO [trainer.py:765] (0/8) Epoch 16, batch 2300, train_loss[loss=3.444, NarTop10Accuracy=0.624, over 5730.00 frames. ], tot_loss[loss=3.509, NarTop10Accuracy=0.6153, over 6064.13 frames. ], batch size: 9, lr: 5.30e-03
2024-08-06 10:22:00,907 INFO [trainer.py:765] (0/8) Epoch 16, batch 2400, train_loss[loss=3.551, NarTop10Accuracy=0.6158, over 6305.00 frames. ], tot_loss[loss=3.503, NarTop10Accuracy=0.6163, over 5898.09 frames. ], batch size: 48, lr: 5.29e-03
2024-08-06 10:22:24,290 INFO [trainer.py:765] (0/8) Epoch 16, batch 2500, train_loss[loss=3.423, NarTop10Accuracy=0.6276, over 5252.00 frames. ], tot_loss[loss=3.479, NarTop10Accuracy=0.6206, over 5533.26 frames. ], batch size: 6, lr: 5.28e-03
2024-08-06 10:22:45,882 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 10:22:45,887 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-16.pt
2024-08-06 10:23:45,727 INFO [trainer.py:765] (0/8) Epoch 17, batch 100, train_loss[loss=3.439, NarTop10Accuracy=0.6323, over 7240.00 frames. ], tot_loss[loss=3.439, NarTop10Accuracy=0.6326, over 2373.95 frames. ], batch size: 30, lr: 5.12e-03
2024-08-06 10:24:19,034 INFO [trainer.py:765] (0/8) Epoch 17, batch 200, train_loss[loss=3.294, NarTop10Accuracy=0.6687, over 6910.00 frames. ], tot_loss[loss=3.438, NarTop10Accuracy=0.6325, over 3864.63 frames. ], batch size: 17, lr: 5.11e-03
2024-08-06 10:24:53,441 INFO [trainer.py:765] (0/8) Epoch 17, batch 300, train_loss[loss=3.667, NarTop10Accuracy=0.5923, over 7201.00 frames. ], tot_loss[loss=3.438, NarTop10Accuracy=0.632, over 4686.93 frames. ], batch size: 22, lr: 5.10e-03
2024-08-06 10:25:28,013 INFO [trainer.py:765] (0/8) Epoch 17, batch 400, train_loss[loss=3.697, NarTop10Accuracy=0.5677, over 5258.00 frames. ], tot_loss[loss=3.437, NarTop10Accuracy=0.6316, over 5135.09 frames. ], batch size: 7, lr: 5.10e-03
2024-08-06 10:25:58,606 INFO [trainer.py:765] (0/8) Epoch 17, batch 500, train_loss[loss=3.525, NarTop10Accuracy=0.6251, over 6124.00 frames. ], tot_loss[loss=3.437, NarTop10Accuracy=0.6311, over 5407.05 frames. ], batch size: 11, lr: 5.09e-03
2024-08-06 10:26:29,756 INFO [trainer.py:765] (0/8) Epoch 17, batch 600, train_loss[loss=3.845, NarTop10Accuracy=0.5448, over 5773.00 frames. ], tot_loss[loss=3.431, NarTop10Accuracy=0.6319, over 5680.41 frames. ], batch size: 9, lr: 5.09e-03
2024-08-06 10:27:07,499 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-42000.pt
2024-08-06 10:27:11,062 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 10:27:17,547 INFO [trainer.py:811] (0/8) Epoch 17, validation: loss=3.327, NarTop10Accuracy=0.6554, over 1907754.00 frames. 
2024-08-06 10:27:17,548 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30501MB
2024-08-06 10:27:18,066 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.474e+02 1.825e+02 1.985e+02 2.150e+02 4.169e+02, threshold=3.970e+02, percent-clipped=0.2
2024-08-06 10:27:18,072 INFO [trainer.py:765] (0/8) Epoch 17, batch 700, train_loss[loss=2.996, NarTop10Accuracy=0.724, over 5076.00 frames. ], tot_loss[loss=3.453, NarTop10Accuracy=0.6272, over 5733.93 frames. ], batch size: 6, lr: 5.08e-03
2024-08-06 10:27:49,841 INFO [trainer.py:765] (0/8) Epoch 17, batch 800, train_loss[loss=3.461, NarTop10Accuracy=0.6324, over 5069.00 frames. ], tot_loss[loss=3.448, NarTop10Accuracy=0.6282, over 5790.92 frames. ], batch size: 6, lr: 5.07e-03
2024-08-06 10:28:24,839 INFO [trainer.py:765] (0/8) Epoch 17, batch 900, train_loss[loss=3.388, NarTop10Accuracy=0.6421, over 6229.00 frames. ], tot_loss[loss=3.448, NarTop10Accuracy=0.628, over 5806.67 frames. ], batch size: 13, lr: 5.07e-03
2024-08-06 10:28:59,684 INFO [trainer.py:765] (0/8) Epoch 17, batch 1000, train_loss[loss=3.279, NarTop10Accuracy=0.6637, over 6228.00 frames. ], tot_loss[loss=3.442, NarTop10Accuracy=0.6294, over 5916.68 frames. ], batch size: 13, lr: 5.06e-03
2024-08-06 10:29:36,659 INFO [trainer.py:765] (0/8) Epoch 17, batch 1100, train_loss[loss=3.225, NarTop10Accuracy=0.6719, over 6796.00 frames. ], tot_loss[loss=3.452, NarTop10Accuracy=0.627, over 5942.04 frames. ], batch size: 17, lr: 5.06e-03
2024-08-06 10:30:08,242 INFO [trainer.py:765] (0/8) Epoch 17, batch 1200, train_loss[loss=3.618, NarTop10Accuracy=0.6015, over 7032.00 frames. ], tot_loss[loss=3.441, NarTop10Accuracy=0.6289, over 5962.02 frames. ], batch size: 30, lr: 5.05e-03
2024-08-06 10:30:47,102 INFO [trainer.py:765] (0/8) Epoch 17, batch 1300, train_loss[loss=3.494, NarTop10Accuracy=0.62, over 5159.00 frames. ], tot_loss[loss=3.451, NarTop10Accuracy=0.6271, over 6030.65 frames. ], batch size: 6, lr: 5.04e-03
2024-08-06 10:31:20,894 INFO [trainer.py:765] (0/8) Epoch 17, batch 1400, train_loss[loss=3.443, NarTop10Accuracy=0.6382, over 6286.00 frames. ], tot_loss[loss=3.459, NarTop10Accuracy=0.6256, over 6042.74 frames. ], batch size: 11, lr: 5.04e-03
2024-08-06 10:31:51,401 INFO [trainer.py:765] (0/8) Epoch 17, batch 1500, train_loss[loss=3.549, NarTop10Accuracy=0.6116, over 6618.00 frames. ], tot_loss[loss=3.454, NarTop10Accuracy=0.6271, over 5998.72 frames. ], batch size: 49, lr: 5.03e-03
2024-08-06 10:32:19,401 INFO [trainer.py:765] (0/8) Epoch 17, batch 1600, train_loss[loss=3.577, NarTop10Accuracy=0.6047, over 7263.00 frames. ], tot_loss[loss=3.461, NarTop10Accuracy=0.6251, over 5979.40 frames. ], batch size: 22, lr: 5.03e-03
2024-08-06 10:32:45,660 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-43000.pt
2024-08-06 10:32:50,394 INFO [trainer.py:765] (0/8) Epoch 17, batch 1700, train_loss[loss=3.65, NarTop10Accuracy=0.5822, over 6639.00 frames. ], tot_loss[loss=3.481, NarTop10Accuracy=0.621, over 5970.67 frames. ], batch size: 14, lr: 5.02e-03
2024-08-06 10:33:17,035 INFO [trainer.py:765] (0/8) Epoch 17, batch 1800, train_loss[loss=3.757, NarTop10Accuracy=0.5539, over 7084.00 frames. ], tot_loss[loss=3.492, NarTop10Accuracy=0.6191, over 6015.86 frames. ], batch size: 22, lr: 5.02e-03
2024-08-06 10:33:43,597 INFO [trainer.py:765] (0/8) Epoch 17, batch 1900, train_loss[loss=3.811, NarTop10Accuracy=0.542, over 5954.00 frames. ], tot_loss[loss=3.487, NarTop10Accuracy=0.6208, over 6033.53 frames. ], batch size: 48, lr: 5.01e-03
2024-08-06 10:34:09,287 INFO [trainer.py:765] (0/8) Epoch 17, batch 2000, train_loss[loss=3.835, NarTop10Accuracy=0.5499, over 5625.00 frames. ], tot_loss[loss=3.489, NarTop10Accuracy=0.6204, over 6015.20 frames. ], batch size: 49, lr: 5.00e-03
2024-08-06 10:34:34,802 INFO [trainer.py:765] (0/8) Epoch 17, batch 2100, train_loss[loss=3.389, NarTop10Accuracy=0.6289, over 3953.00 frames. ], tot_loss[loss=3.497, NarTop10Accuracy=0.6184, over 6005.50 frames. ], batch size: 4, lr: 5.00e-03
2024-08-06 10:35:00,245 INFO [trainer.py:765] (0/8) Epoch 17, batch 2200, train_loss[loss=3.462, NarTop10Accuracy=0.626, over 7251.00 frames. ], tot_loss[loss=3.477, NarTop10Accuracy=0.6225, over 6043.42 frames. ], batch size: 31, lr: 4.99e-03
2024-08-06 10:35:25,732 INFO [trainer.py:765] (0/8) Epoch 17, batch 2300, train_loss[loss=3.174, NarTop10Accuracy=0.6803, over 5822.00 frames. ], tot_loss[loss=3.485, NarTop10Accuracy=0.6208, over 6073.63 frames. ], batch size: 9, lr: 4.99e-03
2024-08-06 10:35:50,526 INFO [trainer.py:765] (0/8) Epoch 17, batch 2400, train_loss[loss=3.817, NarTop10Accuracy=0.5418, over 6006.00 frames. ], tot_loss[loss=3.497, NarTop10Accuracy=0.6182, over 5893.09 frames. ], batch size: 48, lr: 4.98e-03
2024-08-06 10:36:14,105 INFO [trainer.py:765] (0/8) Epoch 17, batch 2500, train_loss[loss=3.472, NarTop10Accuracy=0.6091, over 5018.00 frames. ], tot_loss[loss=3.473, NarTop10Accuracy=0.6224, over 5563.06 frames. ], batch size: 6, lr: 4.98e-03
2024-08-06 10:36:35,891 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 10:36:35,894 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-17.pt
2024-08-06 10:37:32,052 INFO [trainer.py:765] (0/8) Epoch 18, batch 100, train_loss[loss=3.262, NarTop10Accuracy=0.6699, over 7516.00 frames. ], tot_loss[loss=3.431, NarTop10Accuracy=0.6318, over 2383.57 frames. ], batch size: 30, lr: 4.83e-03
2024-08-06 10:37:39,162 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-44000.pt
2024-08-06 10:37:42,569 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 10:37:49,085 INFO [trainer.py:811] (0/8) Epoch 18, validation: loss=3.339, NarTop10Accuracy=0.6526, over 1907754.00 frames. 
2024-08-06 10:37:49,085 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30501MB
2024-08-06 10:37:49,685 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.466e+02 1.841e+02 1.993e+02 2.161e+02 3.871e+02, threshold=3.985e+02, percent-clipped=0.0
2024-08-06 10:38:18,145 INFO [trainer.py:765] (0/8) Epoch 18, batch 200, train_loss[loss=3.517, NarTop10Accuracy=0.61, over 6723.00 frames. ], tot_loss[loss=3.427, NarTop10Accuracy=0.633, over 3879.98 frames. ], batch size: 17, lr: 4.82e-03
2024-08-06 10:38:50,199 INFO [trainer.py:765] (0/8) Epoch 18, batch 300, train_loss[loss=3.41, NarTop10Accuracy=0.6343, over 7139.00 frames. ], tot_loss[loss=3.416, NarTop10Accuracy=0.6351, over 4677.63 frames. ], batch size: 22, lr: 4.81e-03
2024-08-06 10:39:23,743 INFO [trainer.py:765] (0/8) Epoch 18, batch 400, train_loss[loss=3.476, NarTop10Accuracy=0.6295, over 5119.00 frames. ], tot_loss[loss=3.413, NarTop10Accuracy=0.6357, over 5127.61 frames. ], batch size: 7, lr: 4.81e-03
2024-08-06 10:39:54,103 INFO [trainer.py:765] (0/8) Epoch 18, batch 500, train_loss[loss=3.267, NarTop10Accuracy=0.6854, over 6211.00 frames. ], tot_loss[loss=3.408, NarTop10Accuracy=0.6364, over 5412.72 frames. ], batch size: 11, lr: 4.80e-03
2024-08-06 10:40:28,527 INFO [trainer.py:765] (0/8) Epoch 18, batch 600, train_loss[loss=3.565, NarTop10Accuracy=0.6119, over 5777.00 frames. ], tot_loss[loss=3.415, NarTop10Accuracy=0.6355, over 5671.23 frames. ], batch size: 9, lr: 4.80e-03
2024-08-06 10:41:02,143 INFO [trainer.py:765] (0/8) Epoch 18, batch 700, train_loss[loss=3.165, NarTop10Accuracy=0.6882, over 4977.00 frames. ], tot_loss[loss=3.431, NarTop10Accuracy=0.6322, over 5744.77 frames. ], batch size: 6, lr: 4.79e-03
2024-08-06 10:41:38,519 INFO [trainer.py:765] (0/8) Epoch 18, batch 800, train_loss[loss=3.381, NarTop10Accuracy=0.656, over 5088.00 frames. ], tot_loss[loss=3.428, NarTop10Accuracy=0.6326, over 5809.16 frames. ], batch size: 6, lr: 4.79e-03
2024-08-06 10:42:12,611 INFO [trainer.py:765] (0/8) Epoch 18, batch 900, train_loss[loss=3.583, NarTop10Accuracy=0.6133, over 6139.00 frames. ], tot_loss[loss=3.435, NarTop10Accuracy=0.6308, over 5827.63 frames. ], batch size: 13, lr: 4.78e-03
2024-08-06 10:42:46,703 INFO [trainer.py:765] (0/8) Epoch 18, batch 1000, train_loss[loss=3.04, NarTop10Accuracy=0.7079, over 6275.00 frames. ], tot_loss[loss=3.433, NarTop10Accuracy=0.6312, over 5917.41 frames. ], batch size: 13, lr: 4.78e-03
2024-08-06 10:43:24,183 INFO [trainer.py:765] (0/8) Epoch 18, batch 1100, train_loss[loss=3.832, NarTop10Accuracy=0.5548, over 6851.00 frames. ], tot_loss[loss=3.466, NarTop10Accuracy=0.6246, over 5960.72 frames. ], batch size: 17, lr: 4.77e-03
2024-08-06 10:43:28,871 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-45000.pt
2024-08-06 10:44:02,363 INFO [trainer.py:765] (0/8) Epoch 18, batch 1200, train_loss[loss=3.447, NarTop10Accuracy=0.616, over 7430.00 frames. ], tot_loss[loss=3.451, NarTop10Accuracy=0.6273, over 5953.65 frames. ], batch size: 31, lr: 4.77e-03
2024-08-06 10:44:35,919 INFO [trainer.py:765] (0/8) Epoch 18, batch 1300, train_loss[loss=3.222, NarTop10Accuracy=0.6733, over 5074.00 frames. ], tot_loss[loss=3.443, NarTop10Accuracy=0.6285, over 6034.54 frames. ], batch size: 6, lr: 4.76e-03
2024-08-06 10:45:10,238 INFO [trainer.py:765] (0/8) Epoch 18, batch 1400, train_loss[loss=3.506, NarTop10Accuracy=0.6238, over 6010.00 frames. ], tot_loss[loss=3.458, NarTop10Accuracy=0.6258, over 6052.73 frames. ], batch size: 11, lr: 4.76e-03
2024-08-06 10:45:40,976 INFO [trainer.py:765] (0/8) Epoch 18, batch 1500, train_loss[loss=3.797, NarTop10Accuracy=0.5625, over 6089.00 frames. ], tot_loss[loss=3.462, NarTop10Accuracy=0.6242, over 5980.65 frames. ], batch size: 49, lr: 4.75e-03
2024-08-06 10:46:09,055 INFO [trainer.py:765] (0/8) Epoch 18, batch 1600, train_loss[loss=3.339, NarTop10Accuracy=0.652, over 7037.00 frames. ], tot_loss[loss=3.472, NarTop10Accuracy=0.6226, over 5970.39 frames. ], batch size: 22, lr: 4.75e-03
2024-08-06 10:46:35,858 INFO [trainer.py:765] (0/8) Epoch 18, batch 1700, train_loss[loss=3.581, NarTop10Accuracy=0.5917, over 6235.00 frames. ], tot_loss[loss=3.462, NarTop10Accuracy=0.6246, over 5951.66 frames. ], batch size: 13, lr: 4.74e-03
2024-08-06 10:47:02,438 INFO [trainer.py:765] (0/8) Epoch 18, batch 1800, train_loss[loss=3.524, NarTop10Accuracy=0.6118, over 7181.00 frames. ], tot_loss[loss=3.476, NarTop10Accuracy=0.6225, over 6020.04 frames. ], batch size: 22, lr: 4.74e-03
2024-08-06 10:47:29,093 INFO [trainer.py:765] (0/8) Epoch 18, batch 1900, train_loss[loss=3.679, NarTop10Accuracy=0.5866, over 6421.00 frames. ], tot_loss[loss=3.478, NarTop10Accuracy=0.6217, over 6059.00 frames. ], batch size: 49, lr: 4.73e-03
2024-08-06 10:47:54,884 INFO [trainer.py:765] (0/8) Epoch 18, batch 2000, train_loss[loss=3.499, NarTop10Accuracy=0.6279, over 6282.00 frames. ], tot_loss[loss=3.483, NarTop10Accuracy=0.6211, over 6024.30 frames. ], batch size: 51, lr: 4.73e-03
2024-08-06 10:48:20,370 INFO [trainer.py:765] (0/8) Epoch 18, batch 2100, train_loss[loss=3.435, NarTop10Accuracy=0.6313, over 3984.00 frames. ], tot_loss[loss=3.48, NarTop10Accuracy=0.6217, over 6006.34 frames. ], batch size: 4, lr: 4.72e-03
2024-08-06 10:48:24,747 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-46000.pt
2024-08-06 10:48:28,374 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 10:48:35,039 INFO [trainer.py:811] (0/8) Epoch 18, validation: loss=3.307, NarTop10Accuracy=0.6593, over 1907754.00 frames. 
2024-08-06 10:48:35,040 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30501MB
2024-08-06 10:48:35,535 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.484e+02 1.855e+02 2.003e+02 2.193e+02 3.481e+02, threshold=4.005e+02, percent-clipped=0.0
2024-08-06 10:48:56,096 INFO [trainer.py:765] (0/8) Epoch 18, batch 2200, train_loss[loss=3.309, NarTop10Accuracy=0.6538, over 7137.00 frames. ], tot_loss[loss=3.468, NarTop10Accuracy=0.624, over 6040.26 frames. ], batch size: 30, lr: 4.72e-03
2024-08-06 10:49:21,521 INFO [trainer.py:765] (0/8) Epoch 18, batch 2300, train_loss[loss=3.298, NarTop10Accuracy=0.6608, over 5732.00 frames. ], tot_loss[loss=3.472, NarTop10Accuracy=0.6231, over 6082.50 frames. ], batch size: 9, lr: 4.71e-03
2024-08-06 10:49:46,256 INFO [trainer.py:765] (0/8) Epoch 18, batch 2400, train_loss[loss=3.499, NarTop10Accuracy=0.62, over 6395.00 frames. ], tot_loss[loss=3.466, NarTop10Accuracy=0.6237, over 5899.71 frames. ], batch size: 50, lr: 4.71e-03
2024-08-06 10:50:09,708 INFO [trainer.py:765] (0/8) Epoch 18, batch 2500, train_loss[loss=3.239, NarTop10Accuracy=0.6769, over 5079.00 frames. ], tot_loss[loss=3.45, NarTop10Accuracy=0.6267, over 5532.26 frames. ], batch size: 6, lr: 4.70e-03
2024-08-06 10:50:31,270 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 10:50:31,273 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-18.pt
2024-08-06 10:51:33,564 INFO [trainer.py:765] (0/8) Epoch 19, batch 100, train_loss[loss=3.38, NarTop10Accuracy=0.6538, over 7105.00 frames. ], tot_loss[loss=3.397, NarTop10Accuracy=0.6395, over 2361.09 frames. ], batch size: 30, lr: 4.57e-03
2024-08-06 10:52:06,164 INFO [trainer.py:765] (0/8) Epoch 19, batch 200, train_loss[loss=3.632, NarTop10Accuracy=0.5859, over 7006.00 frames. ], tot_loss[loss=3.401, NarTop10Accuracy=0.6389, over 3860.72 frames. ], batch size: 17, lr: 4.56e-03
2024-08-06 10:52:40,031 INFO [trainer.py:765] (0/8) Epoch 19, batch 300, train_loss[loss=3.55, NarTop10Accuracy=0.6134, over 7128.00 frames. ], tot_loss[loss=3.405, NarTop10Accuracy=0.6389, over 4670.97 frames. ], batch size: 22, lr: 4.56e-03
2024-08-06 10:53:12,830 INFO [trainer.py:765] (0/8) Epoch 19, batch 400, train_loss[loss=3.177, NarTop10Accuracy=0.6894, over 5026.00 frames. ], tot_loss[loss=3.411, NarTop10Accuracy=0.6364, over 5131.23 frames. ], batch size: 7, lr: 4.55e-03
2024-08-06 10:53:45,020 INFO [trainer.py:765] (0/8) Epoch 19, batch 500, train_loss[loss=3.365, NarTop10Accuracy=0.648, over 6234.00 frames. ], tot_loss[loss=3.406, NarTop10Accuracy=0.637, over 5398.92 frames. ], batch size: 11, lr: 4.55e-03
2024-08-06 10:53:55,100 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-47000.pt
2024-08-06 10:54:18,600 INFO [trainer.py:765] (0/8) Epoch 19, batch 600, train_loss[loss=3.235, NarTop10Accuracy=0.6758, over 5866.00 frames. ], tot_loss[loss=3.406, NarTop10Accuracy=0.6368, over 5668.87 frames. ], batch size: 9, lr: 4.54e-03
2024-08-06 10:54:54,112 INFO [trainer.py:765] (0/8) Epoch 19, batch 700, train_loss[loss=3.407, NarTop10Accuracy=0.6418, over 5124.00 frames. ], tot_loss[loss=3.421, NarTop10Accuracy=0.634, over 5745.07 frames. ], batch size: 6, lr: 4.54e-03
2024-08-06 10:55:29,925 INFO [trainer.py:765] (0/8) Epoch 19, batch 800, train_loss[loss=3.293, NarTop10Accuracy=0.6604, over 5143.00 frames. ], tot_loss[loss=3.421, NarTop10Accuracy=0.634, over 5787.54 frames. ], batch size: 6, lr: 4.53e-03
2024-08-06 10:56:02,239 INFO [trainer.py:765] (0/8) Epoch 19, batch 900, train_loss[loss=3.489, NarTop10Accuracy=0.619, over 6404.00 frames. ], tot_loss[loss=3.421, NarTop10Accuracy=0.6339, over 5801.49 frames. ], batch size: 13, lr: 4.53e-03
2024-08-06 10:56:38,299 INFO [trainer.py:765] (0/8) Epoch 19, batch 1000, train_loss[loss=3.282, NarTop10Accuracy=0.6481, over 6326.00 frames. ], tot_loss[loss=3.431, NarTop10Accuracy=0.6314, over 5915.69 frames. ], batch size: 13, lr: 4.52e-03
2024-08-06 10:57:15,188 INFO [trainer.py:765] (0/8) Epoch 19, batch 1100, train_loss[loss=3.363, NarTop10Accuracy=0.6531, over 6766.00 frames. ], tot_loss[loss=3.436, NarTop10Accuracy=0.6302, over 5941.02 frames. ], batch size: 17, lr: 4.52e-03
2024-08-06 10:57:46,665 INFO [trainer.py:765] (0/8) Epoch 19, batch 1200, train_loss[loss=3.272, NarTop10Accuracy=0.664, over 7487.00 frames. ], tot_loss[loss=3.435, NarTop10Accuracy=0.6306, over 5938.22 frames. ], batch size: 31, lr: 4.51e-03
2024-08-06 10:58:23,900 INFO [trainer.py:765] (0/8) Epoch 19, batch 1300, train_loss[loss=3.382, NarTop10Accuracy=0.6376, over 5086.00 frames. ], tot_loss[loss=3.441, NarTop10Accuracy=0.6292, over 6006.31 frames. ], batch size: 6, lr: 4.51e-03
2024-08-06 10:58:58,028 INFO [trainer.py:765] (0/8) Epoch 19, batch 1400, train_loss[loss=3.356, NarTop10Accuracy=0.6379, over 6107.00 frames. ], tot_loss[loss=3.446, NarTop10Accuracy=0.628, over 6025.44 frames. ], batch size: 11, lr: 4.50e-03
2024-08-06 10:59:30,769 INFO [trainer.py:765] (0/8) Epoch 19, batch 1500, train_loss[loss=3.729, NarTop10Accuracy=0.5803, over 5679.00 frames. ], tot_loss[loss=3.449, NarTop10Accuracy=0.6276, over 5970.66 frames. ], batch size: 49, lr: 4.50e-03
2024-08-06 10:59:40,830 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-48000.pt
2024-08-06 10:59:44,391 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 10:59:50,899 INFO [trainer.py:811] (0/8) Epoch 19, validation: loss=3.276, NarTop10Accuracy=0.6653, over 1907754.00 frames. 
2024-08-06 10:59:50,899 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30501MB
2024-08-06 10:59:51,426 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.437e+02 1.829e+02 1.984e+02 2.176e+02 3.542e+02, threshold=3.967e+02, percent-clipped=0.0
2024-08-06 11:00:08,816 INFO [trainer.py:765] (0/8) Epoch 19, batch 1600, train_loss[loss=3.712, NarTop10Accuracy=0.5748, over 7197.00 frames. ], tot_loss[loss=3.45, NarTop10Accuracy=0.6276, over 5954.39 frames. ], batch size: 22, lr: 4.49e-03
2024-08-06 11:00:35,588 INFO [trainer.py:765] (0/8) Epoch 19, batch 1700, train_loss[loss=3.552, NarTop10Accuracy=0.5953, over 6299.00 frames. ], tot_loss[loss=3.458, NarTop10Accuracy=0.6258, over 5942.80 frames. ], batch size: 13, lr: 4.49e-03
2024-08-06 11:01:02,257 INFO [trainer.py:765] (0/8) Epoch 19, batch 1800, train_loss[loss=3.259, NarTop10Accuracy=0.6654, over 6922.00 frames. ], tot_loss[loss=3.444, NarTop10Accuracy=0.6281, over 6013.37 frames. ], batch size: 22, lr: 4.49e-03
2024-08-06 11:01:28,930 INFO [trainer.py:765] (0/8) Epoch 19, batch 1900, train_loss[loss=3.616, NarTop10Accuracy=0.6008, over 5937.00 frames. ], tot_loss[loss=3.457, NarTop10Accuracy=0.6258, over 6048.61 frames. ], batch size: 49, lr: 4.48e-03
2024-08-06 11:01:54,633 INFO [trainer.py:765] (0/8) Epoch 19, batch 2000, train_loss[loss=3.422, NarTop10Accuracy=0.6322, over 5901.00 frames. ], tot_loss[loss=3.453, NarTop10Accuracy=0.6265, over 6016.63 frames. ], batch size: 49, lr: 4.48e-03
2024-08-06 11:02:20,186 INFO [trainer.py:765] (0/8) Epoch 19, batch 2100, train_loss[loss=3.031, NarTop10Accuracy=0.6767, over 4762.00 frames. ], tot_loss[loss=3.457, NarTop10Accuracy=0.6255, over 6004.56 frames. ], batch size: 5, lr: 4.47e-03
2024-08-06 11:02:45,694 INFO [trainer.py:765] (0/8) Epoch 19, batch 2200, train_loss[loss=3.52, NarTop10Accuracy=0.6153, over 7486.00 frames. ], tot_loss[loss=3.464, NarTop10Accuracy=0.6245, over 6034.98 frames. ], batch size: 33, lr: 4.47e-03
2024-08-06 11:03:11,131 INFO [trainer.py:765] (0/8) Epoch 19, batch 2300, train_loss[loss=3.363, NarTop10Accuracy=0.6499, over 5772.00 frames. ], tot_loss[loss=3.463, NarTop10Accuracy=0.6246, over 6071.89 frames. ], batch size: 9, lr: 4.46e-03
2024-08-06 11:03:35,950 INFO [trainer.py:765] (0/8) Epoch 19, batch 2400, train_loss[loss=3.61, NarTop10Accuracy=0.5917, over 6172.00 frames. ], tot_loss[loss=3.476, NarTop10Accuracy=0.6219, over 5912.72 frames. ], batch size: 49, lr: 4.46e-03
2024-08-06 11:03:59,406 INFO [trainer.py:765] (0/8) Epoch 19, batch 2500, train_loss[loss=3.427, NarTop10Accuracy=0.6271, over 5131.00 frames. ], tot_loss[loss=3.44, NarTop10Accuracy=0.6286, over 5559.27 frames. ], batch size: 6, lr: 4.45e-03
2024-08-06 11:04:07,805 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-49000.pt
2024-08-06 11:04:24,321 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 11:04:24,323 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-19.pt
2024-08-06 11:05:26,561 INFO [trainer.py:765] (0/8) Epoch 20, batch 100, train_loss[loss=3.57, NarTop10Accuracy=0.6046, over 7170.00 frames. ], tot_loss[loss=3.395, NarTop10Accuracy=0.6392, over 2384.52 frames. ], batch size: 30, lr: 4.33e-03
2024-08-06 11:05:57,409 INFO [trainer.py:765] (0/8) Epoch 20, batch 200, train_loss[loss=3.4, NarTop10Accuracy=0.6418, over 6913.00 frames. ], tot_loss[loss=3.397, NarTop10Accuracy=0.6387, over 3878.91 frames. ], batch size: 17, lr: 4.33e-03
2024-08-06 11:06:30,634 INFO [trainer.py:765] (0/8) Epoch 20, batch 300, train_loss[loss=3.446, NarTop10Accuracy=0.6291, over 7291.00 frames. ], tot_loss[loss=3.399, NarTop10Accuracy=0.6389, over 4677.75 frames. ], batch size: 22, lr: 4.32e-03
2024-08-06 11:07:06,396 INFO [trainer.py:765] (0/8) Epoch 20, batch 400, train_loss[loss=3.313, NarTop10Accuracy=0.6551, over 5135.00 frames. ], tot_loss[loss=3.392, NarTop10Accuracy=0.6402, over 5144.31 frames. ], batch size: 7, lr: 4.32e-03
2024-08-06 11:07:38,166 INFO [trainer.py:765] (0/8) Epoch 20, batch 500, train_loss[loss=3.138, NarTop10Accuracy=0.6831, over 6235.00 frames. ], tot_loss[loss=3.391, NarTop10Accuracy=0.6399, over 5421.23 frames. ], batch size: 11, lr: 4.31e-03
2024-08-06 11:08:11,568 INFO [trainer.py:765] (0/8) Epoch 20, batch 600, train_loss[loss=3.144, NarTop10Accuracy=0.6769, over 5857.00 frames. ], tot_loss[loss=3.388, NarTop10Accuracy=0.6407, over 5691.84 frames. ], batch size: 9, lr: 4.31e-03
2024-08-06 11:08:46,274 INFO [trainer.py:765] (0/8) Epoch 20, batch 700, train_loss[loss=3.271, NarTop10Accuracy=0.6762, over 4949.00 frames. ], tot_loss[loss=3.402, NarTop10Accuracy=0.6374, over 5750.22 frames. ], batch size: 6, lr: 4.31e-03
2024-08-06 11:09:23,425 INFO [trainer.py:765] (0/8) Epoch 20, batch 800, train_loss[loss=3.192, NarTop10Accuracy=0.6872, over 4293.00 frames. ], tot_loss[loss=3.42, NarTop10Accuracy=0.6341, over 5789.81 frames. ], batch size: 5, lr: 4.30e-03
2024-08-06 11:09:53,513 INFO [trainer.py:765] (0/8) Epoch 20, batch 900, train_loss[loss=3.239, NarTop10Accuracy=0.6641, over 6221.00 frames. ], tot_loss[loss=3.424, NarTop10Accuracy=0.6332, over 5821.22 frames. ], batch size: 13, lr: 4.30e-03
2024-08-06 11:10:12,198 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-50000.pt
2024-08-06 11:10:16,238 WARNING [checkpoint.py:343] (0/8) No checkpoints found in exp/valle
2024-08-06 11:10:16,239 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 11:10:23,738 INFO [trainer.py:811] (0/8) Epoch 20, validation: loss=3.279, NarTop10Accuracy=0.6658, over 1907754.00 frames. 
2024-08-06 11:10:23,739 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30501MB
2024-08-06 11:10:24,298 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.491e+02 1.847e+02 2.007e+02 2.180e+02 4.417e+02, threshold=4.013e+02, percent-clipped=0.1
2024-08-06 11:10:42,965 INFO [trainer.py:765] (0/8) Epoch 20, batch 1000, train_loss[loss=3.176, NarTop10Accuracy=0.6844, over 6667.00 frames. ], tot_loss[loss=3.426, NarTop10Accuracy=0.6326, over 5917.17 frames. ], batch size: 14, lr: 4.29e-03
2024-08-06 11:11:21,022 INFO [trainer.py:765] (0/8) Epoch 20, batch 1100, train_loss[loss=3.239, NarTop10Accuracy=0.667, over 6895.00 frames. ], tot_loss[loss=3.431, NarTop10Accuracy=0.6316, over 5958.43 frames. ], batch size: 17, lr: 4.29e-03
2024-08-06 11:11:55,393 INFO [trainer.py:765] (0/8) Epoch 20, batch 1200, train_loss[loss=3.396, NarTop10Accuracy=0.6304, over 7420.00 frames. ], tot_loss[loss=3.43, NarTop10Accuracy=0.6315, over 5956.42 frames. ], batch size: 31, lr: 4.28e-03
2024-08-06 11:12:30,751 INFO [trainer.py:765] (0/8) Epoch 20, batch 1300, train_loss[loss=3.48, NarTop10Accuracy=0.6197, over 5126.00 frames. ], tot_loss[loss=3.417, NarTop10Accuracy=0.6341, over 6021.47 frames. ], batch size: 6, lr: 4.28e-03
2024-08-06 11:13:10,291 INFO [trainer.py:765] (0/8) Epoch 20, batch 1400, train_loss[loss=3.664, NarTop10Accuracy=0.5908, over 6055.00 frames. ], tot_loss[loss=3.44, NarTop10Accuracy=0.6299, over 6050.41 frames. ], batch size: 11, lr: 4.28e-03
2024-08-06 11:13:38,989 INFO [trainer.py:765] (0/8) Epoch 20, batch 1500, train_loss[loss=3.552, NarTop10Accuracy=0.6029, over 6106.00 frames. ], tot_loss[loss=3.446, NarTop10Accuracy=0.6288, over 5986.45 frames. ], batch size: 49, lr: 4.27e-03
2024-08-06 11:14:07,051 INFO [trainer.py:765] (0/8) Epoch 20, batch 1600, train_loss[loss=3.399, NarTop10Accuracy=0.6412, over 7181.00 frames. ], tot_loss[loss=3.445, NarTop10Accuracy=0.6291, over 5960.06 frames. ], batch size: 22, lr: 4.27e-03
2024-08-06 11:14:33,910 INFO [trainer.py:765] (0/8) Epoch 20, batch 1700, train_loss[loss=3.677, NarTop10Accuracy=0.5825, over 6613.00 frames. ], tot_loss[loss=3.441, NarTop10Accuracy=0.6298, over 5941.84 frames. ], batch size: 14, lr: 4.26e-03
2024-08-06 11:15:00,590 INFO [trainer.py:765] (0/8) Epoch 20, batch 1800, train_loss[loss=3.347, NarTop10Accuracy=0.6523, over 7109.00 frames. ], tot_loss[loss=3.446, NarTop10Accuracy=0.6284, over 6014.00 frames. ], batch size: 22, lr: 4.26e-03
2024-08-06 11:15:27,276 INFO [trainer.py:765] (0/8) Epoch 20, batch 1900, train_loss[loss=3.641, NarTop10Accuracy=0.5901, over 6478.00 frames. ], tot_loss[loss=3.456, NarTop10Accuracy=0.6265, over 6044.80 frames. ], batch size: 51, lr: 4.26e-03
2024-08-06 11:15:41,045 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-51000.pt
2024-08-06 11:15:56,438 INFO [trainer.py:765] (0/8) Epoch 20, batch 2000, train_loss[loss=3.533, NarTop10Accuracy=0.6177, over 5881.00 frames. ], tot_loss[loss=3.448, NarTop10Accuracy=0.6282, over 6018.35 frames. ], batch size: 49, lr: 4.25e-03
2024-08-06 11:16:21,957 INFO [trainer.py:765] (0/8) Epoch 20, batch 2100, train_loss[loss=3.102, NarTop10Accuracy=0.7055, over 3844.00 frames. ], tot_loss[loss=3.461, NarTop10Accuracy=0.626, over 5997.16 frames. ], batch size: 4, lr: 4.25e-03
2024-08-06 11:16:47,405 INFO [trainer.py:765] (0/8) Epoch 20, batch 2200, train_loss[loss=3.477, NarTop10Accuracy=0.6204, over 7070.00 frames. ], tot_loss[loss=3.468, NarTop10Accuracy=0.6242, over 6046.12 frames. ], batch size: 30, lr: 4.24e-03
2024-08-06 11:17:12,907 INFO [trainer.py:765] (0/8) Epoch 20, batch 2300, train_loss[loss=3.574, NarTop10Accuracy=0.5969, over 5783.00 frames. ], tot_loss[loss=3.474, NarTop10Accuracy=0.6226, over 6063.83 frames. ], batch size: 9, lr: 4.24e-03
2024-08-06 11:17:37,714 INFO [trainer.py:765] (0/8) Epoch 20, batch 2400, train_loss[loss=3.315, NarTop10Accuracy=0.6505, over 5130.00 frames. ], tot_loss[loss=3.47, NarTop10Accuracy=0.6235, over 5881.42 frames. ], batch size: 7, lr: 4.24e-03
2024-08-06 11:18:01,247 INFO [trainer.py:765] (0/8) Epoch 20, batch 2500, train_loss[loss=3.564, NarTop10Accuracy=0.6051, over 5028.00 frames. ], tot_loss[loss=3.44, NarTop10Accuracy=0.6302, over 5543.41 frames. ], batch size: 6, lr: 4.23e-03
2024-08-06 11:18:22,262 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 11:18:22,264 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-20.pt
2024-08-06 11:19:21,459 INFO [trainer.py:765] (0/8) Epoch 21, batch 100, train_loss[loss=3.248, NarTop10Accuracy=0.6621, over 7339.00 frames. ], tot_loss[loss=3.396, NarTop10Accuracy=0.64, over 2379.39 frames. ], batch size: 32, lr: 4.12e-03
2024-08-06 11:19:56,522 INFO [trainer.py:765] (0/8) Epoch 21, batch 200, train_loss[loss=3.419, NarTop10Accuracy=0.6452, over 7004.00 frames. ], tot_loss[loss=3.396, NarTop10Accuracy=0.6395, over 3868.69 frames. ], batch size: 17, lr: 4.12e-03
2024-08-06 11:20:26,598 INFO [trainer.py:765] (0/8) Epoch 21, batch 300, train_loss[loss=3.643, NarTop10Accuracy=0.5917, over 7038.00 frames. ], tot_loss[loss=3.381, NarTop10Accuracy=0.6423, over 4682.70 frames. ], batch size: 22, lr: 4.11e-03
2024-08-06 11:20:54,240 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-52000.pt
2024-08-06 11:20:57,870 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 11:21:04,970 INFO [trainer.py:811] (0/8) Epoch 21, validation: loss=3.291, NarTop10Accuracy=0.6625, over 1907754.00 frames. 
2024-08-06 11:21:04,970 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 11:21:05,486 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.509e+02 1.858e+02 2.007e+02 2.193e+02 3.729e+02, threshold=4.015e+02, percent-clipped=0.0
2024-08-06 11:21:12,221 INFO [trainer.py:765] (0/8) Epoch 21, batch 400, train_loss[loss=3.504, NarTop10Accuracy=0.609, over 5166.00 frames. ], tot_loss[loss=3.39, NarTop10Accuracy=0.6402, over 5128.82 frames. ], batch size: 7, lr: 4.11e-03
2024-08-06 11:21:47,569 INFO [trainer.py:765] (0/8) Epoch 21, batch 500, train_loss[loss=3.356, NarTop10Accuracy=0.6553, over 6151.00 frames. ], tot_loss[loss=3.385, NarTop10Accuracy=0.6412, over 5401.06 frames. ], batch size: 11, lr: 4.11e-03
2024-08-06 11:22:18,238 INFO [trainer.py:765] (0/8) Epoch 21, batch 600, train_loss[loss=3.737, NarTop10Accuracy=0.5716, over 5728.00 frames. ], tot_loss[loss=3.395, NarTop10Accuracy=0.6391, over 5670.42 frames. ], batch size: 9, lr: 4.10e-03
2024-08-06 11:22:56,843 INFO [trainer.py:765] (0/8) Epoch 21, batch 700, train_loss[loss=3.07, NarTop10Accuracy=0.7053, over 5079.00 frames. ], tot_loss[loss=3.406, NarTop10Accuracy=0.6368, over 5750.59 frames. ], batch size: 6, lr: 4.10e-03
2024-08-06 11:23:33,075 INFO [trainer.py:765] (0/8) Epoch 21, batch 800, train_loss[loss=3.328, NarTop10Accuracy=0.641, over 5085.00 frames. ], tot_loss[loss=3.417, NarTop10Accuracy=0.6347, over 5817.87 frames. ], batch size: 6, lr: 4.09e-03
2024-08-06 11:24:03,022 INFO [trainer.py:765] (0/8) Epoch 21, batch 900, train_loss[loss=3.655, NarTop10Accuracy=0.5791, over 6134.00 frames. ], tot_loss[loss=3.42, NarTop10Accuracy=0.634, over 5830.86 frames. ], batch size: 13, lr: 4.09e-03
2024-08-06 11:24:37,089 INFO [trainer.py:765] (0/8) Epoch 21, batch 1000, train_loss[loss=3.416, NarTop10Accuracy=0.6235, over 6743.00 frames. ], tot_loss[loss=3.43, NarTop10Accuracy=0.632, over 5935.27 frames. ], batch size: 14, lr: 4.09e-03
2024-08-06 11:25:16,429 INFO [trainer.py:765] (0/8) Epoch 21, batch 1100, train_loss[loss=3.661, NarTop10Accuracy=0.5849, over 6898.00 frames. ], tot_loss[loss=3.437, NarTop10Accuracy=0.6305, over 5958.83 frames. ], batch size: 17, lr: 4.08e-03
2024-08-06 11:25:47,740 INFO [trainer.py:765] (0/8) Epoch 21, batch 1200, train_loss[loss=3.46, NarTop10Accuracy=0.6329, over 7371.00 frames. ], tot_loss[loss=3.419, NarTop10Accuracy=0.6339, over 5961.73 frames. ], batch size: 30, lr: 4.08e-03
2024-08-06 11:26:23,057 INFO [trainer.py:765] (0/8) Epoch 21, batch 1300, train_loss[loss=3.579, NarTop10Accuracy=0.6067, over 5001.00 frames. ], tot_loss[loss=3.408, NarTop10Accuracy=0.636, over 6025.63 frames. ], batch size: 6, lr: 4.07e-03
2024-08-06 11:26:49,588 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-53000.pt
2024-08-06 11:27:00,082 INFO [trainer.py:765] (0/8) Epoch 21, batch 1400, train_loss[loss=3.232, NarTop10Accuracy=0.6695, over 6063.00 frames. ], tot_loss[loss=3.422, NarTop10Accuracy=0.6335, over 6051.50 frames. ], batch size: 11, lr: 4.07e-03
2024-08-06 11:27:35,326 INFO [trainer.py:765] (0/8) Epoch 21, batch 1500, train_loss[loss=3.882, NarTop10Accuracy=0.5382, over 6225.00 frames. ], tot_loss[loss=3.435, NarTop10Accuracy=0.6309, over 5980.16 frames. ], batch size: 49, lr: 4.07e-03
2024-08-06 11:28:03,315 INFO [trainer.py:765] (0/8) Epoch 21, batch 1600, train_loss[loss=3.313, NarTop10Accuracy=0.6598, over 7239.00 frames. ], tot_loss[loss=3.431, NarTop10Accuracy=0.6311, over 5951.56 frames. ], batch size: 22, lr: 4.06e-03
2024-08-06 11:28:30,105 INFO [trainer.py:765] (0/8) Epoch 21, batch 1700, train_loss[loss=3.691, NarTop10Accuracy=0.5762, over 6277.00 frames. ], tot_loss[loss=3.438, NarTop10Accuracy=0.6304, over 5939.83 frames. ], batch size: 13, lr: 4.06e-03
2024-08-06 11:28:56,641 INFO [trainer.py:765] (0/8) Epoch 21, batch 1800, train_loss[loss=3.446, NarTop10Accuracy=0.6228, over 7209.00 frames. ], tot_loss[loss=3.435, NarTop10Accuracy=0.6303, over 6010.93 frames. ], batch size: 23, lr: 4.06e-03
2024-08-06 11:29:23,198 INFO [trainer.py:765] (0/8) Epoch 21, batch 1900, train_loss[loss=3.397, NarTop10Accuracy=0.6472, over 6473.00 frames. ], tot_loss[loss=3.443, NarTop10Accuracy=0.629, over 6035.51 frames. ], batch size: 50, lr: 4.05e-03
2024-08-06 11:29:49,028 INFO [trainer.py:765] (0/8) Epoch 21, batch 2000, train_loss[loss=3.477, NarTop10Accuracy=0.6197, over 5303.00 frames. ], tot_loss[loss=3.441, NarTop10Accuracy=0.6291, over 6017.95 frames. ], batch size: 49, lr: 4.05e-03
2024-08-06 11:30:14,529 INFO [trainer.py:765] (0/8) Epoch 21, batch 2100, train_loss[loss=3.358, NarTop10Accuracy=0.6465, over 4752.00 frames. ], tot_loss[loss=3.445, NarTop10Accuracy=0.6281, over 6002.26 frames. ], batch size: 5, lr: 4.04e-03
2024-08-06 11:30:39,871 INFO [trainer.py:765] (0/8) Epoch 21, batch 2200, train_loss[loss=3.682, NarTop10Accuracy=0.5935, over 7128.00 frames. ], tot_loss[loss=3.447, NarTop10Accuracy=0.6281, over 6030.33 frames. ], batch size: 30, lr: 4.04e-03
2024-08-06 11:31:05,472 INFO [trainer.py:765] (0/8) Epoch 21, batch 2300, train_loss[loss=3.389, NarTop10Accuracy=0.6355, over 5699.00 frames. ], tot_loss[loss=3.455, NarTop10Accuracy=0.6266, over 6062.14 frames. ], batch size: 9, lr: 4.04e-03
2024-08-06 11:31:23,873 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-54000.pt
2024-08-06 11:31:27,713 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 11:31:34,439 INFO [trainer.py:811] (0/8) Epoch 21, validation: loss=3.272, NarTop10Accuracy=0.6665, over 1907754.00 frames. 
2024-08-06 11:31:34,439 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 11:31:34,938 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.496e+02 1.892e+02 2.038e+02 2.210e+02 4.910e+02, threshold=4.076e+02, percent-clipped=0.1
2024-08-06 11:31:40,753 INFO [trainer.py:765] (0/8) Epoch 21, batch 2400, train_loss[loss=3.276, NarTop10Accuracy=0.6655, over 5193.00 frames. ], tot_loss[loss=3.452, NarTop10Accuracy=0.6271, over 5873.51 frames. ], batch size: 7, lr: 4.03e-03
2024-08-06 11:32:04,057 INFO [trainer.py:765] (0/8) Epoch 21, batch 2500, train_loss[loss=3.357, NarTop10Accuracy=0.6439, over 5065.00 frames. ], tot_loss[loss=3.431, NarTop10Accuracy=0.6312, over 5530.06 frames. ], batch size: 6, lr: 4.03e-03
2024-08-06 11:32:25,520 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 11:32:25,523 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-21.pt
2024-08-06 11:33:29,682 INFO [trainer.py:765] (0/8) Epoch 22, batch 100, train_loss[loss=3.721, NarTop10Accuracy=0.5773, over 7314.00 frames. ], tot_loss[loss=3.373, NarTop10Accuracy=0.6441, over 2374.95 frames. ], batch size: 31, lr: 3.93e-03
2024-08-06 11:34:05,036 INFO [trainer.py:765] (0/8) Epoch 22, batch 200, train_loss[loss=3.134, NarTop10Accuracy=0.6779, over 6791.00 frames. ], tot_loss[loss=3.364, NarTop10Accuracy=0.6454, over 3876.84 frames. ], batch size: 17, lr: 3.93e-03
2024-08-06 11:34:37,619 INFO [trainer.py:765] (0/8) Epoch 22, batch 300, train_loss[loss=3.199, NarTop10Accuracy=0.6804, over 7117.00 frames. ], tot_loss[loss=3.364, NarTop10Accuracy=0.6458, over 4663.67 frames. ], batch size: 22, lr: 3.92e-03
2024-08-06 11:35:09,968 INFO [trainer.py:765] (0/8) Epoch 22, batch 400, train_loss[loss=3.245, NarTop10Accuracy=0.6756, over 5088.00 frames. ], tot_loss[loss=3.378, NarTop10Accuracy=0.6427, over 5124.95 frames. ], batch size: 7, lr: 3.92e-03
2024-08-06 11:35:42,508 INFO [trainer.py:765] (0/8) Epoch 22, batch 500, train_loss[loss=3.457, NarTop10Accuracy=0.6376, over 6176.00 frames. ], tot_loss[loss=3.386, NarTop10Accuracy=0.6412, over 5407.32 frames. ], batch size: 11, lr: 3.91e-03
2024-08-06 11:36:16,058 INFO [trainer.py:765] (0/8) Epoch 22, batch 600, train_loss[loss=3.138, NarTop10Accuracy=0.7038, over 5700.00 frames. ], tot_loss[loss=3.386, NarTop10Accuracy=0.6411, over 5669.14 frames. ], batch size: 9, lr: 3.91e-03
2024-08-06 11:36:53,858 INFO [trainer.py:765] (0/8) Epoch 22, batch 700, train_loss[loss=3.32, NarTop10Accuracy=0.6515, over 4899.00 frames. ], tot_loss[loss=3.394, NarTop10Accuracy=0.6391, over 5723.82 frames. ], batch size: 6, lr: 3.91e-03
2024-08-06 11:37:23,028 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-55000.pt
2024-08-06 11:37:28,480 INFO [trainer.py:765] (0/8) Epoch 22, batch 800, train_loss[loss=3.227, NarTop10Accuracy=0.6677, over 4946.00 frames. ], tot_loss[loss=3.406, NarTop10Accuracy=0.6366, over 5781.50 frames. ], batch size: 6, lr: 3.90e-03
2024-08-06 11:38:03,950 INFO [trainer.py:765] (0/8) Epoch 22, batch 900, train_loss[loss=3.187, NarTop10Accuracy=0.6772, over 6371.00 frames. ], tot_loss[loss=3.396, NarTop10Accuracy=0.6384, over 5796.06 frames. ], batch size: 13, lr: 3.90e-03
2024-08-06 11:38:38,329 INFO [trainer.py:765] (0/8) Epoch 22, batch 1000, train_loss[loss=3.192, NarTop10Accuracy=0.6724, over 6231.00 frames. ], tot_loss[loss=3.405, NarTop10Accuracy=0.6371, over 5903.61 frames. ], batch size: 13, lr: 3.90e-03
2024-08-06 11:39:14,788 INFO [trainer.py:765] (0/8) Epoch 22, batch 1100, train_loss[loss=3.444, NarTop10Accuracy=0.622, over 6921.00 frames. ], tot_loss[loss=3.41, NarTop10Accuracy=0.6359, over 5938.99 frames. ], batch size: 17, lr: 3.89e-03
2024-08-06 11:39:48,523 INFO [trainer.py:765] (0/8) Epoch 22, batch 1200, train_loss[loss=3.317, NarTop10Accuracy=0.6575, over 7706.00 frames. ], tot_loss[loss=3.411, NarTop10Accuracy=0.6356, over 5936.77 frames. ], batch size: 31, lr: 3.89e-03
2024-08-06 11:40:25,246 INFO [trainer.py:765] (0/8) Epoch 22, batch 1300, train_loss[loss=3.002, NarTop10Accuracy=0.7141, over 4993.00 frames. ], tot_loss[loss=3.412, NarTop10Accuracy=0.6353, over 6001.06 frames. ], batch size: 6, lr: 3.89e-03
2024-08-06 11:41:00,610 INFO [trainer.py:765] (0/8) Epoch 22, batch 1400, train_loss[loss=3.404, NarTop10Accuracy=0.6312, over 6043.00 frames. ], tot_loss[loss=3.417, NarTop10Accuracy=0.6345, over 6011.39 frames. ], batch size: 11, lr: 3.88e-03
2024-08-06 11:41:31,584 INFO [trainer.py:765] (0/8) Epoch 22, batch 1500, train_loss[loss=3.663, NarTop10Accuracy=0.5872, over 6857.00 frames. ], tot_loss[loss=3.422, NarTop10Accuracy=0.633, over 5956.42 frames. ], batch size: 50, lr: 3.88e-03
2024-08-06 11:41:59,677 INFO [trainer.py:765] (0/8) Epoch 22, batch 1600, train_loss[loss=3.228, NarTop10Accuracy=0.6741, over 7293.00 frames. ], tot_loss[loss=3.434, NarTop10Accuracy=0.6306, over 5949.19 frames. ], batch size: 22, lr: 3.88e-03
2024-08-06 11:42:26,464 INFO [trainer.py:765] (0/8) Epoch 22, batch 1700, train_loss[loss=3.444, NarTop10Accuracy=0.6442, over 6255.00 frames. ], tot_loss[loss=3.447, NarTop10Accuracy=0.6286, over 5944.36 frames. ], batch size: 13, lr: 3.87e-03
2024-08-06 11:42:50,722 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-56000.pt
2024-08-06 11:42:54,181 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 11:43:00,818 INFO [trainer.py:811] (0/8) Epoch 22, validation: loss=3.305, NarTop10Accuracy=0.6597, over 1907754.00 frames. 
2024-08-06 11:43:00,819 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 11:43:01,327 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.500e+02 1.900e+02 2.042e+02 2.234e+02 3.494e+02, threshold=4.085e+02, percent-clipped=0.0
2024-08-06 11:43:03,219 INFO [trainer.py:765] (0/8) Epoch 22, batch 1800, train_loss[loss=3.29, NarTop10Accuracy=0.6608, over 7368.00 frames. ], tot_loss[loss=3.439, NarTop10Accuracy=0.6306, over 6007.89 frames. ], batch size: 22, lr: 3.87e-03
2024-08-06 11:43:29,752 INFO [trainer.py:765] (0/8) Epoch 22, batch 1900, train_loss[loss=3.58, NarTop10Accuracy=0.6003, over 6674.00 frames. ], tot_loss[loss=3.442, NarTop10Accuracy=0.6297, over 6039.26 frames. ], batch size: 49, lr: 3.87e-03
2024-08-06 11:43:55,485 INFO [trainer.py:765] (0/8) Epoch 22, batch 2000, train_loss[loss=3.787, NarTop10Accuracy=0.5563, over 6575.00 frames. ], tot_loss[loss=3.433, NarTop10Accuracy=0.6313, over 6019.97 frames. ], batch size: 49, lr: 3.86e-03
2024-08-06 11:44:20,932 INFO [trainer.py:765] (0/8) Epoch 22, batch 2100, train_loss[loss=2.986, NarTop10Accuracy=0.7, over 4955.00 frames. ], tot_loss[loss=3.432, NarTop10Accuracy=0.6314, over 5990.34 frames. ], batch size: 5, lr: 3.86e-03
2024-08-06 11:44:46,456 INFO [trainer.py:765] (0/8) Epoch 22, batch 2200, train_loss[loss=3.837, NarTop10Accuracy=0.5452, over 7035.00 frames. ], tot_loss[loss=3.424, NarTop10Accuracy=0.6327, over 6020.57 frames. ], batch size: 30, lr: 3.86e-03
2024-08-06 11:45:11,882 INFO [trainer.py:765] (0/8) Epoch 22, batch 2300, train_loss[loss=3.288, NarTop10Accuracy=0.6568, over 5819.00 frames. ], tot_loss[loss=3.426, NarTop10Accuracy=0.6321, over 6060.75 frames. ], batch size: 9, lr: 3.85e-03
2024-08-06 11:45:36,583 INFO [trainer.py:765] (0/8) Epoch 22, batch 2400, train_loss[loss=3.599, NarTop10Accuracy=0.6004, over 6093.00 frames. ], tot_loss[loss=3.447, NarTop10Accuracy=0.6284, over 5875.54 frames. ], batch size: 48, lr: 3.85e-03
2024-08-06 11:46:00,081 INFO [trainer.py:765] (0/8) Epoch 22, batch 2500, train_loss[loss=3.13, NarTop10Accuracy=0.6867, over 5043.00 frames. ], tot_loss[loss=3.425, NarTop10Accuracy=0.6326, over 5543.89 frames. ], batch size: 6, lr: 3.85e-03
2024-08-06 11:46:21,449 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 11:46:21,452 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-22.pt
2024-08-06 11:47:20,476 INFO [trainer.py:765] (0/8) Epoch 23, batch 100, train_loss[loss=3.234, NarTop10Accuracy=0.6765, over 7156.00 frames. ], tot_loss[loss=3.366, NarTop10Accuracy=0.6459, over 2370.22 frames. ], batch size: 31, lr: 3.75e-03
2024-08-06 11:47:52,035 INFO [trainer.py:765] (0/8) Epoch 23, batch 200, train_loss[loss=3.46, NarTop10Accuracy=0.6322, over 6843.00 frames. ], tot_loss[loss=3.35, NarTop10Accuracy=0.649, over 3879.56 frames. ], batch size: 17, lr: 3.75e-03
2024-08-06 11:47:54,493 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-57000.pt
2024-08-06 11:48:33,921 INFO [trainer.py:765] (0/8) Epoch 23, batch 300, train_loss[loss=3.322, NarTop10Accuracy=0.6354, over 7175.00 frames. ], tot_loss[loss=3.367, NarTop10Accuracy=0.6452, over 4697.29 frames. ], batch size: 22, lr: 3.75e-03
2024-08-06 11:49:06,656 INFO [trainer.py:765] (0/8) Epoch 23, batch 400, train_loss[loss=3.253, NarTop10Accuracy=0.6755, over 5208.00 frames. ], tot_loss[loss=3.372, NarTop10Accuracy=0.644, over 5135.07 frames. ], batch size: 7, lr: 3.74e-03
2024-08-06 11:49:37,619 INFO [trainer.py:765] (0/8) Epoch 23, batch 500, train_loss[loss=3.601, NarTop10Accuracy=0.6016, over 6186.00 frames. ], tot_loss[loss=3.376, NarTop10Accuracy=0.6435, over 5424.59 frames. ], batch size: 11, lr: 3.74e-03
2024-08-06 11:50:06,740 INFO [trainer.py:765] (0/8) Epoch 23, batch 600, train_loss[loss=3.617, NarTop10Accuracy=0.5901, over 5731.00 frames. ], tot_loss[loss=3.369, NarTop10Accuracy=0.6446, over 5681.99 frames. ], batch size: 9, lr: 3.74e-03
2024-08-06 11:50:47,601 INFO [trainer.py:765] (0/8) Epoch 23, batch 700, train_loss[loss=3.249, NarTop10Accuracy=0.6673, over 4987.00 frames. ], tot_loss[loss=3.377, NarTop10Accuracy=0.6429, over 5729.50 frames. ], batch size: 6, lr: 3.73e-03
2024-08-06 11:51:21,344 INFO [trainer.py:765] (0/8) Epoch 23, batch 800, train_loss[loss=3.39, NarTop10Accuracy=0.642, over 5046.00 frames. ], tot_loss[loss=3.391, NarTop10Accuracy=0.6399, over 5784.69 frames. ], batch size: 6, lr: 3.73e-03
2024-08-06 11:51:52,396 INFO [trainer.py:765] (0/8) Epoch 23, batch 900, train_loss[loss=3.334, NarTop10Accuracy=0.6488, over 6143.00 frames. ], tot_loss[loss=3.385, NarTop10Accuracy=0.6413, over 5803.84 frames. ], batch size: 13, lr: 3.73e-03
2024-08-06 11:52:33,917 INFO [trainer.py:765] (0/8) Epoch 23, batch 1000, train_loss[loss=3.172, NarTop10Accuracy=0.6754, over 6744.00 frames. ], tot_loss[loss=3.394, NarTop10Accuracy=0.6394, over 5906.80 frames. ], batch size: 14, lr: 3.73e-03
2024-08-06 11:53:08,608 INFO [trainer.py:765] (0/8) Epoch 23, batch 1100, train_loss[loss=3.631, NarTop10Accuracy=0.5899, over 7020.00 frames. ], tot_loss[loss=3.411, NarTop10Accuracy=0.6357, over 5941.97 frames. ], batch size: 17, lr: 3.72e-03
2024-08-06 11:53:40,339 INFO [trainer.py:765] (0/8) Epoch 23, batch 1200, train_loss[loss=3.434, NarTop10Accuracy=0.6299, over 7183.00 frames. ], tot_loss[loss=3.422, NarTop10Accuracy=0.6331, over 5947.61 frames. ], batch size: 31, lr: 3.72e-03
2024-08-06 11:53:42,823 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-58000.pt
2024-08-06 11:53:46,299 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 11:53:53,935 INFO [trainer.py:811] (0/8) Epoch 23, validation: loss=3.236, NarTop10Accuracy=0.6739, over 1907754.00 frames. 
2024-08-06 11:53:53,935 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 11:53:54,457 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.540e+02 1.901e+02 2.047e+02 2.234e+02 4.368e+02, threshold=4.093e+02, percent-clipped=0.1
2024-08-06 11:54:30,447 INFO [trainer.py:765] (0/8) Epoch 23, batch 1300, train_loss[loss=3.343, NarTop10Accuracy=0.6414, over 4945.00 frames. ], tot_loss[loss=3.414, NarTop10Accuracy=0.6343, over 6014.42 frames. ], batch size: 6, lr: 3.72e-03
2024-08-06 11:55:04,197 INFO [trainer.py:765] (0/8) Epoch 23, batch 1400, train_loss[loss=3.397, NarTop10Accuracy=0.6363, over 6184.00 frames. ], tot_loss[loss=3.417, NarTop10Accuracy=0.6344, over 6038.82 frames. ], batch size: 11, lr: 3.71e-03
2024-08-06 11:55:35,397 INFO [trainer.py:765] (0/8) Epoch 23, batch 1500, train_loss[loss=3.597, NarTop10Accuracy=0.6022, over 6144.00 frames. ], tot_loss[loss=3.425, NarTop10Accuracy=0.6327, over 5977.67 frames. ], batch size: 49, lr: 3.71e-03
2024-08-06 11:56:03,427 INFO [trainer.py:765] (0/8) Epoch 23, batch 1600, train_loss[loss=3.509, NarTop10Accuracy=0.6345, over 7093.00 frames. ], tot_loss[loss=3.413, NarTop10Accuracy=0.6348, over 5969.81 frames. ], batch size: 22, lr: 3.71e-03
2024-08-06 11:56:30,202 INFO [trainer.py:765] (0/8) Epoch 23, batch 1700, train_loss[loss=3.439, NarTop10Accuracy=0.6223, over 6316.00 frames. ], tot_loss[loss=3.435, NarTop10Accuracy=0.6303, over 5953.36 frames. ], batch size: 13, lr: 3.70e-03
2024-08-06 11:56:56,968 INFO [trainer.py:765] (0/8) Epoch 23, batch 1800, train_loss[loss=3.366, NarTop10Accuracy=0.6486, over 7008.00 frames. ], tot_loss[loss=3.435, NarTop10Accuracy=0.631, over 6007.52 frames. ], batch size: 22, lr: 3.70e-03
2024-08-06 11:57:23,596 INFO [trainer.py:765] (0/8) Epoch 23, batch 1900, train_loss[loss=3.544, NarTop10Accuracy=0.6109, over 6092.00 frames. ], tot_loss[loss=3.439, NarTop10Accuracy=0.6306, over 6049.70 frames. ], batch size: 49, lr: 3.70e-03
2024-08-06 11:57:49,250 INFO [trainer.py:765] (0/8) Epoch 23, batch 2000, train_loss[loss=3.694, NarTop10Accuracy=0.5782, over 6383.00 frames. ], tot_loss[loss=3.433, NarTop10Accuracy=0.6312, over 6015.71 frames. ], batch size: 49, lr: 3.69e-03
2024-08-06 11:58:14,769 INFO [trainer.py:765] (0/8) Epoch 23, batch 2100, train_loss[loss=3.686, NarTop10Accuracy=0.5756, over 3989.00 frames. ], tot_loss[loss=3.426, NarTop10Accuracy=0.6326, over 5990.69 frames. ], batch size: 4, lr: 3.69e-03
2024-08-06 11:58:40,237 INFO [trainer.py:765] (0/8) Epoch 23, batch 2200, train_loss[loss=3.721, NarTop10Accuracy=0.5829, over 7521.00 frames. ], tot_loss[loss=3.423, NarTop10Accuracy=0.6336, over 6029.78 frames. ], batch size: 31, lr: 3.69e-03
2024-08-06 11:58:42,492 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-59000.pt
2024-08-06 11:59:08,915 INFO [trainer.py:765] (0/8) Epoch 23, batch 2300, train_loss[loss=3.402, NarTop10Accuracy=0.632, over 5740.00 frames. ], tot_loss[loss=3.435, NarTop10Accuracy=0.6312, over 6062.11 frames. ], batch size: 9, lr: 3.68e-03
2024-08-06 11:59:33,601 INFO [trainer.py:765] (0/8) Epoch 23, batch 2400, train_loss[loss=3.219, NarTop10Accuracy=0.6564, over 5134.00 frames. ], tot_loss[loss=3.444, NarTop10Accuracy=0.6292, over 5881.67 frames. ], batch size: 7, lr: 3.68e-03
2024-08-06 11:59:57,010 INFO [trainer.py:765] (0/8) Epoch 23, batch 2500, train_loss[loss=3.534, NarTop10Accuracy=0.6211, over 5079.00 frames. ], tot_loss[loss=3.418, NarTop10Accuracy=0.6337, over 5536.53 frames. ], batch size: 6, lr: 3.68e-03
2024-08-06 12:00:17,805 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 12:00:17,808 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-23.pt
2024-08-06 12:01:22,110 INFO [trainer.py:765] (0/8) Epoch 24, batch 100, train_loss[loss=3.572, NarTop10Accuracy=0.6065, over 7255.00 frames. ], tot_loss[loss=3.393, NarTop10Accuracy=0.6404, over 2376.62 frames. ], batch size: 30, lr: 3.59e-03
2024-08-06 12:01:51,341 INFO [trainer.py:765] (0/8) Epoch 24, batch 200, train_loss[loss=3.511, NarTop10Accuracy=0.6158, over 6920.00 frames. ], tot_loss[loss=3.374, NarTop10Accuracy=0.6445, over 3866.49 frames. ], batch size: 17, lr: 3.59e-03
2024-08-06 12:02:23,512 INFO [trainer.py:765] (0/8) Epoch 24, batch 300, train_loss[loss=3.239, NarTop10Accuracy=0.6727, over 7072.00 frames. ], tot_loss[loss=3.368, NarTop10Accuracy=0.6457, over 4683.27 frames. ], batch size: 22, lr: 3.59e-03
2024-08-06 12:03:02,846 INFO [trainer.py:765] (0/8) Epoch 24, batch 400, train_loss[loss=3.071, NarTop10Accuracy=0.6993, over 5173.00 frames. ], tot_loss[loss=3.37, NarTop10Accuracy=0.6454, over 5146.66 frames. ], batch size: 7, lr: 3.59e-03
2024-08-06 12:03:31,256 INFO [trainer.py:765] (0/8) Epoch 24, batch 500, train_loss[loss=3.156, NarTop10Accuracy=0.6839, over 6130.00 frames. ], tot_loss[loss=3.368, NarTop10Accuracy=0.6453, over 5420.38 frames. ], batch size: 11, lr: 3.58e-03
2024-08-06 12:04:00,173 INFO [trainer.py:765] (0/8) Epoch 24, batch 600, train_loss[loss=3.71, NarTop10Accuracy=0.5816, over 5751.00 frames. ], tot_loss[loss=3.374, NarTop10Accuracy=0.6439, over 5687.52 frames. ], batch size: 9, lr: 3.58e-03
2024-08-06 12:04:12,530 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-60000.pt
2024-08-06 12:04:16,120 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 12:04:22,775 INFO [trainer.py:811] (0/8) Epoch 24, validation: loss=3.282, NarTop10Accuracy=0.6644, over 1907754.00 frames. 
2024-08-06 12:04:22,775 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 12:04:23,311 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.497e+02 1.905e+02 2.071e+02 2.258e+02 3.709e+02, threshold=4.142e+02, percent-clipped=0.0
2024-08-06 12:04:51,733 INFO [trainer.py:765] (0/8) Epoch 24, batch 700, train_loss[loss=3.198, NarTop10Accuracy=0.6745, over 4920.00 frames. ], tot_loss[loss=3.381, NarTop10Accuracy=0.6426, over 5748.50 frames. ], batch size: 6, lr: 3.58e-03
2024-08-06 12:05:21,275 INFO [trainer.py:765] (0/8) Epoch 24, batch 800, train_loss[loss=3.489, NarTop10Accuracy=0.6297, over 5147.00 frames. ], tot_loss[loss=3.389, NarTop10Accuracy=0.6405, over 5798.43 frames. ], batch size: 6, lr: 3.57e-03
2024-08-06 12:05:51,754 INFO [trainer.py:765] (0/8) Epoch 24, batch 900, train_loss[loss=3.55, NarTop10Accuracy=0.606, over 6273.00 frames. ], tot_loss[loss=3.383, NarTop10Accuracy=0.6411, over 5806.87 frames. ], batch size: 13, lr: 3.57e-03
2024-08-06 12:06:32,813 INFO [trainer.py:765] (0/8) Epoch 24, batch 1000, train_loss[loss=3.109, NarTop10Accuracy=0.7055, over 6309.00 frames. ], tot_loss[loss=3.39, NarTop10Accuracy=0.6395, over 5905.81 frames. ], batch size: 13, lr: 3.57e-03
2024-08-06 12:07:09,040 INFO [trainer.py:765] (0/8) Epoch 24, batch 1100, train_loss[loss=3.365, NarTop10Accuracy=0.6467, over 6800.00 frames. ], tot_loss[loss=3.4, NarTop10Accuracy=0.6381, over 5944.11 frames. ], batch size: 17, lr: 3.56e-03
2024-08-06 12:07:38,135 INFO [trainer.py:765] (0/8) Epoch 24, batch 1200, train_loss[loss=3.367, NarTop10Accuracy=0.6449, over 7076.00 frames. ], tot_loss[loss=3.404, NarTop10Accuracy=0.6372, over 5942.14 frames. ], batch size: 30, lr: 3.56e-03
2024-08-06 12:08:20,732 INFO [trainer.py:765] (0/8) Epoch 24, batch 1300, train_loss[loss=2.934, NarTop10Accuracy=0.7239, over 4987.00 frames. ], tot_loss[loss=3.4, NarTop10Accuracy=0.6377, over 6006.03 frames. ], batch size: 6, lr: 3.56e-03
2024-08-06 12:08:56,066 INFO [trainer.py:765] (0/8) Epoch 24, batch 1400, train_loss[loss=3.23, NarTop10Accuracy=0.6597, over 6302.00 frames. ], tot_loss[loss=3.418, NarTop10Accuracy=0.6347, over 6026.26 frames. ], batch size: 11, lr: 3.56e-03
2024-08-06 12:09:24,343 INFO [trainer.py:765] (0/8) Epoch 24, batch 1500, train_loss[loss=3.669, NarTop10Accuracy=0.5855, over 6320.00 frames. ], tot_loss[loss=3.423, NarTop10Accuracy=0.6337, over 5961.32 frames. ], batch size: 49, lr: 3.55e-03
2024-08-06 12:09:52,525 INFO [trainer.py:765] (0/8) Epoch 24, batch 1600, train_loss[loss=3.292, NarTop10Accuracy=0.6592, over 7095.00 frames. ], tot_loss[loss=3.415, NarTop10Accuracy=0.635, over 5943.75 frames. ], batch size: 22, lr: 3.55e-03
2024-08-06 12:10:00,006 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-61000.pt
2024-08-06 12:10:22,546 INFO [trainer.py:765] (0/8) Epoch 24, batch 1700, train_loss[loss=3.668, NarTop10Accuracy=0.5776, over 6313.00 frames. ], tot_loss[loss=3.424, NarTop10Accuracy=0.6328, over 5928.34 frames. ], batch size: 13, lr: 3.55e-03
2024-08-06 12:10:49,273 INFO [trainer.py:765] (0/8) Epoch 24, batch 1800, train_loss[loss=3.347, NarTop10Accuracy=0.6509, over 7089.00 frames. ], tot_loss[loss=3.417, NarTop10Accuracy=0.6343, over 6006.83 frames. ], batch size: 22, lr: 3.54e-03
2024-08-06 12:11:15,847 INFO [trainer.py:765] (0/8) Epoch 24, batch 1900, train_loss[loss=3.523, NarTop10Accuracy=0.6194, over 5422.00 frames. ], tot_loss[loss=3.421, NarTop10Accuracy=0.6335, over 6029.65 frames. ], batch size: 48, lr: 3.54e-03
2024-08-06 12:11:41,667 INFO [trainer.py:765] (0/8) Epoch 24, batch 2000, train_loss[loss=3.4, NarTop10Accuracy=0.6412, over 6079.00 frames. ], tot_loss[loss=3.416, NarTop10Accuracy=0.6347, over 6008.55 frames. ], batch size: 48, lr: 3.54e-03
2024-08-06 12:12:07,104 INFO [trainer.py:765] (0/8) Epoch 24, batch 2100, train_loss[loss=3.2, NarTop10Accuracy=0.6913, over 3955.00 frames. ], tot_loss[loss=3.41, NarTop10Accuracy=0.6359, over 5987.42 frames. ], batch size: 4, lr: 3.54e-03
2024-08-06 12:12:33,373 INFO [trainer.py:765] (0/8) Epoch 24, batch 2200, train_loss[loss=3.484, NarTop10Accuracy=0.6189, over 7429.00 frames. ], tot_loss[loss=3.409, NarTop10Accuracy=0.6357, over 6032.35 frames. ], batch size: 31, lr: 3.53e-03
2024-08-06 12:12:58,773 INFO [trainer.py:765] (0/8) Epoch 24, batch 2300, train_loss[loss=3.661, NarTop10Accuracy=0.5825, over 5726.00 frames. ], tot_loss[loss=3.43, NarTop10Accuracy=0.6317, over 6060.25 frames. ], batch size: 9, lr: 3.53e-03
2024-08-06 12:13:23,488 INFO [trainer.py:765] (0/8) Epoch 24, batch 2400, train_loss[loss=3.602, NarTop10Accuracy=0.5975, over 6204.00 frames. ], tot_loss[loss=3.439, NarTop10Accuracy=0.6301, over 5869.15 frames. ], batch size: 49, lr: 3.53e-03
2024-08-06 12:13:47,006 INFO [trainer.py:765] (0/8) Epoch 24, batch 2500, train_loss[loss=3.473, NarTop10Accuracy=0.6325, over 4344.00 frames. ], tot_loss[loss=3.407, NarTop10Accuracy=0.6365, over 5518.77 frames. ], batch size: 5, lr: 3.52e-03
2024-08-06 12:14:08,384 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 12:14:08,388 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-24.pt
2024-08-06 12:14:50,196 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-62000.pt
2024-08-06 12:14:53,735 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 12:15:00,657 INFO [trainer.py:811] (0/8) Epoch 25, validation: loss=3.279, NarTop10Accuracy=0.6656, over 1907754.00 frames. 
2024-08-06 12:15:00,658 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 12:15:01,363 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.547e+02 1.921e+02 2.068e+02 2.276e+02 6.228e+02, threshold=4.136e+02, percent-clipped=0.3
2024-08-06 12:15:17,917 INFO [trainer.py:765] (0/8) Epoch 25, batch 100, train_loss[loss=3.175, NarTop10Accuracy=0.682, over 7596.00 frames. ], tot_loss[loss=3.364, NarTop10Accuracy=0.6472, over 2368.20 frames. ], batch size: 31, lr: 3.45e-03
2024-08-06 12:15:53,499 INFO [trainer.py:765] (0/8) Epoch 25, batch 200, train_loss[loss=3.254, NarTop10Accuracy=0.6574, over 6736.00 frames. ], tot_loss[loss=3.366, NarTop10Accuracy=0.6468, over 3866.68 frames. ], batch size: 17, lr: 3.44e-03
2024-08-06 12:16:23,596 INFO [trainer.py:765] (0/8) Epoch 25, batch 300, train_loss[loss=3.413, NarTop10Accuracy=0.6361, over 7201.00 frames. ], tot_loss[loss=3.363, NarTop10Accuracy=0.6466, over 4674.95 frames. ], batch size: 22, lr: 3.44e-03
2024-08-06 12:16:59,163 INFO [trainer.py:765] (0/8) Epoch 25, batch 400, train_loss[loss=3.491, NarTop10Accuracy=0.6199, over 5014.00 frames. ], tot_loss[loss=3.363, NarTop10Accuracy=0.646, over 5145.83 frames. ], batch size: 7, lr: 3.44e-03
2024-08-06 12:17:32,096 INFO [trainer.py:765] (0/8) Epoch 25, batch 500, train_loss[loss=3.289, NarTop10Accuracy=0.6564, over 6148.00 frames. ], tot_loss[loss=3.356, NarTop10Accuracy=0.6475, over 5416.46 frames. ], batch size: 11, lr: 3.44e-03
2024-08-06 12:18:05,181 INFO [trainer.py:765] (0/8) Epoch 25, batch 600, train_loss[loss=3.236, NarTop10Accuracy=0.6757, over 5904.00 frames. ], tot_loss[loss=3.352, NarTop10Accuracy=0.648, over 5679.35 frames. ], batch size: 9, lr: 3.43e-03
2024-08-06 12:18:39,598 INFO [trainer.py:765] (0/8) Epoch 25, batch 700, train_loss[loss=3.18, NarTop10Accuracy=0.6864, over 5075.00 frames. ], tot_loss[loss=3.359, NarTop10Accuracy=0.6465, over 5740.40 frames. ], batch size: 6, lr: 3.43e-03
2024-08-06 12:19:16,015 INFO [trainer.py:765] (0/8) Epoch 25, batch 800, train_loss[loss=3.039, NarTop10Accuracy=0.7044, over 5054.00 frames. ], tot_loss[loss=3.37, NarTop10Accuracy=0.6444, over 5813.15 frames. ], batch size: 6, lr: 3.43e-03
2024-08-06 12:19:49,558 INFO [trainer.py:765] (0/8) Epoch 25, batch 900, train_loss[loss=3.128, NarTop10Accuracy=0.6871, over 6783.00 frames. ], tot_loss[loss=3.381, NarTop10Accuracy=0.6419, over 5842.67 frames. ], batch size: 14, lr: 3.43e-03
2024-08-06 12:20:23,876 INFO [trainer.py:765] (0/8) Epoch 25, batch 1000, train_loss[loss=3.291, NarTop10Accuracy=0.6531, over 6567.00 frames. ], tot_loss[loss=3.386, NarTop10Accuracy=0.6409, over 5923.10 frames. ], batch size: 14, lr: 3.42e-03
2024-08-06 12:20:40,427 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-63000.pt
2024-08-06 12:21:01,915 INFO [trainer.py:765] (0/8) Epoch 25, batch 1100, train_loss[loss=3.252, NarTop10Accuracy=0.6679, over 6903.00 frames. ], tot_loss[loss=3.399, NarTop10Accuracy=0.6384, over 5951.97 frames. ], batch size: 17, lr: 3.42e-03
2024-08-06 12:21:40,638 INFO [trainer.py:765] (0/8) Epoch 25, batch 1200, train_loss[loss=3.361, NarTop10Accuracy=0.6471, over 7144.00 frames. ], tot_loss[loss=3.383, NarTop10Accuracy=0.6416, over 5942.89 frames. ], batch size: 30, lr: 3.42e-03
2024-08-06 12:22:11,838 INFO [trainer.py:765] (0/8) Epoch 25, batch 1300, train_loss[loss=3.506, NarTop10Accuracy=0.6173, over 5256.00 frames. ], tot_loss[loss=3.382, NarTop10Accuracy=0.6416, over 6008.06 frames. ], batch size: 6, lr: 3.41e-03
2024-08-06 12:22:48,550 INFO [trainer.py:765] (0/8) Epoch 25, batch 1400, train_loss[loss=3.675, NarTop10Accuracy=0.5732, over 6245.00 frames. ], tot_loss[loss=3.391, NarTop10Accuracy=0.6399, over 6023.47 frames. ], batch size: 11, lr: 3.41e-03
2024-08-06 12:23:21,655 INFO [trainer.py:765] (0/8) Epoch 25, batch 1500, train_loss[loss=3.794, NarTop10Accuracy=0.5587, over 6209.00 frames. ], tot_loss[loss=3.395, NarTop10Accuracy=0.6391, over 5957.19 frames. ], batch size: 50, lr: 3.41e-03
2024-08-06 12:23:49,717 INFO [trainer.py:765] (0/8) Epoch 25, batch 1600, train_loss[loss=3.219, NarTop10Accuracy=0.6739, over 7142.00 frames. ], tot_loss[loss=3.404, NarTop10Accuracy=0.6372, over 5943.12 frames. ], batch size: 22, lr: 3.41e-03
2024-08-06 12:24:16,373 INFO [trainer.py:765] (0/8) Epoch 25, batch 1700, train_loss[loss=3.579, NarTop10Accuracy=0.5962, over 6340.00 frames. ], tot_loss[loss=3.419, NarTop10Accuracy=0.6343, over 5929.72 frames. ], batch size: 13, lr: 3.40e-03
2024-08-06 12:24:43,092 INFO [trainer.py:765] (0/8) Epoch 25, batch 1800, train_loss[loss=3.304, NarTop10Accuracy=0.6572, over 7249.00 frames. ], tot_loss[loss=3.419, NarTop10Accuracy=0.6346, over 5992.52 frames. ], batch size: 22, lr: 3.40e-03
2024-08-06 12:25:09,776 INFO [trainer.py:765] (0/8) Epoch 25, batch 1900, train_loss[loss=3.558, NarTop10Accuracy=0.6057, over 5963.00 frames. ], tot_loss[loss=3.44, NarTop10Accuracy=0.6301, over 6027.94 frames. ], batch size: 49, lr: 3.40e-03
2024-08-06 12:25:35,710 INFO [trainer.py:765] (0/8) Epoch 25, batch 2000, train_loss[loss=3.551, NarTop10Accuracy=0.6098, over 6046.00 frames. ], tot_loss[loss=3.413, NarTop10Accuracy=0.6356, over 6021.79 frames. ], batch size: 49, lr: 3.40e-03
2024-08-06 12:25:47,854 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-64000.pt
2024-08-06 12:25:52,168 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 12:25:58,846 INFO [trainer.py:811] (0/8) Epoch 25, validation: loss=3.265, NarTop10Accuracy=0.667, over 1907754.00 frames. 
2024-08-06 12:25:58,847 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 12:25:59,344 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.566e+02 1.947e+02 2.092e+02 2.280e+02 8.190e+02, threshold=4.185e+02, percent-clipped=0.2
2024-08-06 12:26:12,224 INFO [trainer.py:765] (0/8) Epoch 25, batch 2100, train_loss[loss=3.59, NarTop10Accuracy=0.6045, over 3937.00 frames. ], tot_loss[loss=3.413, NarTop10Accuracy=0.6354, over 5997.51 frames. ], batch size: 4, lr: 3.39e-03
2024-08-06 12:26:37,833 INFO [trainer.py:765] (0/8) Epoch 25, batch 2200, train_loss[loss=3.536, NarTop10Accuracy=0.6127, over 7316.00 frames. ], tot_loss[loss=3.408, NarTop10Accuracy=0.6367, over 6036.51 frames. ], batch size: 31, lr: 3.39e-03
2024-08-06 12:27:03,344 INFO [trainer.py:765] (0/8) Epoch 25, batch 2300, train_loss[loss=3.411, NarTop10Accuracy=0.6307, over 5956.00 frames. ], tot_loss[loss=3.415, NarTop10Accuracy=0.6351, over 6062.71 frames. ], batch size: 9, lr: 3.39e-03
2024-08-06 12:27:28,150 INFO [trainer.py:765] (0/8) Epoch 25, batch 2400, train_loss[loss=3.548, NarTop10Accuracy=0.6011, over 6181.00 frames. ], tot_loss[loss=3.428, NarTop10Accuracy=0.6326, over 5877.93 frames. ], batch size: 50, lr: 3.39e-03
2024-08-06 12:27:51,732 INFO [trainer.py:765] (0/8) Epoch 25, batch 2500, train_loss[loss=3.349, NarTop10Accuracy=0.6482, over 4943.00 frames. ], tot_loss[loss=3.406, NarTop10Accuracy=0.6365, over 5535.07 frames. ], batch size: 6, lr: 3.38e-03
2024-08-06 12:28:12,990 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 12:28:12,992 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-25.pt
2024-08-06 12:29:08,881 INFO [trainer.py:765] (0/8) Epoch 26, batch 100, train_loss[loss=3.59, NarTop10Accuracy=0.6065, over 7072.00 frames. ], tot_loss[loss=3.338, NarTop10Accuracy=0.6519, over 2370.30 frames. ], batch size: 30, lr: 3.31e-03
2024-08-06 12:29:44,319 INFO [trainer.py:765] (0/8) Epoch 26, batch 200, train_loss[loss=3.155, NarTop10Accuracy=0.6819, over 6964.00 frames. ], tot_loss[loss=3.345, NarTop10Accuracy=0.6495, over 3865.67 frames. ], batch size: 17, lr: 3.31e-03
2024-08-06 12:30:19,754 INFO [trainer.py:765] (0/8) Epoch 26, batch 300, train_loss[loss=3.222, NarTop10Accuracy=0.6702, over 6967.00 frames. ], tot_loss[loss=3.355, NarTop10Accuracy=0.6472, over 4666.91 frames. ], batch size: 22, lr: 3.31e-03
2024-08-06 12:30:52,510 INFO [trainer.py:765] (0/8) Epoch 26, batch 400, train_loss[loss=3.031, NarTop10Accuracy=0.7061, over 5159.00 frames. ], tot_loss[loss=3.35, NarTop10Accuracy=0.6483, over 5128.15 frames. ], batch size: 7, lr: 3.30e-03
2024-08-06 12:31:11,050 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-65000.pt
2024-08-06 12:31:26,531 INFO [trainer.py:765] (0/8) Epoch 26, batch 500, train_loss[loss=3.356, NarTop10Accuracy=0.6521, over 6111.00 frames. ], tot_loss[loss=3.354, NarTop10Accuracy=0.6479, over 5416.03 frames. ], batch size: 11, lr: 3.30e-03
2024-08-06 12:31:59,782 INFO [trainer.py:765] (0/8) Epoch 26, batch 600, train_loss[loss=3.536, NarTop10Accuracy=0.6105, over 5730.00 frames. ], tot_loss[loss=3.367, NarTop10Accuracy=0.645, over 5673.50 frames. ], batch size: 9, lr: 3.30e-03
2024-08-06 12:32:36,967 INFO [trainer.py:765] (0/8) Epoch 26, batch 700, train_loss[loss=3.292, NarTop10Accuracy=0.6623, over 5004.00 frames. ], tot_loss[loss=3.373, NarTop10Accuracy=0.6444, over 5741.30 frames. ], batch size: 6, lr: 3.30e-03
2024-08-06 12:33:10,809 INFO [trainer.py:765] (0/8) Epoch 26, batch 800, train_loss[loss=3.399, NarTop10Accuracy=0.6487, over 5086.00 frames. ], tot_loss[loss=3.378, NarTop10Accuracy=0.6434, over 5806.52 frames. ], batch size: 6, lr: 3.29e-03
2024-08-06 12:33:46,257 INFO [trainer.py:765] (0/8) Epoch 26, batch 900, train_loss[loss=3.387, NarTop10Accuracy=0.6358, over 6281.00 frames. ], tot_loss[loss=3.393, NarTop10Accuracy=0.6403, over 5802.97 frames. ], batch size: 13, lr: 3.29e-03
2024-08-06 12:34:22,902 INFO [trainer.py:765] (0/8) Epoch 26, batch 1000, train_loss[loss=3.246, NarTop10Accuracy=0.6637, over 6331.00 frames. ], tot_loss[loss=3.387, NarTop10Accuracy=0.6406, over 5916.79 frames. ], batch size: 13, lr: 3.29e-03
2024-08-06 12:34:57,798 INFO [trainer.py:765] (0/8) Epoch 26, batch 1100, train_loss[loss=3.29, NarTop10Accuracy=0.6586, over 6893.00 frames. ], tot_loss[loss=3.38, NarTop10Accuracy=0.6419, over 5958.16 frames. ], batch size: 17, lr: 3.29e-03
2024-08-06 12:35:31,893 INFO [trainer.py:765] (0/8) Epoch 26, batch 1200, train_loss[loss=3.245, NarTop10Accuracy=0.6622, over 7381.00 frames. ], tot_loss[loss=3.37, NarTop10Accuracy=0.6436, over 5952.05 frames. ], batch size: 31, lr: 3.28e-03
2024-08-06 12:36:10,658 INFO [trainer.py:765] (0/8) Epoch 26, batch 1300, train_loss[loss=3.382, NarTop10Accuracy=0.6373, over 5191.00 frames. ], tot_loss[loss=3.373, NarTop10Accuracy=0.6436, over 6024.13 frames. ], batch size: 6, lr: 3.28e-03
2024-08-06 12:36:44,564 INFO [trainer.py:765] (0/8) Epoch 26, batch 1400, train_loss[loss=3.28, NarTop10Accuracy=0.6603, over 6104.00 frames. ], tot_loss[loss=3.383, NarTop10Accuracy=0.6414, over 6033.83 frames. ], batch size: 11, lr: 3.28e-03
2024-08-06 12:37:03,593 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-66000.pt
2024-08-06 12:37:06,979 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 12:37:13,567 INFO [trainer.py:811] (0/8) Epoch 26, validation: loss=3.231, NarTop10Accuracy=0.6753, over 1907754.00 frames. 
2024-08-06 12:37:13,568 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 12:37:14,078 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.547e+02 1.928e+02 2.102e+02 2.299e+02 4.602e+02, threshold=4.203e+02, percent-clipped=0.2
2024-08-06 12:37:23,028 INFO [trainer.py:765] (0/8) Epoch 26, batch 1500, train_loss[loss=3.621, NarTop10Accuracy=0.5993, over 5962.00 frames. ], tot_loss[loss=3.38, NarTop10Accuracy=0.6423, over 5969.17 frames. ], batch size: 49, lr: 3.28e-03
2024-08-06 12:37:51,061 INFO [trainer.py:765] (0/8) Epoch 26, batch 1600, train_loss[loss=3.307, NarTop10Accuracy=0.6505, over 7184.00 frames. ], tot_loss[loss=3.372, NarTop10Accuracy=0.6433, over 5964.51 frames. ], batch size: 22, lr: 3.27e-03
2024-08-06 12:38:17,854 INFO [trainer.py:765] (0/8) Epoch 26, batch 1700, train_loss[loss=3.381, NarTop10Accuracy=0.6425, over 6215.00 frames. ], tot_loss[loss=3.394, NarTop10Accuracy=0.6389, over 5949.93 frames. ], batch size: 13, lr: 3.27e-03
2024-08-06 12:38:44,384 INFO [trainer.py:765] (0/8) Epoch 26, batch 1800, train_loss[loss=3.127, NarTop10Accuracy=0.687, over 7197.00 frames. ], tot_loss[loss=3.398, NarTop10Accuracy=0.6381, over 6010.34 frames. ], batch size: 22, lr: 3.27e-03
2024-08-06 12:39:10,952 INFO [trainer.py:765] (0/8) Epoch 26, batch 1900, train_loss[loss=3.663, NarTop10Accuracy=0.5815, over 5950.00 frames. ], tot_loss[loss=3.409, NarTop10Accuracy=0.6355, over 6058.80 frames. ], batch size: 49, lr: 3.27e-03
2024-08-06 12:39:36,610 INFO [trainer.py:765] (0/8) Epoch 26, batch 2000, train_loss[loss=3.537, NarTop10Accuracy=0.6101, over 5854.00 frames. ], tot_loss[loss=3.412, NarTop10Accuracy=0.6353, over 6004.51 frames. ], batch size: 49, lr: 3.26e-03
2024-08-06 12:40:02,148 INFO [trainer.py:765] (0/8) Epoch 26, batch 2100, train_loss[loss=3.054, NarTop10Accuracy=0.6994, over 4798.00 frames. ], tot_loss[loss=3.399, NarTop10Accuracy=0.6384, over 5977.94 frames. ], batch size: 5, lr: 3.26e-03
2024-08-06 12:40:27,759 INFO [trainer.py:765] (0/8) Epoch 26, batch 2200, train_loss[loss=3.341, NarTop10Accuracy=0.6507, over 7283.00 frames. ], tot_loss[loss=3.4, NarTop10Accuracy=0.638, over 6034.43 frames. ], batch size: 30, lr: 3.26e-03
2024-08-06 12:40:53,233 INFO [trainer.py:765] (0/8) Epoch 26, batch 2300, train_loss[loss=3.278, NarTop10Accuracy=0.6696, over 5823.00 frames. ], tot_loss[loss=3.412, NarTop10Accuracy=0.6362, over 6069.50 frames. ], batch size: 9, lr: 3.26e-03
2024-08-06 12:41:17,931 INFO [trainer.py:765] (0/8) Epoch 26, batch 2400, train_loss[loss=3.386, NarTop10Accuracy=0.6408, over 6094.00 frames. ], tot_loss[loss=3.417, NarTop10Accuracy=0.6347, over 5873.29 frames. ], batch size: 49, lr: 3.25e-03
2024-08-06 12:41:33,149 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-67000.pt
2024-08-06 12:41:44,478 INFO [trainer.py:765] (0/8) Epoch 26, batch 2500, train_loss[loss=3.235, NarTop10Accuracy=0.6787, over 5212.00 frames. ], tot_loss[loss=3.382, NarTop10Accuracy=0.6404, over 5526.74 frames. ], batch size: 6, lr: 3.25e-03
2024-08-06 12:42:05,393 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 12:42:05,396 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-26.pt
2024-08-06 12:43:12,533 INFO [trainer.py:765] (0/8) Epoch 27, batch 100, train_loss[loss=3.536, NarTop10Accuracy=0.6073, over 7347.00 frames. ], tot_loss[loss=3.35, NarTop10Accuracy=0.6496, over 2374.12 frames. ], batch size: 31, lr: 3.19e-03
2024-08-06 12:43:43,576 INFO [trainer.py:765] (0/8) Epoch 27, batch 200, train_loss[loss=3.621, NarTop10Accuracy=0.5893, over 6757.00 frames. ], tot_loss[loss=3.349, NarTop10Accuracy=0.6498, over 3870.67 frames. ], batch size: 17, lr: 3.18e-03
2024-08-06 12:44:13,786 INFO [trainer.py:765] (0/8) Epoch 27, batch 300, train_loss[loss=3.136, NarTop10Accuracy=0.6928, over 7135.00 frames. ], tot_loss[loss=3.333, NarTop10Accuracy=0.6524, over 4682.63 frames. ], batch size: 22, lr: 3.18e-03
2024-08-06 12:44:50,461 INFO [trainer.py:765] (0/8) Epoch 27, batch 400, train_loss[loss=3.169, NarTop10Accuracy=0.688, over 5094.00 frames. ], tot_loss[loss=3.333, NarTop10Accuracy=0.6523, over 5141.44 frames. ], batch size: 7, lr: 3.18e-03
2024-08-06 12:45:20,670 INFO [trainer.py:765] (0/8) Epoch 27, batch 500, train_loss[loss=3.283, NarTop10Accuracy=0.6601, over 6126.00 frames. ], tot_loss[loss=3.333, NarTop10Accuracy=0.6521, over 5411.59 frames. ], batch size: 11, lr: 3.18e-03
2024-08-06 12:45:55,260 INFO [trainer.py:765] (0/8) Epoch 27, batch 600, train_loss[loss=3.255, NarTop10Accuracy=0.6731, over 5782.00 frames. ], tot_loss[loss=3.349, NarTop10Accuracy=0.6489, over 5676.63 frames. ], batch size: 9, lr: 3.17e-03
2024-08-06 12:46:26,747 INFO [trainer.py:765] (0/8) Epoch 27, batch 700, train_loss[loss=3.567, NarTop10Accuracy=0.6038, over 4876.00 frames. ], tot_loss[loss=3.357, NarTop10Accuracy=0.6471, over 5744.84 frames. ], batch size: 6, lr: 3.17e-03
2024-08-06 12:47:05,016 INFO [trainer.py:765] (0/8) Epoch 27, batch 800, train_loss[loss=3.213, NarTop10Accuracy=0.677, over 5124.00 frames. ], tot_loss[loss=3.363, NarTop10Accuracy=0.6459, over 5805.60 frames. ], batch size: 6, lr: 3.17e-03
2024-08-06 12:47:32,741 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-68000.pt
2024-08-06 12:47:36,212 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 12:47:42,765 INFO [trainer.py:811] (0/8) Epoch 27, validation: loss=3.258, NarTop10Accuracy=0.6695, over 1907754.00 frames. 
2024-08-06 12:47:42,766 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 12:47:43,335 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.554e+02 1.939e+02 2.100e+02 2.298e+02 4.859e+02, threshold=4.201e+02, percent-clipped=0.2
2024-08-06 12:47:47,259 INFO [trainer.py:765] (0/8) Epoch 27, batch 900, train_loss[loss=3.36, NarTop10Accuracy=0.6528, over 6199.00 frames. ], tot_loss[loss=3.363, NarTop10Accuracy=0.6458, over 5807.90 frames. ], batch size: 13, lr: 3.17e-03
2024-08-06 12:48:22,862 INFO [trainer.py:765] (0/8) Epoch 27, batch 1000, train_loss[loss=3.252, NarTop10Accuracy=0.6626, over 6310.00 frames. ], tot_loss[loss=3.363, NarTop10Accuracy=0.6453, over 5919.09 frames. ], batch size: 13, lr: 3.17e-03
2024-08-06 12:48:58,085 INFO [trainer.py:765] (0/8) Epoch 27, batch 1100, train_loss[loss=3.666, NarTop10Accuracy=0.5907, over 6960.00 frames. ], tot_loss[loss=3.371, NarTop10Accuracy=0.6436, over 5955.92 frames. ], batch size: 17, lr: 3.16e-03
2024-08-06 12:49:34,896 INFO [trainer.py:765] (0/8) Epoch 27, batch 1200, train_loss[loss=3.218, NarTop10Accuracy=0.6802, over 7397.00 frames. ], tot_loss[loss=3.375, NarTop10Accuracy=0.643, over 5939.12 frames. ], batch size: 31, lr: 3.16e-03
2024-08-06 12:50:06,242 INFO [trainer.py:765] (0/8) Epoch 27, batch 1300, train_loss[loss=3.099, NarTop10Accuracy=0.6923, over 5115.00 frames. ], tot_loss[loss=3.379, NarTop10Accuracy=0.6421, over 6005.58 frames. ], batch size: 6, lr: 3.16e-03
2024-08-06 12:50:42,951 INFO [trainer.py:765] (0/8) Epoch 27, batch 1400, train_loss[loss=3.086, NarTop10Accuracy=0.7041, over 6169.00 frames. ], tot_loss[loss=3.383, NarTop10Accuracy=0.6409, over 6026.90 frames. ], batch size: 11, lr: 3.16e-03
2024-08-06 12:51:11,278 INFO [trainer.py:765] (0/8) Epoch 27, batch 1500, train_loss[loss=3.414, NarTop10Accuracy=0.6303, over 6381.00 frames. ], tot_loss[loss=3.381, NarTop10Accuracy=0.6411, over 5973.28 frames. ], batch size: 49, lr: 3.15e-03
2024-08-06 12:51:39,352 INFO [trainer.py:765] (0/8) Epoch 27, batch 1600, train_loss[loss=3.289, NarTop10Accuracy=0.6561, over 7102.00 frames. ], tot_loss[loss=3.388, NarTop10Accuracy=0.6398, over 5952.38 frames. ], batch size: 22, lr: 3.15e-03
2024-08-06 12:52:06,062 INFO [trainer.py:765] (0/8) Epoch 27, batch 1700, train_loss[loss=3.73, NarTop10Accuracy=0.5806, over 6210.00 frames. ], tot_loss[loss=3.392, NarTop10Accuracy=0.6392, over 5958.37 frames. ], batch size: 13, lr: 3.15e-03
2024-08-06 12:52:32,669 INFO [trainer.py:765] (0/8) Epoch 27, batch 1800, train_loss[loss=3.345, NarTop10Accuracy=0.6567, over 7270.00 frames. ], tot_loss[loss=3.386, NarTop10Accuracy=0.6409, over 6021.99 frames. ], batch size: 22, lr: 3.15e-03
2024-08-06 12:52:55,214 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-69000.pt
2024-08-06 12:53:02,289 INFO [trainer.py:765] (0/8) Epoch 27, batch 1900, train_loss[loss=3.648, NarTop10Accuracy=0.5913, over 6404.00 frames. ], tot_loss[loss=3.405, NarTop10Accuracy=0.637, over 6047.65 frames. ], batch size: 50, lr: 3.14e-03
2024-08-06 12:53:27,998 INFO [trainer.py:765] (0/8) Epoch 27, batch 2000, train_loss[loss=3.478, NarTop10Accuracy=0.6231, over 5784.00 frames. ], tot_loss[loss=3.4, NarTop10Accuracy=0.638, over 6027.85 frames. ], batch size: 49, lr: 3.14e-03
2024-08-06 12:53:53,538 INFO [trainer.py:765] (0/8) Epoch 27, batch 2100, train_loss[loss=3.719, NarTop10Accuracy=0.5732, over 4827.00 frames. ], tot_loss[loss=3.404, NarTop10Accuracy=0.637, over 6006.87 frames. ], batch size: 5, lr: 3.14e-03
2024-08-06 12:54:18,997 INFO [trainer.py:765] (0/8) Epoch 27, batch 2200, train_loss[loss=3.415, NarTop10Accuracy=0.6363, over 7208.00 frames. ], tot_loss[loss=3.399, NarTop10Accuracy=0.6381, over 6044.03 frames. ], batch size: 30, lr: 3.14e-03
2024-08-06 12:54:44,480 INFO [trainer.py:765] (0/8) Epoch 27, batch 2300, train_loss[loss=3.345, NarTop10Accuracy=0.6566, over 5865.00 frames. ], tot_loss[loss=3.411, NarTop10Accuracy=0.6357, over 6068.95 frames. ], batch size: 9, lr: 3.14e-03
2024-08-06 12:55:09,218 INFO [trainer.py:765] (0/8) Epoch 27, batch 2400, train_loss[loss=3.455, NarTop10Accuracy=0.6234, over 5887.00 frames. ], tot_loss[loss=3.424, NarTop10Accuracy=0.6329, over 5893.94 frames. ], batch size: 48, lr: 3.13e-03
2024-08-06 12:55:32,726 INFO [trainer.py:765] (0/8) Epoch 27, batch 2500, train_loss[loss=3.173, NarTop10Accuracy=0.6833, over 5136.00 frames. ], tot_loss[loss=3.387, NarTop10Accuracy=0.6399, over 5549.09 frames. ], batch size: 6, lr: 3.13e-03
2024-08-06 12:55:54,286 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 12:55:54,288 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-27.pt
2024-08-06 12:56:46,803 INFO [trainer.py:765] (0/8) Epoch 28, batch 100, train_loss[loss=3.218, NarTop10Accuracy=0.6725, over 7231.00 frames. ], tot_loss[loss=3.334, NarTop10Accuracy=0.652, over 2368.27 frames. ], batch size: 30, lr: 3.07e-03
2024-08-06 12:57:23,205 INFO [trainer.py:765] (0/8) Epoch 28, batch 200, train_loss[loss=3.393, NarTop10Accuracy=0.6472, over 6872.00 frames. ], tot_loss[loss=3.335, NarTop10Accuracy=0.6523, over 3865.33 frames. ], batch size: 17, lr: 3.07e-03
2024-08-06 12:57:55,704 INFO [trainer.py:765] (0/8) Epoch 28, batch 300, train_loss[loss=3.343, NarTop10Accuracy=0.6469, over 7150.00 frames. ], tot_loss[loss=3.349, NarTop10Accuracy=0.6497, over 4667.55 frames. ], batch size: 22, lr: 3.07e-03
2024-08-06 12:57:56,456 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-70000.pt
2024-08-06 12:58:00,124 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 12:58:06,828 INFO [trainer.py:811] (0/8) Epoch 28, validation: loss=3.275, NarTop10Accuracy=0.6665, over 1907754.00 frames. 
2024-08-06 12:58:06,828 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 12:58:07,333 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.570e+02 1.944e+02 2.106e+02 2.298e+02 4.786e+02, threshold=4.211e+02, percent-clipped=0.1
2024-08-06 12:58:34,932 INFO [trainer.py:765] (0/8) Epoch 28, batch 400, train_loss[loss=3.41, NarTop10Accuracy=0.6251, over 5131.00 frames. ], tot_loss[loss=3.356, NarTop10Accuracy=0.6483, over 5128.33 frames. ], batch size: 7, lr: 3.06e-03
2024-08-06 12:59:11,437 INFO [trainer.py:765] (0/8) Epoch 28, batch 500, train_loss[loss=3.212, NarTop10Accuracy=0.6827, over 6254.00 frames. ], tot_loss[loss=3.347, NarTop10Accuracy=0.6499, over 5403.59 frames. ], batch size: 11, lr: 3.06e-03
2024-08-06 12:59:44,487 INFO [trainer.py:765] (0/8) Epoch 28, batch 600, train_loss[loss=3.206, NarTop10Accuracy=0.6761, over 5758.00 frames. ], tot_loss[loss=3.343, NarTop10Accuracy=0.6501, over 5661.77 frames. ], batch size: 9, lr: 3.06e-03
2024-08-06 13:00:20,012 INFO [trainer.py:765] (0/8) Epoch 28, batch 700, train_loss[loss=3.245, NarTop10Accuracy=0.6625, over 5211.00 frames. ], tot_loss[loss=3.349, NarTop10Accuracy=0.6487, over 5736.47 frames. ], batch size: 6, lr: 3.06e-03
2024-08-06 13:00:56,433 INFO [trainer.py:765] (0/8) Epoch 28, batch 800, train_loss[loss=3.408, NarTop10Accuracy=0.6372, over 5134.00 frames. ], tot_loss[loss=3.344, NarTop10Accuracy=0.6495, over 5798.47 frames. ], batch size: 6, lr: 3.05e-03
2024-08-06 13:01:31,042 INFO [trainer.py:765] (0/8) Epoch 28, batch 900, train_loss[loss=3.311, NarTop10Accuracy=0.6647, over 6239.00 frames. ], tot_loss[loss=3.354, NarTop10Accuracy=0.6473, over 5812.26 frames. ], batch size: 13, lr: 3.05e-03
2024-08-06 13:02:06,494 INFO [trainer.py:765] (0/8) Epoch 28, batch 1000, train_loss[loss=3.462, NarTop10Accuracy=0.6203, over 6336.00 frames. ], tot_loss[loss=3.369, NarTop10Accuracy=0.6442, over 5918.42 frames. ], batch size: 13, lr: 3.05e-03
2024-08-06 13:02:41,229 INFO [trainer.py:765] (0/8) Epoch 28, batch 1100, train_loss[loss=3.334, NarTop10Accuracy=0.6502, over 6863.00 frames. ], tot_loss[loss=3.371, NarTop10Accuracy=0.6437, over 5937.21 frames. ], batch size: 17, lr: 3.05e-03
2024-08-06 13:03:16,895 INFO [trainer.py:765] (0/8) Epoch 28, batch 1200, train_loss[loss=3.52, NarTop10Accuracy=0.6205, over 7466.00 frames. ], tot_loss[loss=3.381, NarTop10Accuracy=0.6414, over 5940.39 frames. ], batch size: 31, lr: 3.05e-03
2024-08-06 13:03:54,153 INFO [trainer.py:765] (0/8) Epoch 28, batch 1300, train_loss[loss=3.371, NarTop10Accuracy=0.6456, over 5117.00 frames. ], tot_loss[loss=3.38, NarTop10Accuracy=0.6415, over 5998.56 frames. ], batch size: 6, lr: 3.04e-03
2024-08-06 13:03:54,981 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-71000.pt
2024-08-06 13:04:28,712 INFO [trainer.py:765] (0/8) Epoch 28, batch 1400, train_loss[loss=3.595, NarTop10Accuracy=0.6051, over 6283.00 frames. ], tot_loss[loss=3.391, NarTop10Accuracy=0.6393, over 6033.16 frames. ], batch size: 11, lr: 3.04e-03
2024-08-06 13:05:02,349 INFO [trainer.py:765] (0/8) Epoch 28, batch 1500, train_loss[loss=3.413, NarTop10Accuracy=0.6369, over 6064.00 frames. ], tot_loss[loss=3.379, NarTop10Accuracy=0.6421, over 5984.40 frames. ], batch size: 48, lr: 3.04e-03
2024-08-06 13:05:30,371 INFO [trainer.py:765] (0/8) Epoch 28, batch 1600, train_loss[loss=3.529, NarTop10Accuracy=0.6094, over 7062.00 frames. ], tot_loss[loss=3.399, NarTop10Accuracy=0.6384, over 5954.51 frames. ], batch size: 22, lr: 3.04e-03
2024-08-06 13:05:57,131 INFO [trainer.py:765] (0/8) Epoch 28, batch 1700, train_loss[loss=3.598, NarTop10Accuracy=0.5925, over 6278.00 frames. ], tot_loss[loss=3.39, NarTop10Accuracy=0.6401, over 5934.82 frames. ], batch size: 13, lr: 3.04e-03
2024-08-06 13:06:23,732 INFO [trainer.py:765] (0/8) Epoch 28, batch 1800, train_loss[loss=3.602, NarTop10Accuracy=0.5948, over 7264.00 frames. ], tot_loss[loss=3.377, NarTop10Accuracy=0.6427, over 6007.83 frames. ], batch size: 22, lr: 3.03e-03
2024-08-06 13:06:50,373 INFO [trainer.py:765] (0/8) Epoch 28, batch 1900, train_loss[loss=3.477, NarTop10Accuracy=0.6202, over 5347.00 frames. ], tot_loss[loss=3.383, NarTop10Accuracy=0.6411, over 6038.12 frames. ], batch size: 49, lr: 3.03e-03
2024-08-06 13:07:16,115 INFO [trainer.py:765] (0/8) Epoch 28, batch 2000, train_loss[loss=3.546, NarTop10Accuracy=0.6063, over 5795.00 frames. ], tot_loss[loss=3.379, NarTop10Accuracy=0.6418, over 6029.68 frames. ], batch size: 49, lr: 3.03e-03
2024-08-06 13:07:41,546 INFO [trainer.py:765] (0/8) Epoch 28, batch 2100, train_loss[loss=3.361, NarTop10Accuracy=0.6198, over 4699.00 frames. ], tot_loss[loss=3.392, NarTop10Accuracy=0.6395, over 6011.65 frames. ], batch size: 5, lr: 3.03e-03
2024-08-06 13:08:06,931 INFO [trainer.py:765] (0/8) Epoch 28, batch 2200, train_loss[loss=3.502, NarTop10Accuracy=0.611, over 7304.00 frames. ], tot_loss[loss=3.395, NarTop10Accuracy=0.6384, over 6047.96 frames. ], batch size: 30, lr: 3.02e-03
2024-08-06 13:08:32,387 INFO [trainer.py:765] (0/8) Epoch 28, batch 2300, train_loss[loss=3.217, NarTop10Accuracy=0.6891, over 5627.00 frames. ], tot_loss[loss=3.404, NarTop10Accuracy=0.6369, over 6063.71 frames. ], batch size: 9, lr: 3.02e-03
2024-08-06 13:08:33,134 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-72000.pt
2024-08-06 13:08:36,743 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 13:08:43,385 INFO [trainer.py:811] (0/8) Epoch 28, validation: loss=3.224, NarTop10Accuracy=0.676, over 1907754.00 frames. 
2024-08-06 13:08:43,385 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 13:08:43,890 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.605e+02 1.997e+02 2.131e+02 2.314e+02 6.875e+02, threshold=4.261e+02, percent-clipped=0.5
2024-08-06 13:09:07,390 INFO [trainer.py:765] (0/8) Epoch 28, batch 2400, train_loss[loss=3.849, NarTop10Accuracy=0.5492, over 5564.00 frames. ], tot_loss[loss=3.417, NarTop10Accuracy=0.6347, over 5876.76 frames. ], batch size: 49, lr: 3.02e-03
2024-08-06 13:09:30,781 INFO [trainer.py:765] (0/8) Epoch 28, batch 2500, train_loss[loss=3.664, NarTop10Accuracy=0.5849, over 4990.00 frames. ], tot_loss[loss=3.391, NarTop10Accuracy=0.6392, over 5546.55 frames. ], batch size: 6, lr: 3.02e-03
2024-08-06 13:09:51,804 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 13:09:51,807 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-28.pt
2024-08-06 13:10:48,192 INFO [trainer.py:765] (0/8) Epoch 29, batch 100, train_loss[loss=3.61, NarTop10Accuracy=0.5972, over 7216.00 frames. ], tot_loss[loss=3.338, NarTop10Accuracy=0.6522, over 2371.98 frames. ], batch size: 30, lr: 2.96e-03
2024-08-06 13:11:20,840 INFO [trainer.py:765] (0/8) Epoch 29, batch 200, train_loss[loss=3.525, NarTop10Accuracy=0.6256, over 7002.00 frames. ], tot_loss[loss=3.336, NarTop10Accuracy=0.6519, over 3879.15 frames. ], batch size: 17, lr: 2.96e-03
2024-08-06 13:11:56,949 INFO [trainer.py:765] (0/8) Epoch 29, batch 300, train_loss[loss=3.156, NarTop10Accuracy=0.6906, over 7269.00 frames. ], tot_loss[loss=3.331, NarTop10Accuracy=0.6532, over 4668.84 frames. ], batch size: 22, lr: 2.96e-03
2024-08-06 13:12:29,716 INFO [trainer.py:765] (0/8) Epoch 29, batch 400, train_loss[loss=3.11, NarTop10Accuracy=0.6906, over 5207.00 frames. ], tot_loss[loss=3.331, NarTop10Accuracy=0.6528, over 5121.79 frames. ], batch size: 7, lr: 2.96e-03
2024-08-06 13:12:59,920 INFO [trainer.py:765] (0/8) Epoch 29, batch 500, train_loss[loss=3.421, NarTop10Accuracy=0.6339, over 6243.00 frames. ], tot_loss[loss=3.332, NarTop10Accuracy=0.6527, over 5399.26 frames. ], batch size: 11, lr: 2.95e-03
2024-08-06 13:13:33,546 INFO [trainer.py:765] (0/8) Epoch 29, batch 600, train_loss[loss=3.518, NarTop10Accuracy=0.5953, over 5719.00 frames. ], tot_loss[loss=3.341, NarTop10Accuracy=0.6506, over 5665.15 frames. ], batch size: 9, lr: 2.95e-03
2024-08-06 13:14:09,936 INFO [trainer.py:765] (0/8) Epoch 29, batch 700, train_loss[loss=3.457, NarTop10Accuracy=0.6255, over 5019.00 frames. ], tot_loss[loss=3.347, NarTop10Accuracy=0.6487, over 5740.43 frames. ], batch size: 6, lr: 2.95e-03
2024-08-06 13:14:16,395 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-73000.pt
2024-08-06 13:14:46,677 INFO [trainer.py:765] (0/8) Epoch 29, batch 800, train_loss[loss=3.469, NarTop10Accuracy=0.6296, over 5239.00 frames. ], tot_loss[loss=3.37, NarTop10Accuracy=0.6441, over 5801.84 frames. ], batch size: 6, lr: 2.95e-03
2024-08-06 13:15:17,113 INFO [trainer.py:765] (0/8) Epoch 29, batch 900, train_loss[loss=3.446, NarTop10Accuracy=0.6296, over 6111.00 frames. ], tot_loss[loss=3.364, NarTop10Accuracy=0.6452, over 5802.20 frames. ], batch size: 13, lr: 2.95e-03
2024-08-06 13:15:59,363 INFO [trainer.py:765] (0/8) Epoch 29, batch 1000, train_loss[loss=3.64, NarTop10Accuracy=0.5853, over 6269.00 frames. ], tot_loss[loss=3.371, NarTop10Accuracy=0.6436, over 5907.58 frames. ], batch size: 13, lr: 2.94e-03
2024-08-06 13:16:31,712 INFO [trainer.py:765] (0/8) Epoch 29, batch 1100, train_loss[loss=3.34, NarTop10Accuracy=0.6505, over 6731.00 frames. ], tot_loss[loss=3.379, NarTop10Accuracy=0.6416, over 5971.35 frames. ], batch size: 17, lr: 2.94e-03
2024-08-06 13:17:04,933 INFO [trainer.py:765] (0/8) Epoch 29, batch 1200, train_loss[loss=3.482, NarTop10Accuracy=0.6249, over 7012.00 frames. ], tot_loss[loss=3.375, NarTop10Accuracy=0.6427, over 5966.53 frames. ], batch size: 30, lr: 2.94e-03
2024-08-06 13:17:43,956 INFO [trainer.py:765] (0/8) Epoch 29, batch 1300, train_loss[loss=3.261, NarTop10Accuracy=0.669, over 5009.00 frames. ], tot_loss[loss=3.372, NarTop10Accuracy=0.6431, over 6026.85 frames. ], batch size: 6, lr: 2.94e-03
2024-08-06 13:18:17,924 INFO [trainer.py:765] (0/8) Epoch 29, batch 1400, train_loss[loss=3.524, NarTop10Accuracy=0.6131, over 6297.00 frames. ], tot_loss[loss=3.392, NarTop10Accuracy=0.6393, over 6042.53 frames. ], batch size: 11, lr: 2.94e-03
2024-08-06 13:18:48,305 INFO [trainer.py:765] (0/8) Epoch 29, batch 1500, train_loss[loss=3.783, NarTop10Accuracy=0.5661, over 5946.00 frames. ], tot_loss[loss=3.385, NarTop10Accuracy=0.6409, over 5961.34 frames. ], batch size: 49, lr: 2.93e-03
2024-08-06 13:19:16,408 INFO [trainer.py:765] (0/8) Epoch 29, batch 1600, train_loss[loss=3.271, NarTop10Accuracy=0.6591, over 7216.00 frames. ], tot_loss[loss=3.382, NarTop10Accuracy=0.6415, over 5957.26 frames. ], batch size: 22, lr: 2.93e-03
2024-08-06 13:19:43,241 INFO [trainer.py:765] (0/8) Epoch 29, batch 1700, train_loss[loss=3.217, NarTop10Accuracy=0.669, over 6759.00 frames. ], tot_loss[loss=3.382, NarTop10Accuracy=0.6411, over 5942.62 frames. ], batch size: 14, lr: 2.93e-03
2024-08-06 13:19:49,090 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-74000.pt
2024-08-06 13:19:52,597 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 13:19:59,386 INFO [trainer.py:811] (0/8) Epoch 29, validation: loss=3.233, NarTop10Accuracy=0.6754, over 1907754.00 frames. 
2024-08-06 13:19:59,386 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 13:19:59,903 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.560e+02 1.964e+02 2.123e+02 2.299e+02 5.520e+02, threshold=4.246e+02, percent-clipped=0.2
2024-08-06 13:20:20,108 INFO [trainer.py:765] (0/8) Epoch 29, batch 1800, train_loss[loss=3.299, NarTop10Accuracy=0.6521, over 7206.00 frames. ], tot_loss[loss=3.389, NarTop10Accuracy=0.6401, over 5987.09 frames. ], batch size: 22, lr: 2.93e-03
2024-08-06 13:20:46,844 INFO [trainer.py:765] (0/8) Epoch 29, batch 1900, train_loss[loss=3.56, NarTop10Accuracy=0.6107, over 6174.00 frames. ], tot_loss[loss=3.404, NarTop10Accuracy=0.6369, over 6030.33 frames. ], batch size: 48, lr: 2.93e-03
2024-08-06 13:21:12,478 INFO [trainer.py:765] (0/8) Epoch 29, batch 2000, train_loss[loss=3.657, NarTop10Accuracy=0.5861, over 5860.00 frames. ], tot_loss[loss=3.399, NarTop10Accuracy=0.6385, over 5998.15 frames. ], batch size: 49, lr: 2.92e-03
2024-08-06 13:21:37,982 INFO [trainer.py:765] (0/8) Epoch 29, batch 2100, train_loss[loss=3.289, NarTop10Accuracy=0.6753, over 4004.00 frames. ], tot_loss[loss=3.395, NarTop10Accuracy=0.639, over 5990.77 frames. ], batch size: 4, lr: 2.92e-03
2024-08-06 13:22:03,360 INFO [trainer.py:765] (0/8) Epoch 29, batch 2200, train_loss[loss=3.292, NarTop10Accuracy=0.6701, over 7471.00 frames. ], tot_loss[loss=3.396, NarTop10Accuracy=0.6392, over 6034.83 frames. ], batch size: 31, lr: 2.92e-03
2024-08-06 13:22:28,831 INFO [trainer.py:765] (0/8) Epoch 29, batch 2300, train_loss[loss=3.28, NarTop10Accuracy=0.6599, over 5783.00 frames. ], tot_loss[loss=3.4, NarTop10Accuracy=0.6381, over 6066.73 frames. ], batch size: 9, lr: 2.92e-03
2024-08-06 13:22:53,620 INFO [trainer.py:765] (0/8) Epoch 29, batch 2400, train_loss[loss=3.759, NarTop10Accuracy=0.5657, over 6202.00 frames. ], tot_loss[loss=3.4, NarTop10Accuracy=0.6379, over 5884.63 frames. ], batch size: 49, lr: 2.92e-03
2024-08-06 13:23:16,978 INFO [trainer.py:765] (0/8) Epoch 29, batch 2500, train_loss[loss=3.107, NarTop10Accuracy=0.6791, over 5033.00 frames. ], tot_loss[loss=3.384, NarTop10Accuracy=0.6407, over 5532.59 frames. ], batch size: 6, lr: 2.91e-03
2024-08-06 13:23:37,887 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 13:23:37,889 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-29.pt
2024-08-06 13:24:38,391 INFO [trainer.py:765] (0/8) Epoch 30, batch 100, train_loss[loss=3.373, NarTop10Accuracy=0.6403, over 7274.00 frames. ], tot_loss[loss=3.292, NarTop10Accuracy=0.6612, over 2358.53 frames. ], batch size: 30, lr: 2.86e-03
2024-08-06 13:24:52,129 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-75000.pt
2024-08-06 13:25:14,782 INFO [trainer.py:765] (0/8) Epoch 30, batch 200, train_loss[loss=3.167, NarTop10Accuracy=0.7004, over 6963.00 frames. ], tot_loss[loss=3.298, NarTop10Accuracy=0.6598, over 3859.04 frames. ], batch size: 17, lr: 2.86e-03
2024-08-06 13:25:46,846 INFO [trainer.py:765] (0/8) Epoch 30, batch 300, train_loss[loss=3.177, NarTop10Accuracy=0.6901, over 6997.00 frames. ], tot_loss[loss=3.311, NarTop10Accuracy=0.6569, over 4677.16 frames. ], batch size: 22, lr: 2.86e-03
2024-08-06 13:26:17,539 INFO [trainer.py:765] (0/8) Epoch 30, batch 400, train_loss[loss=3.432, NarTop10Accuracy=0.6393, over 5031.00 frames. ], tot_loss[loss=3.325, NarTop10Accuracy=0.6544, over 5131.72 frames. ], batch size: 7, lr: 2.86e-03
2024-08-06 13:26:53,920 INFO [trainer.py:765] (0/8) Epoch 30, batch 500, train_loss[loss=3.217, NarTop10Accuracy=0.6696, over 6207.00 frames. ], tot_loss[loss=3.308, NarTop10Accuracy=0.6579, over 5403.46 frames. ], batch size: 11, lr: 2.85e-03
2024-08-06 13:27:25,422 INFO [trainer.py:765] (0/8) Epoch 30, batch 600, train_loss[loss=3.21, NarTop10Accuracy=0.6752, over 5775.00 frames. ], tot_loss[loss=3.317, NarTop10Accuracy=0.6554, over 5661.83 frames. ], batch size: 9, lr: 2.85e-03
2024-08-06 13:28:00,307 INFO [trainer.py:765] (0/8) Epoch 30, batch 700, train_loss[loss=3.39, NarTop10Accuracy=0.6341, over 4982.00 frames. ], tot_loss[loss=3.333, NarTop10Accuracy=0.6521, over 5732.58 frames. ], batch size: 6, lr: 2.85e-03
2024-08-06 13:28:37,477 INFO [trainer.py:765] (0/8) Epoch 30, batch 800, train_loss[loss=3.515, NarTop10Accuracy=0.6169, over 5192.00 frames. ], tot_loss[loss=3.347, NarTop10Accuracy=0.6493, over 5779.40 frames. ], batch size: 6, lr: 2.85e-03
2024-08-06 13:29:10,425 INFO [trainer.py:765] (0/8) Epoch 30, batch 900, train_loss[loss=3.428, NarTop10Accuracy=0.6303, over 6222.00 frames. ], tot_loss[loss=3.357, NarTop10Accuracy=0.6469, over 5813.53 frames. ], batch size: 13, lr: 2.85e-03
2024-08-06 13:29:45,914 INFO [trainer.py:765] (0/8) Epoch 30, batch 1000, train_loss[loss=3.302, NarTop10Accuracy=0.6512, over 6352.00 frames. ], tot_loss[loss=3.372, NarTop10Accuracy=0.6435, over 5916.28 frames. ], batch size: 13, lr: 2.84e-03
2024-08-06 13:30:24,172 INFO [trainer.py:765] (0/8) Epoch 30, batch 1100, train_loss[loss=3.332, NarTop10Accuracy=0.648, over 6998.00 frames. ], tot_loss[loss=3.376, NarTop10Accuracy=0.6425, over 5959.32 frames. ], batch size: 17, lr: 2.84e-03
2024-08-06 13:30:38,001 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-76000.pt
2024-08-06 13:30:41,468 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 13:30:48,195 INFO [trainer.py:811] (0/8) Epoch 30, validation: loss=3.239, NarTop10Accuracy=0.6729, over 1907754.00 frames. 
2024-08-06 13:30:48,196 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 13:30:48,916 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.612e+02 1.985e+02 2.139e+02 2.326e+02 4.628e+02, threshold=4.279e+02, percent-clipped=0.1
2024-08-06 13:31:05,665 INFO [trainer.py:765] (0/8) Epoch 30, batch 1200, train_loss[loss=3.404, NarTop10Accuracy=0.6366, over 7133.00 frames. ], tot_loss[loss=3.372, NarTop10Accuracy=0.6434, over 5958.78 frames. ], batch size: 30, lr: 2.84e-03
2024-08-06 13:31:43,020 INFO [trainer.py:765] (0/8) Epoch 30, batch 1300, train_loss[loss=3.28, NarTop10Accuracy=0.6689, over 4995.00 frames. ], tot_loss[loss=3.36, NarTop10Accuracy=0.646, over 6021.09 frames. ], batch size: 6, lr: 2.84e-03
2024-08-06 13:32:19,324 INFO [trainer.py:765] (0/8) Epoch 30, batch 1400, train_loss[loss=3.536, NarTop10Accuracy=0.6088, over 6129.00 frames. ], tot_loss[loss=3.38, NarTop10Accuracy=0.6424, over 6043.05 frames. ], batch size: 11, lr: 2.84e-03
2024-08-06 13:32:52,335 INFO [trainer.py:765] (0/8) Epoch 30, batch 1500, train_loss[loss=3.535, NarTop10Accuracy=0.6093, over 5704.00 frames. ], tot_loss[loss=3.379, NarTop10Accuracy=0.6429, over 5970.65 frames. ], batch size: 49, lr: 2.83e-03
2024-08-06 13:33:20,407 INFO [trainer.py:765] (0/8) Epoch 30, batch 1600, train_loss[loss=3.806, NarTop10Accuracy=0.5546, over 7032.00 frames. ], tot_loss[loss=3.385, NarTop10Accuracy=0.6413, over 5943.57 frames. ], batch size: 22, lr: 2.83e-03
2024-08-06 13:33:47,200 INFO [trainer.py:765] (0/8) Epoch 30, batch 1700, train_loss[loss=3.47, NarTop10Accuracy=0.619, over 6326.00 frames. ], tot_loss[loss=3.394, NarTop10Accuracy=0.6393, over 5944.98 frames. ], batch size: 13, lr: 2.83e-03
2024-08-06 13:34:13,886 INFO [trainer.py:765] (0/8) Epoch 30, batch 1800, train_loss[loss=3.727, NarTop10Accuracy=0.5606, over 7158.00 frames. ], tot_loss[loss=3.382, NarTop10Accuracy=0.6418, over 6018.83 frames. ], batch size: 22, lr: 2.83e-03
2024-08-06 13:34:40,547 INFO [trainer.py:765] (0/8) Epoch 30, batch 1900, train_loss[loss=3.497, NarTop10Accuracy=0.6177, over 6231.00 frames. ], tot_loss[loss=3.388, NarTop10Accuracy=0.6403, over 6045.56 frames. ], batch size: 49, lr: 2.83e-03
2024-08-06 13:35:06,315 INFO [trainer.py:765] (0/8) Epoch 30, batch 2000, train_loss[loss=3.656, NarTop10Accuracy=0.5895, over 6010.00 frames. ], tot_loss[loss=3.373, NarTop10Accuracy=0.643, over 6006.44 frames. ], batch size: 49, lr: 2.83e-03
2024-08-06 13:35:31,871 INFO [trainer.py:765] (0/8) Epoch 30, batch 2100, train_loss[loss=3.396, NarTop10Accuracy=0.6323, over 3913.00 frames. ], tot_loss[loss=3.387, NarTop10Accuracy=0.6404, over 5975.21 frames. ], batch size: 4, lr: 2.82e-03
2024-08-06 13:35:42,323 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-77000.pt
2024-08-06 13:36:00,553 INFO [trainer.py:765] (0/8) Epoch 30, batch 2200, train_loss[loss=3.552, NarTop10Accuracy=0.6127, over 7269.00 frames. ], tot_loss[loss=3.39, NarTop10Accuracy=0.6398, over 6027.09 frames. ], batch size: 31, lr: 2.82e-03
2024-08-06 13:36:26,029 INFO [trainer.py:765] (0/8) Epoch 30, batch 2300, train_loss[loss=3.849, NarTop10Accuracy=0.5574, over 5773.00 frames. ], tot_loss[loss=3.399, NarTop10Accuracy=0.638, over 6065.11 frames. ], batch size: 9, lr: 2.82e-03
2024-08-06 13:36:50,824 INFO [trainer.py:765] (0/8) Epoch 30, batch 2400, train_loss[loss=3.668, NarTop10Accuracy=0.5916, over 5617.00 frames. ], tot_loss[loss=3.402, NarTop10Accuracy=0.6371, over 5888.91 frames. ], batch size: 50, lr: 2.82e-03
2024-08-06 13:37:14,388 INFO [trainer.py:765] (0/8) Epoch 30, batch 2500, train_loss[loss=3.215, NarTop10Accuracy=0.6699, over 5173.00 frames. ], tot_loss[loss=3.376, NarTop10Accuracy=0.6424, over 5545.35 frames. ], batch size: 6, lr: 2.82e-03
2024-08-06 13:37:36,084 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 13:37:36,087 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-30.pt
2024-08-06 13:38:28,438 INFO [trainer.py:765] (0/8) Epoch 31, batch 100, train_loss[loss=3.081, NarTop10Accuracy=0.698, over 7255.00 frames. ], tot_loss[loss=3.298, NarTop10Accuracy=0.659, over 2371.67 frames. ], batch size: 30, lr: 2.77e-03
2024-08-06 13:39:02,651 INFO [trainer.py:765] (0/8) Epoch 31, batch 200, train_loss[loss=3.131, NarTop10Accuracy=0.6873, over 6922.00 frames. ], tot_loss[loss=3.31, NarTop10Accuracy=0.657, over 3872.76 frames. ], batch size: 17, lr: 2.76e-03
2024-08-06 13:39:34,676 INFO [trainer.py:765] (0/8) Epoch 31, batch 300, train_loss[loss=3.221, NarTop10Accuracy=0.6723, over 7202.00 frames. ], tot_loss[loss=3.305, NarTop10Accuracy=0.658, over 4680.89 frames. ], batch size: 22, lr: 2.76e-03
2024-08-06 13:40:07,363 INFO [trainer.py:765] (0/8) Epoch 31, batch 400, train_loss[loss=3.433, NarTop10Accuracy=0.6382, over 5711.00 frames. ], tot_loss[loss=3.33, NarTop10Accuracy=0.6528, over 5142.12 frames. ], batch size: 8, lr: 2.76e-03
2024-08-06 13:40:37,813 INFO [trainer.py:765] (0/8) Epoch 31, batch 500, train_loss[loss=3.292, NarTop10Accuracy=0.6587, over 6047.00 frames. ], tot_loss[loss=3.321, NarTop10Accuracy=0.6543, over 5413.09 frames. ], batch size: 11, lr: 2.76e-03
2024-08-06 13:40:58,297 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-78000.pt
2024-08-06 13:41:01,893 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 13:41:08,777 INFO [trainer.py:811] (0/8) Epoch 31, validation: loss=3.268, NarTop10Accuracy=0.6673, over 1907754.00 frames. 
2024-08-06 13:41:08,778 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 13:41:09,338 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.599e+02 1.987e+02 2.143e+02 2.328e+02 4.341e+02, threshold=4.287e+02, percent-clipped=0.1
2024-08-06 13:41:20,862 INFO [trainer.py:765] (0/8) Epoch 31, batch 600, train_loss[loss=3.143, NarTop10Accuracy=0.6869, over 5777.00 frames. ], tot_loss[loss=3.335, NarTop10Accuracy=0.6513, over 5678.24 frames. ], batch size: 9, lr: 2.76e-03
2024-08-06 13:41:54,259 INFO [trainer.py:765] (0/8) Epoch 31, batch 700, train_loss[loss=3.274, NarTop10Accuracy=0.6731, over 5052.00 frames. ], tot_loss[loss=3.338, NarTop10Accuracy=0.6508, over 5752.80 frames. ], batch size: 6, lr: 2.76e-03
2024-08-06 13:42:32,158 INFO [trainer.py:765] (0/8) Epoch 31, batch 800, train_loss[loss=3.325, NarTop10Accuracy=0.6583, over 5155.00 frames. ], tot_loss[loss=3.342, NarTop10Accuracy=0.6499, over 5805.86 frames. ], batch size: 6, lr: 2.75e-03
2024-08-06 13:43:06,274 INFO [trainer.py:765] (0/8) Epoch 31, batch 900, train_loss[loss=3.313, NarTop10Accuracy=0.6536, over 6202.00 frames. ], tot_loss[loss=3.332, NarTop10Accuracy=0.6519, over 5830.27 frames. ], batch size: 13, lr: 2.75e-03
2024-08-06 13:43:38,009 INFO [trainer.py:765] (0/8) Epoch 31, batch 1000, train_loss[loss=3.103, NarTop10Accuracy=0.7097, over 6138.00 frames. ], tot_loss[loss=3.339, NarTop10Accuracy=0.6503, over 5939.40 frames. ], batch size: 13, lr: 2.75e-03
2024-08-06 13:44:14,513 INFO [trainer.py:765] (0/8) Epoch 31, batch 1100, train_loss[loss=3.409, NarTop10Accuracy=0.636, over 6922.00 frames. ], tot_loss[loss=3.344, NarTop10Accuracy=0.6492, over 5956.71 frames. ], batch size: 17, lr: 2.75e-03
2024-08-06 13:44:53,786 INFO [trainer.py:765] (0/8) Epoch 31, batch 1200, train_loss[loss=3.313, NarTop10Accuracy=0.6565, over 7196.00 frames. ], tot_loss[loss=3.351, NarTop10Accuracy=0.6476, over 5956.01 frames. ], batch size: 30, lr: 2.75e-03
2024-08-06 13:45:25,076 INFO [trainer.py:765] (0/8) Epoch 31, batch 1300, train_loss[loss=3.16, NarTop10Accuracy=0.6771, over 5039.00 frames. ], tot_loss[loss=3.351, NarTop10Accuracy=0.6475, over 6016.19 frames. ], batch size: 6, lr: 2.75e-03
2024-08-06 13:45:58,740 INFO [trainer.py:765] (0/8) Epoch 31, batch 1400, train_loss[loss=3.245, NarTop10Accuracy=0.6683, over 6126.00 frames. ], tot_loss[loss=3.359, NarTop10Accuracy=0.6461, over 6038.19 frames. ], batch size: 11, lr: 2.74e-03
2024-08-06 13:46:33,490 INFO [trainer.py:765] (0/8) Epoch 31, batch 1500, train_loss[loss=3.503, NarTop10Accuracy=0.6123, over 6260.00 frames. ], tot_loss[loss=3.352, NarTop10Accuracy=0.6475, over 5955.11 frames. ], batch size: 49, lr: 2.74e-03
2024-08-06 13:46:50,119 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-79000.pt
2024-08-06 13:47:04,657 INFO [trainer.py:765] (0/8) Epoch 31, batch 1600, train_loss[loss=3.198, NarTop10Accuracy=0.6743, over 7258.00 frames. ], tot_loss[loss=3.362, NarTop10Accuracy=0.6454, over 5953.78 frames. ], batch size: 22, lr: 2.74e-03
2024-08-06 13:47:31,423 INFO [trainer.py:765] (0/8) Epoch 31, batch 1700, train_loss[loss=3.589, NarTop10Accuracy=0.6061, over 6654.00 frames. ], tot_loss[loss=3.366, NarTop10Accuracy=0.6446, over 5946.60 frames. ], batch size: 14, lr: 2.74e-03
2024-08-06 13:47:58,016 INFO [trainer.py:765] (0/8) Epoch 31, batch 1800, train_loss[loss=3.614, NarTop10Accuracy=0.5922, over 7035.00 frames. ], tot_loss[loss=3.372, NarTop10Accuracy=0.6437, over 6008.46 frames. ], batch size: 22, lr: 2.74e-03
2024-08-06 13:48:24,577 INFO [trainer.py:765] (0/8) Epoch 31, batch 1900, train_loss[loss=3.274, NarTop10Accuracy=0.6572, over 5982.00 frames. ], tot_loss[loss=3.379, NarTop10Accuracy=0.6423, over 6056.28 frames. ], batch size: 49, lr: 2.74e-03
2024-08-06 13:48:50,257 INFO [trainer.py:765] (0/8) Epoch 31, batch 2000, train_loss[loss=3.645, NarTop10Accuracy=0.5836, over 6010.00 frames. ], tot_loss[loss=3.375, NarTop10Accuracy=0.6433, over 6035.63 frames. ], batch size: 49, lr: 2.73e-03
2024-08-06 13:49:15,765 INFO [trainer.py:765] (0/8) Epoch 31, batch 2100, train_loss[loss=3.336, NarTop10Accuracy=0.643, over 4814.00 frames. ], tot_loss[loss=3.366, NarTop10Accuracy=0.6452, over 6017.89 frames. ], batch size: 5, lr: 2.73e-03
2024-08-06 13:49:41,278 INFO [trainer.py:765] (0/8) Epoch 31, batch 2200, train_loss[loss=3.299, NarTop10Accuracy=0.6546, over 7317.00 frames. ], tot_loss[loss=3.363, NarTop10Accuracy=0.6453, over 6038.24 frames. ], batch size: 30, lr: 2.73e-03
2024-08-06 13:50:06,707 INFO [trainer.py:765] (0/8) Epoch 31, batch 2300, train_loss[loss=3.318, NarTop10Accuracy=0.6632, over 5769.00 frames. ], tot_loss[loss=3.384, NarTop10Accuracy=0.6413, over 6067.31 frames. ], batch size: 9, lr: 2.73e-03
2024-08-06 13:50:31,393 INFO [trainer.py:765] (0/8) Epoch 31, batch 2400, train_loss[loss=3.517, NarTop10Accuracy=0.6156, over 5649.00 frames. ], tot_loss[loss=3.403, NarTop10Accuracy=0.6369, over 5890.54 frames. ], batch size: 49, lr: 2.73e-03
2024-08-06 13:50:54,892 INFO [trainer.py:765] (0/8) Epoch 31, batch 2500, train_loss[loss=3.382, NarTop10Accuracy=0.6391, over 4361.00 frames. ], tot_loss[loss=3.371, NarTop10Accuracy=0.6428, over 5525.60 frames. ], batch size: 5, lr: 2.72e-03
2024-08-06 13:51:08,995 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-80000.pt
2024-08-06 13:51:12,534 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 13:51:19,070 INFO [trainer.py:811] (0/8) Epoch 31, validation: loss=3.234, NarTop10Accuracy=0.6746, over 1907754.00 frames. 
2024-08-06 13:51:19,070 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 13:51:19,539 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.591e+02 2.007e+02 2.182e+02 2.368e+02 4.565e+02, threshold=4.363e+02, percent-clipped=0.1
2024-08-06 13:51:26,773 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 13:51:26,776 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-31.pt
2024-08-06 13:52:19,910 INFO [trainer.py:765] (0/8) Epoch 32, batch 100, train_loss[loss=3.27, NarTop10Accuracy=0.667, over 7073.00 frames. ], tot_loss[loss=3.34, NarTop10Accuracy=0.6524, over 2357.15 frames. ], batch size: 30, lr: 2.68e-03
2024-08-06 13:52:52,538 INFO [trainer.py:765] (0/8) Epoch 32, batch 200, train_loss[loss=3.751, NarTop10Accuracy=0.5684, over 6905.00 frames. ], tot_loss[loss=3.336, NarTop10Accuracy=0.6529, over 3866.95 frames. ], batch size: 17, lr: 2.68e-03
2024-08-06 13:53:28,093 INFO [trainer.py:765] (0/8) Epoch 32, batch 300, train_loss[loss=3.333, NarTop10Accuracy=0.6497, over 7083.00 frames. ], tot_loss[loss=3.328, NarTop10Accuracy=0.6538, over 4686.40 frames. ], batch size: 22, lr: 2.68e-03
2024-08-06 13:54:00,887 INFO [trainer.py:765] (0/8) Epoch 32, batch 400, train_loss[loss=3.498, NarTop10Accuracy=0.6127, over 5113.00 frames. ], tot_loss[loss=3.323, NarTop10Accuracy=0.6542, over 5128.34 frames. ], batch size: 7, lr: 2.67e-03
2024-08-06 13:54:32,821 INFO [trainer.py:765] (0/8) Epoch 32, batch 500, train_loss[loss=2.972, NarTop10Accuracy=0.7196, over 6101.00 frames. ], tot_loss[loss=3.306, NarTop10Accuracy=0.6569, over 5405.33 frames. ], batch size: 11, lr: 2.67e-03
2024-08-06 13:55:01,772 INFO [trainer.py:765] (0/8) Epoch 32, batch 600, train_loss[loss=3.59, NarTop10Accuracy=0.6083, over 5757.00 frames. ], tot_loss[loss=3.308, NarTop10Accuracy=0.6568, over 5673.13 frames. ], batch size: 9, lr: 2.67e-03
2024-08-06 13:55:41,511 INFO [trainer.py:765] (0/8) Epoch 32, batch 700, train_loss[loss=3.018, NarTop10Accuracy=0.7216, over 5086.00 frames. ], tot_loss[loss=3.331, NarTop10Accuracy=0.6527, over 5732.10 frames. ], batch size: 6, lr: 2.67e-03
2024-08-06 13:56:13,172 INFO [trainer.py:765] (0/8) Epoch 32, batch 800, train_loss[loss=3.028, NarTop10Accuracy=0.719, over 4998.00 frames. ], tot_loss[loss=3.33, NarTop10Accuracy=0.6526, over 5799.49 frames. ], batch size: 6, lr: 2.67e-03
2024-08-06 13:56:43,166 INFO [trainer.py:765] (0/8) Epoch 32, batch 900, train_loss[loss=3.64, NarTop10Accuracy=0.588, over 6310.00 frames. ], tot_loss[loss=3.328, NarTop10Accuracy=0.6526, over 5832.17 frames. ], batch size: 13, lr: 2.67e-03
2024-08-06 13:57:12,551 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-81000.pt
2024-08-06 13:57:24,520 INFO [trainer.py:765] (0/8) Epoch 32, batch 1000, train_loss[loss=3.593, NarTop10Accuracy=0.5952, over 6351.00 frames. ], tot_loss[loss=3.335, NarTop10Accuracy=0.6511, over 5922.57 frames. ], batch size: 13, lr: 2.66e-03
2024-08-06 13:57:57,452 INFO [trainer.py:765] (0/8) Epoch 32, batch 1100, train_loss[loss=3.112, NarTop10Accuracy=0.6984, over 6865.00 frames. ], tot_loss[loss=3.35, NarTop10Accuracy=0.6479, over 5969.41 frames. ], batch size: 17, lr: 2.66e-03
2024-08-06 13:58:30,541 INFO [trainer.py:765] (0/8) Epoch 32, batch 1200, train_loss[loss=3.105, NarTop10Accuracy=0.6957, over 7266.00 frames. ], tot_loss[loss=3.348, NarTop10Accuracy=0.6481, over 5960.42 frames. ], batch size: 31, lr: 2.66e-03
2024-08-06 13:59:08,259 INFO [trainer.py:765] (0/8) Epoch 32, batch 1300, train_loss[loss=3.087, NarTop10Accuracy=0.6956, over 4302.00 frames. ], tot_loss[loss=3.356, NarTop10Accuracy=0.6465, over 6023.63 frames. ], batch size: 5, lr: 2.66e-03
2024-08-06 13:59:42,265 INFO [trainer.py:765] (0/8) Epoch 32, batch 1400, train_loss[loss=3.329, NarTop10Accuracy=0.6571, over 6053.00 frames. ], tot_loss[loss=3.356, NarTop10Accuracy=0.6465, over 6029.44 frames. ], batch size: 11, lr: 2.66e-03
2024-08-06 14:00:12,976 INFO [trainer.py:765] (0/8) Epoch 32, batch 1500, train_loss[loss=3.699, NarTop10Accuracy=0.5811, over 6483.00 frames. ], tot_loss[loss=3.353, NarTop10Accuracy=0.6469, over 5960.90 frames. ], batch size: 48, lr: 2.66e-03
2024-08-06 14:00:40,824 INFO [trainer.py:765] (0/8) Epoch 32, batch 1600, train_loss[loss=3.178, NarTop10Accuracy=0.673, over 7034.00 frames. ], tot_loss[loss=3.353, NarTop10Accuracy=0.6467, over 5928.65 frames. ], batch size: 22, lr: 2.65e-03
2024-08-06 14:01:07,534 INFO [trainer.py:765] (0/8) Epoch 32, batch 1700, train_loss[loss=3.369, NarTop10Accuracy=0.6484, over 6240.00 frames. ], tot_loss[loss=3.361, NarTop10Accuracy=0.6449, over 5931.02 frames. ], batch size: 13, lr: 2.65e-03
2024-08-06 14:01:34,089 INFO [trainer.py:765] (0/8) Epoch 32, batch 1800, train_loss[loss=3.218, NarTop10Accuracy=0.6746, over 7151.00 frames. ], tot_loss[loss=3.364, NarTop10Accuracy=0.6447, over 5982.39 frames. ], batch size: 22, lr: 2.65e-03
2024-08-06 14:02:00,636 INFO [trainer.py:765] (0/8) Epoch 32, batch 1900, train_loss[loss=3.427, NarTop10Accuracy=0.6275, over 6209.00 frames. ], tot_loss[loss=3.375, NarTop10Accuracy=0.6429, over 6032.82 frames. ], batch size: 49, lr: 2.65e-03
2024-08-06 14:02:20,590 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-82000.pt
2024-08-06 14:02:24,194 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 14:02:30,653 INFO [trainer.py:811] (0/8) Epoch 32, validation: loss=3.204, NarTop10Accuracy=0.6812, over 1907754.00 frames. 
2024-08-06 14:02:30,653 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 14:02:31,152 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.595e+02 2.032e+02 2.200e+02 2.392e+02 6.182e+02, threshold=4.401e+02, percent-clipped=0.1
2024-08-06 14:02:36,383 INFO [trainer.py:765] (0/8) Epoch 32, batch 2000, train_loss[loss=3.504, NarTop10Accuracy=0.6151, over 6045.00 frames. ], tot_loss[loss=3.373, NarTop10Accuracy=0.6438, over 6019.26 frames. ], batch size: 50, lr: 2.65e-03
2024-08-06 14:03:01,698 INFO [trainer.py:765] (0/8) Epoch 32, batch 2100, train_loss[loss=3.239, NarTop10Accuracy=0.6566, over 4710.00 frames. ], tot_loss[loss=3.372, NarTop10Accuracy=0.6439, over 5999.59 frames. ], batch size: 5, lr: 2.65e-03
2024-08-06 14:03:27,177 INFO [trainer.py:765] (0/8) Epoch 32, batch 2200, train_loss[loss=3.622, NarTop10Accuracy=0.6013, over 7088.00 frames. ], tot_loss[loss=3.373, NarTop10Accuracy=0.6435, over 6039.76 frames. ], batch size: 30, lr: 2.64e-03
2024-08-06 14:03:52,585 INFO [trainer.py:765] (0/8) Epoch 32, batch 2300, train_loss[loss=3.712, NarTop10Accuracy=0.5736, over 5855.00 frames. ], tot_loss[loss=3.386, NarTop10Accuracy=0.6408, over 6070.75 frames. ], batch size: 9, lr: 2.64e-03
2024-08-06 14:04:17,274 INFO [trainer.py:765] (0/8) Epoch 32, batch 2400, train_loss[loss=3.601, NarTop10Accuracy=0.6044, over 6410.00 frames. ], tot_loss[loss=3.392, NarTop10Accuracy=0.6398, over 5886.58 frames. ], batch size: 48, lr: 2.64e-03
2024-08-06 14:04:40,635 INFO [trainer.py:765] (0/8) Epoch 32, batch 2500, train_loss[loss=3.193, NarTop10Accuracy=0.6809, over 5074.00 frames. ], tot_loss[loss=3.36, NarTop10Accuracy=0.6458, over 5552.01 frames. ], batch size: 6, lr: 2.64e-03
2024-08-06 14:05:01,806 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 14:05:01,809 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-32.pt
2024-08-06 14:06:02,906 INFO [trainer.py:765] (0/8) Epoch 33, batch 100, train_loss[loss=3.626, NarTop10Accuracy=0.5961, over 7448.00 frames. ], tot_loss[loss=3.338, NarTop10Accuracy=0.6528, over 2369.19 frames. ], batch size: 31, lr: 2.60e-03
2024-08-06 14:06:36,080 INFO [trainer.py:765] (0/8) Epoch 33, batch 200, train_loss[loss=3.403, NarTop10Accuracy=0.6391, over 6779.00 frames. ], tot_loss[loss=3.318, NarTop10Accuracy=0.6561, over 3860.49 frames. ], batch size: 17, lr: 2.59e-03
2024-08-06 14:07:12,147 INFO [trainer.py:765] (0/8) Epoch 33, batch 300, train_loss[loss=3.281, NarTop10Accuracy=0.668, over 7297.00 frames. ], tot_loss[loss=3.314, NarTop10Accuracy=0.6565, over 4674.55 frames. ], batch size: 22, lr: 2.59e-03
2024-08-06 14:07:44,121 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-83000.pt
2024-08-06 14:07:48,256 INFO [trainer.py:765] (0/8) Epoch 33, batch 400, train_loss[loss=3.22, NarTop10Accuracy=0.6712, over 5042.00 frames. ], tot_loss[loss=3.317, NarTop10Accuracy=0.6558, over 5132.82 frames. ], batch size: 7, lr: 2.59e-03
2024-08-06 14:08:18,547 INFO [trainer.py:765] (0/8) Epoch 33, batch 500, train_loss[loss=3.143, NarTop10Accuracy=0.6867, over 6098.00 frames. ], tot_loss[loss=3.326, NarTop10Accuracy=0.6536, over 5404.05 frames. ], batch size: 11, lr: 2.59e-03
2024-08-06 14:08:49,793 INFO [trainer.py:765] (0/8) Epoch 33, batch 600, train_loss[loss=3.043, NarTop10Accuracy=0.7067, over 5739.00 frames. ], tot_loss[loss=3.327, NarTop10Accuracy=0.6528, over 5676.62 frames. ], batch size: 9, lr: 2.59e-03
2024-08-06 14:09:32,926 INFO [trainer.py:765] (0/8) Epoch 33, batch 700, train_loss[loss=3.088, NarTop10Accuracy=0.6963, over 5041.00 frames. ], tot_loss[loss=3.334, NarTop10Accuracy=0.652, over 5735.12 frames. ], batch size: 6, lr: 2.59e-03
2024-08-06 14:10:04,597 INFO [trainer.py:765] (0/8) Epoch 33, batch 800, train_loss[loss=2.994, NarTop10Accuracy=0.6979, over 5127.00 frames. ], tot_loss[loss=3.341, NarTop10Accuracy=0.6502, over 5793.88 frames. ], batch size: 6, lr: 2.58e-03
2024-08-06 14:10:35,387 INFO [trainer.py:765] (0/8) Epoch 33, batch 900, train_loss[loss=3.319, NarTop10Accuracy=0.6511, over 6271.00 frames. ], tot_loss[loss=3.348, NarTop10Accuracy=0.6491, over 5799.96 frames. ], batch size: 13, lr: 2.58e-03
2024-08-06 14:11:15,070 INFO [trainer.py:765] (0/8) Epoch 33, batch 1000, train_loss[loss=3.276, NarTop10Accuracy=0.6493, over 6198.00 frames. ], tot_loss[loss=3.352, NarTop10Accuracy=0.6482, over 5909.75 frames. ], batch size: 13, lr: 2.58e-03
2024-08-06 14:11:47,302 INFO [trainer.py:765] (0/8) Epoch 33, batch 1100, train_loss[loss=3.55, NarTop10Accuracy=0.5997, over 6797.00 frames. ], tot_loss[loss=3.36, NarTop10Accuracy=0.6466, over 5934.00 frames. ], batch size: 17, lr: 2.58e-03
2024-08-06 14:12:20,928 INFO [trainer.py:765] (0/8) Epoch 33, batch 1200, train_loss[loss=3.34, NarTop10Accuracy=0.6361, over 7187.00 frames. ], tot_loss[loss=3.362, NarTop10Accuracy=0.6454, over 5933.53 frames. ], batch size: 31, lr: 2.58e-03
2024-08-06 14:12:57,630 INFO [trainer.py:765] (0/8) Epoch 33, batch 1300, train_loss[loss=3.52, NarTop10Accuracy=0.6062, over 5136.00 frames. ], tot_loss[loss=3.353, NarTop10Accuracy=0.6474, over 6012.87 frames. ], batch size: 6, lr: 2.58e-03
2024-08-06 14:13:30,666 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-84000.pt
2024-08-06 14:13:34,965 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 14:13:41,686 INFO [trainer.py:811] (0/8) Epoch 33, validation: loss=3.242, NarTop10Accuracy=0.6732, over 1907754.00 frames. 
2024-08-06 14:13:41,687 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 14:13:42,264 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.623e+02 2.031e+02 2.174e+02 2.363e+02 4.871e+02, threshold=4.347e+02, percent-clipped=0.1
2024-08-06 14:13:42,803 INFO [trainer.py:765] (0/8) Epoch 33, batch 1400, train_loss[loss=3.294, NarTop10Accuracy=0.6611, over 6135.00 frames. ], tot_loss[loss=3.341, NarTop10Accuracy=0.6496, over 6023.23 frames. ], batch size: 11, lr: 2.58e-03
2024-08-06 14:14:11,246 INFO [trainer.py:765] (0/8) Epoch 33, batch 1500, train_loss[loss=3.43, NarTop10Accuracy=0.6284, over 6351.00 frames. ], tot_loss[loss=3.356, NarTop10Accuracy=0.6466, over 5985.36 frames. ], batch size: 49, lr: 2.57e-03
2024-08-06 14:14:39,191 INFO [trainer.py:765] (0/8) Epoch 33, batch 1600, train_loss[loss=3.319, NarTop10Accuracy=0.6595, over 7159.00 frames. ], tot_loss[loss=3.365, NarTop10Accuracy=0.6449, over 5958.13 frames. ], batch size: 22, lr: 2.57e-03
2024-08-06 14:15:05,858 INFO [trainer.py:765] (0/8) Epoch 33, batch 1700, train_loss[loss=3.577, NarTop10Accuracy=0.6035, over 6305.00 frames. ], tot_loss[loss=3.367, NarTop10Accuracy=0.6442, over 5949.19 frames. ], batch size: 13, lr: 2.57e-03
2024-08-06 14:15:32,589 INFO [trainer.py:765] (0/8) Epoch 33, batch 1800, train_loss[loss=3.473, NarTop10Accuracy=0.6276, over 7164.00 frames. ], tot_loss[loss=3.353, NarTop10Accuracy=0.6472, over 6007.49 frames. ], batch size: 22, lr: 2.57e-03
2024-08-06 14:15:59,214 INFO [trainer.py:765] (0/8) Epoch 33, batch 1900, train_loss[loss=3.472, NarTop10Accuracy=0.6251, over 5754.00 frames. ], tot_loss[loss=3.372, NarTop10Accuracy=0.6441, over 6052.53 frames. ], batch size: 48, lr: 2.57e-03
2024-08-06 14:16:24,895 INFO [trainer.py:765] (0/8) Epoch 33, batch 2000, train_loss[loss=3.435, NarTop10Accuracy=0.6269, over 6121.00 frames. ], tot_loss[loss=3.359, NarTop10Accuracy=0.6467, over 6027.41 frames. ], batch size: 49, lr: 2.57e-03
2024-08-06 14:16:50,350 INFO [trainer.py:765] (0/8) Epoch 33, batch 2100, train_loss[loss=3.532, NarTop10Accuracy=0.6058, over 4666.00 frames. ], tot_loss[loss=3.365, NarTop10Accuracy=0.6453, over 6006.51 frames. ], batch size: 5, lr: 2.56e-03
2024-08-06 14:17:15,825 INFO [trainer.py:765] (0/8) Epoch 33, batch 2200, train_loss[loss=3.533, NarTop10Accuracy=0.6131, over 7126.00 frames. ], tot_loss[loss=3.358, NarTop10Accuracy=0.6466, over 6042.20 frames. ], batch size: 30, lr: 2.56e-03
2024-08-06 14:17:41,309 INFO [trainer.py:765] (0/8) Epoch 33, batch 2300, train_loss[loss=3.456, NarTop10Accuracy=0.6391, over 5716.00 frames. ], tot_loss[loss=3.373, NarTop10Accuracy=0.6437, over 6087.36 frames. ], batch size: 9, lr: 2.56e-03
2024-08-06 14:18:05,030 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-85000.pt
2024-08-06 14:18:10,143 INFO [trainer.py:765] (0/8) Epoch 33, batch 2400, train_loss[loss=3.697, NarTop10Accuracy=0.5805, over 6404.00 frames. ], tot_loss[loss=3.384, NarTop10Accuracy=0.6416, over 5897.97 frames. ], batch size: 48, lr: 2.56e-03
2024-08-06 14:18:33,707 INFO [trainer.py:765] (0/8) Epoch 33, batch 2500, train_loss[loss=3.363, NarTop10Accuracy=0.6418, over 5130.00 frames. ], tot_loss[loss=3.351, NarTop10Accuracy=0.6479, over 5540.90 frames. ], batch size: 6, lr: 2.56e-03
2024-08-06 14:18:54,675 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 14:18:54,678 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-33.pt
2024-08-06 14:19:51,933 INFO [trainer.py:765] (0/8) Epoch 34, batch 100, train_loss[loss=3.141, NarTop10Accuracy=0.6956, over 7159.00 frames. ], tot_loss[loss=3.296, NarTop10Accuracy=0.661, over 2369.73 frames. ], batch size: 30, lr: 2.52e-03
2024-08-06 14:20:24,373 INFO [trainer.py:765] (0/8) Epoch 34, batch 200, train_loss[loss=3.368, NarTop10Accuracy=0.6447, over 6997.00 frames. ], tot_loss[loss=3.283, NarTop10Accuracy=0.663, over 3867.27 frames. ], batch size: 17, lr: 2.52e-03
2024-08-06 14:21:00,842 INFO [trainer.py:765] (0/8) Epoch 34, batch 300, train_loss[loss=3.093, NarTop10Accuracy=0.6944, over 7278.00 frames. ], tot_loss[loss=3.296, NarTop10Accuracy=0.66, over 4682.88 frames. ], batch size: 22, lr: 2.51e-03
2024-08-06 14:21:31,450 INFO [trainer.py:765] (0/8) Epoch 34, batch 400, train_loss[loss=3.134, NarTop10Accuracy=0.6856, over 5233.00 frames. ], tot_loss[loss=3.304, NarTop10Accuracy=0.6581, over 5123.71 frames. ], batch size: 7, lr: 2.51e-03
2024-08-06 14:22:01,876 INFO [trainer.py:765] (0/8) Epoch 34, batch 500, train_loss[loss=3.151, NarTop10Accuracy=0.6834, over 6073.00 frames. ], tot_loss[loss=3.313, NarTop10Accuracy=0.6556, over 5393.84 frames. ], batch size: 11, lr: 2.51e-03
2024-08-06 14:22:36,827 INFO [trainer.py:765] (0/8) Epoch 34, batch 600, train_loss[loss=3.245, NarTop10Accuracy=0.6669, over 5829.00 frames. ], tot_loss[loss=3.314, NarTop10Accuracy=0.6559, over 5659.38 frames. ], batch size: 9, lr: 2.51e-03
2024-08-06 14:23:14,605 INFO [trainer.py:765] (0/8) Epoch 34, batch 700, train_loss[loss=3.149, NarTop10Accuracy=0.6929, over 4919.00 frames. ], tot_loss[loss=3.318, NarTop10Accuracy=0.6547, over 5738.75 frames. ], batch size: 6, lr: 2.51e-03
2024-08-06 14:23:46,606 INFO [trainer.py:765] (0/8) Epoch 34, batch 800, train_loss[loss=3.562, NarTop10Accuracy=0.6108, over 4824.00 frames. ], tot_loss[loss=3.339, NarTop10Accuracy=0.6506, over 5800.48 frames. ], batch size: 6, lr: 2.51e-03
2024-08-06 14:23:50,719 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-86000.pt
2024-08-06 14:23:54,158 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 14:24:00,855 INFO [trainer.py:811] (0/8) Epoch 34, validation: loss=3.226, NarTop10Accuracy=0.6758, over 1907754.00 frames. 
2024-08-06 14:24:00,856 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 14:24:01,413 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.652e+02 2.033e+02 2.200e+02 2.391e+02 5.918e+02, threshold=4.399e+02, percent-clipped=0.1
2024-08-06 14:24:28,899 INFO [trainer.py:765] (0/8) Epoch 34, batch 900, train_loss[loss=3.263, NarTop10Accuracy=0.6598, over 6673.00 frames. ], tot_loss[loss=3.337, NarTop10Accuracy=0.6505, over 5833.17 frames. ], batch size: 14, lr: 2.51e-03
2024-08-06 14:25:05,287 INFO [trainer.py:765] (0/8) Epoch 34, batch 1000, train_loss[loss=3.264, NarTop10Accuracy=0.6657, over 6232.00 frames. ], tot_loss[loss=3.341, NarTop10Accuracy=0.6498, over 5925.37 frames. ], batch size: 13, lr: 2.50e-03
2024-08-06 14:25:37,996 INFO [trainer.py:765] (0/8) Epoch 34, batch 1100, train_loss[loss=3.548, NarTop10Accuracy=0.6182, over 6921.00 frames. ], tot_loss[loss=3.349, NarTop10Accuracy=0.648, over 5956.47 frames. ], batch size: 17, lr: 2.50e-03
2024-08-06 14:26:13,974 INFO [trainer.py:765] (0/8) Epoch 34, batch 1200, train_loss[loss=3.357, NarTop10Accuracy=0.6568, over 7086.00 frames. ], tot_loss[loss=3.341, NarTop10Accuracy=0.6495, over 5955.85 frames. ], batch size: 30, lr: 2.50e-03
2024-08-06 14:26:52,652 INFO [trainer.py:765] (0/8) Epoch 34, batch 1300, train_loss[loss=3.637, NarTop10Accuracy=0.5877, over 5080.00 frames. ], tot_loss[loss=3.342, NarTop10Accuracy=0.6491, over 6024.94 frames. ], batch size: 6, lr: 2.50e-03
2024-08-06 14:27:24,383 INFO [trainer.py:765] (0/8) Epoch 34, batch 1400, train_loss[loss=3.118, NarTop10Accuracy=0.7027, over 6224.00 frames. ], tot_loss[loss=3.348, NarTop10Accuracy=0.6479, over 6048.03 frames. ], batch size: 11, lr: 2.50e-03
2024-08-06 14:27:52,726 INFO [trainer.py:765] (0/8) Epoch 34, batch 1500, train_loss[loss=3.751, NarTop10Accuracy=0.567, over 6755.00 frames. ], tot_loss[loss=3.343, NarTop10Accuracy=0.6495, over 5981.86 frames. ], batch size: 49, lr: 2.50e-03
2024-08-06 14:28:20,672 INFO [trainer.py:765] (0/8) Epoch 34, batch 1600, train_loss[loss=3.378, NarTop10Accuracy=0.6499, over 7198.00 frames. ], tot_loss[loss=3.36, NarTop10Accuracy=0.646, over 5966.01 frames. ], batch size: 22, lr: 2.50e-03
2024-08-06 14:28:47,384 INFO [trainer.py:765] (0/8) Epoch 34, batch 1700, train_loss[loss=3.394, NarTop10Accuracy=0.6391, over 6084.00 frames. ], tot_loss[loss=3.375, NarTop10Accuracy=0.643, over 5937.25 frames. ], batch size: 13, lr: 2.49e-03
2024-08-06 14:29:14,009 INFO [trainer.py:765] (0/8) Epoch 34, batch 1800, train_loss[loss=3.61, NarTop10Accuracy=0.5965, over 7124.00 frames. ], tot_loss[loss=3.368, NarTop10Accuracy=0.6446, over 6010.67 frames. ], batch size: 22, lr: 2.49e-03
2024-08-06 14:29:17,946 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-87000.pt
2024-08-06 14:29:43,752 INFO [trainer.py:765] (0/8) Epoch 34, batch 1900, train_loss[loss=3.684, NarTop10Accuracy=0.5868, over 5995.00 frames. ], tot_loss[loss=3.377, NarTop10Accuracy=0.6424, over 6039.93 frames. ], batch size: 50, lr: 2.49e-03
2024-08-06 14:30:09,515 INFO [trainer.py:765] (0/8) Epoch 34, batch 2000, train_loss[loss=3.587, NarTop10Accuracy=0.6015, over 5902.00 frames. ], tot_loss[loss=3.379, NarTop10Accuracy=0.6424, over 6010.19 frames. ], batch size: 49, lr: 2.49e-03
2024-08-06 14:30:35,015 INFO [trainer.py:765] (0/8) Epoch 34, batch 2100, train_loss[loss=3.2, NarTop10Accuracy=0.6676, over 3914.00 frames. ], tot_loss[loss=3.365, NarTop10Accuracy=0.6451, over 5994.64 frames. ], batch size: 4, lr: 2.49e-03
2024-08-06 14:31:00,510 INFO [trainer.py:765] (0/8) Epoch 34, batch 2200, train_loss[loss=3.368, NarTop10Accuracy=0.6442, over 7212.00 frames. ], tot_loss[loss=3.365, NarTop10Accuracy=0.645, over 6043.81 frames. ], batch size: 30, lr: 2.49e-03
2024-08-06 14:31:25,979 INFO [trainer.py:765] (0/8) Epoch 34, batch 2300, train_loss[loss=3.348, NarTop10Accuracy=0.6416, over 5747.00 frames. ], tot_loss[loss=3.373, NarTop10Accuracy=0.6433, over 6072.35 frames. ], batch size: 9, lr: 2.49e-03
2024-08-06 14:31:50,751 INFO [trainer.py:765] (0/8) Epoch 34, batch 2400, train_loss[loss=3.269, NarTop10Accuracy=0.6634, over 5170.00 frames. ], tot_loss[loss=3.378, NarTop10Accuracy=0.6425, over 5908.34 frames. ], batch size: 7, lr: 2.48e-03
2024-08-06 14:32:14,249 INFO [trainer.py:765] (0/8) Epoch 34, batch 2500, train_loss[loss=3.06, NarTop10Accuracy=0.706, over 5252.00 frames. ], tot_loss[loss=3.36, NarTop10Accuracy=0.6462, over 5555.62 frames. ], batch size: 6, lr: 2.48e-03
2024-08-06 14:32:35,409 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 14:32:35,413 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-34.pt
2024-08-06 14:33:26,337 INFO [trainer.py:765] (0/8) Epoch 35, batch 100, train_loss[loss=3.247, NarTop10Accuracy=0.6673, over 7580.00 frames. ], tot_loss[loss=3.307, NarTop10Accuracy=0.6588, over 2372.96 frames. ], batch size: 31, lr: 2.44e-03
2024-08-06 14:34:03,582 INFO [trainer.py:765] (0/8) Epoch 35, batch 200, train_loss[loss=3.417, NarTop10Accuracy=0.6387, over 6886.00 frames. ], tot_loss[loss=3.301, NarTop10Accuracy=0.6597, over 3873.10 frames. ], batch size: 17, lr: 2.44e-03
2024-08-06 14:34:13,185 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-88000.pt
2024-08-06 14:34:16,712 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 14:34:23,574 INFO [trainer.py:811] (0/8) Epoch 35, validation: loss=3.163, NarTop10Accuracy=0.689, over 1907754.00 frames. 
2024-08-06 14:34:23,575 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 14:34:24,109 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.644e+02 2.042e+02 2.203e+02 2.360e+02 4.181e+02, threshold=4.406e+02, percent-clipped=0.0
2024-08-06 14:34:44,664 INFO [trainer.py:765] (0/8) Epoch 35, batch 300, train_loss[loss=3.678, NarTop10Accuracy=0.5779, over 7099.00 frames. ], tot_loss[loss=3.301, NarTop10Accuracy=0.6592, over 4681.74 frames. ], batch size: 22, lr: 2.44e-03
2024-08-06 14:35:13,542 INFO [trainer.py:765] (0/8) Epoch 35, batch 400, train_loss[loss=3.161, NarTop10Accuracy=0.6888, over 5081.00 frames. ], tot_loss[loss=3.308, NarTop10Accuracy=0.6576, over 5138.55 frames. ], batch size: 7, lr: 2.44e-03
2024-08-06 14:35:48,187 INFO [trainer.py:765] (0/8) Epoch 35, batch 500, train_loss[loss=3.436, NarTop10Accuracy=0.6197, over 6138.00 frames. ], tot_loss[loss=3.307, NarTop10Accuracy=0.6578, over 5406.85 frames. ], batch size: 11, lr: 2.44e-03
2024-08-06 14:36:22,747 INFO [trainer.py:765] (0/8) Epoch 35, batch 600, train_loss[loss=3.254, NarTop10Accuracy=0.6524, over 5716.00 frames. ], tot_loss[loss=3.313, NarTop10Accuracy=0.6562, over 5676.16 frames. ], batch size: 9, lr: 2.44e-03
2024-08-06 14:36:57,825 INFO [trainer.py:765] (0/8) Epoch 35, batch 700, train_loss[loss=3.368, NarTop10Accuracy=0.6467, over 5114.00 frames. ], tot_loss[loss=3.323, NarTop10Accuracy=0.6541, over 5747.68 frames. ], batch size: 6, lr: 2.44e-03
2024-08-06 14:37:29,768 INFO [trainer.py:765] (0/8) Epoch 35, batch 800, train_loss[loss=3.088, NarTop10Accuracy=0.6818, over 4217.00 frames. ], tot_loss[loss=3.323, NarTop10Accuracy=0.6542, over 5808.12 frames. ], batch size: 5, lr: 2.43e-03
2024-08-06 14:38:03,303 INFO [trainer.py:765] (0/8) Epoch 35, batch 900, train_loss[loss=3.257, NarTop10Accuracy=0.6751, over 6320.00 frames. ], tot_loss[loss=3.319, NarTop10Accuracy=0.6548, over 5836.35 frames. ], batch size: 13, lr: 2.43e-03
2024-08-06 14:38:43,708 INFO [trainer.py:765] (0/8) Epoch 35, batch 1000, train_loss[loss=3.588, NarTop10Accuracy=0.6034, over 6209.00 frames. ], tot_loss[loss=3.326, NarTop10Accuracy=0.6537, over 5922.27 frames. ], batch size: 13, lr: 2.43e-03
2024-08-06 14:39:16,566 INFO [trainer.py:765] (0/8) Epoch 35, batch 1100, train_loss[loss=3.538, NarTop10Accuracy=0.6101, over 6746.00 frames. ], tot_loss[loss=3.327, NarTop10Accuracy=0.6528, over 5962.99 frames. ], batch size: 17, lr: 2.43e-03
2024-08-06 14:39:50,837 INFO [trainer.py:765] (0/8) Epoch 35, batch 1200, train_loss[loss=3.217, NarTop10Accuracy=0.6767, over 7632.00 frames. ], tot_loss[loss=3.338, NarTop10Accuracy=0.6505, over 5954.05 frames. ], batch size: 32, lr: 2.43e-03
2024-08-06 14:40:05,950 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-89000.pt
2024-08-06 14:40:33,952 INFO [trainer.py:765] (0/8) Epoch 35, batch 1300, train_loss[loss=3.274, NarTop10Accuracy=0.6665, over 5023.00 frames. ], tot_loss[loss=3.333, NarTop10Accuracy=0.6515, over 6017.40 frames. ], batch size: 6, lr: 2.43e-03
2024-08-06 14:41:03,183 INFO [trainer.py:765] (0/8) Epoch 35, batch 1400, train_loss[loss=3.458, NarTop10Accuracy=0.6212, over 6039.00 frames. ], tot_loss[loss=3.351, NarTop10Accuracy=0.6477, over 6035.73 frames. ], batch size: 11, lr: 2.43e-03
2024-08-06 14:41:33,823 INFO [trainer.py:765] (0/8) Epoch 35, batch 1500, train_loss[loss=3.465, NarTop10Accuracy=0.6258, over 6467.00 frames. ], tot_loss[loss=3.346, NarTop10Accuracy=0.6489, over 5961.48 frames. ], batch size: 49, lr: 2.43e-03
2024-08-06 14:42:01,777 INFO [trainer.py:765] (0/8) Epoch 35, batch 1600, train_loss[loss=3.509, NarTop10Accuracy=0.6057, over 7069.00 frames. ], tot_loss[loss=3.355, NarTop10Accuracy=0.647, over 5955.26 frames. ], batch size: 22, lr: 2.42e-03
2024-08-06 14:42:28,466 INFO [trainer.py:765] (0/8) Epoch 35, batch 1700, train_loss[loss=3.27, NarTop10Accuracy=0.6548, over 6337.00 frames. ], tot_loss[loss=3.348, NarTop10Accuracy=0.6485, over 5938.90 frames. ], batch size: 13, lr: 2.42e-03
2024-08-06 14:42:55,040 INFO [trainer.py:765] (0/8) Epoch 35, batch 1800, train_loss[loss=3.259, NarTop10Accuracy=0.6655, over 6972.00 frames. ], tot_loss[loss=3.354, NarTop10Accuracy=0.6469, over 6019.72 frames. ], batch size: 22, lr: 2.42e-03
2024-08-06 14:43:21,646 INFO [trainer.py:765] (0/8) Epoch 35, batch 1900, train_loss[loss=3.42, NarTop10Accuracy=0.6435, over 5934.00 frames. ], tot_loss[loss=3.356, NarTop10Accuracy=0.6466, over 6043.15 frames. ], batch size: 49, lr: 2.42e-03
2024-08-06 14:43:47,367 INFO [trainer.py:765] (0/8) Epoch 35, batch 2000, train_loss[loss=3.449, NarTop10Accuracy=0.6301, over 5955.00 frames. ], tot_loss[loss=3.353, NarTop10Accuracy=0.6475, over 6001.78 frames. ], batch size: 49, lr: 2.42e-03
2024-08-06 14:44:12,856 INFO [trainer.py:765] (0/8) Epoch 35, batch 2100, train_loss[loss=3.28, NarTop10Accuracy=0.6622, over 4047.00 frames. ], tot_loss[loss=3.356, NarTop10Accuracy=0.6473, over 6007.44 frames. ], batch size: 4, lr: 2.42e-03
2024-08-06 14:44:38,388 INFO [trainer.py:765] (0/8) Epoch 35, batch 2200, train_loss[loss=3.547, NarTop10Accuracy=0.6177, over 7525.00 frames. ], tot_loss[loss=3.353, NarTop10Accuracy=0.6475, over 6046.15 frames. ], batch size: 31, lr: 2.42e-03
2024-08-06 14:44:47,199 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-90000.pt
2024-08-06 14:44:50,805 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 14:44:57,441 INFO [trainer.py:811] (0/8) Epoch 35, validation: loss=3.219, NarTop10Accuracy=0.6773, over 1907754.00 frames. 
2024-08-06 14:44:57,441 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 14:44:57,973 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.690e+02 2.083e+02 2.237e+02 2.412e+02 3.944e+02, threshold=4.474e+02, percent-clipped=0.0
2024-08-06 14:45:14,099 INFO [trainer.py:765] (0/8) Epoch 35, batch 2300, train_loss[loss=3.177, NarTop10Accuracy=0.6805, over 5874.00 frames. ], tot_loss[loss=3.361, NarTop10Accuracy=0.6463, over 6067.67 frames. ], batch size: 9, lr: 2.41e-03
2024-08-06 14:45:38,819 INFO [trainer.py:765] (0/8) Epoch 35, batch 2400, train_loss[loss=3.386, NarTop10Accuracy=0.6475, over 5869.00 frames. ], tot_loss[loss=3.366, NarTop10Accuracy=0.6452, over 5869.71 frames. ], batch size: 48, lr: 2.41e-03
2024-08-06 14:46:02,146 INFO [trainer.py:765] (0/8) Epoch 35, batch 2500, train_loss[loss=3.265, NarTop10Accuracy=0.6726, over 5113.00 frames. ], tot_loss[loss=3.33, NarTop10Accuracy=0.652, over 5540.34 frames. ], batch size: 6, lr: 2.41e-03
2024-08-06 14:46:23,241 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 14:46:23,243 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-35.pt
2024-08-06 14:47:25,441 INFO [trainer.py:765] (0/8) Epoch 36, batch 100, train_loss[loss=3.235, NarTop10Accuracy=0.6733, over 7215.00 frames. ], tot_loss[loss=3.311, NarTop10Accuracy=0.6565, over 2381.26 frames. ], batch size: 31, lr: 2.38e-03
2024-08-06 14:47:58,358 INFO [trainer.py:765] (0/8) Epoch 36, batch 200, train_loss[loss=3.267, NarTop10Accuracy=0.6613, over 6901.00 frames. ], tot_loss[loss=3.295, NarTop10Accuracy=0.66, over 3877.22 frames. ], batch size: 17, lr: 2.37e-03
2024-08-06 14:48:30,724 INFO [trainer.py:765] (0/8) Epoch 36, batch 300, train_loss[loss=3.265, NarTop10Accuracy=0.6733, over 7260.00 frames. ], tot_loss[loss=3.293, NarTop10Accuracy=0.6605, over 4686.20 frames. ], batch size: 22, lr: 2.37e-03
2024-08-06 14:49:04,814 INFO [trainer.py:765] (0/8) Epoch 36, batch 400, train_loss[loss=3.195, NarTop10Accuracy=0.6981, over 5138.00 frames. ], tot_loss[loss=3.295, NarTop10Accuracy=0.6607, over 5154.58 frames. ], batch size: 7, lr: 2.37e-03
2024-08-06 14:49:36,588 INFO [trainer.py:765] (0/8) Epoch 36, batch 500, train_loss[loss=3.456, NarTop10Accuracy=0.6357, over 6170.00 frames. ], tot_loss[loss=3.292, NarTop10Accuracy=0.6613, over 5426.53 frames. ], batch size: 11, lr: 2.37e-03
2024-08-06 14:50:09,654 INFO [trainer.py:765] (0/8) Epoch 36, batch 600, train_loss[loss=3.004, NarTop10Accuracy=0.7149, over 5816.00 frames. ], tot_loss[loss=3.311, NarTop10Accuracy=0.6569, over 5680.17 frames. ], batch size: 9, lr: 2.37e-03
2024-08-06 14:50:29,786 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-91000.pt
2024-08-06 14:50:46,514 INFO [trainer.py:765] (0/8) Epoch 36, batch 700, train_loss[loss=3.165, NarTop10Accuracy=0.6902, over 5038.00 frames. ], tot_loss[loss=3.317, NarTop10Accuracy=0.6557, over 5737.22 frames. ], batch size: 6, lr: 2.37e-03
2024-08-06 14:51:23,702 INFO [trainer.py:765] (0/8) Epoch 36, batch 800, train_loss[loss=3.115, NarTop10Accuracy=0.6728, over 5128.00 frames. ], tot_loss[loss=3.321, NarTop10Accuracy=0.6547, over 5794.96 frames. ], batch size: 6, lr: 2.37e-03
2024-08-06 14:51:54,346 INFO [trainer.py:765] (0/8) Epoch 36, batch 900, train_loss[loss=3.176, NarTop10Accuracy=0.6878, over 6397.00 frames. ], tot_loss[loss=3.322, NarTop10Accuracy=0.654, over 5825.56 frames. ], batch size: 13, lr: 2.36e-03
2024-08-06 14:52:30,324 INFO [trainer.py:765] (0/8) Epoch 36, batch 1000, train_loss[loss=3.113, NarTop10Accuracy=0.7074, over 6363.00 frames. ], tot_loss[loss=3.32, NarTop10Accuracy=0.6545, over 5915.45 frames. ], batch size: 13, lr: 2.36e-03
2024-08-06 14:53:06,863 INFO [trainer.py:765] (0/8) Epoch 36, batch 1100, train_loss[loss=3.299, NarTop10Accuracy=0.6778, over 6814.00 frames. ], tot_loss[loss=3.343, NarTop10Accuracy=0.6501, over 5958.27 frames. ], batch size: 17, lr: 2.36e-03
2024-08-06 14:53:40,248 INFO [trainer.py:765] (0/8) Epoch 36, batch 1200, train_loss[loss=3.277, NarTop10Accuracy=0.6586, over 7475.00 frames. ], tot_loss[loss=3.327, NarTop10Accuracy=0.6534, over 5961.39 frames. ], batch size: 31, lr: 2.36e-03
2024-08-06 14:54:15,855 INFO [trainer.py:765] (0/8) Epoch 36, batch 1300, train_loss[loss=3.269, NarTop10Accuracy=0.659, over 5165.00 frames. ], tot_loss[loss=3.325, NarTop10Accuracy=0.6535, over 6024.05 frames. ], batch size: 6, lr: 2.36e-03
2024-08-06 14:54:51,540 INFO [trainer.py:765] (0/8) Epoch 36, batch 1400, train_loss[loss=3.315, NarTop10Accuracy=0.66, over 6275.00 frames. ], tot_loss[loss=3.332, NarTop10Accuracy=0.6524, over 6046.84 frames. ], batch size: 11, lr: 2.36e-03
2024-08-06 14:55:21,802 INFO [trainer.py:765] (0/8) Epoch 36, batch 1500, train_loss[loss=3.572, NarTop10Accuracy=0.5966, over 5598.00 frames. ], tot_loss[loss=3.337, NarTop10Accuracy=0.6511, over 5979.96 frames. ], batch size: 49, lr: 2.36e-03
2024-08-06 14:55:49,902 INFO [trainer.py:765] (0/8) Epoch 36, batch 1600, train_loss[loss=3.164, NarTop10Accuracy=0.6841, over 6984.00 frames. ], tot_loss[loss=3.34, NarTop10Accuracy=0.6509, over 5941.22 frames. ], batch size: 22, lr: 2.36e-03
2024-08-06 14:56:04,131 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-92000.pt
2024-08-06 14:56:07,767 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 14:56:14,600 INFO [trainer.py:811] (0/8) Epoch 36, validation: loss=3.22, NarTop10Accuracy=0.6784, over 1907754.00 frames. 
2024-08-06 14:56:14,601 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 14:56:15,104 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.690e+02 2.063e+02 2.224e+02 2.398e+02 5.290e+02, threshold=4.447e+02, percent-clipped=0.1
2024-08-06 14:56:27,177 INFO [trainer.py:765] (0/8) Epoch 36, batch 1700, train_loss[loss=3.293, NarTop10Accuracy=0.6632, over 6643.00 frames. ], tot_loss[loss=3.342, NarTop10Accuracy=0.6502, over 5938.46 frames. ], batch size: 14, lr: 2.35e-03
2024-08-06 14:56:53,758 INFO [trainer.py:765] (0/8) Epoch 36, batch 1800, train_loss[loss=3.47, NarTop10Accuracy=0.6207, over 7338.00 frames. ], tot_loss[loss=3.342, NarTop10Accuracy=0.6501, over 6009.30 frames. ], batch size: 22, lr: 2.35e-03
2024-08-06 14:57:20,335 INFO [trainer.py:765] (0/8) Epoch 36, batch 1900, train_loss[loss=3.303, NarTop10Accuracy=0.6553, over 6116.00 frames. ], tot_loss[loss=3.35, NarTop10Accuracy=0.6488, over 6057.87 frames. ], batch size: 50, lr: 2.35e-03
2024-08-06 14:57:46,056 INFO [trainer.py:765] (0/8) Epoch 36, batch 2000, train_loss[loss=3.654, NarTop10Accuracy=0.5832, over 6397.00 frames. ], tot_loss[loss=3.356, NarTop10Accuracy=0.6474, over 6026.92 frames. ], batch size: 49, lr: 2.35e-03
2024-08-06 14:58:11,404 INFO [trainer.py:765] (0/8) Epoch 36, batch 2100, train_loss[loss=3.042, NarTop10Accuracy=0.7175, over 3909.00 frames. ], tot_loss[loss=3.351, NarTop10Accuracy=0.6482, over 6015.42 frames. ], batch size: 4, lr: 2.35e-03
2024-08-06 14:58:36,832 INFO [trainer.py:765] (0/8) Epoch 36, batch 2200, train_loss[loss=3.598, NarTop10Accuracy=0.5882, over 7253.00 frames. ], tot_loss[loss=3.35, NarTop10Accuracy=0.6486, over 6053.70 frames. ], batch size: 30, lr: 2.35e-03
2024-08-06 14:59:02,344 INFO [trainer.py:765] (0/8) Epoch 36, batch 2300, train_loss[loss=3.093, NarTop10Accuracy=0.6936, over 5740.00 frames. ], tot_loss[loss=3.369, NarTop10Accuracy=0.645, over 6070.57 frames. ], batch size: 9, lr: 2.35e-03
2024-08-06 14:59:27,094 INFO [trainer.py:765] (0/8) Epoch 36, batch 2400, train_loss[loss=3.5, NarTop10Accuracy=0.6255, over 5241.00 frames. ], tot_loss[loss=3.376, NarTop10Accuracy=0.6436, over 5894.42 frames. ], batch size: 7, lr: 2.35e-03
2024-08-06 14:59:50,503 INFO [trainer.py:765] (0/8) Epoch 36, batch 2500, train_loss[loss=3.676, NarTop10Accuracy=0.5948, over 4885.00 frames. ], tot_loss[loss=3.357, NarTop10Accuracy=0.6473, over 5532.38 frames. ], batch size: 6, lr: 2.34e-03
2024-08-06 15:00:11,193 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 15:00:11,197 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-36.pt
2024-08-06 15:00:58,572 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-93000.pt
2024-08-06 15:01:14,218 INFO [trainer.py:765] (0/8) Epoch 37, batch 100, train_loss[loss=3.191, NarTop10Accuracy=0.6836, over 7413.00 frames. ], tot_loss[loss=3.291, NarTop10Accuracy=0.6618, over 2380.77 frames. ], batch size: 31, lr: 2.31e-03
2024-08-06 15:01:44,098 INFO [trainer.py:765] (0/8) Epoch 37, batch 200, train_loss[loss=3.085, NarTop10Accuracy=0.6983, over 6909.00 frames. ], tot_loss[loss=3.278, NarTop10Accuracy=0.6638, over 3869.96 frames. ], batch size: 17, lr: 2.31e-03
2024-08-06 15:02:17,383 INFO [trainer.py:765] (0/8) Epoch 37, batch 300, train_loss[loss=3.18, NarTop10Accuracy=0.6889, over 7323.00 frames. ], tot_loss[loss=3.278, NarTop10Accuracy=0.6635, over 4670.79 frames. ], batch size: 23, lr: 2.31e-03
2024-08-06 15:02:48,346 INFO [trainer.py:765] (0/8) Epoch 37, batch 400, train_loss[loss=3.578, NarTop10Accuracy=0.6072, over 5223.00 frames. ], tot_loss[loss=3.293, NarTop10Accuracy=0.6611, over 5122.61 frames. ], batch size: 7, lr: 2.31e-03
2024-08-06 15:03:26,570 INFO [trainer.py:765] (0/8) Epoch 37, batch 500, train_loss[loss=2.978, NarTop10Accuracy=0.7217, over 6189.00 frames. ], tot_loss[loss=3.307, NarTop10Accuracy=0.6583, over 5401.54 frames. ], batch size: 11, lr: 2.30e-03
2024-08-06 15:03:58,033 INFO [trainer.py:765] (0/8) Epoch 37, batch 600, train_loss[loss=3.113, NarTop10Accuracy=0.6895, over 5803.00 frames. ], tot_loss[loss=3.304, NarTop10Accuracy=0.6582, over 5673.00 frames. ], batch size: 9, lr: 2.30e-03
2024-08-06 15:04:30,248 INFO [trainer.py:765] (0/8) Epoch 37, batch 700, train_loss[loss=3.308, NarTop10Accuracy=0.6515, over 5083.00 frames. ], tot_loss[loss=3.31, NarTop10Accuracy=0.6562, over 5741.82 frames. ], batch size: 6, lr: 2.30e-03
2024-08-06 15:05:12,163 INFO [trainer.py:765] (0/8) Epoch 37, batch 800, train_loss[loss=3.201, NarTop10Accuracy=0.668, over 5136.00 frames. ], tot_loss[loss=3.326, NarTop10Accuracy=0.6529, over 5797.89 frames. ], batch size: 6, lr: 2.30e-03
2024-08-06 15:05:40,606 INFO [trainer.py:765] (0/8) Epoch 37, batch 900, train_loss[loss=2.999, NarTop10Accuracy=0.7198, over 6166.00 frames. ], tot_loss[loss=3.32, NarTop10Accuracy=0.654, over 5819.68 frames. ], batch size: 13, lr: 2.30e-03
2024-08-06 15:06:15,608 INFO [trainer.py:765] (0/8) Epoch 37, batch 1000, train_loss[loss=3.163, NarTop10Accuracy=0.6811, over 6164.00 frames. ], tot_loss[loss=3.329, NarTop10Accuracy=0.6524, over 5920.25 frames. ], batch size: 13, lr: 2.30e-03
2024-08-06 15:06:42,490 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-94000.pt
2024-08-06 15:06:46,411 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 15:06:53,168 INFO [trainer.py:811] (0/8) Epoch 37, validation: loss=3.234, NarTop10Accuracy=0.6744, over 1907754.00 frames. 
2024-08-06 15:06:53,169 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 15:06:53,809 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.659e+02 2.068e+02 2.238e+02 2.409e+02 6.392e+02, threshold=4.475e+02, percent-clipped=0.1
2024-08-06 15:07:01,306 INFO [trainer.py:765] (0/8) Epoch 37, batch 1100, train_loss[loss=3.572, NarTop10Accuracy=0.6016, over 6759.00 frames. ], tot_loss[loss=3.325, NarTop10Accuracy=0.6533, over 5954.03 frames. ], batch size: 17, lr: 2.30e-03
2024-08-06 15:07:32,718 INFO [trainer.py:765] (0/8) Epoch 37, batch 1200, train_loss[loss=3.289, NarTop10Accuracy=0.6681, over 7305.00 frames. ], tot_loss[loss=3.318, NarTop10Accuracy=0.6544, over 5957.21 frames. ], batch size: 30, lr: 2.30e-03
2024-08-06 15:08:04,777 INFO [trainer.py:765] (0/8) Epoch 37, batch 1300, train_loss[loss=3.451, NarTop10Accuracy=0.6232, over 4967.00 frames. ], tot_loss[loss=3.322, NarTop10Accuracy=0.6533, over 6022.76 frames. ], batch size: 6, lr: 2.29e-03
2024-08-06 15:08:47,879 INFO [trainer.py:765] (0/8) Epoch 37, batch 1400, train_loss[loss=2.981, NarTop10Accuracy=0.7108, over 6166.00 frames. ], tot_loss[loss=3.337, NarTop10Accuracy=0.6506, over 6042.73 frames. ], batch size: 11, lr: 2.29e-03
2024-08-06 15:09:16,180 INFO [trainer.py:765] (0/8) Epoch 37, batch 1500, train_loss[loss=3.622, NarTop10Accuracy=0.5939, over 6240.00 frames. ], tot_loss[loss=3.33, NarTop10Accuracy=0.6521, over 5967.80 frames. ], batch size: 49, lr: 2.29e-03
2024-08-06 15:09:44,190 INFO [trainer.py:765] (0/8) Epoch 37, batch 1600, train_loss[loss=3.482, NarTop10Accuracy=0.6258, over 7033.00 frames. ], tot_loss[loss=3.331, NarTop10Accuracy=0.6513, over 5946.62 frames. ], batch size: 22, lr: 2.29e-03
2024-08-06 15:10:11,082 INFO [trainer.py:765] (0/8) Epoch 37, batch 1700, train_loss[loss=3.145, NarTop10Accuracy=0.6867, over 6357.00 frames. ], tot_loss[loss=3.334, NarTop10Accuracy=0.651, over 5952.94 frames. ], batch size: 13, lr: 2.29e-03
2024-08-06 15:10:37,752 INFO [trainer.py:765] (0/8) Epoch 37, batch 1800, train_loss[loss=3.495, NarTop10Accuracy=0.6199, over 7340.00 frames. ], tot_loss[loss=3.327, NarTop10Accuracy=0.6525, over 6022.91 frames. ], batch size: 22, lr: 2.29e-03
2024-08-06 15:11:04,270 INFO [trainer.py:765] (0/8) Epoch 37, batch 1900, train_loss[loss=3.621, NarTop10Accuracy=0.5955, over 6243.00 frames. ], tot_loss[loss=3.35, NarTop10Accuracy=0.6479, over 6048.41 frames. ], batch size: 49, lr: 2.29e-03
2024-08-06 15:11:29,941 INFO [trainer.py:765] (0/8) Epoch 37, batch 2000, train_loss[loss=3.532, NarTop10Accuracy=0.6111, over 5832.00 frames. ], tot_loss[loss=3.351, NarTop10Accuracy=0.648, over 6016.96 frames. ], batch size: 48, lr: 2.29e-03
2024-08-06 15:11:48,560 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-95000.pt
2024-08-06 15:11:58,797 INFO [trainer.py:765] (0/8) Epoch 37, batch 2100, train_loss[loss=3.037, NarTop10Accuracy=0.7127, over 3893.00 frames. ], tot_loss[loss=3.344, NarTop10Accuracy=0.6495, over 5994.43 frames. ], batch size: 4, lr: 2.29e-03
2024-08-06 15:12:24,311 INFO [trainer.py:765] (0/8) Epoch 37, batch 2200, train_loss[loss=3.317, NarTop10Accuracy=0.6543, over 7154.00 frames. ], tot_loss[loss=3.343, NarTop10Accuracy=0.6496, over 6028.77 frames. ], batch size: 30, lr: 2.28e-03
2024-08-06 15:12:49,786 INFO [trainer.py:765] (0/8) Epoch 37, batch 2300, train_loss[loss=3.264, NarTop10Accuracy=0.6618, over 5755.00 frames. ], tot_loss[loss=3.35, NarTop10Accuracy=0.648, over 6050.91 frames. ], batch size: 9, lr: 2.28e-03
2024-08-06 15:13:14,526 INFO [trainer.py:765] (0/8) Epoch 37, batch 2400, train_loss[loss=3.186, NarTop10Accuracy=0.6819, over 5228.00 frames. ], tot_loss[loss=3.353, NarTop10Accuracy=0.6476, over 5875.68 frames. ], batch size: 7, lr: 2.28e-03
2024-08-06 15:13:37,942 INFO [trainer.py:765] (0/8) Epoch 37, batch 2500, train_loss[loss=3.831, NarTop10Accuracy=0.5419, over 5144.00 frames. ], tot_loss[loss=3.335, NarTop10Accuracy=0.6509, over 5518.91 frames. ], batch size: 6, lr: 2.28e-03
2024-08-06 15:13:59,219 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 15:13:59,221 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-37.pt
2024-08-06 15:14:50,846 INFO [trainer.py:765] (0/8) Epoch 38, batch 100, train_loss[loss=3.537, NarTop10Accuracy=0.6095, over 7513.00 frames. ], tot_loss[loss=3.315, NarTop10Accuracy=0.6562, over 2367.99 frames. ], batch size: 31, lr: 2.25e-03
2024-08-06 15:15:27,289 INFO [trainer.py:765] (0/8) Epoch 38, batch 200, train_loss[loss=3.172, NarTop10Accuracy=0.6884, over 6845.00 frames. ], tot_loss[loss=3.278, NarTop10Accuracy=0.6638, over 3878.73 frames. ], batch size: 17, lr: 2.25e-03
2024-08-06 15:16:01,281 INFO [trainer.py:765] (0/8) Epoch 38, batch 300, train_loss[loss=3.393, NarTop10Accuracy=0.6398, over 7292.00 frames. ], tot_loss[loss=3.274, NarTop10Accuracy=0.6646, over 4679.86 frames. ], batch size: 22, lr: 2.25e-03
2024-08-06 15:16:32,595 INFO [trainer.py:765] (0/8) Epoch 38, batch 400, train_loss[loss=3.252, NarTop10Accuracy=0.6611, over 5144.00 frames. ], tot_loss[loss=3.277, NarTop10Accuracy=0.6635, over 5125.67 frames. ], batch size: 7, lr: 2.24e-03
2024-08-06 15:17:04,257 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-96000.pt
2024-08-06 15:17:07,576 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 15:17:14,104 INFO [trainer.py:811] (0/8) Epoch 38, validation: loss=3.229, NarTop10Accuracy=0.6755, over 1907754.00 frames. 
2024-08-06 15:17:14,105 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 15:17:14,630 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.687e+02 2.062e+02 2.214e+02 2.396e+02 3.845e+02, threshold=4.429e+02, percent-clipped=0.0
2024-08-06 15:17:16,479 INFO [trainer.py:765] (0/8) Epoch 38, batch 500, train_loss[loss=3.346, NarTop10Accuracy=0.6522, over 6026.00 frames. ], tot_loss[loss=3.27, NarTop10Accuracy=0.6651, over 5382.60 frames. ], batch size: 11, lr: 2.24e-03
2024-08-06 15:17:53,875 INFO [trainer.py:765] (0/8) Epoch 38, batch 600, train_loss[loss=3.278, NarTop10Accuracy=0.67, over 5783.00 frames. ], tot_loss[loss=3.291, NarTop10Accuracy=0.661, over 5658.55 frames. ], batch size: 9, lr: 2.24e-03
2024-08-06 15:18:26,465 INFO [trainer.py:765] (0/8) Epoch 38, batch 700, train_loss[loss=3.241, NarTop10Accuracy=0.6625, over 4988.00 frames. ], tot_loss[loss=3.308, NarTop10Accuracy=0.6572, over 5738.38 frames. ], batch size: 6, lr: 2.24e-03
2024-08-06 15:19:01,129 INFO [trainer.py:765] (0/8) Epoch 38, batch 800, train_loss[loss=3.119, NarTop10Accuracy=0.6906, over 5048.00 frames. ], tot_loss[loss=3.308, NarTop10Accuracy=0.6566, over 5792.08 frames. ], batch size: 6, lr: 2.24e-03
2024-08-06 15:19:36,539 INFO [trainer.py:765] (0/8) Epoch 38, batch 900, train_loss[loss=3.547, NarTop10Accuracy=0.6045, over 6652.00 frames. ], tot_loss[loss=3.315, NarTop10Accuracy=0.6547, over 5807.01 frames. ], batch size: 14, lr: 2.24e-03
2024-08-06 15:20:09,134 INFO [trainer.py:765] (0/8) Epoch 38, batch 1000, train_loss[loss=3.534, NarTop10Accuracy=0.608, over 6234.00 frames. ], tot_loss[loss=3.323, NarTop10Accuracy=0.6535, over 5900.90 frames. ], batch size: 13, lr: 2.24e-03
2024-08-06 15:20:47,345 INFO [trainer.py:765] (0/8) Epoch 38, batch 1100, train_loss[loss=3.34, NarTop10Accuracy=0.6495, over 7053.00 frames. ], tot_loss[loss=3.337, NarTop10Accuracy=0.6509, over 5937.27 frames. ], batch size: 17, lr: 2.24e-03
2024-08-06 15:21:25,594 INFO [trainer.py:765] (0/8) Epoch 38, batch 1200, train_loss[loss=3.389, NarTop10Accuracy=0.6383, over 7150.00 frames. ], tot_loss[loss=3.332, NarTop10Accuracy=0.6516, over 5943.04 frames. ], batch size: 31, lr: 2.23e-03
2024-08-06 15:21:57,556 INFO [trainer.py:765] (0/8) Epoch 38, batch 1300, train_loss[loss=3.143, NarTop10Accuracy=0.6895, over 5109.00 frames. ], tot_loss[loss=3.319, NarTop10Accuracy=0.6543, over 6012.35 frames. ], batch size: 6, lr: 2.23e-03
2024-08-06 15:22:29,467 INFO [trainer.py:765] (0/8) Epoch 38, batch 1400, train_loss[loss=3.028, NarTop10Accuracy=0.7035, over 5981.00 frames. ], tot_loss[loss=3.334, NarTop10Accuracy=0.6516, over 6046.33 frames. ], batch size: 11, lr: 2.23e-03
2024-08-06 15:23:01,195 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-97000.pt
2024-08-06 15:23:06,615 INFO [trainer.py:765] (0/8) Epoch 38, batch 1500, train_loss[loss=3.459, NarTop10Accuracy=0.6308, over 6065.00 frames. ], tot_loss[loss=3.342, NarTop10Accuracy=0.6495, over 5982.05 frames. ], batch size: 49, lr: 2.23e-03
2024-08-06 15:23:34,640 INFO [trainer.py:765] (0/8) Epoch 38, batch 1600, train_loss[loss=3.541, NarTop10Accuracy=0.6073, over 6960.00 frames. ], tot_loss[loss=3.355, NarTop10Accuracy=0.6472, over 5943.97 frames. ], batch size: 22, lr: 2.23e-03
2024-08-06 15:24:01,433 INFO [trainer.py:765] (0/8) Epoch 38, batch 1700, train_loss[loss=3.147, NarTop10Accuracy=0.6991, over 6240.00 frames. ], tot_loss[loss=3.358, NarTop10Accuracy=0.6468, over 5936.85 frames. ], batch size: 13, lr: 2.23e-03
2024-08-06 15:24:28,064 INFO [trainer.py:765] (0/8) Epoch 38, batch 1800, train_loss[loss=3.252, NarTop10Accuracy=0.6728, over 7058.00 frames. ], tot_loss[loss=3.36, NarTop10Accuracy=0.6465, over 5989.95 frames. ], batch size: 22, lr: 2.23e-03
2024-08-06 15:24:54,672 INFO [trainer.py:765] (0/8) Epoch 38, batch 1900, train_loss[loss=3.386, NarTop10Accuracy=0.6449, over 6104.00 frames. ], tot_loss[loss=3.362, NarTop10Accuracy=0.6466, over 6022.17 frames. ], batch size: 50, lr: 2.23e-03
2024-08-06 15:25:20,410 INFO [trainer.py:765] (0/8) Epoch 38, batch 2000, train_loss[loss=3.451, NarTop10Accuracy=0.6309, over 6553.00 frames. ], tot_loss[loss=3.359, NarTop10Accuracy=0.6475, over 6014.71 frames. ], batch size: 49, lr: 2.23e-03
2024-08-06 15:25:45,856 INFO [trainer.py:765] (0/8) Epoch 38, batch 2100, train_loss[loss=3.304, NarTop10Accuracy=0.6584, over 3927.00 frames. ], tot_loss[loss=3.348, NarTop10Accuracy=0.6492, over 6005.97 frames. ], batch size: 4, lr: 2.22e-03
2024-08-06 15:26:11,316 INFO [trainer.py:765] (0/8) Epoch 38, batch 2200, train_loss[loss=3.729, NarTop10Accuracy=0.5734, over 7183.00 frames. ], tot_loss[loss=3.354, NarTop10Accuracy=0.648, over 6037.60 frames. ], batch size: 31, lr: 2.22e-03
2024-08-06 15:26:36,708 INFO [trainer.py:765] (0/8) Epoch 38, batch 2300, train_loss[loss=3.207, NarTop10Accuracy=0.6786, over 5770.00 frames. ], tot_loss[loss=3.365, NarTop10Accuracy=0.6459, over 6061.97 frames. ], batch size: 9, lr: 2.22e-03
2024-08-06 15:27:01,479 INFO [trainer.py:765] (0/8) Epoch 38, batch 2400, train_loss[loss=3.59, NarTop10Accuracy=0.6035, over 6083.00 frames. ], tot_loss[loss=3.379, NarTop10Accuracy=0.6433, over 5883.32 frames. ], batch size: 49, lr: 2.22e-03
2024-08-06 15:27:23,144 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-98000.pt
2024-08-06 15:27:26,744 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 15:27:33,589 INFO [trainer.py:811] (0/8) Epoch 38, validation: loss=3.213, NarTop10Accuracy=0.6782, over 1907754.00 frames. 
2024-08-06 15:27:33,590 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 15:27:34,075 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.659e+02 2.098e+02 2.247e+02 2.437e+02 3.550e+02, threshold=4.494e+02, percent-clipped=0.0
2024-08-06 15:27:35,514 INFO [trainer.py:765] (0/8) Epoch 38, batch 2500, train_loss[loss=2.998, NarTop10Accuracy=0.7071, over 5095.00 frames. ], tot_loss[loss=3.337, NarTop10Accuracy=0.651, over 5531.60 frames. ], batch size: 6, lr: 2.22e-03
2024-08-06 15:27:56,855 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 15:27:56,860 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-38.pt
2024-08-06 15:28:51,227 INFO [trainer.py:765] (0/8) Epoch 39, batch 100, train_loss[loss=3.212, NarTop10Accuracy=0.6722, over 7141.00 frames. ], tot_loss[loss=3.266, NarTop10Accuracy=0.6666, over 2371.47 frames. ], batch size: 30, lr: 2.19e-03
2024-08-06 15:29:28,052 INFO [trainer.py:765] (0/8) Epoch 39, batch 200, train_loss[loss=3.546, NarTop10Accuracy=0.6116, over 6856.00 frames. ], tot_loss[loss=3.258, NarTop10Accuracy=0.667, over 3872.29 frames. ], batch size: 17, lr: 2.19e-03
2024-08-06 15:30:02,018 INFO [trainer.py:765] (0/8) Epoch 39, batch 300, train_loss[loss=3.287, NarTop10Accuracy=0.659, over 7201.00 frames. ], tot_loss[loss=3.279, NarTop10Accuracy=0.6633, over 4681.82 frames. ], batch size: 22, lr: 2.19e-03
2024-08-06 15:30:32,992 INFO [trainer.py:765] (0/8) Epoch 39, batch 400, train_loss[loss=2.964, NarTop10Accuracy=0.7163, over 5237.00 frames. ], tot_loss[loss=3.283, NarTop10Accuracy=0.6621, over 5126.41 frames. ], batch size: 7, lr: 2.19e-03
2024-08-06 15:31:03,569 INFO [trainer.py:765] (0/8) Epoch 39, batch 500, train_loss[loss=3.241, NarTop10Accuracy=0.6684, over 6166.00 frames. ], tot_loss[loss=3.294, NarTop10Accuracy=0.66, over 5399.55 frames. ], batch size: 11, lr: 2.18e-03
2024-08-06 15:31:40,850 INFO [trainer.py:765] (0/8) Epoch 39, batch 600, train_loss[loss=3.156, NarTop10Accuracy=0.688, over 5830.00 frames. ], tot_loss[loss=3.293, NarTop10Accuracy=0.6602, over 5680.82 frames. ], batch size: 9, lr: 2.18e-03
2024-08-06 15:32:14,452 INFO [trainer.py:765] (0/8) Epoch 39, batch 700, train_loss[loss=3.135, NarTop10Accuracy=0.6944, over 5149.00 frames. ], tot_loss[loss=3.306, NarTop10Accuracy=0.6577, over 5754.13 frames. ], batch size: 6, lr: 2.18e-03
2024-08-06 15:32:44,166 INFO [trainer.py:765] (0/8) Epoch 39, batch 800, train_loss[loss=3.377, NarTop10Accuracy=0.6433, over 4989.00 frames. ], tot_loss[loss=3.309, NarTop10Accuracy=0.6573, over 5787.39 frames. ], batch size: 6, lr: 2.18e-03
2024-08-06 15:33:21,117 INFO [trainer.py:765] (0/8) Epoch 39, batch 900, train_loss[loss=3.077, NarTop10Accuracy=0.7075, over 6330.00 frames. ], tot_loss[loss=3.312, NarTop10Accuracy=0.6564, over 5812.17 frames. ], batch size: 13, lr: 2.18e-03
2024-08-06 15:33:28,983 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-99000.pt
2024-08-06 15:34:02,655 INFO [trainer.py:765] (0/8) Epoch 39, batch 1000, train_loss[loss=3.087, NarTop10Accuracy=0.7108, over 6277.00 frames. ], tot_loss[loss=3.311, NarTop10Accuracy=0.6562, over 5915.08 frames. ], batch size: 13, lr: 2.18e-03
2024-08-06 15:34:33,095 INFO [trainer.py:765] (0/8) Epoch 39, batch 1100, train_loss[loss=3.179, NarTop10Accuracy=0.6862, over 6840.00 frames. ], tot_loss[loss=3.316, NarTop10Accuracy=0.6552, over 5944.85 frames. ], batch size: 17, lr: 2.18e-03
2024-08-06 15:35:09,245 INFO [trainer.py:765] (0/8) Epoch 39, batch 1200, train_loss[loss=3.181, NarTop10Accuracy=0.6865, over 6915.00 frames. ], tot_loss[loss=3.32, NarTop10Accuracy=0.6545, over 5941.45 frames. ], batch size: 30, lr: 2.18e-03
2024-08-06 15:35:46,813 INFO [trainer.py:765] (0/8) Epoch 39, batch 1300, train_loss[loss=3.535, NarTop10Accuracy=0.609, over 4289.00 frames. ], tot_loss[loss=3.321, NarTop10Accuracy=0.6546, over 6002.42 frames. ], batch size: 5, lr: 2.18e-03
2024-08-06 15:36:18,850 INFO [trainer.py:765] (0/8) Epoch 39, batch 1400, train_loss[loss=3.11, NarTop10Accuracy=0.7008, over 6070.00 frames. ], tot_loss[loss=3.322, NarTop10Accuracy=0.6545, over 6046.43 frames. ], batch size: 11, lr: 2.17e-03
2024-08-06 15:36:47,214 INFO [trainer.py:765] (0/8) Epoch 39, batch 1500, train_loss[loss=3.363, NarTop10Accuracy=0.65, over 6300.00 frames. ], tot_loss[loss=3.331, NarTop10Accuracy=0.6523, over 5982.05 frames. ], batch size: 48, lr: 2.17e-03
2024-08-06 15:37:15,216 INFO [trainer.py:765] (0/8) Epoch 39, batch 1600, train_loss[loss=3.397, NarTop10Accuracy=0.641, over 7168.00 frames. ], tot_loss[loss=3.346, NarTop10Accuracy=0.6493, over 5969.16 frames. ], batch size: 22, lr: 2.17e-03
2024-08-06 15:37:41,883 INFO [trainer.py:765] (0/8) Epoch 39, batch 1700, train_loss[loss=3.227, NarTop10Accuracy=0.672, over 6578.00 frames. ], tot_loss[loss=3.354, NarTop10Accuracy=0.648, over 5937.14 frames. ], batch size: 14, lr: 2.17e-03
2024-08-06 15:38:08,510 INFO [trainer.py:765] (0/8) Epoch 39, batch 1800, train_loss[loss=3.322, NarTop10Accuracy=0.65, over 7336.00 frames. ], tot_loss[loss=3.343, NarTop10Accuracy=0.6504, over 5999.91 frames. ], batch size: 22, lr: 2.17e-03
2024-08-06 15:38:35,253 INFO [trainer.py:765] (0/8) Epoch 39, batch 1900, train_loss[loss=3.46, NarTop10Accuracy=0.6231, over 6186.00 frames. ], tot_loss[loss=3.354, NarTop10Accuracy=0.648, over 6025.58 frames. ], batch size: 50, lr: 2.17e-03
2024-08-06 15:38:37,990 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-100000.pt
2024-08-06 15:38:41,561 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 15:38:48,262 INFO [trainer.py:811] (0/8) Epoch 39, validation: loss=3.177, NarTop10Accuracy=0.6866, over 1907754.00 frames. 
2024-08-06 15:38:48,262 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 15:38:48,768 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.714e+02 2.106e+02 2.266e+02 2.462e+02 4.274e+02, threshold=4.532e+02, percent-clipped=0.0
2024-08-06 15:39:11,226 INFO [trainer.py:765] (0/8) Epoch 39, batch 2000, train_loss[loss=3.324, NarTop10Accuracy=0.6478, over 6129.00 frames. ], tot_loss[loss=3.337, NarTop10Accuracy=0.6511, over 5992.47 frames. ], batch size: 48, lr: 2.17e-03
2024-08-06 15:39:36,692 INFO [trainer.py:765] (0/8) Epoch 39, batch 2100, train_loss[loss=3.483, NarTop10Accuracy=0.6307, over 4800.00 frames. ], tot_loss[loss=3.348, NarTop10Accuracy=0.6487, over 5986.94 frames. ], batch size: 5, lr: 2.17e-03
2024-08-06 15:40:02,086 INFO [trainer.py:765] (0/8) Epoch 39, batch 2200, train_loss[loss=3.642, NarTop10Accuracy=0.5913, over 7143.00 frames. ], tot_loss[loss=3.346, NarTop10Accuracy=0.6488, over 6030.30 frames. ], batch size: 30, lr: 2.17e-03
2024-08-06 15:40:27,496 INFO [trainer.py:765] (0/8) Epoch 39, batch 2300, train_loss[loss=3.009, NarTop10Accuracy=0.7179, over 5680.00 frames. ], tot_loss[loss=3.349, NarTop10Accuracy=0.6485, over 6068.28 frames. ], batch size: 9, lr: 2.16e-03
2024-08-06 15:40:52,331 INFO [trainer.py:765] (0/8) Epoch 39, batch 2400, train_loss[loss=3.368, NarTop10Accuracy=0.641, over 5974.00 frames. ], tot_loss[loss=3.359, NarTop10Accuracy=0.6467, over 5884.07 frames. ], batch size: 49, lr: 2.16e-03
2024-08-06 15:41:15,695 INFO [trainer.py:765] (0/8) Epoch 39, batch 2500, train_loss[loss=3.513, NarTop10Accuracy=0.6232, over 5076.00 frames. ], tot_loss[loss=3.329, NarTop10Accuracy=0.6526, over 5533.16 frames. ], batch size: 6, lr: 2.16e-03
2024-08-06 15:41:37,020 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 15:41:37,025 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-39.pt
2024-08-06 15:42:35,254 INFO [trainer.py:765] (0/8) Epoch 40, batch 100, train_loss[loss=3.643, NarTop10Accuracy=0.5898, over 7318.00 frames. ], tot_loss[loss=3.316, NarTop10Accuracy=0.6553, over 2361.92 frames. ], batch size: 31, lr: 2.13e-03
2024-08-06 15:43:09,645 INFO [trainer.py:765] (0/8) Epoch 40, batch 200, train_loss[loss=3.438, NarTop10Accuracy=0.6308, over 6828.00 frames. ], tot_loss[loss=3.276, NarTop10Accuracy=0.6637, over 3853.89 frames. ], batch size: 17, lr: 2.13e-03
2024-08-06 15:43:43,738 INFO [trainer.py:765] (0/8) Epoch 40, batch 300, train_loss[loss=3.352, NarTop10Accuracy=0.6378, over 7350.00 frames. ], tot_loss[loss=3.283, NarTop10Accuracy=0.6623, over 4674.59 frames. ], batch size: 22, lr: 2.13e-03
2024-08-06 15:43:52,264 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-101000.pt
2024-08-06 15:44:18,202 INFO [trainer.py:765] (0/8) Epoch 40, batch 400, train_loss[loss=3.053, NarTop10Accuracy=0.7027, over 5066.00 frames. ], tot_loss[loss=3.275, NarTop10Accuracy=0.6638, over 5143.89 frames. ], batch size: 7, lr: 2.13e-03
2024-08-06 15:44:50,257 INFO [trainer.py:765] (0/8) Epoch 40, batch 500, train_loss[loss=3.311, NarTop10Accuracy=0.6664, over 6259.00 frames. ], tot_loss[loss=3.281, NarTop10Accuracy=0.6626, over 5433.20 frames. ], batch size: 11, lr: 2.13e-03
2024-08-06 15:45:25,431 INFO [trainer.py:765] (0/8) Epoch 40, batch 600, train_loss[loss=3.408, NarTop10Accuracy=0.6385, over 5860.00 frames. ], tot_loss[loss=3.299, NarTop10Accuracy=0.6593, over 5692.55 frames. ], batch size: 9, lr: 2.13e-03
2024-08-06 15:45:58,647 INFO [trainer.py:765] (0/8) Epoch 40, batch 700, train_loss[loss=3.383, NarTop10Accuracy=0.6428, over 4959.00 frames. ], tot_loss[loss=3.313, NarTop10Accuracy=0.6561, over 5750.74 frames. ], batch size: 6, lr: 2.13e-03
2024-08-06 15:46:34,887 INFO [trainer.py:765] (0/8) Epoch 40, batch 800, train_loss[loss=3.302, NarTop10Accuracy=0.6584, over 5079.00 frames. ], tot_loss[loss=3.308, NarTop10Accuracy=0.6569, over 5816.45 frames. ], batch size: 6, lr: 2.13e-03
2024-08-06 15:47:07,290 INFO [trainer.py:765] (0/8) Epoch 40, batch 900, train_loss[loss=3.195, NarTop10Accuracy=0.6842, over 6580.00 frames. ], tot_loss[loss=3.306, NarTop10Accuracy=0.6569, over 5829.43 frames. ], batch size: 14, lr: 2.12e-03
2024-08-06 15:47:43,510 INFO [trainer.py:765] (0/8) Epoch 40, batch 1000, train_loss[loss=3.43, NarTop10Accuracy=0.6256, over 6640.00 frames. ], tot_loss[loss=3.313, NarTop10Accuracy=0.6554, over 5917.07 frames. ], batch size: 14, lr: 2.12e-03
2024-08-06 15:48:18,709 INFO [trainer.py:765] (0/8) Epoch 40, batch 1100, train_loss[loss=3.52, NarTop10Accuracy=0.6255, over 6985.00 frames. ], tot_loss[loss=3.318, NarTop10Accuracy=0.6543, over 5954.98 frames. ], batch size: 17, lr: 2.12e-03
2024-08-06 15:48:52,094 INFO [trainer.py:765] (0/8) Epoch 40, batch 1200, train_loss[loss=3.196, NarTop10Accuracy=0.6816, over 7014.00 frames. ], tot_loss[loss=3.319, NarTop10Accuracy=0.6546, over 5947.81 frames. ], batch size: 30, lr: 2.12e-03
2024-08-06 15:49:29,782 INFO [trainer.py:765] (0/8) Epoch 40, batch 1300, train_loss[loss=3.593, NarTop10Accuracy=0.6138, over 5014.00 frames. ], tot_loss[loss=3.322, NarTop10Accuracy=0.6543, over 6026.96 frames. ], batch size: 6, lr: 2.12e-03
2024-08-06 15:49:38,244 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-102000.pt
2024-08-06 15:49:41,956 INFO [trainer.py:803] (0/8) Computing validation loss
2024-08-06 15:49:48,934 INFO [trainer.py:811] (0/8) Epoch 40, validation: loss=3.171, NarTop10Accuracy=0.6871, over 1907754.00 frames. 
2024-08-06 15:49:48,935 INFO [trainer.py:814] (0/8) Maximum memory allocated so far is 30689MB
2024-08-06 15:49:49,615 INFO [optim.py:386] (0/8) Clipping_scale=2.0, grad-norm quartiles 1.708e+02 2.095e+02 2.264e+02 2.441e+02 4.960e+02, threshold=4.528e+02, percent-clipped=0.1
2024-08-06 15:50:12,460 INFO [trainer.py:765] (0/8) Epoch 40, batch 1400, train_loss[loss=3.37, NarTop10Accuracy=0.6521, over 6206.00 frames. ], tot_loss[loss=3.323, NarTop10Accuracy=0.654, over 6019.80 frames. ], batch size: 11, lr: 2.12e-03
2024-08-06 15:50:45,930 INFO [trainer.py:765] (0/8) Epoch 40, batch 1500, train_loss[loss=3.537, NarTop10Accuracy=0.6141, over 5496.00 frames. ], tot_loss[loss=3.329, NarTop10Accuracy=0.6527, over 5956.73 frames. ], batch size: 49, lr: 2.12e-03
2024-08-06 15:51:13,820 INFO [trainer.py:765] (0/8) Epoch 40, batch 1600, train_loss[loss=3.169, NarTop10Accuracy=0.6865, over 7447.00 frames. ], tot_loss[loss=3.321, NarTop10Accuracy=0.6542, over 5946.75 frames. ], batch size: 22, lr: 2.12e-03
2024-08-06 15:51:40,570 INFO [trainer.py:765] (0/8) Epoch 40, batch 1700, train_loss[loss=3.442, NarTop10Accuracy=0.6421, over 6677.00 frames. ], tot_loss[loss=3.335, NarTop10Accuracy=0.6514, over 5938.77 frames. ], batch size: 14, lr: 2.12e-03
2024-08-06 15:52:07,236 INFO [trainer.py:765] (0/8) Epoch 40, batch 1800, train_loss[loss=3.418, NarTop10Accuracy=0.6345, over 7182.00 frames. ], tot_loss[loss=3.338, NarTop10Accuracy=0.6508, over 6006.32 frames. ], batch size: 22, lr: 2.12e-03
2024-08-06 15:52:33,820 INFO [trainer.py:765] (0/8) Epoch 40, batch 1900, train_loss[loss=3.399, NarTop10Accuracy=0.6367, over 5951.00 frames. ], tot_loss[loss=3.349, NarTop10Accuracy=0.6492, over 6044.05 frames. ], batch size: 49, lr: 2.11e-03
2024-08-06 15:52:59,511 INFO [trainer.py:765] (0/8) Epoch 40, batch 2000, train_loss[loss=3.382, NarTop10Accuracy=0.6457, over 5962.00 frames. ], tot_loss[loss=3.351, NarTop10Accuracy=0.6484, over 6012.31 frames. ], batch size: 49, lr: 2.11e-03
2024-08-06 15:53:24,913 INFO [trainer.py:765] (0/8) Epoch 40, batch 2100, train_loss[loss=3.121, NarTop10Accuracy=0.6986, over 4917.00 frames. ], tot_loss[loss=3.347, NarTop10Accuracy=0.6489, over 5993.42 frames. ], batch size: 5, lr: 2.11e-03
2024-08-06 15:53:50,418 INFO [trainer.py:765] (0/8) Epoch 40, batch 2200, train_loss[loss=3.38, NarTop10Accuracy=0.647, over 7279.00 frames. ], tot_loss[loss=3.344, NarTop10Accuracy=0.6498, over 6034.94 frames. ], batch size: 30, lr: 2.11e-03
2024-08-06 15:54:15,886 INFO [trainer.py:765] (0/8) Epoch 40, batch 2300, train_loss[loss=3.344, NarTop10Accuracy=0.6619, over 5761.00 frames. ], tot_loss[loss=3.34, NarTop10Accuracy=0.6505, over 6069.80 frames. ], batch size: 9, lr: 2.11e-03
2024-08-06 15:54:23,200 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/checkpoint-103000.pt
2024-08-06 15:54:43,787 INFO [trainer.py:765] (0/8) Epoch 40, batch 2400, train_loss[loss=3.786, NarTop10Accuracy=0.5617, over 5833.00 frames. ], tot_loss[loss=3.347, NarTop10Accuracy=0.6492, over 5907.90 frames. ], batch size: 51, lr: 2.11e-03
2024-08-06 15:55:07,364 INFO [trainer.py:765] (0/8) Epoch 40, batch 2500, train_loss[loss=3.035, NarTop10Accuracy=0.7183, over 5102.00 frames. ], tot_loss[loss=3.311, NarTop10Accuracy=0.656, over 5547.59 frames. ], batch size: 6, lr: 2.11e-03
2024-08-06 15:55:28,017 INFO [trainer.py:650] (0/8) Reaches end of dataloader.
2024-08-06 15:55:28,019 INFO [checkpoint.py:75] (0/8) Saving checkpoint to exp/valle/epoch-40.pt
2024-08-06 15:55:31,454 INFO [trainer.py:1069] (0/8) Done!