| 2024-08-06 08:06:14,312 INFO [trainer.py:870] (5/8) Training started | |
| 2024-08-06 08:06:14,313 INFO [trainer.py:889] (5/8) Device: cuda:5 | |
| 2024-08-06 08:06:14,314 INFO [trainer.py:890] (5/8) {'best_train_loss': inf, 'best_valid_loss': inf, 'best_train_epoch': -1, 'best_valid_epoch': -1, 'batch_idx_train': 0, 'log_interval': 100, 'reset_interval': 200, 'valid_interval': 2000, 'env_info': {'k2-version': '1.24.3', 'k2-build-type': 'Release', 'k2-with-cuda': True, 'k2-git-sha1': '279b0c87015a615b81b147251814d737a548f397', 'k2-git-date': 'Wed May 24 22:24:09 2023', 'lhotse-version': '1.26.0', 'torch-version': '2.0.1+cu118', 'torch-cuda-available': True, 'torch-cuda-version': '11.8', 'python-version': '3.10', 'icefall-git-branch': None, 'icefall-git-sha1': None, 'icefall-git-date': None, 'icefall-path': '/workspace/icefall_llm', 'k2-path': '/usr/local/lib/python3.10/dist-packages/k2/__init__.py', 'lhotse-path': '/usr/local/lib/python3.10/dist-packages/lhotse/__init__.py', 'hostname': '6867463', 'IP address': '0.104.202.7'}, 'world_size': 8, 'master_port': 12354, 'tensorboard': True, 'num_epochs': 20, 'start_epoch': 1, 'start_batch': 0, 'exp_dir': PosixPath('exp/valle'), 'optimizer_name': 'ScaledAdam', 'scheduler_name': 'Eden', 'base_lr': 0.03, 'warmup_steps': 200, 'seed': 42, 'inf_check': False, 'save_every_n': 20000, 'keep_last_k': 20, 'average_period': 0, 'accumulate_grad_steps': 1, 'dtype': 'bfloat16', 'filter_min_duration': 0.5, 'filter_max_duration': 14.0, 'train_stage': 1, 'visualize': False, 'oom_check': False, 'model_name': 'valle', 'decoder_dim': 1024, 'nhead': 16, 'num_decoder_layers': 12, 'scale_factor': 1.0, 'norm_first': True, 'add_prenet': False, 'prefix_mode': 1, 'share_embedding': True, 'prepend_bos': False, 'num_quantizers': 8, 'scaling_xformers': False, 'manifest_dir': PosixPath('data/tokenized'), 'max_duration': 320, 'bucketing_sampler': True, 'num_buckets': 6, 'concatenate_cuts': False, 'duration_factor': 1.0, 'gap': 0.1, 'on_the_fly_feats': False, 'shuffle': True, 'buffer_size': 40000, 'shuffle_buffer_size': 100000, 'drop_last': False, 'return_cuts': True, 'num_workers': 8, 'enable_spec_aug': False, 'spec_aug_time_warp_factor': 80, 'input_strategy': 'PrecomputedFeatures', 'dataset': 'libritts', 'text_tokens': 'data/tokenized/unique_text_tokens.k2symbols', 'sampling_rate': 24000} | |
| 2024-08-06 08:06:14,314 INFO [trainer.py:892] (5/8) About to create model | |
| 2024-08-06 08:06:15,008 INFO [trainer.py:899] (5/8) Number of model parameters: 367386628 | |
| 2024-08-06 08:06:16,222 INFO [trainer.py:914] (5/8) Using DDP | |
| 2024-08-06 08:06:19,151 INFO [datamodule.py:427] (5/8) About to get train cuts | |
| 2024-08-06 08:06:19,153 INFO [datamodule.py:434] (5/8) About to get dev cuts | |
| 2024-08-06 08:06:19,155 INFO [datamodule.py:292] (5/8) Disable SpecAugment | |
| 2024-08-06 08:06:19,155 INFO [datamodule.py:294] (5/8) About to create train dataset | |
| 2024-08-06 08:06:19,155 INFO [datamodule.py:323] (5/8) Using DynamicBucketingSampler | |
| 2024-08-06 08:06:19,766 INFO [datamodule.py:344] (5/8) About to create train dataloader | |
| 2024-08-06 08:06:19,766 INFO [datamodule.py:367] (5/8) About to create dev dataset | |
| 2024-08-06 08:06:20,091 INFO [datamodule.py:388] (5/8) About to create dev dataloader | |
| 2024-08-06 08:08:02,120 INFO [trainer.py:765] (5/8) Epoch 1, batch 100, train_loss[loss=4.267, ArTop10Accuracy=0.5104, over 13962.00 frames. ], tot_loss[loss=5.049, ArTop10Accuracy=0.3742, over 4764.60 frames. ], batch size: 62, lr: 2.25e-02 | |
| 2024-08-06 08:09:28,828 INFO [trainer.py:765] (5/8) Epoch 1, batch 200, train_loss[loss=4.009, ArTop10Accuracy=0.5501, over 13701.00 frames. ], tot_loss[loss=4.489, ArTop10Accuracy=0.4683, over 7752.76 frames. ], batch size: 34, lr: 3.00e-02 | |
| 2024-08-06 08:10:52,429 INFO [trainer.py:765] (5/8) Epoch 1, batch 300, train_loss[loss=3.902, ArTop10Accuracy=0.5643, over 14151.00 frames. ], tot_loss[loss=4.21, ArTop10Accuracy=0.5149, over 9369.31 frames. ], batch size: 44, lr: 3.00e-02 | |
| 2024-08-06 08:12:12,699 INFO [trainer.py:765] (5/8) Epoch 1, batch 400, train_loss[loss=3.705, ArTop10Accuracy=0.605, over 10998.00 frames. ], tot_loss[loss=4.023, ArTop10Accuracy=0.5465, over 10273.59 frames. ], batch size: 15, lr: 3.00e-02 | |
| 2024-08-06 08:13:40,050 INFO [trainer.py:765] (5/8) Epoch 1, batch 500, train_loss[loss=3.613, ArTop10Accuracy=0.6219, over 12171.00 frames. ], tot_loss[loss=3.878, ArTop10Accuracy=0.5715, over 10848.61 frames. ], batch size: 22, lr: 2.99e-02 | |
| 2024-08-06 08:15:00,243 INFO [trainer.py:765] (5/8) Epoch 1, batch 600, train_loss[loss=3.56, ArTop10Accuracy=0.6298, over 11346.00 frames. ], tot_loss[loss=3.765, ArTop10Accuracy=0.5916, over 11350.53 frames. ], batch size: 18, lr: 2.99e-02 | |
| 2024-08-06 08:16:26,424 INFO [trainer.py:765] (5/8) Epoch 1, batch 700, train_loss[loss=3.414, ArTop10Accuracy=0.6566, over 10089.00 frames. ], tot_loss[loss=3.691, ArTop10Accuracy=0.6047, over 11494.81 frames. ], batch size: 12, lr: 2.99e-02 | |
| 2024-08-06 08:17:43,017 INFO [trainer.py:765] (5/8) Epoch 1, batch 800, train_loss[loss=3.544, ArTop10Accuracy=0.6297, over 9378.00 frames. ], tot_loss[loss=3.625, ArTop10Accuracy=0.6168, over 11636.05 frames. ], batch size: 11, lr: 2.98e-02 | |
| 2024-08-06 08:18:56,151 INFO [trainer.py:765] (5/8) Epoch 1, batch 900, train_loss[loss=3.446, ArTop10Accuracy=0.649, over 12843.00 frames. ], tot_loss[loss=3.57, ArTop10Accuracy=0.6266, over 11681.46 frames. ], batch size: 27, lr: 2.98e-02 | |
| 2024-08-06 08:20:12,862 INFO [trainer.py:765] (5/8) Epoch 1, batch 1000, train_loss[loss=3.474, ArTop10Accuracy=0.6463, over 12768.00 frames. ], tot_loss[loss=3.531, ArTop10Accuracy=0.6336, over 11878.58 frames. ], batch size: 27, lr: 2.97e-02 | |
| 2024-08-06 08:20:13,539 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 9.300e+01 1.871e+02 2.675e+02 4.030e+02 9.119e+03, threshold=5.351e+02, percent-clipped=0.0 | |
| 2024-08-06 08:21:29,155 INFO [trainer.py:765] (5/8) Epoch 1, batch 1100, train_loss[loss=3.411, ArTop10Accuracy=0.6511, over 13878.00 frames. ], tot_loss[loss=3.494, ArTop10Accuracy=0.6401, over 11961.72 frames. ], batch size: 35, lr: 2.96e-02 | |
| 2024-08-06 08:22:45,411 INFO [trainer.py:765] (5/8) Epoch 1, batch 1200, train_loss[loss=3.503, ArTop10Accuracy=0.6388, over 11898.00 frames. ], tot_loss[loss=3.465, ArTop10Accuracy=0.6456, over 11873.58 frames. ], batch size: 101, lr: 2.96e-02 | |
| 2024-08-06 08:23:45,288 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 08:25:36,237 INFO [trainer.py:765] (5/8) Epoch 2, batch 100, train_loss[loss=3.392, ArTop10Accuracy=0.655, over 14622.00 frames. ], tot_loss[loss=3.424, ArTop10Accuracy=0.652, over 4773.72 frames. ], batch size: 62, lr: 2.90e-02 | |
| 2024-08-06 08:26:58,955 INFO [trainer.py:765] (5/8) Epoch 2, batch 200, train_loss[loss=3.378, ArTop10Accuracy=0.6575, over 13425.00 frames. ], tot_loss[loss=3.386, ArTop10Accuracy=0.6597, over 7764.85 frames. ], batch size: 34, lr: 2.89e-02 | |
| 2024-08-06 08:28:25,534 INFO [trainer.py:765] (5/8) Epoch 2, batch 300, train_loss[loss=3.37, ArTop10Accuracy=0.6646, over 14022.00 frames. ], tot_loss[loss=3.371, ArTop10Accuracy=0.6625, over 9389.25 frames. ], batch size: 44, lr: 2.89e-02 | |
| 2024-08-06 08:29:48,637 INFO [trainer.py:765] (5/8) Epoch 2, batch 400, train_loss[loss=3.421, ArTop10Accuracy=0.6513, over 10383.00 frames. ], tot_loss[loss=3.358, ArTop10Accuracy=0.6654, over 10290.14 frames. ], batch size: 14, lr: 2.88e-02 | |
| 2024-08-06 08:31:22,902 INFO [trainer.py:765] (5/8) Epoch 2, batch 500, train_loss[loss=3.412, ArTop10Accuracy=0.6543, over 12753.00 frames. ], tot_loss[loss=3.343, ArTop10Accuracy=0.6681, over 10849.81 frames. ], batch size: 23, lr: 2.87e-02 | |
| 2024-08-06 08:32:45,687 INFO [trainer.py:765] (5/8) Epoch 2, batch 600, train_loss[loss=3.332, ArTop10Accuracy=0.67, over 11454.00 frames. ], tot_loss[loss=3.331, ArTop10Accuracy=0.6706, over 11349.06 frames. ], batch size: 18, lr: 2.86e-02 | |
| 2024-08-06 08:34:13,583 INFO [trainer.py:765] (5/8) Epoch 2, batch 700, train_loss[loss=3.326, ArTop10Accuracy=0.6678, over 10104.00 frames. ], tot_loss[loss=3.326, ArTop10Accuracy=0.6716, over 11504.73 frames. ], batch size: 12, lr: 2.85e-02 | |
| 2024-08-06 08:34:31,175 INFO [trainer.py:803] (5/8) Computing validation loss | |
| 2024-08-06 08:34:40,888 INFO [trainer.py:811] (5/8) Epoch 2, validation: loss=3.277, ArTop10Accuracy=0.6803, over 1827537.00 frames. | |
| 2024-08-06 08:34:40,889 INFO [trainer.py:814] (5/8) Maximum memory allocated so far is 28868MB | |
| 2024-08-06 08:34:41,700 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 7.953e+01 1.592e+02 2.200e+02 3.344e+02 2.949e+03, threshold=4.400e+02, percent-clipped=8.6 | |
| 2024-08-06 08:35:39,877 INFO [trainer.py:765] (5/8) Epoch 2, batch 800, train_loss[loss=3.238, ArTop10Accuracy=0.6932, over 9108.00 frames. ], tot_loss[loss=3.319, ArTop10Accuracy=0.6731, over 11622.67 frames. ], batch size: 11, lr: 2.84e-02 | |
| 2024-08-06 08:36:56,372 INFO [trainer.py:765] (5/8) Epoch 2, batch 900, train_loss[loss=3.164, ArTop10Accuracy=0.6972, over 12777.00 frames. ], tot_loss[loss=3.305, ArTop10Accuracy=0.6756, over 11673.76 frames. ], batch size: 27, lr: 2.83e-02 | |
| 2024-08-06 08:38:10,511 INFO [trainer.py:765] (5/8) Epoch 2, batch 1000, train_loss[loss=3.184, ArTop10Accuracy=0.6983, over 13158.00 frames. ], tot_loss[loss=3.299, ArTop10Accuracy=0.6768, over 11869.74 frames. ], batch size: 27, lr: 2.82e-02 | |
| 2024-08-06 08:39:25,059 INFO [trainer.py:765] (5/8) Epoch 2, batch 1100, train_loss[loss=3.256, ArTop10Accuracy=0.6888, over 13548.00 frames. ], tot_loss[loss=3.292, ArTop10Accuracy=0.6782, over 11932.93 frames. ], batch size: 34, lr: 2.81e-02 | |
| 2024-08-06 08:40:38,220 INFO [trainer.py:765] (5/8) Epoch 2, batch 1200, train_loss[loss=3.328, ArTop10Accuracy=0.6618, over 12423.00 frames. ], tot_loss[loss=3.283, ArTop10Accuracy=0.6796, over 11869.40 frames. ], batch size: 101, lr: 2.80e-02 | |
| 2024-08-06 08:41:38,289 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 08:43:36,649 INFO [trainer.py:765] (5/8) Epoch 3, batch 100, train_loss[loss=3.275, ArTop10Accuracy=0.6816, over 14241.00 frames. ], tot_loss[loss=3.244, ArTop10Accuracy=0.6861, over 4767.69 frames. ], batch size: 62, lr: 2.67e-02 | |
| 2024-08-06 08:45:10,500 INFO [trainer.py:765] (5/8) Epoch 3, batch 200, train_loss[loss=3.273, ArTop10Accuracy=0.6772, over 13731.00 frames. ], tot_loss[loss=3.223, ArTop10Accuracy=0.6902, over 7752.86 frames. ], batch size: 34, lr: 2.66e-02 | |
| 2024-08-06 08:46:29,257 INFO [trainer.py:765] (5/8) Epoch 3, batch 300, train_loss[loss=3.183, ArTop10Accuracy=0.6995, over 14106.00 frames. ], tot_loss[loss=3.205, ArTop10Accuracy=0.6939, over 9389.07 frames. ], batch size: 44, lr: 2.64e-02 | |
| 2024-08-06 08:48:04,218 INFO [trainer.py:765] (5/8) Epoch 3, batch 400, train_loss[loss=3.116, ArTop10Accuracy=0.7163, over 10269.00 frames. ], tot_loss[loss=3.19, ArTop10Accuracy=0.6969, over 10295.87 frames. ], batch size: 14, lr: 2.63e-02 | |
| 2024-08-06 08:48:40,881 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 9.282e+01 1.561e+02 1.981e+02 2.686e+02 1.768e+03, threshold=3.962e+02, percent-clipped=7.6 | |
| 2024-08-06 08:49:25,541 INFO [trainer.py:765] (5/8) Epoch 3, batch 500, train_loss[loss=3.086, ArTop10Accuracy=0.7171, over 12081.00 frames. ], tot_loss[loss=3.174, ArTop10Accuracy=0.7001, over 10836.69 frames. ], batch size: 22, lr: 2.62e-02 | |
| 2024-08-06 08:51:00,476 INFO [trainer.py:765] (5/8) Epoch 3, batch 600, train_loss[loss=3.137, ArTop10Accuracy=0.7034, over 11385.00 frames. ], tot_loss[loss=3.153, ArTop10Accuracy=0.7042, over 11343.41 frames. ], batch size: 18, lr: 2.61e-02 | |
| 2024-08-06 08:52:31,617 INFO [trainer.py:765] (5/8) Epoch 3, batch 700, train_loss[loss=3.084, ArTop10Accuracy=0.7191, over 9357.00 frames. ], tot_loss[loss=3.145, ArTop10Accuracy=0.7056, over 11508.14 frames. ], batch size: 11, lr: 2.60e-02 | |
| 2024-08-06 08:53:57,388 INFO [trainer.py:765] (5/8) Epoch 3, batch 800, train_loss[loss=3.123, ArTop10Accuracy=0.7103, over 9462.00 frames. ], tot_loss[loss=3.138, ArTop10Accuracy=0.7073, over 11610.68 frames. ], batch size: 11, lr: 2.59e-02 | |
| 2024-08-06 08:55:15,117 INFO [trainer.py:765] (5/8) Epoch 3, batch 900, train_loss[loss=2.989, ArTop10Accuracy=0.7359, over 12843.00 frames. ], tot_loss[loss=3.12, ArTop10Accuracy=0.7107, over 11648.07 frames. ], batch size: 27, lr: 2.57e-02 | |
| 2024-08-06 08:56:31,557 INFO [trainer.py:765] (5/8) Epoch 3, batch 1000, train_loss[loss=3.13, ArTop10Accuracy=0.7095, over 13044.00 frames. ], tot_loss[loss=3.112, ArTop10Accuracy=0.712, over 11857.18 frames. ], batch size: 27, lr: 2.56e-02 | |
| 2024-08-06 08:57:46,505 INFO [trainer.py:765] (5/8) Epoch 3, batch 1100, train_loss[loss=3.133, ArTop10Accuracy=0.7092, over 13536.00 frames. ], tot_loss[loss=3.108, ArTop10Accuracy=0.7126, over 11942.38 frames. ], batch size: 34, lr: 2.55e-02 | |
| 2024-08-06 08:59:01,399 INFO [trainer.py:765] (5/8) Epoch 3, batch 1200, train_loss[loss=3.122, ArTop10Accuracy=0.7062, over 11571.00 frames. ], tot_loss[loss=3.098, ArTop10Accuracy=0.7146, over 11849.80 frames. ], batch size: 101, lr: 2.54e-02 | |
| 2024-08-06 09:00:01,730 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 09:01:50,741 INFO [trainer.py:765] (5/8) Epoch 4, batch 100, train_loss[loss=3.116, ArTop10Accuracy=0.7056, over 14949.00 frames. ], tot_loss[loss=3.071, ArTop10Accuracy=0.7187, over 4748.08 frames. ], batch size: 62, lr: 2.38e-02 | |
| 2024-08-06 09:02:52,858 INFO [trainer.py:803] (5/8) Computing validation loss | |
| 2024-08-06 09:03:02,384 INFO [trainer.py:811] (5/8) Epoch 4, validation: loss=2.997, ArTop10Accuracy=0.7338, over 1827537.00 frames. | |
| 2024-08-06 09:03:02,385 INFO [trainer.py:814] (5/8) Maximum memory allocated so far is 29481MB | |
| 2024-08-06 09:03:03,364 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.072e+02 1.499e+02 1.782e+02 2.273e+02 1.100e+03, threshold=3.565e+02, percent-clipped=4.7 | |
| 2024-08-06 09:03:29,273 INFO [trainer.py:765] (5/8) Epoch 4, batch 200, train_loss[loss=2.949, ArTop10Accuracy=0.744, over 13677.00 frames. ], tot_loss[loss=3.048, ArTop10Accuracy=0.7237, over 7770.03 frames. ], batch size: 34, lr: 2.37e-02 | |
| 2024-08-06 09:05:01,732 INFO [trainer.py:765] (5/8) Epoch 4, batch 300, train_loss[loss=3.044, ArTop10Accuracy=0.7282, over 14460.00 frames. ], tot_loss[loss=3.036, ArTop10Accuracy=0.7261, over 9385.26 frames. ], batch size: 45, lr: 2.36e-02 | |
| 2024-08-06 09:06:28,150 INFO [trainer.py:765] (5/8) Epoch 4, batch 400, train_loss[loss=2.864, ArTop10Accuracy=0.7614, over 10161.00 frames. ], tot_loss[loss=3.032, ArTop10Accuracy=0.7271, over 10287.27 frames. ], batch size: 14, lr: 2.34e-02 | |
| 2024-08-06 09:08:01,924 INFO [trainer.py:765] (5/8) Epoch 4, batch 500, train_loss[loss=2.968, ArTop10Accuracy=0.7405, over 12393.00 frames. ], tot_loss[loss=3.022, ArTop10Accuracy=0.729, over 10846.36 frames. ], batch size: 22, lr: 2.33e-02 | |
| 2024-08-06 09:09:28,540 INFO [trainer.py:765] (5/8) Epoch 4, batch 600, train_loss[loss=3.045, ArTop10Accuracy=0.7254, over 11475.00 frames. ], tot_loss[loss=3.019, ArTop10Accuracy=0.7295, over 11367.69 frames. ], batch size: 18, lr: 2.32e-02 | |
| 2024-08-06 09:10:59,865 INFO [trainer.py:765] (5/8) Epoch 4, batch 700, train_loss[loss=2.952, ArTop10Accuracy=0.7418, over 10227.00 frames. ], tot_loss[loss=3.023, ArTop10Accuracy=0.7287, over 11527.79 frames. ], batch size: 12, lr: 2.31e-02 | |
| 2024-08-06 09:12:17,512 INFO [trainer.py:765] (5/8) Epoch 4, batch 800, train_loss[loss=3.026, ArTop10Accuracy=0.7246, over 9501.00 frames. ], tot_loss[loss=3.023, ArTop10Accuracy=0.7287, over 11616.75 frames. ], batch size: 11, lr: 2.30e-02 | |
| 2024-08-06 09:13:33,212 INFO [trainer.py:765] (5/8) Epoch 4, batch 900, train_loss[loss=2.961, ArTop10Accuracy=0.7415, over 12981.00 frames. ], tot_loss[loss=3.014, ArTop10Accuracy=0.7305, over 11675.99 frames. ], batch size: 27, lr: 2.29e-02 | |
| 2024-08-06 09:14:47,519 INFO [trainer.py:765] (5/8) Epoch 4, batch 1000, train_loss[loss=3.037, ArTop10Accuracy=0.7291, over 12837.00 frames. ], tot_loss[loss=3.011, ArTop10Accuracy=0.731, over 11879.91 frames. ], batch size: 27, lr: 2.28e-02 | |
| 2024-08-06 09:16:02,981 INFO [trainer.py:765] (5/8) Epoch 4, batch 1100, train_loss[loss=3.094, ArTop10Accuracy=0.7178, over 13704.00 frames. ], tot_loss[loss=3.013, ArTop10Accuracy=0.7306, over 11943.31 frames. ], batch size: 34, lr: 2.26e-02 | |
| 2024-08-06 09:16:53,291 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.100e+02 1.440e+02 1.636e+02 1.968e+02 7.702e+02, threshold=3.273e+02, percent-clipped=1.3 | |
| 2024-08-06 09:17:18,344 INFO [trainer.py:765] (5/8) Epoch 4, batch 1200, train_loss[loss=3.051, ArTop10Accuracy=0.7233, over 12387.00 frames. ], tot_loss[loss=3.011, ArTop10Accuracy=0.7312, over 11857.01 frames. ], batch size: 101, lr: 2.25e-02 | |
| 2024-08-06 09:18:17,719 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 09:20:17,170 INFO [trainer.py:765] (5/8) Epoch 5, batch 100, train_loss[loss=3.023, ArTop10Accuracy=0.731, over 14667.00 frames. ], tot_loss[loss=2.989, ArTop10Accuracy=0.7344, over 4786.09 frames. ], batch size: 62, lr: 2.10e-02 | |
| 2024-08-06 09:21:52,295 INFO [trainer.py:765] (5/8) Epoch 5, batch 200, train_loss[loss=2.991, ArTop10Accuracy=0.7299, over 13593.00 frames. ], tot_loss[loss=2.981, ArTop10Accuracy=0.736, over 7754.44 frames. ], batch size: 34, lr: 2.09e-02 | |
| 2024-08-06 09:23:19,240 INFO [trainer.py:765] (5/8) Epoch 5, batch 300, train_loss[loss=2.998, ArTop10Accuracy=0.7285, over 14391.00 frames. ], tot_loss[loss=2.972, ArTop10Accuracy=0.7377, over 9395.87 frames. ], batch size: 44, lr: 2.08e-02 | |
| 2024-08-06 09:24:53,536 INFO [trainer.py:765] (5/8) Epoch 5, batch 400, train_loss[loss=2.832, ArTop10Accuracy=0.7682, over 10143.00 frames. ], tot_loss[loss=2.969, ArTop10Accuracy=0.7386, over 10302.88 frames. ], batch size: 14, lr: 2.07e-02 | |
| 2024-08-06 09:26:19,417 INFO [trainer.py:765] (5/8) Epoch 5, batch 500, train_loss[loss=3.012, ArTop10Accuracy=0.7274, over 12156.00 frames. ], tot_loss[loss=2.965, ArTop10Accuracy=0.7393, over 10871.70 frames. ], batch size: 22, lr: 2.06e-02 | |
| 2024-08-06 09:27:49,537 INFO [trainer.py:765] (5/8) Epoch 5, batch 600, train_loss[loss=2.916, ArTop10Accuracy=0.7567, over 11493.00 frames. ], tot_loss[loss=2.962, ArTop10Accuracy=0.7399, over 11392.96 frames. ], batch size: 18, lr: 2.05e-02 | |
| 2024-08-06 09:29:21,669 INFO [trainer.py:765] (5/8) Epoch 5, batch 700, train_loss[loss=2.98, ArTop10Accuracy=0.7363, over 10116.00 frames. ], tot_loss[loss=2.966, ArTop10Accuracy=0.7392, over 11554.05 frames. ], batch size: 12, lr: 2.04e-02 | |
| 2024-08-06 09:30:44,692 INFO [trainer.py:765] (5/8) Epoch 5, batch 800, train_loss[loss=3.029, ArTop10Accuracy=0.7166, over 10044.00 frames. ], tot_loss[loss=2.967, ArTop10Accuracy=0.7391, over 11668.31 frames. ], batch size: 12, lr: 2.03e-02 | |
| 2024-08-06 09:31:51,238 INFO [trainer.py:803] (5/8) Computing validation loss | |
| 2024-08-06 09:32:00,762 INFO [trainer.py:811] (5/8) Epoch 5, validation: loss=2.926, ArTop10Accuracy=0.7466, over 1827537.00 frames. | |
| 2024-08-06 09:32:00,763 INFO [trainer.py:814] (5/8) Maximum memory allocated so far is 29481MB | |
| 2024-08-06 09:32:01,708 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.060e+02 1.349e+02 1.525e+02 1.806e+02 1.007e+03, threshold=3.049e+02, percent-clipped=2.3 | |
| 2024-08-06 09:32:10,553 INFO [trainer.py:765] (5/8) Epoch 5, batch 900, train_loss[loss=2.925, ArTop10Accuracy=0.7434, over 12921.00 frames. ], tot_loss[loss=2.959, ArTop10Accuracy=0.7408, over 11719.21 frames. ], batch size: 27, lr: 2.02e-02 | |
| 2024-08-06 09:33:27,323 INFO [trainer.py:765] (5/8) Epoch 5, batch 1000, train_loss[loss=2.894, ArTop10Accuracy=0.7564, over 12846.00 frames. ], tot_loss[loss=2.96, ArTop10Accuracy=0.7406, over 11894.84 frames. ], batch size: 27, lr: 2.01e-02 | |
| 2024-08-06 09:34:42,300 INFO [trainer.py:765] (5/8) Epoch 5, batch 1100, train_loss[loss=2.979, ArTop10Accuracy=0.7327, over 13755.00 frames. ], tot_loss[loss=2.966, ArTop10Accuracy=0.7394, over 11957.99 frames. ], batch size: 34, lr: 2.00e-02 | |
| 2024-08-06 09:35:56,331 INFO [trainer.py:765] (5/8) Epoch 5, batch 1200, train_loss[loss=3.056, ArTop10Accuracy=0.7168, over 12480.00 frames. ], tot_loss[loss=2.962, ArTop10Accuracy=0.7402, over 11875.87 frames. ], batch size: 101, lr: 1.99e-02 | |
| 2024-08-06 09:36:55,627 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 09:38:52,664 INFO [trainer.py:765] (5/8) Epoch 6, batch 100, train_loss[loss=3.02, ArTop10Accuracy=0.7256, over 14367.00 frames. ], tot_loss[loss=2.952, ArTop10Accuracy=0.7414, over 4758.76 frames. ], batch size: 62, lr: 1.85e-02 | |
| 2024-08-06 09:40:19,833 INFO [trainer.py:765] (5/8) Epoch 6, batch 200, train_loss[loss=2.915, ArTop10Accuracy=0.7538, over 13674.00 frames. ], tot_loss[loss=2.94, ArTop10Accuracy=0.7438, over 7757.35 frames. ], batch size: 34, lr: 1.84e-02 | |
| 2024-08-06 09:41:52,964 INFO [trainer.py:765] (5/8) Epoch 6, batch 300, train_loss[loss=2.919, ArTop10Accuracy=0.7497, over 14718.00 frames. ], tot_loss[loss=2.932, ArTop10Accuracy=0.7457, over 9372.92 frames. ], batch size: 45, lr: 1.83e-02 | |
| 2024-08-06 09:43:17,827 INFO [trainer.py:765] (5/8) Epoch 6, batch 400, train_loss[loss=2.957, ArTop10Accuracy=0.7436, over 10209.00 frames. ], tot_loss[loss=2.93, ArTop10Accuracy=0.7461, over 10291.45 frames. ], batch size: 14, lr: 1.83e-02 | |
| 2024-08-06 09:44:54,128 INFO [trainer.py:765] (5/8) Epoch 6, batch 500, train_loss[loss=2.95, ArTop10Accuracy=0.7403, over 12219.00 frames. ], tot_loss[loss=2.921, ArTop10Accuracy=0.748, over 10847.88 frames. ], batch size: 22, lr: 1.82e-02 | |
| 2024-08-06 09:46:22,872 INFO [trainer.py:765] (5/8) Epoch 6, batch 600, train_loss[loss=2.853, ArTop10Accuracy=0.7617, over 11301.00 frames. ], tot_loss[loss=2.918, ArTop10Accuracy=0.7483, over 11379.44 frames. ], batch size: 18, lr: 1.81e-02 | |
| 2024-08-06 09:46:37,219 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.012e+02 1.339e+02 1.480e+02 1.701e+02 7.506e+02, threshold=2.959e+02, percent-clipped=1.1 | |
| 2024-08-06 09:47:57,870 INFO [trainer.py:765] (5/8) Epoch 6, batch 700, train_loss[loss=2.774, ArTop10Accuracy=0.7798, over 10119.00 frames. ], tot_loss[loss=2.924, ArTop10Accuracy=0.7473, over 11527.84 frames. ], batch size: 12, lr: 1.80e-02 | |
| 2024-08-06 09:49:15,954 INFO [trainer.py:765] (5/8) Epoch 6, batch 800, train_loss[loss=2.951, ArTop10Accuracy=0.7472, over 10164.00 frames. ], tot_loss[loss=2.927, ArTop10Accuracy=0.7466, over 11640.78 frames. ], batch size: 12, lr: 1.79e-02 | |
| 2024-08-06 09:50:32,135 INFO [trainer.py:765] (5/8) Epoch 6, batch 900, train_loss[loss=2.91, ArTop10Accuracy=0.7532, over 12720.00 frames. ], tot_loss[loss=2.921, ArTop10Accuracy=0.7477, over 11692.38 frames. ], batch size: 27, lr: 1.78e-02 | |
| 2024-08-06 09:51:47,298 INFO [trainer.py:765] (5/8) Epoch 6, batch 1000, train_loss[loss=2.88, ArTop10Accuracy=0.7574, over 12849.00 frames. ], tot_loss[loss=2.923, ArTop10Accuracy=0.7472, over 11888.94 frames. ], batch size: 27, lr: 1.77e-02 | |
| 2024-08-06 09:53:00,920 INFO [trainer.py:765] (5/8) Epoch 6, batch 1100, train_loss[loss=2.869, ArTop10Accuracy=0.7568, over 13800.00 frames. ], tot_loss[loss=2.928, ArTop10Accuracy=0.7463, over 11949.08 frames. ], batch size: 34, lr: 1.77e-02 | |
| 2024-08-06 09:54:14,336 INFO [trainer.py:765] (5/8) Epoch 6, batch 1200, train_loss[loss=2.998, ArTop10Accuracy=0.7337, over 12078.00 frames. ], tot_loss[loss=2.926, ArTop10Accuracy=0.7469, over 11859.24 frames. ], batch size: 101, lr: 1.76e-02 | |
| 2024-08-06 09:55:13,309 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 09:57:06,698 INFO [trainer.py:765] (5/8) Epoch 7, batch 100, train_loss[loss=2.986, ArTop10Accuracy=0.7338, over 14781.00 frames. ], tot_loss[loss=2.912, ArTop10Accuracy=0.7488, over 4773.34 frames. ], batch size: 64, lr: 1.64e-02 | |
| 2024-08-06 09:58:39,425 INFO [trainer.py:765] (5/8) Epoch 7, batch 200, train_loss[loss=2.944, ArTop10Accuracy=0.7442, over 13797.00 frames. ], tot_loss[loss=2.904, ArTop10Accuracy=0.7506, over 7754.99 frames. ], batch size: 34, lr: 1.64e-02 | |
| 2024-08-06 10:00:06,083 INFO [trainer.py:765] (5/8) Epoch 7, batch 300, train_loss[loss=2.971, ArTop10Accuracy=0.7391, over 14244.00 frames. ], tot_loss[loss=2.9, ArTop10Accuracy=0.7514, over 9363.68 frames. ], batch size: 44, lr: 1.63e-02 | |
| 2024-08-06 10:00:40,510 INFO [trainer.py:803] (5/8) Computing validation loss | |
| 2024-08-06 10:00:50,245 INFO [trainer.py:811] (5/8) Epoch 7, validation: loss=2.88, ArTop10Accuracy=0.7554, over 1827537.00 frames. | |
| 2024-08-06 10:00:50,246 INFO [trainer.py:814] (5/8) Maximum memory allocated so far is 29481MB | |
| 2024-08-06 10:00:50,976 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.002e+02 1.286e+02 1.429e+02 1.605e+02 1.020e+03, threshold=2.857e+02, percent-clipped=1.5 | |
| 2024-08-06 10:01:49,117 INFO [trainer.py:765] (5/8) Epoch 7, batch 400, train_loss[loss=2.894, ArTop10Accuracy=0.753, over 10830.00 frames. ], tot_loss[loss=2.894, ArTop10Accuracy=0.7526, over 10274.54 frames. ], batch size: 15, lr: 1.62e-02 | |
| 2024-08-06 10:03:21,459 INFO [trainer.py:765] (5/8) Epoch 7, batch 500, train_loss[loss=2.883, ArTop10Accuracy=0.7543, over 12246.00 frames. ], tot_loss[loss=2.891, ArTop10Accuracy=0.7534, over 10842.54 frames. ], batch size: 22, lr: 1.61e-02 | |
| 2024-08-06 10:04:51,882 INFO [trainer.py:765] (5/8) Epoch 7, batch 600, train_loss[loss=2.761, ArTop10Accuracy=0.7776, over 11928.00 frames. ], tot_loss[loss=2.89, ArTop10Accuracy=0.7535, over 11391.82 frames. ], batch size: 19, lr: 1.61e-02 | |
| 2024-08-06 10:06:25,111 INFO [trainer.py:765] (5/8) Epoch 7, batch 700, train_loss[loss=2.848, ArTop10Accuracy=0.7626, over 10137.00 frames. ], tot_loss[loss=2.897, ArTop10Accuracy=0.7521, over 11519.69 frames. ], batch size: 12, lr: 1.60e-02 | |
| 2024-08-06 10:07:46,948 INFO [trainer.py:765] (5/8) Epoch 7, batch 800, train_loss[loss=2.722, ArTop10Accuracy=0.7914, over 10032.00 frames. ], tot_loss[loss=2.896, ArTop10Accuracy=0.7523, over 11636.71 frames. ], batch size: 12, lr: 1.59e-02 | |
| 2024-08-06 10:09:02,824 INFO [trainer.py:765] (5/8) Epoch 7, batch 900, train_loss[loss=2.903, ArTop10Accuracy=0.7506, over 13089.00 frames. ], tot_loss[loss=2.891, ArTop10Accuracy=0.7534, over 11693.06 frames. ], batch size: 27, lr: 1.59e-02 | |
| 2024-08-06 10:10:19,635 INFO [trainer.py:765] (5/8) Epoch 7, batch 1000, train_loss[loss=2.899, ArTop10Accuracy=0.7489, over 12903.00 frames. ], tot_loss[loss=2.895, ArTop10Accuracy=0.7529, over 11887.87 frames. ], batch size: 27, lr: 1.58e-02 | |
| 2024-08-06 10:11:35,208 INFO [trainer.py:765] (5/8) Epoch 7, batch 1100, train_loss[loss=2.961, ArTop10Accuracy=0.7398, over 13668.00 frames. ], tot_loss[loss=2.899, ArTop10Accuracy=0.7522, over 11947.64 frames. ], batch size: 34, lr: 1.57e-02 | |
| 2024-08-06 10:12:48,205 INFO [trainer.py:765] (5/8) Epoch 7, batch 1200, train_loss[loss=3.019, ArTop10Accuracy=0.7253, over 12288.00 frames. ], tot_loss[loss=2.897, ArTop10Accuracy=0.7522, over 11875.89 frames. ], batch size: 101, lr: 1.57e-02 | |
| 2024-08-06 10:13:46,878 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 10:15:03,601 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.017e+02 1.283e+02 1.410e+02 1.601e+02 1.017e+03, threshold=2.820e+02, percent-clipped=0.9 | |
| 2024-08-06 10:15:40,821 INFO [trainer.py:765] (5/8) Epoch 8, batch 100, train_loss[loss=2.939, ArTop10Accuracy=0.7438, over 14424.00 frames. ], tot_loss[loss=2.88, ArTop10Accuracy=0.7551, over 4781.58 frames. ], batch size: 62, lr: 1.47e-02 | |
| 2024-08-06 10:17:12,861 INFO [trainer.py:765] (5/8) Epoch 8, batch 200, train_loss[loss=2.922, ArTop10Accuracy=0.7507, over 13815.00 frames. ], tot_loss[loss=2.872, ArTop10Accuracy=0.7568, over 7780.65 frames. ], batch size: 34, lr: 1.46e-02 | |
| 2024-08-06 10:18:37,898 INFO [trainer.py:765] (5/8) Epoch 8, batch 300, train_loss[loss=2.912, ArTop10Accuracy=0.743, over 14070.00 frames. ], tot_loss[loss=2.868, ArTop10Accuracy=0.7575, over 9380.95 frames. ], batch size: 44, lr: 1.46e-02 | |
| 2024-08-06 10:20:06,342 INFO [trainer.py:765] (5/8) Epoch 8, batch 400, train_loss[loss=2.768, ArTop10Accuracy=0.7759, over 10383.00 frames. ], tot_loss[loss=2.867, ArTop10Accuracy=0.7579, over 10283.30 frames. ], batch size: 14, lr: 1.45e-02 | |
| 2024-08-06 10:21:32,411 INFO [trainer.py:765] (5/8) Epoch 8, batch 500, train_loss[loss=2.79, ArTop10Accuracy=0.7696, over 12282.00 frames. ], tot_loss[loss=2.864, ArTop10Accuracy=0.7583, over 10854.83 frames. ], batch size: 22, lr: 1.45e-02 | |
| 2024-08-06 10:23:00,974 INFO [trainer.py:765] (5/8) Epoch 8, batch 600, train_loss[loss=2.803, ArTop10Accuracy=0.769, over 11862.00 frames. ], tot_loss[loss=2.863, ArTop10Accuracy=0.7586, over 11351.68 frames. ], batch size: 19, lr: 1.44e-02 | |
| 2024-08-06 10:24:37,787 INFO [trainer.py:765] (5/8) Epoch 8, batch 700, train_loss[loss=2.795, ArTop10Accuracy=0.7727, over 9330.00 frames. ], tot_loss[loss=2.87, ArTop10Accuracy=0.7574, over 11504.70 frames. ], batch size: 11, lr: 1.43e-02 | |
| 2024-08-06 10:25:56,085 INFO [trainer.py:765] (5/8) Epoch 8, batch 800, train_loss[loss=2.741, ArTop10Accuracy=0.7868, over 10188.00 frames. ], tot_loss[loss=2.875, ArTop10Accuracy=0.7567, over 11641.71 frames. ], batch size: 12, lr: 1.43e-02 | |
| 2024-08-06 10:27:12,244 INFO [trainer.py:765] (5/8) Epoch 8, batch 900, train_loss[loss=2.89, ArTop10Accuracy=0.7553, over 12810.00 frames. ], tot_loss[loss=2.868, ArTop10Accuracy=0.7581, over 11686.35 frames. ], batch size: 27, lr: 1.42e-02 | |
| 2024-08-06 10:28:25,263 INFO [trainer.py:765] (5/8) Epoch 8, batch 1000, train_loss[loss=2.892, ArTop10Accuracy=0.7476, over 13041.00 frames. ], tot_loss[loss=2.868, ArTop10Accuracy=0.7578, over 11879.42 frames. ], batch size: 27, lr: 1.42e-02 | |
| 2024-08-06 10:29:07,155 INFO [trainer.py:803] (5/8) Computing validation loss | |
| 2024-08-06 10:29:16,831 INFO [trainer.py:811] (5/8) Epoch 8, validation: loss=2.858, ArTop10Accuracy=0.7594, over 1827537.00 frames. | |
| 2024-08-06 10:29:16,831 INFO [trainer.py:814] (5/8) Maximum memory allocated so far is 32717MB | |
| 2024-08-06 10:29:17,490 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.032e+02 1.275e+02 1.390e+02 1.547e+02 3.717e+02, threshold=2.781e+02, percent-clipped=0.7 | |
| 2024-08-06 10:29:51,730 INFO [trainer.py:765] (5/8) Epoch 8, batch 1100, train_loss[loss=2.859, ArTop10Accuracy=0.7609, over 13695.00 frames. ], tot_loss[loss=2.874, ArTop10Accuracy=0.7563, over 11977.45 frames. ], batch size: 34, lr: 1.41e-02 | |
| 2024-08-06 10:31:05,945 INFO [trainer.py:765] (5/8) Epoch 8, batch 1200, train_loss[loss=3.031, ArTop10Accuracy=0.7265, over 12033.00 frames. ], tot_loss[loss=2.875, ArTop10Accuracy=0.7565, over 11874.16 frames. ], batch size: 101, lr: 1.40e-02 | |
| 2024-08-06 10:32:05,333 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 10:34:01,255 INFO [trainer.py:765] (5/8) Epoch 9, batch 100, train_loss[loss=2.908, ArTop10Accuracy=0.7534, over 14391.00 frames. ], tot_loss[loss=2.861, ArTop10Accuracy=0.7583, over 4756.33 frames. ], batch size: 62, lr: 1.32e-02 | |
| 2024-08-06 10:35:31,771 INFO [trainer.py:765] (5/8) Epoch 9, batch 200, train_loss[loss=2.846, ArTop10Accuracy=0.761, over 13746.00 frames. ], tot_loss[loss=2.855, ArTop10Accuracy=0.7598, over 7743.84 frames. ], batch size: 34, lr: 1.32e-02 | |
| 2024-08-06 10:36:57,927 INFO [trainer.py:765] (5/8) Epoch 9, batch 300, train_loss[loss=2.858, ArTop10Accuracy=0.7584, over 14196.00 frames. ], tot_loss[loss=2.85, ArTop10Accuracy=0.7611, over 9395.83 frames. ], batch size: 44, lr: 1.31e-02 | |
| 2024-08-06 10:38:32,696 INFO [trainer.py:765] (5/8) Epoch 9, batch 400, train_loss[loss=2.8, ArTop10Accuracy=0.7774, over 10344.00 frames. ], tot_loss[loss=2.846, ArTop10Accuracy=0.7622, over 10296.88 frames. ], batch size: 14, lr: 1.31e-02 | |
| 2024-08-06 10:39:59,255 INFO [trainer.py:765] (5/8) Epoch 9, batch 500, train_loss[loss=2.821, ArTop10Accuracy=0.7724, over 12003.00 frames. ], tot_loss[loss=2.843, ArTop10Accuracy=0.7627, over 10841.83 frames. ], batch size: 22, lr: 1.30e-02 | |
| 2024-08-06 10:41:29,689 INFO [trainer.py:765] (5/8) Epoch 9, batch 600, train_loss[loss=2.779, ArTop10Accuracy=0.7783, over 11358.00 frames. ], tot_loss[loss=2.844, ArTop10Accuracy=0.7625, over 11341.87 frames. ], batch size: 18, lr: 1.30e-02 | |
| 2024-08-06 10:42:58,440 INFO [trainer.py:765] (5/8) Epoch 9, batch 700, train_loss[loss=2.838, ArTop10Accuracy=0.7582, over 9384.00 frames. ], tot_loss[loss=2.849, ArTop10Accuracy=0.7613, over 11516.65 frames. ], batch size: 11, lr: 1.29e-02 | |
| 2024-08-06 10:44:02,952 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.039e+02 1.253e+02 1.352e+02 1.493e+02 7.010e+02, threshold=2.704e+02, percent-clipped=0.6 | |
| 2024-08-06 10:44:19,668 INFO [trainer.py:765] (5/8) Epoch 9, batch 800, train_loss[loss=2.791, ArTop10Accuracy=0.765, over 9273.00 frames. ], tot_loss[loss=2.853, ArTop10Accuracy=0.7604, over 11630.55 frames. ], batch size: 11, lr: 1.29e-02 | |
| 2024-08-06 10:45:35,718 INFO [trainer.py:765] (5/8) Epoch 9, batch 900, train_loss[loss=2.822, ArTop10Accuracy=0.7728, over 13212.00 frames. ], tot_loss[loss=2.848, ArTop10Accuracy=0.7615, over 11669.97 frames. ], batch size: 28, lr: 1.28e-02 | |
| 2024-08-06 10:46:51,270 INFO [trainer.py:765] (5/8) Epoch 9, batch 1000, train_loss[loss=2.769, ArTop10Accuracy=0.7772, over 12984.00 frames. ], tot_loss[loss=2.851, ArTop10Accuracy=0.7609, over 11902.65 frames. ], batch size: 27, lr: 1.28e-02 | |
| 2024-08-06 10:48:06,247 INFO [trainer.py:765] (5/8) Epoch 9, batch 1100, train_loss[loss=2.867, ArTop10Accuracy=0.7579, over 13776.00 frames. ], tot_loss[loss=2.857, ArTop10Accuracy=0.7596, over 11970.09 frames. ], batch size: 34, lr: 1.28e-02 | |
| 2024-08-06 10:49:21,053 INFO [trainer.py:765] (5/8) Epoch 9, batch 1200, train_loss[loss=2.949, ArTop10Accuracy=0.7393, over 12036.00 frames. ], tot_loss[loss=2.858, ArTop10Accuracy=0.7596, over 11856.75 frames. ], batch size: 101, lr: 1.27e-02 | |
| 2024-08-06 10:50:22,407 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 10:52:12,325 INFO [trainer.py:765] (5/8) Epoch 10, batch 100, train_loss[loss=2.852, ArTop10Accuracy=0.758, over 14463.00 frames. ], tot_loss[loss=2.84, ArTop10Accuracy=0.7628, over 4742.65 frames. ], batch size: 62, lr: 1.20e-02 | |
| 2024-08-06 10:53:44,585 INFO [trainer.py:765] (5/8) Epoch 10, batch 200, train_loss[loss=2.813, ArTop10Accuracy=0.7699, over 13758.00 frames. ], tot_loss[loss=2.831, ArTop10Accuracy=0.7645, over 7757.02 frames. ], batch size: 34, lr: 1.20e-02 | |
| 2024-08-06 10:55:08,089 INFO [trainer.py:765] (5/8) Epoch 10, batch 300, train_loss[loss=2.914, ArTop10Accuracy=0.7479, over 14004.00 frames. ], tot_loss[loss=2.827, ArTop10Accuracy=0.7654, over 9373.26 frames. ], batch size: 44, lr: 1.19e-02 | |
| 2024-08-06 10:56:41,176 INFO [trainer.py:765] (5/8) Epoch 10, batch 400, train_loss[loss=2.773, ArTop10Accuracy=0.7801, over 10770.00 frames. ], tot_loss[loss=2.822, ArTop10Accuracy=0.7663, over 10273.04 frames. ], batch size: 15, lr: 1.19e-02 | |
| 2024-08-06 10:58:04,937 INFO [trainer.py:803] (5/8) Computing validation loss | |
| 2024-08-06 10:58:14,559 INFO [trainer.py:811] (5/8) Epoch 10, validation: loss=2.842, ArTop10Accuracy=0.7624, over 1827537.00 frames. | |
| 2024-08-06 10:58:14,560 INFO [trainer.py:814] (5/8) Maximum memory allocated so far is 32720MB | |
| 2024-08-06 10:58:15,573 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.035e+02 1.228e+02 1.320e+02 1.458e+02 6.096e+02, threshold=2.641e+02, percent-clipped=0.6 | |
| 2024-08-06 10:58:15,577 INFO [trainer.py:765] (5/8) Epoch 10, batch 500, train_loss[loss=2.785, ArTop10Accuracy=0.773, over 11997.00 frames. ], tot_loss[loss=2.816, ArTop10Accuracy=0.7675, over 10852.04 frames. ], batch size: 22, lr: 1.19e-02 | |
| 2024-08-06 10:59:42,814 INFO [trainer.py:765] (5/8) Epoch 10, batch 600, train_loss[loss=2.791, ArTop10Accuracy=0.7766, over 11439.00 frames. ], tot_loss[loss=2.82, ArTop10Accuracy=0.7666, over 11373.96 frames. ], batch size: 18, lr: 1.18e-02 | |
| 2024-08-06 11:01:18,107 INFO [trainer.py:765] (5/8) Epoch 10, batch 700, train_loss[loss=2.726, ArTop10Accuracy=0.79, over 9531.00 frames. ], tot_loss[loss=2.824, ArTop10Accuracy=0.7659, over 11496.29 frames. ], batch size: 11, lr: 1.18e-02 | |
| 2024-08-06 11:02:36,917 INFO [trainer.py:765] (5/8) Epoch 10, batch 800, train_loss[loss=2.834, ArTop10Accuracy=0.7639, over 9489.00 frames. ], tot_loss[loss=2.83, ArTop10Accuracy=0.7648, over 11633.51 frames. ], batch size: 11, lr: 1.17e-02 | |
| 2024-08-06 11:03:51,212 INFO [trainer.py:765] (5/8) Epoch 10, batch 900, train_loss[loss=2.879, ArTop10Accuracy=0.7498, over 12795.00 frames. ], tot_loss[loss=2.825, ArTop10Accuracy=0.7655, over 11693.93 frames. ], batch size: 27, lr: 1.17e-02 | |
| 2024-08-06 11:05:06,351 INFO [trainer.py:765] (5/8) Epoch 10, batch 1000, train_loss[loss=2.797, ArTop10Accuracy=0.7712, over 12963.00 frames. ], tot_loss[loss=2.826, ArTop10Accuracy=0.7653, over 11888.02 frames. ], batch size: 27, lr: 1.17e-02 | |
| 2024-08-06 11:06:21,721 INFO [trainer.py:765] (5/8) Epoch 10, batch 1100, train_loss[loss=2.854, ArTop10Accuracy=0.7611, over 13578.00 frames. ], tot_loss[loss=2.833, ArTop10Accuracy=0.7641, over 11957.09 frames. ], batch size: 34, lr: 1.16e-02 | |
| 2024-08-06 11:07:34,771 INFO [trainer.py:765] (5/8) Epoch 10, batch 1200, train_loss[loss=2.927, ArTop10Accuracy=0.7442, over 12105.00 frames. ], tot_loss[loss=2.835, ArTop10Accuracy=0.7636, over 11866.59 frames. ], batch size: 101, lr: 1.16e-02 | |
| 2024-08-06 11:08:33,901 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 11:10:29,955 INFO [trainer.py:765] (5/8) Epoch 11, batch 100, train_loss[loss=2.92, ArTop10Accuracy=0.7496, over 14313.00 frames. ], tot_loss[loss=2.817, ArTop10Accuracy=0.7665, over 4766.81 frames. ], batch size: 62, lr: 1.10e-02 | |
| 2024-08-06 11:12:04,674 INFO [trainer.py:765] (5/8) Epoch 11, batch 200, train_loss[loss=2.854, ArTop10Accuracy=0.7574, over 13833.00 frames. ], tot_loss[loss=2.813, ArTop10Accuracy=0.7673, over 7738.81 frames. ], batch size: 35, lr: 1.10e-02 | |
| 2024-08-06 11:12:22,826 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 9.884e+01 1.240e+02 1.333e+02 1.457e+02 6.939e+02, threshold=2.667e+02, percent-clipped=0.1 | |
| 2024-08-06 11:13:31,549 INFO [trainer.py:765] (5/8) Epoch 11, batch 300, train_loss[loss=2.892, ArTop10Accuracy=0.757, over 14262.00 frames. ], tot_loss[loss=2.809, ArTop10Accuracy=0.7685, over 9368.06 frames. ], batch size: 44, lr: 1.09e-02 | |
| 2024-08-06 11:15:03,269 INFO [trainer.py:765] (5/8) Epoch 11, batch 400, train_loss[loss=2.728, ArTop10Accuracy=0.7802, over 10251.00 frames. ], tot_loss[loss=2.805, ArTop10Accuracy=0.7695, over 10282.75 frames. ], batch size: 14, lr: 1.09e-02 | |
| 2024-08-06 11:16:29,637 INFO [trainer.py:765] (5/8) Epoch 11, batch 500, train_loss[loss=2.883, ArTop10Accuracy=0.752, over 12291.00 frames. ], tot_loss[loss=2.802, ArTop10Accuracy=0.7702, over 10837.26 frames. ], batch size: 22, lr: 1.09e-02 | |
| 2024-08-06 11:18:00,517 INFO [trainer.py:765] (5/8) Epoch 11, batch 600, train_loss[loss=2.792, ArTop10Accuracy=0.7794, over 11952.00 frames. ], tot_loss[loss=2.804, ArTop10Accuracy=0.7697, over 11348.71 frames. ], batch size: 19, lr: 1.08e-02 | |
| 2024-08-06 11:19:34,514 INFO [trainer.py:765] (5/8) Epoch 11, batch 700, train_loss[loss=2.643, ArTop10Accuracy=0.8037, over 10194.00 frames. ], tot_loss[loss=2.808, ArTop10Accuracy=0.7689, over 11504.54 frames. ], batch size: 12, lr: 1.08e-02 | |
| 2024-08-06 11:20:55,484 INFO [trainer.py:765] (5/8) Epoch 11, batch 800, train_loss[loss=2.71, ArTop10Accuracy=0.797, over 9279.00 frames. ], tot_loss[loss=2.814, ArTop10Accuracy=0.7677, over 11622.96 frames. ], batch size: 11, lr: 1.07e-02 | |
| 2024-08-06 11:22:13,706 INFO [trainer.py:765] (5/8) Epoch 11, batch 900, train_loss[loss=2.818, ArTop10Accuracy=0.7658, over 13197.00 frames. ], tot_loss[loss=2.807, ArTop10Accuracy=0.7691, over 11678.82 frames. ], batch size: 28, lr: 1.07e-02 | |
| 2024-08-06 11:23:31,799 INFO [trainer.py:765] (5/8) Epoch 11, batch 1000, train_loss[loss=2.855, ArTop10Accuracy=0.7635, over 12798.00 frames. ], tot_loss[loss=2.812, ArTop10Accuracy=0.7684, over 11876.21 frames. ], batch size: 27, lr: 1.07e-02 | |
| 2024-08-06 11:24:46,902 INFO [trainer.py:765] (5/8) Epoch 11, batch 1100, train_loss[loss=2.822, ArTop10Accuracy=0.77, over 13593.00 frames. ], tot_loss[loss=2.817, ArTop10Accuracy=0.7673, over 11945.29 frames. ], batch size: 34, lr: 1.06e-02 | |
| 2024-08-06 11:26:00,733 INFO [trainer.py:765] (5/8) Epoch 11, batch 1200, train_loss[loss=2.923, ArTop10Accuracy=0.7414, over 12033.00 frames. ], tot_loss[loss=2.822, ArTop10Accuracy=0.7665, over 11870.12 frames. ], batch size: 101, lr: 1.06e-02 | |
| 2024-08-06 11:26:15,848 INFO [trainer.py:803] (5/8) Computing validation loss | |
| 2024-08-06 11:26:25,556 INFO [trainer.py:811] (5/8) Epoch 11, validation: loss=2.831, ArTop10Accuracy=0.7643, over 1827537.00 frames. | |
| 2024-08-06 11:26:25,557 INFO [trainer.py:814] (5/8) Maximum memory allocated so far is 33004MB | |
| 2024-08-06 11:26:26,186 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.029e+02 1.251e+02 1.335e+02 1.441e+02 2.942e+02, threshold=2.669e+02, percent-clipped=0.1 | |
| 2024-08-06 11:27:09,681 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 11:29:03,451 INFO [trainer.py:765] (5/8) Epoch 12, batch 100, train_loss[loss=2.86, ArTop10Accuracy=0.7608, over 14634.00 frames. ], tot_loss[loss=2.805, ArTop10Accuracy=0.7687, over 4753.89 frames. ], batch size: 63, lr: 1.01e-02 | |
| 2024-08-06 11:30:30,674 INFO [trainer.py:765] (5/8) Epoch 12, batch 200, train_loss[loss=2.759, ArTop10Accuracy=0.7796, over 13662.00 frames. ], tot_loss[loss=2.799, ArTop10Accuracy=0.7702, over 7750.57 frames. ], batch size: 34, lr: 1.01e-02 | |
| 2024-08-06 11:31:57,655 INFO [trainer.py:765] (5/8) Epoch 12, batch 300, train_loss[loss=2.788, ArTop10Accuracy=0.7695, over 14505.00 frames. ], tot_loss[loss=2.792, ArTop10Accuracy=0.772, over 9362.26 frames. ], batch size: 44, lr: 1.01e-02 | |
| 2024-08-06 11:33:30,737 INFO [trainer.py:765] (5/8) Epoch 12, batch 400, train_loss[loss=2.694, ArTop10Accuracy=0.7914, over 10332.00 frames. ], tot_loss[loss=2.791, ArTop10Accuracy=0.7723, over 10271.04 frames. ], batch size: 14, lr: 1.00e-02 | |
| 2024-08-06 11:34:55,733 INFO [trainer.py:765] (5/8) Epoch 12, batch 500, train_loss[loss=2.766, ArTop10Accuracy=0.7771, over 12225.00 frames. ], tot_loss[loss=2.786, ArTop10Accuracy=0.7732, over 10823.88 frames. ], batch size: 22, lr: 1.00e-02 | |
| 2024-08-06 11:36:29,361 INFO [trainer.py:765] (5/8) Epoch 12, batch 600, train_loss[loss=2.677, ArTop10Accuracy=0.792, over 11376.00 frames. ], tot_loss[loss=2.792, ArTop10Accuracy=0.7719, over 11337.58 frames. ], batch size: 18, lr: 9.97e-03 | |
| 2024-08-06 11:38:00,343 INFO [trainer.py:765] (5/8) Epoch 12, batch 700, train_loss[loss=2.785, ArTop10Accuracy=0.7774, over 9279.00 frames. ], tot_loss[loss=2.796, ArTop10Accuracy=0.7713, over 11485.09 frames. ], batch size: 11, lr: 9.93e-03 | |
| 2024-08-06 11:39:23,610 INFO [trainer.py:765] (5/8) Epoch 12, batch 800, train_loss[loss=2.656, ArTop10Accuracy=0.7993, over 10065.00 frames. ], tot_loss[loss=2.801, ArTop10Accuracy=0.7704, over 11623.21 frames. ], batch size: 12, lr: 9.90e-03 | |
| 2024-08-06 11:40:39,889 INFO [trainer.py:765] (5/8) Epoch 12, batch 900, train_loss[loss=2.795, ArTop10Accuracy=0.771, over 12933.00 frames. ], tot_loss[loss=2.797, ArTop10Accuracy=0.7712, over 11693.14 frames. ], batch size: 27, lr: 9.87e-03 | |
| 2024-08-06 11:41:13,995 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.041e+02 1.248e+02 1.348e+02 1.459e+02 5.540e+02, threshold=2.695e+02, percent-clipped=0.3 | |
| 2024-08-06 11:41:56,189 INFO [trainer.py:765] (5/8) Epoch 12, batch 1000, train_loss[loss=2.816, ArTop10Accuracy=0.7683, over 12939.00 frames. ], tot_loss[loss=2.8, ArTop10Accuracy=0.7707, over 11895.98 frames. ], batch size: 27, lr: 9.85e-03 | |
| 2024-08-06 11:43:14,320 INFO [trainer.py:765] (5/8) Epoch 12, batch 1100, train_loss[loss=2.788, ArTop10Accuracy=0.7715, over 13668.00 frames. ], tot_loss[loss=2.803, ArTop10Accuracy=0.7701, over 11955.74 frames. ], batch size: 34, lr: 9.82e-03 | |
| 2024-08-06 11:44:26,155 INFO [trainer.py:765] (5/8) Epoch 12, batch 1200, train_loss[loss=2.949, ArTop10Accuracy=0.7443, over 12600.00 frames. ], tot_loss[loss=2.804, ArTop10Accuracy=0.7698, over 11871.36 frames. ], batch size: 101, lr: 9.79e-03 | |
| 2024-08-06 11:45:26,265 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 11:47:26,600 INFO [trainer.py:765] (5/8) Epoch 13, batch 100, train_loss[loss=2.828, ArTop10Accuracy=0.7665, over 14454.00 frames. ], tot_loss[loss=2.795, ArTop10Accuracy=0.7707, over 4769.49 frames. ], batch size: 62, lr: 9.37e-03 | |
| 2024-08-06 11:48:54,779 INFO [trainer.py:765] (5/8) Epoch 13, batch 200, train_loss[loss=2.763, ArTop10Accuracy=0.7791, over 13644.00 frames. ], tot_loss[loss=2.785, ArTop10Accuracy=0.773, over 7764.91 frames. ], batch size: 34, lr: 9.34e-03 | |
| 2024-08-06 11:50:20,516 INFO [trainer.py:765] (5/8) Epoch 13, batch 300, train_loss[loss=2.849, ArTop10Accuracy=0.7621, over 14187.00 frames. ], tot_loss[loss=2.778, ArTop10Accuracy=0.7741, over 9372.51 frames. ], batch size: 45, lr: 9.31e-03 | |
| 2024-08-06 11:51:48,765 INFO [trainer.py:765] (5/8) Epoch 13, batch 400, train_loss[loss=2.674, ArTop10Accuracy=0.7952, over 10284.00 frames. ], tot_loss[loss=2.777, ArTop10Accuracy=0.7745, over 10272.45 frames. ], batch size: 14, lr: 9.28e-03 | |
| 2024-08-06 11:53:13,407 INFO [trainer.py:765] (5/8) Epoch 13, batch 500, train_loss[loss=2.734, ArTop10Accuracy=0.7823, over 12261.00 frames. ], tot_loss[loss=2.77, ArTop10Accuracy=0.7759, over 10847.65 frames. ], batch size: 22, lr: 9.26e-03 | |
| 2024-08-06 11:54:52,223 INFO [trainer.py:765] (5/8) Epoch 13, batch 600, train_loss[loss=2.73, ArTop10Accuracy=0.7878, over 11385.00 frames. ], tot_loss[loss=2.777, ArTop10Accuracy=0.7746, over 11374.02 frames. ], batch size: 18, lr: 9.23e-03 | |
| 2024-08-06 11:55:47,080 INFO [trainer.py:803] (5/8) Computing validation loss | |
| 2024-08-06 11:55:56,835 INFO [trainer.py:811] (5/8) Epoch 13, validation: loss=2.824, ArTop10Accuracy=0.7662, over 1827537.00 frames. | |
| 2024-08-06 11:55:56,835 INFO [trainer.py:814] (5/8) Maximum memory allocated so far is 33004MB | |
| 2024-08-06 11:55:57,712 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.064e+02 1.255e+02 1.343e+02 1.452e+02 4.888e+02, threshold=2.687e+02, percent-clipped=0.1 | |
| 2024-08-06 11:56:28,465 INFO [trainer.py:765] (5/8) Epoch 13, batch 700, train_loss[loss=2.767, ArTop10Accuracy=0.7719, over 10383.00 frames. ], tot_loss[loss=2.779, ArTop10Accuracy=0.7743, over 11511.62 frames. ], batch size: 12, lr: 9.20e-03 | |
| 2024-08-06 11:57:46,684 INFO [trainer.py:765] (5/8) Epoch 13, batch 800, train_loss[loss=2.706, ArTop10Accuracy=0.789, over 10110.00 frames. ], tot_loss[loss=2.783, ArTop10Accuracy=0.7736, over 11648.41 frames. ], batch size: 12, lr: 9.18e-03 | |
| 2024-08-06 11:59:03,287 INFO [trainer.py:765] (5/8) Epoch 13, batch 900, train_loss[loss=2.762, ArTop10Accuracy=0.7781, over 12837.00 frames. ], tot_loss[loss=2.78, ArTop10Accuracy=0.7742, over 11698.79 frames. ], batch size: 27, lr: 9.15e-03 | |
| 2024-08-06 12:00:19,174 INFO [trainer.py:765] (5/8) Epoch 13, batch 1000, train_loss[loss=2.702, ArTop10Accuracy=0.7918, over 13026.00 frames. ], tot_loss[loss=2.783, ArTop10Accuracy=0.774, over 11892.62 frames. ], batch size: 27, lr: 9.13e-03 | |
| 2024-08-06 12:01:34,881 INFO [trainer.py:765] (5/8) Epoch 13, batch 1100, train_loss[loss=2.782, ArTop10Accuracy=0.7729, over 13821.00 frames. ], tot_loss[loss=2.791, ArTop10Accuracy=0.7723, over 11969.07 frames. ], batch size: 34, lr: 9.10e-03 | |
| 2024-08-06 12:02:48,663 INFO [trainer.py:765] (5/8) Epoch 13, batch 1200, train_loss[loss=2.887, ArTop10Accuracy=0.7551, over 12504.00 frames. ], tot_loss[loss=2.79, ArTop10Accuracy=0.7727, over 11873.91 frames. ], batch size: 101, lr: 9.08e-03 | |
| 2024-08-06 12:03:48,484 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 12:05:45,334 INFO [trainer.py:765] (5/8) Epoch 14, batch 100, train_loss[loss=2.834, ArTop10Accuracy=0.7626, over 14121.00 frames. ], tot_loss[loss=2.778, ArTop10Accuracy=0.7746, over 4755.27 frames. ], batch size: 62, lr: 8.71e-03 | |
| 2024-08-06 12:07:16,604 INFO [trainer.py:765] (5/8) Epoch 14, batch 200, train_loss[loss=2.797, ArTop10Accuracy=0.7675, over 13734.00 frames. ], tot_loss[loss=2.767, ArTop10Accuracy=0.7765, over 7756.67 frames. ], batch size: 34, lr: 8.69e-03 | |
| 2024-08-06 12:08:44,311 INFO [trainer.py:765] (5/8) Epoch 14, batch 300, train_loss[loss=2.841, ArTop10Accuracy=0.76, over 14163.00 frames. ], tot_loss[loss=2.766, ArTop10Accuracy=0.7764, over 9360.38 frames. ], batch size: 44, lr: 8.66e-03 | |
| 2024-08-06 12:10:01,130 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.072e+02 1.266e+02 1.374e+02 1.483e+02 6.480e+02, threshold=2.748e+02, percent-clipped=0.2 | |
| 2024-08-06 12:10:10,226 INFO [trainer.py:765] (5/8) Epoch 14, batch 400, train_loss[loss=2.808, ArTop10Accuracy=0.7624, over 10140.00 frames. ], tot_loss[loss=2.766, ArTop10Accuracy=0.7767, over 10281.93 frames. ], batch size: 14, lr: 8.64e-03 | |
| 2024-08-06 12:11:36,150 INFO [trainer.py:765] (5/8) Epoch 14, batch 500, train_loss[loss=2.77, ArTop10Accuracy=0.7749, over 12003.00 frames. ], tot_loss[loss=2.758, ArTop10Accuracy=0.7782, over 10833.99 frames. ], batch size: 22, lr: 8.62e-03 | |
| 2024-08-06 12:13:05,993 INFO [trainer.py:765] (5/8) Epoch 14, batch 600, train_loss[loss=2.66, ArTop10Accuracy=0.7942, over 11427.00 frames. ], tot_loss[loss=2.764, ArTop10Accuracy=0.7773, over 11372.99 frames. ], batch size: 18, lr: 8.59e-03 | |
| 2024-08-06 12:14:38,552 INFO [trainer.py:765] (5/8) Epoch 14, batch 700, train_loss[loss=2.681, ArTop10Accuracy=0.7937, over 10236.00 frames. ], tot_loss[loss=2.769, ArTop10Accuracy=0.7763, over 11512.30 frames. ], batch size: 12, lr: 8.57e-03 | |
| 2024-08-06 12:15:58,069 INFO [trainer.py:765] (5/8) Epoch 14, batch 800, train_loss[loss=2.677, ArTop10Accuracy=0.7916, over 9333.00 frames. ], tot_loss[loss=2.771, ArTop10Accuracy=0.7759, over 11636.50 frames. ], batch size: 11, lr: 8.55e-03 | |
| 2024-08-06 12:17:12,866 INFO [trainer.py:765] (5/8) Epoch 14, batch 900, train_loss[loss=2.798, ArTop10Accuracy=0.7775, over 12933.00 frames. ], tot_loss[loss=2.764, ArTop10Accuracy=0.7774, over 11669.10 frames. ], batch size: 27, lr: 8.52e-03 | |
| 2024-08-06 12:18:29,615 INFO [trainer.py:765] (5/8) Epoch 14, batch 1000, train_loss[loss=2.859, ArTop10Accuracy=0.7621, over 13071.00 frames. ], tot_loss[loss=2.769, ArTop10Accuracy=0.7765, over 11877.89 frames. ], batch size: 27, lr: 8.50e-03 | |
| 2024-08-06 12:19:45,377 INFO [trainer.py:765] (5/8) Epoch 14, batch 1100, train_loss[loss=2.76, ArTop10Accuracy=0.7808, over 13647.00 frames. ], tot_loss[loss=2.778, ArTop10Accuracy=0.7747, over 11945.73 frames. ], batch size: 34, lr: 8.48e-03 | |
| 2024-08-06 12:20:59,279 INFO [trainer.py:765] (5/8) Epoch 14, batch 1200, train_loss[loss=2.936, ArTop10Accuracy=0.7421, over 11955.00 frames. ], tot_loss[loss=2.779, ArTop10Accuracy=0.7744, over 11860.24 frames. ], batch size: 101, lr: 8.46e-03 | |
| 2024-08-06 12:21:57,889 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 12:23:51,962 INFO [trainer.py:765] (5/8) Epoch 15, batch 100, train_loss[loss=2.862, ArTop10Accuracy=0.7631, over 14766.00 frames. ], tot_loss[loss=2.767, ArTop10Accuracy=0.7762, over 4772.35 frames. ], batch size: 63, lr: 8.14e-03 | |
| 2024-08-06 12:24:00,598 INFO [trainer.py:803] (5/8) Computing validation loss | |
| 2024-08-06 12:24:10,290 INFO [trainer.py:811] (5/8) Epoch 15, validation: loss=2.819, ArTop10Accuracy=0.7675, over 1827537.00 frames. | |
| 2024-08-06 12:24:10,291 INFO [trainer.py:814] (5/8) Maximum memory allocated so far is 33004MB | |
| 2024-08-06 12:24:11,094 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.080e+02 1.284e+02 1.371e+02 1.488e+02 4.667e+02, threshold=2.743e+02, percent-clipped=0.2 | |
| 2024-08-06 12:25:29,989 INFO [trainer.py:765] (5/8) Epoch 15, batch 200, train_loss[loss=2.767, ArTop10Accuracy=0.7776, over 13629.00 frames. ], tot_loss[loss=2.759, ArTop10Accuracy=0.7778, over 7753.42 frames. ], batch size: 34, lr: 8.12e-03 | |
| 2024-08-06 12:26:58,695 INFO [trainer.py:765] (5/8) Epoch 15, batch 300, train_loss[loss=2.774, ArTop10Accuracy=0.7769, over 13872.00 frames. ], tot_loss[loss=2.75, ArTop10Accuracy=0.7796, over 9374.66 frames. ], batch size: 44, lr: 8.09e-03 | |
| 2024-08-06 12:28:28,535 INFO [trainer.py:765] (5/8) Epoch 15, batch 400, train_loss[loss=2.682, ArTop10Accuracy=0.7932, over 10188.00 frames. ], tot_loss[loss=2.748, ArTop10Accuracy=0.7801, over 10289.44 frames. ], batch size: 14, lr: 8.07e-03 | |
| 2024-08-06 12:29:54,031 INFO [trainer.py:765] (5/8) Epoch 15, batch 500, train_loss[loss=2.693, ArTop10Accuracy=0.7891, over 12228.00 frames. ], tot_loss[loss=2.747, ArTop10Accuracy=0.7803, over 10854.86 frames. ], batch size: 22, lr: 8.05e-03 | |
| 2024-08-06 12:31:23,293 INFO [trainer.py:765] (5/8) Epoch 15, batch 600, train_loss[loss=2.795, ArTop10Accuracy=0.7702, over 11559.00 frames. ], tot_loss[loss=2.751, ArTop10Accuracy=0.7795, over 11372.03 frames. ], batch size: 18, lr: 8.03e-03 | |
| 2024-08-06 12:32:53,176 INFO [trainer.py:765] (5/8) Epoch 15, batch 700, train_loss[loss=2.946, ArTop10Accuracy=0.7441, over 9426.00 frames. ], tot_loss[loss=2.755, ArTop10Accuracy=0.7785, over 11509.96 frames. ], batch size: 11, lr: 8.01e-03 | |
| 2024-08-06 12:34:18,254 INFO [trainer.py:765] (5/8) Epoch 15, batch 800, train_loss[loss=2.731, ArTop10Accuracy=0.7853, over 10077.00 frames. ], tot_loss[loss=2.759, ArTop10Accuracy=0.7778, over 11655.46 frames. ], batch size: 12, lr: 7.99e-03 | |
| 2024-08-06 12:35:34,726 INFO [trainer.py:765] (5/8) Epoch 15, batch 900, train_loss[loss=2.748, ArTop10Accuracy=0.7775, over 12918.00 frames. ], tot_loss[loss=2.757, ArTop10Accuracy=0.7782, over 11703.63 frames. ], batch size: 27, lr: 7.97e-03 | |
| 2024-08-06 12:36:50,540 INFO [trainer.py:765] (5/8) Epoch 15, batch 1000, train_loss[loss=2.679, ArTop10Accuracy=0.7947, over 12975.00 frames. ], tot_loss[loss=2.758, ArTop10Accuracy=0.7781, over 11898.75 frames. ], batch size: 27, lr: 7.95e-03 | |
| 2024-08-06 12:38:05,179 INFO [trainer.py:765] (5/8) Epoch 15, batch 1100, train_loss[loss=2.721, ArTop10Accuracy=0.7841, over 13785.00 frames. ], tot_loss[loss=2.765, ArTop10Accuracy=0.7767, over 11973.07 frames. ], batch size: 34, lr: 7.93e-03 | |
| 2024-08-06 12:38:12,841 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.080e+02 1.293e+02 1.379e+02 1.467e+02 2.824e+02, threshold=2.759e+02, percent-clipped=0.1 | |
| 2024-08-06 12:39:18,789 INFO [trainer.py:765] (5/8) Epoch 15, batch 1200, train_loss[loss=2.935, ArTop10Accuracy=0.7409, over 12156.00 frames. ], tot_loss[loss=2.767, ArTop10Accuracy=0.7763, over 11863.35 frames. ], batch size: 101, lr: 7.91e-03 | |
| 2024-08-06 12:40:18,769 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 12:42:17,619 INFO [trainer.py:765] (5/8) Epoch 16, batch 100, train_loss[loss=2.761, ArTop10Accuracy=0.7795, over 14676.00 frames. ], tot_loss[loss=2.752, ArTop10Accuracy=0.779, over 4775.40 frames. ], batch size: 62, lr: 7.63e-03 | |
| 2024-08-06 12:43:49,563 INFO [trainer.py:765] (5/8) Epoch 16, batch 200, train_loss[loss=2.658, ArTop10Accuracy=0.7943, over 13419.00 frames. ], tot_loss[loss=2.746, ArTop10Accuracy=0.7803, over 7764.60 frames. ], batch size: 34, lr: 7.61e-03 | |
| 2024-08-06 12:45:18,502 INFO [trainer.py:765] (5/8) Epoch 16, batch 300, train_loss[loss=2.827, ArTop10Accuracy=0.7627, over 14460.00 frames. ], tot_loss[loss=2.742, ArTop10Accuracy=0.7812, over 9390.93 frames. ], batch size: 44, lr: 7.59e-03 | |
| 2024-08-06 12:46:45,209 INFO [trainer.py:765] (5/8) Epoch 16, batch 400, train_loss[loss=2.647, ArTop10Accuracy=0.8053, over 10230.00 frames. ], tot_loss[loss=2.739, ArTop10Accuracy=0.7819, over 10315.27 frames. ], batch size: 14, lr: 7.58e-03 | |
| 2024-08-06 12:48:16,311 INFO [trainer.py:765] (5/8) Epoch 16, batch 500, train_loss[loss=2.762, ArTop10Accuracy=0.7751, over 12336.00 frames. ], tot_loss[loss=2.737, ArTop10Accuracy=0.7821, over 10875.67 frames. ], batch size: 22, lr: 7.56e-03 | |
| 2024-08-06 12:49:46,642 INFO [trainer.py:765] (5/8) Epoch 16, batch 600, train_loss[loss=2.76, ArTop10Accuracy=0.7807, over 11430.00 frames. ], tot_loss[loss=2.74, ArTop10Accuracy=0.7817, over 11381.50 frames. ], batch size: 18, lr: 7.54e-03 | |
| 2024-08-06 12:51:23,681 INFO [trainer.py:765] (5/8) Epoch 16, batch 700, train_loss[loss=2.55, ArTop10Accuracy=0.8138, over 9411.00 frames. ], tot_loss[loss=2.74, ArTop10Accuracy=0.7817, over 11524.49 frames. ], batch size: 11, lr: 7.52e-03 | |
| 2024-08-06 12:52:43,501 INFO [trainer.py:765] (5/8) Epoch 16, batch 800, train_loss[loss=2.647, ArTop10Accuracy=0.8003, over 10152.00 frames. ], tot_loss[loss=2.745, ArTop10Accuracy=0.7807, over 11650.13 frames. ], batch size: 12, lr: 7.51e-03 | |
| 2024-08-06 12:53:06,015 INFO [trainer.py:803] (5/8) Computing validation loss | |
| 2024-08-06 12:53:15,497 INFO [trainer.py:811] (5/8) Epoch 16, validation: loss=2.816, ArTop10Accuracy=0.7678, over 1827537.00 frames. | |
| 2024-08-06 12:53:15,497 INFO [trainer.py:814] (5/8) Maximum memory allocated so far is 33004MB | |
| 2024-08-06 12:53:16,186 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.112e+02 1.291e+02 1.391e+02 1.487e+02 3.459e+02, threshold=2.783e+02, percent-clipped=0.1 | |
| 2024-08-06 12:54:06,480 INFO [trainer.py:765] (5/8) Epoch 16, batch 900, train_loss[loss=2.733, ArTop10Accuracy=0.7827, over 13089.00 frames. ], tot_loss[loss=2.74, ArTop10Accuracy=0.7817, over 11714.19 frames. ], batch size: 27, lr: 7.49e-03 | |
| 2024-08-06 12:55:19,791 INFO [trainer.py:765] (5/8) Epoch 16, batch 1000, train_loss[loss=2.714, ArTop10Accuracy=0.7895, over 12903.00 frames. ], tot_loss[loss=2.744, ArTop10Accuracy=0.7809, over 11908.25 frames. ], batch size: 27, lr: 7.47e-03 | |
| 2024-08-06 12:56:33,163 INFO [trainer.py:765] (5/8) Epoch 16, batch 1100, train_loss[loss=2.734, ArTop10Accuracy=0.7784, over 13548.00 frames. ], tot_loss[loss=2.756, ArTop10Accuracy=0.7786, over 11968.05 frames. ], batch size: 34, lr: 7.45e-03 | |
| 2024-08-06 12:57:48,485 INFO [trainer.py:765] (5/8) Epoch 16, batch 1200, train_loss[loss=2.914, ArTop10Accuracy=0.7486, over 12042.00 frames. ], tot_loss[loss=2.755, ArTop10Accuracy=0.7789, over 11883.70 frames. ], batch size: 101, lr: 7.44e-03 | |
| 2024-08-06 12:58:48,462 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 13:00:47,900 INFO [trainer.py:765] (5/8) Epoch 17, batch 100, train_loss[loss=2.79, ArTop10Accuracy=0.7739, over 14514.00 frames. ], tot_loss[loss=2.744, ArTop10Accuracy=0.7802, over 4746.89 frames. ], batch size: 62, lr: 7.18e-03 | |
| 2024-08-06 13:02:19,302 INFO [trainer.py:765] (5/8) Epoch 17, batch 200, train_loss[loss=2.762, ArTop10Accuracy=0.7752, over 13503.00 frames. ], tot_loss[loss=2.737, ArTop10Accuracy=0.7818, over 7745.07 frames. ], batch size: 34, lr: 7.17e-03 | |
| 2024-08-06 13:03:45,517 INFO [trainer.py:765] (5/8) Epoch 17, batch 300, train_loss[loss=2.704, ArTop10Accuracy=0.7863, over 14109.00 frames. ], tot_loss[loss=2.728, ArTop10Accuracy=0.7837, over 9363.13 frames. ], batch size: 44, lr: 7.15e-03 | |
| 2024-08-06 13:05:21,760 INFO [trainer.py:765] (5/8) Epoch 17, batch 400, train_loss[loss=2.597, ArTop10Accuracy=0.8068, over 10308.00 frames. ], tot_loss[loss=2.727, ArTop10Accuracy=0.784, over 10298.50 frames. ], batch size: 14, lr: 7.14e-03 | |
| 2024-08-06 13:06:47,021 INFO [trainer.py:765] (5/8) Epoch 17, batch 500, train_loss[loss=2.65, ArTop10Accuracy=0.7994, over 12138.00 frames. ], tot_loss[loss=2.723, ArTop10Accuracy=0.7848, over 10855.95 frames. ], batch size: 22, lr: 7.12e-03 | |
| 2024-08-06 13:07:39,878 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.140e+02 1.293e+02 1.386e+02 1.488e+02 3.253e+02, threshold=2.772e+02, percent-clipped=0.1 | |
| 2024-08-06 13:08:22,688 INFO [trainer.py:765] (5/8) Epoch 17, batch 600, train_loss[loss=2.731, ArTop10Accuracy=0.7827, over 11508.00 frames. ], tot_loss[loss=2.729, ArTop10Accuracy=0.7836, over 11353.38 frames. ], batch size: 18, lr: 7.10e-03 | |
| 2024-08-06 13:09:54,835 INFO [trainer.py:765] (5/8) Epoch 17, batch 700, train_loss[loss=2.661, ArTop10Accuracy=0.8012, over 10047.00 frames. ], tot_loss[loss=2.728, ArTop10Accuracy=0.7837, over 11499.97 frames. ], batch size: 12, lr: 7.09e-03 | |
| 2024-08-06 13:11:19,480 INFO [trainer.py:765] (5/8) Epoch 17, batch 800, train_loss[loss=2.683, ArTop10Accuracy=0.7903, over 9426.00 frames. ], tot_loss[loss=2.731, ArTop10Accuracy=0.7833, over 11612.45 frames. ], batch size: 11, lr: 7.07e-03 | |
| 2024-08-06 13:12:35,669 INFO [trainer.py:765] (5/8) Epoch 17, batch 900, train_loss[loss=2.679, ArTop10Accuracy=0.7909, over 13299.00 frames. ], tot_loss[loss=2.728, ArTop10Accuracy=0.784, over 11667.58 frames. ], batch size: 28, lr: 7.06e-03 | |
| 2024-08-06 13:13:53,061 INFO [trainer.py:765] (5/8) Epoch 17, batch 1000, train_loss[loss=2.747, ArTop10Accuracy=0.7768, over 12849.00 frames. ], tot_loss[loss=2.733, ArTop10Accuracy=0.7829, over 11871.51 frames. ], batch size: 27, lr: 7.04e-03 | |
| 2024-08-06 13:15:08,484 INFO [trainer.py:765] (5/8) Epoch 17, batch 1100, train_loss[loss=2.783, ArTop10Accuracy=0.7756, over 13917.00 frames. ], tot_loss[loss=2.743, ArTop10Accuracy=0.781, over 11963.27 frames. ], batch size: 34, lr: 7.02e-03 | |
| 2024-08-06 13:16:22,388 INFO [trainer.py:765] (5/8) Epoch 17, batch 1200, train_loss[loss=2.881, ArTop10Accuracy=0.7528, over 11922.00 frames. ], tot_loss[loss=2.742, ArTop10Accuracy=0.7812, over 11883.35 frames. ], batch size: 101, lr: 7.01e-03 | |
| 2024-08-06 13:17:21,256 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 13:19:15,994 INFO [trainer.py:765] (5/8) Epoch 18, batch 100, train_loss[loss=2.836, ArTop10Accuracy=0.7618, over 14517.00 frames. ], tot_loss[loss=2.734, ArTop10Accuracy=0.7823, over 4741.07 frames. ], batch size: 62, lr: 6.78e-03 | |
| 2024-08-06 13:20:46,600 INFO [trainer.py:765] (5/8) Epoch 18, batch 200, train_loss[loss=2.73, ArTop10Accuracy=0.7836, over 13572.00 frames. ], tot_loss[loss=2.723, ArTop10Accuracy=0.7844, over 7726.79 frames. ], batch size: 34, lr: 6.77e-03 | |
| 2024-08-06 13:21:55,105 INFO [trainer.py:803] (5/8) Computing validation loss | |
| 2024-08-06 13:22:04,751 INFO [trainer.py:811] (5/8) Epoch 18, validation: loss=2.817, ArTop10Accuracy=0.768, over 1827537.00 frames. | |
| 2024-08-06 13:22:04,752 INFO [trainer.py:814] (5/8) Maximum memory allocated so far is 33004MB | |
| 2024-08-06 13:22:05,473 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.131e+02 1.323e+02 1.409e+02 1.514e+02 3.209e+02, threshold=2.818e+02, percent-clipped=0.1 | |
| 2024-08-06 13:22:26,581 INFO [trainer.py:765] (5/8) Epoch 18, batch 300, train_loss[loss=2.857, ArTop10Accuracy=0.7546, over 14451.00 frames. ], tot_loss[loss=2.716, ArTop10Accuracy=0.7859, over 9365.24 frames. ], batch size: 44, lr: 6.76e-03 | |
| 2024-08-06 13:23:57,930 INFO [trainer.py:765] (5/8) Epoch 18, batch 400, train_loss[loss=2.635, ArTop10Accuracy=0.7974, over 10905.00 frames. ], tot_loss[loss=2.714, ArTop10Accuracy=0.7862, over 10266.45 frames. ], batch size: 15, lr: 6.74e-03 | |
| 2024-08-06 13:25:34,013 INFO [trainer.py:765] (5/8) Epoch 18, batch 500, train_loss[loss=2.698, ArTop10Accuracy=0.7882, over 12288.00 frames. ], tot_loss[loss=2.708, ArTop10Accuracy=0.7875, over 10841.88 frames. ], batch size: 22, lr: 6.73e-03 | |
| 2024-08-06 13:27:00,633 INFO [trainer.py:765] (5/8) Epoch 18, batch 600, train_loss[loss=2.6, ArTop10Accuracy=0.8053, over 11361.00 frames. ], tot_loss[loss=2.715, ArTop10Accuracy=0.786, over 11357.44 frames. ], batch size: 18, lr: 6.71e-03 | |
| 2024-08-06 13:28:33,583 INFO [trainer.py:765] (5/8) Epoch 18, batch 700, train_loss[loss=2.631, ArTop10Accuracy=0.7964, over 9192.00 frames. ], tot_loss[loss=2.72, ArTop10Accuracy=0.785, over 11507.07 frames. ], batch size: 11, lr: 6.70e-03 | |
| 2024-08-06 13:29:54,984 INFO [trainer.py:765] (5/8) Epoch 18, batch 800, train_loss[loss=2.628, ArTop10Accuracy=0.8056, over 9312.00 frames. ], tot_loss[loss=2.721, ArTop10Accuracy=0.7848, over 11606.69 frames. ], batch size: 11, lr: 6.68e-03 | |
| 2024-08-06 13:31:12,519 INFO [trainer.py:765] (5/8) Epoch 18, batch 900, train_loss[loss=2.791, ArTop10Accuracy=0.7691, over 12933.00 frames. ], tot_loss[loss=2.719, ArTop10Accuracy=0.7852, over 11666.15 frames. ], batch size: 27, lr: 6.67e-03 | |
| 2024-08-06 13:32:26,551 INFO [trainer.py:765] (5/8) Epoch 18, batch 1000, train_loss[loss=2.696, ArTop10Accuracy=0.7862, over 12858.00 frames. ], tot_loss[loss=2.729, ArTop10Accuracy=0.7835, over 11872.20 frames. ], batch size: 27, lr: 6.66e-03 | |
| 2024-08-06 13:33:41,497 INFO [trainer.py:765] (5/8) Epoch 18, batch 1100, train_loss[loss=2.694, ArTop10Accuracy=0.788, over 13674.00 frames. ], tot_loss[loss=2.736, ArTop10Accuracy=0.7822, over 11940.32 frames. ], batch size: 34, lr: 6.64e-03 | |
| 2024-08-06 13:34:54,674 INFO [trainer.py:765] (5/8) Epoch 18, batch 1200, train_loss[loss=2.855, ArTop10Accuracy=0.7583, over 12042.00 frames. ], tot_loss[loss=2.736, ArTop10Accuracy=0.782, over 11858.01 frames. ], batch size: 101, lr: 6.63e-03 | |
| 2024-08-06 13:35:51,064 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.124e+02 1.340e+02 1.433e+02 1.533e+02 2.444e+02, threshold=2.867e+02, percent-clipped=0.0 | |
| 2024-08-06 13:35:54,247 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 13:37:48,624 INFO [trainer.py:765] (5/8) Epoch 19, batch 100, train_loss[loss=2.784, ArTop10Accuracy=0.7731, over 14520.00 frames. ], tot_loss[loss=2.72, ArTop10Accuracy=0.7851, over 4772.92 frames. ], batch size: 62, lr: 6.43e-03 | |
| 2024-08-06 13:39:23,257 INFO [trainer.py:765] (5/8) Epoch 19, batch 200, train_loss[loss=2.721, ArTop10Accuracy=0.7922, over 13578.00 frames. ], tot_loss[loss=2.714, ArTop10Accuracy=0.7863, over 7762.37 frames. ], batch size: 34, lr: 6.41e-03 | |
| 2024-08-06 13:40:48,360 INFO [trainer.py:765] (5/8) Epoch 19, batch 300, train_loss[loss=2.743, ArTop10Accuracy=0.7811, over 14157.00 frames. ], tot_loss[loss=2.712, ArTop10Accuracy=0.7867, over 9395.02 frames. ], batch size: 44, lr: 6.40e-03 | |
| 2024-08-06 13:42:21,067 INFO [trainer.py:765] (5/8) Epoch 19, batch 400, train_loss[loss=2.584, ArTop10Accuracy=0.8123, over 10896.00 frames. ], tot_loss[loss=2.706, ArTop10Accuracy=0.7878, over 10297.34 frames. ], batch size: 15, lr: 6.39e-03 | |
| 2024-08-06 13:43:44,955 INFO [trainer.py:765] (5/8) Epoch 19, batch 500, train_loss[loss=2.74, ArTop10Accuracy=0.7834, over 12279.00 frames. ], tot_loss[loss=2.701, ArTop10Accuracy=0.7886, over 10864.55 frames. ], batch size: 22, lr: 6.37e-03 | |
| 2024-08-06 13:45:16,682 INFO [trainer.py:765] (5/8) Epoch 19, batch 600, train_loss[loss=2.702, ArTop10Accuracy=0.7867, over 11439.00 frames. ], tot_loss[loss=2.706, ArTop10Accuracy=0.788, over 11385.29 frames. ], batch size: 18, lr: 6.36e-03 | |
| 2024-08-06 13:46:48,324 INFO [trainer.py:765] (5/8) Epoch 19, batch 700, train_loss[loss=2.638, ArTop10Accuracy=0.8023, over 10281.00 frames. ], tot_loss[loss=2.71, ArTop10Accuracy=0.7873, over 11528.57 frames. ], batch size: 12, lr: 6.35e-03 | |
| 2024-08-06 13:48:11,884 INFO [trainer.py:765] (5/8) Epoch 19, batch 800, train_loss[loss=2.69, ArTop10Accuracy=0.7886, over 9447.00 frames. ], tot_loss[loss=2.716, ArTop10Accuracy=0.786, over 11643.01 frames. ], batch size: 11, lr: 6.34e-03 | |
| 2024-08-06 13:49:27,259 INFO [trainer.py:765] (5/8) Epoch 19, batch 900, train_loss[loss=2.693, ArTop10Accuracy=0.7895, over 12918.00 frames. ], tot_loss[loss=2.709, ArTop10Accuracy=0.7871, over 11699.57 frames. ], batch size: 27, lr: 6.32e-03 | |
| 2024-08-06 13:50:40,655 INFO [trainer.py:803] (5/8) Computing validation loss | |
| 2024-08-06 13:50:50,537 INFO [trainer.py:811] (5/8) Epoch 19, validation: loss=2.818, ArTop10Accuracy=0.7679, over 1827537.00 frames. | |
| 2024-08-06 13:50:50,537 INFO [trainer.py:814] (5/8) Maximum memory allocated so far is 33004MB | |
| 2024-08-06 13:50:51,490 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.161e+02 1.371e+02 1.455e+02 1.550e+02 3.697e+02, threshold=2.909e+02, percent-clipped=0.2 | |
| 2024-08-06 13:50:52,916 INFO [trainer.py:765] (5/8) Epoch 19, batch 1000, train_loss[loss=2.719, ArTop10Accuracy=0.7876, over 12822.00 frames. ], tot_loss[loss=2.717, ArTop10Accuracy=0.7857, over 11884.36 frames. ], batch size: 27, lr: 6.31e-03 | |
| 2024-08-06 13:52:08,266 INFO [trainer.py:765] (5/8) Epoch 19, batch 1100, train_loss[loss=2.753, ArTop10Accuracy=0.7806, over 13827.00 frames. ], tot_loss[loss=2.725, ArTop10Accuracy=0.7842, over 11932.60 frames. ], batch size: 34, lr: 6.30e-03 | |
| 2024-08-06 13:53:22,314 INFO [trainer.py:765] (5/8) Epoch 19, batch 1200, train_loss[loss=2.875, ArTop10Accuracy=0.754, over 12696.00 frames. ], tot_loss[loss=2.727, ArTop10Accuracy=0.7839, over 11864.89 frames. ], batch size: 101, lr: 6.28e-03 | |
| 2024-08-06 13:54:21,608 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 13:56:12,905 INFO [trainer.py:765] (5/8) Epoch 20, batch 100, train_loss[loss=2.771, ArTop10Accuracy=0.774, over 14373.00 frames. ], tot_loss[loss=2.713, ArTop10Accuracy=0.7863, over 4760.92 frames. ], batch size: 62, lr: 6.10e-03 | |
| 2024-08-06 13:57:42,495 INFO [trainer.py:765] (5/8) Epoch 20, batch 200, train_loss[loss=2.687, ArTop10Accuracy=0.7925, over 13419.00 frames. ], tot_loss[loss=2.704, ArTop10Accuracy=0.788, over 7758.04 frames. ], batch size: 34, lr: 6.09e-03 | |
| 2024-08-06 13:59:15,430 INFO [trainer.py:765] (5/8) Epoch 20, batch 300, train_loss[loss=2.741, ArTop10Accuracy=0.7801, over 13872.00 frames. ], tot_loss[loss=2.698, ArTop10Accuracy=0.7891, over 9376.52 frames. ], batch size: 44, lr: 6.08e-03 | |
| 2024-08-06 14:00:44,357 INFO [trainer.py:765] (5/8) Epoch 20, batch 400, train_loss[loss=2.606, ArTop10Accuracy=0.8038, over 10281.00 frames. ], tot_loss[loss=2.703, ArTop10Accuracy=0.7882, over 10286.27 frames. ], batch size: 14, lr: 6.07e-03 | |
| 2024-08-06 14:02:14,855 INFO [trainer.py:765] (5/8) Epoch 20, batch 500, train_loss[loss=2.693, ArTop10Accuracy=0.7924, over 12243.00 frames. ], tot_loss[loss=2.7, ArTop10Accuracy=0.7887, over 10847.60 frames. ], batch size: 22, lr: 6.06e-03 | |
| 2024-08-06 14:03:40,856 INFO [trainer.py:765] (5/8) Epoch 20, batch 600, train_loss[loss=2.575, ArTop10Accuracy=0.8142, over 11583.00 frames. ], tot_loss[loss=2.701, ArTop10Accuracy=0.7885, over 11371.01 frames. ], batch size: 18, lr: 6.04e-03 | |
| 2024-08-06 14:05:13,864 INFO [trainer.py:765] (5/8) Epoch 20, batch 700, train_loss[loss=2.636, ArTop10Accuracy=0.8101, over 10137.00 frames. ], tot_loss[loss=2.704, ArTop10Accuracy=0.788, over 11525.42 frames. ], batch size: 12, lr: 6.03e-03 | |
| 2024-08-06 14:05:30,791 INFO [optim.py:386] (5/8) Clipping_scale=2.0, grad-norm quartiles 1.180e+02 1.365e+02 1.456e+02 1.550e+02 3.525e+02, threshold=2.913e+02, percent-clipped=0.1 | |
| 2024-08-06 14:06:34,509 INFO [trainer.py:765] (5/8) Epoch 20, batch 800, train_loss[loss=2.63, ArTop10Accuracy=0.805, over 9363.00 frames. ], tot_loss[loss=2.708, ArTop10Accuracy=0.7871, over 11647.49 frames. ], batch size: 11, lr: 6.02e-03 | |
| 2024-08-06 14:07:50,944 INFO [trainer.py:765] (5/8) Epoch 20, batch 900, train_loss[loss=2.69, ArTop10Accuracy=0.7948, over 12888.00 frames. ], tot_loss[loss=2.704, ArTop10Accuracy=0.7881, over 11693.94 frames. ], batch size: 27, lr: 6.01e-03 | |
| 2024-08-06 14:09:07,173 INFO [trainer.py:765] (5/8) Epoch 20, batch 1000, train_loss[loss=2.754, ArTop10Accuracy=0.7762, over 12837.00 frames. ], tot_loss[loss=2.709, ArTop10Accuracy=0.7873, over 11889.47 frames. ], batch size: 27, lr: 6.00e-03 | |
| 2024-08-06 14:10:21,209 INFO [trainer.py:765] (5/8) Epoch 20, batch 1100, train_loss[loss=2.775, ArTop10Accuracy=0.776, over 13869.00 frames. ], tot_loss[loss=2.716, ArTop10Accuracy=0.7859, over 11949.23 frames. ], batch size: 35, lr: 5.99e-03 | |
| 2024-08-06 14:11:37,813 INFO [trainer.py:765] (5/8) Epoch 20, batch 1200, train_loss[loss=2.857, ArTop10Accuracy=0.7624, over 11853.00 frames. ], tot_loss[loss=2.715, ArTop10Accuracy=0.7862, over 11859.92 frames. ], batch size: 101, lr: 5.98e-03 | |
| 2024-08-06 14:12:37,384 INFO [trainer.py:650] (5/8) Reaches end of dataloader. | |
| 2024-08-06 14:12:37,386 INFO [trainer.py:1069] (5/8) Done! | |