| Use bfloat16 for core module. | |
| Use parallel attention and MLP in ViViT. | |
| Epoch 001/400 | |
| Train loss: 111647336.00 correlation: 0.0122 | |
| Validation loss: 200300080.00 correlation: 0.0267 | |
| Elapse: 537.13s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 002/400 | |
| Train loss: 97648256.00 correlation: 0.0367 | |
| Validation loss: 199289632.00 correlation: 0.0355 | |
| Elapse: 539.72s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 003/400 | |
| Train loss: 96321792.00 correlation: 0.0500 | |
| Validation loss: 198299696.00 correlation: 0.0446 | |
| Elapse: 548.37s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 004/400 | |
| Train loss: 95213544.00 correlation: 0.0617 | |
| Validation loss: 196983504.00 correlation: 0.0527 | |
| Elapse: 552.04s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 005/400 | |
| Train loss: 93915056.00 correlation: 0.0756 | |
| Validation loss: 195022704.00 correlation: 0.0660 | |
| Elapse: 551.29s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 006/400 | |
| Train loss: 92500376.00 correlation: 0.0909 | |
| Validation loss: 193045536.00 correlation: 0.0815 | |
| Elapse: 549.45s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 007/400 | |
| Train loss: 91511136.00 correlation: 0.1012 | |
| Validation loss: 191545056.00 correlation: 0.0905 | |
| Elapse: 547.11s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 008/400 | |
| Train loss: 90428944.00 correlation: 0.1113 | |
| Validation loss: 189976016.00 correlation: 0.0998 | |
| Elapse: 544.76s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 009/400 | |
| Train loss: 89301664.00 correlation: 0.1228 | |
| Validation loss: 188514336.00 correlation: 0.1074 | |
| Elapse: 541.79s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 010/400 | |
| Train loss: 88238616.00 correlation: 0.1335 | |
| Validation loss: 186730592.00 correlation: 0.1194 | |
| Elapse: 540.44s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 011/400 | |
| Train loss: 87239336.00 correlation: 0.1436 | |
| Validation loss: 185287104.00 correlation: 0.1296 | |
| Elapse: 539.71s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 012/400 | |
| Train loss: 86205256.00 correlation: 0.1540 | |
| Validation loss: 183237792.00 correlation: 0.1419 | |
| Elapse: 540.02s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 013/400 | |
| Train loss: 85068472.00 correlation: 0.1655 | |
| Validation loss: 181721296.00 correlation: 0.1515 | |
| Elapse: 539.81s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 014/400 | |
| Train loss: 84160560.00 correlation: 0.1745 | |
| Validation loss: 180469664.00 correlation: 0.1595 | |
| Elapse: 540.34s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 015/400 | |
| Train loss: 83386128.00 correlation: 0.1821 | |
| Validation loss: 179517792.00 correlation: 0.1658 | |
| Elapse: 540.81s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 016/400 | |
| Train loss: 82805200.00 correlation: 0.1877 | |
| Validation loss: 178801584.00 correlation: 0.1698 | |
| Elapse: 540.70s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 017/400 | |
| Train loss: 82275736.00 correlation: 0.1926 | |
| Validation loss: 177842080.00 correlation: 0.1761 | |
| Elapse: 540.76s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 018/400 | |
| Train loss: 81662840.00 correlation: 0.1987 | |
| Validation loss: 176971248.00 correlation: 0.1820 | |
| Elapse: 541.07s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 019/400 | |
| Train loss: 81147088.00 correlation: 0.2038 | |
| Validation loss: 176423600.00 correlation: 0.1858 | |
| Elapse: 541.20s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 020/400 | |
| Train loss: 80748784.00 correlation: 0.2079 | |
| Validation loss: 175690976.00 correlation: 0.1908 | |
| Elapse: 541.22s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 021/400 | |
| Train loss: 80313880.00 correlation: 0.2122 | |
| Validation loss: 175389216.00 correlation: 0.1929 | |
| Elapse: 540.65s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 022/400 | |
| Train loss: 79957744.00 correlation: 0.2153 | |
| Validation loss: 174862864.00 correlation: 0.1968 | |
| Elapse: 540.96s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 023/400 | |
| Train loss: 79621360.00 correlation: 0.2186 | |
| Validation loss: 174437632.00 correlation: 0.1996 | |
| Elapse: 541.11s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 024/400 | |
| Train loss: 79374368.00 correlation: 0.2211 | |
| Validation loss: 173783968.00 correlation: 0.2031 | |
| Elapse: 542.03s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 025/400 | |
| Train loss: 79085072.00 correlation: 0.2240 | |
| Validation loss: 173357520.00 correlation: 0.2059 | |
| Elapse: 541.66s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 026/400 | |
| Train loss: 78723728.00 correlation: 0.2273 | |
| Validation loss: 173125120.00 correlation: 0.2087 | |
| Elapse: 542.13s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 027/400 | |
| Train loss: 78456624.00 correlation: 0.2299 | |
| Validation loss: 172740432.00 correlation: 0.2102 | |
| Elapse: 541.72s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 028/400 | |
| Train loss: 78264240.00 correlation: 0.2316 | |
| Validation loss: 172426416.00 correlation: 0.2125 | |
| Elapse: 541.90s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 029/400 | |
| Train loss: 78095968.00 correlation: 0.2335 | |
| Validation loss: 172338688.00 correlation: 0.2139 | |
| Elapse: 542.04s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 030/400 | |
| Train loss: 77880056.00 correlation: 0.2357 | |
| Validation loss: 171972944.00 correlation: 0.2151 | |
| Elapse: 541.96s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 031/400 | |
| Train loss: 77782920.00 correlation: 0.2367 | |
| Validation loss: 171662176.00 correlation: 0.2171 | |
| Elapse: 542.59s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 032/400 | |
| Train loss: 77596272.00 correlation: 0.2382 | |
| Validation loss: 171484624.00 correlation: 0.2187 | |
| Elapse: 543.51s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 033/400 | |
| Train loss: 77378984.00 correlation: 0.2403 | |
| Validation loss: 171180896.00 correlation: 0.2208 | |
| Elapse: 542.09s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 034/400 | |
| Train loss: 77329272.00 correlation: 0.2411 | |
| Validation loss: 171382000.00 correlation: 0.2191 | |
| Elapse: 541.49s | |
| Epoch 035/400 | |
| Train loss: 77173400.00 correlation: 0.2426 | |
| Validation loss: 170990288.00 correlation: 0.2215 | |
| Elapse: 542.22s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 036/400 | |
| Train loss: 77103352.00 correlation: 0.2430 | |
| Validation loss: 171064224.00 correlation: 0.2212 | |
| Elapse: 542.85s | |
| Epoch 037/400 | |
| Train loss: 76961264.00 correlation: 0.2447 | |
| Validation loss: 170693952.00 correlation: 0.2239 | |
| Elapse: 542.80s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 038/400 | |
| Train loss: 76838424.00 correlation: 0.2456 | |
| Validation loss: 170481408.00 correlation: 0.2246 | |
| Elapse: 542.40s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 039/400 | |
| Train loss: 76726736.00 correlation: 0.2469 | |
| Validation loss: 170405888.00 correlation: 0.2257 | |
| Elapse: 541.86s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 040/400 | |
| Train loss: 76625736.00 correlation: 0.2481 | |
| Validation loss: 170183216.00 correlation: 0.2271 | |
| Elapse: 542.67s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 041/400 | |
| Train loss: 76633712.00 correlation: 0.2477 | |
| Validation loss: 170194400.00 correlation: 0.2270 | |
| Elapse: 542.30s | |
| Epoch 042/400 | |
| Train loss: 76442136.00 correlation: 0.2493 | |
| Validation loss: 170452384.00 correlation: 0.2263 | |
| Elapse: 542.26s | |
| Epoch 043/400 | |
| Train loss: 76492200.00 correlation: 0.2491 | |
| Validation loss: 170249952.00 correlation: 0.2260 | |
| Elapse: 542.10s | |
| Epoch 044/400 | |
| Train loss: 76279480.00 correlation: 0.2510 | |
| Validation loss: 169784496.00 correlation: 0.2295 | |
| Elapse: 541.77s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 045/400 | |
| Train loss: 76193080.00 correlation: 0.2520 | |
| Validation loss: 169944928.00 correlation: 0.2278 | |
| Elapse: 542.23s | |
| Epoch 046/400 | |
| Train loss: 76107936.00 correlation: 0.2526 | |
| Validation loss: 169860016.00 correlation: 0.2283 | |
| Elapse: 542.90s | |
| Epoch 047/400 | |
| Train loss: 76111024.00 correlation: 0.2527 | |
| Validation loss: 169548304.00 correlation: 0.2308 | |
| Elapse: 541.89s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 048/400 | |
| Train loss: 76049672.00 correlation: 0.2534 | |
| Validation loss: 169801264.00 correlation: 0.2296 | |
| Elapse: 543.95s | |
| Epoch 049/400 | |
| Train loss: 75997232.00 correlation: 0.2537 | |
| Validation loss: 169721248.00 correlation: 0.2301 | |
| Elapse: 543.00s | |
| Epoch 050/400 | |
| Train loss: 75961488.00 correlation: 0.2545 | |
| Validation loss: 169722912.00 correlation: 0.2293 | |
| Elapse: 542.91s | |
| Epoch 051/400 | |
| Train loss: 75908376.00 correlation: 0.2548 | |
| Validation loss: 169474704.00 correlation: 0.2310 | |
| Elapse: 542.56s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 052/400 | |
| Train loss: 75771856.00 correlation: 0.2561 | |
| Validation loss: 170132720.00 correlation: 0.2272 | |
| Elapse: 542.05s | |
| Epoch 053/400 | |
| Train loss: 75913696.00 correlation: 0.2547 | |
| Validation loss: 169491888.00 correlation: 0.2309 | |
| Elapse: 542.57s | |
| Epoch 054/400 | |
| Train loss: 75764272.00 correlation: 0.2560 | |
| Validation loss: 169266704.00 correlation: 0.2330 | |
| Elapse: 542.91s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 055/400 | |
| Train loss: 75670816.00 correlation: 0.2569 | |
| Validation loss: 169200320.00 correlation: 0.2327 | |
| Elapse: 542.21s | |
| Epoch 056/400 | |
| Train loss: 75680240.00 correlation: 0.2570 | |
| Validation loss: 169253344.00 correlation: 0.2322 | |
| Elapse: 542.73s | |
| Epoch 057/400 | |
| Train loss: 75623792.00 correlation: 0.2573 | |
| Validation loss: 169158704.00 correlation: 0.2336 | |
| Elapse: 544.29s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 058/400 | |
| Train loss: 75518968.00 correlation: 0.2584 | |
| Validation loss: 169124832.00 correlation: 0.2339 | |
| Elapse: 542.64s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 059/400 | |
| Train loss: 75452928.00 correlation: 0.2591 | |
| Validation loss: 169483008.00 correlation: 0.2320 | |
| Elapse: 542.34s | |
| Epoch 060/400 | |
| Train loss: 75478992.00 correlation: 0.2590 | |
| Validation loss: 169223728.00 correlation: 0.2337 | |
| Elapse: 543.01s | |
| Epoch 061/400 | |
| Train loss: 75414704.00 correlation: 0.2597 | |
| Validation loss: 169588672.00 correlation: 0.2307 | |
| Elapse: 542.79s | |
| Epoch 062/400 | |
| Train loss: 75631880.00 correlation: 0.2578 | |
| Validation loss: 169325440.00 correlation: 0.2326 | |
| Elapse: 543.75s | |
| Epoch 063/400 | |
| Train loss: 75359640.00 correlation: 0.2602 | |
| Validation loss: 169091040.00 correlation: 0.2339 | |
| Elapse: 543.16s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 064/400 | |
| Train loss: 75319080.00 correlation: 0.2605 | |
| Validation loss: 169416560.00 correlation: 0.2321 | |
| Elapse: 543.52s | |
| Epoch 065/400 | |
| Train loss: 75306104.00 correlation: 0.2609 | |
| Validation loss: 169176224.00 correlation: 0.2331 | |
| Elapse: 542.80s | |
| Epoch 066/400 | |
| Train loss: 75301592.00 correlation: 0.2609 | |
| Validation loss: 169371616.00 correlation: 0.2327 | |
| Elapse: 543.10s | |
| Epoch 067/400 | |
| Train loss: 75389672.00 correlation: 0.2601 | |
| Validation loss: 169169312.00 correlation: 0.2332 | |
| Elapse: 544.11s | |
| Epoch 068/400 | |
| Train loss: 75361008.00 correlation: 0.2604 | |
| Validation loss: 169366960.00 correlation: 0.2315 | |
| Elapse: 543.42s | |
| Loaded checkpoint from epoch 63 (correlation: 0.2339). | |
| Reduce learning rate of core to 1.4400e-03 (num. reduce: 1). | |
| Reduce learning rate of readouts to 1.0800e-03 (num. reduce: 1). | |
| Reduce learning rate of shifters to 1.0800e-03 (num. reduce: 1). | |
| Epoch 069/400 | |
| Train loss: 73278464.00 correlation: 0.2784 | |
| Validation loss: 167482848.00 correlation: 0.2447 | |
| Elapse: 544.59s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 070/400 | |
| Train loss: 72562424.00 correlation: 0.2849 | |
| Validation loss: 167277872.00 correlation: 0.2456 | |
| Elapse: 544.90s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 071/400 | |
| Train loss: 72429480.00 correlation: 0.2862 | |
| Validation loss: 167337568.00 correlation: 0.2452 | |
| Elapse: 544.84s | |
| Epoch 072/400 | |
| Train loss: 72339856.00 correlation: 0.2872 | |
| Validation loss: 167472656.00 correlation: 0.2442 | |
| Elapse: 544.89s | |
| Epoch 073/400 | |
| Train loss: 72270912.00 correlation: 0.2877 | |
| Validation loss: 167356592.00 correlation: 0.2443 | |
| Elapse: 545.34s | |
| Epoch 074/400 | |
| Train loss: 72207960.00 correlation: 0.2884 | |
| Validation loss: 167467040.00 correlation: 0.2434 | |
| Elapse: 545.38s | |
| Epoch 075/400 | |
| Train loss: 72186752.00 correlation: 0.2885 | |
| Validation loss: 167473600.00 correlation: 0.2434 | |
| Elapse: 545.13s | |
| Loaded checkpoint from epoch 70 (correlation: 0.2456). | |
| Reduce learning rate of core to 4.3200e-04 (num. reduce: 1). | |
| Reduce learning rate of readouts to 3.2400e-04 (num. reduce: 1). | |
| Reduce learning rate of shifters to 3.2400e-04 (num. reduce: 1). | |
| Epoch 076/400 | |
| Train loss: 71743648.00 correlation: 0.2919 | |
| Validation loss: 166945040.00 correlation: 0.2479 | |
| Elapse: 544.86s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 077/400 | |
| Train loss: 71453328.00 correlation: 0.2949 | |
| Validation loss: 166933984.00 correlation: 0.2480 | |
| Elapse: 545.20s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 078/400 | |
| Train loss: 71407184.00 correlation: 0.2951 | |
| Validation loss: 166839616.00 correlation: 0.2480 | |
| Elapse: 545.38s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 079/400 | |
| Train loss: 71308560.00 correlation: 0.2960 | |
| Validation loss: 166820688.00 correlation: 0.2484 | |
| Elapse: 545.26s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 080/400 | |
| Train loss: 71223520.00 correlation: 0.2968 | |
| Validation loss: 166801616.00 correlation: 0.2487 | |
| Elapse: 545.81s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 081/400 | |
| Train loss: 71151600.00 correlation: 0.2975 | |
| Validation loss: 166777584.00 correlation: 0.2485 | |
| Elapse: 545.15s | |
| Epoch 082/400 | |
| Train loss: 71166128.00 correlation: 0.2973 | |
| Validation loss: 166806736.00 correlation: 0.2484 | |
| Elapse: 545.51s | |
| Epoch 083/400 | |
| Train loss: 71003768.00 correlation: 0.2991 | |
| Validation loss: 166794000.00 correlation: 0.2487 | |
| Elapse: 545.39s | |
| Epoch 084/400 | |
| Train loss: 71028680.00 correlation: 0.2987 | |
| Validation loss: 166740976.00 correlation: 0.2486 | |
| Elapse: 545.50s | |
| Epoch 085/400 | |
| Train loss: 70987920.00 correlation: 0.2989 | |
| Validation loss: 166771344.00 correlation: 0.2483 | |
| Elapse: 545.38s | |
| Loaded checkpoint from epoch 80 (correlation: 0.2487). | |
| Reduce learning rate of core to 1.2960e-04 (num. reduce: 1). | |
| Reduce learning rate of readouts to 9.7200e-05 (num. reduce: 1). | |
| Reduce learning rate of shifters to 9.7200e-05 (num. reduce: 1). | |
| Epoch 086/400 | |
| Train loss: 70886192.00 correlation: 0.2998 | |
| Validation loss: 166696544.00 correlation: 0.2492 | |
| Elapse: 545.68s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 087/400 | |
| Train loss: 70891072.00 correlation: 0.2998 | |
| Validation loss: 166700640.00 correlation: 0.2493 | |
| Elapse: 545.36s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 088/400 | |
| Train loss: 70786208.00 correlation: 0.3009 | |
| Validation loss: 166720928.00 correlation: 0.2490 | |
| Elapse: 545.75s | |
| Epoch 089/400 | |
| Train loss: 70840864.00 correlation: 0.3002 | |
| Validation loss: 166705888.00 correlation: 0.2491 | |
| Elapse: 546.01s | |
| Epoch 090/400 | |
| Train loss: 70781008.00 correlation: 0.3008 | |
| Validation loss: 166757248.00 correlation: 0.2489 | |
| Elapse: 545.02s | |
| Epoch 091/400 | |
| Train loss: 70768960.00 correlation: 0.3009 | |
| Validation loss: 166698640.00 correlation: 0.2489 | |
| Elapse: 545.07s | |
| Epoch 092/400 | |
| Train loss: 70700664.00 correlation: 0.3016 | |
| Validation loss: 166705392.00 correlation: 0.2491 | |
| Elapse: 545.85s | |
| Loaded checkpoint from epoch 87 (correlation: 0.2493). | |
| Reduce learning rate of core to 3.8880e-05 (num. reduce: 1). | |
| Reduce learning rate of readouts to 2.9160e-05 (num. reduce: 1). | |
| Reduce learning rate of shifters to 2.9160e-05 (num. reduce: 1). | |
| Epoch 093/400 | |
| Train loss: 70743624.00 correlation: 0.3012 | |
| Validation loss: 166685456.00 correlation: 0.2494 | |
| Elapse: 545.69s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 094/400 | |
| Train loss: 70752552.00 correlation: 0.3012 | |
| Validation loss: 166676048.00 correlation: 0.2493 | |
| Elapse: 545.53s | |
| Epoch 095/400 | |
| Train loss: 70729328.00 correlation: 0.3013 | |
| Validation loss: 166677072.00 correlation: 0.2494 | |
| Elapse: 545.12s | |
| Epoch 096/400 | |
| Train loss: 70661304.00 correlation: 0.3018 | |
| Validation loss: 166687824.00 correlation: 0.2493 | |
| Elapse: 545.30s | |
| Epoch 097/400 | |
| Train loss: 70768896.00 correlation: 0.3006 | |
| Validation loss: 166658272.00 correlation: 0.2492 | |
| Elapse: 545.47s | |
| Epoch 098/400 | |
| Train loss: 70673976.00 correlation: 0.3017 | |
| Validation loss: 166648288.00 correlation: 0.2495 | |
| Elapse: 545.22s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 099/400 | |
| Train loss: 70635672.00 correlation: 0.3022 | |
| Validation loss: 166648032.00 correlation: 0.2495 | |
| Elapse: 545.17s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 100/400 | |
| Train loss: 70748008.00 correlation: 0.3010 | |
| Validation loss: 166647312.00 correlation: 0.2495 | |
| Elapse: 545.61s | |
| Epoch 101/400 | |
| Train loss: 70682408.00 correlation: 0.3016 | |
| Validation loss: 166662208.00 correlation: 0.2494 | |
| Elapse: 545.56s | |
| Epoch 102/400 | |
| Train loss: 70642688.00 correlation: 0.3020 | |
| Validation loss: 166662624.00 correlation: 0.2495 | |
| Elapse: 545.17s | |
| Epoch 103/400 | |
| Train loss: 70638368.00 correlation: 0.3019 | |
| Validation loss: 166645120.00 correlation: 0.2495 | |
| Elapse: 545.25s | |
| Epoch 104/400 | |
| Train loss: 70563872.00 correlation: 0.3027 | |
| Validation loss: 166649696.00 correlation: 0.2495 | |
| Elapse: 545.70s | |
| Loaded checkpoint from epoch 99 (correlation: 0.2495). | |
| Reduce learning rate of core to 1.1664e-05 (num. reduce: 1). | |
| Reduce learning rate of readouts to 8.7480e-06 (num. reduce: 1). | |
| Reduce learning rate of shifters to 8.7480e-06 (num. reduce: 1). | |
| Epoch 105/400 | |
| Train loss: 70679696.00 correlation: 0.3019 | |
| Validation loss: 166646944.00 correlation: 0.2495 | |
| Elapse: 545.44s | |
| Epoch 106/400 | |
| Train loss: 70623656.00 correlation: 0.3021 | |
| Validation loss: 166651056.00 correlation: 0.2495 | |
| Elapse: 545.46s | |
| Epoch 107/400 | |
| Train loss: 70689912.00 correlation: 0.3013 | |
| Validation loss: 166642080.00 correlation: 0.2495 | |
| Elapse: 545.17s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 108/400 | |
| Train loss: 70636352.00 correlation: 0.3019 | |
| Validation loss: 166643376.00 correlation: 0.2495 | |
| Elapse: 545.12s | |
| Epoch 109/400 | |
| Train loss: 70622056.00 correlation: 0.3021 | |
| Validation loss: 166645664.00 correlation: 0.2495 | |
| Elapse: 545.91s | |
| Epoch 110/400 | |
| Train loss: 70689848.00 correlation: 0.3013 | |
| Validation loss: 166646544.00 correlation: 0.2495 | |
| Elapse: 544.95s | |
| Epoch 111/400 | |
| Train loss: 70594792.00 correlation: 0.3025 | |
| Validation loss: 166655344.00 correlation: 0.2494 | |
| Elapse: 545.66s | |
| Epoch 112/400 | |
| Train loss: 70636904.00 correlation: 0.3020 | |
| Validation loss: 166638224.00 correlation: 0.2495 | |
| Elapse: 544.74s | |
| Loaded checkpoint from epoch 107 (correlation: 0.2495). | |
| Reduce learning rate of core to 3.4992e-06 (num. reduce: 1). | |
| Reduce learning rate of readouts to 2.6244e-06 (num. reduce: 1). | |
| Reduce learning rate of shifters to 2.6244e-06 (num. reduce: 1). | |
| Epoch 113/400 | |
| Train loss: 70679760.00 correlation: 0.3016 | |
| Validation loss: 166645488.00 correlation: 0.2495 | |
| Elapse: 543.62s | |
| Epoch 114/400 | |
| Train loss: 70609840.00 correlation: 0.3026 | |
| Validation loss: 166641568.00 correlation: 0.2495 | |
| Elapse: 543.57s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/028/ckpt/model_state.pt. | |
| Epoch 115/400 | |
| Train loss: 70676264.00 correlation: 0.3017 | |
| Validation loss: 166645824.00 correlation: 0.2495 | |
| Elapse: 545.22s | |
| Epoch 116/400 | |
| Train loss: 70640760.00 correlation: 0.3023 | |
| Validation loss: 166649888.00 correlation: 0.2495 | |
| Elapse: 545.14s | |
| Epoch 117/400 | |
| Train loss: 70651464.00 correlation: 0.3017 | |
| Validation loss: 166644896.00 correlation: 0.2495 | |
| Elapse: 545.75s | |
| Epoch 118/400 | |
| Train loss: 70647664.00 correlation: 0.3020 | |
| Validation loss: 166642016.00 correlation: 0.2495 | |
| Elapse: 551.10s | |
| Epoch 119/400 | |
| Train loss: 70647632.00 correlation: 0.3017 | |
| Validation loss: 166642992.00 correlation: 0.2495 | |
| Elapse: 545.02s | |
| Loaded checkpoint from epoch 114 (correlation: 0.2495). | |
| Reduce learning rate of core to 1.0498e-06 (num. reduce: 1). | |
| Reduce learning rate of readouts to 7.8732e-07 (num. reduce: 1). | |
| Reduce learning rate of shifters to 7.8732e-07 (num. reduce: 1). | |
| Epoch 120/400 | |
| Train loss: 70673976.00 correlation: 0.3014 | |
| Validation loss: 166641376.00 correlation: 0.2495 | |
| Elapse: 544.88s | |
| Epoch 121/400 | |
| Train loss: 70599064.00 correlation: 0.3023 | |
| Validation loss: 166642464.00 correlation: 0.2495 | |
| Elapse: 544.91s | |
| Epoch 122/400 | |
| Train loss: 70592552.00 correlation: 0.3025 | |
| Validation loss: 166642720.00 correlation: 0.2495 | |
| Elapse: 544.80s | |
| Epoch 123/400 | |
| Train loss: 70644184.00 correlation: 0.3019 | |
| Validation loss: 166643072.00 correlation: 0.2495 | |
| Elapse: 544.57s | |
| Epoch 124/400 | |
| Train loss: 70629504.00 correlation: 0.3022 | |
| Validation loss: 166642208.00 correlation: 0.2495 | |
| Elapse: 545.38s | |
| Loaded checkpoint from epoch 114 (correlation: 0.2495). | |
| Reduce learning rate of core to 3.1493e-07 (num. reduce: 2). | |
| Reduce learning rate of readouts to 2.3620e-07 (num. reduce: 2). | |
| Reduce learning rate of shifters to 2.3620e-07 (num. reduce: 2). | |
| Epoch 125/400 | |
| Train loss: 70617168.00 correlation: 0.3023 | |
| Validation loss: 166641952.00 correlation: 0.2495 | |
| Elapse: 544.97s | |
| Epoch 126/400 | |
| Train loss: 70613328.00 correlation: 0.3026 | |
| Validation loss: 166642704.00 correlation: 0.2495 | |
| Elapse: 544.92s | |
| Epoch 127/400 | |
| Train loss: 70681712.00 correlation: 0.3016 | |
| Validation loss: 166642848.00 correlation: 0.2495 | |
| Elapse: 544.98s | |
| Epoch 128/400 | |
| Train loss: 70630672.00 correlation: 0.3021 | |
| Validation loss: 166642912.00 correlation: 0.2495 | |
| Elapse: 544.96s | |
| Epoch 129/400 | |
| Train loss: 70600632.00 correlation: 0.3025 | |
| Validation loss: 166642848.00 correlation: 0.2495 | |
| Elapse: 545.27s | |
| Model has not improved after 2 LR reductions. | |
| Loaded checkpoint from epoch 114 (correlation: 0.2495). | |
| ValidationA: 0.2465 B: 0.2783 C: 0.2688 D: 0.2292 E: 0.2366 F: 0.2341 G: 0.2549 H: 0.2330 I: 0.2567 J: 0.2571 average: 0.2495 | |
| Results saved to /home/storage/runs/vivit_ensemble/028. | |