| Use bfloat16 for core module. | |
| Use parallel attention and MLP in ViViT. | |
| Epoch 001/400 | |
| Train loss: 108875712.00 correlation: 0.0142 | |
| Validation loss: 200009632.00 correlation: 0.0280 | |
| Elapse: 536.31s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 002/400 | |
| Train loss: 97286888.00 correlation: 0.0397 | |
| Validation loss: 199179056.00 correlation: 0.0389 | |
| Elapse: 540.45s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 003/400 | |
| Train loss: 96065840.00 correlation: 0.0525 | |
| Validation loss: 198705632.00 correlation: 0.0414 | |
| Elapse: 548.06s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 004/400 | |
| Train loss: 95196720.00 correlation: 0.0616 | |
| Validation loss: 197267888.00 correlation: 0.0520 | |
| Elapse: 550.50s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 005/400 | |
| Train loss: 94175976.00 correlation: 0.0728 | |
| Validation loss: 195531712.00 correlation: 0.0637 | |
| Elapse: 549.52s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 006/400 | |
| Train loss: 92696320.00 correlation: 0.0883 | |
| Validation loss: 193088208.00 correlation: 0.0800 | |
| Elapse: 547.83s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 007/400 | |
| Train loss: 91328912.00 correlation: 0.1027 | |
| Validation loss: 190954368.00 correlation: 0.0931 | |
| Elapse: 546.41s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 008/400 | |
| Train loss: 90154608.00 correlation: 0.1149 | |
| Validation loss: 189186224.00 correlation: 0.1027 | |
| Elapse: 544.21s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 009/400 | |
| Train loss: 88984904.00 correlation: 0.1267 | |
| Validation loss: 187096768.00 correlation: 0.1152 | |
| Elapse: 542.67s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 010/400 | |
| Train loss: 87719928.00 correlation: 0.1389 | |
| Validation loss: 185224656.00 correlation: 0.1287 | |
| Elapse: 541.24s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 011/400 | |
| Train loss: 86455896.00 correlation: 0.1522 | |
| Validation loss: 183495936.00 correlation: 0.1386 | |
| Elapse: 540.40s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 012/400 | |
| Train loss: 85286208.00 correlation: 0.1638 | |
| Validation loss: 181750576.00 correlation: 0.1502 | |
| Elapse: 539.78s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 013/400 | |
| Train loss: 84359632.00 correlation: 0.1728 | |
| Validation loss: 180806192.00 correlation: 0.1570 | |
| Elapse: 540.59s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 014/400 | |
| Train loss: 83654496.00 correlation: 0.1800 | |
| Validation loss: 179662112.00 correlation: 0.1648 | |
| Elapse: 539.99s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 015/400 | |
| Train loss: 82936160.00 correlation: 0.1870 | |
| Validation loss: 178832416.00 correlation: 0.1697 | |
| Elapse: 540.96s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 016/400 | |
| Train loss: 82256432.00 correlation: 0.1933 | |
| Validation loss: 177940560.00 correlation: 0.1755 | |
| Elapse: 541.08s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 017/400 | |
| Train loss: 81816704.00 correlation: 0.1974 | |
| Validation loss: 177331792.00 correlation: 0.1789 | |
| Elapse: 541.24s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 018/400 | |
| Train loss: 81317624.00 correlation: 0.2024 | |
| Validation loss: 176620096.00 correlation: 0.1839 | |
| Elapse: 541.38s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 019/400 | |
| Train loss: 80761904.00 correlation: 0.2073 | |
| Validation loss: 175956080.00 correlation: 0.1882 | |
| Elapse: 540.85s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 020/400 | |
| Train loss: 80425104.00 correlation: 0.2111 | |
| Validation loss: 175399376.00 correlation: 0.1923 | |
| Elapse: 541.23s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 021/400 | |
| Train loss: 80006704.00 correlation: 0.2149 | |
| Validation loss: 174810944.00 correlation: 0.1963 | |
| Elapse: 540.64s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 022/400 | |
| Train loss: 79605576.00 correlation: 0.2188 | |
| Validation loss: 174441312.00 correlation: 0.1990 | |
| Elapse: 541.29s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 023/400 | |
| Train loss: 79238384.00 correlation: 0.2225 | |
| Validation loss: 173628144.00 correlation: 0.2041 | |
| Elapse: 541.23s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 024/400 | |
| Train loss: 78942312.00 correlation: 0.2253 | |
| Validation loss: 173239408.00 correlation: 0.2069 | |
| Elapse: 541.38s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 025/400 | |
| Train loss: 78635512.00 correlation: 0.2284 | |
| Validation loss: 172979872.00 correlation: 0.2084 | |
| Elapse: 541.43s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 026/400 | |
| Train loss: 78411792.00 correlation: 0.2303 | |
| Validation loss: 172513792.00 correlation: 0.2110 | |
| Elapse: 541.42s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 027/400 | |
| Train loss: 78185872.00 correlation: 0.2329 | |
| Validation loss: 172230208.00 correlation: 0.2128 | |
| Elapse: 540.87s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 028/400 | |
| Train loss: 77904656.00 correlation: 0.2353 | |
| Validation loss: 172162992.00 correlation: 0.2139 | |
| Elapse: 541.44s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 029/400 | |
| Train loss: 77741728.00 correlation: 0.2368 | |
| Validation loss: 171699152.00 correlation: 0.2166 | |
| Elapse: 542.05s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 030/400 | |
| Train loss: 77624216.00 correlation: 0.2381 | |
| Validation loss: 171523792.00 correlation: 0.2179 | |
| Elapse: 541.70s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 031/400 | |
| Train loss: 77405640.00 correlation: 0.2398 | |
| Validation loss: 171512912.00 correlation: 0.2182 | |
| Elapse: 541.80s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 032/400 | |
| Train loss: 77315544.00 correlation: 0.2411 | |
| Validation loss: 171059296.00 correlation: 0.2209 | |
| Elapse: 541.92s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 033/400 | |
| Train loss: 77214704.00 correlation: 0.2420 | |
| Validation loss: 171228784.00 correlation: 0.2201 | |
| Elapse: 541.77s | |
| Epoch 034/400 | |
| Train loss: 77108704.00 correlation: 0.2433 | |
| Validation loss: 170971952.00 correlation: 0.2219 | |
| Elapse: 541.81s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 035/400 | |
| Train loss: 76970384.00 correlation: 0.2445 | |
| Validation loss: 170653312.00 correlation: 0.2236 | |
| Elapse: 542.50s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 036/400 | |
| Train loss: 76829184.00 correlation: 0.2459 | |
| Validation loss: 170723072.00 correlation: 0.2228 | |
| Elapse: 542.11s | |
| Epoch 037/400 | |
| Train loss: 76814728.00 correlation: 0.2461 | |
| Validation loss: 170317536.00 correlation: 0.2260 | |
| Elapse: 542.26s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 038/400 | |
| Train loss: 76648232.00 correlation: 0.2476 | |
| Validation loss: 170352096.00 correlation: 0.2263 | |
| Elapse: 542.30s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 039/400 | |
| Train loss: 76570480.00 correlation: 0.2485 | |
| Validation loss: 170048544.00 correlation: 0.2276 | |
| Elapse: 542.30s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 040/400 | |
| Train loss: 76481936.00 correlation: 0.2492 | |
| Validation loss: 170178064.00 correlation: 0.2271 | |
| Elapse: 542.72s | |
| Epoch 041/400 | |
| Train loss: 76392648.00 correlation: 0.2501 | |
| Validation loss: 169916672.00 correlation: 0.2292 | |
| Elapse: 542.22s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 042/400 | |
| Train loss: 76277856.00 correlation: 0.2512 | |
| Validation loss: 169939984.00 correlation: 0.2288 | |
| Elapse: 542.30s | |
| Epoch 043/400 | |
| Train loss: 76200912.00 correlation: 0.2518 | |
| Validation loss: 169966848.00 correlation: 0.2293 | |
| Elapse: 543.01s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 044/400 | |
| Train loss: 76180752.00 correlation: 0.2520 | |
| Validation loss: 169804672.00 correlation: 0.2301 | |
| Elapse: 541.51s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 045/400 | |
| Train loss: 76124944.00 correlation: 0.2529 | |
| Validation loss: 169699056.00 correlation: 0.2297 | |
| Elapse: 541.18s | |
| Epoch 046/400 | |
| Train loss: 75982544.00 correlation: 0.2542 | |
| Validation loss: 169554976.00 correlation: 0.2308 | |
| Elapse: 541.59s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 047/400 | |
| Train loss: 75966320.00 correlation: 0.2542 | |
| Validation loss: 169422160.00 correlation: 0.2317 | |
| Elapse: 540.36s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 048/400 | |
| Train loss: 75923248.00 correlation: 0.2547 | |
| Validation loss: 169780352.00 correlation: 0.2287 | |
| Elapse: 540.75s | |
| Epoch 049/400 | |
| Train loss: 75947968.00 correlation: 0.2547 | |
| Validation loss: 169837856.00 correlation: 0.2298 | |
| Elapse: 541.57s | |
| Epoch 050/400 | |
| Train loss: 75809040.00 correlation: 0.2556 | |
| Validation loss: 169605200.00 correlation: 0.2306 | |
| Elapse: 541.98s | |
| Epoch 051/400 | |
| Train loss: 75789808.00 correlation: 0.2558 | |
| Validation loss: 169401440.00 correlation: 0.2307 | |
| Elapse: 542.15s | |
| Epoch 052/400 | |
| Train loss: 75764560.00 correlation: 0.2561 | |
| Validation loss: 169899840.00 correlation: 0.2291 | |
| Elapse: 541.98s | |
| Loaded checkpoint from epoch 47 (correlation: 0.2317). | |
| Reduce learning rate of core to 1.4400e-03 (num. reduce: 1). | |
| Reduce learning rate of readouts to 1.0800e-03 (num. reduce: 1). | |
| Reduce learning rate of shifters to 1.0800e-03 (num. reduce: 1). | |
| Epoch 053/400 | |
| Train loss: 73947496.00 correlation: 0.2716 | |
| Validation loss: 167854176.00 correlation: 0.2423 | |
| Elapse: 542.40s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 054/400 | |
| Train loss: 73334280.00 correlation: 0.2771 | |
| Validation loss: 167810464.00 correlation: 0.2430 | |
| Elapse: 543.06s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 055/400 | |
| Train loss: 73178416.00 correlation: 0.2788 | |
| Validation loss: 167610272.00 correlation: 0.2431 | |
| Elapse: 543.11s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 056/400 | |
| Train loss: 73106944.00 correlation: 0.2793 | |
| Validation loss: 167676960.00 correlation: 0.2432 | |
| Elapse: 543.44s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 057/400 | |
| Train loss: 73098200.00 correlation: 0.2798 | |
| Validation loss: 167731616.00 correlation: 0.2427 | |
| Elapse: 543.05s | |
| Epoch 058/400 | |
| Train loss: 73030688.00 correlation: 0.2803 | |
| Validation loss: 167715872.00 correlation: 0.2427 | |
| Elapse: 543.72s | |
| Epoch 059/400 | |
| Train loss: 72944768.00 correlation: 0.2814 | |
| Validation loss: 167514192.00 correlation: 0.2440 | |
| Elapse: 543.63s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 060/400 | |
| Train loss: 72929872.00 correlation: 0.2815 | |
| Validation loss: 167771024.00 correlation: 0.2423 | |
| Elapse: 543.29s | |
| Epoch 061/400 | |
| Train loss: 72848184.00 correlation: 0.2822 | |
| Validation loss: 167556976.00 correlation: 0.2437 | |
| Elapse: 543.81s | |
| Epoch 062/400 | |
| Train loss: 72894440.00 correlation: 0.2819 | |
| Validation loss: 167687056.00 correlation: 0.2429 | |
| Elapse: 543.05s | |
| Epoch 063/400 | |
| Train loss: 72843624.00 correlation: 0.2824 | |
| Validation loss: 167582688.00 correlation: 0.2431 | |
| Elapse: 544.00s | |
| Epoch 064/400 | |
| Train loss: 72864712.00 correlation: 0.2818 | |
| Validation loss: 167485088.00 correlation: 0.2435 | |
| Elapse: 543.20s | |
| Loaded checkpoint from epoch 59 (correlation: 0.2440). | |
| Reduce learning rate of core to 4.3200e-04 (num. reduce: 1). | |
| Reduce learning rate of readouts to 3.2400e-04 (num. reduce: 1). | |
| Reduce learning rate of shifters to 3.2400e-04 (num. reduce: 1). | |
| Epoch 065/400 | |
| Train loss: 72100880.00 correlation: 0.2884 | |
| Validation loss: 167043168.00 correlation: 0.2470 | |
| Elapse: 543.83s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 066/400 | |
| Train loss: 71817264.00 correlation: 0.2907 | |
| Validation loss: 166928480.00 correlation: 0.2480 | |
| Elapse: 543.10s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 067/400 | |
| Train loss: 71765488.00 correlation: 0.2914 | |
| Validation loss: 166935472.00 correlation: 0.2473 | |
| Elapse: 543.05s | |
| Epoch 068/400 | |
| Train loss: 71709384.00 correlation: 0.2919 | |
| Validation loss: 166836192.00 correlation: 0.2479 | |
| Elapse: 544.05s | |
| Epoch 069/400 | |
| Train loss: 71609624.00 correlation: 0.2931 | |
| Validation loss: 166932704.00 correlation: 0.2478 | |
| Elapse: 544.51s | |
| Epoch 070/400 | |
| Train loss: 71532048.00 correlation: 0.2937 | |
| Validation loss: 166911168.00 correlation: 0.2475 | |
| Elapse: 543.81s | |
| Epoch 071/400 | |
| Train loss: 71531912.00 correlation: 0.2938 | |
| Validation loss: 166931488.00 correlation: 0.2474 | |
| Elapse: 543.44s | |
| Loaded checkpoint from epoch 66 (correlation: 0.2480). | |
| Reduce learning rate of core to 1.2960e-04 (num. reduce: 1). | |
| Reduce learning rate of readouts to 9.7200e-05 (num. reduce: 1). | |
| Reduce learning rate of shifters to 9.7200e-05 (num. reduce: 1). | |
| Epoch 072/400 | |
| Train loss: 71521936.00 correlation: 0.2938 | |
| Validation loss: 166835152.00 correlation: 0.2481 | |
| Elapse: 544.28s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 073/400 | |
| Train loss: 71422952.00 correlation: 0.2945 | |
| Validation loss: 166809568.00 correlation: 0.2486 | |
| Elapse: 542.98s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 074/400 | |
| Train loss: 71402352.00 correlation: 0.2945 | |
| Validation loss: 166765920.00 correlation: 0.2487 | |
| Elapse: 544.22s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 075/400 | |
| Train loss: 71357648.00 correlation: 0.2949 | |
| Validation loss: 166766240.00 correlation: 0.2487 | |
| Elapse: 544.05s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 076/400 | |
| Train loss: 71315664.00 correlation: 0.2955 | |
| Validation loss: 166761216.00 correlation: 0.2489 | |
| Elapse: 544.69s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 077/400 | |
| Train loss: 71363472.00 correlation: 0.2948 | |
| Validation loss: 166736352.00 correlation: 0.2489 | |
| Elapse: 544.27s | |
| Epoch 078/400 | |
| Train loss: 71227736.00 correlation: 0.2964 | |
| Validation loss: 166733360.00 correlation: 0.2489 | |
| Elapse: 544.22s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 079/400 | |
| Train loss: 71277464.00 correlation: 0.2955 | |
| Validation loss: 166735664.00 correlation: 0.2488 | |
| Elapse: 543.94s | |
| Epoch 080/400 | |
| Train loss: 71206488.00 correlation: 0.2965 | |
| Validation loss: 166742432.00 correlation: 0.2489 | |
| Elapse: 544.10s | |
| Epoch 081/400 | |
| Train loss: 71163264.00 correlation: 0.2968 | |
| Validation loss: 166738592.00 correlation: 0.2489 | |
| Elapse: 544.08s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 082/400 | |
| Train loss: 71186960.00 correlation: 0.2968 | |
| Validation loss: 166722656.00 correlation: 0.2489 | |
| Elapse: 544.32s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 083/400 | |
| Train loss: 71185200.00 correlation: 0.2968 | |
| Validation loss: 166706704.00 correlation: 0.2489 | |
| Elapse: 543.20s | |
| Epoch 084/400 | |
| Train loss: 71172336.00 correlation: 0.2970 | |
| Validation loss: 166667952.00 correlation: 0.2490 | |
| Elapse: 544.33s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 085/400 | |
| Train loss: 71094784.00 correlation: 0.2977 | |
| Validation loss: 166736224.00 correlation: 0.2490 | |
| Elapse: 544.02s | |
| Epoch 086/400 | |
| Train loss: 71121680.00 correlation: 0.2972 | |
| Validation loss: 166702080.00 correlation: 0.2491 | |
| Elapse: 544.15s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 087/400 | |
| Train loss: 71068680.00 correlation: 0.2980 | |
| Validation loss: 166759776.00 correlation: 0.2488 | |
| Elapse: 544.35s | |
| Epoch 088/400 | |
| Train loss: 71037376.00 correlation: 0.2982 | |
| Validation loss: 166717840.00 correlation: 0.2488 | |
| Elapse: 544.48s | |
| Epoch 089/400 | |
| Train loss: 71044336.00 correlation: 0.2979 | |
| Validation loss: 166688608.00 correlation: 0.2491 | |
| Elapse: 543.63s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 090/400 | |
| Train loss: 71028720.00 correlation: 0.2984 | |
| Validation loss: 166697808.00 correlation: 0.2488 | |
| Elapse: 544.01s | |
| Epoch 091/400 | |
| Train loss: 70956424.00 correlation: 0.2991 | |
| Validation loss: 166683104.00 correlation: 0.2491 | |
| Elapse: 544.36s | |
| Epoch 092/400 | |
| Train loss: 71034488.00 correlation: 0.2982 | |
| Validation loss: 166691024.00 correlation: 0.2491 | |
| Elapse: 544.53s | |
| Epoch 093/400 | |
| Train loss: 71008336.00 correlation: 0.2981 | |
| Validation loss: 166667472.00 correlation: 0.2493 | |
| Elapse: 544.24s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 094/400 | |
| Train loss: 70955584.00 correlation: 0.2990 | |
| Validation loss: 166707392.00 correlation: 0.2487 | |
| Elapse: 543.78s | |
| Epoch 095/400 | |
| Train loss: 70927976.00 correlation: 0.2993 | |
| Validation loss: 166647504.00 correlation: 0.2491 | |
| Elapse: 543.94s | |
| Epoch 096/400 | |
| Train loss: 70877440.00 correlation: 0.2996 | |
| Validation loss: 166701728.00 correlation: 0.2486 | |
| Elapse: 543.45s | |
| Epoch 097/400 | |
| Train loss: 70917616.00 correlation: 0.2991 | |
| Validation loss: 166641920.00 correlation: 0.2492 | |
| Elapse: 544.10s | |
| Epoch 098/400 | |
| Train loss: 70867216.00 correlation: 0.2999 | |
| Validation loss: 166664160.00 correlation: 0.2492 | |
| Elapse: 544.30s | |
| Loaded checkpoint from epoch 93 (correlation: 0.2493). | |
| Reduce learning rate of core to 3.8880e-05 (num. reduce: 1). | |
| Reduce learning rate of readouts to 2.9160e-05 (num. reduce: 1). | |
| Reduce learning rate of shifters to 2.9160e-05 (num. reduce: 1). | |
| Epoch 099/400 | |
| Train loss: 70901904.00 correlation: 0.2993 | |
| Validation loss: 166624560.00 correlation: 0.2493 | |
| Elapse: 544.44s | |
| Epoch 100/400 | |
| Train loss: 70815976.00 correlation: 0.3000 | |
| Validation loss: 166631168.00 correlation: 0.2494 | |
| Elapse: 544.02s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 101/400 | |
| Train loss: 70876152.00 correlation: 0.2993 | |
| Validation loss: 166642400.00 correlation: 0.2492 | |
| Elapse: 544.50s | |
| Epoch 102/400 | |
| Train loss: 70783744.00 correlation: 0.3006 | |
| Validation loss: 166643056.00 correlation: 0.2492 | |
| Elapse: 544.34s | |
| Epoch 103/400 | |
| Train loss: 70822832.00 correlation: 0.3002 | |
| Validation loss: 166621408.00 correlation: 0.2494 | |
| Elapse: 544.59s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 104/400 | |
| Train loss: 70795760.00 correlation: 0.3005 | |
| Validation loss: 166626384.00 correlation: 0.2494 | |
| Elapse: 544.65s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/015/ckpt/model_state.pt. | |
| Epoch 105/400 | |
| Train loss: 70758416.00 correlation: 0.3004 | |
| Validation loss: 166603600.00 correlation: 0.2494 | |
| Elapse: 544.30s | |
| Epoch 106/400 | |
| Train loss: 70836856.00 correlation: 0.2999 | |
| Validation loss: 166631360.00 correlation: 0.2493 | |
| Elapse: 544.28s | |
| Epoch 107/400 | |
| Train loss: 70783560.00 correlation: 0.3006 | |
| Validation loss: 166638304.00 correlation: 0.2493 | |
| Elapse: 544.44s | |
| Epoch 108/400 | |
| Train loss: 70793304.00 correlation: 0.3004 | |
| Validation loss: 166632480.00 correlation: 0.2492 | |
| Elapse: 544.19s | |
| Epoch 109/400 | |
| Train loss: 70730064.00 correlation: 0.3012 | |
| Validation loss: 166617632.00 correlation: 0.2493 | |
| Elapse: 543.89s | |
| Loaded checkpoint from epoch 104 (correlation: 0.2494). | |
| Reduce learning rate of core to 1.1664e-05 (num. reduce: 1). | |
| Reduce learning rate of readouts to 8.7480e-06 (num. reduce: 1). | |
| Reduce learning rate of shifters to 8.7480e-06 (num. reduce: 1). | |
| Epoch 110/400 | |
| Train loss: 70741584.00 correlation: 0.3007 | |
| Validation loss: 166614400.00 correlation: 0.2493 | |
| Elapse: 544.26s | |
| Epoch 111/400 | |
| Train loss: 70763320.00 correlation: 0.3005 | |
| Validation loss: 166633072.00 correlation: 0.2493 | |
| Elapse: 544.57s | |
| Epoch 112/400 | |
| Train loss: 70721984.00 correlation: 0.3011 | |
| Validation loss: 166633552.00 correlation: 0.2493 | |
| Elapse: 544.64s | |
| Epoch 113/400 | |
| Train loss: 70737856.00 correlation: 0.3009 | |
| Validation loss: 166608208.00 correlation: 0.2494 | |
| Elapse: 544.39s | |
| Epoch 114/400 | |
| Train loss: 70798224.00 correlation: 0.3000 | |
| Validation loss: 166619792.00 correlation: 0.2494 | |
| Elapse: 543.95s | |
| Loaded checkpoint from epoch 104 (correlation: 0.2494). | |
| Reduce learning rate of core to 3.4992e-06 (num. reduce: 2). | |
| Reduce learning rate of readouts to 2.6244e-06 (num. reduce: 2). | |
| Reduce learning rate of shifters to 2.6244e-06 (num. reduce: 2). | |
| Epoch 115/400 | |
| Train loss: 70736776.00 correlation: 0.3008 | |
| Validation loss: 166615600.00 correlation: 0.2494 | |
| Elapse: 544.31s | |
| Epoch 116/400 | |
| Train loss: 70760608.00 correlation: 0.3006 | |
| Validation loss: 166622592.00 correlation: 0.2493 | |
| Elapse: 544.61s | |
| Epoch 117/400 | |
| Train loss: 70707912.00 correlation: 0.3011 | |
| Validation loss: 166622416.00 correlation: 0.2494 | |
| Elapse: 544.73s | |
| Epoch 118/400 | |
| Train loss: 70797264.00 correlation: 0.3002 | |
| Validation loss: 166620240.00 correlation: 0.2493 | |
| Elapse: 545.29s | |
| Epoch 119/400 | |
| Train loss: 70782240.00 correlation: 0.3003 | |
| Validation loss: 166620000.00 correlation: 0.2493 | |
| Elapse: 544.33s | |
| Model has not improved after 2 LR reductions. | |
| Loaded checkpoint from epoch 104 (correlation: 0.2494). | |
| ValidationA: 0.2428 B: 0.2746 C: 0.2716 D: 0.2360 E: 0.2351 F: 0.2335 G: 0.2531 H: 0.2331 I: 0.2568 J: 0.2576 average: 0.2494 | |
| Results saved to /home/storage/runs/vivit_ensemble/015. | |