| Use bfloat16 for core module. | |
| Use parallel attention and MLP in ViViT. | |
| Epoch 001/400 | |
| Train loss: 121388096.00 correlation: 0.0109 | |
| Validation loss: 200734000.00 correlation: 0.0221 | |
| Elapse: 522.91s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 002/400 | |
| Train loss: 98318960.00 correlation: 0.0308 | |
| Validation loss: 199955104.00 correlation: 0.0339 | |
| Elapse: 525.64s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 003/400 | |
| Train loss: 97097640.00 correlation: 0.0413 | |
| Validation loss: 199270752.00 correlation: 0.0396 | |
| Elapse: 531.95s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 004/400 | |
| Train loss: 96457752.00 correlation: 0.0479 | |
| Validation loss: 198723456.00 correlation: 0.0421 | |
| Elapse: 536.86s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 005/400 | |
| Train loss: 95920056.00 correlation: 0.0541 | |
| Validation loss: 197835248.00 correlation: 0.0461 | |
| Elapse: 538.89s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 006/400 | |
| Train loss: 95131240.00 correlation: 0.0621 | |
| Validation loss: 196873360.00 correlation: 0.0519 | |
| Elapse: 542.34s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 007/400 | |
| Train loss: 94226616.00 correlation: 0.0710 | |
| Validation loss: 195726576.00 correlation: 0.0578 | |
| Elapse: 541.43s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 008/400 | |
| Train loss: 93506680.00 correlation: 0.0783 | |
| Validation loss: 194520016.00 correlation: 0.0654 | |
| Elapse: 541.56s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 009/400 | |
| Train loss: 92727504.00 correlation: 0.0871 | |
| Validation loss: 193465776.00 correlation: 0.0716 | |
| Elapse: 543.31s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 010/400 | |
| Train loss: 91843728.00 correlation: 0.0966 | |
| Validation loss: 192176672.00 correlation: 0.0791 | |
| Elapse: 541.68s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 011/400 | |
| Train loss: 90936304.00 correlation: 0.1063 | |
| Validation loss: 190695968.00 correlation: 0.0928 | |
| Elapse: 540.29s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 012/400 | |
| Train loss: 90051912.00 correlation: 0.1153 | |
| Validation loss: 189604784.00 correlation: 0.1011 | |
| Elapse: 542.29s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 013/400 | |
| Train loss: 89180568.00 correlation: 0.1245 | |
| Validation loss: 188031648.00 correlation: 0.1112 | |
| Elapse: 542.62s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 014/400 | |
| Train loss: 88307792.00 correlation: 0.1337 | |
| Validation loss: 186159440.00 correlation: 0.1226 | |
| Elapse: 540.81s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 015/400 | |
| Train loss: 87278960.00 correlation: 0.1436 | |
| Validation loss: 184683440.00 correlation: 0.1313 | |
| Elapse: 543.63s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 016/400 | |
| Train loss: 86609264.00 correlation: 0.1505 | |
| Validation loss: 183679264.00 correlation: 0.1374 | |
| Elapse: 544.07s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 017/400 | |
| Train loss: 85933704.00 correlation: 0.1573 | |
| Validation loss: 182858960.00 correlation: 0.1427 | |
| Elapse: 543.60s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 018/400 | |
| Train loss: 85316224.00 correlation: 0.1634 | |
| Validation loss: 181758608.00 correlation: 0.1502 | |
| Elapse: 543.09s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 019/400 | |
| Train loss: 84800144.00 correlation: 0.1686 | |
| Validation loss: 181267248.00 correlation: 0.1538 | |
| Elapse: 545.10s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 020/400 | |
| Train loss: 84276656.00 correlation: 0.1737 | |
| Validation loss: 180435344.00 correlation: 0.1578 | |
| Elapse: 545.34s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 021/400 | |
| Train loss: 83717728.00 correlation: 0.1793 | |
| Validation loss: 179796752.00 correlation: 0.1639 | |
| Elapse: 545.93s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 022/400 | |
| Train loss: 83306056.00 correlation: 0.1836 | |
| Validation loss: 178887952.00 correlation: 0.1689 | |
| Elapse: 545.45s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 023/400 | |
| Train loss: 82787680.00 correlation: 0.1887 | |
| Validation loss: 177999440.00 correlation: 0.1749 | |
| Elapse: 545.47s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 024/400 | |
| Train loss: 82335960.00 correlation: 0.1933 | |
| Validation loss: 177650256.00 correlation: 0.1774 | |
| Elapse: 545.75s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 025/400 | |
| Train loss: 82013392.00 correlation: 0.1962 | |
| Validation loss: 177572752.00 correlation: 0.1786 | |
| Elapse: 545.69s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 026/400 | |
| Train loss: 81675472.00 correlation: 0.1996 | |
| Validation loss: 176845120.00 correlation: 0.1841 | |
| Elapse: 545.58s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 027/400 | |
| Train loss: 81393008.00 correlation: 0.2025 | |
| Validation loss: 176056032.00 correlation: 0.1886 | |
| Elapse: 545.76s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 028/400 | |
| Train loss: 80972248.00 correlation: 0.2065 | |
| Validation loss: 175799008.00 correlation: 0.1898 | |
| Elapse: 545.76s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 029/400 | |
| Train loss: 80749744.00 correlation: 0.2086 | |
| Validation loss: 175376608.00 correlation: 0.1918 | |
| Elapse: 546.06s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 030/400 | |
| Train loss: 80453840.00 correlation: 0.2117 | |
| Validation loss: 174830400.00 correlation: 0.1967 | |
| Elapse: 546.10s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 031/400 | |
| Train loss: 80186704.00 correlation: 0.2145 | |
| Validation loss: 174322544.00 correlation: 0.1984 | |
| Elapse: 545.45s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 032/400 | |
| Train loss: 79856432.00 correlation: 0.2176 | |
| Validation loss: 174104672.00 correlation: 0.2003 | |
| Elapse: 545.77s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 033/400 | |
| Train loss: 79589576.00 correlation: 0.2200 | |
| Validation loss: 173607248.00 correlation: 0.2031 | |
| Elapse: 545.80s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 034/400 | |
| Train loss: 79380120.00 correlation: 0.2221 | |
| Validation loss: 173319472.00 correlation: 0.2055 | |
| Elapse: 545.70s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 035/400 | |
| Train loss: 79158640.00 correlation: 0.2243 | |
| Validation loss: 172901472.00 correlation: 0.2089 | |
| Elapse: 545.61s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 036/400 | |
| Train loss: 78927200.00 correlation: 0.2266 | |
| Validation loss: 172636560.00 correlation: 0.2097 | |
| Elapse: 546.53s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 037/400 | |
| Train loss: 78682176.00 correlation: 0.2288 | |
| Validation loss: 172407184.00 correlation: 0.2118 | |
| Elapse: 545.83s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 038/400 | |
| Train loss: 78582504.00 correlation: 0.2299 | |
| Validation loss: 172238496.00 correlation: 0.2137 | |
| Elapse: 542.71s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 039/400 | |
| Train loss: 78397632.00 correlation: 0.2314 | |
| Validation loss: 171969648.00 correlation: 0.2147 | |
| Elapse: 545.53s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 040/400 | |
| Train loss: 78234600.00 correlation: 0.2330 | |
| Validation loss: 171758000.00 correlation: 0.2170 | |
| Elapse: 545.77s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 041/400 | |
| Train loss: 78047656.00 correlation: 0.2349 | |
| Validation loss: 171651168.00 correlation: 0.2175 | |
| Elapse: 545.10s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 042/400 | |
| Train loss: 77868808.00 correlation: 0.2362 | |
| Validation loss: 171537312.00 correlation: 0.2173 | |
| Elapse: 545.58s | |
| Epoch 043/400 | |
| Train loss: 77839128.00 correlation: 0.2370 | |
| Validation loss: 171300016.00 correlation: 0.2185 | |
| Elapse: 546.30s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 044/400 | |
| Train loss: 77629288.00 correlation: 0.2390 | |
| Validation loss: 171374944.00 correlation: 0.2191 | |
| Elapse: 546.42s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 045/400 | |
| Train loss: 77550776.00 correlation: 0.2396 | |
| Validation loss: 171207344.00 correlation: 0.2194 | |
| Elapse: 545.96s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 046/400 | |
| Train loss: 77437640.00 correlation: 0.2408 | |
| Validation loss: 171176240.00 correlation: 0.2198 | |
| Elapse: 546.38s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 047/400 | |
| Train loss: 77397712.00 correlation: 0.2412 | |
| Validation loss: 170808800.00 correlation: 0.2224 | |
| Elapse: 545.84s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 048/400 | |
| Train loss: 77307432.00 correlation: 0.2422 | |
| Validation loss: 170735520.00 correlation: 0.2233 | |
| Elapse: 546.33s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 049/400 | |
| Train loss: 77119432.00 correlation: 0.2439 | |
| Validation loss: 170442368.00 correlation: 0.2245 | |
| Elapse: 545.45s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 050/400 | |
| Train loss: 76988832.00 correlation: 0.2449 | |
| Validation loss: 170672960.00 correlation: 0.2232 | |
| Elapse: 544.34s | |
| Epoch 051/400 | |
| Train loss: 76992464.00 correlation: 0.2450 | |
| Validation loss: 170562112.00 correlation: 0.2244 | |
| Elapse: 545.56s | |
| Epoch 052/400 | |
| Train loss: 76925112.00 correlation: 0.2452 | |
| Validation loss: 170404016.00 correlation: 0.2248 | |
| Elapse: 545.72s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 053/400 | |
| Train loss: 76844464.00 correlation: 0.2460 | |
| Validation loss: 170137952.00 correlation: 0.2261 | |
| Elapse: 545.92s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 054/400 | |
| Train loss: 76812920.00 correlation: 0.2469 | |
| Validation loss: 170023120.00 correlation: 0.2259 | |
| Elapse: 545.56s | |
| Epoch 055/400 | |
| Train loss: 76702536.00 correlation: 0.2475 | |
| Validation loss: 170054384.00 correlation: 0.2263 | |
| Elapse: 544.95s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 056/400 | |
| Train loss: 76612752.00 correlation: 0.2485 | |
| Validation loss: 170032864.00 correlation: 0.2272 | |
| Elapse: 545.49s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 057/400 | |
| Train loss: 76617696.00 correlation: 0.2485 | |
| Validation loss: 169630816.00 correlation: 0.2293 | |
| Elapse: 545.90s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 058/400 | |
| Train loss: 76465880.00 correlation: 0.2499 | |
| Validation loss: 169825600.00 correlation: 0.2283 | |
| Elapse: 545.37s | |
| Epoch 059/400 | |
| Train loss: 76456912.00 correlation: 0.2499 | |
| Validation loss: 169766688.00 correlation: 0.2284 | |
| Elapse: 545.61s | |
| Epoch 060/400 | |
| Train loss: 76426872.00 correlation: 0.2501 | |
| Validation loss: 169651072.00 correlation: 0.2298 | |
| Elapse: 545.66s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 061/400 | |
| Train loss: 76272176.00 correlation: 0.2515 | |
| Validation loss: 169694896.00 correlation: 0.2290 | |
| Elapse: 544.26s | |
| Epoch 062/400 | |
| Train loss: 76381040.00 correlation: 0.2506 | |
| Validation loss: 169396208.00 correlation: 0.2316 | |
| Elapse: 544.91s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 063/400 | |
| Train loss: 76206744.00 correlation: 0.2523 | |
| Validation loss: 169683888.00 correlation: 0.2297 | |
| Elapse: 544.69s | |
| Epoch 064/400 | |
| Train loss: 76167792.00 correlation: 0.2527 | |
| Validation loss: 169194928.00 correlation: 0.2329 | |
| Elapse: 545.55s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 065/400 | |
| Train loss: 76127384.00 correlation: 0.2528 | |
| Validation loss: 169763808.00 correlation: 0.2287 | |
| Elapse: 545.54s | |
| Epoch 066/400 | |
| Train loss: 76104336.00 correlation: 0.2531 | |
| Validation loss: 169165280.00 correlation: 0.2327 | |
| Elapse: 545.65s | |
| Epoch 067/400 | |
| Train loss: 76021512.00 correlation: 0.2539 | |
| Validation loss: 169219872.00 correlation: 0.2317 | |
| Elapse: 545.30s | |
| Epoch 068/400 | |
| Train loss: 76148160.00 correlation: 0.2528 | |
| Validation loss: 169344656.00 correlation: 0.2314 | |
| Elapse: 545.59s | |
| Epoch 069/400 | |
| Train loss: 75939984.00 correlation: 0.2545 | |
| Validation loss: 169043376.00 correlation: 0.2339 | |
| Elapse: 545.18s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 070/400 | |
| Train loss: 75822872.00 correlation: 0.2557 | |
| Validation loss: 169167760.00 correlation: 0.2328 | |
| Elapse: 544.88s | |
| Epoch 071/400 | |
| Train loss: 75873072.00 correlation: 0.2551 | |
| Validation loss: 169123712.00 correlation: 0.2336 | |
| Elapse: 545.60s | |
| Epoch 072/400 | |
| Train loss: 75867984.00 correlation: 0.2552 | |
| Validation loss: 168947232.00 correlation: 0.2344 | |
| Elapse: 545.61s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 073/400 | |
| Train loss: 75840912.00 correlation: 0.2555 | |
| Validation loss: 169026784.00 correlation: 0.2348 | |
| Elapse: 545.60s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 074/400 | |
| Train loss: 75855384.00 correlation: 0.2555 | |
| Validation loss: 168842592.00 correlation: 0.2341 | |
| Elapse: 545.99s | |
| Epoch 075/400 | |
| Train loss: 75737744.00 correlation: 0.2565 | |
| Validation loss: 168971312.00 correlation: 0.2336 | |
| Elapse: 545.40s | |
| Epoch 076/400 | |
| Train loss: 75735656.00 correlation: 0.2565 | |
| Validation loss: 168829920.00 correlation: 0.2338 | |
| Elapse: 546.00s | |
| Epoch 077/400 | |
| Train loss: 75607888.00 correlation: 0.2576 | |
| Validation loss: 168935968.00 correlation: 0.2349 | |
| Elapse: 545.47s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 078/400 | |
| Train loss: 75679128.00 correlation: 0.2571 | |
| Validation loss: 168987280.00 correlation: 0.2328 | |
| Elapse: 545.11s | |
| Epoch 079/400 | |
| Train loss: 75621152.00 correlation: 0.2574 | |
| Validation loss: 168822512.00 correlation: 0.2347 | |
| Elapse: 545.59s | |
| Epoch 080/400 | |
| Train loss: 75483728.00 correlation: 0.2590 | |
| Validation loss: 168774416.00 correlation: 0.2347 | |
| Elapse: 545.47s | |
| Epoch 081/400 | |
| Train loss: 75531304.00 correlation: 0.2586 | |
| Validation loss: 169126304.00 correlation: 0.2331 | |
| Elapse: 545.29s | |
| Epoch 082/400 | |
| Train loss: 75558736.00 correlation: 0.2583 | |
| Validation loss: 168670160.00 correlation: 0.2354 | |
| Elapse: 545.72s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 083/400 | |
| Train loss: 75590176.00 correlation: 0.2579 | |
| Validation loss: 169091472.00 correlation: 0.2330 | |
| Elapse: 545.76s | |
| Epoch 084/400 | |
| Train loss: 75406416.00 correlation: 0.2595 | |
| Validation loss: 169084192.00 correlation: 0.2329 | |
| Elapse: 545.78s | |
| Epoch 085/400 | |
| Train loss: 75376912.00 correlation: 0.2599 | |
| Validation loss: 168861952.00 correlation: 0.2341 | |
| Elapse: 545.89s | |
| Epoch 086/400 | |
| Train loss: 75349048.00 correlation: 0.2604 | |
| Validation loss: 168882304.00 correlation: 0.2349 | |
| Elapse: 545.73s | |
| Epoch 087/400 | |
| Train loss: 75302984.00 correlation: 0.2607 | |
| Validation loss: 169006784.00 correlation: 0.2333 | |
| Elapse: 545.26s | |
| Loaded checkpoint from epoch 82 (correlation: 0.2354). | |
| Reduce learning rate of core to 1.4400e-03 (num. reduce: 1). | |
| Reduce learning rate of readouts to 1.0800e-03 (num. reduce: 1). | |
| Reduce learning rate of shifters to 1.0800e-03 (num. reduce: 1). | |
| Epoch 088/400 | |
| Train loss: 73472912.00 correlation: 0.2760 | |
| Validation loss: 166967968.00 correlation: 0.2466 | |
| Elapse: 546.46s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 089/400 | |
| Train loss: 72834528.00 correlation: 0.2820 | |
| Validation loss: 166834160.00 correlation: 0.2485 | |
| Elapse: 546.50s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 090/400 | |
| Train loss: 72607408.00 correlation: 0.2844 | |
| Validation loss: 166764992.00 correlation: 0.2490 | |
| Elapse: 546.18s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 091/400 | |
| Train loss: 72571872.00 correlation: 0.2846 | |
| Validation loss: 166824368.00 correlation: 0.2476 | |
| Elapse: 546.25s | |
| Epoch 092/400 | |
| Train loss: 72491600.00 correlation: 0.2854 | |
| Validation loss: 166791872.00 correlation: 0.2474 | |
| Elapse: 545.90s | |
| Epoch 093/400 | |
| Train loss: 72479560.00 correlation: 0.2856 | |
| Validation loss: 166900064.00 correlation: 0.2473 | |
| Elapse: 546.88s | |
| Epoch 094/400 | |
| Train loss: 72391640.00 correlation: 0.2867 | |
| Validation loss: 166895872.00 correlation: 0.2471 | |
| Elapse: 546.67s | |
| Epoch 095/400 | |
| Train loss: 72406800.00 correlation: 0.2865 | |
| Validation loss: 166785824.00 correlation: 0.2477 | |
| Elapse: 546.76s | |
| Loaded checkpoint from epoch 90 (correlation: 0.2490). | |
| Reduce learning rate of core to 4.3200e-04 (num. reduce: 1). | |
| Reduce learning rate of readouts to 3.2400e-04 (num. reduce: 1). | |
| Reduce learning rate of shifters to 3.2400e-04 (num. reduce: 1). | |
| Epoch 096/400 | |
| Train loss: 71820392.00 correlation: 0.2914 | |
| Validation loss: 166357664.00 correlation: 0.2510 | |
| Elapse: 546.29s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 097/400 | |
| Train loss: 71646128.00 correlation: 0.2926 | |
| Validation loss: 166358480.00 correlation: 0.2512 | |
| Elapse: 546.63s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 098/400 | |
| Train loss: 71469944.00 correlation: 0.2943 | |
| Validation loss: 166261168.00 correlation: 0.2520 | |
| Elapse: 547.33s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 099/400 | |
| Train loss: 71457032.00 correlation: 0.2945 | |
| Validation loss: 166242256.00 correlation: 0.2517 | |
| Elapse: 547.22s | |
| Epoch 100/400 | |
| Train loss: 71381520.00 correlation: 0.2953 | |
| Validation loss: 166269072.00 correlation: 0.2516 | |
| Elapse: 546.93s | |
| Epoch 101/400 | |
| Train loss: 71365904.00 correlation: 0.2954 | |
| Validation loss: 166306016.00 correlation: 0.2512 | |
| Elapse: 547.14s | |
| Epoch 102/400 | |
| Train loss: 71313408.00 correlation: 0.2956 | |
| Validation loss: 166202720.00 correlation: 0.2520 | |
| Elapse: 547.18s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 103/400 | |
| Train loss: 71246520.00 correlation: 0.2963 | |
| Validation loss: 166201664.00 correlation: 0.2519 | |
| Elapse: 546.59s | |
| Epoch 104/400 | |
| Train loss: 71237544.00 correlation: 0.2967 | |
| Validation loss: 166225280.00 correlation: 0.2516 | |
| Elapse: 546.47s | |
| Epoch 105/400 | |
| Train loss: 71175248.00 correlation: 0.2973 | |
| Validation loss: 166234304.00 correlation: 0.2515 | |
| Elapse: 546.73s | |
| Epoch 106/400 | |
| Train loss: 71139096.00 correlation: 0.2976 | |
| Validation loss: 166179840.00 correlation: 0.2518 | |
| Elapse: 547.27s | |
| Epoch 107/400 | |
| Train loss: 71152664.00 correlation: 0.2972 | |
| Validation loss: 166277328.00 correlation: 0.2509 | |
| Elapse: 547.28s | |
| Loaded checkpoint from epoch 102 (correlation: 0.2520). | |
| Reduce learning rate of core to 1.2960e-04 (num. reduce: 1). | |
| Reduce learning rate of readouts to 9.7200e-05 (num. reduce: 1). | |
| Reduce learning rate of shifters to 9.7200e-05 (num. reduce: 1). | |
| Epoch 108/400 | |
| Train loss: 70975272.00 correlation: 0.2990 | |
| Validation loss: 166102416.00 correlation: 0.2526 | |
| Elapse: 547.38s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 109/400 | |
| Train loss: 70927400.00 correlation: 0.2990 | |
| Validation loss: 166090848.00 correlation: 0.2526 | |
| Elapse: 547.10s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 110/400 | |
| Train loss: 70928176.00 correlation: 0.2993 | |
| Validation loss: 166087520.00 correlation: 0.2526 | |
| Elapse: 547.24s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 111/400 | |
| Train loss: 70948912.00 correlation: 0.2989 | |
| Validation loss: 166067424.00 correlation: 0.2526 | |
| Elapse: 547.30s | |
| Epoch 112/400 | |
| Train loss: 70843856.00 correlation: 0.2999 | |
| Validation loss: 166089904.00 correlation: 0.2527 | |
| Elapse: 547.44s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 113/400 | |
| Train loss: 70930864.00 correlation: 0.2991 | |
| Validation loss: 166048192.00 correlation: 0.2527 | |
| Elapse: 547.22s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 114/400 | |
| Train loss: 70888224.00 correlation: 0.2998 | |
| Validation loss: 166076336.00 correlation: 0.2525 | |
| Elapse: 547.07s | |
| Epoch 115/400 | |
| Train loss: 70807400.00 correlation: 0.3003 | |
| Validation loss: 166078992.00 correlation: 0.2524 | |
| Elapse: 547.34s | |
| Epoch 116/400 | |
| Train loss: 70784824.00 correlation: 0.3006 | |
| Validation loss: 166036512.00 correlation: 0.2528 | |
| Elapse: 547.33s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 117/400 | |
| Train loss: 70760888.00 correlation: 0.3007 | |
| Validation loss: 166109792.00 correlation: 0.2523 | |
| Elapse: 547.36s | |
| Epoch 118/400 | |
| Train loss: 70793224.00 correlation: 0.3003 | |
| Validation loss: 166033088.00 correlation: 0.2527 | |
| Elapse: 547.66s | |
| Epoch 119/400 | |
| Train loss: 70801416.00 correlation: 0.3002 | |
| Validation loss: 166062960.00 correlation: 0.2529 | |
| Elapse: 547.68s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 120/400 | |
| Train loss: 70748944.00 correlation: 0.3009 | |
| Validation loss: 166063152.00 correlation: 0.2524 | |
| Elapse: 547.18s | |
| Epoch 121/400 | |
| Train loss: 70736464.00 correlation: 0.3010 | |
| Validation loss: 166058976.00 correlation: 0.2528 | |
| Elapse: 547.53s | |
| Epoch 122/400 | |
| Train loss: 70653008.00 correlation: 0.3019 | |
| Validation loss: 166041920.00 correlation: 0.2527 | |
| Elapse: 547.68s | |
| Epoch 123/400 | |
| Train loss: 70723656.00 correlation: 0.3011 | |
| Validation loss: 166053344.00 correlation: 0.2525 | |
| Elapse: 547.37s | |
| Epoch 124/400 | |
| Train loss: 70689936.00 correlation: 0.3016 | |
| Validation loss: 166053472.00 correlation: 0.2526 | |
| Elapse: 547.93s | |
| Loaded checkpoint from epoch 119 (correlation: 0.2529). | |
| Reduce learning rate of core to 3.8880e-05 (num. reduce: 1). | |
| Reduce learning rate of readouts to 2.9160e-05 (num. reduce: 1). | |
| Reduce learning rate of shifters to 2.9160e-05 (num. reduce: 1). | |
| Epoch 125/400 | |
| Train loss: 70604440.00 correlation: 0.3023 | |
| Validation loss: 166025312.00 correlation: 0.2529 | |
| Elapse: 547.63s | |
| Epoch 126/400 | |
| Train loss: 70608160.00 correlation: 0.3021 | |
| Validation loss: 166010512.00 correlation: 0.2530 | |
| Elapse: 547.36s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 127/400 | |
| Train loss: 70626448.00 correlation: 0.3020 | |
| Validation loss: 166023648.00 correlation: 0.2528 | |
| Elapse: 547.13s | |
| Epoch 128/400 | |
| Train loss: 70612096.00 correlation: 0.3018 | |
| Validation loss: 166023216.00 correlation: 0.2529 | |
| Elapse: 546.99s | |
| Epoch 129/400 | |
| Train loss: 70635640.00 correlation: 0.3020 | |
| Validation loss: 166036208.00 correlation: 0.2528 | |
| Elapse: 547.23s | |
| Epoch 130/400 | |
| Train loss: 70666920.00 correlation: 0.3015 | |
| Validation loss: 166035808.00 correlation: 0.2527 | |
| Elapse: 546.99s | |
| Epoch 131/400 | |
| Train loss: 70564592.00 correlation: 0.3028 | |
| Validation loss: 166005856.00 correlation: 0.2528 | |
| Elapse: 547.30s | |
| Loaded checkpoint from epoch 126 (correlation: 0.2530). | |
| Reduce learning rate of core to 1.1664e-05 (num. reduce: 1). | |
| Reduce learning rate of readouts to 8.7480e-06 (num. reduce: 1). | |
| Reduce learning rate of shifters to 8.7480e-06 (num. reduce: 1). | |
| Epoch 132/400 | |
| Train loss: 70603848.00 correlation: 0.3022 | |
| Validation loss: 166004944.00 correlation: 0.2530 | |
| Elapse: 547.32s | |
| Checkpoint saved to /home/storage/runs/vivit_ensemble/009/ckpt/model_state.pt. | |
| Epoch 133/400 | |
| Train loss: 70533872.00 correlation: 0.3031 | |
| Validation loss: 166019360.00 correlation: 0.2529 | |
| Elapse: 547.50s | |
| Epoch 134/400 | |
| Train loss: 70635248.00 correlation: 0.3019 | |
| Validation loss: 166021888.00 correlation: 0.2529 | |
| Elapse: 547.34s | |
| Epoch 135/400 | |
| Train loss: 70620176.00 correlation: 0.3020 | |
| Validation loss: 166009888.00 correlation: 0.2529 | |
| Elapse: 546.38s | |
| Epoch 136/400 | |
| Train loss: 70572368.00 correlation: 0.3023 | |
| Validation loss: 166012144.00 correlation: 0.2529 | |
| Elapse: 547.10s | |
| Epoch 137/400 | |
| Train loss: 70658640.00 correlation: 0.3016 | |
| Validation loss: 166018080.00 correlation: 0.2529 | |
| Elapse: 546.90s | |
| Loaded checkpoint from epoch 132 (correlation: 0.2530). | |
| Reduce learning rate of core to 3.4992e-06 (num. reduce: 1). | |
| Reduce learning rate of readouts to 2.6244e-06 (num. reduce: 1). | |
| Reduce learning rate of shifters to 2.6244e-06 (num. reduce: 1). | |
| Epoch 138/400 | |
| Train loss: 70545520.00 correlation: 0.3026 | |
| Validation loss: 166011936.00 correlation: 0.2530 | |
| Elapse: 546.72s | |
| Epoch 139/400 | |
| Train loss: 70622800.00 correlation: 0.3018 | |
| Validation loss: 166013088.00 correlation: 0.2529 | |
| Elapse: 547.01s | |
| Epoch 140/400 | |
| Train loss: 70572936.00 correlation: 0.3025 | |
| Validation loss: 166013696.00 correlation: 0.2529 | |
| Elapse: 547.38s | |
| Epoch 141/400 | |
| Train loss: 70624016.00 correlation: 0.3020 | |
| Validation loss: 166012496.00 correlation: 0.2530 | |
| Elapse: 546.98s | |
| Epoch 142/400 | |
| Train loss: 70651392.00 correlation: 0.3018 | |
| Validation loss: 166017120.00 correlation: 0.2529 | |
| Elapse: 547.57s | |
| Loaded checkpoint from epoch 132 (correlation: 0.2530). | |
| Reduce learning rate of core to 1.0498e-06 (num. reduce: 2). | |
| Reduce learning rate of readouts to 7.8732e-07 (num. reduce: 2). | |
| Reduce learning rate of shifters to 7.8732e-07 (num. reduce: 2). | |
| Epoch 143/400 | |
| Train loss: 70616608.00 correlation: 0.3018 | |
| Validation loss: 166009488.00 correlation: 0.2530 | |
| Elapse: 547.46s | |
| Epoch 144/400 | |
| Train loss: 70557224.00 correlation: 0.3027 | |
| Validation loss: 166010240.00 correlation: 0.2530 | |
| Elapse: 547.42s | |
| Epoch 145/400 | |
| Train loss: 70617064.00 correlation: 0.3020 | |
| Validation loss: 166011824.00 correlation: 0.2530 | |
| Elapse: 547.48s | |
| Epoch 146/400 | |
| Train loss: 70601720.00 correlation: 0.3022 | |
| Validation loss: 166010880.00 correlation: 0.2530 | |
| Elapse: 547.26s | |
| Epoch 147/400 | |
| Train loss: 70616768.00 correlation: 0.3021 | |
| Validation loss: 166012944.00 correlation: 0.2530 | |
| Elapse: 547.43s | |
| Model has not improved after 2 LR reductions. | |
| Loaded checkpoint from epoch 132 (correlation: 0.2530). | |
| ValidationA: 0.2508 B: 0.2804 C: 0.2750 D: 0.2393 E: 0.2390 F: 0.2346 G: 0.2573 H: 0.2379 I: 0.2555 J: 0.2599 average: 0.2530 | |
| Results saved to /home/storage/runs/vivit_ensemble/009. | |