[2024-11-22 07:16:22,127][__main__][INFO] - Using GPU for training. Main device : cuda:0
[2024-11-22 07:16:22,127][__main__][INFO] - Using multiple GPUs : [0, 1, 2, 3]
[2024-11-22 07:16:30,565][__main__][INFO] - PSF shape : torch.Size([1, 380, 507, 3])
[2024-11-22 07:16:30,587][__main__][INFO] - PSF min : 0.0
[2024-11-22 07:16:30,596][__main__][INFO] - PSF max : 0.015610030852258205
[2024-11-22 07:16:30,596][__main__][INFO] - PSF dtype : torch.float32
[2024-11-22 07:16:30,621][__main__][INFO] - PSF norm : 1.000004768371582
[2024-11-22 07:16:30,622][__main__][INFO] - Reconstruction a few images with ADMM...
[2024-11-22 07:16:31,437][__main__][INFO] - Cropped shape :  (262, 215, 3)
[2024-11-22 07:16:33,560][__main__][INFO] - Train test size : 22100
[2024-11-22 07:16:33,561][__main__][INFO] - Test test size : 3900
[2024-11-22 07:16:33,685][__main__][INFO] - Training model with 8092627 parameters
[2024-11-22 07:16:33,686][__main__][INFO] - -- Pre-process model with 3913696 parameters
[2024-11-22 07:16:33,686][__main__][INFO] - -- Post-process model with 4051168 parameters
[2024-11-22 07:16:33,687][__main__][INFO] - -- PSF network model with 127740 parameters
[2024-11-22 07:16:33,687][__main__][INFO] - Setup time : 0.1261594295501709 s
[2024-11-22 07:16:33,687][__main__][INFO] - PSF shape : torch.Size([1, 380, 507, 3])
[2024-11-22 07:16:33,687][__main__][INFO] - Results saved in /root/LenslessPiCam/outputs/2024-11-22/07-16-15
[2024-11-22 07:27:06,280][__main__][INFO] - Epoch 0 with learning rate [0.0001]
[2024-11-22 09:34:19,610][__main__][INFO] - loss : 0.48878798563555753
[2024-11-22 09:44:52,971][__main__][INFO] - Epoch 1 with learning rate [0.0001]
[2024-11-22 11:51:20,822][__main__][INFO] - loss : 0.3643008540820223
[2024-11-22 12:01:54,206][__main__][INFO] - Epoch 2 with learning rate [0.0001]
[2024-11-22 14:08:25,175][__main__][INFO] - loss : 0.3395183362383629
[2024-11-22 14:18:55,174][__main__][INFO] - Epoch 3 with learning rate [0.0001]
[2024-11-22 16:25:25,100][__main__][INFO] - loss : 0.3232957250176519
[2024-11-22 16:36:00,017][__main__][INFO] - Epoch 4 with learning rate [0.0001]
[2024-11-22 18:42:32,408][__main__][INFO] - loss : 0.31140302979568435
[2024-11-22 18:53:04,220][__main__][INFO] - Epoch 5 with learning rate [0.0001]
[2024-11-22 20:59:39,401][__main__][INFO] - loss : 0.3028789611234926
[2024-11-22 21:10:06,879][__main__][INFO] - Epoch 6 with learning rate [0.0001]
[2024-11-22 23:16:37,192][__main__][INFO] - loss : 0.2965210252865411
[2024-11-22 23:27:12,362][__main__][INFO] - Epoch 7 with learning rate [0.0001]
[2024-11-23 01:33:44,819][__main__][INFO] - loss : 0.29124890148100324
[2024-11-23 01:44:22,279][__main__][INFO] - Epoch 8 with learning rate [0.0001]
[2024-11-23 03:50:39,690][__main__][INFO] - loss : 0.2866853000972063
[2024-11-23 04:01:13,846][__main__][INFO] - Epoch 9 with learning rate [0.0001]
[2024-11-23 06:07:39,147][__main__][INFO] - loss : 0.2828487040311496
[2024-11-23 06:18:10,284][__main__][INFO] - Epoch 10 with learning rate [0.0001]
[2024-11-23 08:24:32,395][__main__][INFO] - loss : 0.2795618656749642
[2024-11-23 08:34:57,917][__main__][INFO] - Epoch 11 with learning rate [0.0001]
[2024-11-23 10:41:15,280][__main__][INFO] - loss : 0.27629247227405596
[2024-11-23 10:51:49,516][__main__][INFO] - Epoch 12 with learning rate [0.0001]
[2024-11-23 12:58:13,018][__main__][INFO] - loss : 0.2734775769063253
[2024-11-23 13:08:44,125][__main__][INFO] - Epoch 13 with learning rate [0.0001]
[2024-11-23 15:15:06,712][__main__][INFO] - loss : 0.27096088547242747
[2024-11-23 15:25:41,849][__main__][INFO] - Epoch 14 with learning rate [0.0001]
[2024-11-23 17:31:57,147][__main__][INFO] - loss : 0.2685547446907917
[2024-11-23 17:42:30,547][__main__][INFO] - Epoch 15 with learning rate [0.0001]
[2024-11-23 19:48:56,991][__main__][INFO] - loss : 0.26624881224125213
[2024-11-23 19:59:25,581][__main__][INFO] - Epoch 16 with learning rate [0.0001]
[2024-11-23 22:05:56,406][__main__][INFO] - loss : 0.2644316919788511
[2024-11-23 22:16:30,621][__main__][INFO] - Epoch 17 with learning rate [0.0001]
[2024-11-24 00:22:56,594][__main__][INFO] - loss : 0.2625052805373038
[2024-11-24 00:33:27,767][__main__][INFO] - Epoch 18 with learning rate [0.0001]
[2024-11-24 02:39:50,086][__main__][INFO] - loss : 0.26052219921647135
[2024-11-24 02:50:17,976][__main__][INFO] - Epoch 19 with learning rate [0.0001]
[2024-11-24 04:56:31,927][__main__][INFO] - loss : 0.2588567895452356
[2024-11-24 05:06:59,427][__main__][INFO] - Epoch 20 with learning rate [0.0001]
[2024-11-24 07:13:26,694][__main__][INFO] - loss : 0.2570547045544824
[2024-11-24 07:24:00,803][__main__][INFO] - Epoch 21 with learning rate [0.0001]
[2024-11-24 09:30:33,357][__main__][INFO] - loss : 0.2555194438097166
[2024-11-24 09:41:06,281][__main__][INFO] - Epoch 22 with learning rate [0.0001]
[2024-11-24 11:47:16,365][__main__][INFO] - loss : 0.2541234564079951
[2024-11-24 11:57:50,898][__main__][INFO] - Epoch 23 with learning rate [0.0001]
[2024-11-24 14:04:08,710][__main__][INFO] - loss : 0.2527223272857609
[2024-11-24 14:14:38,347][__main__][INFO] - Epoch 24 with learning rate [0.0001]
[2024-11-24 16:20:34,843][__main__][INFO] - loss : 0.2512848522841128
[2024-11-24 16:31:01,421][__main__][INFO] - Train time [hour] : 57.24062335365348 h
[2024-11-24 16:31:01,422][__main__][INFO] - Results saved in /root/LenslessPiCam/outputs/2024-11-22/07-16-15