[2021-06-11 01:06:26,576][root][INFO] - train: phase: train is_train: true exp_name: kitti_ours_randomcrop log_dir: /rhome/jhou/data/checkpoint/pri3d_now/finetune/kitti_ours_randomcrop batch_size: 8 report_batch: 10 report_epoch: 5 workers: 4 write_result: false optimizer: optimizer: SGD lr: 0.01 sgd_momentum: 0.9 sgd_dampening: 0.1 adam_beta1: 0.9 adam_beta2: 0.999 weight_decay: 0.0 bn_momentum: 0.02 scheduler: scheduler: PolyLR max_epochs: 80 decay_epochs: - 80 - 120 - 160 lr_decay: 0.1 step_size: 80 poly_power: 0.9 distributed: num_gpus: 1 finetune: model: Semantic2D weights: /rhome/jhou/data/checkpoint/pri3d_now/pretrain/kitti/lr001_bound_long/0/models/checkpoint10.pth pretrained: imagenet backbone: Res50UNet dataset: name: KITTI size: - 240 - 320 augment: false chunk: false npoints: 80000 resize: false colorjitter: false random_crop: true [2021-06-11 01:06:31,182][root][INFO] - Loss weights from content: tensor([0.0000, 0.9990, 0.9990, 0.9990, 0.9990, 0.9990, 0.9990, 0.9990, 0.9990, 0.9990, 0.9990, 0.9990, 0.9990, 0.9990, 0.9990, 0.9990, 0.9990, 0.9990, 0.9990, 0.9990]) [2021-06-11 01:06:33,267][root][INFO] - Ignoring class 0 in IoU evaluation [2021-06-11 01:06:34,740][root][INFO] - ===> Number of trainable parameters: Semantic2D: 175319892 [2021-06-11 01:06:34,740][root][INFO] - ===> Loading weights: /rhome/jhou/data/checkpoint/pri3d_now/pretrain/kitti/lr001_bound_long/0/models/checkpoint10.pth [2021-06-11 01:06:36,174][root][INFO] - ===> Loaded weights: /rhome/jhou/data/checkpoint/pri3d_now/pretrain/kitti/lr001_bound_long/0/models/checkpoint10.pth [2021-06-11 01:06:36,371][root][INFO] - Training in device: cuda [2021-06-11 01:06:38,949][root][INFO] - Lr: 9.994e-03 | Epoch: [0][0/20] | Time 2.152 | Data 1.117 | Loss 3.2944 | IoU 0.009 [2021-06-11 01:06:44,924][root][INFO] - Lr: 9.938e-03 | Epoch: [0][10/20] | Time 0.769 | Data 0.180 | Loss 2.2270 | IoU 0.047 [2021-06-11 01:06:50,961][root][INFO] - 0/5 [2021-06-11 01:06:52,301][root][INFO] - 1/5 [2021-06-11 01:06:53,581][root][INFO] - 2/5 [2021-06-11 01:06:54,846][root][INFO] - 3/5 [2021-06-11 01:06:56,110][root][INFO] - 4/5 [2021-06-11 01:06:57,449][root][INFO] - Validation set: Time avg per batch 1.392 Loss avg 1.6222 IoU avg 0.101 [2021-06-11 01:06:57,465][root][INFO] - ---> save checkpoint [2021-06-11 01:07:02,377][root][INFO] - Lr: 9.882e-03 | Epoch: [1][0/20] | Time 1.837 | Data 1.204 | Loss 1.8792 | IoU 0.063 [2021-06-11 01:07:08,410][root][INFO] - Lr: 9.826e-03 | Epoch: [1][10/20] | Time 0.720 | Data 0.161 | Loss 1.6930 | IoU 0.086 [2021-06-11 01:07:15,542][root][INFO] - Lr: 9.769e-03 | Epoch: [2][0/20] | Time 1.575 | Data 0.970 | Loss 1.5274 | IoU 0.103 [2021-06-11 01:07:21,642][root][INFO] - Lr: 9.713e-03 | Epoch: [2][10/20] | Time 0.700 | Data 0.141 | Loss 1.4086 | IoU 0.121 [2021-06-11 01:07:28,851][root][INFO] - Lr: 9.656e-03 | Epoch: [3][0/20] | Time 1.550 | Data 0.951 | Loss 1.3228 | IoU 0.131 [2021-06-11 01:07:35,022][root][INFO] - Lr: 9.600e-03 | Epoch: [3][10/20] | Time 0.708 | Data 0.141 | Loss 1.2499 | IoU 0.141 [2021-06-11 01:07:42,408][root][INFO] - Lr: 9.543e-03 | Epoch: [4][0/20] | Time 1.745 | Data 1.139 | Loss 1.1912 | IoU 0.147 [2021-06-11 01:07:48,571][root][INFO] - Lr: 9.487e-03 | Epoch: [4][10/20] | Time 0.721 | Data 0.156 | Loss 1.1384 | IoU 0.153 [2021-06-11 01:07:55,732][root][INFO] - Lr: 9.430e-03 | Epoch: [5][0/20] | Time 1.529 | Data 0.921 | Loss 1.0964 | IoU 0.158 [2021-06-11 01:08:01,838][root][INFO] - Lr: 9.374e-03 | Epoch: [5][10/20] | Time 0.694 | Data 0.130 | Loss 1.0650 | IoU 0.162 [2021-06-11 01:08:08,038][root][INFO] - 0/5 [2021-06-11 01:08:09,347][root][INFO] - 1/5 [2021-06-11 01:08:10,645][root][INFO] - 2/5 [2021-06-11 01:08:11,925][root][INFO] - 3/5 [2021-06-11 01:08:13,227][root][INFO] - 4/5 [2021-06-11 01:08:14,600][root][INFO] - Validation set: Time avg per batch 1.416 Loss avg 1.2118 IoU avg 0.173 [2021-06-11 01:08:14,610][root][INFO] - ---> save checkpoint [2021-06-11 01:08:19,378][root][INFO] - Lr: 9.317e-03 | Epoch: [6][0/20] | Time 1.665 | Data 1.055 | Loss 1.0434 | IoU 0.164 [2021-06-11 01:08:25,507][root][INFO] - Lr: 9.260e-03 | Epoch: [6][10/20] | Time 0.710 | Data 0.143 | Loss 1.0235 | IoU 0.167 [2021-06-11 01:08:32,875][root][INFO] - Lr: 9.204e-03 | Epoch: [7][0/20] | Time 1.680 | Data 1.100 | Loss 0.9997 | IoU 0.170 [2021-06-11 01:08:39,096][root][INFO] - Lr: 9.147e-03 | Epoch: [7][10/20] | Time 0.721 | Data 0.156 | Loss 0.9828 | IoU 0.172 [2021-06-11 01:08:46,881][root][INFO] - Lr: 9.090e-03 | Epoch: [8][0/20] | Time 2.142 | Data 1.521 | Loss 0.9654 | IoU 0.174 [2021-06-11 01:08:52,982][root][INFO] - Lr: 9.033e-03 | Epoch: [8][10/20] | Time 0.749 | Data 0.183 | Loss 0.9481 | IoU 0.177 [2021-06-11 01:09:00,435][root][INFO] - Lr: 8.977e-03 | Epoch: [9][0/20] | Time 1.767 | Data 1.108 | Loss 0.9304 | IoU 0.180 [2021-06-11 01:09:06,659][root][INFO] - Lr: 8.920e-03 | Epoch: [9][10/20] | Time 0.730 | Data 0.151 | Loss 0.9169 | IoU 0.183 [2021-06-11 01:09:14,182][root][INFO] - Lr: 8.863e-03 | Epoch: [10][0/20] | Time 1.875 | Data 1.243 | Loss 0.9089 | IoU 0.184 [2021-06-11 01:09:20,300][root][INFO] - Lr: 8.806e-03 | Epoch: [10][10/20] | Time 0.727 | Data 0.159 | Loss 0.8969 | IoU 0.188 [2021-06-11 01:09:26,487][root][INFO] - 0/5 [2021-06-11 01:09:27,749][root][INFO] - 1/5 [2021-06-11 01:09:29,003][root][INFO] - 2/5 [2021-06-11 01:09:30,254][root][INFO] - 3/5 [2021-06-11 01:09:31,522][root][INFO] - 4/5 [2021-06-11 01:09:32,874][root][INFO] - Validation set: Time avg per batch 1.372 Loss avg 0.8625 IoU avg 0.223 [2021-06-11 01:09:32,888][root][INFO] - ---> save checkpoint [2021-06-11 01:09:37,281][root][INFO] - Lr: 8.749e-03 | Epoch: [11][0/20] | Time 1.384 | Data 0.775 | Loss 0.8801 | IoU 0.191 [2021-06-11 01:09:43,533][root][INFO] - Lr: 8.692e-03 | Epoch: [11][10/20] | Time 0.697 | Data 0.125 | Loss 0.8659 | IoU 0.194 [2021-06-11 01:09:50,995][root][INFO] - Lr: 8.634e-03 | Epoch: [12][0/20] | Time 1.815 | Data 1.144 | Loss 0.8523 | IoU 0.196 [2021-06-11 01:09:57,135][root][INFO] - Lr: 8.577e-03 | Epoch: [12][10/20] | Time 0.724 | Data 0.151 | Loss 0.8429 | IoU 0.198 [2021-06-11 01:10:04,774][root][INFO] - Lr: 8.520e-03 | Epoch: [13][0/20] | Time 1.964 | Data 1.366 | Loss 0.8299 | IoU 0.200 [2021-06-11 01:10:10,971][root][INFO] - Lr: 8.463e-03 | Epoch: [13][10/20] | Time 0.742 | Data 0.170 | Loss 0.8178 | IoU 0.202 [2021-06-11 01:10:18,347][root][INFO] - Lr: 8.406e-03 | Epoch: [14][0/20] | Time 1.658 | Data 1.046 | Loss 0.8065 | IoU 0.204 [2021-06-11 01:10:24,613][root][INFO] - Lr: 8.348e-03 | Epoch: [14][10/20] | Time 0.725 | Data 0.151 | Loss 0.7993 | IoU 0.206 [2021-06-11 01:10:32,484][root][INFO] - Lr: 8.291e-03 | Epoch: [15][0/20] | Time 2.169 | Data 1.541 | Loss 0.7902 | IoU 0.208 [2021-06-11 01:10:38,623][root][INFO] - Lr: 8.233e-03 | Epoch: [15][10/20] | Time 0.757 | Data 0.187 | Loss 0.7839 | IoU 0.210 [2021-06-11 01:10:44,762][root][INFO] - 0/5 [2021-06-11 01:10:46,054][root][INFO] - 1/5 [2021-06-11 01:10:47,313][root][INFO] - 2/5 [2021-06-11 01:10:48,563][root][INFO] - 3/5 [2021-06-11 01:10:49,818][root][INFO] - 4/5 [2021-06-11 01:10:51,194][root][INFO] - Validation set: Time avg per batch 1.365 Loss avg 0.9953 IoU avg 0.209 [2021-06-11 01:10:52,921][root][INFO] - Lr: 8.176e-03 | Epoch: [16][0/20] | Time 1.630 | Data 1.041 | Loss 0.7733 | IoU 0.212 [2021-06-11 01:10:59,177][root][INFO] - Lr: 8.118e-03 | Epoch: [16][10/20] | Time 0.718 | Data 0.148 | Loss 0.7638 | IoU 0.214 [2021-06-11 01:11:06,679][root][INFO] - Lr: 8.061e-03 | Epoch: [17][0/20] | Time 1.825 | Data 1.203 | Loss 0.7572 | IoU 0.215 [2021-06-11 01:11:12,785][root][INFO] - Lr: 8.003e-03 | Epoch: [17][10/20] | Time 0.721 | Data 0.153 | Loss 0.7483 | IoU 0.217 [2021-06-11 01:11:20,027][root][INFO] - Lr: 7.946e-03 | Epoch: [18][0/20] | Time 1.580 | Data 0.958 | Loss 0.7393 | IoU 0.219 [2021-06-11 01:11:26,214][root][INFO] - Lr: 7.888e-03 | Epoch: [18][10/20] | Time 0.707 | Data 0.135 | Loss 0.7313 | IoU 0.221 [2021-06-11 01:11:33,832][root][INFO] - Lr: 7.830e-03 | Epoch: [19][0/20] | Time 1.921 | Data 1.306 | Loss 0.7214 | IoU 0.223 [2021-06-11 01:11:40,011][root][INFO] - Lr: 7.772e-03 | Epoch: [19][10/20] | Time 0.740 | Data 0.169 | Loss 0.7121 | IoU 0.225 [2021-06-11 01:11:47,194][root][INFO] - Lr: 7.715e-03 | Epoch: [20][0/20] | Time 1.488 | Data 0.899 | Loss 0.7030 | IoU 0.227 [2021-06-11 01:11:53,433][root][INFO] - Lr: 7.657e-03 | Epoch: [20][10/20] | Time 0.703 | Data 0.133 | Loss 0.6962 | IoU 0.229 [2021-06-11 01:11:59,579][root][INFO] - 0/5 [2021-06-11 01:12:00,922][root][INFO] - 1/5 [2021-06-11 01:12:02,221][root][INFO] - 2/5 [2021-06-11 01:12:03,541][root][INFO] - 3/5 [2021-06-11 01:12:04,843][root][INFO] - 4/5 [2021-06-11 01:12:06,199][root][INFO] - Validation set: Time avg per batch 1.416 Loss avg 0.8920 IoU avg 0.236 [2021-06-11 01:12:06,219][root][INFO] - ---> save checkpoint [2021-06-11 01:12:10,654][root][INFO] - Lr: 7.599e-03 | Epoch: [21][0/20] | Time 1.589 | Data 0.941 | Loss 0.6887 | IoU 0.231 [2021-06-11 01:12:16,794][root][INFO] - Lr: 7.541e-03 | Epoch: [21][10/20] | Time 0.704 | Data 0.133 | Loss 0.6824 | IoU 0.233 [2021-06-11 01:12:24,307][root][INFO] - Lr: 7.483e-03 | Epoch: [22][0/20] | Time 1.827 | Data 1.208 | Loss 0.6751 | IoU 0.234 [2021-06-11 01:12:30,466][root][INFO] - Lr: 7.425e-03 | Epoch: [22][10/20] | Time 0.730 | Data 0.162 | Loss 0.6685 | IoU 0.236 [2021-06-11 01:12:37,755][root][INFO] - Lr: 7.367e-03 | Epoch: [23][0/20] | Time 1.660 | Data 1.045 | Loss 0.6609 | IoU 0.238 [2021-06-11 01:12:43,882][root][INFO] - Lr: 7.308e-03 | Epoch: [23][10/20] | Time 0.708 | Data 0.140 | Loss 0.6560 | IoU 0.240 [2021-06-11 01:12:50,930][root][INFO] - Lr: 7.250e-03 | Epoch: [24][0/20] | Time 1.295 | Data 0.681 | Loss 0.6498 | IoU 0.242 [2021-06-11 01:12:57,168][root][INFO] - Lr: 7.192e-03 | Epoch: [24][10/20] | Time 0.685 | Data 0.115 | Loss 0.6428 | IoU 0.243 [2021-06-11 01:13:04,636][root][INFO] - Lr: 7.133e-03 | Epoch: [25][0/20] | Time 1.781 | Data 1.125 | Loss 0.6387 | IoU 0.245 [2021-06-11 01:13:10,795][root][INFO] - Lr: 7.075e-03 | Epoch: [25][10/20] | Time 0.725 | Data 0.155 | Loss 0.6329 | IoU 0.247 [2021-06-11 01:13:16,912][root][INFO] - 0/5 [2021-06-11 01:13:18,202][root][INFO] - 1/5 [2021-06-11 01:13:19,491][root][INFO] - 2/5 [2021-06-11 01:13:20,748][root][INFO] - 3/5 [2021-06-11 01:13:22,011][root][INFO] - 4/5 [2021-06-11 01:13:23,389][root][INFO] - Validation set: Time avg per batch 1.380 Loss avg 0.9234 IoU avg 0.247 [2021-06-11 01:13:23,404][root][INFO] - ---> save checkpoint [2021-06-11 01:13:27,924][root][INFO] - Lr: 7.017e-03 | Epoch: [26][0/20] | Time 1.525 | Data 0.904 | Loss 0.6277 | IoU 0.248 [2021-06-11 01:13:34,058][root][INFO] - Lr: 6.958e-03 | Epoch: [26][10/20] | Time 0.698 | Data 0.129 | Loss 0.6224 | IoU 0.250 [2021-06-11 01:13:41,350][root][INFO] - Lr: 6.900e-03 | Epoch: [27][0/20] | Time 1.619 | Data 1.000 | Loss 0.6170 | IoU 0.252 [2021-06-11 01:13:47,570][root][INFO] - Lr: 6.841e-03 | Epoch: [27][10/20] | Time 0.716 | Data 0.144 | Loss 0.6122 | IoU 0.253 [2021-06-11 01:13:54,738][root][INFO] - Lr: 6.782e-03 | Epoch: [28][0/20] | Time 1.486 | Data 0.887 | Loss 0.6064 | IoU 0.254 [2021-06-11 01:14:00,980][root][INFO] - Lr: 6.724e-03 | Epoch: [28][10/20] | Time 0.705 | Data 0.134 | Loss 0.6017 | IoU 0.256 [2021-06-11 01:14:08,137][root][INFO] - Lr: 6.665e-03 | Epoch: [29][0/20] | Time 1.490 | Data 0.893 | Loss 0.5971 | IoU 0.257 [2021-06-11 01:14:14,304][root][INFO] - Lr: 6.606e-03 | Epoch: [29][10/20] | Time 0.698 | Data 0.131 | Loss 0.5924 | IoU 0.259 [2021-06-11 01:14:21,760][root][INFO] - Lr: 6.547e-03 | Epoch: [30][0/20] | Time 1.743 | Data 1.112 | Loss 0.5879 | IoU 0.260 [2021-06-11 01:14:27,950][root][INFO] - Lr: 6.488e-03 | Epoch: [30][10/20] | Time 0.726 | Data 0.153 | Loss 0.5833 | IoU 0.261 [2021-06-11 01:14:34,046][root][INFO] - 0/5 [2021-06-11 01:14:35,340][root][INFO] - 1/5 [2021-06-11 01:14:36,633][root][INFO] - 2/5 [2021-06-11 01:14:37,928][root][INFO] - 3/5 [2021-06-11 01:14:39,202][root][INFO] - 4/5 [2021-06-11 01:14:40,576][root][INFO] - Validation set: Time avg per batch 1.387 Loss avg 1.2349 IoU avg 0.223 [2021-06-11 01:14:42,347][root][INFO] - Lr: 6.429e-03 | Epoch: [31][0/20] | Time 1.609 | Data 0.998 | Loss 0.5805 | IoU 0.262 [2021-06-11 01:14:48,534][root][INFO] - Lr: 6.370e-03 | Epoch: [31][10/20] | Time 0.716 | Data 0.150 | Loss 0.5765 | IoU 0.264 [2021-06-11 01:14:55,719][root][INFO] - Lr: 6.311e-03 | Epoch: [32][0/20] | Time 1.494 | Data 0.862 | Loss 0.5724 | IoU 0.265 [2021-06-11 01:15:01,914][root][INFO] - Lr: 6.252e-03 | Epoch: [32][10/20] | Time 0.702 | Data 0.127 | Loss 0.5694 | IoU 0.266 [2021-06-11 01:15:09,309][root][INFO] - Lr: 6.192e-03 | Epoch: [33][0/20] | Time 1.689 | Data 1.091 | Loss 0.5658 | IoU 0.267 [2021-06-11 01:15:15,481][root][INFO] - Lr: 6.133e-03 | Epoch: [33][10/20] | Time 0.716 | Data 0.146 | Loss 0.5626 | IoU 0.268 [2021-06-11 01:15:22,750][root][INFO] - Lr: 6.074e-03 | Epoch: [34][0/20] | Time 1.573 | Data 0.962 | Loss 0.5581 | IoU 0.269 [2021-06-11 01:15:28,928][root][INFO] - Lr: 6.014e-03 | Epoch: [34][10/20] | Time 0.708 | Data 0.137 | Loss 0.5538 | IoU 0.271 [2021-06-11 01:15:36,332][root][INFO] - Lr: 5.955e-03 | Epoch: [35][0/20] | Time 1.682 | Data 1.068 | Loss 0.5499 | IoU 0.272 [2021-06-11 01:15:42,593][root][INFO] - Lr: 5.895e-03 | Epoch: [35][10/20] | Time 0.727 | Data 0.156 | Loss 0.5464 | IoU 0.273 [2021-06-11 01:15:48,709][root][INFO] - 0/5 [2021-06-11 01:15:50,006][root][INFO] - 1/5 [2021-06-11 01:15:51,310][root][INFO] - 2/5 [2021-06-11 01:15:52,594][root][INFO] - 3/5 [2021-06-11 01:15:53,874][root][INFO] - 4/5 [2021-06-11 01:15:55,211][root][INFO] - Validation set: Time avg per batch 1.385 Loss avg 0.8498 IoU avg 0.262 [2021-06-11 01:15:55,224][root][INFO] - ---> save checkpoint [2021-06-11 01:15:59,843][root][INFO] - Lr: 5.836e-03 | Epoch: [36][0/20] | Time 1.633 | Data 1.026 | Loss 0.5423 | IoU 0.275 [2021-06-11 01:16:06,068][root][INFO] - Lr: 5.776e-03 | Epoch: [36][10/20] | Time 0.719 | Data 0.149 | Loss 0.5389 | IoU 0.276 [2021-06-11 01:16:13,290][root][INFO] - Lr: 5.716e-03 | Epoch: [37][0/20] | Time 1.558 | Data 0.955 | Loss 0.5349 | IoU 0.277 [2021-06-11 01:16:19,455][root][INFO] - Lr: 5.656e-03 | Epoch: [37][10/20] | Time 0.704 | Data 0.137 | Loss 0.5310 | IoU 0.278 [2021-06-11 01:16:26,751][root][INFO] - Lr: 5.596e-03 | Epoch: [38][0/20] | Time 1.635 | Data 1.028 | Loss 0.5274 | IoU 0.279 [2021-06-11 01:16:32,911][root][INFO] - Lr: 5.536e-03 | Epoch: [38][10/20] | Time 0.709 | Data 0.141 | Loss 0.5244 | IoU 0.280 [2021-06-11 01:16:40,496][root][INFO] - Lr: 5.476e-03 | Epoch: [39][0/20] | Time 1.874 | Data 1.251 | Loss 0.5213 | IoU 0.281 [2021-06-11 01:16:46,672][root][INFO] - Lr: 5.416e-03 | Epoch: [39][10/20] | Time 0.736 | Data 0.167 | Loss 0.5185 | IoU 0.282 [2021-06-11 01:16:54,124][root][INFO] - Lr: 5.356e-03 | Epoch: [40][0/20] | Time 1.762 | Data 1.180 | Loss 0.5153 | IoU 0.283 [2021-06-11 01:17:00,289][root][INFO] - Lr: 5.296e-03 | Epoch: [40][10/20] | Time 0.723 | Data 0.156 | Loss 0.5117 | IoU 0.285 [2021-06-11 01:17:06,413][root][INFO] - 0/5 [2021-06-11 01:17:07,706][root][INFO] - 1/5 [2021-06-11 01:17:08,997][root][INFO] - 2/5 [2021-06-11 01:17:10,289][root][INFO] - 3/5 [2021-06-11 01:17:11,570][root][INFO] - 4/5 [2021-06-11 01:17:12,942][root][INFO] - Validation set: Time avg per batch 1.396 Loss avg 0.7111 IoU avg 0.290 [2021-06-11 01:17:12,962][root][INFO] - ---> save checkpoint [2021-06-11 01:17:17,388][root][INFO] - Lr: 5.235e-03 | Epoch: [41][0/20] | Time 1.460 | Data 0.848 | Loss 0.5085 | IoU 0.286 [2021-06-11 01:17:23,503][root][INFO] - Lr: 5.175e-03 | Epoch: [41][10/20] | Time 0.692 | Data 0.125 | Loss 0.5056 | IoU 0.287 [2021-06-11 01:17:31,006][root][INFO] - Lr: 5.114e-03 | Epoch: [42][0/20] | Time 1.816 | Data 1.192 | Loss 0.5025 | IoU 0.288 [2021-06-11 01:17:37,130][root][INFO] - Lr: 5.054e-03 | Epoch: [42][10/20] | Time 0.724 | Data 0.156 | Loss 0.4991 | IoU 0.289 [2021-06-11 01:17:44,715][root][INFO] - Lr: 4.993e-03 | Epoch: [43][0/20] | Time 1.956 | Data 1.315 | Loss 0.4963 | IoU 0.290 [2021-06-11 01:17:50,869][root][INFO] - Lr: 4.932e-03 | Epoch: [43][10/20] | Time 0.738 | Data 0.167 | Loss 0.4933 | IoU 0.291 [2021-06-11 01:17:58,030][root][INFO] - Lr: 4.871e-03 | Epoch: [44][0/20] | Time 1.514 | Data 0.862 | Loss 0.4904 | IoU 0.292 [2021-06-11 01:18:04,162][root][INFO] - Lr: 4.810e-03 | Epoch: [44][10/20] | Time 0.697 | Data 0.126 | Loss 0.4877 | IoU 0.293 [2021-06-11 01:18:11,265][root][INFO] - Lr: 4.749e-03 | Epoch: [45][0/20] | Time 1.452 | Data 0.859 | Loss 0.4849 | IoU 0.294 [2021-06-11 01:18:17,485][root][INFO] - Lr: 4.688e-03 | Epoch: [45][10/20] | Time 0.698 | Data 0.128 | Loss 0.4818 | IoU 0.295 [2021-06-11 01:18:23,614][root][INFO] - 0/5 [2021-06-11 01:18:24,945][root][INFO] - 1/5 [2021-06-11 01:18:26,257][root][INFO] - 2/5 [2021-06-11 01:18:27,540][root][INFO] - 3/5 [2021-06-11 01:18:28,834][root][INFO] - 4/5 [2021-06-11 01:18:30,212][root][INFO] - Validation set: Time avg per batch 1.404 Loss avg 0.6109 IoU avg 0.312 [2021-06-11 01:18:30,230][root][INFO] - ---> save checkpoint [2021-06-11 01:18:34,910][root][INFO] - Lr: 4.627e-03 | Epoch: [46][0/20] | Time 1.474 | Data 0.883 | Loss 0.4789 | IoU 0.296 [2021-06-11 01:18:41,068][root][INFO] - Lr: 4.566e-03 | Epoch: [46][10/20] | Time 0.697 | Data 0.129 | Loss 0.4765 | IoU 0.297 [2021-06-11 01:18:48,445][root][INFO] - Lr: 4.504e-03 | Epoch: [47][0/20] | Time 1.723 | Data 1.126 | Loss 0.4738 | IoU 0.298 [2021-06-11 01:18:54,625][root][INFO] - Lr: 4.443e-03 | Epoch: [47][10/20] | Time 0.721 | Data 0.150 | Loss 0.4714 | IoU 0.299 [2021-06-11 01:19:01,957][root][INFO] - Lr: 4.381e-03 | Epoch: [48][0/20] | Time 1.664 | Data 1.078 | Loss 0.4691 | IoU 0.300 [2021-06-11 01:19:08,130][root][INFO] - Lr: 4.320e-03 | Epoch: [48][10/20] | Time 0.713 | Data 0.147 | Loss 0.4668 | IoU 0.300 [2021-06-11 01:19:15,542][root][INFO] - Lr: 4.258e-03 | Epoch: [49][0/20] | Time 1.760 | Data 1.165 | Loss 0.4645 | IoU 0.301 [2021-06-11 01:19:21,651][root][INFO] - Lr: 4.196e-03 | Epoch: [49][10/20] | Time 0.716 | Data 0.151 | Loss 0.4619 | IoU 0.302 [2021-06-11 01:19:28,963][root][INFO] - Lr: 4.134e-03 | Epoch: [50][0/20] | Time 1.675 | Data 1.073 | Loss 0.4596 | IoU 0.303 [2021-06-11 01:19:35,142][root][INFO] - Lr: 4.072e-03 | Epoch: [50][10/20] | Time 0.715 | Data 0.145 | Loss 0.4574 | IoU 0.304 [2021-06-11 01:19:41,266][root][INFO] - 0/5 [2021-06-11 01:19:42,553][root][INFO] - 1/5 [2021-06-11 01:19:43,871][root][INFO] - 2/5 [2021-06-11 01:19:45,151][root][INFO] - 3/5 [2021-06-11 01:19:46,474][root][INFO] - 4/5 [2021-06-11 01:19:47,854][root][INFO] - Validation set: Time avg per batch 1.403 Loss avg 0.6322 IoU avg 0.314 [2021-06-11 01:19:47,870][root][INFO] - ---> save checkpoint [2021-06-11 01:19:52,725][root][INFO] - Lr: 4.010e-03 | Epoch: [51][0/20] | Time 1.810 | Data 1.197 | Loss 0.4553 | IoU 0.304 [2021-06-11 01:19:58,835][root][INFO] - Lr: 3.948e-03 | Epoch: [51][10/20] | Time 0.721 | Data 0.155 | Loss 0.4528 | IoU 0.305 [2021-06-11 01:20:05,845][root][INFO] - Lr: 3.885e-03 | Epoch: [52][0/20] | Time 1.376 | Data 0.745 | Loss 0.4507 | IoU 0.306 [2021-06-11 01:20:11,991][root][INFO] - Lr: 3.823e-03 | Epoch: [52][10/20] | Time 0.683 | Data 0.115 | Loss 0.4485 | IoU 0.307 [2021-06-11 01:20:19,268][root][INFO] - Lr: 3.760e-03 | Epoch: [53][0/20] | Time 1.615 | Data 1.024 | Loss 0.4460 | IoU 0.308 [2021-06-11 01:20:25,430][root][INFO] - Lr: 3.697e-03 | Epoch: [53][10/20] | Time 0.707 | Data 0.139 | Loss 0.4438 | IoU 0.309 [2021-06-11 01:20:32,748][root][INFO] - Lr: 3.635e-03 | Epoch: [54][0/20] | Time 1.669 | Data 1.081 | Loss 0.4419 | IoU 0.309 [2021-06-11 01:20:38,904][root][INFO] - Lr: 3.572e-03 | Epoch: [54][10/20] | Time 0.711 | Data 0.144 | Loss 0.4398 | IoU 0.310 [2021-06-11 01:20:46,015][root][INFO] - Lr: 3.508e-03 | Epoch: [55][0/20] | Time 1.423 | Data 0.827 | Loss 0.4375 | IoU 0.311 [2021-06-11 01:20:52,189][root][INFO] - Lr: 3.445e-03 | Epoch: [55][10/20] | Time 0.693 | Data 0.126 | Loss 0.4355 | IoU 0.312 [2021-06-11 01:20:58,294][root][INFO] - 0/5 [2021-06-11 01:20:59,562][root][INFO] - 1/5 [2021-06-11 01:21:00,847][root][INFO] - 2/5 [2021-06-11 01:21:02,105][root][INFO] - 3/5 [2021-06-11 01:21:03,401][root][INFO] - 4/5 [2021-06-11 01:21:04,767][root][INFO] - Validation set: Time avg per batch 1.378 Loss avg 0.6352 IoU avg 0.312 [2021-06-11 01:21:06,466][root][INFO] - Lr: 3.382e-03 | Epoch: [56][0/20] | Time 1.563 | Data 0.983 | Loss 0.4336 | IoU 0.312 [2021-06-11 01:21:12,588][root][INFO] - Lr: 3.318e-03 | Epoch: [56][10/20] | Time 0.703 | Data 0.141 | Loss 0.4318 | IoU 0.313 [2021-06-11 01:21:19,980][root][INFO] - Lr: 3.255e-03 | Epoch: [57][0/20] | Time 1.706 | Data 1.099 | Loss 0.4298 | IoU 0.314 [2021-06-11 01:21:26,123][root][INFO] - Lr: 3.191e-03 | Epoch: [57][10/20] | Time 0.716 | Data 0.150 | Loss 0.4278 | IoU 0.315 [2021-06-11 01:21:33,141][root][INFO] - Lr: 3.127e-03 | Epoch: [58][0/20] | Time 1.357 | Data 0.727 | Loss 0.4257 | IoU 0.315 [2021-06-11 01:21:39,383][root][INFO] - Lr: 3.063e-03 | Epoch: [58][10/20] | Time 0.692 | Data 0.121 | Loss 0.4238 | IoU 0.316 [2021-06-11 01:21:46,848][root][INFO] - Lr: 2.999e-03 | Epoch: [59][0/20] | Time 1.798 | Data 1.205 | Loss 0.4224 | IoU 0.317 [2021-06-11 01:21:52,995][root][INFO] - Lr: 2.935e-03 | Epoch: [59][10/20] | Time 0.723 | Data 0.156 | Loss 0.4205 | IoU 0.317 [2021-06-11 01:22:00,315][root][INFO] - Lr: 2.870e-03 | Epoch: [60][0/20] | Time 1.636 | Data 1.037 | Loss 0.4186 | IoU 0.318 [2021-06-11 01:22:06,483][root][INFO] - Lr: 2.805e-03 | Epoch: [60][10/20] | Time 0.711 | Data 0.144 | Loss 0.4168 | IoU 0.319 [2021-06-11 01:22:12,514][root][INFO] - 0/5 [2021-06-11 01:22:13,804][root][INFO] - 1/5 [2021-06-11 01:22:15,094][root][INFO] - 2/5 [2021-06-11 01:22:16,385][root][INFO] - 3/5 [2021-06-11 01:22:17,667][root][INFO] - 4/5 [2021-06-11 01:22:19,081][root][INFO] - Validation set: Time avg per batch 1.372 Loss avg 0.5375 IoU avg 0.324 [2021-06-11 01:22:19,103][root][INFO] - ---> save checkpoint [2021-06-11 01:22:23,682][root][INFO] - Lr: 2.741e-03 | Epoch: [61][0/20] | Time 1.533 | Data 0.928 | Loss 0.4150 | IoU 0.320 [2021-06-11 01:22:29,833][root][INFO] - Lr: 2.676e-03 | Epoch: [61][10/20] | Time 0.702 | Data 0.134 | Loss 0.4133 | IoU 0.321 [2021-06-11 01:22:36,794][root][INFO] - Lr: 2.610e-03 | Epoch: [62][0/20] | Time 1.317 | Data 0.719 | Loss 0.4115 | IoU 0.321 [2021-06-11 01:22:43,012][root][INFO] - Lr: 2.545e-03 | Epoch: [62][10/20] | Time 0.687 | Data 0.113 | Loss 0.4096 | IoU 0.322 [2021-06-11 01:22:50,177][root][INFO] - Lr: 2.480e-03 | Epoch: [63][0/20] | Time 1.508 | Data 0.902 | Loss 0.4081 | IoU 0.323 [2021-06-11 01:22:56,363][root][INFO] - Lr: 2.414e-03 | Epoch: [63][10/20] | Time 0.700 | Data 0.130 | Loss 0.4063 | IoU 0.323 [2021-06-11 01:23:03,836][root][INFO] - Lr: 2.348e-03 | Epoch: [64][0/20] | Time 1.670 | Data 1.083 | Loss 0.4047 | IoU 0.324 [2021-06-11 01:23:10,004][root][INFO] - Lr: 2.282e-03 | Epoch: [64][10/20] | Time 0.714 | Data 0.145 | Loss 0.4032 | IoU 0.325 [2021-06-11 01:23:17,230][root][INFO] - Lr: 2.215e-03 | Epoch: [65][0/20] | Time 1.580 | Data 0.982 | Loss 0.4015 | IoU 0.325 [2021-06-11 01:23:23,452][root][INFO] - Lr: 2.149e-03 | Epoch: [65][10/20] | Time 0.710 | Data 0.138 | Loss 0.4000 | IoU 0.326 [2021-06-11 01:23:29,757][root][INFO] - 0/5 [2021-06-11 01:23:31,053][root][INFO] - 1/5 [2021-06-11 01:23:32,339][root][INFO] - 2/5 [2021-06-11 01:23:33,617][root][INFO] - 3/5 [2021-06-11 01:23:34,898][root][INFO] - 4/5 [2021-06-11 01:23:36,295][root][INFO] - Validation set: Time avg per batch 1.421 Loss avg 0.5432 IoU avg 0.321 [2021-06-11 01:23:38,206][root][INFO] - Lr: 2.082e-03 | Epoch: [66][0/20] | Time 1.823 | Data 1.206 | Loss 0.3984 | IoU 0.327 [2021-06-11 01:23:44,325][root][INFO] - Lr: 2.015e-03 | Epoch: [66][10/20] | Time 0.722 | Data 0.154 | Loss 0.3969 | IoU 0.327 [2021-06-11 01:23:51,607][root][INFO] - Lr: 1.948e-03 | Epoch: [67][0/20] | Time 1.621 | Data 1.034 | Loss 0.3954 | IoU 0.328 [2021-06-11 01:23:57,741][root][INFO] - Lr: 1.880e-03 | Epoch: [67][10/20] | Time 0.707 | Data 0.141 | Loss 0.3940 | IoU 0.328 [2021-06-11 01:24:05,374][root][INFO] - Lr: 1.812e-03 | Epoch: [68][0/20] | Time 1.898 | Data 1.271 | Loss 0.3924 | IoU 0.329 [2021-06-11 01:24:11,576][root][INFO] - Lr: 1.744e-03 | Epoch: [68][10/20] | Time 0.742 | Data 0.171 | Loss 0.3908 | IoU 0.330 [2021-06-11 01:24:18,938][root][INFO] - Lr: 1.676e-03 | Epoch: [69][0/20] | Time 1.704 | Data 1.137 | Loss 0.3893 | IoU 0.330 [2021-06-11 01:24:25,095][root][INFO] - Lr: 1.607e-03 | Epoch: [69][10/20] | Time 0.715 | Data 0.150 | Loss 0.3878 | IoU 0.331 [2021-06-11 01:24:32,412][root][INFO] - Lr: 1.538e-03 | Epoch: [70][0/20] | Time 1.651 | Data 1.032 | Loss 0.3864 | IoU 0.331 [2021-06-11 01:24:38,667][root][INFO] - Lr: 1.469e-03 | Epoch: [70][10/20] | Time 0.720 | Data 0.149 | Loss 0.3851 | IoU 0.332 [2021-06-11 01:24:44,798][root][INFO] - 0/5 [2021-06-11 01:24:46,102][root][INFO] - 1/5 [2021-06-11 01:24:47,415][root][INFO] - 2/5 [2021-06-11 01:24:48,723][root][INFO] - 3/5 [2021-06-11 01:24:50,010][root][INFO] - 4/5 [2021-06-11 01:24:51,395][root][INFO] - Validation set: Time avg per batch 1.403 Loss avg 0.5271 IoU avg 0.326 [2021-06-11 01:24:51,412][root][INFO] - ---> save checkpoint [2021-06-11 01:24:56,053][root][INFO] - Lr: 1.399e-03 | Epoch: [71][0/20] | Time 1.590 | Data 0.930 | Loss 0.3836 | IoU 0.333 [2021-06-11 01:25:02,151][root][INFO] - Lr: 1.329e-03 | Epoch: [71][10/20] | Time 0.699 | Data 0.131 | Loss 0.3823 | IoU 0.333 [2021-06-11 01:25:09,514][root][INFO] - Lr: 1.258e-03 | Epoch: [72][0/20] | Time 1.692 | Data 1.088 | Loss 0.3810 | IoU 0.334 [2021-06-11 01:25:15,638][root][INFO] - Lr: 1.187e-03 | Epoch: [72][10/20] | Time 0.711 | Data 0.145 | Loss 0.3796 | IoU 0.334 [2021-06-11 01:25:23,074][root][INFO] - Lr: 1.116e-03 | Epoch: [73][0/20] | Time 1.749 | Data 1.120 | Loss 0.3782 | IoU 0.335 [2021-06-11 01:25:29,183][root][INFO] - Lr: 1.044e-03 | Epoch: [73][10/20] | Time 0.719 | Data 0.151 | Loss 0.3767 | IoU 0.335 [2021-06-11 01:25:36,639][root][INFO] - Lr: 9.712e-04 | Epoch: [74][0/20] | Time 1.789 | Data 1.190 | Loss 0.3753 | IoU 0.336 [2021-06-11 01:25:42,818][root][INFO] - Lr: 8.981e-04 | Epoch: [74][10/20] | Time 0.725 | Data 0.156 | Loss 0.3741 | IoU 0.337 [2021-06-11 01:25:50,291][root][INFO] - Lr: 8.242e-04 | Epoch: [75][0/20] | Time 1.823 | Data 1.217 | Loss 0.3727 | IoU 0.337 [2021-06-11 01:25:56,498][root][INFO] - Lr: 7.497e-04 | Epoch: [75][10/20] | Time 0.730 | Data 0.159 | Loss 0.3715 | IoU 0.338 [2021-06-11 01:26:02,625][root][INFO] - 0/5 [2021-06-11 01:26:03,952][root][INFO] - 1/5 [2021-06-11 01:26:05,240][root][INFO] - 2/5 [2021-06-11 01:26:06,520][root][INFO] - 3/5 [2021-06-11 01:26:07,794][root][INFO] - 4/5 [2021-06-11 01:26:09,168][root][INFO] - Validation set: Time avg per batch 1.395 Loss avg 0.5089 IoU avg 0.332 [2021-06-11 01:26:09,184][root][INFO] - ---> save checkpoint [2021-06-11 01:26:13,966][root][INFO] - Lr: 6.743e-04 | Epoch: [76][0/20] | Time 1.488 | Data 0.873 | Loss 0.3703 | IoU 0.338 [2021-06-11 01:26:20,117][root][INFO] - Lr: 5.979e-04 | Epoch: [76][10/20] | Time 0.699 | Data 0.131 | Loss 0.3691 | IoU 0.339 [2021-06-11 01:26:27,073][root][INFO] - Lr: 5.205e-04 | Epoch: [77][0/20] | Time 1.297 | Data 0.699 | Loss 0.3679 | IoU 0.339 [2021-06-11 01:26:33,210][root][INFO] - Lr: 4.417e-04 | Epoch: [77][10/20] | Time 0.678 | Data 0.113 | Loss 0.3667 | IoU 0.340 [2021-06-11 01:26:40,446][root][INFO] - Lr: 3.613e-04 | Epoch: [78][0/20] | Time 1.572 | Data 0.977 | Loss 0.3654 | IoU 0.340 [2021-06-11 01:26:46,652][root][INFO] - Lr: 2.789e-04 | Epoch: [78][10/20] | Time 0.709 | Data 0.138 | Loss 0.3643 | IoU 0.341 [2021-06-11 01:26:53,737][root][INFO] - Lr: 1.936e-04 | Epoch: [79][0/20] | Time 1.434 | Data 0.857 | Loss 0.3630 | IoU 0.341 [2021-06-11 01:26:59,865][root][INFO] - Lr: 1.038e-04 | Epoch: [79][10/20] | Time 0.688 | Data 0.124 | Loss 0.3619 | IoU 0.342 [2021-06-11 01:27:05,453][root][INFO] - Finished Training [2021-06-11 01:27:05,464][submitit][INFO] - Job completed successfully