[2021-07-07 09:43:42,381][root][INFO] - ===> Configurations [2021-07-07 09:43:42,395][root][INFO] - net: model: Res16UNet34C conv1_kernel_size: 3 weights: /rhome/jhou/data/checkpoint/data_efficient3d/publish/pretrained/partition8_4096_100k.pth weights_for_inner_model: false dilations: - 1 - 1 - 1 - 1 wrapper_type: null wrapper_region_type: 1 wrapper_kernel_size: 3 wrapper_lr: 0.1 meanfield_iterations: 10 crf_spatial_sigma: 1 crf_chromatic_sigma: 12 optimizer: optimizer: SGD lr: 0.1 sgd_momentum: 0.9 sgd_dampening: 0.1 adam_beta1: 0.9 adam_beta2: 0.999 weight_decay: 0.0001 param_histogram_freq: 100 save_param_histogram: false iter_size: 1 bn_momentum: 0.02 scheduler: PolyLR max_iter: 60000 step_size: 20000.0 step_gamma: 0.1 poly_power: 0.9 exp_gamma: 0.95 exp_step_size: 445 data: dataset: StanfordArea5Dataset train_file: null voxel_size: 0.05 data_dir: data sampled_inds: null temporal_dilation: 30 temporal_numseq: 3 point_lim: -1 pre_point_lim: -1 batch_size: 48 cache_data: false num_workers: 6 num_val_workers: 1 ignore_label: 255 return_transformation: false ignore_duplicate_class: false partial_crop: 0 train_limit_numpoints: 0 synthia_path: /home/chrischoy/datasets/Synthia/Synthia4D synthia_camera_path: /home/chrischoy/datasets/Synthia/%s/CameraParams/ synthia_camera_intrinsic_file: intrinsics.txt synthia_camera_extrinsics_file: Stereo_Right/Omni_F/%s.txt temporal_rand_dilation: false temporal_rand_numseq: false scannet_path: /home/chrischoy/datasets/scannet/scannet_preprocessed stanford3d_path: /rhome/jhou/data/dataset/stanford3d/pointcloud/ train: is_train: true stat_freq: 5 val_freq: 200 empty_cache_freq: 1 train_phase: train val_phase: val overwrite_weights: true resume: true resume_optimizer: True, eval_upsample: false lenient_weight_loading: true distributed: distributed_world_size: 8 distributed_rank: 0 distributed_backend: nccl distributed_init_method: null distributed_port: 10010 device_id: 0 distributed_no_spawn: true ddp_backend: c10d bucket_cap_mb: 25 augmentation: use_feat_aug: true data_aug_color_trans_ratio: 0.05 data_aug_color_jitter_std: 0.005 normalize_color: true data_aug_scale_min: 0.9 data_aug_scale_max: 1.1 data_aug_hue_max: 0.5 data_aug_saturation_max: 0.2 test: visualize: false save_features: false save_feat_dir: outputs/feat test_phase: test test_stat_freq: 100 evaluate_benchmark: false misc: is_cuda: true load_path: null log_step: 50 log_level: INFO num_gpus: 8 seed: 123 log_dir: reproduce4 load_bn: all_bn resume_config: null train_stuff: false [2021-07-07 09:43:42,396][root][INFO] - ===> Initializing dataloader [2021-07-07 09:43:42,399][root][INFO] - voxel size: 0.05 [2021-07-07 09:43:42,399][root][INFO] - Loading StanfordArea5Dataset DatasetPhase.Train: ['area1.txt', 'area2.txt', 'area3.txt', 'area4.txt', 'area6.txt'] [2021-07-07 09:43:42,402][root][INFO] - voxel size: 0.05 [2021-07-07 09:43:42,402][root][INFO] - Loading StanfordArea5Dataset DatasetPhase.Val: area5.txt [2021-07-07 09:43:42,404][root][INFO] - voxel size: 0.05 [2021-07-07 09:43:42,404][root][INFO] - Loading StanfordArea5Dataset DatasetPhase.Test: area5.txt [2021-07-07 09:43:42,405][root][INFO] - ===> Building model [2021-07-07 09:43:42,760][root][INFO] - ===> Number of trainable parameters: Res16UNet34C: 37845965 [2021-07-07 09:43:42,765][root][INFO] - Res16UNet34C( (conv0p1s1): MinkowskiConvolution(in=3, out=32, region_type=RegionType.HYPERCUBE, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (bn0): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (conv1p1s2): MinkowskiConvolution(in=32, out=32, region_type=RegionType.HYPERCUBE, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bn1): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block1): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=32, out=32, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=32, out=32, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=32, out=32, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=32, out=32, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (conv2p2s2): MinkowskiConvolution(in=32, out=32, region_type=RegionType.HYPERCUBE, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bn2): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block2): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=32, out=64, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=64, out=64, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=32, out=64, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=64, out=64, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=64, out=64, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (2): BasicBlock( (conv1): MinkowskiConvolution(in=64, out=64, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=64, out=64, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (conv3p4s2): MinkowskiConvolution(in=64, out=64, region_type=RegionType.HYPERCUBE, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bn3): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block3): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=64, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=64, out=128, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (2): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (3): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (conv4p8s2): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYPERCUBE, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bn4): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block4): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=128, out=256, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (2): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (3): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (4): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (5): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (convtr4p16s2): MinkowskiConvolutionTranspose(in=256, out=256, region_type=RegionType.HYPERCUBE, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bntr4): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block5): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=384, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=384, out=256, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (convtr5p8s2): MinkowskiConvolutionTranspose(in=256, out=128, region_type=RegionType.HYPERCUBE, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bntr5): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block6): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=192, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=192, out=128, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (convtr6p4s2): MinkowskiConvolutionTranspose(in=128, out=96, region_type=RegionType.HYPERCUBE, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bntr6): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block7): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=96, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=96, out=96, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=128, out=96, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=96, out=96, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=96, out=96, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (convtr7p2s2): MinkowskiConvolutionTranspose(in=96, out=96, region_type=RegionType.HYPERCUBE, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bntr7): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block8): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=96, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=96, out=96, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=128, out=96, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=96, out=96, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=96, out=96, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (final): MinkowskiConvolution(in=96, out=13, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (relu): MinkowskiReLU() ) [2021-07-07 09:43:42,769][root][INFO] - ===> Loading weights: /rhome/jhou/data/checkpoint/data_efficient3d/publish/pretrained/partition8_4096_100k.pth [2021-07-07 09:43:42,996][root][INFO] - Loading weights:conv0p1s1.kernel, bn0.bn.weight, bn0.bn.bias, bn0.bn.running_mean, bn0.bn.running_var, bn0.bn.num_batches_tracked, conv1p1s2.kernel, bn1.bn.weight, bn1.bn.bias, bn1.bn.running_mean, bn1.bn.running_var, bn1.bn.num_batches_tracked, block1.0.conv1.kernel, block1.0.norm1.bn.weight, block1.0.norm1.bn.bias, block1.0.norm1.bn.running_mean, block1.0.norm1.bn.running_var, block1.0.norm1.bn.num_batches_tracked, block1.0.conv2.kernel, block1.0.norm2.bn.weight, block1.0.norm2.bn.bias, block1.0.norm2.bn.running_mean, block1.0.norm2.bn.running_var, block1.0.norm2.bn.num_batches_tracked, block1.1.conv1.kernel, block1.1.norm1.bn.weight, block1.1.norm1.bn.bias, block1.1.norm1.bn.running_mean, block1.1.norm1.bn.running_var, block1.1.norm1.bn.num_batches_tracked, block1.1.conv2.kernel, block1.1.norm2.bn.weight, block1.1.norm2.bn.bias, block1.1.norm2.bn.running_mean, block1.1.norm2.bn.running_var, block1.1.norm2.bn.num_batches_tracked, conv2p2s2.kernel, bn2.bn.weight, bn2.bn.bias, bn2.bn.running_mean, bn2.bn.running_var, bn2.bn.num_batches_tracked, block2.0.conv1.kernel, block2.0.norm1.bn.weight, block2.0.norm1.bn.bias, block2.0.norm1.bn.running_mean, block2.0.norm1.bn.running_var, block2.0.norm1.bn.num_batches_tracked, block2.0.conv2.kernel, block2.0.norm2.bn.weight, block2.0.norm2.bn.bias, block2.0.norm2.bn.running_mean, block2.0.norm2.bn.running_var, block2.0.norm2.bn.num_batches_tracked, block2.0.downsample.0.kernel, block2.0.downsample.1.bn.weight, block2.0.downsample.1.bn.bias, block2.0.downsample.1.bn.running_mean, block2.0.downsample.1.bn.running_var, block2.0.downsample.1.bn.num_batches_tracked, block2.1.conv1.kernel, block2.1.norm1.bn.weight, block2.1.norm1.bn.bias, block2.1.norm1.bn.running_mean, block2.1.norm1.bn.running_var, block2.1.norm1.bn.num_batches_tracked, block2.1.conv2.kernel, block2.1.norm2.bn.weight, block2.1.norm2.bn.bias, block2.1.norm2.bn.running_mean, block2.1.norm2.bn.running_var, block2.1.norm2.bn.num_batches_tracked, block2.2.conv1.kernel, block2.2.norm1.bn.weight, block2.2.norm1.bn.bias, block2.2.norm1.bn.running_mean, block2.2.norm1.bn.running_var, block2.2.norm1.bn.num_batches_tracked, block2.2.conv2.kernel, block2.2.norm2.bn.weight, block2.2.norm2.bn.bias, block2.2.norm2.bn.running_mean, block2.2.norm2.bn.running_var, block2.2.norm2.bn.num_batches_tracked, conv3p4s2.kernel, bn3.bn.weight, bn3.bn.bias, bn3.bn.running_mean, bn3.bn.running_var, bn3.bn.num_batches_tracked, block3.0.conv1.kernel, block3.0.norm1.bn.weight, block3.0.norm1.bn.bias, block3.0.norm1.bn.running_mean, block3.0.norm1.bn.running_var, block3.0.norm1.bn.num_batches_tracked, block3.0.conv2.kernel, block3.0.norm2.bn.weight, block3.0.norm2.bn.bias, block3.0.norm2.bn.running_mean, block3.0.norm2.bn.running_var, block3.0.norm2.bn.num_batches_tracked, block3.0.downsample.0.kernel, block3.0.downsample.1.bn.weight, block3.0.downsample.1.bn.bias, block3.0.downsample.1.bn.running_mean, block3.0.downsample.1.bn.running_var, block3.0.downsample.1.bn.num_batches_tracked, block3.1.conv1.kernel, block3.1.norm1.bn.weight, block3.1.norm1.bn.bias, block3.1.norm1.bn.running_mean, block3.1.norm1.bn.running_var, block3.1.norm1.bn.num_batches_tracked, block3.1.conv2.kernel, block3.1.norm2.bn.weight, block3.1.norm2.bn.bias, block3.1.norm2.bn.running_mean, block3.1.norm2.bn.running_var, block3.1.norm2.bn.num_batches_tracked, block3.2.conv1.kernel, block3.2.norm1.bn.weight, block3.2.norm1.bn.bias, block3.2.norm1.bn.running_mean, block3.2.norm1.bn.running_var, block3.2.norm1.bn.num_batches_tracked, block3.2.conv2.kernel, block3.2.norm2.bn.weight, block3.2.norm2.bn.bias, block3.2.norm2.bn.running_mean, block3.2.norm2.bn.running_var, block3.2.norm2.bn.num_batches_tracked, block3.3.conv1.kernel, block3.3.norm1.bn.weight, block3.3.norm1.bn.bias, block3.3.norm1.bn.running_mean, block3.3.norm1.bn.running_var, block3.3.norm1.bn.num_batches_tracked, block3.3.conv2.kernel, block3.3.norm2.bn.weight, block3.3.norm2.bn.bias, block3.3.norm2.bn.running_mean, block3.3.norm2.bn.running_var, block3.3.norm2.bn.num_batches_tracked, conv4p8s2.kernel, bn4.bn.weight, bn4.bn.bias, bn4.bn.running_mean, bn4.bn.running_var, bn4.bn.num_batches_tracked, block4.0.conv1.kernel, block4.0.norm1.bn.weight, block4.0.norm1.bn.bias, block4.0.norm1.bn.running_mean, block4.0.norm1.bn.running_var, block4.0.norm1.bn.num_batches_tracked, block4.0.conv2.kernel, block4.0.norm2.bn.weight, block4.0.norm2.bn.bias, block4.0.norm2.bn.running_mean, block4.0.norm2.bn.running_var, block4.0.norm2.bn.num_batches_tracked, block4.0.downsample.0.kernel, block4.0.downsample.1.bn.weight, block4.0.downsample.1.bn.bias, block4.0.downsample.1.bn.running_mean, block4.0.downsample.1.bn.running_var, block4.0.downsample.1.bn.num_batches_tracked, block4.1.conv1.kernel, block4.1.norm1.bn.weight, block4.1.norm1.bn.bias, block4.1.norm1.bn.running_mean, block4.1.norm1.bn.running_var, block4.1.norm1.bn.num_batches_tracked, block4.1.conv2.kernel, block4.1.norm2.bn.weight, block4.1.norm2.bn.bias, block4.1.norm2.bn.running_mean, block4.1.norm2.bn.running_var, block4.1.norm2.bn.num_batches_tracked, block4.2.conv1.kernel, block4.2.norm1.bn.weight, block4.2.norm1.bn.bias, block4.2.norm1.bn.running_mean, block4.2.norm1.bn.running_var, block4.2.norm1.bn.num_batches_tracked, block4.2.conv2.kernel, block4.2.norm2.bn.weight, block4.2.norm2.bn.bias, block4.2.norm2.bn.running_mean, block4.2.norm2.bn.running_var, block4.2.norm2.bn.num_batches_tracked, block4.3.conv1.kernel, block4.3.norm1.bn.weight, block4.3.norm1.bn.bias, block4.3.norm1.bn.running_mean, block4.3.norm1.bn.running_var, block4.3.norm1.bn.num_batches_tracked, block4.3.conv2.kernel, block4.3.norm2.bn.weight, block4.3.norm2.bn.bias, block4.3.norm2.bn.running_mean, block4.3.norm2.bn.running_var, block4.3.norm2.bn.num_batches_tracked, block4.4.conv1.kernel, block4.4.norm1.bn.weight, block4.4.norm1.bn.bias, block4.4.norm1.bn.running_mean, block4.4.norm1.bn.running_var, block4.4.norm1.bn.num_batches_tracked, block4.4.conv2.kernel, block4.4.norm2.bn.weight, block4.4.norm2.bn.bias, block4.4.norm2.bn.running_mean, block4.4.norm2.bn.running_var, block4.4.norm2.bn.num_batches_tracked, block4.5.conv1.kernel, block4.5.norm1.bn.weight, block4.5.norm1.bn.bias, block4.5.norm1.bn.running_mean, block4.5.norm1.bn.running_var, block4.5.norm1.bn.num_batches_tracked, block4.5.conv2.kernel, block4.5.norm2.bn.weight, block4.5.norm2.bn.bias, block4.5.norm2.bn.running_mean, block4.5.norm2.bn.running_var, block4.5.norm2.bn.num_batches_tracked, convtr4p16s2.kernel, bntr4.bn.weight, bntr4.bn.bias, bntr4.bn.running_mean, bntr4.bn.running_var, bntr4.bn.num_batches_tracked, block5.0.conv1.kernel, block5.0.norm1.bn.weight, block5.0.norm1.bn.bias, block5.0.norm1.bn.running_mean, block5.0.norm1.bn.running_var, block5.0.norm1.bn.num_batches_tracked, block5.0.conv2.kernel, block5.0.norm2.bn.weight, block5.0.norm2.bn.bias, block5.0.norm2.bn.running_mean, block5.0.norm2.bn.running_var, block5.0.norm2.bn.num_batches_tracked, block5.0.downsample.0.kernel, block5.0.downsample.1.bn.weight, block5.0.downsample.1.bn.bias, block5.0.downsample.1.bn.running_mean, block5.0.downsample.1.bn.running_var, block5.0.downsample.1.bn.num_batches_tracked, block5.1.conv1.kernel, block5.1.norm1.bn.weight, block5.1.norm1.bn.bias, block5.1.norm1.bn.running_mean, block5.1.norm1.bn.running_var, block5.1.norm1.bn.num_batches_tracked, block5.1.conv2.kernel, block5.1.norm2.bn.weight, block5.1.norm2.bn.bias, block5.1.norm2.bn.running_mean, block5.1.norm2.bn.running_var, block5.1.norm2.bn.num_batches_tracked, convtr5p8s2.kernel, bntr5.bn.weight, bntr5.bn.bias, bntr5.bn.running_mean, bntr5.bn.running_var, bntr5.bn.num_batches_tracked, block6.0.conv1.kernel, block6.0.norm1.bn.weight, block6.0.norm1.bn.bias, block6.0.norm1.bn.running_mean, block6.0.norm1.bn.running_var, block6.0.norm1.bn.num_batches_tracked, block6.0.conv2.kernel, block6.0.norm2.bn.weight, block6.0.norm2.bn.bias, block6.0.norm2.bn.running_mean, block6.0.norm2.bn.running_var, block6.0.norm2.bn.num_batches_tracked, block6.0.downsample.0.kernel, block6.0.downsample.1.bn.weight, block6.0.downsample.1.bn.bias, block6.0.downsample.1.bn.running_mean, block6.0.downsample.1.bn.running_var, block6.0.downsample.1.bn.num_batches_tracked, block6.1.conv1.kernel, block6.1.norm1.bn.weight, block6.1.norm1.bn.bias, block6.1.norm1.bn.running_mean, block6.1.norm1.bn.running_var, block6.1.norm1.bn.num_batches_tracked, block6.1.conv2.kernel, block6.1.norm2.bn.weight, block6.1.norm2.bn.bias, block6.1.norm2.bn.running_mean, block6.1.norm2.bn.running_var, block6.1.norm2.bn.num_batches_tracked, convtr6p4s2.kernel, bntr6.bn.weight, bntr6.bn.bias, bntr6.bn.running_mean, bntr6.bn.running_var, bntr6.bn.num_batches_tracked, block7.0.conv1.kernel, block7.0.norm1.bn.weight, block7.0.norm1.bn.bias, block7.0.norm1.bn.running_mean, block7.0.norm1.bn.running_var, block7.0.norm1.bn.num_batches_tracked, block7.0.conv2.kernel, block7.0.norm2.bn.weight, block7.0.norm2.bn.bias, block7.0.norm2.bn.running_mean, block7.0.norm2.bn.running_var, block7.0.norm2.bn.num_batches_tracked, block7.0.downsample.0.kernel, block7.0.downsample.1.bn.weight, block7.0.downsample.1.bn.bias, block7.0.downsample.1.bn.running_mean, block7.0.downsample.1.bn.running_var, block7.0.downsample.1.bn.num_batches_tracked, block7.1.conv1.kernel, block7.1.norm1.bn.weight, block7.1.norm1.bn.bias, block7.1.norm1.bn.running_mean, block7.1.norm1.bn.running_var, block7.1.norm1.bn.num_batches_tracked, block7.1.conv2.kernel, block7.1.norm2.bn.weight, block7.1.norm2.bn.bias, block7.1.norm2.bn.running_mean, block7.1.norm2.bn.running_var, block7.1.norm2.bn.num_batches_tracked, convtr7p2s2.kernel, bntr7.bn.weight, bntr7.bn.bias, bntr7.bn.running_mean, bntr7.bn.running_var, bntr7.bn.num_batches_tracked, block8.0.conv1.kernel, block8.0.norm1.bn.weight, block8.0.norm1.bn.bias, block8.0.norm1.bn.running_mean, block8.0.norm1.bn.running_var, block8.0.norm1.bn.num_batches_tracked, block8.0.conv2.kernel, block8.0.norm2.bn.weight, block8.0.norm2.bn.bias, block8.0.norm2.bn.running_mean, block8.0.norm2.bn.running_var, block8.0.norm2.bn.num_batches_tracked, block8.0.downsample.0.kernel, block8.0.downsample.1.bn.weight, block8.0.downsample.1.bn.bias, block8.0.downsample.1.bn.running_mean, block8.0.downsample.1.bn.running_var, block8.0.downsample.1.bn.num_batches_tracked, block8.1.conv1.kernel, block8.1.norm1.bn.weight, block8.1.norm1.bn.bias, block8.1.norm1.bn.running_mean, block8.1.norm1.bn.running_var, block8.1.norm1.bn.num_batches_tracked, block8.1.conv2.kernel, block8.1.norm2.bn.weight, block8.1.norm2.bn.bias, block8.1.norm2.bn.running_mean, block8.1.norm2.bn.running_var, block8.1.norm2.bn.num_batches_tracked [2021-07-07 09:43:47,952][root][INFO] - => no weights.pth [2021-07-07 09:43:47,997][root][INFO] - ===> Start training on 8 GPUs, batch-size=48 [2021-07-07 09:44:06,120][root][INFO] - ===> Epoch[1](1/5): Loss 2.6084, Sem 2.6084 LR: 1.000e-01 Score 2.326 Data time: 9.2610, Forward time: 2.9780, Backward time: 5.5589, DDP time: 0.0030, Total iter time: 17.8702 [2021-07-07 09:44:17,610][root][INFO] - ===> Epoch[1](5/5): Loss 2.2260, Sem 2.2260 LR: 9.999e-02 Score 28.263 Data time: 0.2141, Forward time: 1.7050, Backward time: 2.0524, DDP time: 0.0042, Total iter time: 2.8133 [2021-07-07 09:44:32,309][root][INFO] - ===> Epoch[2](10/5): Loss 1.5998, Sem 1.5998 LR: 9.999e-02 Score 51.471 Data time: 0.2297, Forward time: 1.4801, Backward time: 1.7023, DDP time: 0.0046, Total iter time: 2.8772 [2021-07-07 09:44:46,359][root][INFO] - ===> Epoch[3](15/5): Loss 1.2524, Sem 1.2524 LR: 9.998e-02 Score 65.453 Data time: 0.3312, Forward time: 1.4146, Backward time: 1.5026, DDP time: 0.0045, Total iter time: 2.7498 [2021-07-07 09:44:59,451][root][INFO] - ===> Epoch[4](20/5): Loss 1.0577, Sem 1.0577 LR: 9.997e-02 Score 67.507 Data time: 0.2914, Forward time: 1.3707, Backward time: 1.3807, DDP time: 0.0043, Total iter time: 2.5706 [2021-07-07 09:45:13,337][root][INFO] - ===> Epoch[5](25/5): Loss 0.9758, Sem 0.9758 LR: 9.996e-02 Score 68.928 Data time: 0.2133, Forward time: 1.3464, Backward time: 1.3527, DDP time: 0.0044, Total iter time: 2.7385 [2021-07-07 09:45:28,517][root][INFO] - ===> Epoch[6](30/5): Loss 0.9008, Sem 0.9008 LR: 9.995e-02 Score 71.278 Data time: 0.2938, Forward time: 1.3580, Backward time: 1.3358, DDP time: 0.0044, Total iter time: 2.9908 [2021-07-07 09:45:42,890][root][INFO] - ===> Epoch[7](35/5): Loss 0.8164, Sem 0.8164 LR: 9.995e-02 Score 73.702 Data time: 0.2668, Forward time: 1.3548, Backward time: 1.3139, DDP time: 0.0044, Total iter time: 2.8152 [2021-07-07 09:45:56,850][root][INFO] - ===> Epoch[8](40/5): Loss 0.7451, Sem 0.7451 LR: 9.994e-02 Score 76.016 Data time: 0.2514, Forward time: 1.3559, Backward time: 1.2873, DDP time: 0.0043, Total iter time: 2.7407 [2021-07-07 09:46:12,133][root][INFO] - ===> Epoch[9](45/5): Loss 0.7050, Sem 0.7050 LR: 9.993e-02 Score 77.761 Data time: 0.2795, Forward time: 1.3576, Backward time: 1.2921, DDP time: 0.0042, Total iter time: 3.0132 [2021-07-07 09:46:25,802][root][INFO] - ===> Epoch[10](50/5): Loss 0.6716, Sem 0.6716 LR: 9.992e-02 Score 78.507 Data time: 0.2447, Forward time: 1.3446, Backward time: 1.2790, DDP time: 0.0041, Total iter time: 2.6733 [2021-07-07 09:46:39,770][root][INFO] - ===> Epoch[11](55/5): Loss 0.5911, Sem 0.5911 LR: 9.992e-02 Score 81.743 Data time: 0.2883, Forward time: 1.3383, Backward time: 1.2670, DDP time: 0.0043, Total iter time: 2.7459 [2021-07-07 09:46:54,800][root][INFO] - ===> Epoch[12](60/5): Loss 0.5700, Sem 0.5700 LR: 9.991e-02 Score 81.863 Data time: 0.2156, Forward time: 1.3379, Backward time: 1.2756, DDP time: 0.0043, Total iter time: 2.9585 [2021-07-07 09:47:08,560][root][INFO] - ===> Epoch[13](65/5): Loss 0.5243, Sem 0.5243 LR: 9.990e-02 Score 83.168 Data time: 0.2347, Forward time: 1.3314, Backward time: 1.2677, DDP time: 0.0043, Total iter time: 2.6978 [2021-07-07 09:47:22,732][root][INFO] - ===> Epoch[14](70/5): Loss 0.4946, Sem 0.4946 LR: 9.989e-02 Score 83.894 Data time: 0.1903, Forward time: 1.3166, Backward time: 1.2790, DDP time: 0.0042, Total iter time: 2.7893 [2021-07-07 09:47:35,781][root][INFO] - ===> Epoch[15](75/5): Loss 0.4498, Sem 0.4498 LR: 9.989e-02 Score 85.459 Data time: 0.2278, Forward time: 1.3130, Backward time: 1.2630, DDP time: 0.0042, Total iter time: 2.5571 [2021-07-07 09:47:50,253][root][INFO] - ===> Epoch[16](80/5): Loss 0.4129, Sem 0.4129 LR: 9.988e-02 Score 86.494 Data time: 0.2186, Forward time: 1.3071, Backward time: 1.2691, DDP time: 0.0042, Total iter time: 2.8442 [2021-07-07 09:48:04,926][root][INFO] - ===> Epoch[17](85/5): Loss 0.3954, Sem 0.3954 LR: 9.987e-02 Score 86.782 Data time: 0.2941, Forward time: 1.3105, Backward time: 1.2644, DDP time: 0.0042, Total iter time: 2.8784 [2021-07-07 09:48:20,558][root][INFO] - ===> Epoch[18](90/5): Loss 0.3908, Sem 0.3908 LR: 9.986e-02 Score 87.109 Data time: 0.2223, Forward time: 1.3150, Backward time: 1.2736, DDP time: 0.0042, Total iter time: 3.0839 [2021-07-07 09:48:34,506][root][INFO] - ===> Epoch[19](95/5): Loss 0.3524, Sem 0.3524 LR: 9.986e-02 Score 88.176 Data time: 0.2445, Forward time: 1.3135, Backward time: 1.2687, DDP time: 0.0042, Total iter time: 2.7524 [2021-07-07 09:48:48,859][root][INFO] - ===> Epoch[20](100/5): Loss 0.3401, Sem 0.3401 LR: 9.985e-02 Score 88.602 Data time: 0.1985, Forward time: 1.3153, Backward time: 1.2671, DDP time: 0.0042, Total iter time: 2.8140 [2021-07-07 09:49:03,348][root][INFO] - ===> Epoch[21](105/5): Loss 0.3409, Sem 0.3409 LR: 9.984e-02 Score 88.843 Data time: 0.2177, Forward time: 1.3139, Backward time: 1.2691, DDP time: 0.0042, Total iter time: 2.8501 [2021-07-07 09:49:17,705][root][INFO] - ===> Epoch[22](110/5): Loss 0.3002, Sem 0.3002 LR: 9.983e-02 Score 90.055 Data time: 0.2463, Forward time: 1.3079, Backward time: 1.2726, DDP time: 0.0042, Total iter time: 2.8151 [2021-07-07 09:49:31,720][root][INFO] - ===> Epoch[23](115/5): Loss 0.3116, Sem 0.3116 LR: 9.983e-02 Score 89.596 Data time: 0.2337, Forward time: 1.3118, Backward time: 1.2646, DDP time: 0.0043, Total iter time: 2.7465 [2021-07-07 09:49:45,973][root][INFO] - ===> Epoch[24](120/5): Loss 0.2982, Sem 0.2982 LR: 9.982e-02 Score 90.094 Data time: 0.2104, Forward time: 1.3083, Backward time: 1.2670, DDP time: 0.0042, Total iter time: 2.8005 [2021-07-07 09:49:59,839][root][INFO] - ===> Epoch[25](125/5): Loss 0.2806, Sem 0.2806 LR: 9.981e-02 Score 90.639 Data time: 0.2103, Forward time: 1.3088, Backward time: 1.2624, DDP time: 0.0042, Total iter time: 2.7149 [2021-07-07 09:50:13,607][root][INFO] - ===> Epoch[26](130/5): Loss 0.2693, Sem 0.2693 LR: 9.980e-02 Score 91.167 Data time: 0.2365, Forward time: 1.3032, Backward time: 1.2627, DDP time: 0.0042, Total iter time: 2.7040 [2021-07-07 09:50:27,975][root][INFO] - ===> Epoch[27](135/5): Loss 0.2546, Sem 0.2546 LR: 9.980e-02 Score 91.588 Data time: 0.2374, Forward time: 1.3035, Backward time: 1.2618, DDP time: 0.0043, Total iter time: 2.8216 [2021-07-07 09:50:41,922][root][INFO] - ===> Epoch[28](140/5): Loss 0.2628, Sem 0.2628 LR: 9.979e-02 Score 91.377 Data time: 0.3168, Forward time: 1.3047, Backward time: 1.2543, DDP time: 0.0043, Total iter time: 2.7355 [2021-07-07 09:50:56,252][root][INFO] - ===> Epoch[29](145/5): Loss 0.2445, Sem 0.2445 LR: 9.978e-02 Score 91.830 Data time: 0.1897, Forward time: 1.3016, Backward time: 1.2587, DDP time: 0.0043, Total iter time: 2.8245 [2021-07-07 09:51:10,065][root][INFO] - ===> Epoch[30](150/5): Loss 0.2153, Sem 0.2153 LR: 9.977e-02 Score 92.879 Data time: 0.2245, Forward time: 1.3020, Backward time: 1.2546, DDP time: 0.0043, Total iter time: 2.7013 [2021-07-07 09:51:24,407][root][INFO] - ===> Epoch[31](155/5): Loss 0.2322, Sem 0.2322 LR: 9.977e-02 Score 92.237 Data time: 0.2119, Forward time: 1.2975, Backward time: 1.2587, DDP time: 0.0043, Total iter time: 2.7995 [2021-07-07 09:51:38,271][root][INFO] - ===> Epoch[32](160/5): Loss 0.2184, Sem 0.2184 LR: 9.976e-02 Score 92.770 Data time: 0.2566, Forward time: 1.2952, Backward time: 1.2572, DDP time: 0.0043, Total iter time: 2.7275 [2021-07-07 09:51:52,603][root][INFO] - ===> Epoch[33](165/5): Loss 0.2119, Sem 0.2119 LR: 9.975e-02 Score 92.959 Data time: 0.2611, Forward time: 1.2950, Backward time: 1.2566, DDP time: 0.0043, Total iter time: 2.8212 [2021-07-07 09:52:06,471][root][INFO] - ===> Epoch[34](170/5): Loss 0.2152, Sem 0.2152 LR: 9.974e-02 Score 92.934 Data time: 0.1654, Forward time: 1.2935, Backward time: 1.2574, DDP time: 0.0043, Total iter time: 2.7292 [2021-07-07 09:52:20,050][root][INFO] - ===> Epoch[35](175/5): Loss 0.1981, Sem 0.1981 LR: 9.974e-02 Score 93.372 Data time: 0.1929, Forward time: 1.2891, Backward time: 1.2587, DDP time: 0.0043, Total iter time: 2.6687 [2021-07-07 09:52:35,691][root][INFO] - ===> Epoch[36](180/5): Loss 0.1962, Sem 0.1962 LR: 9.973e-02 Score 93.478 Data time: 0.2063, Forward time: 1.2918, Backward time: 1.2632, DDP time: 0.0044, Total iter time: 3.0681 [2021-07-07 09:52:50,159][root][INFO] - ===> Epoch[37](185/5): Loss 0.1802, Sem 0.1802 LR: 9.972e-02 Score 94.036 Data time: 0.2305, Forward time: 1.2925, Backward time: 1.2631, DDP time: 0.0043, Total iter time: 2.8428 [2021-07-07 09:53:05,516][root][INFO] - ===> Epoch[38](190/5): Loss 0.1885, Sem 0.1885 LR: 9.971e-02 Score 93.731 Data time: 0.2657, Forward time: 1.2920, Backward time: 1.2674, DDP time: 0.0043, Total iter time: 3.0135 [2021-07-07 09:53:19,764][root][INFO] - ===> Epoch[39](195/5): Loss 0.1887, Sem 0.1887 LR: 9.971e-02 Score 93.686 Data time: 0.1821, Forward time: 1.2898, Backward time: 1.2697, DDP time: 0.0044, Total iter time: 2.7996 [2021-07-07 09:53:34,051][root][INFO] - ===> Epoch[40](200/5): Loss 0.1914, Sem 0.1914 LR: 9.970e-02 Score 93.638 Data time: 0.2106, Forward time: 1.2886, Backward time: 1.2710, DDP time: 0.0043, Total iter time: 2.8150 [2021-07-07 09:53:34,053][root][INFO] - ===> Start testing [2021-07-07 09:54:10,778][root][INFO] - 68/68: Data time: 0.0421, Iter time: 0.1284 Loss 0.379 (AVG: 0.373) Score 85.964 (AVG: 89.335) mIOU 64.195 mAP 83.619 mAcc 72.120 IOU: 59.409 0.000 66.078 73.039 93.677 87.153 33.402 71.332 98.650 39.904 79.269 85.031 47.589 mAP: 83.272 6.316 90.045 90.004 99.247 97.813 69.316 93.088 99.797 83.580 94.257 95.093 85.219 mAcc: 74.906 0.000 81.968 80.606 97.222 95.554 38.521 86.520 99.636 41.999 84.365 94.915 61.345 [2021-07-07 09:54:10,778][root][INFO] - Finished test. Elapsed time: 36.7240 [2021-07-07 09:54:11,731][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2021-07-07 09:54:11,732][root][INFO] - Current best mIoU: 64.195 at iter 1 [2021-07-07 09:54:12,669][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 09:54:26,303][root][INFO] - ===> Epoch[41](205/5): Loss 0.1848, Sem 0.1848 LR: 9.969e-02 Score 93.791 Data time: 0.2667, Forward time: 1.2897, Backward time: 1.2657, DDP time: 0.0043, Total iter time: 2.6746 [2021-07-07 09:54:40,754][root][INFO] - ===> Epoch[42](210/5): Loss 0.1848, Sem 0.1848 LR: 9.968e-02 Score 93.763 Data time: 0.2182, Forward time: 1.2875, Backward time: 1.2686, DDP time: 0.0043, Total iter time: 2.8475 [2021-07-07 09:54:55,407][root][INFO] - ===> Epoch[43](215/5): Loss 0.1744, Sem 0.1744 LR: 9.968e-02 Score 94.092 Data time: 0.1954, Forward time: 1.2886, Backward time: 1.2697, DDP time: 0.0043, Total iter time: 2.8804 [2021-07-07 09:55:08,749][root][INFO] - ===> Epoch[44](220/5): Loss 0.1796, Sem 0.1796 LR: 9.967e-02 Score 93.853 Data time: 0.2173, Forward time: 1.2861, Backward time: 1.2676, DDP time: 0.0043, Total iter time: 2.6025 [2021-07-07 09:55:22,805][root][INFO] - ===> Epoch[45](225/5): Loss 0.1625, Sem 0.1625 LR: 9.966e-02 Score 94.571 Data time: 0.2672, Forward time: 1.2857, Backward time: 1.2660, DDP time: 0.0043, Total iter time: 2.7675 [2021-07-07 09:55:37,365][root][INFO] - ===> Epoch[46](230/5): Loss 0.1556, Sem 0.1556 LR: 9.965e-02 Score 94.775 Data time: 0.2892, Forward time: 1.2848, Backward time: 1.2663, DDP time: 0.0043, Total iter time: 2.8528 [2021-07-07 09:55:50,996][root][INFO] - ===> Epoch[47](235/5): Loss 0.1625, Sem 0.1625 LR: 9.965e-02 Score 94.594 Data time: 0.2374, Forward time: 1.2845, Backward time: 1.2632, DDP time: 0.0043, Total iter time: 2.6620 [2021-07-07 09:56:05,173][root][INFO] - ===> Epoch[48](240/5): Loss 0.1572, Sem 0.1572 LR: 9.964e-02 Score 94.701 Data time: 0.3095, Forward time: 1.2827, Backward time: 1.2626, DDP time: 0.0043, Total iter time: 2.7788 [2021-07-07 09:56:18,011][root][INFO] - ===> Epoch[49](245/5): Loss 0.1506, Sem 0.1506 LR: 9.963e-02 Score 94.881 Data time: 0.1969, Forward time: 1.2777, Backward time: 1.2623, DDP time: 0.0043, Total iter time: 2.5305 [2021-07-07 09:56:31,664][root][INFO] - ===> Epoch[50](250/5): Loss 0.1524, Sem 0.1524 LR: 9.962e-02 Score 94.833 Data time: 0.2456, Forward time: 1.2765, Backward time: 1.2607, DDP time: 0.0044, Total iter time: 2.6797 [2021-07-07 09:56:45,370][root][INFO] - ===> Epoch[51](255/5): Loss 0.1463, Sem 0.1463 LR: 9.962e-02 Score 95.079 Data time: 0.1872, Forward time: 1.2756, Backward time: 1.2603, DDP time: 0.0044, Total iter time: 2.6940 [2021-07-07 09:56:59,106][root][INFO] - ===> Epoch[52](260/5): Loss 0.1482, Sem 0.1482 LR: 9.961e-02 Score 94.965 Data time: 0.2161, Forward time: 1.2763, Backward time: 1.2580, DDP time: 0.0044, Total iter time: 2.7015 [2021-07-07 09:57:13,972][root][INFO] - ===> Epoch[53](265/5): Loss 0.1428, Sem 0.1428 LR: 9.960e-02 Score 95.182 Data time: 0.1652, Forward time: 1.2753, Backward time: 1.2626, DDP time: 0.0044, Total iter time: 2.9428 [2021-07-07 09:57:26,845][root][INFO] - ===> Epoch[54](270/5): Loss 0.1412, Sem 0.1412 LR: 9.959e-02 Score 95.215 Data time: 0.2036, Forward time: 1.2720, Backward time: 1.2615, DDP time: 0.0044, Total iter time: 2.5411 [2021-07-07 09:57:41,528][root][INFO] - ===> Epoch[55](275/5): Loss 0.1331, Sem 0.1331 LR: 9.959e-02 Score 95.489 Data time: 0.2426, Forward time: 1.2714, Backward time: 1.2634, DDP time: 0.0044, Total iter time: 2.8893 [2021-07-07 09:57:56,401][root][INFO] - ===> Epoch[56](280/5): Loss 0.1383, Sem 0.1383 LR: 9.958e-02 Score 95.318 Data time: 0.2311, Forward time: 1.2702, Backward time: 1.2667, DDP time: 0.0044, Total iter time: 2.9308 [2021-07-07 09:58:10,039][root][INFO] - ===> Epoch[57](285/5): Loss 0.1365, Sem 0.1365 LR: 9.957e-02 Score 95.398 Data time: 0.2057, Forward time: 1.2689, Backward time: 1.2664, DDP time: 0.0044, Total iter time: 2.6867 [2021-07-07 09:58:24,008][root][INFO] - ===> Epoch[58](290/5): Loss 0.1365, Sem 0.1365 LR: 9.956e-02 Score 95.378 Data time: 0.2446, Forward time: 1.2688, Backward time: 1.2654, DDP time: 0.0044, Total iter time: 2.7481 [2021-07-07 09:58:38,707][root][INFO] - ===> Epoch[59](295/5): Loss 0.1412, Sem 0.1412 LR: 9.956e-02 Score 95.197 Data time: 0.2119, Forward time: 1.2680, Backward time: 1.2679, DDP time: 0.0044, Total iter time: 2.8930 [2021-07-07 09:58:51,906][root][INFO] - ===> Epoch[60](300/5): Loss 0.1334, Sem 0.1334 LR: 9.955e-02 Score 95.519 Data time: 0.2347, Forward time: 1.2658, Backward time: 1.2667, DDP time: 0.0044, Total iter time: 2.6026 [2021-07-07 09:59:06,123][root][INFO] - ===> Epoch[61](305/5): Loss 0.1441, Sem 0.1441 LR: 9.954e-02 Score 95.085 Data time: 0.1972, Forward time: 1.2656, Backward time: 1.2674, DDP time: 0.0044, Total iter time: 2.7985 [2021-07-07 09:59:20,113][root][INFO] - ===> Epoch[62](310/5): Loss 0.1351, Sem 0.1351 LR: 9.953e-02 Score 95.459 Data time: 0.2336, Forward time: 1.2646, Backward time: 1.2676, DDP time: 0.0044, Total iter time: 2.7545 [2021-07-07 09:59:34,146][root][INFO] - ===> Epoch[63](315/5): Loss 0.1353, Sem 0.1353 LR: 9.953e-02 Score 95.374 Data time: 0.2647, Forward time: 1.2656, Backward time: 1.2653, DDP time: 0.0044, Total iter time: 2.7490 [2021-07-07 09:59:48,094][root][INFO] - ===> Epoch[64](320/5): Loss 0.1324, Sem 0.1324 LR: 9.952e-02 Score 95.496 Data time: 0.2485, Forward time: 1.2650, Backward time: 1.2650, DDP time: 0.0044, Total iter time: 2.7513 [2021-07-07 10:00:03,133][root][INFO] - ===> Epoch[65](325/5): Loss 0.1264, Sem 0.1264 LR: 9.951e-02 Score 95.664 Data time: 0.2686, Forward time: 1.2683, Backward time: 1.2637, DDP time: 0.0044, Total iter time: 2.9547 [2021-07-07 10:00:17,167][root][INFO] - ===> Epoch[66](330/5): Loss 0.1226, Sem 0.1226 LR: 9.950e-02 Score 95.849 Data time: 0.2101, Forward time: 1.2675, Backward time: 1.2642, DDP time: 0.0044, Total iter time: 2.7624 [2021-07-07 10:00:31,693][root][INFO] - ===> Epoch[67](335/5): Loss 0.1154, Sem 0.1154 LR: 9.950e-02 Score 96.090 Data time: 0.2280, Forward time: 1.2663, Backward time: 1.2661, DDP time: 0.0044, Total iter time: 2.8445 [2021-07-07 10:00:45,279][root][INFO] - ===> Epoch[68](340/5): Loss 0.1175, Sem 0.1175 LR: 9.949e-02 Score 95.981 Data time: 0.1859, Forward time: 1.2653, Backward time: 1.2657, DDP time: 0.0044, Total iter time: 2.6660 [2021-07-07 10:00:59,993][root][INFO] - ===> Epoch[69](345/5): Loss 0.1178, Sem 0.1178 LR: 9.948e-02 Score 96.033 Data time: 0.2368, Forward time: 1.2649, Backward time: 1.2673, DDP time: 0.0044, Total iter time: 2.8936 [2021-07-07 10:01:13,520][root][INFO] - ===> Epoch[70](350/5): Loss 0.1150, Sem 0.1150 LR: 9.947e-02 Score 96.039 Data time: 0.1973, Forward time: 1.2620, Backward time: 1.2684, DDP time: 0.0044, Total iter time: 2.6552 [2021-07-07 10:01:27,338][root][INFO] - ===> Epoch[71](355/5): Loss 0.1125, Sem 0.1125 LR: 9.947e-02 Score 96.155 Data time: 0.2526, Forward time: 1.2610, Backward time: 1.2679, DDP time: 0.0044, Total iter time: 2.7164 [2021-07-07 10:01:42,225][root][INFO] - ===> Epoch[72](360/5): Loss 0.1104, Sem 0.1104 LR: 9.946e-02 Score 96.302 Data time: 0.2518, Forward time: 1.2620, Backward time: 1.2681, DDP time: 0.0044, Total iter time: 2.9031 [2021-07-07 10:01:55,413][root][INFO] - ===> Epoch[73](365/5): Loss 0.1037, Sem 0.1037 LR: 9.945e-02 Score 96.429 Data time: 0.2281, Forward time: 1.2619, Backward time: 1.2655, DDP time: 0.0044, Total iter time: 2.5881 [2021-07-07 10:02:08,838][root][INFO] - ===> Epoch[74](370/5): Loss 0.1158, Sem 0.1158 LR: 9.944e-02 Score 96.080 Data time: 0.2823, Forward time: 1.2599, Backward time: 1.2646, DDP time: 0.0044, Total iter time: 2.6305 [2021-07-07 10:02:22,777][root][INFO] - ===> Epoch[75](375/5): Loss 0.1046, Sem 0.1046 LR: 9.944e-02 Score 96.445 Data time: 0.2972, Forward time: 1.2600, Backward time: 1.2630, DDP time: 0.0044, Total iter time: 2.7378 [2021-07-07 10:02:37,507][root][INFO] - ===> Epoch[76](380/5): Loss 0.1085, Sem 0.1085 LR: 9.943e-02 Score 96.239 Data time: 0.2161, Forward time: 1.2590, Backward time: 1.2652, DDP time: 0.0044, Total iter time: 2.8838 [2021-07-07 10:02:51,818][root][INFO] - ===> Epoch[77](385/5): Loss 0.1016, Sem 0.1016 LR: 9.942e-02 Score 96.515 Data time: 0.2840, Forward time: 1.2596, Backward time: 1.2643, DDP time: 0.0044, Total iter time: 2.8079 [2021-07-07 10:03:05,445][root][INFO] - ===> Epoch[78](390/5): Loss 0.1044, Sem 0.1044 LR: 9.941e-02 Score 96.389 Data time: 0.2198, Forward time: 1.2580, Backward time: 1.2645, DDP time: 0.0044, Total iter time: 2.6838 [2021-07-07 10:03:19,589][root][INFO] - ===> Epoch[79](395/5): Loss 0.1008, Sem 0.1008 LR: 9.941e-02 Score 96.553 Data time: 0.2655, Forward time: 1.2594, Backward time: 1.2628, DDP time: 0.0044, Total iter time: 2.7818 [2021-07-07 10:03:33,834][root][INFO] - ===> Epoch[80](400/5): Loss 0.1006, Sem 0.1006 LR: 9.940e-02 Score 96.551 Data time: 0.2005, Forward time: 1.2596, Backward time: 1.2631, DDP time: 0.0044, Total iter time: 2.8028 [2021-07-07 10:03:33,836][root][INFO] - ===> Start testing [2021-07-07 10:04:08,572][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1339 Loss 0.363 (AVG: 0.374) Score 87.708 (AVG: 90.237) mIOU 68.197 mAP 84.563 mAcc 75.480 IOU: 62.671 0.000 70.635 75.389 94.707 89.802 38.088 69.965 98.693 66.540 81.036 84.806 54.226 mAP: 83.237 4.061 94.975 90.990 99.375 98.411 66.299 92.909 99.824 89.853 94.117 95.154 90.116 mAcc: 74.999 0.000 90.555 83.288 97.292 94.112 46.636 79.612 99.717 68.923 89.927 96.077 60.110 [2021-07-07 10:04:08,573][root][INFO] - Finished test. Elapsed time: 34.7362 [2021-07-07 10:04:09,628][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2021-07-07 10:04:09,629][root][INFO] - Current best mIoU: 68.197 at iter 1 [2021-07-07 10:04:10,530][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 10:04:23,941][root][INFO] - ===> Epoch[81](405/5): Loss 0.0975, Sem 0.0975 LR: 9.939e-02 Score 96.705 Data time: 0.2375, Forward time: 1.2579, Backward time: 1.2629, DDP time: 0.0044, Total iter time: 2.6457 [2021-07-07 10:04:37,781][root][INFO] - ===> Epoch[82](410/5): Loss 0.0995, Sem 0.0995 LR: 9.938e-02 Score 96.558 Data time: 0.2603, Forward time: 1.2581, Backward time: 1.2615, DDP time: 0.0044, Total iter time: 2.7173 [2021-07-07 10:04:52,249][root][INFO] - ===> Epoch[83](415/5): Loss 0.0978, Sem 0.0978 LR: 9.938e-02 Score 96.669 Data time: 0.1912, Forward time: 1.2564, Backward time: 1.2641, DDP time: 0.0044, Total iter time: 2.8421 [2021-07-07 10:05:06,501][root][INFO] - ===> Epoch[84](420/5): Loss 0.1016, Sem 0.1016 LR: 9.937e-02 Score 96.491 Data time: 0.2063, Forward time: 1.2563, Backward time: 1.2646, DDP time: 0.0044, Total iter time: 2.7967 [2021-07-07 10:05:20,243][root][INFO] - ===> Epoch[85](425/5): Loss 0.0924, Sem 0.0924 LR: 9.936e-02 Score 96.831 Data time: 0.2364, Forward time: 1.2561, Backward time: 1.2638, DDP time: 0.0044, Total iter time: 2.7067 [2021-07-07 10:05:34,582][root][INFO] - ===> Epoch[86](430/5): Loss 0.0960, Sem 0.0960 LR: 9.935e-02 Score 96.675 Data time: 0.2304, Forward time: 1.2556, Backward time: 1.2647, DDP time: 0.0044, Total iter time: 2.8249 [2021-07-07 10:05:48,008][root][INFO] - ===> Epoch[87](435/5): Loss 0.0914, Sem 0.0914 LR: 9.935e-02 Score 96.885 Data time: 0.2270, Forward time: 1.2546, Backward time: 1.2639, DDP time: 0.0044, Total iter time: 2.6324 [2021-07-07 10:06:01,779][root][INFO] - ===> Epoch[88](440/5): Loss 0.0972, Sem 0.0972 LR: 9.934e-02 Score 96.652 Data time: 0.1844, Forward time: 1.2537, Backward time: 1.2643, DDP time: 0.0044, Total iter time: 2.7074 [2021-07-07 10:06:16,268][root][INFO] - ===> Epoch[89](445/5): Loss 0.0895, Sem 0.0895 LR: 9.933e-02 Score 96.941 Data time: 0.2076, Forward time: 1.2541, Backward time: 1.2648, DDP time: 0.0044, Total iter time: 2.8441 [2021-07-07 10:06:29,735][root][INFO] - ===> Epoch[90](450/5): Loss 0.0925, Sem 0.0925 LR: 9.932e-02 Score 96.804 Data time: 0.2482, Forward time: 1.2537, Backward time: 1.2634, DDP time: 0.0044, Total iter time: 2.6432 [2021-07-07 10:06:43,963][root][INFO] - ===> Epoch[91](455/5): Loss 0.0922, Sem 0.0922 LR: 9.932e-02 Score 96.785 Data time: 0.1989, Forward time: 1.2530, Backward time: 1.2644, DDP time: 0.0044, Total iter time: 2.7895 [2021-07-07 10:06:56,878][root][INFO] - ===> Epoch[92](460/5): Loss 0.0874, Sem 0.0874 LR: 9.931e-02 Score 96.982 Data time: 0.2713, Forward time: 1.2521, Backward time: 1.2621, DDP time: 0.0044, Total iter time: 2.5268 [2021-07-07 10:07:10,903][root][INFO] - ===> Epoch[93](465/5): Loss 0.0859, Sem 0.0859 LR: 9.930e-02 Score 97.048 Data time: 0.2920, Forward time: 1.2528, Backward time: 1.2604, DDP time: 0.0044, Total iter time: 2.7478 [2021-07-07 10:07:25,391][root][INFO] - ===> Epoch[94](470/5): Loss 0.0910, Sem 0.0910 LR: 9.929e-02 Score 96.842 Data time: 0.2496, Forward time: 1.2529, Backward time: 1.2608, DDP time: 0.0044, Total iter time: 2.8508 [2021-07-07 10:07:40,700][root][INFO] - ===> Epoch[95](475/5): Loss 0.0839, Sem 0.0839 LR: 9.929e-02 Score 97.082 Data time: 0.2407, Forward time: 1.2537, Backward time: 1.2622, DDP time: 0.0044, Total iter time: 2.9980 [2021-07-07 10:07:54,558][root][INFO] - ===> Epoch[96](480/5): Loss 0.0908, Sem 0.0908 LR: 9.928e-02 Score 96.893 Data time: 0.2290, Forward time: 1.2547, Backward time: 1.2605, DDP time: 0.0044, Total iter time: 2.7109 [2021-07-07 10:08:08,465][root][INFO] - ===> Epoch[97](485/5): Loss 0.0855, Sem 0.0855 LR: 9.927e-02 Score 97.030 Data time: 0.2581, Forward time: 1.2550, Backward time: 1.2596, DDP time: 0.0044, Total iter time: 2.7406 [2021-07-07 10:08:23,296][root][INFO] - ===> Epoch[98](490/5): Loss 0.0867, Sem 0.0867 LR: 9.926e-02 Score 97.010 Data time: 0.2626, Forward time: 1.2557, Backward time: 1.2600, DDP time: 0.0044, Total iter time: 2.9202 [2021-07-07 10:08:37,618][root][INFO] - ===> Epoch[99](495/5): Loss 0.0836, Sem 0.0836 LR: 9.926e-02 Score 97.080 Data time: 0.2248, Forward time: 1.2557, Backward time: 1.2605, DDP time: 0.0044, Total iter time: 2.8203 [2021-07-07 10:08:51,487][root][INFO] - ===> Epoch[100](500/5): Loss 0.0849, Sem 0.0849 LR: 9.925e-02 Score 97.063 Data time: 0.2599, Forward time: 1.2553, Backward time: 1.2600, DDP time: 0.0044, Total iter time: 2.7282 [2021-07-07 10:09:04,579][root][INFO] - ===> Epoch[101](505/5): Loss 0.0889, Sem 0.0889 LR: 9.924e-02 Score 96.930 Data time: 0.1918, Forward time: 1.2544, Backward time: 1.2592, DDP time: 0.0044, Total iter time: 2.5703 [2021-07-07 10:09:18,447][root][INFO] - ===> Epoch[102](510/5): Loss 0.0840, Sem 0.0840 LR: 9.923e-02 Score 97.104 Data time: 0.2715, Forward time: 1.2529, Backward time: 1.2596, DDP time: 0.0044, Total iter time: 2.7198 [2021-07-07 10:09:32,739][root][INFO] - ===> Epoch[103](515/5): Loss 0.0848, Sem 0.0848 LR: 9.923e-02 Score 97.078 Data time: 0.2253, Forward time: 1.2528, Backward time: 1.2601, DDP time: 0.0044, Total iter time: 2.8241 [2021-07-07 10:09:47,320][root][INFO] - ===> Epoch[104](520/5): Loss 0.0849, Sem 0.0849 LR: 9.922e-02 Score 97.101 Data time: 0.2698, Forward time: 1.2535, Backward time: 1.2600, DDP time: 0.0044, Total iter time: 2.8674 [2021-07-07 10:10:01,451][root][INFO] - ===> Epoch[105](525/5): Loss 0.0840, Sem 0.0840 LR: 9.921e-02 Score 97.092 Data time: 0.2049, Forward time: 1.2538, Backward time: 1.2600, DDP time: 0.0044, Total iter time: 2.7869 [2021-07-07 10:10:15,026][root][INFO] - ===> Epoch[106](530/5): Loss 0.0814, Sem 0.0814 LR: 9.920e-02 Score 97.176 Data time: 0.2178, Forward time: 1.2545, Backward time: 1.2585, DDP time: 0.0044, Total iter time: 2.6750 [2021-07-07 10:10:29,245][root][INFO] - ===> Epoch[107](535/5): Loss 0.0799, Sem 0.0799 LR: 9.920e-02 Score 97.227 Data time: 0.2289, Forward time: 1.2546, Backward time: 1.2586, DDP time: 0.0044, Total iter time: 2.7979 [2021-07-07 10:10:43,312][root][INFO] - ===> Epoch[108](540/5): Loss 0.0850, Sem 0.0850 LR: 9.919e-02 Score 97.054 Data time: 0.2692, Forward time: 1.2541, Backward time: 1.2586, DDP time: 0.0044, Total iter time: 2.7737 [2021-07-07 10:10:57,387][root][INFO] - ===> Epoch[109](545/5): Loss 0.0811, Sem 0.0811 LR: 9.918e-02 Score 97.217 Data time: 0.2712, Forward time: 1.2548, Backward time: 1.2575, DDP time: 0.0044, Total iter time: 2.7649 [2021-07-07 10:11:11,332][root][INFO] - ===> Epoch[110](550/5): Loss 0.0779, Sem 0.0779 LR: 9.917e-02 Score 97.291 Data time: 0.2094, Forward time: 1.2557, Backward time: 1.2565, DDP time: 0.0044, Total iter time: 2.7367 [2021-07-07 10:11:25,302][root][INFO] - ===> Epoch[111](555/5): Loss 0.0827, Sem 0.0827 LR: 9.917e-02 Score 97.141 Data time: 0.2680, Forward time: 1.2556, Backward time: 1.2560, DDP time: 0.0044, Total iter time: 2.7502 [2021-07-07 10:11:38,521][root][INFO] - ===> Epoch[112](560/5): Loss 0.0773, Sem 0.0773 LR: 9.916e-02 Score 97.276 Data time: 0.1860, Forward time: 1.2554, Backward time: 1.2550, DDP time: 0.0044, Total iter time: 2.5922 [2021-07-07 10:11:52,672][root][INFO] - ===> Epoch[113](565/5): Loss 0.0778, Sem 0.0778 LR: 9.915e-02 Score 97.282 Data time: 0.2278, Forward time: 1.2552, Backward time: 1.2553, DDP time: 0.0044, Total iter time: 2.7841 [2021-07-07 10:12:06,045][root][INFO] - ===> Epoch[114](570/5): Loss 0.0788, Sem 0.0788 LR: 9.914e-02 Score 97.253 Data time: 0.2362, Forward time: 1.2552, Backward time: 1.2539, DDP time: 0.0044, Total iter time: 2.6176 [2021-07-07 10:12:20,731][root][INFO] - ===> Epoch[115](575/5): Loss 0.0808, Sem 0.0808 LR: 9.914e-02 Score 97.185 Data time: 0.2397, Forward time: 1.2564, Backward time: 1.2537, DDP time: 0.0044, Total iter time: 2.8868 [2021-07-07 10:12:34,273][root][INFO] - ===> Epoch[116](580/5): Loss 0.0804, Sem 0.0804 LR: 9.913e-02 Score 97.247 Data time: 0.2041, Forward time: 1.2572, Backward time: 1.2522, DDP time: 0.0044, Total iter time: 2.6590 [2021-07-07 10:12:47,899][root][INFO] - ===> Epoch[117](585/5): Loss 0.0811, Sem 0.0811 LR: 9.912e-02 Score 97.227 Data time: 0.2089, Forward time: 1.2567, Backward time: 1.2520, DDP time: 0.0044, Total iter time: 2.6848 [2021-07-07 10:13:01,895][root][INFO] - ===> Epoch[118](590/5): Loss 0.0787, Sem 0.0787 LR: 9.911e-02 Score 97.276 Data time: 0.2763, Forward time: 1.2557, Backward time: 1.2523, DDP time: 0.0044, Total iter time: 2.7477 [2021-07-07 10:13:15,488][root][INFO] - ===> Epoch[119](595/5): Loss 0.0772, Sem 0.0772 LR: 9.911e-02 Score 97.315 Data time: 0.2656, Forward time: 1.2562, Backward time: 1.2508, DDP time: 0.0044, Total iter time: 2.6725 [2021-07-07 10:13:30,221][root][INFO] - ===> Epoch[120](600/5): Loss 0.0743, Sem 0.0743 LR: 9.910e-02 Score 97.384 Data time: 0.2561, Forward time: 1.2557, Backward time: 1.2520, DDP time: 0.0044, Total iter time: 2.8947 [2021-07-07 10:13:30,225][root][INFO] - ===> Start testing [2021-07-07 10:14:03,656][root][INFO] - 68/68: Data time: 0.0016, Iter time: 0.1223 Loss 0.186 (AVG: 0.412) Score 93.559 (AVG: 90.450) mIOU 68.555 mAP 83.663 mAcc 75.326 IOU: 63.310 0.000 73.177 76.243 94.614 89.679 29.411 78.678 98.586 65.517 79.999 85.390 56.616 mAP: 84.299 5.468 90.865 90.680 99.253 98.583 58.556 95.653 99.830 83.979 94.176 94.813 91.460 mAcc: 80.327 0.000 88.471 82.068 96.642 95.794 33.743 90.196 99.653 66.525 87.029 95.498 63.291 [2021-07-07 10:14:03,656][root][INFO] - Finished test. Elapsed time: 33.4306 [2021-07-07 10:14:04,713][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2021-07-07 10:14:04,714][root][INFO] - Current best mIoU: 68.555 at iter 1 [2021-07-07 10:14:05,649][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 10:14:19,896][root][INFO] - ===> Epoch[121](605/5): Loss 0.0708, Sem 0.0708 LR: 9.909e-02 Score 97.560 Data time: 0.2680, Forward time: 1.2563, Backward time: 1.2514, DDP time: 0.0044, Total iter time: 2.8026 [2021-07-07 10:14:33,571][root][INFO] - ===> Epoch[122](610/5): Loss 0.0785, Sem 0.0785 LR: 9.908e-02 Score 97.223 Data time: 0.2284, Forward time: 1.2569, Backward time: 1.2501, DDP time: 0.0044, Total iter time: 2.6859 [2021-07-07 10:14:47,602][root][INFO] - ===> Epoch[123](615/5): Loss 0.0720, Sem 0.0720 LR: 9.908e-02 Score 97.466 Data time: 0.1967, Forward time: 1.2572, Backward time: 1.2500, DDP time: 0.0044, Total iter time: 2.7616 [2021-07-07 10:15:01,789][root][INFO] - ===> Epoch[124](620/5): Loss 0.0707, Sem 0.0707 LR: 9.907e-02 Score 97.580 Data time: 0.2427, Forward time: 1.2574, Backward time: 1.2499, DDP time: 0.0044, Total iter time: 2.7805 [2021-07-07 10:15:15,428][root][INFO] - ===> Epoch[125](625/5): Loss 0.0763, Sem 0.0763 LR: 9.906e-02 Score 97.352 Data time: 0.1859, Forward time: 1.2576, Backward time: 1.2492, DDP time: 0.0044, Total iter time: 2.6730 [2021-07-07 10:15:30,577][root][INFO] - ===> Epoch[126](630/5): Loss 0.0704, Sem 0.0704 LR: 9.905e-02 Score 97.537 Data time: 0.2294, Forward time: 1.2577, Backward time: 1.2508, DDP time: 0.0044, Total iter time: 2.9849 [2021-07-07 10:15:44,401][root][INFO] - ===> Epoch[127](635/5): Loss 0.0712, Sem 0.0712 LR: 9.905e-02 Score 97.504 Data time: 0.2310, Forward time: 1.2575, Backward time: 1.2505, DDP time: 0.0044, Total iter time: 2.7150 [2021-07-07 10:15:58,650][root][INFO] - ===> Epoch[128](640/5): Loss 0.0718, Sem 0.0718 LR: 9.904e-02 Score 97.529 Data time: 0.2019, Forward time: 1.2568, Backward time: 1.2516, DDP time: 0.0044, Total iter time: 2.8045 [2021-07-07 10:16:12,734][root][INFO] - ===> Epoch[129](645/5): Loss 0.0722, Sem 0.0722 LR: 9.903e-02 Score 97.464 Data time: 0.1455, Forward time: 1.2564, Backward time: 1.2526, DDP time: 0.0044, Total iter time: 2.7776 [2021-07-07 10:16:26,881][root][INFO] - ===> Epoch[130](650/5): Loss 0.0724, Sem 0.0724 LR: 9.902e-02 Score 97.467 Data time: 0.2787, Forward time: 1.2571, Backward time: 1.2517, DDP time: 0.0044, Total iter time: 2.7861 [2021-07-07 10:16:41,285][root][INFO] - ===> Epoch[131](655/5): Loss 0.0710, Sem 0.0710 LR: 9.902e-02 Score 97.535 Data time: 0.2052, Forward time: 1.2570, Backward time: 1.2525, DDP time: 0.0044, Total iter time: 2.8425 [2021-07-07 10:16:55,404][root][INFO] - ===> Epoch[132](660/5): Loss 0.0719, Sem 0.0719 LR: 9.901e-02 Score 97.490 Data time: 0.2447, Forward time: 1.2575, Backward time: 1.2519, DDP time: 0.0044, Total iter time: 2.7794 [2021-07-07 10:17:08,994][root][INFO] - ===> Epoch[133](665/5): Loss 0.0697, Sem 0.0697 LR: 9.900e-02 Score 97.586 Data time: 0.2289, Forward time: 1.2583, Backward time: 1.2504, DDP time: 0.0044, Total iter time: 2.6751 [2021-07-07 10:17:23,369][root][INFO] - ===> Epoch[134](670/5): Loss 0.0704, Sem 0.0704 LR: 9.899e-02 Score 97.538 Data time: 0.2051, Forward time: 1.2578, Backward time: 1.2514, DDP time: 0.0044, Total iter time: 2.8358 [2021-07-07 10:17:37,121][root][INFO] - ===> Epoch[135](675/5): Loss 0.0676, Sem 0.0676 LR: 9.899e-02 Score 97.634 Data time: 0.2604, Forward time: 1.2581, Backward time: 1.2504, DDP time: 0.0044, Total iter time: 2.7077 [2021-07-07 10:17:51,410][root][INFO] - ===> Epoch[136](680/5): Loss 0.0657, Sem 0.0657 LR: 9.898e-02 Score 97.667 Data time: 0.2029, Forward time: 1.2574, Backward time: 1.2516, DDP time: 0.0044, Total iter time: 2.8163 [2021-07-07 10:18:04,920][root][INFO] - ===> Epoch[137](685/5): Loss 0.0672, Sem 0.0672 LR: 9.897e-02 Score 97.666 Data time: 0.2141, Forward time: 1.2567, Backward time: 1.2515, DDP time: 0.0044, Total iter time: 2.6620 [2021-07-07 10:18:18,348][root][INFO] - ===> Epoch[138](690/5): Loss 0.0705, Sem 0.0705 LR: 9.896e-02 Score 97.554 Data time: 0.2694, Forward time: 1.2562, Backward time: 1.2507, DDP time: 0.0044, Total iter time: 2.6282 [2021-07-07 10:18:32,418][root][INFO] - ===> Epoch[139](695/5): Loss 0.0693, Sem 0.0693 LR: 9.896e-02 Score 97.595 Data time: 0.1980, Forward time: 1.2561, Backward time: 1.2510, DDP time: 0.0044, Total iter time: 2.7671 [2021-07-07 10:18:46,464][root][INFO] - ===> Epoch[140](700/5): Loss 0.0658, Sem 0.0658 LR: 9.895e-02 Score 97.685 Data time: 0.2101, Forward time: 1.2562, Backward time: 1.2510, DDP time: 0.0044, Total iter time: 2.7616 [2021-07-07 10:19:00,884][root][INFO] - ===> Epoch[141](705/5): Loss 0.0699, Sem 0.0699 LR: 9.894e-02 Score 97.572 Data time: 0.2036, Forward time: 1.2558, Backward time: 1.2519, DDP time: 0.0044, Total iter time: 2.8389 [2021-07-07 10:19:15,041][root][INFO] - ===> Epoch[142](710/5): Loss 0.0653, Sem 0.0653 LR: 9.893e-02 Score 97.686 Data time: 0.2745, Forward time: 1.2565, Backward time: 1.2509, DDP time: 0.0044, Total iter time: 2.7666 [2021-07-07 10:19:29,204][root][INFO] - ===> Epoch[143](715/5): Loss 0.0653, Sem 0.0653 LR: 9.893e-02 Score 97.715 Data time: 0.2101, Forward time: 1.2571, Backward time: 1.2505, DDP time: 0.0044, Total iter time: 2.7795 [2021-07-07 10:19:42,652][root][INFO] - ===> Epoch[144](720/5): Loss 0.0641, Sem 0.0641 LR: 9.892e-02 Score 97.746 Data time: 0.1853, Forward time: 1.2576, Backward time: 1.2494, DDP time: 0.0044, Total iter time: 2.6266 [2021-07-07 10:19:56,438][root][INFO] - ===> Epoch[145](725/5): Loss 0.0616, Sem 0.0616 LR: 9.891e-02 Score 97.823 Data time: 0.2371, Forward time: 1.2579, Backward time: 1.2486, DDP time: 0.0044, Total iter time: 2.7070 [2021-07-07 10:20:10,100][root][INFO] - ===> Epoch[146](730/5): Loss 0.0659, Sem 0.0659 LR: 9.890e-02 Score 97.699 Data time: 0.2441, Forward time: 1.2580, Backward time: 1.2479, DDP time: 0.0044, Total iter time: 2.6931 [2021-07-07 10:20:24,268][root][INFO] - ===> Epoch[147](735/5): Loss 0.0666, Sem 0.0666 LR: 9.890e-02 Score 97.655 Data time: 0.2510, Forward time: 1.2586, Backward time: 1.2473, DDP time: 0.0044, Total iter time: 2.7934 [2021-07-07 10:20:38,519][root][INFO] - ===> Epoch[148](740/5): Loss 0.0627, Sem 0.0627 LR: 9.889e-02 Score 97.819 Data time: 0.2136, Forward time: 1.2588, Backward time: 1.2474, DDP time: 0.0044, Total iter time: 2.8078 [2021-07-07 10:20:52,569][root][INFO] - ===> Epoch[149](745/5): Loss 0.0652, Sem 0.0652 LR: 9.888e-02 Score 97.689 Data time: 0.1770, Forward time: 1.2576, Backward time: 1.2489, DDP time: 0.0044, Total iter time: 2.7691 [2021-07-07 10:21:06,660][root][INFO] - ===> Epoch[150](750/5): Loss 0.0667, Sem 0.0667 LR: 9.887e-02 Score 97.657 Data time: 0.3008, Forward time: 1.2575, Backward time: 1.2485, DDP time: 0.0044, Total iter time: 2.7722 [2021-07-07 10:21:20,812][root][INFO] - ===> Epoch[151](755/5): Loss 0.0673, Sem 0.0673 LR: 9.887e-02 Score 97.644 Data time: 0.1728, Forward time: 1.2569, Backward time: 1.2495, DDP time: 0.0044, Total iter time: 2.7917 [2021-07-07 10:21:34,879][root][INFO] - ===> Epoch[152](760/5): Loss 0.0622, Sem 0.0622 LR: 9.886e-02 Score 97.852 Data time: 0.2073, Forward time: 1.2567, Backward time: 1.2499, DDP time: 0.0044, Total iter time: 2.7691 [2021-07-07 10:21:49,657][root][INFO] - ===> Epoch[153](765/5): Loss 0.0646, Sem 0.0646 LR: 9.885e-02 Score 97.701 Data time: 0.2410, Forward time: 1.2566, Backward time: 1.2507, DDP time: 0.0044, Total iter time: 2.9051 [2021-07-07 10:22:03,336][root][INFO] - ===> Epoch[154](770/5): Loss 0.0645, Sem 0.0645 LR: 9.884e-02 Score 97.723 Data time: 0.2615, Forward time: 1.2561, Backward time: 1.2504, DDP time: 0.0044, Total iter time: 2.6838 [2021-07-07 10:22:18,380][root][INFO] - ===> Epoch[155](775/5): Loss 0.0668, Sem 0.0668 LR: 9.884e-02 Score 97.629 Data time: 0.2068, Forward time: 1.2556, Backward time: 1.2522, DDP time: 0.0043, Total iter time: 2.9629 [2021-07-07 10:22:31,944][root][INFO] - ===> Epoch[156](780/5): Loss 0.0665, Sem 0.0665 LR: 9.883e-02 Score 97.631 Data time: 0.2786, Forward time: 1.2553, Backward time: 1.2514, DDP time: 0.0043, Total iter time: 2.6666 [2021-07-07 10:22:45,400][root][INFO] - ===> Epoch[157](785/5): Loss 0.0672, Sem 0.0672 LR: 9.882e-02 Score 97.650 Data time: 0.2506, Forward time: 1.2546, Backward time: 1.2512, DDP time: 0.0043, Total iter time: 2.6462 [2021-07-07 10:22:58,874][root][INFO] - ===> Epoch[158](790/5): Loss 0.0636, Sem 0.0636 LR: 9.881e-02 Score 97.761 Data time: 0.2322, Forward time: 1.2545, Backward time: 1.2506, DDP time: 0.0043, Total iter time: 2.6466 [2021-07-07 10:23:12,941][root][INFO] - ===> Epoch[159](795/5): Loss 0.0640, Sem 0.0640 LR: 9.881e-02 Score 97.771 Data time: 0.2166, Forward time: 1.2543, Backward time: 1.2508, DDP time: 0.0043, Total iter time: 2.7651 [2021-07-07 10:23:27,426][root][INFO] - ===> Epoch[160](800/5): Loss 0.0615, Sem 0.0615 LR: 9.880e-02 Score 97.854 Data time: 0.2430, Forward time: 1.2544, Backward time: 1.2511, DDP time: 0.0043, Total iter time: 2.8532 [2021-07-07 10:23:27,427][root][INFO] - ===> Start testing [2021-07-07 10:24:00,726][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1267 Loss 0.145 (AVG: 0.411) Score 95.171 (AVG: 90.806) mIOU 70.335 mAP 85.298 mAcc 76.228 IOU: 64.675 0.000 77.175 76.324 95.047 90.712 30.525 80.801 98.667 79.359 80.644 85.147 55.284 mAP: 85.190 5.302 92.963 90.987 99.189 98.719 62.647 96.574 99.832 95.452 94.209 95.345 92.470 mAcc: 81.243 0.000 83.990 83.063 97.067 96.583 32.163 88.581 99.478 84.096 88.142 96.286 60.271 [2021-07-07 10:24:00,726][root][INFO] - Finished test. Elapsed time: 33.2983 [2021-07-07 10:24:01,804][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2021-07-07 10:24:01,805][root][INFO] - Current best mIoU: 70.335 at iter 1 [2021-07-07 10:24:02,750][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 10:24:15,433][root][INFO] - ===> Epoch[161](805/5): Loss 0.0605, Sem 0.0605 LR: 9.879e-02 Score 97.863 Data time: 0.2240, Forward time: 1.2544, Backward time: 1.2494, DDP time: 0.0043, Total iter time: 2.4806 [2021-07-07 10:24:29,594][root][INFO] - ===> Epoch[162](810/5): Loss 0.0613, Sem 0.0613 LR: 9.878e-02 Score 97.837 Data time: 0.2117, Forward time: 1.2543, Backward time: 1.2496, DDP time: 0.0043, Total iter time: 2.7797 [2021-07-07 10:24:44,387][root][INFO] - ===> Epoch[163](815/5): Loss 0.0626, Sem 0.0626 LR: 9.878e-02 Score 97.781 Data time: 0.2731, Forward time: 1.2550, Backward time: 1.2495, DDP time: 0.0043, Total iter time: 2.9001 [2021-07-07 10:24:58,037][root][INFO] - ===> Epoch[164](820/5): Loss 0.0616, Sem 0.0616 LR: 9.877e-02 Score 97.827 Data time: 0.1848, Forward time: 1.2538, Backward time: 1.2504, DDP time: 0.0043, Total iter time: 2.6845 [2021-07-07 10:25:12,728][root][INFO] - ===> Epoch[165](825/5): Loss 0.0586, Sem 0.0586 LR: 9.876e-02 Score 97.948 Data time: 0.2476, Forward time: 1.2544, Backward time: 1.2503, DDP time: 0.0043, Total iter time: 2.8807 [2021-07-07 10:25:25,867][root][INFO] - ===> Epoch[166](830/5): Loss 0.0588, Sem 0.0588 LR: 9.875e-02 Score 97.899 Data time: 0.1884, Forward time: 1.2540, Backward time: 1.2499, DDP time: 0.0043, Total iter time: 2.5848 [2021-07-07 10:25:39,206][root][INFO] - ===> Epoch[167](835/5): Loss 0.0608, Sem 0.0608 LR: 9.875e-02 Score 97.910 Data time: 0.1475, Forward time: 1.2536, Backward time: 1.2500, DDP time: 0.0043, Total iter time: 2.6346 [2021-07-07 10:25:53,737][root][INFO] - ===> Epoch[168](840/5): Loss 0.0612, Sem 0.0612 LR: 9.874e-02 Score 97.823 Data time: 0.2264, Forward time: 1.2530, Backward time: 1.2510, DDP time: 0.0043, Total iter time: 2.8582 [2021-07-07 10:26:07,140][root][INFO] - ===> Epoch[169](845/5): Loss 0.0609, Sem 0.0609 LR: 9.873e-02 Score 97.870 Data time: 0.2742, Forward time: 1.2537, Backward time: 1.2493, DDP time: 0.0043, Total iter time: 2.6301 [2021-07-07 10:26:21,620][root][INFO] - ===> Epoch[170](850/5): Loss 0.0589, Sem 0.0589 LR: 9.872e-02 Score 97.943 Data time: 0.2765, Forward time: 1.2541, Backward time: 1.2492, DDP time: 0.0043, Total iter time: 2.8562 [2021-07-07 10:26:35,897][root][INFO] - ===> Epoch[171](855/5): Loss 0.0588, Sem 0.0588 LR: 9.872e-02 Score 97.913 Data time: 0.2611, Forward time: 1.2537, Backward time: 1.2496, DDP time: 0.0043, Total iter time: 2.8065 [2021-07-07 10:26:48,835][root][INFO] - ===> Epoch[172](860/5): Loss 0.0613, Sem 0.0613 LR: 9.871e-02 Score 97.858 Data time: 0.2135, Forward time: 1.2534, Backward time: 1.2487, DDP time: 0.0043, Total iter time: 2.5491 [2021-07-07 10:27:03,229][root][INFO] - ===> Epoch[173](865/5): Loss 0.0569, Sem 0.0569 LR: 9.870e-02 Score 98.001 Data time: 0.2226, Forward time: 1.2530, Backward time: 1.2495, DDP time: 0.0043, Total iter time: 2.8309 [2021-07-07 10:27:17,279][root][INFO] - ===> Epoch[174](870/5): Loss 0.0550, Sem 0.0550 LR: 9.869e-02 Score 98.034 Data time: 0.2554, Forward time: 1.2530, Backward time: 1.2494, DDP time: 0.0043, Total iter time: 2.7678 [2021-07-07 10:27:31,291][root][INFO] - ===> Epoch[175](875/5): Loss 0.0545, Sem 0.0545 LR: 9.869e-02 Score 98.076 Data time: 0.2005, Forward time: 1.2530, Backward time: 1.2494, DDP time: 0.0043, Total iter time: 2.7589 [2021-07-07 10:27:44,860][root][INFO] - ===> Epoch[176](880/5): Loss 0.0550, Sem 0.0550 LR: 9.868e-02 Score 98.048 Data time: 0.2196, Forward time: 1.2531, Backward time: 1.2489, DDP time: 0.0043, Total iter time: 2.6711 [2021-07-07 10:27:59,451][root][INFO] - ===> Epoch[177](885/5): Loss 0.0575, Sem 0.0575 LR: 9.867e-02 Score 97.959 Data time: 0.2906, Forward time: 1.2531, Backward time: 1.2491, DDP time: 0.0043, Total iter time: 2.8697 [2021-07-07 10:28:13,614][root][INFO] - ===> Epoch[178](890/5): Loss 0.0544, Sem 0.0544 LR: 9.866e-02 Score 98.085 Data time: 0.2640, Forward time: 1.2536, Backward time: 1.2485, DDP time: 0.0043, Total iter time: 2.7719 [2021-07-07 10:28:28,077][root][INFO] - ===> Epoch[179](895/5): Loss 0.0547, Sem 0.0547 LR: 9.866e-02 Score 98.043 Data time: 0.1916, Forward time: 1.2539, Backward time: 1.2488, DDP time: 0.0043, Total iter time: 2.8477 [2021-07-07 10:28:42,264][root][INFO] - ===> Epoch[180](900/5): Loss 0.0526, Sem 0.0526 LR: 9.865e-02 Score 98.156 Data time: 0.1826, Forward time: 1.2541, Backward time: 1.2490, DDP time: 0.0043, Total iter time: 2.7882 [2021-07-07 10:28:56,511][root][INFO] - ===> Epoch[181](905/5): Loss 0.0571, Sem 0.0571 LR: 9.864e-02 Score 97.987 Data time: 0.2557, Forward time: 1.2547, Backward time: 1.2484, DDP time: 0.0043, Total iter time: 2.7992 [2021-07-07 10:29:09,442][root][INFO] - ===> Epoch[182](910/5): Loss 0.0555, Sem 0.0555 LR: 9.863e-02 Score 98.050 Data time: 0.1534, Forward time: 1.2544, Backward time: 1.2479, DDP time: 0.0043, Total iter time: 2.5541 [2021-07-07 10:29:23,872][root][INFO] - ===> Epoch[183](915/5): Loss 0.0564, Sem 0.0564 LR: 9.863e-02 Score 97.996 Data time: 0.2074, Forward time: 1.2542, Backward time: 1.2486, DDP time: 0.0043, Total iter time: 2.8382 [2021-07-07 10:29:38,074][root][INFO] - ===> Epoch[184](920/5): Loss 0.0542, Sem 0.0542 LR: 9.862e-02 Score 98.070 Data time: 0.1800, Forward time: 1.2539, Backward time: 1.2493, DDP time: 0.0043, Total iter time: 2.7902 [2021-07-07 10:29:52,125][root][INFO] - ===> Epoch[185](925/5): Loss 0.0548, Sem 0.0548 LR: 9.861e-02 Score 98.056 Data time: 0.2264, Forward time: 1.2543, Backward time: 1.2488, DDP time: 0.0043, Total iter time: 2.7581 [2021-07-07 10:30:05,575][root][INFO] - ===> Epoch[186](930/5): Loss 0.0556, Sem 0.0556 LR: 9.860e-02 Score 98.027 Data time: 0.1978, Forward time: 1.2542, Backward time: 1.2484, DDP time: 0.0043, Total iter time: 2.6412 [2021-07-07 10:30:19,533][root][INFO] - ===> Epoch[187](935/5): Loss 0.0509, Sem 0.0509 LR: 9.860e-02 Score 98.207 Data time: 0.1927, Forward time: 1.2539, Backward time: 1.2488, DDP time: 0.0043, Total iter time: 2.7509 [2021-07-07 10:30:33,273][root][INFO] - ===> Epoch[188](940/5): Loss 0.0539, Sem 0.0539 LR: 9.859e-02 Score 98.069 Data time: 0.2738, Forward time: 1.2541, Backward time: 1.2480, DDP time: 0.0043, Total iter time: 2.6988 [2021-07-07 10:30:46,783][root][INFO] - ===> Epoch[189](945/5): Loss 0.0564, Sem 0.0564 LR: 9.858e-02 Score 98.014 Data time: 0.2513, Forward time: 1.2540, Backward time: 1.2474, DDP time: 0.0043, Total iter time: 2.6498 [2021-07-07 10:31:01,493][root][INFO] - ===> Epoch[190](950/5): Loss 0.0558, Sem 0.0558 LR: 9.857e-02 Score 98.031 Data time: 0.1752, Forward time: 1.2543, Backward time: 1.2480, DDP time: 0.0043, Total iter time: 2.8894 [2021-07-07 10:31:15,323][root][INFO] - ===> Epoch[191](955/5): Loss 0.0539, Sem 0.0539 LR: 9.857e-02 Score 98.083 Data time: 0.2577, Forward time: 1.2548, Backward time: 1.2472, DDP time: 0.0043, Total iter time: 2.7092 [2021-07-07 10:31:30,055][root][INFO] - ===> Epoch[192](960/5): Loss 0.0564, Sem 0.0564 LR: 9.856e-02 Score 98.011 Data time: 0.1807, Forward time: 1.2541, Backward time: 1.2487, DDP time: 0.0043, Total iter time: 2.9021 [2021-07-07 10:31:43,662][root][INFO] - ===> Epoch[193](965/5): Loss 0.0530, Sem 0.0530 LR: 9.855e-02 Score 98.120 Data time: 0.2421, Forward time: 1.2535, Backward time: 1.2487, DDP time: 0.0043, Total iter time: 2.6744 [2021-07-07 10:31:57,677][root][INFO] - ===> Epoch[194](970/5): Loss 0.0548, Sem 0.0548 LR: 9.854e-02 Score 98.067 Data time: 0.2562, Forward time: 1.2532, Backward time: 1.2488, DDP time: 0.0043, Total iter time: 2.7515 [2021-07-07 10:32:11,574][root][INFO] - ===> Epoch[195](975/5): Loss 0.0559, Sem 0.0559 LR: 9.854e-02 Score 98.026 Data time: 0.2567, Forward time: 1.2537, Backward time: 1.2480, DDP time: 0.0043, Total iter time: 2.7205 [2021-07-07 10:32:25,607][root][INFO] - ===> Epoch[196](980/5): Loss 0.0549, Sem 0.0549 LR: 9.853e-02 Score 98.046 Data time: 0.1557, Forward time: 1.2531, Backward time: 1.2490, DDP time: 0.0043, Total iter time: 2.7662 [2021-07-07 10:32:40,023][root][INFO] - ===> Epoch[197](985/5): Loss 0.0583, Sem 0.0583 LR: 9.852e-02 Score 97.959 Data time: 0.2104, Forward time: 1.2532, Backward time: 1.2493, DDP time: 0.0043, Total iter time: 2.8407 [2021-07-07 10:32:54,476][root][INFO] - ===> Epoch[198](990/5): Loss 0.0543, Sem 0.0543 LR: 9.851e-02 Score 98.084 Data time: 0.1500, Forward time: 1.2527, Backward time: 1.2505, DDP time: 0.0043, Total iter time: 2.8424 [2021-07-07 10:33:08,617][root][INFO] - ===> Epoch[199](995/5): Loss 0.0563, Sem 0.0563 LR: 9.851e-02 Score 98.023 Data time: 0.2116, Forward time: 1.2530, Backward time: 1.2504, DDP time: 0.0043, Total iter time: 2.7814 [2021-07-07 10:33:23,200][root][INFO] - ===> Epoch[200](1000/5): Loss 0.0548, Sem 0.0548 LR: 9.850e-02 Score 98.085 Data time: 0.2173, Forward time: 1.2524, Backward time: 1.2515, DDP time: 0.0043, Total iter time: 2.8745 [2021-07-07 10:33:23,201][root][INFO] - ===> Start testing [2021-07-07 10:33:56,101][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1202 Loss 0.307 (AVG: 0.408) Score 91.027 (AVG: 90.833) mIOU 70.348 mAP 85.473 mAcc 76.924 IOU: 64.696 0.054 78.294 77.510 94.768 90.475 28.433 80.521 98.593 79.181 79.500 85.788 56.706 mAP: 85.365 11.757 92.273 91.742 99.070 98.530 62.698 97.381 99.833 90.971 94.054 95.130 92.341 mAcc: 79.245 0.547 87.048 86.578 96.210 96.128 31.288 93.372 99.323 83.416 87.466 95.372 64.018 [2021-07-07 10:33:56,101][root][INFO] - Finished test. Elapsed time: 32.8992 [2021-07-07 10:33:57,170][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2021-07-07 10:33:57,171][root][INFO] - Current best mIoU: 70.348 at iter 1 [2021-07-07 10:33:58,137][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 10:34:11,990][root][INFO] - ===> Epoch[201](1005/5): Loss 0.0545, Sem 0.0545 LR: 9.849e-02 Score 98.069 Data time: 0.2201, Forward time: 1.2521, Backward time: 1.2516, DDP time: 0.0043, Total iter time: 2.7267 [2021-07-07 10:34:25,908][root][INFO] - ===> Epoch[202](1010/5): Loss 0.0534, Sem 0.0534 LR: 9.848e-02 Score 98.128 Data time: 0.2293, Forward time: 1.2518, Backward time: 1.2517, DDP time: 0.0043, Total iter time: 2.7413 [2021-07-07 10:34:40,217][root][INFO] - ===> Epoch[203](1015/5): Loss 0.0533, Sem 0.0533 LR: 9.848e-02 Score 98.115 Data time: 0.1947, Forward time: 1.2514, Backward time: 1.2525, DDP time: 0.0043, Total iter time: 2.8220 [2021-07-07 10:34:53,440][root][INFO] - ===> Epoch[204](1020/5): Loss 0.0543, Sem 0.0543 LR: 9.847e-02 Score 98.098 Data time: 0.1996, Forward time: 1.2508, Backward time: 1.2524, DDP time: 0.0043, Total iter time: 2.5980 [2021-07-07 10:35:07,341][root][INFO] - ===> Epoch[205](1025/5): Loss 0.0520, Sem 0.0520 LR: 9.846e-02 Score 98.175 Data time: 0.2061, Forward time: 1.2510, Backward time: 1.2521, DDP time: 0.0043, Total iter time: 2.7232 [2021-07-07 10:35:21,154][root][INFO] - ===> Epoch[206](1030/5): Loss 0.0503, Sem 0.0503 LR: 9.845e-02 Score 98.195 Data time: 0.2739, Forward time: 1.2513, Backward time: 1.2513, DDP time: 0.0043, Total iter time: 2.7079 [2021-07-07 10:35:35,323][root][INFO] - ===> Epoch[207](1035/5): Loss 0.0483, Sem 0.0483 LR: 9.845e-02 Score 98.293 Data time: 0.2686, Forward time: 1.2519, Backward time: 1.2507, DDP time: 0.0043, Total iter time: 2.7865 [2021-07-07 10:35:49,148][root][INFO] - ===> Epoch[208](1040/5): Loss 0.0524, Sem 0.0524 LR: 9.844e-02 Score 98.123 Data time: 0.2952, Forward time: 1.2523, Backward time: 1.2498, DDP time: 0.0043, Total iter time: 2.7132 [2021-07-07 10:36:02,456][root][INFO] - ===> Epoch[209](1045/5): Loss 0.0524, Sem 0.0524 LR: 9.843e-02 Score 98.158 Data time: 0.2144, Forward time: 1.2522, Backward time: 1.2493, DDP time: 0.0043, Total iter time: 2.6203 [2021-07-07 10:36:16,654][root][INFO] - ===> Epoch[210](1050/5): Loss 0.0518, Sem 0.0518 LR: 9.842e-02 Score 98.161 Data time: 0.1522, Forward time: 1.2521, Backward time: 1.2498, DDP time: 0.0043, Total iter time: 2.8036 [2021-07-07 10:36:30,939][root][INFO] - ===> Epoch[211](1055/5): Loss 0.0499, Sem 0.0499 LR: 9.842e-02 Score 98.239 Data time: 0.2083, Forward time: 1.2522, Backward time: 1.2501, DDP time: 0.0043, Total iter time: 2.8156 [2021-07-07 10:36:44,715][root][INFO] - ===> Epoch[212](1060/5): Loss 0.0557, Sem 0.0557 LR: 9.841e-02 Score 98.035 Data time: 0.2041, Forward time: 1.2516, Backward time: 1.2505, DDP time: 0.0043, Total iter time: 2.7138 [2021-07-07 10:36:57,688][root][INFO] - ===> Epoch[213](1065/5): Loss 0.0539, Sem 0.0539 LR: 9.840e-02 Score 98.067 Data time: 0.1968, Forward time: 1.2512, Backward time: 1.2501, DDP time: 0.0043, Total iter time: 2.5513 [2021-07-07 10:37:12,589][root][INFO] - ===> Epoch[214](1070/5): Loss 0.0551, Sem 0.0551 LR: 9.839e-02 Score 98.078 Data time: 0.2883, Forward time: 1.2508, Backward time: 1.2510, DDP time: 0.0043, Total iter time: 2.9292 [2021-07-07 10:37:26,386][root][INFO] - ===> Epoch[215](1075/5): Loss 0.0541, Sem 0.0541 LR: 9.839e-02 Score 98.086 Data time: 0.2312, Forward time: 1.2508, Backward time: 1.2507, DDP time: 0.0043, Total iter time: 2.7105 [2021-07-07 10:37:40,193][root][INFO] - ===> Epoch[216](1080/5): Loss 0.0543, Sem 0.0543 LR: 9.838e-02 Score 98.066 Data time: 0.2168, Forward time: 1.2511, Backward time: 1.2502, DDP time: 0.0043, Total iter time: 2.7180 [2021-07-07 10:37:53,764][root][INFO] - ===> Epoch[217](1085/5): Loss 0.0524, Sem 0.0524 LR: 9.837e-02 Score 98.165 Data time: 0.2284, Forward time: 1.2513, Backward time: 1.2496, DDP time: 0.0043, Total iter time: 2.6680 [2021-07-07 10:38:08,568][root][INFO] - ===> Epoch[218](1090/5): Loss 0.0540, Sem 0.0540 LR: 9.836e-02 Score 98.172 Data time: 0.2041, Forward time: 1.2516, Backward time: 1.2500, DDP time: 0.0043, Total iter time: 2.9103 [2021-07-07 10:38:23,166][root][INFO] - ===> Epoch[219](1095/5): Loss 0.0514, Sem 0.0514 LR: 9.836e-02 Score 98.198 Data time: 0.2777, Forward time: 1.2516, Backward time: 1.2503, DDP time: 0.0043, Total iter time: 2.8672 [2021-07-07 10:38:36,990][root][INFO] - ===> Epoch[220](1100/5): Loss 0.0514, Sem 0.0514 LR: 9.835e-02 Score 98.161 Data time: 0.2472, Forward time: 1.2516, Backward time: 1.2500, DDP time: 0.0043, Total iter time: 2.7184 [2021-07-07 10:38:51,234][root][INFO] - ===> Epoch[221](1105/5): Loss 0.0502, Sem 0.0502 LR: 9.834e-02 Score 98.222 Data time: 0.2501, Forward time: 1.2517, Backward time: 1.2500, DDP time: 0.0043, Total iter time: 2.8067 [2021-07-07 10:39:04,911][root][INFO] - ===> Epoch[222](1110/5): Loss 0.0482, Sem 0.0482 LR: 9.833e-02 Score 98.282 Data time: 0.2510, Forward time: 1.2515, Backward time: 1.2498, DDP time: 0.0043, Total iter time: 2.6895 [2021-07-07 10:39:18,092][root][INFO] - ===> Epoch[223](1115/5): Loss 0.0537, Sem 0.0537 LR: 9.833e-02 Score 98.093 Data time: 0.2088, Forward time: 1.2515, Backward time: 1.2491, DDP time: 0.0043, Total iter time: 2.5884 [2021-07-07 10:39:32,265][root][INFO] - ===> Epoch[224](1120/5): Loss 0.0522, Sem 0.0522 LR: 9.832e-02 Score 98.166 Data time: 0.2019, Forward time: 1.2519, Backward time: 1.2489, DDP time: 0.0043, Total iter time: 2.7894 [2021-07-07 10:39:47,096][root][INFO] - ===> Epoch[225](1125/5): Loss 0.0505, Sem 0.0505 LR: 9.831e-02 Score 98.204 Data time: 0.2066, Forward time: 1.2515, Backward time: 1.2501, DDP time: 0.0043, Total iter time: 2.9259 [2021-07-07 10:40:00,690][root][INFO] - ===> Epoch[226](1130/5): Loss 0.0510, Sem 0.0510 LR: 9.830e-02 Score 98.179 Data time: 0.2448, Forward time: 1.2514, Backward time: 1.2497, DDP time: 0.0043, Total iter time: 2.6691 [2021-07-07 10:40:14,316][root][INFO] - ===> Epoch[227](1135/5): Loss 0.0526, Sem 0.0526 LR: 9.830e-02 Score 98.140 Data time: 0.2691, Forward time: 1.2515, Backward time: 1.2490, DDP time: 0.0043, Total iter time: 2.6714 [2021-07-07 10:40:28,153][root][INFO] - ===> Epoch[228](1140/5): Loss 0.0508, Sem 0.0508 LR: 9.829e-02 Score 98.217 Data time: 0.2285, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0043, Total iter time: 2.7273 [2021-07-07 10:40:42,148][root][INFO] - ===> Epoch[229](1145/5): Loss 0.0503, Sem 0.0503 LR: 9.828e-02 Score 98.225 Data time: 0.1949, Forward time: 1.2521, Backward time: 1.2483, DDP time: 0.0043, Total iter time: 2.7561 [2021-07-07 10:40:55,693][root][INFO] - ===> Epoch[230](1150/5): Loss 0.0488, Sem 0.0488 LR: 9.827e-02 Score 98.268 Data time: 0.2257, Forward time: 1.2524, Backward time: 1.2476, DDP time: 0.0043, Total iter time: 2.6561 [2021-07-07 10:41:08,794][root][INFO] - ===> Epoch[231](1155/5): Loss 0.0482, Sem 0.0482 LR: 9.827e-02 Score 98.292 Data time: 0.1628, Forward time: 1.2522, Backward time: 1.2474, DDP time: 0.0043, Total iter time: 2.5811 [2021-07-07 10:41:24,252][root][INFO] - ===> Epoch[232](1160/5): Loss 0.0505, Sem 0.0505 LR: 9.826e-02 Score 98.209 Data time: 0.2531, Forward time: 1.2526, Backward time: 1.2480, DDP time: 0.0043, Total iter time: 3.0457 [2021-07-07 10:41:38,866][root][INFO] - ===> Epoch[233](1165/5): Loss 0.0487, Sem 0.0487 LR: 9.825e-02 Score 98.287 Data time: 0.2533, Forward time: 1.2528, Backward time: 1.2482, DDP time: 0.0043, Total iter time: 2.8806 [2021-07-07 10:41:52,688][root][INFO] - ===> Epoch[234](1170/5): Loss 0.0491, Sem 0.0491 LR: 9.824e-02 Score 98.276 Data time: 0.2106, Forward time: 1.2526, Backward time: 1.2482, DDP time: 0.0043, Total iter time: 2.7198 [2021-07-07 10:42:06,350][root][INFO] - ===> Epoch[235](1175/5): Loss 0.0502, Sem 0.0502 LR: 9.824e-02 Score 98.221 Data time: 0.2446, Forward time: 1.2522, Backward time: 1.2482, DDP time: 0.0043, Total iter time: 2.6891 [2021-07-07 10:42:20,093][root][INFO] - ===> Epoch[236](1180/5): Loss 0.0451, Sem 0.0451 LR: 9.823e-02 Score 98.395 Data time: 0.2126, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0043, Total iter time: 2.6960 [2021-07-07 10:42:34,850][root][INFO] - ===> Epoch[237](1185/5): Loss 0.0488, Sem 0.0488 LR: 9.822e-02 Score 98.267 Data time: 0.2998, Forward time: 1.2517, Backward time: 1.2487, DDP time: 0.0043, Total iter time: 2.8904 [2021-07-07 10:42:48,362][root][INFO] - ===> Epoch[238](1190/5): Loss 0.0476, Sem 0.0476 LR: 9.821e-02 Score 98.351 Data time: 0.1913, Forward time: 1.2510, Backward time: 1.2491, DDP time: 0.0043, Total iter time: 2.6637 [2021-07-07 10:43:03,369][root][INFO] - ===> Epoch[239](1195/5): Loss 0.0480, Sem 0.0480 LR: 9.821e-02 Score 98.321 Data time: 0.2006, Forward time: 1.2507, Backward time: 1.2502, DDP time: 0.0043, Total iter time: 2.9467 [2021-07-07 10:43:17,029][root][INFO] - ===> Epoch[240](1200/5): Loss 0.0478, Sem 0.0478 LR: 9.820e-02 Score 98.334 Data time: 0.1642, Forward time: 1.2512, Backward time: 1.2497, DDP time: 0.0043, Total iter time: 2.6873 [2021-07-07 10:43:17,030][root][INFO] - ===> Start testing [2021-07-07 10:43:53,432][root][INFO] - 68/68: Data time: 0.0384, Iter time: 0.1303 Loss 0.291 (AVG: 0.433) Score 91.438 (AVG: 90.878) mIOU 69.778 mAP 85.738 mAcc 76.853 IOU: 64.499 0.054 66.680 77.830 95.236 90.789 33.202 80.931 98.700 77.398 79.317 85.606 56.878 mAP: 83.362 14.034 95.224 90.598 99.059 98.745 63.046 96.631 99.844 92.013 94.349 94.645 93.045 mAcc: 78.082 0.547 93.432 86.016 96.620 96.055 35.670 87.908 99.649 79.459 86.864 96.099 62.689 [2021-07-07 10:43:53,432][root][INFO] - Finished test. Elapsed time: 36.4015 [2021-07-07 10:43:54,532][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 10:44:07,625][root][INFO] - ===> Epoch[241](1205/5): Loss 0.0500, Sem 0.0500 LR: 9.819e-02 Score 98.254 Data time: 0.2177, Forward time: 1.2509, Backward time: 1.2492, DDP time: 0.0043, Total iter time: 2.5812 [2021-07-07 10:44:21,606][root][INFO] - ===> Epoch[242](1210/5): Loss 0.0486, Sem 0.0486 LR: 9.818e-02 Score 98.278 Data time: 0.2143, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0043, Total iter time: 2.7573 [2021-07-07 10:44:35,351][root][INFO] - ===> Epoch[243](1215/5): Loss 0.0496, Sem 0.0496 LR: 9.818e-02 Score 98.254 Data time: 0.1614, Forward time: 1.2519, Backward time: 1.2484, DDP time: 0.0043, Total iter time: 2.7176 [2021-07-07 10:44:49,988][root][INFO] - ===> Epoch[244](1220/5): Loss 0.0488, Sem 0.0488 LR: 9.817e-02 Score 98.257 Data time: 0.2915, Forward time: 1.2522, Backward time: 1.2484, DDP time: 0.0043, Total iter time: 2.8751 [2021-07-07 10:45:03,984][root][INFO] - ===> Epoch[245](1225/5): Loss 0.0474, Sem 0.0474 LR: 9.816e-02 Score 98.321 Data time: 0.1948, Forward time: 1.2518, Backward time: 1.2489, DDP time: 0.0043, Total iter time: 2.7562 [2021-07-07 10:45:17,522][root][INFO] - ===> Epoch[246](1230/5): Loss 0.0482, Sem 0.0482 LR: 9.815e-02 Score 98.290 Data time: 0.2901, Forward time: 1.2518, Backward time: 1.2481, DDP time: 0.0043, Total iter time: 2.6548 [2021-07-07 10:45:31,028][root][INFO] - ===> Epoch[247](1235/5): Loss 0.0477, Sem 0.0477 LR: 9.815e-02 Score 98.301 Data time: 0.2734, Forward time: 1.2522, Backward time: 1.2472, DDP time: 0.0043, Total iter time: 2.6528 [2021-07-07 10:45:44,515][root][INFO] - ===> Epoch[248](1240/5): Loss 0.0473, Sem 0.0473 LR: 9.814e-02 Score 98.332 Data time: 0.1877, Forward time: 1.2523, Backward time: 1.2468, DDP time: 0.0043, Total iter time: 2.6581 [2021-07-07 10:45:59,081][root][INFO] - ===> Epoch[249](1245/5): Loss 0.0481, Sem 0.0481 LR: 9.813e-02 Score 98.277 Data time: 0.2170, Forward time: 1.2526, Backward time: 1.2470, DDP time: 0.0043, Total iter time: 2.8650 [2021-07-07 10:46:12,912][root][INFO] - ===> Epoch[250](1250/5): Loss 0.0477, Sem 0.0477 LR: 9.812e-02 Score 98.329 Data time: 0.1818, Forward time: 1.2523, Backward time: 1.2472, DDP time: 0.0043, Total iter time: 2.7238 [2021-07-07 10:46:26,509][root][INFO] - ===> Epoch[251](1255/5): Loss 0.0462, Sem 0.0462 LR: 9.812e-02 Score 98.357 Data time: 0.2148, Forward time: 1.2522, Backward time: 1.2471, DDP time: 0.0043, Total iter time: 2.6749 [2021-07-07 10:46:40,191][root][INFO] - ===> Epoch[252](1260/5): Loss 0.0480, Sem 0.0480 LR: 9.811e-02 Score 98.316 Data time: 0.2007, Forward time: 1.2521, Backward time: 1.2469, DDP time: 0.0043, Total iter time: 2.6861 [2021-07-07 10:46:54,153][root][INFO] - ===> Epoch[253](1265/5): Loss 0.0458, Sem 0.0458 LR: 9.810e-02 Score 98.373 Data time: 0.2477, Forward time: 1.2521, Backward time: 1.2468, DDP time: 0.0043, Total iter time: 2.7520 [2021-07-07 10:47:08,110][root][INFO] - ===> Epoch[254](1270/5): Loss 0.0464, Sem 0.0464 LR: 9.809e-02 Score 98.361 Data time: 0.2262, Forward time: 1.2528, Backward time: 1.2461, DDP time: 0.0043, Total iter time: 2.7412 [2021-07-07 10:47:22,262][root][INFO] - ===> Epoch[255](1275/5): Loss 0.0459, Sem 0.0459 LR: 9.809e-02 Score 98.388 Data time: 0.2126, Forward time: 1.2528, Backward time: 1.2463, DDP time: 0.0043, Total iter time: 2.7920 [2021-07-07 10:47:36,569][root][INFO] - ===> Epoch[256](1280/5): Loss 0.0456, Sem 0.0456 LR: 9.808e-02 Score 98.376 Data time: 0.1767, Forward time: 1.2520, Backward time: 1.2474, DDP time: 0.0043, Total iter time: 2.8139 [2021-07-07 10:47:50,812][root][INFO] - ===> Epoch[257](1285/5): Loss 0.0490, Sem 0.0490 LR: 9.807e-02 Score 98.326 Data time: 0.2849, Forward time: 1.2525, Backward time: 1.2468, DDP time: 0.0043, Total iter time: 2.8005 [2021-07-07 10:48:05,938][root][INFO] - ===> Epoch[258](1290/5): Loss 0.0494, Sem 0.0494 LR: 9.806e-02 Score 98.239 Data time: 0.2883, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0043, Total iter time: 2.9755 [2021-07-07 10:48:19,570][root][INFO] - ===> Epoch[259](1295/5): Loss 0.0476, Sem 0.0476 LR: 9.806e-02 Score 98.313 Data time: 0.2884, Forward time: 1.2529, Backward time: 1.2465, DDP time: 0.0043, Total iter time: 2.6870 [2021-07-07 10:48:33,130][root][INFO] - ===> Epoch[260](1300/5): Loss 0.0458, Sem 0.0458 LR: 9.805e-02 Score 98.379 Data time: 0.2532, Forward time: 1.2532, Backward time: 1.2458, DDP time: 0.0043, Total iter time: 2.6555 [2021-07-07 10:48:47,678][root][INFO] - ===> Epoch[261](1305/5): Loss 0.0462, Sem 0.0462 LR: 9.804e-02 Score 98.346 Data time: 0.2402, Forward time: 1.2534, Backward time: 1.2459, DDP time: 0.0043, Total iter time: 2.8543 [2021-07-07 10:49:02,021][root][INFO] - ===> Epoch[262](1310/5): Loss 0.0434, Sem 0.0434 LR: 9.803e-02 Score 98.451 Data time: 0.3105, Forward time: 1.2533, Backward time: 1.2459, DDP time: 0.0043, Total iter time: 2.8281 [2021-07-07 10:49:16,127][root][INFO] - ===> Epoch[263](1315/5): Loss 0.0471, Sem 0.0471 LR: 9.803e-02 Score 98.316 Data time: 0.2272, Forward time: 1.2534, Backward time: 1.2458, DDP time: 0.0043, Total iter time: 2.7721 [2021-07-07 10:49:29,772][root][INFO] - ===> Epoch[264](1320/5): Loss 0.0453, Sem 0.0453 LR: 9.802e-02 Score 98.388 Data time: 0.1646, Forward time: 1.2533, Backward time: 1.2459, DDP time: 0.0043, Total iter time: 2.6876 [2021-07-07 10:49:43,932][root][INFO] - ===> Epoch[265](1325/5): Loss 0.0448, Sem 0.0448 LR: 9.801e-02 Score 98.406 Data time: 0.1576, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0043, Total iter time: 2.7782 [2021-07-07 10:49:58,063][root][INFO] - ===> Epoch[266](1330/5): Loss 0.0453, Sem 0.0453 LR: 9.800e-02 Score 98.389 Data time: 0.2996, Forward time: 1.2535, Backward time: 1.2458, DDP time: 0.0043, Total iter time: 2.7771 [2021-07-07 10:50:12,226][root][INFO] - ===> Epoch[267](1335/5): Loss 0.0470, Sem 0.0470 LR: 9.800e-02 Score 98.349 Data time: 0.2504, Forward time: 1.2537, Backward time: 1.2456, DDP time: 0.0043, Total iter time: 2.7889 [2021-07-07 10:50:26,588][root][INFO] - ===> Epoch[268](1340/5): Loss 0.0450, Sem 0.0450 LR: 9.799e-02 Score 98.396 Data time: 0.1991, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0043, Total iter time: 2.8208 [2021-07-07 10:50:40,964][root][INFO] - ===> Epoch[269](1345/5): Loss 0.0460, Sem 0.0460 LR: 9.798e-02 Score 98.363 Data time: 0.1823, Forward time: 1.2535, Backward time: 1.2465, DDP time: 0.0043, Total iter time: 2.8334 [2021-07-07 10:50:54,282][root][INFO] - ===> Epoch[270](1350/5): Loss 0.0459, Sem 0.0459 LR: 9.797e-02 Score 98.364 Data time: 0.2026, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0043, Total iter time: 2.6165 [2021-07-07 10:51:08,318][root][INFO] - ===> Epoch[271](1355/5): Loss 0.0434, Sem 0.0434 LR: 9.797e-02 Score 98.451 Data time: 0.2405, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0043, Total iter time: 2.7566 [2021-07-07 10:51:22,627][root][INFO] - ===> Epoch[272](1360/5): Loss 0.0445, Sem 0.0445 LR: 9.796e-02 Score 98.412 Data time: 0.2550, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0043, Total iter time: 2.8176 [2021-07-07 10:51:37,337][root][INFO] - ===> Epoch[273](1365/5): Loss 0.0433, Sem 0.0433 LR: 9.795e-02 Score 98.466 Data time: 0.2457, Forward time: 1.2532, Backward time: 1.2467, DDP time: 0.0043, Total iter time: 2.8973 [2021-07-07 10:51:51,822][root][INFO] - ===> Epoch[274](1370/5): Loss 0.0419, Sem 0.0419 LR: 9.794e-02 Score 98.509 Data time: 0.2770, Forward time: 1.2531, Backward time: 1.2469, DDP time: 0.0043, Total iter time: 2.8407 [2021-07-07 10:52:06,179][root][INFO] - ===> Epoch[275](1375/5): Loss 0.0442, Sem 0.0442 LR: 9.794e-02 Score 98.418 Data time: 0.1862, Forward time: 1.2530, Backward time: 1.2473, DDP time: 0.0043, Total iter time: 2.8194 [2021-07-07 10:52:19,800][root][INFO] - ===> Epoch[276](1380/5): Loss 0.0439, Sem 0.0439 LR: 9.793e-02 Score 98.440 Data time: 0.2411, Forward time: 1.2530, Backward time: 1.2470, DDP time: 0.0043, Total iter time: 2.6773 [2021-07-07 10:52:33,797][root][INFO] - ===> Epoch[277](1385/5): Loss 0.0438, Sem 0.0438 LR: 9.792e-02 Score 98.445 Data time: 0.2511, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0043, Total iter time: 2.7475 [2021-07-07 10:52:48,017][root][INFO] - ===> Epoch[278](1390/5): Loss 0.0433, Sem 0.0433 LR: 9.791e-02 Score 98.447 Data time: 0.1925, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0043, Total iter time: 2.7995 [2021-07-07 10:53:02,189][root][INFO] - ===> Epoch[279](1395/5): Loss 0.0445, Sem 0.0445 LR: 9.791e-02 Score 98.428 Data time: 0.2458, Forward time: 1.2527, Backward time: 1.2474, DDP time: 0.0043, Total iter time: 2.7825 [2021-07-07 10:53:15,842][root][INFO] - ===> Epoch[280](1400/5): Loss 0.0413, Sem 0.0413 LR: 9.790e-02 Score 98.519 Data time: 0.2548, Forward time: 1.2522, Backward time: 1.2475, DDP time: 0.0043, Total iter time: 2.6917 [2021-07-07 10:53:15,844][root][INFO] - ===> Start testing [2021-07-07 10:53:52,030][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1190 Loss 0.281 (AVG: 0.436) Score 91.599 (AVG: 90.931) mIOU 70.482 mAP 85.041 mAcc 76.365 IOU: 64.172 0.000 79.553 76.317 94.902 91.530 32.257 81.115 98.677 74.537 80.487 86.026 56.699 mAP: 85.147 6.431 91.125 90.967 99.084 98.625 62.603 96.727 99.842 92.995 94.523 95.156 92.304 mAcc: 79.942 0.000 86.427 83.302 96.525 96.120 34.249 91.944 99.642 76.588 89.731 96.465 61.815 [2021-07-07 10:53:52,030][root][INFO] - Finished test. Elapsed time: 36.1861 [2021-07-07 10:53:53,158][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2021-07-07 10:53:53,159][root][INFO] - Current best mIoU: 70.482 at iter 1 [2021-07-07 10:53:54,159][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 10:54:06,209][root][INFO] - ===> Epoch[281](1405/5): Loss 0.0425, Sem 0.0425 LR: 9.789e-02 Score 98.481 Data time: 0.1925, Forward time: 1.2513, Backward time: 1.2471, DDP time: 0.0043, Total iter time: 2.3741 [2021-07-07 10:54:19,965][root][INFO] - ===> Epoch[282](1410/5): Loss 0.0413, Sem 0.0413 LR: 9.788e-02 Score 98.536 Data time: 0.2488, Forward time: 1.2518, Backward time: 1.2464, DDP time: 0.0043, Total iter time: 2.6993 [2021-07-07 10:54:34,176][root][INFO] - ===> Epoch[283](1415/5): Loss 0.0454, Sem 0.0454 LR: 9.788e-02 Score 98.392 Data time: 0.3520, Forward time: 1.2521, Backward time: 1.2458, DDP time: 0.0043, Total iter time: 2.7924 [2021-07-07 10:54:47,800][root][INFO] - ===> Epoch[284](1420/5): Loss 0.0455, Sem 0.0455 LR: 9.787e-02 Score 98.447 Data time: 0.1825, Forward time: 1.2517, Backward time: 1.2460, DDP time: 0.0043, Total iter time: 2.6733 [2021-07-07 10:55:01,663][root][INFO] - ===> Epoch[285](1425/5): Loss 0.0450, Sem 0.0450 LR: 9.786e-02 Score 98.415 Data time: 0.2316, Forward time: 1.2519, Backward time: 1.2457, DDP time: 0.0043, Total iter time: 2.7241 [2021-07-07 10:55:15,477][root][INFO] - ===> Epoch[286](1430/5): Loss 0.0438, Sem 0.0438 LR: 9.785e-02 Score 98.420 Data time: 0.1941, Forward time: 1.2518, Backward time: 1.2457, DDP time: 0.0043, Total iter time: 2.7186 [2021-07-07 10:55:28,519][root][INFO] - ===> Epoch[287](1435/5): Loss 0.0446, Sem 0.0446 LR: 9.784e-02 Score 98.403 Data time: 0.1631, Forward time: 1.2516, Backward time: 1.2455, DDP time: 0.0043, Total iter time: 2.5625 [2021-07-07 10:55:43,009][root][INFO] - ===> Epoch[288](1440/5): Loss 0.0447, Sem 0.0447 LR: 9.784e-02 Score 98.427 Data time: 0.2364, Forward time: 1.2514, Backward time: 1.2460, DDP time: 0.0043, Total iter time: 2.8519 [2021-07-07 10:55:56,703][root][INFO] - ===> Epoch[289](1445/5): Loss 0.0429, Sem 0.0429 LR: 9.783e-02 Score 98.470 Data time: 0.2439, Forward time: 1.2515, Backward time: 1.2456, DDP time: 0.0043, Total iter time: 2.6896 [2021-07-07 10:56:10,731][root][INFO] - ===> Epoch[290](1450/5): Loss 0.0439, Sem 0.0439 LR: 9.782e-02 Score 98.453 Data time: 0.2309, Forward time: 1.2517, Backward time: 1.2454, DDP time: 0.0043, Total iter time: 2.7601 [2021-07-07 10:56:25,340][root][INFO] - ===> Epoch[291](1455/5): Loss 0.0440, Sem 0.0440 LR: 9.781e-02 Score 98.431 Data time: 0.2408, Forward time: 1.2517, Backward time: 1.2457, DDP time: 0.0043, Total iter time: 2.8662 [2021-07-07 10:56:38,968][root][INFO] - ===> Epoch[292](1460/5): Loss 0.0424, Sem 0.0424 LR: 9.781e-02 Score 98.515 Data time: 0.2093, Forward time: 1.2517, Backward time: 1.2454, DDP time: 0.0043, Total iter time: 2.6771 [2021-07-07 10:56:53,398][root][INFO] - ===> Epoch[293](1465/5): Loss 0.0434, Sem 0.0434 LR: 9.780e-02 Score 98.457 Data time: 0.1974, Forward time: 1.2512, Backward time: 1.2462, DDP time: 0.0043, Total iter time: 2.8347 [2021-07-07 10:57:07,942][root][INFO] - ===> Epoch[294](1470/5): Loss 0.0408, Sem 0.0408 LR: 9.779e-02 Score 98.531 Data time: 0.2182, Forward time: 1.2519, Backward time: 1.2459, DDP time: 0.0043, Total iter time: 2.8422 [2021-07-07 10:57:21,271][root][INFO] - ===> Epoch[295](1475/5): Loss 0.0458, Sem 0.0458 LR: 9.778e-02 Score 98.378 Data time: 0.2500, Forward time: 1.2518, Backward time: 1.2454, DDP time: 0.0043, Total iter time: 2.6199 [2021-07-07 10:57:36,143][root][INFO] - ===> Epoch[296](1480/5): Loss 0.0444, Sem 0.0444 LR: 9.778e-02 Score 98.437 Data time: 0.2770, Forward time: 1.2522, Backward time: 1.2454, DDP time: 0.0043, Total iter time: 2.9240 [2021-07-07 10:57:49,465][root][INFO] - ===> Epoch[297](1485/5): Loss 0.0437, Sem 0.0437 LR: 9.777e-02 Score 98.438 Data time: 0.2529, Forward time: 1.2520, Backward time: 1.2450, DDP time: 0.0043, Total iter time: 2.6228 [2021-07-07 10:58:03,994][root][INFO] - ===> Epoch[298](1490/5): Loss 0.0434, Sem 0.0434 LR: 9.776e-02 Score 98.456 Data time: 0.2665, Forward time: 1.2521, Backward time: 1.2452, DDP time: 0.0043, Total iter time: 2.8543 [2021-07-07 10:58:18,058][root][INFO] - ===> Epoch[299](1495/5): Loss 0.0399, Sem 0.0399 LR: 9.775e-02 Score 98.566 Data time: 0.2604, Forward time: 1.2524, Backward time: 1.2448, DDP time: 0.0043, Total iter time: 2.7711 [2021-07-07 10:58:31,807][root][INFO] - ===> Epoch[300](1500/5): Loss 0.0400, Sem 0.0400 LR: 9.775e-02 Score 98.581 Data time: 0.2924, Forward time: 1.2523, Backward time: 1.2444, DDP time: 0.0043, Total iter time: 2.6876 [2021-07-07 10:58:44,971][root][INFO] - ===> Epoch[301](1505/5): Loss 0.0412, Sem 0.0412 LR: 9.774e-02 Score 98.522 Data time: 0.2121, Forward time: 1.2524, Backward time: 1.2438, DDP time: 0.0043, Total iter time: 2.5938 [2021-07-07 10:58:59,033][root][INFO] - ===> Epoch[302](1510/5): Loss 0.0398, Sem 0.0398 LR: 9.773e-02 Score 98.577 Data time: 0.1876, Forward time: 1.2522, Backward time: 1.2442, DDP time: 0.0043, Total iter time: 2.7621 [2021-07-07 10:59:12,016][root][INFO] - ===> Epoch[303](1515/5): Loss 0.0415, Sem 0.0415 LR: 9.772e-02 Score 98.516 Data time: 0.2207, Forward time: 1.2523, Backward time: 1.2435, DDP time: 0.0043, Total iter time: 2.5527 [2021-07-07 10:59:26,684][root][INFO] - ===> Epoch[304](1520/5): Loss 0.0425, Sem 0.0425 LR: 9.772e-02 Score 98.479 Data time: 0.2597, Forward time: 1.2526, Backward time: 1.2435, DDP time: 0.0043, Total iter time: 2.8803 [2021-07-07 10:59:40,891][root][INFO] - ===> Epoch[305](1525/5): Loss 0.0432, Sem 0.0432 LR: 9.771e-02 Score 98.447 Data time: 0.2005, Forward time: 1.2529, Backward time: 1.2434, DDP time: 0.0043, Total iter time: 2.8015 [2021-07-07 10:59:54,826][root][INFO] - ===> Epoch[306](1530/5): Loss 0.0423, Sem 0.0423 LR: 9.770e-02 Score 98.507 Data time: 0.2000, Forward time: 1.2526, Backward time: 1.2437, DDP time: 0.0043, Total iter time: 2.7355 [2021-07-07 11:00:09,414][root][INFO] - ===> Epoch[307](1535/5): Loss 0.0409, Sem 0.0409 LR: 9.769e-02 Score 98.558 Data time: 0.1911, Forward time: 1.2525, Backward time: 1.2443, DDP time: 0.0043, Total iter time: 2.8731 [2021-07-07 11:00:23,306][root][INFO] - ===> Epoch[308](1540/5): Loss 0.0395, Sem 0.0395 LR: 9.769e-02 Score 98.592 Data time: 0.2172, Forward time: 1.2524, Backward time: 1.2443, DDP time: 0.0043, Total iter time: 2.7323 [2021-07-07 11:00:37,081][root][INFO] - ===> Epoch[309](1545/5): Loss 0.0396, Sem 0.0396 LR: 9.768e-02 Score 98.585 Data time: 0.2909, Forward time: 1.2526, Backward time: 1.2438, DDP time: 0.0043, Total iter time: 2.6939 [2021-07-07 11:00:51,953][root][INFO] - ===> Epoch[310](1550/5): Loss 0.0418, Sem 0.0418 LR: 9.767e-02 Score 98.502 Data time: 0.1964, Forward time: 1.2525, Backward time: 1.2444, DDP time: 0.0043, Total iter time: 2.9198 [2021-07-07 11:01:05,719][root][INFO] - ===> Epoch[311](1555/5): Loss 0.0394, Sem 0.0394 LR: 9.766e-02 Score 98.602 Data time: 0.2056, Forward time: 1.2522, Backward time: 1.2446, DDP time: 0.0043, Total iter time: 2.7083 [2021-07-07 11:01:20,712][root][INFO] - ===> Epoch[312](1560/5): Loss 0.0403, Sem 0.0403 LR: 9.766e-02 Score 98.581 Data time: 0.2848, Forward time: 1.2527, Backward time: 1.2446, DDP time: 0.0043, Total iter time: 2.9409 [2021-07-07 11:01:34,525][root][INFO] - ===> Epoch[313](1565/5): Loss 0.0393, Sem 0.0393 LR: 9.765e-02 Score 98.582 Data time: 0.2664, Forward time: 1.2528, Backward time: 1.2442, DDP time: 0.0043, Total iter time: 2.7218 [2021-07-07 11:01:48,098][root][INFO] - ===> Epoch[314](1570/5): Loss 0.0415, Sem 0.0415 LR: 9.764e-02 Score 98.512 Data time: 0.1914, Forward time: 1.2528, Backward time: 1.2441, DDP time: 0.0043, Total iter time: 2.6779 [2021-07-07 11:02:02,103][root][INFO] - ===> Epoch[315](1575/5): Loss 0.0396, Sem 0.0396 LR: 9.763e-02 Score 98.579 Data time: 0.2635, Forward time: 1.2534, Backward time: 1.2434, DDP time: 0.0043, Total iter time: 2.7513 [2021-07-07 11:02:16,436][root][INFO] - ===> Epoch[316](1580/5): Loss 0.0389, Sem 0.0389 LR: 9.763e-02 Score 98.607 Data time: 0.2042, Forward time: 1.2535, Backward time: 1.2436, DDP time: 0.0043, Total iter time: 2.8279 [2021-07-07 11:02:31,043][root][INFO] - ===> Epoch[317](1585/5): Loss 0.0398, Sem 0.0398 LR: 9.762e-02 Score 98.580 Data time: 0.2279, Forward time: 1.2536, Backward time: 1.2438, DDP time: 0.0043, Total iter time: 2.8836 [2021-07-07 11:02:45,160][root][INFO] - ===> Epoch[318](1590/5): Loss 0.0387, Sem 0.0387 LR: 9.761e-02 Score 98.610 Data time: 0.2186, Forward time: 1.2535, Backward time: 1.2440, DDP time: 0.0043, Total iter time: 2.7811 [2021-07-07 11:02:59,534][root][INFO] - ===> Epoch[319](1595/5): Loss 0.0401, Sem 0.0401 LR: 9.760e-02 Score 98.572 Data time: 0.2785, Forward time: 1.2539, Backward time: 1.2437, DDP time: 0.0043, Total iter time: 2.8235 [2021-07-07 11:03:14,463][root][INFO] - ===> Epoch[320](1600/5): Loss 0.0396, Sem 0.0396 LR: 9.760e-02 Score 98.580 Data time: 0.2088, Forward time: 1.2537, Backward time: 1.2445, DDP time: 0.0043, Total iter time: 2.9491 [2021-07-07 11:03:14,464][root][INFO] - ===> Start testing [2021-07-07 11:03:50,213][root][INFO] - 68/68: Data time: 0.0018, Iter time: 0.1368 Loss 0.231 (AVG: 0.470) Score 91.891 (AVG: 90.744) mIOU 69.946 mAP 84.482 mAcc 75.873 IOU: 65.018 0.012 77.434 77.482 93.836 90.894 29.414 84.000 97.424 70.732 80.700 85.104 57.245 mAP: 85.010 9.755 91.373 91.220 98.801 98.743 60.158 96.884 99.509 84.446 94.217 95.023 93.123 mAcc: 80.937 0.091 88.630 82.896 95.546 96.519 31.431 89.603 99.469 72.553 89.933 96.665 62.081 [2021-07-07 11:03:50,213][root][INFO] - Finished test. Elapsed time: 35.7484 [2021-07-07 11:03:51,286][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 11:04:05,265][root][INFO] - ===> Epoch[321](1605/5): Loss 0.0402, Sem 0.0402 LR: 9.759e-02 Score 98.584 Data time: 0.2755, Forward time: 1.2535, Backward time: 1.2444, DDP time: 0.0043, Total iter time: 2.7439 [2021-07-07 11:04:19,676][root][INFO] - ===> Epoch[322](1610/5): Loss 0.0389, Sem 0.0389 LR: 9.758e-02 Score 98.607 Data time: 0.2354, Forward time: 1.2536, Backward time: 1.2445, DDP time: 0.0043, Total iter time: 2.8264 [2021-07-07 11:04:33,605][root][INFO] - ===> Epoch[323](1615/5): Loss 0.0401, Sem 0.0401 LR: 9.757e-02 Score 98.582 Data time: 0.1989, Forward time: 1.2537, Backward time: 1.2445, DDP time: 0.0043, Total iter time: 2.7457 [2021-07-07 11:04:47,436][root][INFO] - ===> Epoch[324](1620/5): Loss 0.0382, Sem 0.0382 LR: 9.757e-02 Score 98.618 Data time: 0.2200, Forward time: 1.2535, Backward time: 1.2446, DDP time: 0.0043, Total iter time: 2.7267 [2021-07-07 11:05:01,565][root][INFO] - ===> Epoch[325](1625/5): Loss 0.0379, Sem 0.0379 LR: 9.756e-02 Score 98.650 Data time: 0.2425, Forward time: 1.2537, Backward time: 1.2444, DDP time: 0.0043, Total iter time: 2.7809 [2021-07-07 11:05:14,855][root][INFO] - ===> Epoch[326](1630/5): Loss 0.0382, Sem 0.0382 LR: 9.755e-02 Score 98.631 Data time: 0.2647, Forward time: 1.2536, Backward time: 1.2440, DDP time: 0.0043, Total iter time: 2.6171 [2021-07-07 11:05:28,352][root][INFO] - ===> Epoch[327](1635/5): Loss 0.0399, Sem 0.0399 LR: 9.754e-02 Score 98.583 Data time: 0.2653, Forward time: 1.2538, Backward time: 1.2434, DDP time: 0.0043, Total iter time: 2.6510 [2021-07-07 11:05:41,914][root][INFO] - ===> Epoch[328](1640/5): Loss 0.0390, Sem 0.0390 LR: 9.754e-02 Score 98.612 Data time: 0.1671, Forward time: 1.2538, Backward time: 1.2432, DDP time: 0.0043, Total iter time: 2.6700 [2021-07-07 11:05:56,103][root][INFO] - ===> Epoch[329](1645/5): Loss 0.0387, Sem 0.0387 LR: 9.753e-02 Score 98.619 Data time: 0.2110, Forward time: 1.2537, Backward time: 1.2435, DDP time: 0.0043, Total iter time: 2.7850 [2021-07-07 11:06:09,355][root][INFO] - ===> Epoch[330](1650/5): Loss 0.0381, Sem 0.0381 LR: 9.752e-02 Score 98.645 Data time: 0.2460, Forward time: 1.2536, Backward time: 1.2431, DDP time: 0.0043, Total iter time: 2.5978 [2021-07-07 11:06:23,477][root][INFO] - ===> Epoch[331](1655/5): Loss 0.0376, Sem 0.0376 LR: 9.751e-02 Score 98.664 Data time: 0.2010, Forward time: 1.2538, Backward time: 1.2430, DDP time: 0.0043, Total iter time: 2.7785 [2021-07-07 11:06:36,957][root][INFO] - ===> Epoch[332](1660/5): Loss 0.0372, Sem 0.0372 LR: 9.751e-02 Score 98.677 Data time: 0.1896, Forward time: 1.2538, Backward time: 1.2428, DDP time: 0.0043, Total iter time: 2.6502 [2021-07-07 11:06:50,717][root][INFO] - ===> Epoch[333](1665/5): Loss 0.0394, Sem 0.0394 LR: 9.750e-02 Score 98.594 Data time: 0.2028, Forward time: 1.2539, Backward time: 1.2426, DDP time: 0.0043, Total iter time: 2.7098 [2021-07-07 11:07:04,657][root][INFO] - ===> Epoch[334](1670/5): Loss 0.0383, Sem 0.0383 LR: 9.749e-02 Score 98.633 Data time: 0.2744, Forward time: 1.2540, Backward time: 1.2424, DDP time: 0.0043, Total iter time: 2.7387 [2021-07-07 11:07:18,732][root][INFO] - ===> Epoch[335](1675/5): Loss 0.0371, Sem 0.0371 LR: 9.748e-02 Score 98.675 Data time: 0.2513, Forward time: 1.2541, Backward time: 1.2422, DDP time: 0.0043, Total iter time: 2.7663 [2021-07-07 11:07:32,319][root][INFO] - ===> Epoch[336](1680/5): Loss 0.0401, Sem 0.0401 LR: 9.748e-02 Score 98.573 Data time: 0.2418, Forward time: 1.2539, Backward time: 1.2420, DDP time: 0.0043, Total iter time: 2.6708 [2021-07-07 11:07:46,878][root][INFO] - ===> Epoch[337](1685/5): Loss 0.0383, Sem 0.0383 LR: 9.747e-02 Score 98.646 Data time: 0.2729, Forward time: 1.2540, Backward time: 1.2422, DDP time: 0.0043, Total iter time: 2.8651 [2021-07-07 11:08:01,325][root][INFO] - ===> Epoch[338](1690/5): Loss 0.0372, Sem 0.0372 LR: 9.746e-02 Score 98.665 Data time: 0.2654, Forward time: 1.2538, Backward time: 1.2424, DDP time: 0.0043, Total iter time: 2.8358 [2021-07-07 11:08:14,952][root][INFO] - ===> Epoch[339](1695/5): Loss 0.0382, Sem 0.0382 LR: 9.745e-02 Score 98.638 Data time: 0.2030, Forward time: 1.2535, Backward time: 1.2426, DDP time: 0.0043, Total iter time: 2.6833 [2021-07-07 11:08:29,938][root][INFO] - ===> Epoch[340](1700/5): Loss 0.0388, Sem 0.0388 LR: 9.745e-02 Score 98.619 Data time: 0.2300, Forward time: 1.2536, Backward time: 1.2430, DDP time: 0.0043, Total iter time: 2.9540 [2021-07-07 11:08:44,438][root][INFO] - ===> Epoch[341](1705/5): Loss 0.0375, Sem 0.0375 LR: 9.744e-02 Score 98.670 Data time: 0.2397, Forward time: 1.2533, Backward time: 1.2435, DDP time: 0.0043, Total iter time: 2.8561 [2021-07-07 11:08:57,900][root][INFO] - ===> Epoch[342](1710/5): Loss 0.0384, Sem 0.0384 LR: 9.743e-02 Score 98.638 Data time: 0.2425, Forward time: 1.2533, Backward time: 1.2432, DDP time: 0.0043, Total iter time: 2.6506 [2021-07-07 11:09:11,716][root][INFO] - ===> Epoch[343](1715/5): Loss 0.0385, Sem 0.0385 LR: 9.742e-02 Score 98.637 Data time: 0.2056, Forward time: 1.2535, Backward time: 1.2429, DDP time: 0.0043, Total iter time: 2.7014 [2021-07-07 11:09:26,343][root][INFO] - ===> Epoch[344](1720/5): Loss 0.0386, Sem 0.0386 LR: 9.742e-02 Score 98.636 Data time: 0.2299, Forward time: 1.2538, Backward time: 1.2429, DDP time: 0.0043, Total iter time: 2.8754 [2021-07-07 11:09:40,315][root][INFO] - ===> Epoch[345](1725/5): Loss 0.0381, Sem 0.0381 LR: 9.741e-02 Score 98.665 Data time: 0.1938, Forward time: 1.2541, Backward time: 1.2427, DDP time: 0.0043, Total iter time: 2.7394 [2021-07-07 11:09:54,372][root][INFO] - ===> Epoch[346](1730/5): Loss 0.0382, Sem 0.0382 LR: 9.740e-02 Score 98.631 Data time: 0.2320, Forward time: 1.2542, Backward time: 1.2426, DDP time: 0.0043, Total iter time: 2.7583 [2021-07-07 11:10:08,038][root][INFO] - ===> Epoch[347](1735/5): Loss 0.0384, Sem 0.0384 LR: 9.739e-02 Score 98.626 Data time: 0.2395, Forward time: 1.2540, Backward time: 1.2425, DDP time: 0.0043, Total iter time: 2.6918 [2021-07-07 11:10:21,813][root][INFO] - ===> Epoch[348](1740/5): Loss 0.0390, Sem 0.0390 LR: 9.739e-02 Score 98.617 Data time: 0.2688, Forward time: 1.2538, Backward time: 1.2424, DDP time: 0.0043, Total iter time: 2.6861 [2021-07-07 11:10:35,430][root][INFO] - ===> Epoch[349](1745/5): Loss 0.0398, Sem 0.0398 LR: 9.738e-02 Score 98.625 Data time: 0.2178, Forward time: 1.2536, Backward time: 1.2424, DDP time: 0.0043, Total iter time: 2.6783 [2021-07-07 11:10:49,389][root][INFO] - ===> Epoch[350](1750/5): Loss 0.0414, Sem 0.0414 LR: 9.737e-02 Score 98.552 Data time: 0.2614, Forward time: 1.2537, Backward time: 1.2422, DDP time: 0.0043, Total iter time: 2.7521 [2021-07-07 11:11:03,332][root][INFO] - ===> Epoch[351](1755/5): Loss 0.0398, Sem 0.0398 LR: 9.736e-02 Score 98.575 Data time: 0.2107, Forward time: 1.2537, Backward time: 1.2422, DDP time: 0.0043, Total iter time: 2.7487 [2021-07-07 11:11:16,603][root][INFO] - ===> Epoch[352](1760/5): Loss 0.0395, Sem 0.0395 LR: 9.736e-02 Score 98.578 Data time: 0.2383, Forward time: 1.2537, Backward time: 1.2418, DDP time: 0.0043, Total iter time: 2.6151 [2021-07-07 11:11:30,279][root][INFO] - ===> Epoch[353](1765/5): Loss 0.0403, Sem 0.0403 LR: 9.735e-02 Score 98.566 Data time: 0.1957, Forward time: 1.2536, Backward time: 1.2418, DDP time: 0.0043, Total iter time: 2.6941 [2021-07-07 11:11:43,657][root][INFO] - ===> Epoch[354](1770/5): Loss 0.0394, Sem 0.0394 LR: 9.734e-02 Score 98.606 Data time: 0.2684, Forward time: 1.2536, Backward time: 1.2413, DDP time: 0.0043, Total iter time: 2.6303 [2021-07-07 11:11:57,235][root][INFO] - ===> Epoch[355](1775/5): Loss 0.0385, Sem 0.0385 LR: 9.733e-02 Score 98.638 Data time: 0.2175, Forward time: 1.2535, Backward time: 1.2412, DDP time: 0.0043, Total iter time: 2.6644 [2021-07-07 11:12:11,901][root][INFO] - ===> Epoch[356](1780/5): Loss 0.0387, Sem 0.0387 LR: 9.733e-02 Score 98.616 Data time: 0.1806, Forward time: 1.2534, Backward time: 1.2418, DDP time: 0.0043, Total iter time: 2.8824 [2021-07-07 11:12:26,698][root][INFO] - ===> Epoch[357](1785/5): Loss 0.0403, Sem 0.0403 LR: 9.732e-02 Score 98.552 Data time: 0.2840, Forward time: 1.2537, Backward time: 1.2418, DDP time: 0.0043, Total iter time: 2.9117 [2021-07-07 11:12:39,728][root][INFO] - ===> Epoch[358](1790/5): Loss 0.0375, Sem 0.0375 LR: 9.731e-02 Score 98.663 Data time: 0.2659, Forward time: 1.2536, Backward time: 1.2412, DDP time: 0.0043, Total iter time: 2.5586 [2021-07-07 11:12:54,355][root][INFO] - ===> Epoch[359](1795/5): Loss 0.0388, Sem 0.0388 LR: 9.730e-02 Score 98.607 Data time: 0.2678, Forward time: 1.2535, Backward time: 1.2415, DDP time: 0.0043, Total iter time: 2.8788 [2021-07-07 11:13:07,991][root][INFO] - ===> Epoch[360](1800/5): Loss 0.0389, Sem 0.0389 LR: 9.730e-02 Score 98.618 Data time: 0.2104, Forward time: 1.2536, Backward time: 1.2412, DDP time: 0.0043, Total iter time: 2.6824 [2021-07-07 11:13:07,992][root][INFO] - ===> Start testing [2021-07-07 11:13:42,418][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.2435 Loss 0.216 (AVG: 0.470) Score 93.186 (AVG: 90.823) mIOU 70.508 mAP 85.130 mAcc 76.171 IOU: 64.274 0.081 79.016 76.955 94.702 91.252 25.497 84.023 98.757 82.804 79.915 85.358 53.976 mAP: 84.744 12.677 91.919 90.555 99.021 98.635 56.081 97.374 99.842 95.158 93.835 95.090 91.761 mAcc: 78.447 0.729 86.381 84.045 96.959 95.042 26.724 93.156 99.650 85.320 88.881 96.944 57.942 [2021-07-07 11:13:42,418][root][INFO] - Finished test. Elapsed time: 34.4252 [2021-07-07 11:13:43,495][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2021-07-07 11:13:43,496][root][INFO] - Current best mIoU: 70.508 at iter 1 [2021-07-07 11:13:44,439][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 11:13:58,040][root][INFO] - ===> Epoch[361](1805/5): Loss 0.0402, Sem 0.0402 LR: 9.729e-02 Score 98.567 Data time: 0.2357, Forward time: 1.2537, Backward time: 1.2410, DDP time: 0.0043, Total iter time: 2.6754 [2021-07-07 11:14:12,467][root][INFO] - ===> Epoch[362](1810/5): Loss 0.0388, Sem 0.0388 LR: 9.728e-02 Score 98.628 Data time: 0.2260, Forward time: 1.2534, Backward time: 1.2414, DDP time: 0.0043, Total iter time: 2.8372 [2021-07-07 11:14:26,102][root][INFO] - ===> Epoch[363](1815/5): Loss 0.0366, Sem 0.0366 LR: 9.727e-02 Score 98.691 Data time: 0.2345, Forward time: 1.2537, Backward time: 1.2409, DDP time: 0.0043, Total iter time: 2.6748 [2021-07-07 11:14:39,824][root][INFO] - ===> Epoch[364](1820/5): Loss 0.0364, Sem 0.0364 LR: 9.727e-02 Score 98.700 Data time: 0.2580, Forward time: 1.2539, Backward time: 1.2405, DDP time: 0.0043, Total iter time: 2.6929 [2021-07-07 11:14:53,835][root][INFO] - ===> Epoch[365](1825/5): Loss 0.0377, Sem 0.0377 LR: 9.726e-02 Score 98.660 Data time: 0.2331, Forward time: 1.2541, Backward time: 1.2403, DDP time: 0.0043, Total iter time: 2.7592 [2021-07-07 11:15:07,866][root][INFO] - ===> Epoch[366](1830/5): Loss 0.0360, Sem 0.0360 LR: 9.725e-02 Score 98.723 Data time: 0.2096, Forward time: 1.2542, Backward time: 1.2403, DDP time: 0.0043, Total iter time: 2.7687 [2021-07-07 11:15:21,459][root][INFO] - ===> Epoch[367](1835/5): Loss 0.0385, Sem 0.0385 LR: 9.724e-02 Score 98.618 Data time: 0.2080, Forward time: 1.2541, Backward time: 1.2402, DDP time: 0.0043, Total iter time: 2.6723 [2021-07-07 11:15:35,945][root][INFO] - ===> Epoch[368](1840/5): Loss 0.0376, Sem 0.0376 LR: 9.724e-02 Score 98.655 Data time: 0.2429, Forward time: 1.2541, Backward time: 1.2404, DDP time: 0.0043, Total iter time: 2.8507 [2021-07-07 11:15:49,530][root][INFO] - ===> Epoch[369](1845/5): Loss 0.0412, Sem 0.0412 LR: 9.723e-02 Score 98.554 Data time: 0.1913, Forward time: 1.2536, Backward time: 1.2407, DDP time: 0.0043, Total iter time: 2.6712 [2021-07-07 11:16:02,986][root][INFO] - ===> Epoch[370](1850/5): Loss 0.0385, Sem 0.0385 LR: 9.722e-02 Score 98.643 Data time: 0.2308, Forward time: 1.2536, Backward time: 1.2404, DDP time: 0.0043, Total iter time: 2.6331 [2021-07-07 11:16:16,871][root][INFO] - ===> Epoch[371](1855/5): Loss 0.0404, Sem 0.0404 LR: 9.721e-02 Score 98.589 Data time: 0.1861, Forward time: 1.2538, Backward time: 1.2403, DDP time: 0.0043, Total iter time: 2.7381 [2021-07-07 11:16:30,139][root][INFO] - ===> Epoch[372](1860/5): Loss 0.0403, Sem 0.0403 LR: 9.721e-02 Score 98.571 Data time: 0.3112, Forward time: 1.2535, Backward time: 1.2399, DDP time: 0.0043, Total iter time: 2.6014 [2021-07-07 11:16:44,088][root][INFO] - ===> Epoch[373](1865/5): Loss 0.0399, Sem 0.0399 LR: 9.720e-02 Score 98.593 Data time: 0.2914, Forward time: 1.2537, Backward time: 1.2395, DDP time: 0.0043, Total iter time: 2.7331 [2021-07-07 11:16:57,515][root][INFO] - ===> Epoch[374](1870/5): Loss 0.0385, Sem 0.0385 LR: 9.719e-02 Score 98.642 Data time: 0.2515, Forward time: 1.2538, Backward time: 1.2391, DDP time: 0.0043, Total iter time: 2.6404 [2021-07-07 11:17:11,119][root][INFO] - ===> Epoch[375](1875/5): Loss 0.0391, Sem 0.0391 LR: 9.718e-02 Score 98.630 Data time: 0.2914, Forward time: 1.2538, Backward time: 1.2387, DDP time: 0.0043, Total iter time: 2.6769 [2021-07-07 11:17:25,464][root][INFO] - ===> Epoch[376](1880/5): Loss 0.0402, Sem 0.0402 LR: 9.718e-02 Score 98.564 Data time: 0.2314, Forward time: 1.2543, Backward time: 1.2385, DDP time: 0.0043, Total iter time: 2.8278 [2021-07-07 11:17:38,848][root][INFO] - ===> Epoch[377](1885/5): Loss 0.0365, Sem 0.0365 LR: 9.717e-02 Score 98.693 Data time: 0.2221, Forward time: 1.2544, Backward time: 1.2380, DDP time: 0.0043, Total iter time: 2.6333 [2021-07-07 11:17:53,026][root][INFO] - ===> Epoch[378](1890/5): Loss 0.0383, Sem 0.0383 LR: 9.716e-02 Score 98.637 Data time: 0.1996, Forward time: 1.2544, Backward time: 1.2382, DDP time: 0.0043, Total iter time: 2.7847 [2021-07-07 11:18:06,668][root][INFO] - ===> Epoch[379](1895/5): Loss 0.0370, Sem 0.0370 LR: 9.715e-02 Score 98.681 Data time: 0.2847, Forward time: 1.2541, Backward time: 1.2381, DDP time: 0.0043, Total iter time: 2.6885 [2021-07-07 11:18:20,502][root][INFO] - ===> Epoch[380](1900/5): Loss 0.0385, Sem 0.0385 LR: 9.715e-02 Score 98.651 Data time: 0.2684, Forward time: 1.2539, Backward time: 1.2381, DDP time: 0.0043, Total iter time: 2.7067 [2021-07-07 11:18:34,012][root][INFO] - ===> Epoch[381](1905/5): Loss 0.0373, Sem 0.0373 LR: 9.714e-02 Score 98.669 Data time: 0.2215, Forward time: 1.2538, Backward time: 1.2379, DDP time: 0.0043, Total iter time: 2.6582 [2021-07-07 11:18:47,965][root][INFO] - ===> Epoch[382](1910/5): Loss 0.0375, Sem 0.0375 LR: 9.713e-02 Score 98.663 Data time: 0.1911, Forward time: 1.2538, Backward time: 1.2380, DDP time: 0.0043, Total iter time: 2.7478 [2021-07-07 11:19:02,023][root][INFO] - ===> Epoch[383](1915/5): Loss 0.0380, Sem 0.0380 LR: 9.712e-02 Score 98.641 Data time: 0.2998, Forward time: 1.2538, Backward time: 1.2378, DDP time: 0.0043, Total iter time: 2.7566 [2021-07-07 11:19:16,090][root][INFO] - ===> Epoch[384](1920/5): Loss 0.0372, Sem 0.0372 LR: 9.712e-02 Score 98.667 Data time: 0.2430, Forward time: 1.2539, Backward time: 1.2377, DDP time: 0.0043, Total iter time: 2.7670 [2021-07-07 11:19:30,808][root][INFO] - ===> Epoch[385](1925/5): Loss 0.0365, Sem 0.0365 LR: 9.711e-02 Score 98.686 Data time: 0.1976, Forward time: 1.2539, Backward time: 1.2382, DDP time: 0.0043, Total iter time: 2.8985 [2021-07-07 11:19:44,988][root][INFO] - ===> Epoch[386](1930/5): Loss 0.0377, Sem 0.0377 LR: 9.710e-02 Score 98.656 Data time: 0.2015, Forward time: 1.2536, Backward time: 1.2386, DDP time: 0.0043, Total iter time: 2.7865 [2021-07-07 11:19:59,910][root][INFO] - ===> Epoch[387](1935/5): Loss 0.0379, Sem 0.0379 LR: 9.709e-02 Score 98.657 Data time: 0.2310, Forward time: 1.2534, Backward time: 1.2392, DDP time: 0.0043, Total iter time: 2.9371 [2021-07-07 11:20:13,831][root][INFO] - ===> Epoch[388](1940/5): Loss 0.0361, Sem 0.0361 LR: 9.709e-02 Score 98.713 Data time: 0.2635, Forward time: 1.2535, Backward time: 1.2390, DDP time: 0.0043, Total iter time: 2.7289 [2021-07-07 11:20:27,042][root][INFO] - ===> Epoch[389](1945/5): Loss 0.0369, Sem 0.0369 LR: 9.708e-02 Score 98.680 Data time: 0.2063, Forward time: 1.2533, Backward time: 1.2388, DDP time: 0.0043, Total iter time: 2.5839 [2021-07-07 11:20:41,519][root][INFO] - ===> Epoch[390](1950/5): Loss 0.0345, Sem 0.0345 LR: 9.707e-02 Score 98.772 Data time: 0.2532, Forward time: 1.2532, Backward time: 1.2390, DDP time: 0.0043, Total iter time: 2.8392 [2021-07-07 11:20:54,678][root][INFO] - ===> Epoch[391](1955/5): Loss 0.0356, Sem 0.0356 LR: 9.706e-02 Score 98.716 Data time: 0.2507, Forward time: 1.2533, Backward time: 1.2385, DDP time: 0.0043, Total iter time: 2.5865 [2021-07-07 11:21:07,924][root][INFO] - ===> Epoch[392](1960/5): Loss 0.0358, Sem 0.0358 LR: 9.706e-02 Score 98.723 Data time: 0.1912, Forward time: 1.2534, Backward time: 1.2381, DDP time: 0.0043, Total iter time: 2.6014 [2021-07-07 11:21:22,287][root][INFO] - ===> Epoch[393](1965/5): Loss 0.0383, Sem 0.0383 LR: 9.705e-02 Score 98.662 Data time: 0.2770, Forward time: 1.2534, Backward time: 1.2381, DDP time: 0.0043, Total iter time: 2.8154 [2021-07-07 11:21:36,739][root][INFO] - ===> Epoch[394](1970/5): Loss 0.0341, Sem 0.0341 LR: 9.704e-02 Score 98.801 Data time: 0.2776, Forward time: 1.2536, Backward time: 1.2380, DDP time: 0.0042, Total iter time: 2.8277 [2021-07-07 11:21:50,371][root][INFO] - ===> Epoch[395](1975/5): Loss 0.0348, Sem 0.0348 LR: 9.703e-02 Score 98.755 Data time: 0.2321, Forward time: 1.2537, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6770 [2021-07-07 11:22:04,344][root][INFO] - ===> Epoch[396](1980/5): Loss 0.0366, Sem 0.0366 LR: 9.703e-02 Score 98.689 Data time: 0.2665, Forward time: 1.2539, Backward time: 1.2373, DDP time: 0.0042, Total iter time: 2.7395 [2021-07-07 11:22:18,364][root][INFO] - ===> Epoch[397](1985/5): Loss 0.0350, Sem 0.0350 LR: 9.702e-02 Score 98.759 Data time: 0.2715, Forward time: 1.2542, Backward time: 1.2370, DDP time: 0.0042, Total iter time: 2.7377 [2021-07-07 11:22:32,619][root][INFO] - ===> Epoch[398](1990/5): Loss 0.0351, Sem 0.0351 LR: 9.701e-02 Score 98.752 Data time: 0.2261, Forward time: 1.2545, Backward time: 1.2369, DDP time: 0.0042, Total iter time: 2.8117 [2021-07-07 11:22:47,022][root][INFO] - ===> Epoch[399](1995/5): Loss 0.0347, Sem 0.0347 LR: 9.700e-02 Score 98.760 Data time: 0.1803, Forward time: 1.2545, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.8445 [2021-07-07 11:23:00,978][root][INFO] - ===> Epoch[400](2000/5): Loss 0.0346, Sem 0.0346 LR: 9.699e-02 Score 98.759 Data time: 0.2326, Forward time: 1.2547, Backward time: 1.2369, DDP time: 0.0042, Total iter time: 2.7434 [2021-07-07 11:23:00,980][root][INFO] - ===> Start testing [2021-07-07 11:23:36,567][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1231 Loss 0.260 (AVG: 0.447) Score 92.132 (AVG: 91.282) mIOU 71.139 mAP 85.388 mAcc 77.570 IOU: 65.295 0.084 77.396 77.721 95.265 90.062 36.602 81.349 98.551 79.512 80.055 86.945 55.975 mAP: 85.031 9.832 92.574 90.215 98.896 98.556 64.469 97.200 99.842 91.257 93.914 95.411 92.851 mAcc: 80.537 0.729 90.331 84.840 96.928 96.621 38.839 92.224 99.365 82.015 88.535 96.467 60.984 [2021-07-07 11:23:36,567][root][INFO] - Finished test. Elapsed time: 35.5868 [2021-07-07 11:23:37,678][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2021-07-07 11:23:37,679][root][INFO] - Current best mIoU: 71.139 at iter 1 [2021-07-07 11:23:38,632][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 11:23:51,571][root][INFO] - ===> Epoch[401](2005/5): Loss 0.0352, Sem 0.0352 LR: 9.699e-02 Score 98.736 Data time: 0.1587, Forward time: 1.2543, Backward time: 1.2369, DDP time: 0.0042, Total iter time: 2.5572 [2021-07-07 11:24:04,905][root][INFO] - ===> Epoch[402](2010/5): Loss 0.0343, Sem 0.0343 LR: 9.698e-02 Score 98.760 Data time: 0.2057, Forward time: 1.2542, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.6225 [2021-07-07 11:24:18,667][root][INFO] - ===> Epoch[403](2015/5): Loss 0.0357, Sem 0.0357 LR: 9.697e-02 Score 98.726 Data time: 0.2563, Forward time: 1.2543, Backward time: 1.2366, DDP time: 0.0042, Total iter time: 2.6939 [2021-07-07 11:24:32,673][root][INFO] - ===> Epoch[404](2020/5): Loss 0.0340, Sem 0.0340 LR: 9.696e-02 Score 98.794 Data time: 0.2066, Forward time: 1.2541, Backward time: 1.2367, DDP time: 0.0042, Total iter time: 2.7626 [2021-07-07 11:24:46,709][root][INFO] - ===> Epoch[405](2025/5): Loss 0.0359, Sem 0.0359 LR: 9.696e-02 Score 98.712 Data time: 0.2883, Forward time: 1.2544, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7531 [2021-07-07 11:24:59,556][root][INFO] - ===> Epoch[406](2030/5): Loss 0.0344, Sem 0.0344 LR: 9.695e-02 Score 98.769 Data time: 0.1786, Forward time: 1.2541, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.5233 [2021-07-07 11:25:14,174][root][INFO] - ===> Epoch[407](2035/5): Loss 0.0360, Sem 0.0360 LR: 9.694e-02 Score 98.705 Data time: 0.2129, Forward time: 1.2545, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.8787 [2021-07-07 11:25:28,528][root][INFO] - ===> Epoch[408](2040/5): Loss 0.0328, Sem 0.0328 LR: 9.693e-02 Score 98.812 Data time: 0.2579, Forward time: 1.2541, Backward time: 1.2366, DDP time: 0.0042, Total iter time: 2.8182 [2021-07-07 11:25:43,179][root][INFO] - ===> Epoch[409](2045/5): Loss 0.0349, Sem 0.0349 LR: 9.693e-02 Score 98.762 Data time: 0.2443, Forward time: 1.2542, Backward time: 1.2367, DDP time: 0.0042, Total iter time: 2.8514 [2021-07-07 11:25:57,437][root][INFO] - ===> Epoch[410](2050/5): Loss 0.0349, Sem 0.0349 LR: 9.692e-02 Score 98.770 Data time: 0.2184, Forward time: 1.2543, Backward time: 1.2367, DDP time: 0.0042, Total iter time: 2.7953 [2021-07-07 11:26:11,650][root][INFO] - ===> Epoch[411](2055/5): Loss 0.0349, Sem 0.0349 LR: 9.691e-02 Score 98.755 Data time: 0.2440, Forward time: 1.2545, Backward time: 1.2366, DDP time: 0.0042, Total iter time: 2.7884 [2021-07-07 11:26:26,149][root][INFO] - ===> Epoch[412](2060/5): Loss 0.0345, Sem 0.0345 LR: 9.690e-02 Score 98.772 Data time: 0.2919, Forward time: 1.2546, Backward time: 1.2366, DDP time: 0.0042, Total iter time: 2.8548 [2021-07-07 11:26:39,237][root][INFO] - ===> Epoch[413](2065/5): Loss 0.0348, Sem 0.0348 LR: 9.690e-02 Score 98.759 Data time: 0.2042, Forward time: 1.2544, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.5651 [2021-07-07 11:26:53,162][root][INFO] - ===> Epoch[414](2070/5): Loss 0.0331, Sem 0.0331 LR: 9.689e-02 Score 98.804 Data time: 0.2234, Forward time: 1.2542, Backward time: 1.2365, DDP time: 0.0042, Total iter time: 2.7353 [2021-07-07 11:27:07,201][root][INFO] - ===> Epoch[415](2075/5): Loss 0.0351, Sem 0.0351 LR: 9.688e-02 Score 98.744 Data time: 0.2074, Forward time: 1.2544, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7609 [2021-07-07 11:27:21,052][root][INFO] - ===> Epoch[416](2080/5): Loss 0.0331, Sem 0.0331 LR: 9.687e-02 Score 98.808 Data time: 0.1846, Forward time: 1.2544, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7256 [2021-07-07 11:27:34,673][root][INFO] - ===> Epoch[417](2085/5): Loss 0.0362, Sem 0.0362 LR: 9.687e-02 Score 98.724 Data time: 0.2094, Forward time: 1.2543, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.6754 [2021-07-07 11:27:49,058][root][INFO] - ===> Epoch[418](2090/5): Loss 0.0343, Sem 0.0343 LR: 9.686e-02 Score 98.778 Data time: 0.2318, Forward time: 1.2542, Backward time: 1.2367, DDP time: 0.0042, Total iter time: 2.8296 [2021-07-07 11:28:03,328][root][INFO] - ===> Epoch[419](2095/5): Loss 0.0342, Sem 0.0342 LR: 9.685e-02 Score 98.789 Data time: 0.2347, Forward time: 1.2541, Backward time: 1.2369, DDP time: 0.0042, Total iter time: 2.7977 [2021-07-07 11:28:16,694][root][INFO] - ===> Epoch[420](2100/5): Loss 0.0352, Sem 0.0352 LR: 9.684e-02 Score 98.739 Data time: 0.1968, Forward time: 1.2541, Backward time: 1.2367, DDP time: 0.0042, Total iter time: 2.6400 [2021-07-07 11:28:30,242][root][INFO] - ===> Epoch[421](2105/5): Loss 0.0329, Sem 0.0329 LR: 9.684e-02 Score 98.812 Data time: 0.2114, Forward time: 1.2541, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.6657 [2021-07-07 11:28:43,872][root][INFO] - ===> Epoch[422](2110/5): Loss 0.0348, Sem 0.0348 LR: 9.683e-02 Score 98.767 Data time: 0.1697, Forward time: 1.2543, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6755 [2021-07-07 11:28:58,245][root][INFO] - ===> Epoch[423](2115/5): Loss 0.0351, Sem 0.0351 LR: 9.682e-02 Score 98.757 Data time: 0.2542, Forward time: 1.2543, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8210 [2021-07-07 11:29:12,819][root][INFO] - ===> Epoch[424](2120/5): Loss 0.0338, Sem 0.0338 LR: 9.681e-02 Score 98.802 Data time: 0.2340, Forward time: 1.2538, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.8670 [2021-07-07 11:29:26,032][root][INFO] - ===> Epoch[425](2125/5): Loss 0.0348, Sem 0.0348 LR: 9.681e-02 Score 98.742 Data time: 0.1858, Forward time: 1.2537, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.5775 [2021-07-07 11:29:39,325][root][INFO] - ===> Epoch[426](2130/5): Loss 0.0355, Sem 0.0355 LR: 9.680e-02 Score 98.724 Data time: 0.2441, Forward time: 1.2538, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.6115 [2021-07-07 11:29:53,616][root][INFO] - ===> Epoch[427](2135/5): Loss 0.0349, Sem 0.0349 LR: 9.679e-02 Score 98.752 Data time: 0.2140, Forward time: 1.2539, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.8075 [2021-07-07 11:30:08,798][root][INFO] - ===> Epoch[428](2140/5): Loss 0.0343, Sem 0.0343 LR: 9.678e-02 Score 98.782 Data time: 0.2598, Forward time: 1.2539, Backward time: 1.2369, DDP time: 0.0042, Total iter time: 2.9908 [2021-07-07 11:30:21,775][root][INFO] - ===> Epoch[429](2145/5): Loss 0.0354, Sem 0.0354 LR: 9.678e-02 Score 98.747 Data time: 0.1940, Forward time: 1.2536, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.5490 [2021-07-07 11:30:35,699][root][INFO] - ===> Epoch[430](2150/5): Loss 0.0334, Sem 0.0334 LR: 9.677e-02 Score 98.807 Data time: 0.2034, Forward time: 1.2537, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.7388 [2021-07-07 11:30:48,593][root][INFO] - ===> Epoch[431](2155/5): Loss 0.0358, Sem 0.0358 LR: 9.676e-02 Score 98.724 Data time: 0.2499, Forward time: 1.2536, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.5220 [2021-07-07 11:31:02,431][root][INFO] - ===> Epoch[432](2160/5): Loss 0.0338, Sem 0.0338 LR: 9.675e-02 Score 98.794 Data time: 0.2804, Forward time: 1.2535, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.7244 [2021-07-07 11:31:17,362][root][INFO] - ===> Epoch[433](2165/5): Loss 0.0330, Sem 0.0330 LR: 9.675e-02 Score 98.806 Data time: 0.1596, Forward time: 1.2538, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.9401 [2021-07-07 11:31:30,668][root][INFO] - ===> Epoch[434](2170/5): Loss 0.0334, Sem 0.0334 LR: 9.674e-02 Score 98.807 Data time: 0.2402, Forward time: 1.2539, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.6185 [2021-07-07 11:31:44,180][root][INFO] - ===> Epoch[435](2175/5): Loss 0.0329, Sem 0.0329 LR: 9.673e-02 Score 98.823 Data time: 0.2641, Forward time: 1.2541, Backward time: 1.2355, DDP time: 0.0042, Total iter time: 2.6533 [2021-07-07 11:31:58,654][root][INFO] - ===> Epoch[436](2180/5): Loss 0.0332, Sem 0.0332 LR: 9.672e-02 Score 98.813 Data time: 0.2039, Forward time: 1.2542, Backward time: 1.2357, DDP time: 0.0042, Total iter time: 2.8394 [2021-07-07 11:32:12,267][root][INFO] - ===> Epoch[437](2185/5): Loss 0.0345, Sem 0.0345 LR: 9.672e-02 Score 98.762 Data time: 0.1942, Forward time: 1.2543, Backward time: 1.2355, DDP time: 0.0042, Total iter time: 2.6796 [2021-07-07 11:32:26,426][root][INFO] - ===> Epoch[438](2190/5): Loss 0.0343, Sem 0.0343 LR: 9.671e-02 Score 98.769 Data time: 0.2383, Forward time: 1.2542, Backward time: 1.2357, DDP time: 0.0042, Total iter time: 2.7852 [2021-07-07 11:32:40,365][root][INFO] - ===> Epoch[439](2195/5): Loss 0.0332, Sem 0.0332 LR: 9.670e-02 Score 98.820 Data time: 0.2180, Forward time: 1.2542, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.7499 [2021-07-07 11:32:54,966][root][INFO] - ===> Epoch[440](2200/5): Loss 0.0325, Sem 0.0325 LR: 9.669e-02 Score 98.843 Data time: 0.2477, Forward time: 1.2544, Backward time: 1.2357, DDP time: 0.0042, Total iter time: 2.8727 [2021-07-07 11:32:54,968][root][INFO] - ===> Start testing [2021-07-07 11:33:29,839][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1229 Loss 0.378 (AVG: 0.454) Score 90.439 (AVG: 90.968) mIOU 70.833 mAP 85.271 mAcc 76.724 IOU: 65.097 0.000 78.776 78.021 94.547 90.808 38.840 81.200 98.616 75.986 79.067 85.832 54.038 mAP: 85.430 8.987 90.567 91.188 98.985 98.089 68.553 96.489 99.849 89.342 93.584 95.570 91.891 mAcc: 81.003 0.000 86.827 85.703 95.754 96.073 45.667 88.708 99.488 77.733 86.055 96.502 57.902 [2021-07-07 11:33:29,839][root][INFO] - Finished test. Elapsed time: 34.8700 [2021-07-07 11:33:30,851][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 11:33:43,641][root][INFO] - ===> Epoch[441](2205/5): Loss 0.0336, Sem 0.0336 LR: 9.669e-02 Score 98.791 Data time: 0.1669, Forward time: 1.2540, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.5193 [2021-07-07 11:33:56,729][root][INFO] - ===> Epoch[442](2210/5): Loss 0.0348, Sem 0.0348 LR: 9.668e-02 Score 98.748 Data time: 0.2822, Forward time: 1.2541, Backward time: 1.2351, DDP time: 0.0042, Total iter time: 2.5600 [2021-07-07 11:34:11,317][root][INFO] - ===> Epoch[443](2215/5): Loss 0.0342, Sem 0.0342 LR: 9.667e-02 Score 98.779 Data time: 0.2213, Forward time: 1.2539, Backward time: 1.2355, DDP time: 0.0042, Total iter time: 2.8670 [2021-07-07 11:34:24,945][root][INFO] - ===> Epoch[444](2220/5): Loss 0.0331, Sem 0.0331 LR: 9.666e-02 Score 98.809 Data time: 0.2290, Forward time: 1.2539, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.6804 [2021-07-07 11:34:39,756][root][INFO] - ===> Epoch[445](2225/5): Loss 0.0324, Sem 0.0324 LR: 9.666e-02 Score 98.845 Data time: 0.2720, Forward time: 1.2541, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.9089 [2021-07-07 11:34:53,565][root][INFO] - ===> Epoch[446](2230/5): Loss 0.0332, Sem 0.0332 LR: 9.665e-02 Score 98.808 Data time: 0.2165, Forward time: 1.2543, Backward time: 1.2352, DDP time: 0.0042, Total iter time: 2.7254 [2021-07-07 11:35:07,800][root][INFO] - ===> Epoch[447](2235/5): Loss 0.0341, Sem 0.0341 LR: 9.664e-02 Score 98.793 Data time: 0.1613, Forward time: 1.2541, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.8030 [2021-07-07 11:35:22,104][root][INFO] - ===> Epoch[448](2240/5): Loss 0.0351, Sem 0.0351 LR: 9.663e-02 Score 98.769 Data time: 0.2280, Forward time: 1.2539, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.8092 [2021-07-07 11:35:35,353][root][INFO] - ===> Epoch[449](2245/5): Loss 0.0331, Sem 0.0331 LR: 9.663e-02 Score 98.816 Data time: 0.2332, Forward time: 1.2539, Backward time: 1.2355, DDP time: 0.0042, Total iter time: 2.6027 [2021-07-07 11:35:49,204][root][INFO] - ===> Epoch[450](2250/5): Loss 0.0354, Sem 0.0354 LR: 9.662e-02 Score 98.736 Data time: 0.2408, Forward time: 1.2540, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.7010 [2021-07-07 11:36:03,599][root][INFO] - ===> Epoch[451](2255/5): Loss 0.0328, Sem 0.0328 LR: 9.661e-02 Score 98.817 Data time: 0.2373, Forward time: 1.2541, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.8185 [2021-07-07 11:36:16,645][root][INFO] - ===> Epoch[452](2260/5): Loss 0.0352, Sem 0.0352 LR: 9.660e-02 Score 98.758 Data time: 0.2696, Forward time: 1.2540, Backward time: 1.2349, DDP time: 0.0042, Total iter time: 2.5637 [2021-07-07 11:36:31,494][root][INFO] - ===> Epoch[453](2265/5): Loss 0.0351, Sem 0.0351 LR: 9.660e-02 Score 98.747 Data time: 0.2763, Forward time: 1.2541, Backward time: 1.2351, DDP time: 0.0042, Total iter time: 2.9264 [2021-07-07 11:36:45,947][root][INFO] - ===> Epoch[454](2270/5): Loss 0.0337, Sem 0.0337 LR: 9.659e-02 Score 98.805 Data time: 0.2405, Forward time: 1.2540, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.8369 [2021-07-07 11:36:59,992][root][INFO] - ===> Epoch[455](2275/5): Loss 0.0356, Sem 0.0356 LR: 9.658e-02 Score 98.720 Data time: 0.2476, Forward time: 1.2540, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.7400 [2021-07-07 11:37:13,920][root][INFO] - ===> Epoch[456](2280/5): Loss 0.0345, Sem 0.0345 LR: 9.657e-02 Score 98.778 Data time: 0.2263, Forward time: 1.2539, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.7343 [2021-07-07 11:37:28,070][root][INFO] - ===> Epoch[457](2285/5): Loss 0.0331, Sem 0.0331 LR: 9.657e-02 Score 98.808 Data time: 0.2809, Forward time: 1.2541, Backward time: 1.2350, DDP time: 0.0042, Total iter time: 2.7783 [2021-07-07 11:37:42,035][root][INFO] - ===> Epoch[458](2290/5): Loss 0.0335, Sem 0.0335 LR: 9.656e-02 Score 98.798 Data time: 0.2807, Forward time: 1.2542, Backward time: 1.2349, DDP time: 0.0042, Total iter time: 2.7519 [2021-07-07 11:37:55,697][root][INFO] - ===> Epoch[459](2295/5): Loss 0.0357, Sem 0.0357 LR: 9.655e-02 Score 98.731 Data time: 0.1953, Forward time: 1.2543, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.6906 [2021-07-07 11:38:09,449][root][INFO] - ===> Epoch[460](2300/5): Loss 0.0342, Sem 0.0342 LR: 9.654e-02 Score 98.779 Data time: 0.1864, Forward time: 1.2542, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 2.6998 [2021-07-07 11:38:23,379][root][INFO] - ===> Epoch[461](2305/5): Loss 0.0315, Sem 0.0315 LR: 9.654e-02 Score 98.882 Data time: 0.2024, Forward time: 1.2540, Backward time: 1.2350, DDP time: 0.0042, Total iter time: 2.7465 [2021-07-07 11:38:36,701][root][INFO] - ===> Epoch[462](2310/5): Loss 0.0343, Sem 0.0343 LR: 9.653e-02 Score 98.776 Data time: 0.2368, Forward time: 1.2540, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.6150 [2021-07-07 11:38:50,852][root][INFO] - ===> Epoch[463](2315/5): Loss 0.0323, Sem 0.0323 LR: 9.652e-02 Score 98.837 Data time: 0.1978, Forward time: 1.2541, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7782 [2021-07-07 11:39:04,668][root][INFO] - ===> Epoch[464](2320/5): Loss 0.0322, Sem 0.0322 LR: 9.651e-02 Score 98.854 Data time: 0.2334, Forward time: 1.2539, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 2.7163 [2021-07-07 11:39:18,688][root][INFO] - ===> Epoch[465](2325/5): Loss 0.0329, Sem 0.0329 LR: 9.651e-02 Score 98.823 Data time: 0.2386, Forward time: 1.2540, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7525 [2021-07-07 11:39:32,936][root][INFO] - ===> Epoch[466](2330/5): Loss 0.0310, Sem 0.0310 LR: 9.650e-02 Score 98.894 Data time: 0.1730, Forward time: 1.2543, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7985 [2021-07-07 11:39:46,965][root][INFO] - ===> Epoch[467](2335/5): Loss 0.0322, Sem 0.0322 LR: 9.649e-02 Score 98.847 Data time: 0.2510, Forward time: 1.2543, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7582 [2021-07-07 11:40:00,712][root][INFO] - ===> Epoch[468](2340/5): Loss 0.0308, Sem 0.0308 LR: 9.648e-02 Score 98.898 Data time: 0.2109, Forward time: 1.2545, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.6961 [2021-07-07 11:40:15,137][root][INFO] - ===> Epoch[469](2345/5): Loss 0.0314, Sem 0.0314 LR: 9.648e-02 Score 98.869 Data time: 0.2535, Forward time: 1.2548, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.8310 [2021-07-07 11:40:30,043][root][INFO] - ===> Epoch[470](2350/5): Loss 0.0306, Sem 0.0306 LR: 9.647e-02 Score 98.906 Data time: 0.2320, Forward time: 1.2545, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 2.9349 [2021-07-07 11:40:44,298][root][INFO] - ===> Epoch[471](2355/5): Loss 0.0327, Sem 0.0327 LR: 9.646e-02 Score 98.826 Data time: 0.1825, Forward time: 1.2544, Backward time: 1.2351, DDP time: 0.0042, Total iter time: 2.8094 [2021-07-07 11:40:58,486][root][INFO] - ===> Epoch[472](2360/5): Loss 0.0303, Sem 0.0303 LR: 9.645e-02 Score 98.911 Data time: 0.2298, Forward time: 1.2546, Backward time: 1.2350, DDP time: 0.0042, Total iter time: 2.7729 [2021-07-07 11:41:12,993][root][INFO] - ===> Epoch[473](2365/5): Loss 0.0310, Sem 0.0310 LR: 9.645e-02 Score 98.881 Data time: 0.2160, Forward time: 1.2546, Backward time: 1.2352, DDP time: 0.0042, Total iter time: 2.8525 [2021-07-07 11:41:27,072][root][INFO] - ===> Epoch[474](2370/5): Loss 0.0314, Sem 0.0314 LR: 9.644e-02 Score 98.876 Data time: 0.1920, Forward time: 1.2546, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.7750 [2021-07-07 11:41:41,558][root][INFO] - ===> Epoch[475](2375/5): Loss 0.0305, Sem 0.0305 LR: 9.643e-02 Score 98.908 Data time: 0.2684, Forward time: 1.2547, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.8466 [2021-07-07 11:41:55,217][root][INFO] - ===> Epoch[476](2380/5): Loss 0.0305, Sem 0.0305 LR: 9.642e-02 Score 98.905 Data time: 0.1734, Forward time: 1.2549, Backward time: 1.2352, DDP time: 0.0042, Total iter time: 2.6897 [2021-07-07 11:42:10,067][root][INFO] - ===> Epoch[477](2385/5): Loss 0.0308, Sem 0.0308 LR: 9.642e-02 Score 98.917 Data time: 0.2713, Forward time: 1.2553, Backward time: 1.2350, DDP time: 0.0042, Total iter time: 2.9222 [2021-07-07 11:42:24,274][root][INFO] - ===> Epoch[478](2390/5): Loss 0.0323, Sem 0.0323 LR: 9.641e-02 Score 98.840 Data time: 0.1603, Forward time: 1.2549, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.8096 [2021-07-07 11:42:38,225][root][INFO] - ===> Epoch[479](2395/5): Loss 0.0318, Sem 0.0318 LR: 9.640e-02 Score 98.856 Data time: 0.2160, Forward time: 1.2547, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.7376 [2021-07-07 11:42:52,048][root][INFO] - ===> Epoch[480](2400/5): Loss 0.0326, Sem 0.0326 LR: 9.639e-02 Score 98.838 Data time: 0.2379, Forward time: 1.2546, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.7127 [2021-07-07 11:42:52,050][root][INFO] - ===> Start testing [2021-07-07 11:43:26,268][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1399 Loss 0.298 (AVG: 0.477) Score 91.152 (AVG: 91.025) mIOU 69.695 mAP 84.855 mAcc 76.696 IOU: 64.390 0.119 76.182 77.542 94.594 89.661 26.373 80.356 98.609 75.589 78.723 86.801 57.093 mAP: 83.904 11.422 93.497 90.657 98.872 98.708 60.569 97.423 99.839 86.366 93.607 95.604 92.644 mAcc: 79.128 0.638 91.254 86.100 95.978 98.098 27.561 94.732 99.428 78.177 87.454 96.437 62.059 [2021-07-07 11:43:26,269][root][INFO] - Finished test. Elapsed time: 34.2178 [2021-07-07 11:43:27,369][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 11:43:39,579][root][INFO] - ===> Epoch[481](2405/5): Loss 0.0339, Sem 0.0339 LR: 9.639e-02 Score 98.791 Data time: 0.2421, Forward time: 1.2543, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.4040 [2021-07-07 11:43:53,364][root][INFO] - ===> Epoch[482](2410/5): Loss 0.0337, Sem 0.0337 LR: 9.638e-02 Score 98.789 Data time: 0.2431, Forward time: 1.2543, Backward time: 1.2352, DDP time: 0.0042, Total iter time: 2.7086 [2021-07-07 11:44:07,260][root][INFO] - ===> Epoch[483](2415/5): Loss 0.0327, Sem 0.0327 LR: 9.637e-02 Score 98.813 Data time: 0.2507, Forward time: 1.2543, Backward time: 1.2351, DDP time: 0.0042, Total iter time: 2.7236 [2021-07-07 11:44:20,665][root][INFO] - ===> Epoch[484](2420/5): Loss 0.0350, Sem 0.0350 LR: 9.636e-02 Score 98.759 Data time: 0.2286, Forward time: 1.2544, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 2.6291 [2021-07-07 11:44:35,851][root][INFO] - ===> Epoch[485](2425/5): Loss 0.0334, Sem 0.0334 LR: 9.636e-02 Score 98.797 Data time: 0.2260, Forward time: 1.2546, Backward time: 1.2351, DDP time: 0.0042, Total iter time: 2.9799 [2021-07-07 11:44:49,656][root][INFO] - ===> Epoch[486](2430/5): Loss 0.0326, Sem 0.0326 LR: 9.635e-02 Score 98.839 Data time: 0.2625, Forward time: 1.2547, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 2.7142 [2021-07-07 11:45:02,437][root][INFO] - ===> Epoch[487](2435/5): Loss 0.0342, Sem 0.0342 LR: 9.634e-02 Score 98.758 Data time: 0.3428, Forward time: 1.2546, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.5013 [2021-07-07 11:45:16,269][root][INFO] - ===> Epoch[488](2440/5): Loss 0.0320, Sem 0.0320 LR: 9.633e-02 Score 98.853 Data time: 0.2540, Forward time: 1.2547, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.7197 [2021-07-07 11:45:30,484][root][INFO] - ===> Epoch[489](2445/5): Loss 0.0331, Sem 0.0331 LR: 9.632e-02 Score 98.838 Data time: 0.1708, Forward time: 1.2550, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7966 [2021-07-07 11:45:44,906][root][INFO] - ===> Epoch[490](2450/5): Loss 0.0324, Sem 0.0324 LR: 9.632e-02 Score 98.851 Data time: 0.2200, Forward time: 1.2548, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.8345 [2021-07-07 11:45:57,858][root][INFO] - ===> Epoch[491](2455/5): Loss 0.0306, Sem 0.0306 LR: 9.631e-02 Score 98.907 Data time: 0.2656, Forward time: 1.2546, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.5324 [2021-07-07 11:46:11,351][root][INFO] - ===> Epoch[492](2460/5): Loss 0.0316, Sem 0.0316 LR: 9.630e-02 Score 98.867 Data time: 0.2712, Forward time: 1.2545, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6445 [2021-07-07 11:46:25,125][root][INFO] - ===> Epoch[493](2465/5): Loss 0.0331, Sem 0.0331 LR: 9.629e-02 Score 98.815 Data time: 0.2377, Forward time: 1.2545, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7110 [2021-07-07 11:46:39,937][root][INFO] - ===> Epoch[494](2470/5): Loss 0.0309, Sem 0.0309 LR: 9.629e-02 Score 98.905 Data time: 0.2354, Forward time: 1.2549, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.9052 [2021-07-07 11:46:54,098][root][INFO] - ===> Epoch[495](2475/5): Loss 0.0314, Sem 0.0314 LR: 9.628e-02 Score 98.867 Data time: 0.2262, Forward time: 1.2550, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7875 [2021-07-07 11:47:07,656][root][INFO] - ===> Epoch[496](2480/5): Loss 0.0324, Sem 0.0324 LR: 9.627e-02 Score 98.849 Data time: 0.1752, Forward time: 1.2548, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6728 [2021-07-07 11:47:21,882][root][INFO] - ===> Epoch[497](2485/5): Loss 0.0307, Sem 0.0307 LR: 9.626e-02 Score 98.910 Data time: 0.1668, Forward time: 1.2545, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.8076 [2021-07-07 11:47:36,218][root][INFO] - ===> Epoch[498](2490/5): Loss 0.0318, Sem 0.0318 LR: 9.626e-02 Score 98.860 Data time: 0.2256, Forward time: 1.2544, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.8155 [2021-07-07 11:47:50,877][root][INFO] - ===> Epoch[499](2495/5): Loss 0.0305, Sem 0.0305 LR: 9.625e-02 Score 98.899 Data time: 0.2739, Forward time: 1.2546, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.8827 [2021-07-07 11:48:04,168][root][INFO] - ===> Epoch[500](2500/5): Loss 0.0301, Sem 0.0301 LR: 9.624e-02 Score 98.921 Data time: 0.2090, Forward time: 1.2546, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.6057 [2021-07-07 11:48:17,554][root][INFO] - ===> Epoch[501](2505/5): Loss 0.0304, Sem 0.0304 LR: 9.623e-02 Score 98.907 Data time: 0.2312, Forward time: 1.2546, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.6270 [2021-07-07 11:48:31,658][root][INFO] - ===> Epoch[502](2510/5): Loss 0.0317, Sem 0.0317 LR: 9.623e-02 Score 98.863 Data time: 0.2522, Forward time: 1.2548, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7698 [2021-07-07 11:48:45,323][root][INFO] - ===> Epoch[503](2515/5): Loss 0.0305, Sem 0.0305 LR: 9.622e-02 Score 98.905 Data time: 0.2249, Forward time: 1.2549, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.6890 [2021-07-07 11:48:59,118][root][INFO] - ===> Epoch[504](2520/5): Loss 0.0316, Sem 0.0316 LR: 9.621e-02 Score 98.862 Data time: 0.2243, Forward time: 1.2548, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7116 [2021-07-07 11:49:12,801][root][INFO] - ===> Epoch[505](2525/5): Loss 0.0290, Sem 0.0290 LR: 9.620e-02 Score 98.962 Data time: 0.2505, Forward time: 1.2547, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6847 [2021-07-07 11:49:27,097][root][INFO] - ===> Epoch[506](2530/5): Loss 0.0307, Sem 0.0307 LR: 9.620e-02 Score 98.896 Data time: 0.2293, Forward time: 1.2547, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8165 [2021-07-07 11:49:40,745][root][INFO] - ===> Epoch[507](2535/5): Loss 0.0313, Sem 0.0313 LR: 9.619e-02 Score 98.881 Data time: 0.1873, Forward time: 1.2547, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6836 [2021-07-07 11:49:54,873][root][INFO] - ===> Epoch[508](2540/5): Loss 0.0309, Sem 0.0309 LR: 9.618e-02 Score 98.878 Data time: 0.2655, Forward time: 1.2545, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7762 [2021-07-07 11:50:08,826][root][INFO] - ===> Epoch[509](2545/5): Loss 0.0302, Sem 0.0302 LR: 9.617e-02 Score 98.936 Data time: 0.2685, Forward time: 1.2546, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7329 [2021-07-07 11:50:22,592][root][INFO] - ===> Epoch[510](2550/5): Loss 0.0289, Sem 0.0289 LR: 9.617e-02 Score 98.960 Data time: 0.1895, Forward time: 1.2546, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7091 [2021-07-07 11:50:37,258][root][INFO] - ===> Epoch[511](2555/5): Loss 0.0304, Sem 0.0304 LR: 9.616e-02 Score 98.927 Data time: 0.2157, Forward time: 1.2547, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8814 [2021-07-07 11:50:50,927][root][INFO] - ===> Epoch[512](2560/5): Loss 0.0321, Sem 0.0321 LR: 9.615e-02 Score 98.868 Data time: 0.2179, Forward time: 1.2550, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.6865 [2021-07-07 11:51:04,981][root][INFO] - ===> Epoch[513](2565/5): Loss 0.0311, Sem 0.0311 LR: 9.614e-02 Score 98.881 Data time: 0.2549, Forward time: 1.2553, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7494 [2021-07-07 11:51:18,856][root][INFO] - ===> Epoch[514](2570/5): Loss 0.0320, Sem 0.0320 LR: 9.614e-02 Score 98.855 Data time: 0.2212, Forward time: 1.2555, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7300 [2021-07-07 11:51:32,298][root][INFO] - ===> Epoch[515](2575/5): Loss 0.0313, Sem 0.0313 LR: 9.613e-02 Score 98.885 Data time: 0.2353, Forward time: 1.2556, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6422 [2021-07-07 11:51:47,076][root][INFO] - ===> Epoch[516](2580/5): Loss 0.0293, Sem 0.0293 LR: 9.612e-02 Score 98.950 Data time: 0.1656, Forward time: 1.2556, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.9139 [2021-07-07 11:52:01,419][root][INFO] - ===> Epoch[517](2585/5): Loss 0.0338, Sem 0.0338 LR: 9.611e-02 Score 98.822 Data time: 0.2364, Forward time: 1.2557, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8104 [2021-07-07 11:52:15,134][root][INFO] - ===> Epoch[518](2590/5): Loss 0.0307, Sem 0.0307 LR: 9.611e-02 Score 98.889 Data time: 0.1936, Forward time: 1.2556, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.6929 [2021-07-07 11:52:28,849][root][INFO] - ===> Epoch[519](2595/5): Loss 0.0309, Sem 0.0309 LR: 9.610e-02 Score 98.895 Data time: 0.1885, Forward time: 1.2556, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.6869 [2021-07-07 11:52:43,231][root][INFO] - ===> Epoch[520](2600/5): Loss 0.0317, Sem 0.0317 LR: 9.609e-02 Score 98.864 Data time: 0.2218, Forward time: 1.2558, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8346 [2021-07-07 11:52:43,232][root][INFO] - ===> Start testing [2021-07-07 11:53:18,758][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1223 Loss 0.574 (AVG: 0.471) Score 86.770 (AVG: 91.026) mIOU 70.913 mAP 84.637 mAcc 76.906 IOU: 64.465 0.000 79.611 77.593 94.062 90.517 31.304 84.994 98.671 79.596 78.638 86.200 56.221 mAP: 84.988 6.770 91.946 90.941 98.649 98.309 61.288 97.378 99.845 88.535 92.785 95.648 93.201 mAcc: 81.619 0.000 88.568 84.620 95.373 96.288 33.417 93.851 99.461 82.478 86.794 96.810 60.502 [2021-07-07 11:53:18,759][root][INFO] - Finished test. Elapsed time: 35.5263 [2021-07-07 11:53:19,842][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 11:53:32,791][root][INFO] - ===> Epoch[521](2605/5): Loss 0.0306, Sem 0.0306 LR: 9.608e-02 Score 98.903 Data time: 0.1678, Forward time: 1.2556, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.5600 [2021-07-07 11:53:47,611][root][INFO] - ===> Epoch[522](2610/5): Loss 0.0312, Sem 0.0312 LR: 9.608e-02 Score 98.889 Data time: 0.2406, Forward time: 1.2558, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.9077 [2021-07-07 11:54:02,058][root][INFO] - ===> Epoch[523](2615/5): Loss 0.0302, Sem 0.0302 LR: 9.607e-02 Score 98.909 Data time: 0.2672, Forward time: 1.2560, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8360 [2021-07-07 11:54:17,037][root][INFO] - ===> Epoch[524](2620/5): Loss 0.0305, Sem 0.0305 LR: 9.606e-02 Score 98.916 Data time: 0.1889, Forward time: 1.2559, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.9476 [2021-07-07 11:54:31,103][root][INFO] - ===> Epoch[525](2625/5): Loss 0.0322, Sem 0.0322 LR: 9.605e-02 Score 98.843 Data time: 0.2673, Forward time: 1.2558, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7670 [2021-07-07 11:54:45,144][root][INFO] - ===> Epoch[526](2630/5): Loss 0.0314, Sem 0.0314 LR: 9.605e-02 Score 98.884 Data time: 0.2291, Forward time: 1.2559, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7597 [2021-07-07 11:54:58,862][root][INFO] - ===> Epoch[527](2635/5): Loss 0.0299, Sem 0.0299 LR: 9.604e-02 Score 98.929 Data time: 0.2360, Forward time: 1.2562, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6953 [2021-07-07 11:55:12,664][root][INFO] - ===> Epoch[528](2640/5): Loss 0.0298, Sem 0.0298 LR: 9.603e-02 Score 98.922 Data time: 0.2266, Forward time: 1.2563, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7065 [2021-07-07 11:55:26,606][root][INFO] - ===> Epoch[529](2645/5): Loss 0.0314, Sem 0.0314 LR: 9.602e-02 Score 98.875 Data time: 0.2603, Forward time: 1.2563, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7456 [2021-07-07 11:55:40,305][root][INFO] - ===> Epoch[530](2650/5): Loss 0.0304, Sem 0.0304 LR: 9.602e-02 Score 98.907 Data time: 0.2623, Forward time: 1.2561, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6854 [2021-07-07 11:55:54,865][root][INFO] - ===> Epoch[531](2655/5): Loss 0.0309, Sem 0.0309 LR: 9.601e-02 Score 98.893 Data time: 0.2175, Forward time: 1.2562, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8652 [2021-07-07 11:56:09,507][root][INFO] - ===> Epoch[532](2660/5): Loss 0.0316, Sem 0.0316 LR: 9.600e-02 Score 98.901 Data time: 0.2619, Forward time: 1.2561, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8807 [2021-07-07 11:56:23,786][root][INFO] - ===> Epoch[533](2665/5): Loss 0.0355, Sem 0.0355 LR: 9.599e-02 Score 98.780 Data time: 0.2148, Forward time: 1.2559, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.8111 [2021-07-07 11:56:38,020][root][INFO] - ===> Epoch[534](2670/5): Loss 0.0368, Sem 0.0368 LR: 9.599e-02 Score 98.732 Data time: 0.2308, Forward time: 1.2560, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.8086 [2021-07-07 11:56:52,768][root][INFO] - ===> Epoch[535](2675/5): Loss 0.0351, Sem 0.0351 LR: 9.598e-02 Score 98.759 Data time: 0.2409, Forward time: 1.2564, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.8987 [2021-07-07 11:57:06,512][root][INFO] - ===> Epoch[536](2680/5): Loss 0.0380, Sem 0.0380 LR: 9.597e-02 Score 98.638 Data time: 0.1967, Forward time: 1.2564, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7101 [2021-07-07 11:57:20,572][root][INFO] - ===> Epoch[537](2685/5): Loss 0.0349, Sem 0.0349 LR: 9.596e-02 Score 98.763 Data time: 0.2786, Forward time: 1.2564, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7711 [2021-07-07 11:57:34,776][root][INFO] - ===> Epoch[538](2690/5): Loss 0.0346, Sem 0.0346 LR: 9.596e-02 Score 98.765 Data time: 0.2258, Forward time: 1.2564, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7942 [2021-07-07 11:57:48,237][root][INFO] - ===> Epoch[539](2695/5): Loss 0.0346, Sem 0.0346 LR: 9.595e-02 Score 98.766 Data time: 0.2277, Forward time: 1.2564, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6421 [2021-07-07 11:58:01,868][root][INFO] - ===> Epoch[540](2700/5): Loss 0.0346, Sem 0.0346 LR: 9.594e-02 Score 98.779 Data time: 0.2430, Forward time: 1.2564, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6731 [2021-07-07 11:58:16,179][root][INFO] - ===> Epoch[541](2705/5): Loss 0.0328, Sem 0.0328 LR: 9.593e-02 Score 98.831 Data time: 0.2684, Forward time: 1.2564, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8210 [2021-07-07 11:58:30,575][root][INFO] - ===> Epoch[542](2710/5): Loss 0.0319, Sem 0.0319 LR: 9.593e-02 Score 98.864 Data time: 0.2322, Forward time: 1.2565, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8258 [2021-07-07 11:58:44,466][root][INFO] - ===> Epoch[543](2715/5): Loss 0.0361, Sem 0.0361 LR: 9.592e-02 Score 98.755 Data time: 0.1713, Forward time: 1.2565, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7363 [2021-07-07 11:58:57,670][root][INFO] - ===> Epoch[544](2720/5): Loss 0.0391, Sem 0.0391 LR: 9.591e-02 Score 98.622 Data time: 0.2987, Forward time: 1.2564, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.5935 [2021-07-07 11:59:11,973][root][INFO] - ===> Epoch[545](2725/5): Loss 0.0419, Sem 0.0419 LR: 9.590e-02 Score 98.584 Data time: 0.2587, Forward time: 1.2567, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8101 [2021-07-07 11:59:25,906][root][INFO] - ===> Epoch[546](2730/5): Loss 0.0428, Sem 0.0428 LR: 9.590e-02 Score 98.510 Data time: 0.2886, Forward time: 1.2570, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7335 [2021-07-07 11:59:39,810][root][INFO] - ===> Epoch[547](2735/5): Loss 0.0420, Sem 0.0420 LR: 9.589e-02 Score 98.528 Data time: 0.2599, Forward time: 1.2569, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7413 [2021-07-07 11:59:53,399][root][INFO] - ===> Epoch[548](2740/5): Loss 0.0415, Sem 0.0415 LR: 9.588e-02 Score 98.582 Data time: 0.2412, Forward time: 1.2570, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6670 [2021-07-07 12:00:07,633][root][INFO] - ===> Epoch[549](2745/5): Loss 0.0401, Sem 0.0401 LR: 9.587e-02 Score 98.602 Data time: 0.2862, Forward time: 1.2569, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8049 [2021-07-07 12:00:21,611][root][INFO] - ===> Epoch[550](2750/5): Loss 0.0403, Sem 0.0403 LR: 9.587e-02 Score 98.593 Data time: 0.2139, Forward time: 1.2565, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7260 [2021-07-07 12:00:34,998][root][INFO] - ===> Epoch[551](2755/5): Loss 0.0416, Sem 0.0416 LR: 9.586e-02 Score 98.565 Data time: 0.1934, Forward time: 1.2564, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6225 [2021-07-07 12:00:48,616][root][INFO] - ===> Epoch[552](2760/5): Loss 0.0410, Sem 0.0410 LR: 9.585e-02 Score 98.582 Data time: 0.2007, Forward time: 1.2563, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6696 [2021-07-07 12:01:02,952][root][INFO] - ===> Epoch[553](2765/5): Loss 0.0376, Sem 0.0376 LR: 9.584e-02 Score 98.672 Data time: 0.1872, Forward time: 1.2563, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.8156 [2021-07-07 12:01:16,475][root][INFO] - ===> Epoch[554](2770/5): Loss 0.0377, Sem 0.0377 LR: 9.584e-02 Score 98.665 Data time: 0.1833, Forward time: 1.2561, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.6526 [2021-07-07 12:01:30,494][root][INFO] - ===> Epoch[555](2775/5): Loss 0.0352, Sem 0.0352 LR: 9.583e-02 Score 98.746 Data time: 0.1825, Forward time: 1.2561, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7411 [2021-07-07 12:01:44,239][root][INFO] - ===> Epoch[556](2780/5): Loss 0.0343, Sem 0.0343 LR: 9.582e-02 Score 98.783 Data time: 0.2709, Forward time: 1.2563, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6957 [2021-07-07 12:01:57,456][root][INFO] - ===> Epoch[557](2785/5): Loss 0.0322, Sem 0.0322 LR: 9.581e-02 Score 98.838 Data time: 0.2651, Forward time: 1.2561, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.5970 [2021-07-07 12:02:11,701][root][INFO] - ===> Epoch[558](2790/5): Loss 0.0346, Sem 0.0346 LR: 9.581e-02 Score 98.774 Data time: 0.2105, Forward time: 1.2561, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8039 [2021-07-07 12:02:25,380][root][INFO] - ===> Epoch[559](2795/5): Loss 0.0325, Sem 0.0325 LR: 9.580e-02 Score 98.865 Data time: 0.2508, Forward time: 1.2561, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.6823 [2021-07-07 12:02:40,032][root][INFO] - ===> Epoch[560](2800/5): Loss 0.0318, Sem 0.0318 LR: 9.579e-02 Score 98.876 Data time: 0.2547, Forward time: 1.2562, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8748 [2021-07-07 12:02:40,034][root][INFO] - ===> Start testing [2021-07-07 12:03:16,400][root][INFO] - 68/68: Data time: 0.0476, Iter time: 0.1274 Loss 0.245 (AVG: 0.444) Score 92.441 (AVG: 90.944) mIOU 70.886 mAP 85.010 mAcc 77.231 IOU: 64.469 0.112 78.362 77.395 94.303 91.364 37.986 80.569 98.718 77.549 80.039 85.883 54.767 mAP: 85.214 5.842 93.455 90.602 99.010 98.808 68.469 95.966 99.853 87.305 93.627 95.327 91.659 mAcc: 81.205 0.547 90.323 83.863 95.938 97.147 43.371 87.965 99.544 79.676 88.843 96.353 59.230 [2021-07-07 12:03:16,401][root][INFO] - Finished test. Elapsed time: 36.3660 [2021-07-07 12:03:17,539][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 12:03:30,953][root][INFO] - ===> Epoch[561](2805/5): Loss 0.0327, Sem 0.0327 LR: 9.578e-02 Score 98.842 Data time: 0.1755, Forward time: 1.2559, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6476 [2021-07-07 12:03:45,455][root][INFO] - ===> Epoch[562](2810/5): Loss 0.0317, Sem 0.0317 LR: 9.578e-02 Score 98.858 Data time: 0.1803, Forward time: 1.2559, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.8561 [2021-07-07 12:03:59,974][root][INFO] - ===> Epoch[563](2815/5): Loss 0.0311, Sem 0.0311 LR: 9.577e-02 Score 98.877 Data time: 0.2267, Forward time: 1.2561, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8347 [2021-07-07 12:04:13,731][root][INFO] - ===> Epoch[564](2820/5): Loss 0.0331, Sem 0.0331 LR: 9.576e-02 Score 98.802 Data time: 0.2271, Forward time: 1.2560, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7049 [2021-07-07 12:04:27,439][root][INFO] - ===> Epoch[565](2825/5): Loss 0.0309, Sem 0.0309 LR: 9.575e-02 Score 98.908 Data time: 0.2747, Forward time: 1.2561, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6920 [2021-07-07 12:04:41,467][root][INFO] - ===> Epoch[566](2830/5): Loss 0.0310, Sem 0.0310 LR: 9.574e-02 Score 98.888 Data time: 0.1511, Forward time: 1.2562, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7667 [2021-07-07 12:04:55,079][root][INFO] - ===> Epoch[567](2835/5): Loss 0.0317, Sem 0.0317 LR: 9.574e-02 Score 98.861 Data time: 0.1551, Forward time: 1.2560, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.6924 [2021-07-07 12:05:09,293][root][INFO] - ===> Epoch[568](2840/5): Loss 0.0288, Sem 0.0288 LR: 9.573e-02 Score 98.962 Data time: 0.2311, Forward time: 1.2562, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7984 [2021-07-07 12:05:23,245][root][INFO] - ===> Epoch[569](2845/5): Loss 0.0306, Sem 0.0306 LR: 9.572e-02 Score 98.908 Data time: 0.2578, Forward time: 1.2563, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7454 [2021-07-07 12:05:36,969][root][INFO] - ===> Epoch[570](2850/5): Loss 0.0314, Sem 0.0314 LR: 9.571e-02 Score 98.886 Data time: 0.2515, Forward time: 1.2563, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.6966 [2021-07-07 12:05:50,380][root][INFO] - ===> Epoch[571](2855/5): Loss 0.0315, Sem 0.0315 LR: 9.571e-02 Score 98.864 Data time: 0.1819, Forward time: 1.2562, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6342 [2021-07-07 12:06:03,798][root][INFO] - ===> Epoch[572](2860/5): Loss 0.0315, Sem 0.0315 LR: 9.570e-02 Score 98.872 Data time: 0.2363, Forward time: 1.2562, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6380 [2021-07-07 12:06:18,628][root][INFO] - ===> Epoch[573](2865/5): Loss 0.0315, Sem 0.0315 LR: 9.569e-02 Score 98.881 Data time: 0.2298, Forward time: 1.2563, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.9290 [2021-07-07 12:06:32,560][root][INFO] - ===> Epoch[574](2870/5): Loss 0.0311, Sem 0.0311 LR: 9.568e-02 Score 98.887 Data time: 0.2336, Forward time: 1.2561, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7403 [2021-07-07 12:06:47,501][root][INFO] - ===> Epoch[575](2875/5): Loss 0.0289, Sem 0.0289 LR: 9.568e-02 Score 98.960 Data time: 0.2173, Forward time: 1.2562, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.9503 [2021-07-07 12:07:00,983][root][INFO] - ===> Epoch[576](2880/5): Loss 0.0294, Sem 0.0294 LR: 9.567e-02 Score 98.951 Data time: 0.1793, Forward time: 1.2562, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6494 [2021-07-07 12:07:15,115][root][INFO] - ===> Epoch[577](2885/5): Loss 0.0291, Sem 0.0291 LR: 9.566e-02 Score 98.951 Data time: 0.1927, Forward time: 1.2564, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7767 [2021-07-07 12:07:29,437][root][INFO] - ===> Epoch[578](2890/5): Loss 0.0273, Sem 0.0273 LR: 9.565e-02 Score 99.018 Data time: 0.3158, Forward time: 1.2566, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8052 [2021-07-07 12:07:43,607][root][INFO] - ===> Epoch[579](2895/5): Loss 0.0285, Sem 0.0285 LR: 9.565e-02 Score 98.983 Data time: 0.3193, Forward time: 1.2566, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7811 [2021-07-07 12:07:57,796][root][INFO] - ===> Epoch[580](2900/5): Loss 0.0288, Sem 0.0288 LR: 9.564e-02 Score 98.964 Data time: 0.2902, Forward time: 1.2567, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7915 [2021-07-07 12:08:11,821][root][INFO] - ===> Epoch[581](2905/5): Loss 0.0277, Sem 0.0277 LR: 9.563e-02 Score 99.011 Data time: 0.2244, Forward time: 1.2568, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7489 [2021-07-07 12:08:25,887][root][INFO] - ===> Epoch[582](2910/5): Loss 0.0287, Sem 0.0287 LR: 9.562e-02 Score 98.978 Data time: 0.2492, Forward time: 1.2570, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7597 [2021-07-07 12:08:40,165][root][INFO] - ===> Epoch[583](2915/5): Loss 0.0286, Sem 0.0286 LR: 9.562e-02 Score 98.974 Data time: 0.1936, Forward time: 1.2573, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8125 [2021-07-07 12:08:53,998][root][INFO] - ===> Epoch[584](2920/5): Loss 0.0306, Sem 0.0306 LR: 9.561e-02 Score 98.909 Data time: 0.2216, Forward time: 1.2572, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7130 [2021-07-07 12:09:08,052][root][INFO] - ===> Epoch[585](2925/5): Loss 0.0284, Sem 0.0284 LR: 9.560e-02 Score 98.989 Data time: 0.2072, Forward time: 1.2572, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7659 [2021-07-07 12:09:21,927][root][INFO] - ===> Epoch[586](2930/5): Loss 0.0285, Sem 0.0285 LR: 9.559e-02 Score 98.979 Data time: 0.3211, Forward time: 1.2574, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.7286 [2021-07-07 12:09:36,827][root][INFO] - ===> Epoch[587](2935/5): Loss 0.0275, Sem 0.0275 LR: 9.559e-02 Score 99.010 Data time: 0.2630, Forward time: 1.2576, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.9323 [2021-07-07 12:09:50,527][root][INFO] - ===> Epoch[588](2940/5): Loss 0.0284, Sem 0.0284 LR: 9.558e-02 Score 98.978 Data time: 0.2571, Forward time: 1.2576, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.6904 [2021-07-07 12:10:04,748][root][INFO] - ===> Epoch[589](2945/5): Loss 0.0275, Sem 0.0275 LR: 9.557e-02 Score 99.010 Data time: 0.2290, Forward time: 1.2577, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.7873 [2021-07-07 12:10:18,741][root][INFO] - ===> Epoch[590](2950/5): Loss 0.0282, Sem 0.0282 LR: 9.556e-02 Score 98.980 Data time: 0.2127, Forward time: 1.2578, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.7424 [2021-07-07 12:10:32,917][root][INFO] - ===> Epoch[591](2955/5): Loss 0.0293, Sem 0.0293 LR: 9.556e-02 Score 98.954 Data time: 0.2519, Forward time: 1.2580, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.7857 [2021-07-07 12:10:46,307][root][INFO] - ===> Epoch[592](2960/5): Loss 0.0286, Sem 0.0286 LR: 9.555e-02 Score 98.984 Data time: 0.2250, Forward time: 1.2581, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.6399 [2021-07-07 12:11:00,291][root][INFO] - ===> Epoch[593](2965/5): Loss 0.0304, Sem 0.0304 LR: 9.554e-02 Score 98.927 Data time: 0.1849, Forward time: 1.2581, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.7554 [2021-07-07 12:11:13,917][root][INFO] - ===> Epoch[594](2970/5): Loss 0.0295, Sem 0.0295 LR: 9.553e-02 Score 98.944 Data time: 0.1744, Forward time: 1.2582, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.6708 [2021-07-07 12:11:28,489][root][INFO] - ===> Epoch[595](2975/5): Loss 0.0297, Sem 0.0297 LR: 9.553e-02 Score 98.939 Data time: 0.3090, Forward time: 1.2585, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.8650 [2021-07-07 12:11:42,430][root][INFO] - ===> Epoch[596](2980/5): Loss 0.0297, Sem 0.0297 LR: 9.552e-02 Score 98.930 Data time: 0.2211, Forward time: 1.2587, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7359 [2021-07-07 12:11:56,858][root][INFO] - ===> Epoch[597](2985/5): Loss 0.0306, Sem 0.0306 LR: 9.551e-02 Score 98.913 Data time: 0.2338, Forward time: 1.2586, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.8393 [2021-07-07 12:12:11,120][root][INFO] - ===> Epoch[598](2990/5): Loss 0.0302, Sem 0.0302 LR: 9.550e-02 Score 98.918 Data time: 0.2400, Forward time: 1.2586, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.8011 [2021-07-07 12:12:24,876][root][INFO] - ===> Epoch[599](2995/5): Loss 0.0330, Sem 0.0330 LR: 9.550e-02 Score 98.851 Data time: 0.1783, Forward time: 1.2586, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.7011 [2021-07-07 12:12:38,796][root][INFO] - ===> Epoch[600](3000/5): Loss 0.0317, Sem 0.0317 LR: 9.549e-02 Score 98.875 Data time: 0.2534, Forward time: 1.2585, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.7290 [2021-07-07 12:12:38,799][root][INFO] - ===> Start testing [2021-07-07 12:13:12,901][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1287 Loss 0.261 (AVG: 0.448) Score 92.550 (AVG: 91.193) mIOU 71.280 mAP 85.131 mAcc 77.565 IOU: 64.345 0.104 80.044 77.139 94.549 91.374 38.277 85.055 98.703 75.601 79.462 86.476 55.507 mAP: 85.206 6.429 95.233 90.147 99.059 98.577 66.585 97.944 99.843 86.569 94.004 95.633 91.480 mAcc: 80.075 0.456 91.381 83.667 96.473 97.221 40.673 96.090 99.468 77.309 88.512 96.517 60.499 [2021-07-07 12:13:12,901][root][INFO] - Finished test. Elapsed time: 34.1013 [2021-07-07 12:13:14,010][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2021-07-07 12:13:14,011][root][INFO] - Current best mIoU: 71.280 at iter 1 [2021-07-07 12:13:14,983][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 12:13:27,193][root][INFO] - ===> Epoch[601](3005/5): Loss 0.0319, Sem 0.0319 LR: 9.548e-02 Score 98.863 Data time: 0.2176, Forward time: 1.2582, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.4027 [2021-07-07 12:13:41,526][root][INFO] - ===> Epoch[602](3010/5): Loss 0.0311, Sem 0.0311 LR: 9.547e-02 Score 98.911 Data time: 0.2527, Forward time: 1.2585, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.8233 [2021-07-07 12:13:55,072][root][INFO] - ===> Epoch[603](3015/5): Loss 0.0309, Sem 0.0309 LR: 9.547e-02 Score 98.887 Data time: 0.1799, Forward time: 1.2585, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.6806 [2021-07-07 12:14:09,796][root][INFO] - ===> Epoch[604](3020/5): Loss 0.0321, Sem 0.0321 LR: 9.546e-02 Score 98.850 Data time: 0.2365, Forward time: 1.2586, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.8943 [2021-07-07 12:14:24,316][root][INFO] - ===> Epoch[605](3025/5): Loss 0.0310, Sem 0.0310 LR: 9.545e-02 Score 98.892 Data time: 0.2373, Forward time: 1.2588, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.8611 [2021-07-07 12:14:37,628][root][INFO] - ===> Epoch[606](3030/5): Loss 0.0303, Sem 0.0303 LR: 9.544e-02 Score 98.907 Data time: 0.2485, Forward time: 1.2589, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.6121 [2021-07-07 12:14:52,275][root][INFO] - ===> Epoch[607](3035/5): Loss 0.0311, Sem 0.0311 LR: 9.544e-02 Score 98.878 Data time: 0.2404, Forward time: 1.2590, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.8901 [2021-07-07 12:15:06,145][root][INFO] - ===> Epoch[608](3040/5): Loss 0.0306, Sem 0.0306 LR: 9.543e-02 Score 98.913 Data time: 0.2197, Forward time: 1.2590, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7187 [2021-07-07 12:15:20,952][root][INFO] - ===> Epoch[609](3045/5): Loss 0.0291, Sem 0.0291 LR: 9.542e-02 Score 98.956 Data time: 0.2081, Forward time: 1.2590, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.9255 [2021-07-07 12:15:34,744][root][INFO] - ===> Epoch[610](3050/5): Loss 0.0299, Sem 0.0299 LR: 9.541e-02 Score 98.928 Data time: 0.2924, Forward time: 1.2589, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7041 [2021-07-07 12:15:49,637][root][INFO] - ===> Epoch[611](3055/5): Loss 0.0290, Sem 0.0290 LR: 9.541e-02 Score 98.963 Data time: 0.2524, Forward time: 1.2590, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.9126 [2021-07-07 12:16:03,625][root][INFO] - ===> Epoch[612](3060/5): Loss 0.0296, Sem 0.0296 LR: 9.540e-02 Score 98.954 Data time: 0.2714, Forward time: 1.2591, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7550 [2021-07-07 12:16:18,064][root][INFO] - ===> Epoch[613](3065/5): Loss 0.0305, Sem 0.0305 LR: 9.539e-02 Score 98.915 Data time: 0.2329, Forward time: 1.2590, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8391 [2021-07-07 12:16:32,134][root][INFO] - ===> Epoch[614](3070/5): Loss 0.0309, Sem 0.0309 LR: 9.538e-02 Score 98.901 Data time: 0.2563, Forward time: 1.2592, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7618 [2021-07-07 12:16:46,459][root][INFO] - ===> Epoch[615](3075/5): Loss 0.0318, Sem 0.0318 LR: 9.538e-02 Score 98.853 Data time: 0.1554, Forward time: 1.2592, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8152 [2021-07-07 12:17:00,388][root][INFO] - ===> Epoch[616](3080/5): Loss 0.0292, Sem 0.0292 LR: 9.537e-02 Score 98.959 Data time: 0.2226, Forward time: 1.2593, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7425 [2021-07-07 12:17:14,036][root][INFO] - ===> Epoch[617](3085/5): Loss 0.0311, Sem 0.0311 LR: 9.536e-02 Score 98.903 Data time: 0.1913, Forward time: 1.2591, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6858 [2021-07-07 12:17:27,598][root][INFO] - ===> Epoch[618](3090/5): Loss 0.0306, Sem 0.0306 LR: 9.535e-02 Score 98.903 Data time: 0.2076, Forward time: 1.2591, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6700 [2021-07-07 12:17:42,394][root][INFO] - ===> Epoch[619](3095/5): Loss 0.0309, Sem 0.0309 LR: 9.535e-02 Score 98.895 Data time: 0.2131, Forward time: 1.2591, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.9163 [2021-07-07 12:17:56,912][root][INFO] - ===> Epoch[620](3100/5): Loss 0.0295, Sem 0.0295 LR: 9.534e-02 Score 98.936 Data time: 0.2401, Forward time: 1.2588, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.8560 [2021-07-07 12:18:11,745][root][INFO] - ===> Epoch[621](3105/5): Loss 0.0281, Sem 0.0281 LR: 9.533e-02 Score 98.991 Data time: 0.2388, Forward time: 1.2588, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.9158 [2021-07-07 12:18:24,829][root][INFO] - ===> Epoch[622](3110/5): Loss 0.0285, Sem 0.0285 LR: 9.532e-02 Score 98.986 Data time: 0.2209, Forward time: 1.2587, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.5691 [2021-07-07 12:18:38,642][root][INFO] - ===> Epoch[623](3115/5): Loss 0.0286, Sem 0.0286 LR: 9.532e-02 Score 98.984 Data time: 0.2230, Forward time: 1.2587, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.7082 [2021-07-07 12:18:52,575][root][INFO] - ===> Epoch[624](3120/5): Loss 0.0276, Sem 0.0276 LR: 9.531e-02 Score 99.012 Data time: 0.1923, Forward time: 1.2586, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.7451 [2021-07-07 12:19:06,674][root][INFO] - ===> Epoch[625](3125/5): Loss 0.0287, Sem 0.0287 LR: 9.530e-02 Score 98.963 Data time: 0.2620, Forward time: 1.2586, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.7683 [2021-07-07 12:19:20,634][root][INFO] - ===> Epoch[626](3130/5): Loss 0.0286, Sem 0.0286 LR: 9.529e-02 Score 98.986 Data time: 0.2107, Forward time: 1.2584, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.7549 [2021-07-07 12:19:35,101][root][INFO] - ===> Epoch[627](3135/5): Loss 0.0278, Sem 0.0278 LR: 9.529e-02 Score 99.003 Data time: 0.2881, Forward time: 1.2584, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.8517 [2021-07-07 12:19:48,626][root][INFO] - ===> Epoch[628](3140/5): Loss 0.0278, Sem 0.0278 LR: 9.528e-02 Score 99.002 Data time: 0.2132, Forward time: 1.2583, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.6667 [2021-07-07 12:20:02,319][root][INFO] - ===> Epoch[629](3145/5): Loss 0.0281, Sem 0.0281 LR: 9.527e-02 Score 98.986 Data time: 0.2332, Forward time: 1.2584, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.6977 [2021-07-07 12:20:16,174][root][INFO] - ===> Epoch[630](3150/5): Loss 0.0269, Sem 0.0269 LR: 9.526e-02 Score 99.030 Data time: 0.3134, Forward time: 1.2585, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.7167 [2021-07-07 12:20:30,297][root][INFO] - ===> Epoch[631](3155/5): Loss 0.0293, Sem 0.0293 LR: 9.525e-02 Score 98.962 Data time: 0.1926, Forward time: 1.2583, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.7944 [2021-07-07 12:20:44,088][root][INFO] - ===> Epoch[632](3160/5): Loss 0.0265, Sem 0.0265 LR: 9.525e-02 Score 99.047 Data time: 0.2291, Forward time: 1.2584, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.7138 [2021-07-07 12:20:58,181][root][INFO] - ===> Epoch[633](3165/5): Loss 0.0284, Sem 0.0284 LR: 9.524e-02 Score 98.976 Data time: 0.2537, Forward time: 1.2584, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.7770 [2021-07-07 12:21:12,465][root][INFO] - ===> Epoch[634](3170/5): Loss 0.0270, Sem 0.0270 LR: 9.523e-02 Score 99.042 Data time: 0.1838, Forward time: 1.2584, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.8166 [2021-07-07 12:21:26,976][root][INFO] - ===> Epoch[635](3175/5): Loss 0.0283, Sem 0.0283 LR: 9.522e-02 Score 98.986 Data time: 0.2448, Forward time: 1.2585, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.8432 [2021-07-07 12:21:41,583][root][INFO] - ===> Epoch[636](3180/5): Loss 0.0281, Sem 0.0281 LR: 9.522e-02 Score 99.004 Data time: 0.2188, Forward time: 1.2584, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.8835 [2021-07-07 12:21:55,830][root][INFO] - ===> Epoch[637](3185/5): Loss 0.0293, Sem 0.0293 LR: 9.521e-02 Score 98.943 Data time: 0.1568, Forward time: 1.2582, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8208 [2021-07-07 12:22:09,217][root][INFO] - ===> Epoch[638](3190/5): Loss 0.0300, Sem 0.0300 LR: 9.520e-02 Score 98.918 Data time: 0.2340, Forward time: 1.2583, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.6305 [2021-07-07 12:22:22,993][root][INFO] - ===> Epoch[639](3195/5): Loss 0.0305, Sem 0.0305 LR: 9.519e-02 Score 98.913 Data time: 0.2169, Forward time: 1.2583, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.7113 [2021-07-07 12:22:36,485][root][INFO] - ===> Epoch[640](3200/5): Loss 0.0292, Sem 0.0292 LR: 9.519e-02 Score 98.955 Data time: 0.1780, Forward time: 1.2583, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.6564 [2021-07-07 12:22:36,486][root][INFO] - ===> Start testing [2021-07-07 12:23:11,407][root][INFO] - 68/68: Data time: 0.0350, Iter time: 0.1227 Loss 0.243 (AVG: 0.475) Score 93.424 (AVG: 91.132) mIOU 70.297 mAP 84.845 mAcc 76.534 IOU: 64.049 0.018 77.362 77.694 94.363 91.081 34.391 83.234 98.654 69.615 79.958 86.554 56.889 mAP: 84.793 6.315 94.153 91.317 98.965 98.400 64.211 97.380 99.831 85.988 93.808 95.130 92.687 mAcc: 79.530 0.091 89.888 84.350 96.142 96.269 36.272 92.702 99.391 71.971 89.905 97.025 61.412 [2021-07-07 12:23:11,407][root][INFO] - Finished test. Elapsed time: 34.9206 [2021-07-07 12:23:12,523][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 12:23:24,832][root][INFO] - ===> Epoch[641](3205/5): Loss 0.0287, Sem 0.0287 LR: 9.518e-02 Score 98.960 Data time: 0.2161, Forward time: 1.2580, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.4208 [2021-07-07 12:23:38,899][root][INFO] - ===> Epoch[642](3210/5): Loss 0.0287, Sem 0.0287 LR: 9.517e-02 Score 98.963 Data time: 0.3140, Forward time: 1.2582, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.7600 [2021-07-07 12:23:53,363][root][INFO] - ===> Epoch[643](3215/5): Loss 0.0290, Sem 0.0290 LR: 9.516e-02 Score 98.952 Data time: 0.1918, Forward time: 1.2582, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.8379 [2021-07-07 12:24:07,844][root][INFO] - ===> Epoch[644](3220/5): Loss 0.0276, Sem 0.0276 LR: 9.516e-02 Score 99.005 Data time: 0.2150, Forward time: 1.2581, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.8558 [2021-07-07 12:24:21,641][root][INFO] - ===> Epoch[645](3225/5): Loss 0.0309, Sem 0.0309 LR: 9.515e-02 Score 98.923 Data time: 0.3016, Forward time: 1.2580, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.7115 [2021-07-07 12:24:35,438][root][INFO] - ===> Epoch[646](3230/5): Loss 0.0310, Sem 0.0310 LR: 9.514e-02 Score 98.915 Data time: 0.2181, Forward time: 1.2580, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.7176 [2021-07-07 12:24:49,541][root][INFO] - ===> Epoch[647](3235/5): Loss 0.0318, Sem 0.0318 LR: 9.513e-02 Score 98.860 Data time: 0.2510, Forward time: 1.2581, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.7615 [2021-07-07 12:25:03,797][root][INFO] - ===> Epoch[648](3240/5): Loss 0.0316, Sem 0.0316 LR: 9.513e-02 Score 98.874 Data time: 0.2020, Forward time: 1.2580, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.8005 [2021-07-07 12:25:17,926][root][INFO] - ===> Epoch[649](3245/5): Loss 0.0304, Sem 0.0304 LR: 9.512e-02 Score 98.910 Data time: 0.2517, Forward time: 1.2580, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.7751 [2021-07-07 12:25:31,300][root][INFO] - ===> Epoch[650](3250/5): Loss 0.0304, Sem 0.0304 LR: 9.511e-02 Score 98.909 Data time: 0.2503, Forward time: 1.2579, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.6189 [2021-07-07 12:25:45,193][root][INFO] - ===> Epoch[651](3255/5): Loss 0.0293, Sem 0.0293 LR: 9.510e-02 Score 98.947 Data time: 0.2280, Forward time: 1.2581, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.7233 [2021-07-07 12:25:59,541][root][INFO] - ===> Epoch[652](3260/5): Loss 0.0292, Sem 0.0292 LR: 9.510e-02 Score 98.953 Data time: 0.2609, Forward time: 1.2581, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.8274 [2021-07-07 12:26:13,264][root][INFO] - ===> Epoch[653](3265/5): Loss 0.0278, Sem 0.0278 LR: 9.509e-02 Score 98.999 Data time: 0.1864, Forward time: 1.2581, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.6899 [2021-07-07 12:26:27,710][root][INFO] - ===> Epoch[654](3270/5): Loss 0.0285, Sem 0.0285 LR: 9.508e-02 Score 98.984 Data time: 0.1871, Forward time: 1.2581, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.8434 [2021-07-07 12:26:42,362][root][INFO] - ===> Epoch[655](3275/5): Loss 0.0273, Sem 0.0273 LR: 9.507e-02 Score 99.017 Data time: 0.2443, Forward time: 1.2581, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.8822 [2021-07-07 12:26:56,030][root][INFO] - ===> Epoch[656](3280/5): Loss 0.0283, Sem 0.0283 LR: 9.507e-02 Score 98.983 Data time: 0.2404, Forward time: 1.2579, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.6928 [2021-07-07 12:27:09,693][root][INFO] - ===> Epoch[657](3285/5): Loss 0.0293, Sem 0.0293 LR: 9.506e-02 Score 98.963 Data time: 0.1751, Forward time: 1.2577, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.6899 [2021-07-07 12:27:24,060][root][INFO] - ===> Epoch[658](3290/5): Loss 0.0291, Sem 0.0291 LR: 9.505e-02 Score 98.973 Data time: 0.2655, Forward time: 1.2577, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.8227 [2021-07-07 12:27:38,702][root][INFO] - ===> Epoch[659](3295/5): Loss 0.0298, Sem 0.0298 LR: 9.504e-02 Score 98.945 Data time: 0.2696, Forward time: 1.2578, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8680 [2021-07-07 12:27:51,835][root][INFO] - ===> Epoch[660](3300/5): Loss 0.0309, Sem 0.0309 LR: 9.504e-02 Score 98.891 Data time: 0.1822, Forward time: 1.2576, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.5806 [2021-07-07 12:28:06,425][root][INFO] - ===> Epoch[661](3305/5): Loss 0.0305, Sem 0.0305 LR: 9.503e-02 Score 98.907 Data time: 0.2906, Forward time: 1.2574, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8690 [2021-07-07 12:28:20,631][root][INFO] - ===> Epoch[662](3310/5): Loss 0.0325, Sem 0.0325 LR: 9.502e-02 Score 98.828 Data time: 0.2412, Forward time: 1.2576, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7929 [2021-07-07 12:28:34,412][root][INFO] - ===> Epoch[663](3315/5): Loss 0.0303, Sem 0.0303 LR: 9.501e-02 Score 98.913 Data time: 0.1357, Forward time: 1.2575, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7094 [2021-07-07 12:28:48,301][root][INFO] - ===> Epoch[664](3320/5): Loss 0.0299, Sem 0.0299 LR: 9.501e-02 Score 98.951 Data time: 0.1995, Forward time: 1.2575, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7350 [2021-07-07 12:29:02,465][root][INFO] - ===> Epoch[665](3325/5): Loss 0.0303, Sem 0.0303 LR: 9.500e-02 Score 98.936 Data time: 0.2618, Forward time: 1.2575, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7746 [2021-07-07 12:29:16,214][root][INFO] - ===> Epoch[666](3330/5): Loss 0.0294, Sem 0.0294 LR: 9.499e-02 Score 98.945 Data time: 0.2230, Forward time: 1.2576, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7012 [2021-07-07 12:29:29,887][root][INFO] - ===> Epoch[667](3335/5): Loss 0.0291, Sem 0.0291 LR: 9.498e-02 Score 98.968 Data time: 0.2047, Forward time: 1.2576, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.6933 [2021-07-07 12:29:43,493][root][INFO] - ===> Epoch[668](3340/5): Loss 0.0286, Sem 0.0286 LR: 9.498e-02 Score 98.982 Data time: 0.2360, Forward time: 1.2577, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.6794 [2021-07-07 12:29:57,463][root][INFO] - ===> Epoch[669](3345/5): Loss 0.0264, Sem 0.0264 LR: 9.497e-02 Score 99.051 Data time: 0.2179, Forward time: 1.2578, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.7527 [2021-07-07 12:30:10,812][root][INFO] - ===> Epoch[670](3350/5): Loss 0.0287, Sem 0.0287 LR: 9.496e-02 Score 98.975 Data time: 0.2174, Forward time: 1.2576, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.6254 [2021-07-07 12:30:25,046][root][INFO] - ===> Epoch[671](3355/5): Loss 0.0276, Sem 0.0276 LR: 9.495e-02 Score 99.007 Data time: 0.2399, Forward time: 1.2575, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.8009 [2021-07-07 12:30:38,644][root][INFO] - ===> Epoch[672](3360/5): Loss 0.0285, Sem 0.0285 LR: 9.495e-02 Score 98.983 Data time: 0.2252, Forward time: 1.2575, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.6536 [2021-07-07 12:30:53,393][root][INFO] - ===> Epoch[673](3365/5): Loss 0.0281, Sem 0.0281 LR: 9.494e-02 Score 98.982 Data time: 0.2264, Forward time: 1.2574, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.9060 [2021-07-07 12:31:08,372][root][INFO] - ===> Epoch[674](3370/5): Loss 0.0260, Sem 0.0260 LR: 9.493e-02 Score 99.069 Data time: 0.2616, Forward time: 1.2573, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.9504 [2021-07-07 12:31:22,300][root][INFO] - ===> Epoch[675](3375/5): Loss 0.0266, Sem 0.0266 LR: 9.492e-02 Score 99.042 Data time: 0.2305, Forward time: 1.2574, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7432 [2021-07-07 12:31:36,126][root][INFO] - ===> Epoch[676](3380/5): Loss 0.0285, Sem 0.0285 LR: 9.492e-02 Score 98.986 Data time: 0.2309, Forward time: 1.2573, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7246 [2021-07-07 12:31:49,723][root][INFO] - ===> Epoch[677](3385/5): Loss 0.0290, Sem 0.0290 LR: 9.491e-02 Score 98.969 Data time: 0.2081, Forward time: 1.2573, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6687 [2021-07-07 12:32:04,371][root][INFO] - ===> Epoch[678](3390/5): Loss 0.0267, Sem 0.0267 LR: 9.490e-02 Score 99.036 Data time: 0.2017, Forward time: 1.2573, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8787 [2021-07-07 12:32:17,566][root][INFO] - ===> Epoch[679](3395/5): Loss 0.0284, Sem 0.0284 LR: 9.489e-02 Score 98.976 Data time: 0.2421, Forward time: 1.2572, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.5863 [2021-07-07 12:32:30,507][root][INFO] - ===> Epoch[680](3400/5): Loss 0.0281, Sem 0.0281 LR: 9.489e-02 Score 98.996 Data time: 0.1988, Forward time: 1.2571, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.5420 [2021-07-07 12:32:30,508][root][INFO] - ===> Start testing [2021-07-07 12:33:05,818][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1352 Loss 0.300 (AVG: 0.465) Score 92.299 (AVG: 91.248) mIOU 71.531 mAP 85.188 mAcc 77.841 IOU: 65.036 0.000 79.379 77.443 95.051 91.883 36.379 81.259 98.672 83.050 78.776 87.075 55.901 mAP: 85.638 7.039 92.043 90.354 99.039 98.727 66.146 97.641 99.839 89.614 93.479 95.145 92.745 mAcc: 81.031 0.000 85.164 83.379 96.881 97.136 38.940 95.255 99.427 87.204 88.364 96.310 62.840 [2021-07-07 12:33:05,819][root][INFO] - Finished test. Elapsed time: 35.3091 [2021-07-07 12:33:06,883][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2021-07-07 12:33:06,884][root][INFO] - Current best mIoU: 71.531 at iter 1 [2021-07-07 12:33:07,829][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 12:33:21,636][root][INFO] - ===> Epoch[681](3405/5): Loss 0.0270, Sem 0.0270 LR: 9.488e-02 Score 99.026 Data time: 0.1952, Forward time: 1.2571, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7228 [2021-07-07 12:33:36,450][root][INFO] - ===> Epoch[682](3410/5): Loss 0.0271, Sem 0.0271 LR: 9.487e-02 Score 99.030 Data time: 0.1932, Forward time: 1.2570, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.9246 [2021-07-07 12:33:51,761][root][INFO] - ===> Epoch[683](3415/5): Loss 0.0267, Sem 0.0267 LR: 9.486e-02 Score 99.035 Data time: 0.2267, Forward time: 1.2571, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 3.0173 [2021-07-07 12:34:05,997][root][INFO] - ===> Epoch[684](3420/5): Loss 0.0269, Sem 0.0269 LR: 9.486e-02 Score 99.030 Data time: 0.1923, Forward time: 1.2571, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.8055 [2021-07-07 12:34:19,562][root][INFO] - ===> Epoch[685](3425/5): Loss 0.0273, Sem 0.0273 LR: 9.485e-02 Score 99.023 Data time: 0.1895, Forward time: 1.2570, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.6681 [2021-07-07 12:34:34,161][root][INFO] - ===> Epoch[686](3430/5): Loss 0.0257, Sem 0.0257 LR: 9.484e-02 Score 99.080 Data time: 0.2197, Forward time: 1.2569, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8769 [2021-07-07 12:34:48,185][root][INFO] - ===> Epoch[687](3435/5): Loss 0.0254, Sem 0.0254 LR: 9.483e-02 Score 99.089 Data time: 0.2758, Forward time: 1.2567, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7628 [2021-07-07 12:35:01,936][root][INFO] - ===> Epoch[688](3440/5): Loss 0.0254, Sem 0.0254 LR: 9.482e-02 Score 99.087 Data time: 0.2438, Forward time: 1.2567, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7128 [2021-07-07 12:35:15,989][root][INFO] - ===> Epoch[689](3445/5): Loss 0.0268, Sem 0.0268 LR: 9.482e-02 Score 99.033 Data time: 0.2159, Forward time: 1.2566, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7606 [2021-07-07 12:35:30,214][root][INFO] - ===> Epoch[690](3450/5): Loss 0.0270, Sem 0.0270 LR: 9.481e-02 Score 99.029 Data time: 0.2067, Forward time: 1.2566, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8012 [2021-07-07 12:35:44,376][root][INFO] - ===> Epoch[691](3455/5): Loss 0.0280, Sem 0.0280 LR: 9.480e-02 Score 98.995 Data time: 0.2132, Forward time: 1.2566, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7788 [2021-07-07 12:35:58,390][root][INFO] - ===> Epoch[692](3460/5): Loss 0.0258, Sem 0.0258 LR: 9.479e-02 Score 99.071 Data time: 0.2042, Forward time: 1.2569, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7462 [2021-07-07 12:36:13,117][root][INFO] - ===> Epoch[693](3465/5): Loss 0.0280, Sem 0.0280 LR: 9.479e-02 Score 98.999 Data time: 0.1699, Forward time: 1.2568, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.9077 [2021-07-07 12:36:27,217][root][INFO] - ===> Epoch[694](3470/5): Loss 0.0288, Sem 0.0288 LR: 9.478e-02 Score 98.963 Data time: 0.2304, Forward time: 1.2569, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7705 [2021-07-07 12:36:41,750][root][INFO] - ===> Epoch[695](3475/5): Loss 0.0267, Sem 0.0267 LR: 9.477e-02 Score 99.037 Data time: 0.2550, Forward time: 1.2569, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8688 [2021-07-07 12:36:56,428][root][INFO] - ===> Epoch[696](3480/5): Loss 0.0275, Sem 0.0275 LR: 9.476e-02 Score 99.024 Data time: 0.2655, Forward time: 1.2568, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.8912 [2021-07-07 12:37:10,980][root][INFO] - ===> Epoch[697](3485/5): Loss 0.0263, Sem 0.0263 LR: 9.476e-02 Score 99.049 Data time: 0.2332, Forward time: 1.2568, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8571 [2021-07-07 12:37:24,772][root][INFO] - ===> Epoch[698](3490/5): Loss 0.0286, Sem 0.0286 LR: 9.475e-02 Score 98.986 Data time: 0.2416, Forward time: 1.2568, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7131 [2021-07-07 12:37:38,835][root][INFO] - ===> Epoch[699](3495/5): Loss 0.0294, Sem 0.0294 LR: 9.474e-02 Score 98.940 Data time: 0.2294, Forward time: 1.2567, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7761 [2021-07-07 12:37:53,467][root][INFO] - ===> Epoch[700](3500/5): Loss 0.0284, Sem 0.0284 LR: 9.473e-02 Score 98.982 Data time: 0.2046, Forward time: 1.2566, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8738 [2021-07-07 12:38:07,380][root][INFO] - ===> Epoch[701](3505/5): Loss 0.0306, Sem 0.0306 LR: 9.473e-02 Score 98.938 Data time: 0.2170, Forward time: 1.2565, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7448 [2021-07-07 12:38:21,324][root][INFO] - ===> Epoch[702](3510/5): Loss 0.0300, Sem 0.0300 LR: 9.472e-02 Score 98.934 Data time: 0.2523, Forward time: 1.2564, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7400 [2021-07-07 12:38:35,465][root][INFO] - ===> Epoch[703](3515/5): Loss 0.0288, Sem 0.0288 LR: 9.471e-02 Score 98.975 Data time: 0.2591, Forward time: 1.2565, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7792 [2021-07-07 12:38:50,320][root][INFO] - ===> Epoch[704](3520/5): Loss 0.0284, Sem 0.0284 LR: 9.470e-02 Score 98.976 Data time: 0.2390, Forward time: 1.2567, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.9354 [2021-07-07 12:39:03,532][root][INFO] - ===> Epoch[705](3525/5): Loss 0.0287, Sem 0.0287 LR: 9.470e-02 Score 98.967 Data time: 0.2142, Forward time: 1.2567, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.5839 [2021-07-07 12:39:18,396][root][INFO] - ===> Epoch[706](3530/5): Loss 0.0286, Sem 0.0286 LR: 9.469e-02 Score 98.986 Data time: 0.2658, Forward time: 1.2566, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.9187 [2021-07-07 12:39:33,113][root][INFO] - ===> Epoch[707](3535/5): Loss 0.0289, Sem 0.0289 LR: 9.468e-02 Score 98.982 Data time: 0.2841, Forward time: 1.2567, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8858 [2021-07-07 12:39:46,915][root][INFO] - ===> Epoch[708](3540/5): Loss 0.0276, Sem 0.0276 LR: 9.467e-02 Score 99.020 Data time: 0.2294, Forward time: 1.2566, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7245 [2021-07-07 12:40:00,871][root][INFO] - ===> Epoch[709](3545/5): Loss 0.0276, Sem 0.0276 LR: 9.467e-02 Score 99.010 Data time: 0.2893, Forward time: 1.2566, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7475 [2021-07-07 12:40:15,503][root][INFO] - ===> Epoch[710](3550/5): Loss 0.0264, Sem 0.0264 LR: 9.466e-02 Score 99.048 Data time: 0.2270, Forward time: 1.2567, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8766 [2021-07-07 12:40:29,667][root][INFO] - ===> Epoch[711](3555/5): Loss 0.0281, Sem 0.0281 LR: 9.465e-02 Score 98.997 Data time: 0.2702, Forward time: 1.2567, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7887 [2021-07-07 12:40:43,242][root][INFO] - ===> Epoch[712](3560/5): Loss 0.0271, Sem 0.0271 LR: 9.464e-02 Score 99.030 Data time: 0.2100, Forward time: 1.2567, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.6630 [2021-07-07 12:40:57,321][root][INFO] - ===> Epoch[713](3565/5): Loss 0.0269, Sem 0.0269 LR: 9.464e-02 Score 99.036 Data time: 0.2573, Forward time: 1.2567, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7647 [2021-07-07 12:41:11,419][root][INFO] - ===> Epoch[714](3570/5): Loss 0.0264, Sem 0.0264 LR: 9.463e-02 Score 99.049 Data time: 0.1506, Forward time: 1.2567, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7708 [2021-07-07 12:41:25,045][root][INFO] - ===> Epoch[715](3575/5): Loss 0.0264, Sem 0.0264 LR: 9.462e-02 Score 99.054 Data time: 0.2610, Forward time: 1.2566, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.6834 [2021-07-07 12:41:38,585][root][INFO] - ===> Epoch[716](3580/5): Loss 0.0278, Sem 0.0278 LR: 9.461e-02 Score 99.018 Data time: 0.2574, Forward time: 1.2566, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6586 [2021-07-07 12:41:52,095][root][INFO] - ===> Epoch[717](3585/5): Loss 0.0275, Sem 0.0275 LR: 9.461e-02 Score 99.007 Data time: 0.2230, Forward time: 1.2566, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6502 [2021-07-07 12:42:05,697][root][INFO] - ===> Epoch[718](3590/5): Loss 0.0268, Sem 0.0268 LR: 9.460e-02 Score 99.050 Data time: 0.2439, Forward time: 1.2567, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6634 [2021-07-07 12:42:19,543][root][INFO] - ===> Epoch[719](3595/5): Loss 0.0277, Sem 0.0277 LR: 9.459e-02 Score 99.007 Data time: 0.2313, Forward time: 1.2566, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7209 [2021-07-07 12:42:32,266][root][INFO] - ===> Epoch[720](3600/5): Loss 0.0269, Sem 0.0269 LR: 9.458e-02 Score 99.026 Data time: 0.1967, Forward time: 1.2565, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.5049 [2021-07-07 12:42:32,267][root][INFO] - ===> Start testing [2021-07-07 12:43:07,735][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1223 Loss 0.256 (AVG: 0.448) Score 93.498 (AVG: 91.127) mIOU 71.131 mAP 84.953 mAcc 77.176 IOU: 63.992 0.154 79.826 77.509 94.861 91.322 36.388 82.820 98.562 75.539 79.636 86.795 57.304 mAP: 85.420 8.462 89.955 90.521 98.948 98.326 67.704 97.722 99.835 85.797 93.518 95.155 93.028 mAcc: 81.941 1.276 87.499 84.243 96.647 96.139 39.979 93.606 99.360 76.776 87.763 95.779 62.279 [2021-07-07 12:43:07,735][root][INFO] - Finished test. Elapsed time: 35.4678 [2021-07-07 12:43:08,871][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 12:43:21,561][root][INFO] - ===> Epoch[721](3605/5): Loss 0.0254, Sem 0.0254 LR: 9.458e-02 Score 99.097 Data time: 0.2059, Forward time: 1.2564, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.4968 [2021-07-07 12:43:35,995][root][INFO] - ===> Epoch[722](3610/5): Loss 0.0261, Sem 0.0261 LR: 9.457e-02 Score 99.076 Data time: 0.2852, Forward time: 1.2565, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8304 [2021-07-07 12:43:50,192][root][INFO] - ===> Epoch[723](3615/5): Loss 0.0253, Sem 0.0253 LR: 9.456e-02 Score 99.093 Data time: 0.2834, Forward time: 1.2564, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7878 [2021-07-07 12:44:03,586][root][INFO] - ===> Epoch[724](3620/5): Loss 0.0264, Sem 0.0264 LR: 9.455e-02 Score 99.063 Data time: 0.2350, Forward time: 1.2563, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.6297 [2021-07-07 12:44:17,801][root][INFO] - ===> Epoch[725](3625/5): Loss 0.0260, Sem 0.0260 LR: 9.455e-02 Score 99.053 Data time: 0.2023, Forward time: 1.2563, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7998 [2021-07-07 12:44:32,145][root][INFO] - ===> Epoch[726](3630/5): Loss 0.0253, Sem 0.0253 LR: 9.454e-02 Score 99.090 Data time: 0.2266, Forward time: 1.2563, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8187 [2021-07-07 12:44:46,292][root][INFO] - ===> Epoch[727](3635/5): Loss 0.0264, Sem 0.0264 LR: 9.453e-02 Score 99.053 Data time: 0.2410, Forward time: 1.2562, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7812 [2021-07-07 12:44:59,815][root][INFO] - ===> Epoch[728](3640/5): Loss 0.0259, Sem 0.0259 LR: 9.452e-02 Score 99.073 Data time: 0.2166, Forward time: 1.2561, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.6486 [2021-07-07 12:45:13,468][root][INFO] - ===> Epoch[729](3645/5): Loss 0.0272, Sem 0.0272 LR: 9.452e-02 Score 99.034 Data time: 0.1836, Forward time: 1.2560, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.6784 [2021-07-07 12:45:27,250][root][INFO] - ===> Epoch[730](3650/5): Loss 0.0256, Sem 0.0256 LR: 9.451e-02 Score 99.084 Data time: 0.2579, Forward time: 1.2559, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7122 [2021-07-07 12:45:41,190][root][INFO] - ===> Epoch[731](3655/5): Loss 0.0262, Sem 0.0262 LR: 9.450e-02 Score 99.052 Data time: 0.2305, Forward time: 1.2558, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7435 [2021-07-07 12:45:54,440][root][INFO] - ===> Epoch[732](3660/5): Loss 0.0248, Sem 0.0248 LR: 9.449e-02 Score 99.099 Data time: 0.1750, Forward time: 1.2557, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.6143 [2021-07-07 12:46:09,426][root][INFO] - ===> Epoch[733](3665/5): Loss 0.0257, Sem 0.0257 LR: 9.449e-02 Score 99.072 Data time: 0.2577, Forward time: 1.2557, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.9552 [2021-07-07 12:46:22,699][root][INFO] - ===> Epoch[734](3670/5): Loss 0.0262, Sem 0.0262 LR: 9.448e-02 Score 99.057 Data time: 0.1964, Forward time: 1.2557, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6114 [2021-07-07 12:46:37,004][root][INFO] - ===> Epoch[735](3675/5): Loss 0.0252, Sem 0.0252 LR: 9.447e-02 Score 99.096 Data time: 0.2609, Forward time: 1.2557, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8147 [2021-07-07 12:46:49,851][root][INFO] - ===> Epoch[736](3680/5): Loss 0.0262, Sem 0.0262 LR: 9.446e-02 Score 99.059 Data time: 0.2380, Forward time: 1.2557, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.5198 [2021-07-07 12:47:04,102][root][INFO] - ===> Epoch[737](3685/5): Loss 0.0249, Sem 0.0249 LR: 9.446e-02 Score 99.112 Data time: 0.2018, Forward time: 1.2557, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.8035 [2021-07-07 12:47:18,440][root][INFO] - ===> Epoch[738](3690/5): Loss 0.0258, Sem 0.0258 LR: 9.445e-02 Score 99.073 Data time: 0.1803, Forward time: 1.2557, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.8219 [2021-07-07 12:47:32,556][root][INFO] - ===> Epoch[739](3695/5): Loss 0.0252, Sem 0.0252 LR: 9.444e-02 Score 99.092 Data time: 0.2134, Forward time: 1.2556, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7753 [2021-07-07 12:47:46,776][root][INFO] - ===> Epoch[740](3700/5): Loss 0.0257, Sem 0.0257 LR: 9.443e-02 Score 99.082 Data time: 0.2438, Forward time: 1.2558, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8000 [2021-07-07 12:48:00,256][root][INFO] - ===> Epoch[741](3705/5): Loss 0.0265, Sem 0.0265 LR: 9.443e-02 Score 99.034 Data time: 0.1948, Forward time: 1.2558, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.6452 [2021-07-07 12:48:14,202][root][INFO] - ===> Epoch[742](3710/5): Loss 0.0260, Sem 0.0260 LR: 9.442e-02 Score 99.069 Data time: 0.2495, Forward time: 1.2559, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7374 [2021-07-07 12:48:28,282][root][INFO] - ===> Epoch[743](3715/5): Loss 0.0262, Sem 0.0262 LR: 9.441e-02 Score 99.073 Data time: 0.2102, Forward time: 1.2561, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7725 [2021-07-07 12:48:43,182][root][INFO] - ===> Epoch[744](3720/5): Loss 0.0246, Sem 0.0246 LR: 9.440e-02 Score 99.119 Data time: 0.2219, Forward time: 1.2561, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.9348 [2021-07-07 12:48:58,049][root][INFO] - ===> Epoch[745](3725/5): Loss 0.0274, Sem 0.0274 LR: 9.439e-02 Score 99.026 Data time: 0.2032, Forward time: 1.2561, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.9278 [2021-07-07 12:49:11,431][root][INFO] - ===> Epoch[746](3730/5): Loss 0.0267, Sem 0.0267 LR: 9.439e-02 Score 99.042 Data time: 0.2094, Forward time: 1.2561, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6270 [2021-07-07 12:49:25,632][root][INFO] - ===> Epoch[747](3735/5): Loss 0.0250, Sem 0.0250 LR: 9.438e-02 Score 99.106 Data time: 0.1834, Forward time: 1.2561, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.8025 [2021-07-07 12:49:39,800][root][INFO] - ===> Epoch[748](3740/5): Loss 0.0265, Sem 0.0265 LR: 9.437e-02 Score 99.054 Data time: 0.1938, Forward time: 1.2560, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7880 [2021-07-07 12:49:54,388][root][INFO] - ===> Epoch[749](3745/5): Loss 0.0241, Sem 0.0241 LR: 9.436e-02 Score 99.131 Data time: 0.2030, Forward time: 1.2560, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8660 [2021-07-07 12:50:08,382][root][INFO] - ===> Epoch[750](3750/5): Loss 0.0268, Sem 0.0268 LR: 9.436e-02 Score 99.036 Data time: 0.2132, Forward time: 1.2558, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7549 [2021-07-07 12:50:22,262][root][INFO] - ===> Epoch[751](3755/5): Loss 0.0247, Sem 0.0247 LR: 9.435e-02 Score 99.115 Data time: 0.2813, Forward time: 1.2558, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7266 [2021-07-07 12:50:35,313][root][INFO] - ===> Epoch[752](3760/5): Loss 0.0248, Sem 0.0248 LR: 9.434e-02 Score 99.100 Data time: 0.1493, Forward time: 1.2555, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.5649 [2021-07-07 12:50:49,782][root][INFO] - ===> Epoch[753](3765/5): Loss 0.0244, Sem 0.0244 LR: 9.433e-02 Score 99.116 Data time: 0.2499, Forward time: 1.2554, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.8449 [2021-07-07 12:51:04,688][root][INFO] - ===> Epoch[754](3770/5): Loss 0.0242, Sem 0.0242 LR: 9.433e-02 Score 99.128 Data time: 0.2699, Forward time: 1.2554, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.9296 [2021-07-07 12:51:18,872][root][INFO] - ===> Epoch[755](3775/5): Loss 0.0245, Sem 0.0245 LR: 9.432e-02 Score 99.121 Data time: 0.2129, Forward time: 1.2553, Backward time: 1.2345, DDP time: 0.0042, Total iter time: 2.7901 [2021-07-07 12:51:33,532][root][INFO] - ===> Epoch[756](3780/5): Loss 0.0248, Sem 0.0248 LR: 9.431e-02 Score 99.114 Data time: 0.2821, Forward time: 1.2556, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.8819 [2021-07-07 12:51:47,733][root][INFO] - ===> Epoch[757](3785/5): Loss 0.0240, Sem 0.0240 LR: 9.430e-02 Score 99.146 Data time: 0.2467, Forward time: 1.2556, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7997 [2021-07-07 12:52:01,096][root][INFO] - ===> Epoch[758](3790/5): Loss 0.0260, Sem 0.0260 LR: 9.430e-02 Score 99.067 Data time: 0.2320, Forward time: 1.2556, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.6176 [2021-07-07 12:52:15,737][root][INFO] - ===> Epoch[759](3795/5): Loss 0.0247, Sem 0.0247 LR: 9.429e-02 Score 99.108 Data time: 0.2721, Forward time: 1.2556, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.8626 [2021-07-07 12:52:30,255][root][INFO] - ===> Epoch[760](3800/5): Loss 0.0248, Sem 0.0248 LR: 9.428e-02 Score 99.110 Data time: 0.2080, Forward time: 1.2557, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.8523 [2021-07-07 12:52:30,256][root][INFO] - ===> Start testing [2021-07-07 12:53:06,556][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1253 Loss 0.312 (AVG: 0.474) Score 92.373 (AVG: 91.255) mIOU 71.431 mAP 85.310 mAcc 77.453 IOU: 64.881 0.074 82.592 78.314 94.531 91.727 31.094 83.252 98.638 79.742 79.803 86.530 57.425 mAP: 85.399 13.151 95.148 91.411 99.114 98.575 61.892 97.806 99.847 84.714 93.264 95.381 93.325 mAcc: 79.247 0.547 90.471 86.037 96.368 96.282 33.374 94.898 99.441 82.251 89.604 96.456 61.914 [2021-07-07 12:53:06,557][root][INFO] - Finished test. Elapsed time: 36.2997 [2021-07-07 12:53:07,593][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 12:53:20,173][root][INFO] - ===> Epoch[761](3805/5): Loss 0.0255, Sem 0.0255 LR: 9.427e-02 Score 99.088 Data time: 0.1719, Forward time: 1.2556, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.4838 [2021-07-07 12:53:34,582][root][INFO] - ===> Epoch[762](3810/5): Loss 0.0250, Sem 0.0250 LR: 9.427e-02 Score 99.103 Data time: 0.2800, Forward time: 1.2556, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8226 [2021-07-07 12:53:48,173][root][INFO] - ===> Epoch[763](3815/5): Loss 0.0239, Sem 0.0239 LR: 9.426e-02 Score 99.146 Data time: 0.2076, Forward time: 1.2557, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.6773 [2021-07-07 12:54:01,612][root][INFO] - ===> Epoch[764](3820/5): Loss 0.0273, Sem 0.0273 LR: 9.425e-02 Score 99.026 Data time: 0.2364, Forward time: 1.2557, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.6429 [2021-07-07 12:54:16,617][root][INFO] - ===> Epoch[765](3825/5): Loss 0.0244, Sem 0.0244 LR: 9.424e-02 Score 99.119 Data time: 0.1929, Forward time: 1.2558, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.9636 [2021-07-07 12:54:29,746][root][INFO] - ===> Epoch[766](3830/5): Loss 0.0243, Sem 0.0243 LR: 9.424e-02 Score 99.123 Data time: 0.2331, Forward time: 1.2557, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.5784 [2021-07-07 12:54:43,821][root][INFO] - ===> Epoch[767](3835/5): Loss 0.0247, Sem 0.0247 LR: 9.423e-02 Score 99.109 Data time: 0.2598, Forward time: 1.2558, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7654 [2021-07-07 12:54:57,400][root][INFO] - ===> Epoch[768](3840/5): Loss 0.0249, Sem 0.0249 LR: 9.422e-02 Score 99.104 Data time: 0.2539, Forward time: 1.2557, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6678 [2021-07-07 12:55:10,357][root][INFO] - ===> Epoch[769](3845/5): Loss 0.0254, Sem 0.0254 LR: 9.421e-02 Score 99.092 Data time: 0.2032, Forward time: 1.2557, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.5356 [2021-07-07 12:55:24,410][root][INFO] - ===> Epoch[770](3850/5): Loss 0.0240, Sem 0.0240 LR: 9.421e-02 Score 99.136 Data time: 0.2552, Forward time: 1.2558, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7554 [2021-07-07 12:55:38,533][root][INFO] - ===> Epoch[771](3855/5): Loss 0.0246, Sem 0.0246 LR: 9.420e-02 Score 99.115 Data time: 0.2016, Forward time: 1.2556, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7813 [2021-07-07 12:55:52,904][root][INFO] - ===> Epoch[772](3860/5): Loss 0.0244, Sem 0.0244 LR: 9.419e-02 Score 99.123 Data time: 0.2586, Forward time: 1.2557, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8204 [2021-07-07 12:56:06,102][root][INFO] - ===> Epoch[773](3865/5): Loss 0.0243, Sem 0.0243 LR: 9.418e-02 Score 99.129 Data time: 0.2715, Forward time: 1.2557, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.5844 [2021-07-07 12:56:20,317][root][INFO] - ===> Epoch[774](3870/5): Loss 0.0239, Sem 0.0239 LR: 9.418e-02 Score 99.145 Data time: 0.2151, Forward time: 1.2557, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7976 [2021-07-07 12:56:34,578][root][INFO] - ===> Epoch[775](3875/5): Loss 0.0236, Sem 0.0236 LR: 9.417e-02 Score 99.145 Data time: 0.2322, Forward time: 1.2557, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.8097 [2021-07-07 12:56:48,452][root][INFO] - ===> Epoch[776](3880/5): Loss 0.0235, Sem 0.0235 LR: 9.416e-02 Score 99.158 Data time: 0.1805, Forward time: 1.2556, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7326 [2021-07-07 12:57:01,824][root][INFO] - ===> Epoch[777](3885/5): Loss 0.0249, Sem 0.0249 LR: 9.415e-02 Score 99.097 Data time: 0.2325, Forward time: 1.2555, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6334 [2021-07-07 12:57:15,504][root][INFO] - ===> Epoch[778](3890/5): Loss 0.0245, Sem 0.0245 LR: 9.415e-02 Score 99.118 Data time: 0.2128, Forward time: 1.2554, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6971 [2021-07-07 12:57:28,732][root][INFO] - ===> Epoch[779](3895/5): Loss 0.0240, Sem 0.0240 LR: 9.414e-02 Score 99.132 Data time: 0.1968, Forward time: 1.2554, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.6020 [2021-07-07 12:57:42,810][root][INFO] - ===> Epoch[780](3900/5): Loss 0.0241, Sem 0.0241 LR: 9.413e-02 Score 99.126 Data time: 0.1867, Forward time: 1.2556, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7735 [2021-07-07 12:57:57,036][root][INFO] - ===> Epoch[781](3905/5): Loss 0.0245, Sem 0.0245 LR: 9.412e-02 Score 99.122 Data time: 0.2075, Forward time: 1.2556, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.8060 [2021-07-07 12:58:10,871][root][INFO] - ===> Epoch[782](3910/5): Loss 0.0232, Sem 0.0232 LR: 9.412e-02 Score 99.161 Data time: 0.2658, Forward time: 1.2556, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7107 [2021-07-07 12:58:25,472][root][INFO] - ===> Epoch[783](3915/5): Loss 0.0247, Sem 0.0247 LR: 9.411e-02 Score 99.108 Data time: 0.1903, Forward time: 1.2557, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.8726 [2021-07-07 12:58:39,419][root][INFO] - ===> Epoch[784](3920/5): Loss 0.0241, Sem 0.0241 LR: 9.410e-02 Score 99.137 Data time: 0.2212, Forward time: 1.2558, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7427 [2021-07-07 12:58:53,621][root][INFO] - ===> Epoch[785](3925/5): Loss 0.0245, Sem 0.0245 LR: 9.409e-02 Score 99.124 Data time: 0.1968, Forward time: 1.2556, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8000 [2021-07-07 12:59:08,172][root][INFO] - ===> Epoch[786](3930/5): Loss 0.0234, Sem 0.0234 LR: 9.409e-02 Score 99.152 Data time: 0.2198, Forward time: 1.2557, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8721 [2021-07-07 12:59:22,148][root][INFO] - ===> Epoch[787](3935/5): Loss 0.0243, Sem 0.0243 LR: 9.408e-02 Score 99.133 Data time: 0.2844, Forward time: 1.2556, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7412 [2021-07-07 12:59:36,896][root][INFO] - ===> Epoch[788](3940/5): Loss 0.0247, Sem 0.0247 LR: 9.407e-02 Score 99.114 Data time: 0.2394, Forward time: 1.2558, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.9098 [2021-07-07 12:59:51,512][root][INFO] - ===> Epoch[789](3945/5): Loss 0.0240, Sem 0.0240 LR: 9.406e-02 Score 99.130 Data time: 0.2615, Forward time: 1.2559, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8769 [2021-07-07 13:00:05,635][root][INFO] - ===> Epoch[790](3950/5): Loss 0.0259, Sem 0.0259 LR: 9.406e-02 Score 99.072 Data time: 0.1931, Forward time: 1.2558, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7730 [2021-07-07 13:00:20,952][root][INFO] - ===> Epoch[791](3955/5): Loss 0.0258, Sem 0.0258 LR: 9.405e-02 Score 99.078 Data time: 0.3026, Forward time: 1.2559, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 3.0125 [2021-07-07 13:00:34,892][root][INFO] - ===> Epoch[792](3960/5): Loss 0.0257, Sem 0.0257 LR: 9.404e-02 Score 99.074 Data time: 0.1561, Forward time: 1.2559, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7495 [2021-07-07 13:00:48,543][root][INFO] - ===> Epoch[793](3965/5): Loss 0.0254, Sem 0.0254 LR: 9.403e-02 Score 99.094 Data time: 0.2427, Forward time: 1.2559, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.6758 [2021-07-07 13:01:02,958][root][INFO] - ===> Epoch[794](3970/5): Loss 0.0267, Sem 0.0267 LR: 9.402e-02 Score 99.043 Data time: 0.2246, Forward time: 1.2558, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.8309 [2021-07-07 13:01:16,146][root][INFO] - ===> Epoch[795](3975/5): Loss 0.0250, Sem 0.0250 LR: 9.402e-02 Score 99.098 Data time: 0.1748, Forward time: 1.2558, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.5890 [2021-07-07 13:01:30,988][root][INFO] - ===> Epoch[796](3980/5): Loss 0.0241, Sem 0.0241 LR: 9.401e-02 Score 99.137 Data time: 0.1630, Forward time: 1.2558, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.9242 [2021-07-07 13:01:45,430][root][INFO] - ===> Epoch[797](3985/5): Loss 0.0240, Sem 0.0240 LR: 9.400e-02 Score 99.136 Data time: 0.2939, Forward time: 1.2559, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.8284 [2021-07-07 13:01:59,998][root][INFO] - ===> Epoch[798](3990/5): Loss 0.0243, Sem 0.0243 LR: 9.399e-02 Score 99.131 Data time: 0.2803, Forward time: 1.2561, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8666 [2021-07-07 13:02:13,885][root][INFO] - ===> Epoch[799](3995/5): Loss 0.0245, Sem 0.0245 LR: 9.399e-02 Score 99.122 Data time: 0.2093, Forward time: 1.2562, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.7310 [2021-07-07 13:02:27,743][root][INFO] - ===> Epoch[800](4000/5): Loss 0.0246, Sem 0.0246 LR: 9.398e-02 Score 99.115 Data time: 0.1945, Forward time: 1.2561, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7364 [2021-07-07 13:02:27,746][root][INFO] - ===> Start testing [2021-07-07 13:03:03,734][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1258 Loss 0.281 (AVG: 0.482) Score 92.376 (AVG: 91.181) mIOU 70.890 mAP 85.130 mAcc 76.837 IOU: 64.557 0.118 81.620 77.572 94.740 91.395 34.633 81.326 98.656 74.297 78.893 86.705 57.062 mAP: 85.205 10.818 94.263 90.358 99.172 98.283 64.071 97.416 99.815 86.144 92.893 95.390 92.859 mAcc: 79.836 0.820 90.050 84.688 96.602 95.148 37.380 92.993 99.474 75.829 88.469 96.816 60.773 [2021-07-07 13:03:03,735][root][INFO] - Finished test. Elapsed time: 35.9884 [2021-07-07 13:03:04,849][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 13:03:18,329][root][INFO] - ===> Epoch[801](4005/5): Loss 0.0235, Sem 0.0235 LR: 9.397e-02 Score 99.145 Data time: 0.1775, Forward time: 1.2561, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.6541 [2021-07-07 13:03:32,211][root][INFO] - ===> Epoch[802](4010/5): Loss 0.0246, Sem 0.0246 LR: 9.396e-02 Score 99.127 Data time: 0.2260, Forward time: 1.2561, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.7239 [2021-07-07 13:03:46,635][root][INFO] - ===> Epoch[803](4015/5): Loss 0.0234, Sem 0.0234 LR: 9.396e-02 Score 99.162 Data time: 0.2199, Forward time: 1.2562, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8507 [2021-07-07 13:04:01,044][root][INFO] - ===> Epoch[804](4020/5): Loss 0.0244, Sem 0.0244 LR: 9.395e-02 Score 99.121 Data time: 0.1675, Forward time: 1.2561, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.8451 [2021-07-07 13:04:15,278][root][INFO] - ===> Epoch[805](4025/5): Loss 0.0252, Sem 0.0252 LR: 9.394e-02 Score 99.094 Data time: 0.1418, Forward time: 1.2563, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.8115 [2021-07-07 13:04:28,987][root][INFO] - ===> Epoch[806](4030/5): Loss 0.0235, Sem 0.0235 LR: 9.393e-02 Score 99.151 Data time: 0.2325, Forward time: 1.2564, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.6981 [2021-07-07 13:04:43,126][root][INFO] - ===> Epoch[807](4035/5): Loss 0.0234, Sem 0.0234 LR: 9.393e-02 Score 99.161 Data time: 0.2402, Forward time: 1.2566, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.7854 [2021-07-07 13:04:57,699][root][INFO] - ===> Epoch[808](4040/5): Loss 0.0241, Sem 0.0241 LR: 9.392e-02 Score 99.141 Data time: 0.2701, Forward time: 1.2567, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.8663 [2021-07-07 13:05:11,869][root][INFO] - ===> Epoch[809](4045/5): Loss 0.0234, Sem 0.0234 LR: 9.391e-02 Score 99.159 Data time: 0.2407, Forward time: 1.2568, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7739 [2021-07-07 13:05:26,350][root][INFO] - ===> Epoch[810](4050/5): Loss 0.0242, Sem 0.0242 LR: 9.390e-02 Score 99.127 Data time: 0.2384, Forward time: 1.2569, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8429 [2021-07-07 13:05:40,660][root][INFO] - ===> Epoch[811](4055/5): Loss 0.0235, Sem 0.0235 LR: 9.390e-02 Score 99.151 Data time: 0.2818, Forward time: 1.2569, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8101 [2021-07-07 13:05:53,928][root][INFO] - ===> Epoch[812](4060/5): Loss 0.0239, Sem 0.0239 LR: 9.389e-02 Score 99.141 Data time: 0.2480, Forward time: 1.2569, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6011 [2021-07-07 13:06:07,802][root][INFO] - ===> Epoch[813](4065/5): Loss 0.0236, Sem 0.0236 LR: 9.388e-02 Score 99.143 Data time: 0.2329, Forward time: 1.2569, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7204 [2021-07-07 13:06:22,496][root][INFO] - ===> Epoch[814](4070/5): Loss 0.0231, Sem 0.0231 LR: 9.387e-02 Score 99.168 Data time: 0.2403, Forward time: 1.2570, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8768 [2021-07-07 13:06:36,065][root][INFO] - ===> Epoch[815](4075/5): Loss 0.0236, Sem 0.0236 LR: 9.387e-02 Score 99.146 Data time: 0.1687, Forward time: 1.2569, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.6705 [2021-07-07 13:06:49,507][root][INFO] - ===> Epoch[816](4080/5): Loss 0.0246, Sem 0.0246 LR: 9.386e-02 Score 99.115 Data time: 0.2305, Forward time: 1.2569, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6441 [2021-07-07 13:07:03,663][root][INFO] - ===> Epoch[817](4085/5): Loss 0.0243, Sem 0.0243 LR: 9.385e-02 Score 99.124 Data time: 0.1885, Forward time: 1.2570, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7852 [2021-07-07 13:07:17,743][root][INFO] - ===> Epoch[818](4090/5): Loss 0.0227, Sem 0.0227 LR: 9.384e-02 Score 99.179 Data time: 0.2373, Forward time: 1.2570, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7733 [2021-07-07 13:07:32,068][root][INFO] - ===> Epoch[819](4095/5): Loss 0.0240, Sem 0.0240 LR: 9.384e-02 Score 99.140 Data time: 0.2273, Forward time: 1.2570, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8147 [2021-07-07 13:07:46,645][root][INFO] - ===> Epoch[820](4100/5): Loss 0.0240, Sem 0.0240 LR: 9.383e-02 Score 99.137 Data time: 0.2978, Forward time: 1.2570, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8580 [2021-07-07 13:08:01,024][root][INFO] - ===> Epoch[821](4105/5): Loss 0.0232, Sem 0.0232 LR: 9.382e-02 Score 99.164 Data time: 0.3085, Forward time: 1.2571, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8189 [2021-07-07 13:08:15,151][root][INFO] - ===> Epoch[822](4110/5): Loss 0.0238, Sem 0.0238 LR: 9.381e-02 Score 99.141 Data time: 0.2114, Forward time: 1.2572, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7854 [2021-07-07 13:08:29,218][root][INFO] - ===> Epoch[823](4115/5): Loss 0.0240, Sem 0.0240 LR: 9.381e-02 Score 99.132 Data time: 0.2178, Forward time: 1.2572, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7602 [2021-07-07 13:08:43,124][root][INFO] - ===> Epoch[824](4120/5): Loss 0.0243, Sem 0.0243 LR: 9.380e-02 Score 99.143 Data time: 0.2387, Forward time: 1.2572, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7383 [2021-07-07 13:08:57,436][root][INFO] - ===> Epoch[825](4125/5): Loss 0.0237, Sem 0.0237 LR: 9.379e-02 Score 99.149 Data time: 0.2987, Forward time: 1.2572, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8193 [2021-07-07 13:09:11,004][root][INFO] - ===> Epoch[826](4130/5): Loss 0.0237, Sem 0.0237 LR: 9.378e-02 Score 99.153 Data time: 0.1818, Forward time: 1.2572, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6605 [2021-07-07 13:09:24,930][root][INFO] - ===> Epoch[827](4135/5): Loss 0.0239, Sem 0.0239 LR: 9.378e-02 Score 99.147 Data time: 0.2655, Forward time: 1.2572, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7361 [2021-07-07 13:09:38,555][root][INFO] - ===> Epoch[828](4140/5): Loss 0.0240, Sem 0.0240 LR: 9.377e-02 Score 99.146 Data time: 0.2064, Forward time: 1.2571, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6897 [2021-07-07 13:09:51,877][root][INFO] - ===> Epoch[829](4145/5): Loss 0.0226, Sem 0.0226 LR: 9.376e-02 Score 99.181 Data time: 0.2674, Forward time: 1.2571, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6189 [2021-07-07 13:10:05,407][root][INFO] - ===> Epoch[830](4150/5): Loss 0.0235, Sem 0.0235 LR: 9.375e-02 Score 99.162 Data time: 0.2269, Forward time: 1.2570, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6599 [2021-07-07 13:10:19,146][root][INFO] - ===> Epoch[831](4155/5): Loss 0.0240, Sem 0.0240 LR: 9.375e-02 Score 99.137 Data time: 0.1809, Forward time: 1.2570, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6981 [2021-07-07 13:10:34,141][root][INFO] - ===> Epoch[832](4160/5): Loss 0.0231, Sem 0.0231 LR: 9.374e-02 Score 99.168 Data time: 0.2365, Forward time: 1.2570, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.9388 [2021-07-07 13:10:48,380][root][INFO] - ===> Epoch[833](4165/5): Loss 0.0253, Sem 0.0253 LR: 9.373e-02 Score 99.097 Data time: 0.2543, Forward time: 1.2570, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7827 [2021-07-07 13:11:02,075][root][INFO] - ===> Epoch[834](4170/5): Loss 0.0238, Sem 0.0238 LR: 9.372e-02 Score 99.140 Data time: 0.2564, Forward time: 1.2570, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6840 [2021-07-07 13:11:16,980][root][INFO] - ===> Epoch[835](4175/5): Loss 0.0236, Sem 0.0236 LR: 9.372e-02 Score 99.148 Data time: 0.2100, Forward time: 1.2569, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.9335 [2021-07-07 13:11:30,581][root][INFO] - ===> Epoch[836](4180/5): Loss 0.0235, Sem 0.0235 LR: 9.371e-02 Score 99.153 Data time: 0.2417, Forward time: 1.2569, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6800 [2021-07-07 13:11:44,179][root][INFO] - ===> Epoch[837](4185/5): Loss 0.0229, Sem 0.0229 LR: 9.370e-02 Score 99.175 Data time: 0.1724, Forward time: 1.2568, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6737 [2021-07-07 13:11:59,043][root][INFO] - ===> Epoch[838](4190/5): Loss 0.0237, Sem 0.0237 LR: 9.369e-02 Score 99.141 Data time: 0.2566, Forward time: 1.2570, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.9130 [2021-07-07 13:12:12,756][root][INFO] - ===> Epoch[839](4195/5): Loss 0.0241, Sem 0.0241 LR: 9.369e-02 Score 99.137 Data time: 0.2381, Forward time: 1.2570, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7015 [2021-07-07 13:12:27,518][root][INFO] - ===> Epoch[840](4200/5): Loss 0.0242, Sem 0.0242 LR: 9.368e-02 Score 99.125 Data time: 0.2291, Forward time: 1.2570, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.9051 [2021-07-07 13:12:27,521][root][INFO] - ===> Start testing [2021-07-07 13:13:02,906][root][INFO] - 68/68: Data time: 0.0320, Iter time: 0.1306 Loss 0.269 (AVG: 0.495) Score 93.539 (AVG: 91.093) mIOU 70.602 mAP 84.854 mAcc 76.338 IOU: 64.594 0.091 81.157 76.801 94.866 91.164 35.299 82.090 98.548 71.139 79.249 86.505 56.321 mAP: 84.982 10.934 92.477 89.482 99.011 98.159 65.820 97.381 99.836 82.739 93.116 95.726 93.438 mAcc: 80.968 0.638 87.677 82.519 97.072 95.502 38.326 92.253 99.277 72.346 88.673 96.965 60.173 [2021-07-07 13:13:02,907][root][INFO] - Finished test. Elapsed time: 35.3851 [2021-07-07 13:13:03,968][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 13:13:17,204][root][INFO] - ===> Epoch[841](4205/5): Loss 0.0234, Sem 0.0234 LR: 9.367e-02 Score 99.154 Data time: 0.2521, Forward time: 1.2569, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6016 [2021-07-07 13:13:31,760][root][INFO] - ===> Epoch[842](4210/5): Loss 0.0232, Sem 0.0232 LR: 9.366e-02 Score 99.163 Data time: 0.2748, Forward time: 1.2570, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8528 [2021-07-07 13:13:45,202][root][INFO] - ===> Epoch[843](4215/5): Loss 0.0235, Sem 0.0235 LR: 9.365e-02 Score 99.149 Data time: 0.1992, Forward time: 1.2570, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.6374 [2021-07-07 13:13:58,574][root][INFO] - ===> Epoch[844](4220/5): Loss 0.0237, Sem 0.0237 LR: 9.365e-02 Score 99.146 Data time: 0.2140, Forward time: 1.2571, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6316 [2021-07-07 13:14:12,750][root][INFO] - ===> Epoch[845](4225/5): Loss 0.0233, Sem 0.0233 LR: 9.364e-02 Score 99.158 Data time: 0.2145, Forward time: 1.2570, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7753 [2021-07-07 13:14:26,647][root][INFO] - ===> Epoch[846](4230/5): Loss 0.0233, Sem 0.0233 LR: 9.363e-02 Score 99.165 Data time: 0.2634, Forward time: 1.2571, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7414 [2021-07-07 13:14:40,829][root][INFO] - ===> Epoch[847](4235/5): Loss 0.0241, Sem 0.0241 LR: 9.362e-02 Score 99.137 Data time: 0.2554, Forward time: 1.2571, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7868 [2021-07-07 13:14:55,486][root][INFO] - ===> Epoch[848](4240/5): Loss 0.0263, Sem 0.0263 LR: 9.362e-02 Score 99.056 Data time: 0.2931, Forward time: 1.2571, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8740 [2021-07-07 13:15:10,523][root][INFO] - ===> Epoch[849](4245/5): Loss 0.0292, Sem 0.0292 LR: 9.361e-02 Score 99.009 Data time: 0.1417, Forward time: 1.2569, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.9702 [2021-07-07 13:15:24,464][root][INFO] - ===> Epoch[850](4250/5): Loss 0.0331, Sem 0.0331 LR: 9.360e-02 Score 98.833 Data time: 0.2423, Forward time: 1.2569, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7461 [2021-07-07 13:15:38,845][root][INFO] - ===> Epoch[851](4255/5): Loss 0.0314, Sem 0.0314 LR: 9.359e-02 Score 98.913 Data time: 0.2484, Forward time: 1.2569, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8274 [2021-07-07 13:15:52,287][root][INFO] - ===> Epoch[852](4260/5): Loss 0.0328, Sem 0.0328 LR: 9.359e-02 Score 98.848 Data time: 0.1823, Forward time: 1.2569, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6367 [2021-07-07 13:16:05,841][root][INFO] - ===> Epoch[853](4265/5): Loss 0.0295, Sem 0.0295 LR: 9.358e-02 Score 98.963 Data time: 0.1942, Forward time: 1.2569, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6708 [2021-07-07 13:16:20,607][root][INFO] - ===> Epoch[854](4270/5): Loss 0.0322, Sem 0.0322 LR: 9.357e-02 Score 98.885 Data time: 0.2157, Forward time: 1.2570, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.9049 [2021-07-07 13:16:35,241][root][INFO] - ===> Epoch[855](4275/5): Loss 0.0314, Sem 0.0314 LR: 9.356e-02 Score 98.905 Data time: 0.2152, Forward time: 1.2570, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8659 [2021-07-07 13:16:48,688][root][INFO] - ===> Epoch[856](4280/5): Loss 0.0327, Sem 0.0327 LR: 9.356e-02 Score 98.855 Data time: 0.2359, Forward time: 1.2569, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.6412 [2021-07-07 13:17:03,369][root][INFO] - ===> Epoch[857](4285/5): Loss 0.0332, Sem 0.0332 LR: 9.355e-02 Score 98.842 Data time: 0.2310, Forward time: 1.2571, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8859 [2021-07-07 13:17:17,742][root][INFO] - ===> Epoch[858](4290/5): Loss 0.0319, Sem 0.0319 LR: 9.354e-02 Score 98.877 Data time: 0.2647, Forward time: 1.2571, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8270 [2021-07-07 13:17:31,460][root][INFO] - ===> Epoch[859](4295/5): Loss 0.0334, Sem 0.0334 LR: 9.353e-02 Score 98.832 Data time: 0.1918, Forward time: 1.2570, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7040 [2021-07-07 13:17:45,669][root][INFO] - ===> Epoch[860](4300/5): Loss 0.0312, Sem 0.0312 LR: 9.353e-02 Score 98.888 Data time: 0.2122, Forward time: 1.2570, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.8001 [2021-07-07 13:17:59,394][root][INFO] - ===> Epoch[861](4305/5): Loss 0.0279, Sem 0.0279 LR: 9.352e-02 Score 99.008 Data time: 0.2109, Forward time: 1.2570, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7065 [2021-07-07 13:18:13,892][root][INFO] - ===> Epoch[862](4310/5): Loss 0.0322, Sem 0.0322 LR: 9.351e-02 Score 98.874 Data time: 0.1702, Forward time: 1.2571, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.8491 [2021-07-07 13:18:28,319][root][INFO] - ===> Epoch[863](4315/5): Loss 0.0282, Sem 0.0282 LR: 9.350e-02 Score 98.985 Data time: 0.2538, Forward time: 1.2572, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.8392 [2021-07-07 13:18:41,769][root][INFO] - ===> Epoch[864](4320/5): Loss 0.0282, Sem 0.0282 LR: 9.350e-02 Score 99.000 Data time: 0.2432, Forward time: 1.2572, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.6475 [2021-07-07 13:18:55,395][root][INFO] - ===> Epoch[865](4325/5): Loss 0.0265, Sem 0.0265 LR: 9.349e-02 Score 99.068 Data time: 0.2339, Forward time: 1.2573, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6845 [2021-07-07 13:19:09,134][root][INFO] - ===> Epoch[866](4330/5): Loss 0.0274, Sem 0.0274 LR: 9.348e-02 Score 99.028 Data time: 0.2320, Forward time: 1.2573, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6903 [2021-07-07 13:19:22,708][root][INFO] - ===> Epoch[867](4335/5): Loss 0.0263, Sem 0.0263 LR: 9.347e-02 Score 99.054 Data time: 0.2325, Forward time: 1.2573, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6579 [2021-07-07 13:19:37,853][root][INFO] - ===> Epoch[868](4340/5): Loss 0.0256, Sem 0.0256 LR: 9.347e-02 Score 99.083 Data time: 0.2630, Forward time: 1.2574, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.9746 [2021-07-07 13:19:51,229][root][INFO] - ===> Epoch[869](4345/5): Loss 0.0248, Sem 0.0248 LR: 9.346e-02 Score 99.098 Data time: 0.2208, Forward time: 1.2573, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6178 [2021-07-07 13:20:04,613][root][INFO] - ===> Epoch[870](4350/5): Loss 0.0275, Sem 0.0275 LR: 9.345e-02 Score 99.028 Data time: 0.3228, Forward time: 1.2573, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6280 [2021-07-07 13:20:20,455][root][INFO] - ===> Epoch[871](4355/5): Loss 0.0250, Sem 0.0250 LR: 9.344e-02 Score 99.116 Data time: 0.2066, Forward time: 1.2573, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 3.1228 [2021-07-07 13:20:34,069][root][INFO] - ===> Epoch[872](4360/5): Loss 0.0241, Sem 0.0241 LR: 9.344e-02 Score 99.134 Data time: 0.2347, Forward time: 1.2574, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6763 [2021-07-07 13:20:48,334][root][INFO] - ===> Epoch[873](4365/5): Loss 0.0244, Sem 0.0244 LR: 9.343e-02 Score 99.134 Data time: 0.2988, Forward time: 1.2573, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7978 [2021-07-07 13:21:01,624][root][INFO] - ===> Epoch[874](4370/5): Loss 0.0252, Sem 0.0252 LR: 9.342e-02 Score 99.090 Data time: 0.2592, Forward time: 1.2574, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.6108 [2021-07-07 13:21:16,085][root][INFO] - ===> Epoch[875](4375/5): Loss 0.0248, Sem 0.0248 LR: 9.341e-02 Score 99.107 Data time: 0.2428, Forward time: 1.2574, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.8435 [2021-07-07 13:21:30,336][root][INFO] - ===> Epoch[876](4380/5): Loss 0.0255, Sem 0.0255 LR: 9.341e-02 Score 99.091 Data time: 0.1921, Forward time: 1.2575, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.8145 [2021-07-07 13:21:44,225][root][INFO] - ===> Epoch[877](4385/5): Loss 0.0248, Sem 0.0248 LR: 9.340e-02 Score 99.106 Data time: 0.2315, Forward time: 1.2576, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7305 [2021-07-07 13:21:57,893][root][INFO] - ===> Epoch[878](4390/5): Loss 0.0263, Sem 0.0263 LR: 9.339e-02 Score 99.068 Data time: 0.1917, Forward time: 1.2575, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6938 [2021-07-07 13:22:12,148][root][INFO] - ===> Epoch[879](4395/5): Loss 0.0262, Sem 0.0262 LR: 9.338e-02 Score 99.050 Data time: 0.2111, Forward time: 1.2575, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7992 [2021-07-07 13:22:25,661][root][INFO] - ===> Epoch[880](4400/5): Loss 0.0253, Sem 0.0253 LR: 9.338e-02 Score 99.100 Data time: 0.1897, Forward time: 1.2575, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.6597 [2021-07-07 13:22:25,662][root][INFO] - ===> Start testing [2021-07-07 13:22:59,970][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1205 Loss 0.267 (AVG: 0.483) Score 92.277 (AVG: 90.952) mIOU 70.519 mAP 86.432 mAcc 77.943 IOU: 63.801 0.442 80.130 75.557 95.446 91.050 35.187 80.794 98.654 70.775 79.951 86.437 58.529 mAP: 85.419 29.643 91.137 89.073 98.941 98.556 65.597 97.199 99.835 87.460 93.533 94.320 92.902 mAcc: 81.657 3.920 88.079 82.381 96.886 97.263 37.867 96.601 99.465 82.094 87.871 95.157 64.018 [2021-07-07 13:22:59,971][root][INFO] - Finished test. Elapsed time: 34.3075 [2021-07-07 13:23:01,011][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 13:23:15,021][root][INFO] - ===> Epoch[881](4405/5): Loss 0.0247, Sem 0.0247 LR: 9.337e-02 Score 99.124 Data time: 0.2405, Forward time: 1.2576, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7615 [2021-07-07 13:23:28,618][root][INFO] - ===> Epoch[882](4410/5): Loss 0.0244, Sem 0.0244 LR: 9.336e-02 Score 99.136 Data time: 0.2493, Forward time: 1.2576, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.6568 [2021-07-07 13:23:43,220][root][INFO] - ===> Epoch[883](4415/5): Loss 0.0244, Sem 0.0244 LR: 9.335e-02 Score 99.122 Data time: 0.2505, Forward time: 1.2577, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.8682 [2021-07-07 13:23:57,140][root][INFO] - ===> Epoch[884](4420/5): Loss 0.0241, Sem 0.0241 LR: 9.335e-02 Score 99.138 Data time: 0.1890, Forward time: 1.2577, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7409 [2021-07-07 13:24:10,832][root][INFO] - ===> Epoch[885](4425/5): Loss 0.0245, Sem 0.0245 LR: 9.334e-02 Score 99.115 Data time: 0.2094, Forward time: 1.2577, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7004 [2021-07-07 13:24:24,608][root][INFO] - ===> Epoch[886](4430/5): Loss 0.0231, Sem 0.0231 LR: 9.333e-02 Score 99.175 Data time: 0.2253, Forward time: 1.2576, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7101 [2021-07-07 13:24:38,752][root][INFO] - ===> Epoch[887](4435/5): Loss 0.0236, Sem 0.0236 LR: 9.332e-02 Score 99.149 Data time: 0.2467, Forward time: 1.2576, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7779 [2021-07-07 13:24:52,684][root][INFO] - ===> Epoch[888](4440/5): Loss 0.0236, Sem 0.0236 LR: 9.331e-02 Score 99.143 Data time: 0.2735, Forward time: 1.2576, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7306 [2021-07-07 13:25:06,246][root][INFO] - ===> Epoch[889](4445/5): Loss 0.0237, Sem 0.0237 LR: 9.331e-02 Score 99.146 Data time: 0.2090, Forward time: 1.2576, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6659 [2021-07-07 13:25:20,664][root][INFO] - ===> Epoch[890](4450/5): Loss 0.0227, Sem 0.0227 LR: 9.330e-02 Score 99.184 Data time: 0.2540, Forward time: 1.2576, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.8286 [2021-07-07 13:25:35,290][root][INFO] - ===> Epoch[891](4455/5): Loss 0.0228, Sem 0.0228 LR: 9.329e-02 Score 99.175 Data time: 0.2849, Forward time: 1.2577, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.8766 [2021-07-07 13:25:49,725][root][INFO] - ===> Epoch[892](4460/5): Loss 0.0244, Sem 0.0244 LR: 9.328e-02 Score 99.118 Data time: 0.2022, Forward time: 1.2576, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8496 [2021-07-07 13:26:04,242][root][INFO] - ===> Epoch[893](4465/5): Loss 0.0239, Sem 0.0239 LR: 9.328e-02 Score 99.137 Data time: 0.2373, Forward time: 1.2576, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.8444 [2021-07-07 13:26:17,655][root][INFO] - ===> Epoch[894](4470/5): Loss 0.0231, Sem 0.0231 LR: 9.327e-02 Score 99.176 Data time: 0.2253, Forward time: 1.2576, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6389 [2021-07-07 13:26:31,264][root][INFO] - ===> Epoch[895](4475/5): Loss 0.0226, Sem 0.0226 LR: 9.326e-02 Score 99.185 Data time: 0.2205, Forward time: 1.2576, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.6603 [2021-07-07 13:26:45,000][root][INFO] - ===> Epoch[896](4480/5): Loss 0.0233, Sem 0.0233 LR: 9.325e-02 Score 99.159 Data time: 0.2194, Forward time: 1.2576, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.6967 [2021-07-07 13:26:58,702][root][INFO] - ===> Epoch[897](4485/5): Loss 0.0226, Sem 0.0226 LR: 9.325e-02 Score 99.181 Data time: 0.2396, Forward time: 1.2575, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.6914 [2021-07-07 13:27:13,338][root][INFO] - ===> Epoch[898](4490/5): Loss 0.0233, Sem 0.0233 LR: 9.324e-02 Score 99.161 Data time: 0.2676, Forward time: 1.2575, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.8721 [2021-07-07 13:27:26,898][root][INFO] - ===> Epoch[899](4495/5): Loss 0.0229, Sem 0.0229 LR: 9.323e-02 Score 99.178 Data time: 0.1991, Forward time: 1.2575, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.6723 [2021-07-07 13:27:39,944][root][INFO] - ===> Epoch[900](4500/5): Loss 0.0230, Sem 0.0230 LR: 9.322e-02 Score 99.167 Data time: 0.2344, Forward time: 1.2574, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.5623 [2021-07-07 13:27:53,265][root][INFO] - ===> Epoch[901](4505/5): Loss 0.0218, Sem 0.0218 LR: 9.322e-02 Score 99.217 Data time: 0.2765, Forward time: 1.2574, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.6099 [2021-07-07 13:28:07,343][root][INFO] - ===> Epoch[902](4510/5): Loss 0.0222, Sem 0.0222 LR: 9.321e-02 Score 99.202 Data time: 0.2115, Forward time: 1.2575, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7750 [2021-07-07 13:28:20,743][root][INFO] - ===> Epoch[903](4515/5): Loss 0.0236, Sem 0.0236 LR: 9.320e-02 Score 99.152 Data time: 0.2431, Forward time: 1.2575, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6376 [2021-07-07 13:28:34,501][root][INFO] - ===> Epoch[904](4520/5): Loss 0.0233, Sem 0.0233 LR: 9.319e-02 Score 99.157 Data time: 0.2573, Forward time: 1.2575, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6918 [2021-07-07 13:28:48,061][root][INFO] - ===> Epoch[905](4525/5): Loss 0.0224, Sem 0.0224 LR: 9.319e-02 Score 99.194 Data time: 0.2426, Forward time: 1.2575, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6670 [2021-07-07 13:29:01,817][root][INFO] - ===> Epoch[906](4530/5): Loss 0.0228, Sem 0.0228 LR: 9.318e-02 Score 99.176 Data time: 0.2485, Forward time: 1.2575, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6977 [2021-07-07 13:29:15,582][root][INFO] - ===> Epoch[907](4535/5): Loss 0.0232, Sem 0.0232 LR: 9.317e-02 Score 99.163 Data time: 0.3225, Forward time: 1.2575, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7022 [2021-07-07 13:29:29,953][root][INFO] - ===> Epoch[908](4540/5): Loss 0.0226, Sem 0.0226 LR: 9.316e-02 Score 99.195 Data time: 0.2219, Forward time: 1.2576, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.8318 [2021-07-07 13:29:43,829][root][INFO] - ===> Epoch[909](4545/5): Loss 0.0229, Sem 0.0229 LR: 9.316e-02 Score 99.174 Data time: 0.3077, Forward time: 1.2577, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7372 [2021-07-07 13:29:57,661][root][INFO] - ===> Epoch[910](4550/5): Loss 0.0225, Sem 0.0225 LR: 9.315e-02 Score 99.187 Data time: 0.2037, Forward time: 1.2578, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7161 [2021-07-07 13:30:11,390][root][INFO] - ===> Epoch[911](4555/5): Loss 0.0235, Sem 0.0235 LR: 9.314e-02 Score 99.154 Data time: 0.2190, Forward time: 1.2578, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7019 [2021-07-07 13:30:25,133][root][INFO] - ===> Epoch[912](4560/5): Loss 0.0226, Sem 0.0226 LR: 9.313e-02 Score 99.190 Data time: 0.2914, Forward time: 1.2578, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7071 [2021-07-07 13:30:38,888][root][INFO] - ===> Epoch[913](4565/5): Loss 0.0227, Sem 0.0227 LR: 9.313e-02 Score 99.181 Data time: 0.1945, Forward time: 1.2577, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7061 [2021-07-07 13:30:53,446][root][INFO] - ===> Epoch[914](4570/5): Loss 0.0231, Sem 0.0231 LR: 9.312e-02 Score 99.168 Data time: 0.2194, Forward time: 1.2577, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8605 [2021-07-07 13:31:07,862][root][INFO] - ===> Epoch[915](4575/5): Loss 0.0219, Sem 0.0219 LR: 9.311e-02 Score 99.212 Data time: 0.2343, Forward time: 1.2577, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8374 [2021-07-07 13:31:22,789][root][INFO] - ===> Epoch[916](4580/5): Loss 0.0220, Sem 0.0220 LR: 9.310e-02 Score 99.206 Data time: 0.2683, Forward time: 1.2577, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.9326 [2021-07-07 13:31:36,763][root][INFO] - ===> Epoch[917](4585/5): Loss 0.0212, Sem 0.0212 LR: 9.310e-02 Score 99.236 Data time: 0.2262, Forward time: 1.2579, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7534 [2021-07-07 13:31:50,859][root][INFO] - ===> Epoch[918](4590/5): Loss 0.0212, Sem 0.0212 LR: 9.309e-02 Score 99.232 Data time: 0.2044, Forward time: 1.2578, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7692 [2021-07-07 13:32:05,178][root][INFO] - ===> Epoch[919](4595/5): Loss 0.0224, Sem 0.0224 LR: 9.308e-02 Score 99.206 Data time: 0.2302, Forward time: 1.2579, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8136 [2021-07-07 13:32:18,984][root][INFO] - ===> Epoch[920](4600/5): Loss 0.0228, Sem 0.0228 LR: 9.307e-02 Score 99.182 Data time: 0.2777, Forward time: 1.2580, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7197 [2021-07-07 13:32:18,985][root][INFO] - ===> Start testing [2021-07-07 13:32:54,641][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1267 Loss 0.382 (AVG: 0.479) Score 90.404 (AVG: 91.294) mIOU 70.766 mAP 84.756 mAcc 77.033 IOU: 65.350 0.066 77.418 78.447 95.342 91.677 28.148 81.970 98.680 78.874 80.232 86.384 57.372 mAP: 85.650 10.278 93.360 91.070 98.872 98.437 57.279 96.832 99.840 89.699 93.383 94.920 92.204 mAcc: 78.044 0.456 89.297 87.011 96.961 95.868 29.878 92.114 99.499 82.547 89.471 96.590 63.698 [2021-07-07 13:32:54,641][root][INFO] - Finished test. Elapsed time: 35.6549 [2021-07-07 13:32:55,731][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 13:33:08,438][root][INFO] - ===> Epoch[921](4605/5): Loss 0.0249, Sem 0.0249 LR: 9.307e-02 Score 99.103 Data time: 0.1818, Forward time: 1.2579, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.5081 [2021-07-07 13:33:22,190][root][INFO] - ===> Epoch[922](4610/5): Loss 0.0232, Sem 0.0232 LR: 9.306e-02 Score 99.178 Data time: 0.2202, Forward time: 1.2580, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7127 [2021-07-07 13:33:35,419][root][INFO] - ===> Epoch[923](4615/5): Loss 0.0232, Sem 0.0232 LR: 9.305e-02 Score 99.162 Data time: 0.1800, Forward time: 1.2579, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.5994 [2021-07-07 13:33:48,820][root][INFO] - ===> Epoch[924](4620/5): Loss 0.0230, Sem 0.0230 LR: 9.304e-02 Score 99.171 Data time: 0.2210, Forward time: 1.2579, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6376 [2021-07-07 13:34:03,089][root][INFO] - ===> Epoch[925](4625/5): Loss 0.0226, Sem 0.0226 LR: 9.304e-02 Score 99.187 Data time: 0.1680, Forward time: 1.2580, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8035 [2021-07-07 13:34:17,740][root][INFO] - ===> Epoch[926](4630/5): Loss 0.0226, Sem 0.0226 LR: 9.303e-02 Score 99.190 Data time: 0.1805, Forward time: 1.2579, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8812 [2021-07-07 13:34:30,999][root][INFO] - ===> Epoch[927](4635/5): Loss 0.0222, Sem 0.0222 LR: 9.302e-02 Score 99.198 Data time: 0.2550, Forward time: 1.2579, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.6024 [2021-07-07 13:34:46,541][root][INFO] - ===> Epoch[928](4640/5): Loss 0.0235, Sem 0.0235 LR: 9.301e-02 Score 99.144 Data time: 0.1893, Forward time: 1.2579, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 3.0556 [2021-07-07 13:35:00,711][root][INFO] - ===> Epoch[929](4645/5): Loss 0.0223, Sem 0.0223 LR: 9.300e-02 Score 99.203 Data time: 0.2443, Forward time: 1.2580, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7920 [2021-07-07 13:35:14,423][root][INFO] - ===> Epoch[930](4650/5): Loss 0.0222, Sem 0.0222 LR: 9.300e-02 Score 99.198 Data time: 0.1604, Forward time: 1.2579, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6954 [2021-07-07 13:35:29,412][root][INFO] - ===> Epoch[931](4655/5): Loss 0.0219, Sem 0.0219 LR: 9.299e-02 Score 99.214 Data time: 0.2359, Forward time: 1.2580, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.9531 [2021-07-07 13:35:43,662][root][INFO] - ===> Epoch[932](4660/5): Loss 0.0230, Sem 0.0230 LR: 9.298e-02 Score 99.170 Data time: 0.2203, Forward time: 1.2579, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8101 [2021-07-07 13:35:58,193][root][INFO] - ===> Epoch[933](4665/5): Loss 0.0224, Sem 0.0224 LR: 9.297e-02 Score 99.192 Data time: 0.2596, Forward time: 1.2580, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8447 [2021-07-07 13:36:11,472][root][INFO] - ===> Epoch[934](4670/5): Loss 0.0229, Sem 0.0229 LR: 9.297e-02 Score 99.174 Data time: 0.2040, Forward time: 1.2580, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.6100 [2021-07-07 13:36:25,513][root][INFO] - ===> Epoch[935](4675/5): Loss 0.0236, Sem 0.0236 LR: 9.296e-02 Score 99.148 Data time: 0.1846, Forward time: 1.2580, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7650 [2021-07-07 13:36:39,614][root][INFO] - ===> Epoch[936](4680/5): Loss 0.0224, Sem 0.0224 LR: 9.295e-02 Score 99.191 Data time: 0.2262, Forward time: 1.2579, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7657 [2021-07-07 13:36:53,899][root][INFO] - ===> Epoch[937](4685/5): Loss 0.0226, Sem 0.0226 LR: 9.294e-02 Score 99.196 Data time: 0.2086, Forward time: 1.2580, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8045 [2021-07-07 13:37:08,260][root][INFO] - ===> Epoch[938](4690/5): Loss 0.0219, Sem 0.0219 LR: 9.294e-02 Score 99.215 Data time: 0.1592, Forward time: 1.2580, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8307 [2021-07-07 13:37:22,253][root][INFO] - ===> Epoch[939](4695/5): Loss 0.0233, Sem 0.0233 LR: 9.293e-02 Score 99.167 Data time: 0.2721, Forward time: 1.2579, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7530 [2021-07-07 13:37:36,002][root][INFO] - ===> Epoch[940](4700/5): Loss 0.0232, Sem 0.0232 LR: 9.292e-02 Score 99.168 Data time: 0.1710, Forward time: 1.2577, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7156 [2021-07-07 13:37:50,443][root][INFO] - ===> Epoch[941](4705/5): Loss 0.0224, Sem 0.0224 LR: 9.291e-02 Score 99.189 Data time: 0.1684, Forward time: 1.2576, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8421 [2021-07-07 13:38:04,043][root][INFO] - ===> Epoch[942](4710/5): Loss 0.0219, Sem 0.0219 LR: 9.291e-02 Score 99.206 Data time: 0.2454, Forward time: 1.2576, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6759 [2021-07-07 13:38:18,100][root][INFO] - ===> Epoch[943](4715/5): Loss 0.0223, Sem 0.0223 LR: 9.290e-02 Score 99.197 Data time: 0.2505, Forward time: 1.2575, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7539 [2021-07-07 13:38:32,305][root][INFO] - ===> Epoch[944](4720/5): Loss 0.0215, Sem 0.0215 LR: 9.289e-02 Score 99.220 Data time: 0.2496, Forward time: 1.2575, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7906 [2021-07-07 13:38:46,866][root][INFO] - ===> Epoch[945](4725/5): Loss 0.0214, Sem 0.0214 LR: 9.288e-02 Score 99.228 Data time: 0.1873, Forward time: 1.2576, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8749 [2021-07-07 13:39:00,859][root][INFO] - ===> Epoch[946](4730/5): Loss 0.0227, Sem 0.0227 LR: 9.288e-02 Score 99.181 Data time: 0.2512, Forward time: 1.2577, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7425 [2021-07-07 13:39:14,835][root][INFO] - ===> Epoch[947](4735/5): Loss 0.0221, Sem 0.0221 LR: 9.287e-02 Score 99.209 Data time: 0.2209, Forward time: 1.2578, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7570 [2021-07-07 13:39:28,671][root][INFO] - ===> Epoch[948](4740/5): Loss 0.0223, Sem 0.0223 LR: 9.286e-02 Score 99.201 Data time: 0.2159, Forward time: 1.2578, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7276 [2021-07-07 13:39:42,571][root][INFO] - ===> Epoch[949](4745/5): Loss 0.0219, Sem 0.0219 LR: 9.285e-02 Score 99.210 Data time: 0.2769, Forward time: 1.2578, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7257 [2021-07-07 13:39:56,210][root][INFO] - ===> Epoch[950](4750/5): Loss 0.0219, Sem 0.0219 LR: 9.285e-02 Score 99.208 Data time: 0.2844, Forward time: 1.2578, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6804 [2021-07-07 13:40:10,261][root][INFO] - ===> Epoch[951](4755/5): Loss 0.0222, Sem 0.0222 LR: 9.284e-02 Score 99.206 Data time: 0.2865, Forward time: 1.2578, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7551 [2021-07-07 13:40:24,858][root][INFO] - ===> Epoch[952](4760/5): Loss 0.0222, Sem 0.0222 LR: 9.283e-02 Score 99.220 Data time: 0.2357, Forward time: 1.2578, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8706 [2021-07-07 13:40:39,490][root][INFO] - ===> Epoch[953](4765/5): Loss 0.0230, Sem 0.0230 LR: 9.282e-02 Score 99.180 Data time: 0.2496, Forward time: 1.2579, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8782 [2021-07-07 13:40:53,402][root][INFO] - ===> Epoch[954](4770/5): Loss 0.0215, Sem 0.0215 LR: 9.282e-02 Score 99.225 Data time: 0.2292, Forward time: 1.2580, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7306 [2021-07-07 13:41:07,404][root][INFO] - ===> Epoch[955](4775/5): Loss 0.0219, Sem 0.0219 LR: 9.281e-02 Score 99.212 Data time: 0.2115, Forward time: 1.2581, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7551 [2021-07-07 13:41:20,980][root][INFO] - ===> Epoch[956](4780/5): Loss 0.0218, Sem 0.0218 LR: 9.280e-02 Score 99.217 Data time: 0.2042, Forward time: 1.2581, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6728 [2021-07-07 13:41:34,364][root][INFO] - ===> Epoch[957](4785/5): Loss 0.0226, Sem 0.0226 LR: 9.279e-02 Score 99.189 Data time: 0.2231, Forward time: 1.2580, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6281 [2021-07-07 13:41:48,970][root][INFO] - ===> Epoch[958](4790/5): Loss 0.0212, Sem 0.0212 LR: 9.279e-02 Score 99.238 Data time: 0.2530, Forward time: 1.2582, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8713 [2021-07-07 13:42:02,154][root][INFO] - ===> Epoch[959](4795/5): Loss 0.0219, Sem 0.0219 LR: 9.278e-02 Score 99.200 Data time: 0.2106, Forward time: 1.2581, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.5864 [2021-07-07 13:42:15,899][root][INFO] - ===> Epoch[960](4800/5): Loss 0.0218, Sem 0.0218 LR: 9.277e-02 Score 99.214 Data time: 0.1922, Forward time: 1.2582, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7027 [2021-07-07 13:42:15,900][root][INFO] - ===> Start testing [2021-07-07 13:42:50,058][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1200 Loss 0.258 (AVG: 0.494) Score 93.379 (AVG: 91.113) mIOU 70.542 mAP 85.062 mAcc 76.837 IOU: 64.418 0.049 79.671 77.530 95.253 90.715 25.447 83.519 98.523 79.026 80.155 86.239 56.500 mAP: 85.269 10.871 94.672 90.237 98.990 98.305 56.518 97.678 99.845 92.526 93.302 95.227 92.371 mAcc: 80.110 0.273 90.336 84.773 96.937 97.165 26.501 94.045 99.199 84.195 87.494 96.746 61.101 [2021-07-07 13:42:50,058][root][INFO] - Finished test. Elapsed time: 34.1572 [2021-07-07 13:42:51,211][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 13:43:04,865][root][INFO] - ===> Epoch[961](4805/5): Loss 0.0219, Sem 0.0219 LR: 9.276e-02 Score 99.205 Data time: 0.2047, Forward time: 1.2583, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6921 [2021-07-07 13:43:18,473][root][INFO] - ===> Epoch[962](4810/5): Loss 0.0212, Sem 0.0212 LR: 9.276e-02 Score 99.237 Data time: 0.2340, Forward time: 1.2583, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6698 [2021-07-07 13:43:33,063][root][INFO] - ===> Epoch[963](4815/5): Loss 0.0215, Sem 0.0215 LR: 9.275e-02 Score 99.223 Data time: 0.2580, Forward time: 1.2582, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8728 [2021-07-07 13:43:46,789][root][INFO] - ===> Epoch[964](4820/5): Loss 0.0213, Sem 0.0213 LR: 9.274e-02 Score 99.233 Data time: 0.2224, Forward time: 1.2582, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6958 [2021-07-07 13:44:01,294][root][INFO] - ===> Epoch[965](4825/5): Loss 0.0217, Sem 0.0217 LR: 9.273e-02 Score 99.218 Data time: 0.2978, Forward time: 1.2582, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8616 [2021-07-07 13:44:15,289][root][INFO] - ===> Epoch[966](4830/5): Loss 0.0211, Sem 0.0211 LR: 9.273e-02 Score 99.234 Data time: 0.2344, Forward time: 1.2581, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7532 [2021-07-07 13:44:29,380][root][INFO] - ===> Epoch[967](4835/5): Loss 0.0209, Sem 0.0209 LR: 9.272e-02 Score 99.241 Data time: 0.2577, Forward time: 1.2581, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7732 [2021-07-07 13:44:43,203][root][INFO] - ===> Epoch[968](4840/5): Loss 0.0216, Sem 0.0216 LR: 9.271e-02 Score 99.222 Data time: 0.2610, Forward time: 1.2582, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7129 [2021-07-07 13:44:57,089][root][INFO] - ===> Epoch[969](4845/5): Loss 0.0212, Sem 0.0212 LR: 9.270e-02 Score 99.235 Data time: 0.2540, Forward time: 1.2581, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7282 [2021-07-07 13:45:11,665][root][INFO] - ===> Epoch[970](4850/5): Loss 0.0217, Sem 0.0217 LR: 9.269e-02 Score 99.218 Data time: 0.2363, Forward time: 1.2582, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8699 [2021-07-07 13:45:26,050][root][INFO] - ===> Epoch[971](4855/5): Loss 0.0217, Sem 0.0217 LR: 9.269e-02 Score 99.213 Data time: 0.1867, Forward time: 1.2581, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8415 [2021-07-07 13:45:39,503][root][INFO] - ===> Epoch[972](4860/5): Loss 0.0209, Sem 0.0209 LR: 9.268e-02 Score 99.247 Data time: 0.2269, Forward time: 1.2581, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6298 [2021-07-07 13:45:53,631][root][INFO] - ===> Epoch[973](4865/5): Loss 0.0204, Sem 0.0204 LR: 9.267e-02 Score 99.265 Data time: 0.2327, Forward time: 1.2579, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7788 [2021-07-07 13:46:07,325][root][INFO] - ===> Epoch[974](4870/5): Loss 0.0226, Sem 0.0226 LR: 9.266e-02 Score 99.193 Data time: 0.2527, Forward time: 1.2581, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6859 [2021-07-07 13:46:20,722][root][INFO] - ===> Epoch[975](4875/5): Loss 0.0215, Sem 0.0215 LR: 9.266e-02 Score 99.220 Data time: 0.1784, Forward time: 1.2582, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6401 [2021-07-07 13:46:35,642][root][INFO] - ===> Epoch[976](4880/5): Loss 0.0211, Sem 0.0211 LR: 9.265e-02 Score 99.235 Data time: 0.2638, Forward time: 1.2583, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.9398 [2021-07-07 13:46:49,345][root][INFO] - ===> Epoch[977](4885/5): Loss 0.0209, Sem 0.0209 LR: 9.264e-02 Score 99.239 Data time: 0.1755, Forward time: 1.2582, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6921 [2021-07-07 13:47:02,933][root][INFO] - ===> Epoch[978](4890/5): Loss 0.0209, Sem 0.0209 LR: 9.263e-02 Score 99.238 Data time: 0.1846, Forward time: 1.2583, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6755 [2021-07-07 13:47:17,120][root][INFO] - ===> Epoch[979](4895/5): Loss 0.0224, Sem 0.0224 LR: 9.263e-02 Score 99.192 Data time: 0.2419, Forward time: 1.2583, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7878 [2021-07-07 13:47:30,671][root][INFO] - ===> Epoch[980](4900/5): Loss 0.0211, Sem 0.0211 LR: 9.262e-02 Score 99.250 Data time: 0.1780, Forward time: 1.2583, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6659 [2021-07-07 13:47:45,366][root][INFO] - ===> Epoch[981](4905/5): Loss 0.0217, Sem 0.0217 LR: 9.261e-02 Score 99.217 Data time: 0.2465, Forward time: 1.2583, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8810 [2021-07-07 13:47:59,123][root][INFO] - ===> Epoch[982](4910/5): Loss 0.0210, Sem 0.0210 LR: 9.260e-02 Score 99.241 Data time: 0.2341, Forward time: 1.2583, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7078 [2021-07-07 13:48:12,927][root][INFO] - ===> Epoch[983](4915/5): Loss 0.0221, Sem 0.0221 LR: 9.260e-02 Score 99.202 Data time: 0.1785, Forward time: 1.2582, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7070 [2021-07-07 13:48:27,199][root][INFO] - ===> Epoch[984](4920/5): Loss 0.0224, Sem 0.0224 LR: 9.259e-02 Score 99.215 Data time: 0.3298, Forward time: 1.2582, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8029 [2021-07-07 13:48:41,810][root][INFO] - ===> Epoch[985](4925/5): Loss 0.0220, Sem 0.0220 LR: 9.258e-02 Score 99.210 Data time: 0.2129, Forward time: 1.2581, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.8842 [2021-07-07 13:48:55,304][root][INFO] - ===> Epoch[986](4930/5): Loss 0.0230, Sem 0.0230 LR: 9.257e-02 Score 99.170 Data time: 0.2495, Forward time: 1.2580, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.6431 [2021-07-07 13:49:09,699][root][INFO] - ===> Epoch[987](4935/5): Loss 0.0222, Sem 0.0222 LR: 9.257e-02 Score 99.200 Data time: 0.2183, Forward time: 1.2579, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8209 [2021-07-07 13:49:24,325][root][INFO] - ===> Epoch[988](4940/5): Loss 0.0221, Sem 0.0221 LR: 9.256e-02 Score 99.197 Data time: 0.2874, Forward time: 1.2578, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8816 [2021-07-07 13:49:38,250][root][INFO] - ===> Epoch[989](4945/5): Loss 0.0230, Sem 0.0230 LR: 9.255e-02 Score 99.154 Data time: 0.2797, Forward time: 1.2578, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7275 [2021-07-07 13:49:52,607][root][INFO] - ===> Epoch[990](4950/5): Loss 0.0214, Sem 0.0214 LR: 9.254e-02 Score 99.229 Data time: 0.2420, Forward time: 1.2579, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8256 [2021-07-07 13:50:05,994][root][INFO] - ===> Epoch[991](4955/5): Loss 0.0237, Sem 0.0237 LR: 9.254e-02 Score 99.163 Data time: 0.2273, Forward time: 1.2579, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6270 [2021-07-07 13:50:20,851][root][INFO] - ===> Epoch[992](4960/5): Loss 0.0221, Sem 0.0221 LR: 9.253e-02 Score 99.194 Data time: 0.2496, Forward time: 1.2579, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.9149 [2021-07-07 13:50:34,594][root][INFO] - ===> Epoch[993](4965/5): Loss 0.0239, Sem 0.0239 LR: 9.252e-02 Score 99.142 Data time: 0.2204, Forward time: 1.2580, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7007 [2021-07-07 13:50:48,179][root][INFO] - ===> Epoch[994](4970/5): Loss 0.0221, Sem 0.0221 LR: 9.251e-02 Score 99.204 Data time: 0.2259, Forward time: 1.2581, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6703 [2021-07-07 13:51:01,648][root][INFO] - ===> Epoch[995](4975/5): Loss 0.0230, Sem 0.0230 LR: 9.251e-02 Score 99.176 Data time: 0.2358, Forward time: 1.2581, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6422 [2021-07-07 13:51:15,404][root][INFO] - ===> Epoch[996](4980/5): Loss 0.0224, Sem 0.0224 LR: 9.250e-02 Score 99.197 Data time: 0.2427, Forward time: 1.2582, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.6901 [2021-07-07 13:51:30,575][root][INFO] - ===> Epoch[997](4985/5): Loss 0.0215, Sem 0.0215 LR: 9.249e-02 Score 99.227 Data time: 0.2138, Forward time: 1.2583, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.9799 [2021-07-07 13:51:44,521][root][INFO] - ===> Epoch[998](4990/5): Loss 0.0221, Sem 0.0221 LR: 9.248e-02 Score 99.202 Data time: 0.2174, Forward time: 1.2584, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7497 [2021-07-07 13:51:58,851][root][INFO] - ===> Epoch[999](4995/5): Loss 0.0215, Sem 0.0215 LR: 9.248e-02 Score 99.225 Data time: 0.2029, Forward time: 1.2584, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8208 [2021-07-07 13:52:12,672][root][INFO] - ===> Epoch[1000](5000/5): Loss 0.0226, Sem 0.0226 LR: 9.247e-02 Score 99.189 Data time: 0.2784, Forward time: 1.2583, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7199 [2021-07-07 13:52:12,674][root][INFO] - ===> Start testing [2021-07-07 13:52:47,763][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1214 Loss 0.354 (AVG: 0.472) Score 92.222 (AVG: 91.284) mIOU 71.491 mAP 85.178 mAcc 77.738 IOU: 64.261 0.107 81.595 77.902 94.871 91.579 33.005 81.620 98.605 79.812 79.812 86.978 59.239 mAP: 85.595 12.688 94.149 90.529 98.902 98.549 61.664 97.512 99.813 86.376 92.574 95.603 93.355 mAcc: 78.738 0.820 89.888 84.888 96.650 96.254 35.684 94.043 99.317 81.946 90.353 96.509 65.498 [2021-07-07 13:52:47,763][root][INFO] - Finished test. Elapsed time: 35.0885 [2021-07-07 13:52:48,921][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 13:53:01,519][root][INFO] - ===> Epoch[1001](5005/5): Loss 0.0211, Sem 0.0211 LR: 9.246e-02 Score 99.232 Data time: 0.1902, Forward time: 1.2582, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.4856 [2021-07-07 13:53:15,755][root][INFO] - ===> Epoch[1002](5010/5): Loss 0.0206, Sem 0.0206 LR: 9.245e-02 Score 99.264 Data time: 0.2821, Forward time: 1.2583, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7973 [2021-07-07 13:53:29,057][root][INFO] - ===> Epoch[1003](5015/5): Loss 0.0217, Sem 0.0217 LR: 9.245e-02 Score 99.211 Data time: 0.2540, Forward time: 1.2583, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6069 [2021-07-07 13:53:43,107][root][INFO] - ===> Epoch[1004](5020/5): Loss 0.0211, Sem 0.0211 LR: 9.244e-02 Score 99.237 Data time: 0.2407, Forward time: 1.2582, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7649 [2021-07-07 13:53:57,307][root][INFO] - ===> Epoch[1005](5025/5): Loss 0.0202, Sem 0.0202 LR: 9.243e-02 Score 99.265 Data time: 0.1981, Forward time: 1.2583, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7940 [2021-07-07 13:54:11,689][root][INFO] - ===> Epoch[1006](5030/5): Loss 0.0222, Sem 0.0222 LR: 9.242e-02 Score 99.202 Data time: 0.2116, Forward time: 1.2583, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8307 [2021-07-07 13:54:25,166][root][INFO] - ===> Epoch[1007](5035/5): Loss 0.0211, Sem 0.0211 LR: 9.241e-02 Score 99.235 Data time: 0.2019, Forward time: 1.2583, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6469 [2021-07-07 13:54:39,216][root][INFO] - ===> Epoch[1008](5040/5): Loss 0.0213, Sem 0.0213 LR: 9.241e-02 Score 99.227 Data time: 0.2468, Forward time: 1.2584, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7636 [2021-07-07 13:54:52,378][root][INFO] - ===> Epoch[1009](5045/5): Loss 0.0215, Sem 0.0215 LR: 9.240e-02 Score 99.225 Data time: 0.2117, Forward time: 1.2584, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.5874 [2021-07-07 13:55:06,439][root][INFO] - ===> Epoch[1010](5050/5): Loss 0.0209, Sem 0.0209 LR: 9.239e-02 Score 99.242 Data time: 0.2517, Forward time: 1.2586, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7652 [2021-07-07 13:55:21,209][root][INFO] - ===> Epoch[1011](5055/5): Loss 0.0213, Sem 0.0213 LR: 9.238e-02 Score 99.245 Data time: 0.2485, Forward time: 1.2586, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.9105 [2021-07-07 13:55:34,961][root][INFO] - ===> Epoch[1012](5060/5): Loss 0.0200, Sem 0.0200 LR: 9.238e-02 Score 99.279 Data time: 0.2815, Forward time: 1.2586, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7070 [2021-07-07 13:55:49,184][root][INFO] - ===> Epoch[1013](5065/5): Loss 0.0205, Sem 0.0205 LR: 9.237e-02 Score 99.258 Data time: 0.2829, Forward time: 1.2586, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7809 [2021-07-07 13:56:03,065][root][INFO] - ===> Epoch[1014](5070/5): Loss 0.0209, Sem 0.0209 LR: 9.236e-02 Score 99.253 Data time: 0.1957, Forward time: 1.2585, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7248 [2021-07-07 13:56:17,189][root][INFO] - ===> Epoch[1015](5075/5): Loss 0.0209, Sem 0.0209 LR: 9.235e-02 Score 99.245 Data time: 0.2511, Forward time: 1.2586, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7832 [2021-07-07 13:56:31,954][root][INFO] - ===> Epoch[1016](5080/5): Loss 0.0203, Sem 0.0203 LR: 9.235e-02 Score 99.266 Data time: 0.2870, Forward time: 1.2586, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8979 [2021-07-07 13:56:46,307][root][INFO] - ===> Epoch[1017](5085/5): Loss 0.0203, Sem 0.0203 LR: 9.234e-02 Score 99.271 Data time: 0.2735, Forward time: 1.2586, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8117 [2021-07-07 13:57:00,084][root][INFO] - ===> Epoch[1018](5090/5): Loss 0.0213, Sem 0.0213 LR: 9.233e-02 Score 99.232 Data time: 0.2498, Forward time: 1.2587, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7025 [2021-07-07 13:57:14,442][root][INFO] - ===> Epoch[1019](5095/5): Loss 0.0200, Sem 0.0200 LR: 9.232e-02 Score 99.279 Data time: 0.2411, Forward time: 1.2588, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.8199 [2021-07-07 13:57:29,143][root][INFO] - ===> Epoch[1020](5100/5): Loss 0.0207, Sem 0.0207 LR: 9.232e-02 Score 99.254 Data time: 0.2671, Forward time: 1.2587, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8877 [2021-07-07 13:57:43,335][root][INFO] - ===> Epoch[1021](5105/5): Loss 0.0222, Sem 0.0222 LR: 9.231e-02 Score 99.208 Data time: 0.2457, Forward time: 1.2587, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7929 [2021-07-07 13:57:57,825][root][INFO] - ===> Epoch[1022](5110/5): Loss 0.0215, Sem 0.0215 LR: 9.230e-02 Score 99.224 Data time: 0.2656, Forward time: 1.2588, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8390 [2021-07-07 13:58:11,804][root][INFO] - ===> Epoch[1023](5115/5): Loss 0.0208, Sem 0.0208 LR: 9.229e-02 Score 99.248 Data time: 0.2872, Forward time: 1.2588, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7569 [2021-07-07 13:58:26,607][root][INFO] - ===> Epoch[1024](5120/5): Loss 0.0220, Sem 0.0220 LR: 9.229e-02 Score 99.212 Data time: 0.2037, Forward time: 1.2589, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.9059 [2021-07-07 13:58:40,285][root][INFO] - ===> Epoch[1025](5125/5): Loss 0.0214, Sem 0.0214 LR: 9.228e-02 Score 99.231 Data time: 0.2254, Forward time: 1.2588, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.6951 [2021-07-07 13:58:53,800][root][INFO] - ===> Epoch[1026](5130/5): Loss 0.0218, Sem 0.0218 LR: 9.227e-02 Score 99.211 Data time: 0.2235, Forward time: 1.2588, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6415 [2021-07-07 13:59:07,355][root][INFO] - ===> Epoch[1027](5135/5): Loss 0.0220, Sem 0.0220 LR: 9.226e-02 Score 99.214 Data time: 0.1582, Forward time: 1.2588, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6745 [2021-07-07 13:59:21,571][root][INFO] - ===> Epoch[1028](5140/5): Loss 0.0213, Sem 0.0213 LR: 9.226e-02 Score 99.237 Data time: 0.2359, Forward time: 1.2588, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7901 [2021-07-07 13:59:35,709][root][INFO] - ===> Epoch[1029](5145/5): Loss 0.0223, Sem 0.0223 LR: 9.225e-02 Score 99.190 Data time: 0.2447, Forward time: 1.2588, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7774 [2021-07-07 13:59:49,777][root][INFO] - ===> Epoch[1030](5150/5): Loss 0.0212, Sem 0.0212 LR: 9.224e-02 Score 99.229 Data time: 0.2386, Forward time: 1.2589, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.7715 [2021-07-07 14:00:03,115][root][INFO] - ===> Epoch[1031](5155/5): Loss 0.0226, Sem 0.0226 LR: 9.223e-02 Score 99.190 Data time: 0.2250, Forward time: 1.2589, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.6299 [2021-07-07 14:00:19,039][root][INFO] - ===> Epoch[1032](5160/5): Loss 0.0219, Sem 0.0219 LR: 9.223e-02 Score 99.209 Data time: 0.2981, Forward time: 1.2591, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 3.1333 [2021-07-07 14:00:32,562][root][INFO] - ===> Epoch[1033](5165/5): Loss 0.0224, Sem 0.0224 LR: 9.222e-02 Score 99.198 Data time: 0.2386, Forward time: 1.2591, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.6507 [2021-07-07 14:00:46,407][root][INFO] - ===> Epoch[1034](5170/5): Loss 0.0251, Sem 0.0251 LR: 9.221e-02 Score 99.105 Data time: 0.2289, Forward time: 1.2591, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.7100 [2021-07-07 14:01:00,537][root][INFO] - ===> Epoch[1035](5175/5): Loss 0.0231, Sem 0.0231 LR: 9.220e-02 Score 99.163 Data time: 0.2303, Forward time: 1.2592, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.7767 [2021-07-07 14:01:14,715][root][INFO] - ===> Epoch[1036](5180/5): Loss 0.0242, Sem 0.0242 LR: 9.220e-02 Score 99.143 Data time: 0.2217, Forward time: 1.2592, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.7838 [2021-07-07 14:01:28,217][root][INFO] - ===> Epoch[1037](5185/5): Loss 0.0262, Sem 0.0262 LR: 9.219e-02 Score 99.052 Data time: 0.2441, Forward time: 1.2593, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.6523 [2021-07-07 14:01:42,049][root][INFO] - ===> Epoch[1038](5190/5): Loss 0.0237, Sem 0.0237 LR: 9.218e-02 Score 99.157 Data time: 0.2221, Forward time: 1.2594, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.7127 [2021-07-07 14:01:55,526][root][INFO] - ===> Epoch[1039](5195/5): Loss 0.0234, Sem 0.0234 LR: 9.217e-02 Score 99.155 Data time: 0.2151, Forward time: 1.2592, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.6556 [2021-07-07 14:02:09,034][root][INFO] - ===> Epoch[1040](5200/5): Loss 0.0236, Sem 0.0236 LR: 9.217e-02 Score 99.148 Data time: 0.2384, Forward time: 1.2593, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.6490 [2021-07-07 14:02:09,036][root][INFO] - ===> Start testing [2021-07-07 14:02:45,255][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1204 Loss 0.246 (AVG: 0.512) Score 94.349 (AVG: 90.930) mIOU 69.922 mAP 85.800 mAcc 77.134 IOU: 64.345 0.693 76.317 77.081 94.503 91.582 23.354 79.684 98.709 78.412 79.432 86.371 58.497 mAP: 84.857 30.845 92.537 89.339 98.783 98.391 54.755 97.407 99.803 88.505 93.245 94.936 92.003 mAcc: 78.133 5.196 89.580 84.876 96.460 95.508 24.892 94.462 99.602 84.461 89.479 96.458 63.628 [2021-07-07 14:02:45,255][root][INFO] - Finished test. Elapsed time: 36.2193 [2021-07-07 14:02:46,265][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 14:02:59,137][root][INFO] - ===> Epoch[1041](5205/5): Loss 0.0230, Sem 0.0230 LR: 9.216e-02 Score 99.166 Data time: 0.1982, Forward time: 1.2592, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.5373 [2021-07-07 14:03:13,129][root][INFO] - ===> Epoch[1042](5210/5): Loss 0.0229, Sem 0.0229 LR: 9.215e-02 Score 99.170 Data time: 0.2400, Forward time: 1.2592, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.7397 [2021-07-07 14:03:26,552][root][INFO] - ===> Epoch[1043](5215/5): Loss 0.0223, Sem 0.0223 LR: 9.214e-02 Score 99.200 Data time: 0.2012, Forward time: 1.2592, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.6409 [2021-07-07 14:03:40,712][root][INFO] - ===> Epoch[1044](5220/5): Loss 0.0228, Sem 0.0228 LR: 9.213e-02 Score 99.186 Data time: 0.2596, Forward time: 1.2593, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.7704 [2021-07-07 14:03:55,144][root][INFO] - ===> Epoch[1045](5225/5): Loss 0.0216, Sem 0.0216 LR: 9.213e-02 Score 99.234 Data time: 0.1756, Forward time: 1.2593, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.8377 [2021-07-07 14:04:10,560][root][INFO] - ===> Epoch[1046](5230/5): Loss 0.0225, Sem 0.0225 LR: 9.212e-02 Score 99.188 Data time: 0.2456, Forward time: 1.2594, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 3.0246 [2021-07-07 14:04:25,140][root][INFO] - ===> Epoch[1047](5235/5): Loss 0.0226, Sem 0.0226 LR: 9.211e-02 Score 99.183 Data time: 0.2581, Forward time: 1.2594, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.8649 [2021-07-07 14:04:39,615][root][INFO] - ===> Epoch[1048](5240/5): Loss 0.0228, Sem 0.0228 LR: 9.210e-02 Score 99.179 Data time: 0.1801, Forward time: 1.2593, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.8595 [2021-07-07 14:04:52,895][root][INFO] - ===> Epoch[1049](5245/5): Loss 0.0221, Sem 0.0221 LR: 9.210e-02 Score 99.203 Data time: 0.2302, Forward time: 1.2593, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.5985 [2021-07-07 14:05:07,553][root][INFO] - ===> Epoch[1050](5250/5): Loss 0.0224, Sem 0.0224 LR: 9.209e-02 Score 99.203 Data time: 0.2271, Forward time: 1.2594, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.8814 [2021-07-07 14:05:20,672][root][INFO] - ===> Epoch[1051](5255/5): Loss 0.0219, Sem 0.0219 LR: 9.208e-02 Score 99.211 Data time: 0.2180, Forward time: 1.2593, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.5798 [2021-07-07 14:05:34,894][root][INFO] - ===> Epoch[1052](5260/5): Loss 0.0224, Sem 0.0224 LR: 9.207e-02 Score 99.196 Data time: 0.2177, Forward time: 1.2593, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.7838 [2021-07-07 14:05:47,873][root][INFO] - ===> Epoch[1053](5265/5): Loss 0.0227, Sem 0.0227 LR: 9.207e-02 Score 99.181 Data time: 0.2021, Forward time: 1.2593, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.5408 [2021-07-07 14:06:01,829][root][INFO] - ===> Epoch[1054](5270/5): Loss 0.0218, Sem 0.0218 LR: 9.206e-02 Score 99.220 Data time: 0.2685, Forward time: 1.2593, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.7450 [2021-07-07 14:06:16,700][root][INFO] - ===> Epoch[1055](5275/5): Loss 0.0220, Sem 0.0220 LR: 9.205e-02 Score 99.203 Data time: 0.2833, Forward time: 1.2593, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.9205 [2021-07-07 14:06:30,329][root][INFO] - ===> Epoch[1056](5280/5): Loss 0.0216, Sem 0.0216 LR: 9.204e-02 Score 99.221 Data time: 0.2077, Forward time: 1.2594, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.6890 [2021-07-07 14:06:44,876][root][INFO] - ===> Epoch[1057](5285/5): Loss 0.0223, Sem 0.0223 LR: 9.204e-02 Score 99.201 Data time: 0.2872, Forward time: 1.2594, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.8610 [2021-07-07 14:06:58,992][root][INFO] - ===> Epoch[1058](5290/5): Loss 0.0228, Sem 0.0228 LR: 9.203e-02 Score 99.183 Data time: 0.2453, Forward time: 1.2593, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.7796 [2021-07-07 14:07:12,504][root][INFO] - ===> Epoch[1059](5295/5): Loss 0.0209, Sem 0.0209 LR: 9.202e-02 Score 99.251 Data time: 0.2009, Forward time: 1.2592, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.6477 [2021-07-07 14:07:26,727][root][INFO] - ===> Epoch[1060](5300/5): Loss 0.0209, Sem 0.0209 LR: 9.201e-02 Score 99.248 Data time: 0.2464, Forward time: 1.2592, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.8010 [2021-07-07 14:07:40,722][root][INFO] - ===> Epoch[1061](5305/5): Loss 0.0213, Sem 0.0213 LR: 9.201e-02 Score 99.224 Data time: 0.2288, Forward time: 1.2591, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.7472 [2021-07-07 14:07:54,484][root][INFO] - ===> Epoch[1062](5310/5): Loss 0.0206, Sem 0.0206 LR: 9.200e-02 Score 99.257 Data time: 0.2125, Forward time: 1.2590, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.7007 [2021-07-07 14:08:08,823][root][INFO] - ===> Epoch[1063](5315/5): Loss 0.0203, Sem 0.0203 LR: 9.199e-02 Score 99.262 Data time: 0.2582, Forward time: 1.2591, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.8193 [2021-07-07 14:08:22,790][root][INFO] - ===> Epoch[1064](5320/5): Loss 0.0213, Sem 0.0213 LR: 9.198e-02 Score 99.235 Data time: 0.2559, Forward time: 1.2591, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.7391 [2021-07-07 14:08:37,024][root][INFO] - ===> Epoch[1065](5325/5): Loss 0.0212, Sem 0.0212 LR: 9.198e-02 Score 99.226 Data time: 0.2148, Forward time: 1.2591, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.8016 [2021-07-07 14:08:50,758][root][INFO] - ===> Epoch[1066](5330/5): Loss 0.0212, Sem 0.0212 LR: 9.197e-02 Score 99.232 Data time: 0.2757, Forward time: 1.2592, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.6841 [2021-07-07 14:09:04,596][root][INFO] - ===> Epoch[1067](5335/5): Loss 0.0201, Sem 0.0201 LR: 9.196e-02 Score 99.269 Data time: 0.2459, Forward time: 1.2591, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.7187 [2021-07-07 14:09:17,813][root][INFO] - ===> Epoch[1068](5340/5): Loss 0.0214, Sem 0.0214 LR: 9.195e-02 Score 99.225 Data time: 0.1728, Forward time: 1.2591, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.5941 [2021-07-07 14:09:32,010][root][INFO] - ===> Epoch[1069](5345/5): Loss 0.0202, Sem 0.0202 LR: 9.195e-02 Score 99.279 Data time: 0.3152, Forward time: 1.2592, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.7768 [2021-07-07 14:09:45,681][root][INFO] - ===> Epoch[1070](5350/5): Loss 0.0211, Sem 0.0211 LR: 9.194e-02 Score 99.232 Data time: 0.2832, Forward time: 1.2593, Backward time: 1.2301, DDP time: 0.0042, Total iter time: 2.6942 [2021-07-07 14:09:58,955][root][INFO] - ===> Epoch[1071](5355/5): Loss 0.0224, Sem 0.0224 LR: 9.193e-02 Score 99.195 Data time: 0.2360, Forward time: 1.2594, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.6069 [2021-07-07 14:10:13,052][root][INFO] - ===> Epoch[1072](5360/5): Loss 0.0201, Sem 0.0201 LR: 9.192e-02 Score 99.272 Data time: 0.1842, Forward time: 1.2594, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7791 [2021-07-07 14:10:27,673][root][INFO] - ===> Epoch[1073](5365/5): Loss 0.0207, Sem 0.0207 LR: 9.192e-02 Score 99.253 Data time: 0.2958, Forward time: 1.2595, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.8792 [2021-07-07 14:10:41,686][root][INFO] - ===> Epoch[1074](5370/5): Loss 0.0201, Sem 0.0201 LR: 9.191e-02 Score 99.275 Data time: 0.1891, Forward time: 1.2595, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7563 [2021-07-07 14:10:55,183][root][INFO] - ===> Epoch[1075](5375/5): Loss 0.0215, Sem 0.0215 LR: 9.190e-02 Score 99.227 Data time: 0.2637, Forward time: 1.2595, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.6417 [2021-07-07 14:11:09,084][root][INFO] - ===> Epoch[1076](5380/5): Loss 0.0203, Sem 0.0203 LR: 9.189e-02 Score 99.260 Data time: 0.2265, Forward time: 1.2595, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7336 [2021-07-07 14:11:23,096][root][INFO] - ===> Epoch[1077](5385/5): Loss 0.0216, Sem 0.0216 LR: 9.189e-02 Score 99.222 Data time: 0.2405, Forward time: 1.2594, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7566 [2021-07-07 14:11:37,170][root][INFO] - ===> Epoch[1078](5390/5): Loss 0.0214, Sem 0.0214 LR: 9.188e-02 Score 99.230 Data time: 0.1807, Forward time: 1.2595, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7669 [2021-07-07 14:11:51,531][root][INFO] - ===> Epoch[1079](5395/5): Loss 0.0199, Sem 0.0199 LR: 9.187e-02 Score 99.285 Data time: 0.2399, Forward time: 1.2594, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.8245 [2021-07-07 14:12:05,624][root][INFO] - ===> Epoch[1080](5400/5): Loss 0.0210, Sem 0.0210 LR: 9.186e-02 Score 99.243 Data time: 0.2289, Forward time: 1.2595, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7697 [2021-07-07 14:12:05,626][root][INFO] - ===> Start testing [2021-07-07 14:12:40,412][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1249 Loss 0.218 (AVG: 0.513) Score 94.680 (AVG: 91.063) mIOU 70.822 mAP 84.542 mAcc 77.010 IOU: 64.174 0.013 79.212 76.632 94.888 92.126 27.064 81.471 98.676 81.403 79.586 86.514 58.927 mAP: 84.584 9.319 93.938 89.368 98.839 98.561 58.446 97.670 99.812 87.301 92.570 95.086 93.548 mAcc: 80.297 0.091 88.295 82.749 96.739 97.059 28.660 95.359 99.533 83.475 89.079 96.597 63.196 [2021-07-07 14:12:40,412][root][INFO] - Finished test. Elapsed time: 34.7849 [2021-07-07 14:12:41,559][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 14:12:55,346][root][INFO] - ===> Epoch[1081](5405/5): Loss 0.0214, Sem 0.0214 LR: 9.185e-02 Score 99.244 Data time: 0.2049, Forward time: 1.2594, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.7209 [2021-07-07 14:13:10,465][root][INFO] - ===> Epoch[1082](5410/5): Loss 0.0205, Sem 0.0205 LR: 9.185e-02 Score 99.258 Data time: 0.2403, Forward time: 1.2596, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.9714 [2021-07-07 14:13:24,599][root][INFO] - ===> Epoch[1083](5415/5): Loss 0.0197, Sem 0.0197 LR: 9.184e-02 Score 99.284 Data time: 0.2296, Forward time: 1.2596, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.7672 [2021-07-07 14:13:38,881][root][INFO] - ===> Epoch[1084](5420/5): Loss 0.0205, Sem 0.0205 LR: 9.183e-02 Score 99.256 Data time: 0.2613, Forward time: 1.2596, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.7978 [2021-07-07 14:13:52,423][root][INFO] - ===> Epoch[1085](5425/5): Loss 0.0207, Sem 0.0207 LR: 9.182e-02 Score 99.254 Data time: 0.2140, Forward time: 1.2596, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.6488 [2021-07-07 14:14:06,039][root][INFO] - ===> Epoch[1086](5430/5): Loss 0.0194, Sem 0.0194 LR: 9.182e-02 Score 99.292 Data time: 0.3059, Forward time: 1.2597, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6631 [2021-07-07 14:14:19,928][root][INFO] - ===> Epoch[1087](5435/5): Loss 0.0204, Sem 0.0204 LR: 9.181e-02 Score 99.262 Data time: 0.2003, Forward time: 1.2597, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7333 [2021-07-07 14:14:34,416][root][INFO] - ===> Epoch[1088](5440/5): Loss 0.0205, Sem 0.0205 LR: 9.180e-02 Score 99.259 Data time: 0.1909, Forward time: 1.2598, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8533 [2021-07-07 14:14:49,457][root][INFO] - ===> Epoch[1089](5445/5): Loss 0.0213, Sem 0.0213 LR: 9.179e-02 Score 99.229 Data time: 0.2371, Forward time: 1.2597, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.9736 [2021-07-07 14:15:02,981][root][INFO] - ===> Epoch[1090](5450/5): Loss 0.0194, Sem 0.0194 LR: 9.179e-02 Score 99.289 Data time: 0.1405, Forward time: 1.2597, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.6589 [2021-07-07 14:15:17,654][root][INFO] - ===> Epoch[1091](5455/5): Loss 0.0216, Sem 0.0216 LR: 9.178e-02 Score 99.222 Data time: 0.2425, Forward time: 1.2597, Backward time: 1.2301, DDP time: 0.0042, Total iter time: 2.8901 [2021-07-07 14:15:33,725][root][INFO] - ===> Epoch[1092](5460/5): Loss 0.0206, Sem 0.0206 LR: 9.177e-02 Score 99.258 Data time: 0.2839, Forward time: 1.2600, Backward time: 1.2301, DDP time: 0.0042, Total iter time: 3.1609 [2021-07-07 14:15:48,134][root][INFO] - ===> Epoch[1093](5465/5): Loss 0.0204, Sem 0.0204 LR: 9.176e-02 Score 99.261 Data time: 0.2741, Forward time: 1.2600, Backward time: 1.2301, DDP time: 0.0042, Total iter time: 2.8301 [2021-07-07 14:16:01,731][root][INFO] - ===> Epoch[1094](5470/5): Loss 0.0213, Sem 0.0213 LR: 9.176e-02 Score 99.231 Data time: 0.2066, Forward time: 1.2600, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.6571 [2021-07-07 14:16:15,749][root][INFO] - ===> Epoch[1095](5475/5): Loss 0.0202, Sem 0.0202 LR: 9.175e-02 Score 99.275 Data time: 0.2851, Forward time: 1.2601, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7531 [2021-07-07 14:16:29,824][root][INFO] - ===> Epoch[1096](5480/5): Loss 0.0201, Sem 0.0201 LR: 9.174e-02 Score 99.275 Data time: 0.2586, Forward time: 1.2602, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7758 [2021-07-07 14:16:43,860][root][INFO] - ===> Epoch[1097](5485/5): Loss 0.0202, Sem 0.0202 LR: 9.173e-02 Score 99.277 Data time: 0.2546, Forward time: 1.2602, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7458 [2021-07-07 14:16:57,054][root][INFO] - ===> Epoch[1098](5490/5): Loss 0.0198, Sem 0.0198 LR: 9.173e-02 Score 99.276 Data time: 0.2405, Forward time: 1.2603, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.5949 [2021-07-07 14:17:11,172][root][INFO] - ===> Epoch[1099](5495/5): Loss 0.0192, Sem 0.0192 LR: 9.172e-02 Score 99.306 Data time: 0.2459, Forward time: 1.2603, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7743 [2021-07-07 14:17:24,879][root][INFO] - ===> Epoch[1100](5500/5): Loss 0.0209, Sem 0.0209 LR: 9.171e-02 Score 99.247 Data time: 0.2400, Forward time: 1.2602, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.6981 [2021-07-07 14:17:40,203][root][INFO] - ===> Epoch[1101](5505/5): Loss 0.0210, Sem 0.0210 LR: 9.170e-02 Score 99.242 Data time: 0.2513, Forward time: 1.2603, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 3.0180 [2021-07-07 14:17:54,846][root][INFO] - ===> Epoch[1102](5510/5): Loss 0.0204, Sem 0.0204 LR: 9.170e-02 Score 99.262 Data time: 0.2132, Forward time: 1.2603, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8775 [2021-07-07 14:18:08,670][root][INFO] - ===> Epoch[1103](5515/5): Loss 0.0207, Sem 0.0207 LR: 9.169e-02 Score 99.259 Data time: 0.1730, Forward time: 1.2604, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7234 [2021-07-07 14:18:23,276][root][INFO] - ===> Epoch[1104](5520/5): Loss 0.0209, Sem 0.0209 LR: 9.168e-02 Score 99.250 Data time: 0.2273, Forward time: 1.2604, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.8784 [2021-07-07 14:18:37,259][root][INFO] - ===> Epoch[1105](5525/5): Loss 0.0202, Sem 0.0202 LR: 9.167e-02 Score 99.279 Data time: 0.2406, Forward time: 1.2604, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7546 [2021-07-07 14:18:50,700][root][INFO] - ===> Epoch[1106](5530/5): Loss 0.0207, Sem 0.0207 LR: 9.167e-02 Score 99.249 Data time: 0.1687, Forward time: 1.2604, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.6398 [2021-07-07 14:19:04,770][root][INFO] - ===> Epoch[1107](5535/5): Loss 0.0198, Sem 0.0198 LR: 9.166e-02 Score 99.291 Data time: 0.2060, Forward time: 1.2603, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7765 [2021-07-07 14:19:18,415][root][INFO] - ===> Epoch[1108](5540/5): Loss 0.0203, Sem 0.0203 LR: 9.165e-02 Score 99.259 Data time: 0.2042, Forward time: 1.2603, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.6809 [2021-07-07 14:19:32,772][root][INFO] - ===> Epoch[1109](5545/5): Loss 0.0204, Sem 0.0204 LR: 9.164e-02 Score 99.270 Data time: 0.2554, Forward time: 1.2604, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8228 [2021-07-07 14:19:46,471][root][INFO] - ===> Epoch[1110](5550/5): Loss 0.0208, Sem 0.0208 LR: 9.164e-02 Score 99.253 Data time: 0.1983, Forward time: 1.2605, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6931 [2021-07-07 14:19:59,785][root][INFO] - ===> Epoch[1111](5555/5): Loss 0.0211, Sem 0.0211 LR: 9.163e-02 Score 99.237 Data time: 0.1784, Forward time: 1.2606, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.6177 [2021-07-07 14:20:13,586][root][INFO] - ===> Epoch[1112](5560/5): Loss 0.0208, Sem 0.0208 LR: 9.162e-02 Score 99.252 Data time: 0.2568, Forward time: 1.2606, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7170 [2021-07-07 14:20:27,915][root][INFO] - ===> Epoch[1113](5565/5): Loss 0.0199, Sem 0.0199 LR: 9.161e-02 Score 99.280 Data time: 0.1845, Forward time: 1.2607, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.8193 [2021-07-07 14:20:41,727][root][INFO] - ===> Epoch[1114](5570/5): Loss 0.0200, Sem 0.0200 LR: 9.160e-02 Score 99.281 Data time: 0.2370, Forward time: 1.2608, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7026 [2021-07-07 14:20:55,271][root][INFO] - ===> Epoch[1115](5575/5): Loss 0.0210, Sem 0.0210 LR: 9.160e-02 Score 99.246 Data time: 0.2208, Forward time: 1.2609, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.6582 [2021-07-07 14:21:08,597][root][INFO] - ===> Epoch[1116](5580/5): Loss 0.0204, Sem 0.0204 LR: 9.159e-02 Score 99.268 Data time: 0.1746, Forward time: 1.2609, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.6244 [2021-07-07 14:21:23,254][root][INFO] - ===> Epoch[1117](5585/5): Loss 0.0209, Sem 0.0209 LR: 9.158e-02 Score 99.243 Data time: 0.1950, Forward time: 1.2607, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.8794 [2021-07-07 14:21:36,850][root][INFO] - ===> Epoch[1118](5590/5): Loss 0.0196, Sem 0.0196 LR: 9.157e-02 Score 99.294 Data time: 0.2248, Forward time: 1.2606, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.6652 [2021-07-07 14:21:50,372][root][INFO] - ===> Epoch[1119](5595/5): Loss 0.0203, Sem 0.0203 LR: 9.157e-02 Score 99.270 Data time: 0.2157, Forward time: 1.2607, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.6604 [2021-07-07 14:22:05,091][root][INFO] - ===> Epoch[1120](5600/5): Loss 0.0195, Sem 0.0195 LR: 9.156e-02 Score 99.294 Data time: 0.1954, Forward time: 1.2607, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.8959 [2021-07-07 14:22:05,093][root][INFO] - ===> Start testing [2021-07-07 14:22:40,263][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1349 Loss 0.293 (AVG: 0.501) Score 93.170 (AVG: 91.039) mIOU 70.963 mAP 84.482 mAcc 77.020 IOU: 64.401 0.000 78.757 77.969 94.315 92.197 27.398 83.206 98.636 81.078 80.260 86.033 58.268 mAP: 85.038 11.572 92.382 89.694 98.925 98.296 58.238 97.619 99.816 85.537 93.206 94.708 93.236 mAcc: 80.715 0.000 89.159 85.144 95.922 97.005 29.286 93.795 99.430 83.662 87.819 96.301 63.024 [2021-07-07 14:22:40,264][root][INFO] - Finished test. Elapsed time: 35.1698 [2021-07-07 14:22:41,441][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 14:22:54,526][root][INFO] - ===> Epoch[1121](5605/5): Loss 0.0205, Sem 0.0205 LR: 9.155e-02 Score 99.264 Data time: 0.2264, Forward time: 1.2606, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.5711 [2021-07-07 14:23:08,608][root][INFO] - ===> Epoch[1122](5610/5): Loss 0.0204, Sem 0.0204 LR: 9.154e-02 Score 99.265 Data time: 0.2603, Forward time: 1.2606, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7670 [2021-07-07 14:23:21,613][root][INFO] - ===> Epoch[1123](5615/5): Loss 0.0214, Sem 0.0214 LR: 9.154e-02 Score 99.229 Data time: 0.2268, Forward time: 1.2605, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.5555 [2021-07-07 14:23:35,459][root][INFO] - ===> Epoch[1124](5620/5): Loss 0.0201, Sem 0.0201 LR: 9.153e-02 Score 99.272 Data time: 0.2206, Forward time: 1.2605, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7245 [2021-07-07 14:23:49,409][root][INFO] - ===> Epoch[1125](5625/5): Loss 0.0208, Sem 0.0208 LR: 9.152e-02 Score 99.248 Data time: 0.2014, Forward time: 1.2604, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7515 [2021-07-07 14:24:03,333][root][INFO] - ===> Epoch[1126](5630/5): Loss 0.0203, Sem 0.0203 LR: 9.151e-02 Score 99.265 Data time: 0.2286, Forward time: 1.2605, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7325 [2021-07-07 14:24:16,760][root][INFO] - ===> Epoch[1127](5635/5): Loss 0.0210, Sem 0.0210 LR: 9.151e-02 Score 99.241 Data time: 0.2432, Forward time: 1.2604, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.6303 [2021-07-07 14:24:30,581][root][INFO] - ===> Epoch[1128](5640/5): Loss 0.0207, Sem 0.0207 LR: 9.150e-02 Score 99.257 Data time: 0.2658, Forward time: 1.2604, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.7146 [2021-07-07 14:24:44,508][root][INFO] - ===> Epoch[1129](5645/5): Loss 0.0206, Sem 0.0206 LR: 9.149e-02 Score 99.253 Data time: 0.2093, Forward time: 1.2605, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.7262 [2021-07-07 14:24:58,113][root][INFO] - ===> Epoch[1130](5650/5): Loss 0.0193, Sem 0.0193 LR: 9.148e-02 Score 99.305 Data time: 0.2163, Forward time: 1.2605, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.6777 [2021-07-07 14:25:12,579][root][INFO] - ===> Epoch[1131](5655/5): Loss 0.0205, Sem 0.0205 LR: 9.148e-02 Score 99.256 Data time: 0.2038, Forward time: 1.2603, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.8498 [2021-07-07 14:25:27,184][root][INFO] - ===> Epoch[1132](5660/5): Loss 0.0218, Sem 0.0218 LR: 9.147e-02 Score 99.249 Data time: 0.2101, Forward time: 1.2602, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.8791 [2021-07-07 14:25:41,097][root][INFO] - ===> Epoch[1133](5665/5): Loss 0.0213, Sem 0.0213 LR: 9.146e-02 Score 99.233 Data time: 0.1720, Forward time: 1.2601, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7344 [2021-07-07 14:25:54,396][root][INFO] - ===> Epoch[1134](5670/5): Loss 0.0208, Sem 0.0208 LR: 9.145e-02 Score 99.250 Data time: 0.2758, Forward time: 1.2600, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.6064 [2021-07-07 14:26:09,642][root][INFO] - ===> Epoch[1135](5675/5): Loss 0.0199, Sem 0.0199 LR: 9.145e-02 Score 99.281 Data time: 0.2505, Forward time: 1.2602, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.9780 [2021-07-07 14:26:24,103][root][INFO] - ===> Epoch[1136](5680/5): Loss 0.0204, Sem 0.0204 LR: 9.144e-02 Score 99.263 Data time: 0.1964, Forward time: 1.2602, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8464 [2021-07-07 14:26:37,478][root][INFO] - ===> Epoch[1137](5685/5): Loss 0.0210, Sem 0.0210 LR: 9.143e-02 Score 99.248 Data time: 0.2333, Forward time: 1.2602, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.6340 [2021-07-07 14:26:51,144][root][INFO] - ===> Epoch[1138](5690/5): Loss 0.0213, Sem 0.0213 LR: 9.142e-02 Score 99.223 Data time: 0.2453, Forward time: 1.2602, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.6876 [2021-07-07 14:27:05,139][root][INFO] - ===> Epoch[1139](5695/5): Loss 0.0217, Sem 0.0217 LR: 9.142e-02 Score 99.225 Data time: 0.2113, Forward time: 1.2601, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7581 [2021-07-07 14:27:18,622][root][INFO] - ===> Epoch[1140](5700/5): Loss 0.0200, Sem 0.0200 LR: 9.141e-02 Score 99.278 Data time: 0.1816, Forward time: 1.2600, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6579 [2021-07-07 14:27:32,771][root][INFO] - ===> Epoch[1141](5705/5): Loss 0.0196, Sem 0.0196 LR: 9.140e-02 Score 99.290 Data time: 0.2374, Forward time: 1.2600, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7719 [2021-07-07 14:27:46,463][root][INFO] - ===> Epoch[1142](5710/5): Loss 0.0196, Sem 0.0196 LR: 9.139e-02 Score 99.288 Data time: 0.1910, Forward time: 1.2600, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6950 [2021-07-07 14:28:00,265][root][INFO] - ===> Epoch[1143](5715/5): Loss 0.0201, Sem 0.0201 LR: 9.139e-02 Score 99.271 Data time: 0.2310, Forward time: 1.2600, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7104 [2021-07-07 14:28:14,319][root][INFO] - ===> Epoch[1144](5720/5): Loss 0.0194, Sem 0.0194 LR: 9.138e-02 Score 99.296 Data time: 0.2416, Forward time: 1.2601, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7600 [2021-07-07 14:28:28,486][root][INFO] - ===> Epoch[1145](5725/5): Loss 0.0214, Sem 0.0214 LR: 9.137e-02 Score 99.236 Data time: 0.2086, Forward time: 1.2602, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7947 [2021-07-07 14:28:42,386][root][INFO] - ===> Epoch[1146](5730/5): Loss 0.0203, Sem 0.0203 LR: 9.136e-02 Score 99.270 Data time: 0.2125, Forward time: 1.2603, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7390 [2021-07-07 14:28:55,855][root][INFO] - ===> Epoch[1147](5735/5): Loss 0.0202, Sem 0.0202 LR: 9.136e-02 Score 99.280 Data time: 0.1483, Forward time: 1.2602, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.6491 [2021-07-07 14:29:10,737][root][INFO] - ===> Epoch[1148](5740/5): Loss 0.0196, Sem 0.0196 LR: 9.135e-02 Score 99.294 Data time: 0.1821, Forward time: 1.2603, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.9380 [2021-07-07 14:29:24,698][root][INFO] - ===> Epoch[1149](5745/5): Loss 0.0205, Sem 0.0205 LR: 9.134e-02 Score 99.265 Data time: 0.2147, Forward time: 1.2602, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7526 [2021-07-07 14:29:39,110][root][INFO] - ===> Epoch[1150](5750/5): Loss 0.0197, Sem 0.0197 LR: 9.133e-02 Score 99.293 Data time: 0.2081, Forward time: 1.2603, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8316 [2021-07-07 14:29:53,121][root][INFO] - ===> Epoch[1151](5755/5): Loss 0.0201, Sem 0.0201 LR: 9.132e-02 Score 99.271 Data time: 0.2545, Forward time: 1.2604, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7545 [2021-07-07 14:30:07,378][root][INFO] - ===> Epoch[1152](5760/5): Loss 0.0205, Sem 0.0205 LR: 9.132e-02 Score 99.260 Data time: 0.2400, Forward time: 1.2603, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7992 [2021-07-07 14:30:21,042][root][INFO] - ===> Epoch[1153](5765/5): Loss 0.0196, Sem 0.0196 LR: 9.131e-02 Score 99.292 Data time: 0.2002, Forward time: 1.2603, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.6878 [2021-07-07 14:30:35,812][root][INFO] - ===> Epoch[1154](5770/5): Loss 0.0222, Sem 0.0222 LR: 9.130e-02 Score 99.220 Data time: 0.2275, Forward time: 1.2603, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.9056 [2021-07-07 14:30:49,644][root][INFO] - ===> Epoch[1155](5775/5): Loss 0.0220, Sem 0.0220 LR: 9.129e-02 Score 99.212 Data time: 0.3010, Forward time: 1.2602, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7117 [2021-07-07 14:31:03,792][root][INFO] - ===> Epoch[1156](5780/5): Loss 0.0231, Sem 0.0231 LR: 9.129e-02 Score 99.186 Data time: 0.2365, Forward time: 1.2602, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7660 [2021-07-07 14:31:18,119][root][INFO] - ===> Epoch[1157](5785/5): Loss 0.0236, Sem 0.0236 LR: 9.128e-02 Score 99.165 Data time: 0.2322, Forward time: 1.2602, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8144 [2021-07-07 14:31:31,946][root][INFO] - ===> Epoch[1158](5790/5): Loss 0.0234, Sem 0.0234 LR: 9.127e-02 Score 99.155 Data time: 0.2046, Forward time: 1.2602, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7258 [2021-07-07 14:31:45,894][root][INFO] - ===> Epoch[1159](5795/5): Loss 0.0222, Sem 0.0222 LR: 9.126e-02 Score 99.208 Data time: 0.2507, Forward time: 1.2602, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7408 [2021-07-07 14:31:59,900][root][INFO] - ===> Epoch[1160](5800/5): Loss 0.0219, Sem 0.0219 LR: 9.126e-02 Score 99.215 Data time: 0.2900, Forward time: 1.2603, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7546 [2021-07-07 14:31:59,902][root][INFO] - ===> Start testing [2021-07-07 14:32:35,217][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1274 Loss 0.253 (AVG: 0.538) Score 93.597 (AVG: 90.503) mIOU 68.667 mAP 83.035 mAcc 74.742 IOU: 64.012 0.000 78.699 77.673 94.152 90.483 19.825 78.592 98.618 73.672 79.556 84.906 52.477 mAP: 84.118 6.755 93.118 90.380 98.962 98.620 49.143 96.045 99.827 84.263 93.114 94.427 90.679 mAcc: 80.550 0.000 87.826 85.545 95.291 97.600 21.123 88.963 99.310 77.161 86.490 97.158 54.626 [2021-07-07 14:32:35,217][root][INFO] - Finished test. Elapsed time: 35.3141 [2021-07-07 14:32:36,311][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 14:32:49,791][root][INFO] - ===> Epoch[1161](5805/5): Loss 0.0239, Sem 0.0239 LR: 9.125e-02 Score 99.157 Data time: 0.1482, Forward time: 1.2602, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6591 [2021-07-07 14:33:03,412][root][INFO] - ===> Epoch[1162](5810/5): Loss 0.0213, Sem 0.0213 LR: 9.124e-02 Score 99.239 Data time: 0.2221, Forward time: 1.2602, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6836 [2021-07-07 14:33:17,326][root][INFO] - ===> Epoch[1163](5815/5): Loss 0.0209, Sem 0.0209 LR: 9.123e-02 Score 99.246 Data time: 0.2277, Forward time: 1.2603, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7401 [2021-07-07 14:33:31,719][root][INFO] - ===> Epoch[1164](5820/5): Loss 0.0210, Sem 0.0210 LR: 9.123e-02 Score 99.242 Data time: 0.2859, Forward time: 1.2604, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.8245 [2021-07-07 14:33:45,754][root][INFO] - ===> Epoch[1165](5825/5): Loss 0.0208, Sem 0.0208 LR: 9.122e-02 Score 99.253 Data time: 0.2799, Forward time: 1.2604, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7614 [2021-07-07 14:33:59,582][root][INFO] - ===> Epoch[1166](5830/5): Loss 0.0205, Sem 0.0205 LR: 9.121e-02 Score 99.256 Data time: 0.2337, Forward time: 1.2603, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7137 [2021-07-07 14:34:14,080][root][INFO] - ===> Epoch[1167](5835/5): Loss 0.0194, Sem 0.0194 LR: 9.120e-02 Score 99.302 Data time: 0.2680, Forward time: 1.2604, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.8482 [2021-07-07 14:34:27,670][root][INFO] - ===> Epoch[1168](5840/5): Loss 0.0208, Sem 0.0208 LR: 9.120e-02 Score 99.248 Data time: 0.1945, Forward time: 1.2603, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.6769 [2021-07-07 14:34:41,545][root][INFO] - ===> Epoch[1169](5845/5): Loss 0.0206, Sem 0.0206 LR: 9.119e-02 Score 99.273 Data time: 0.2344, Forward time: 1.2602, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7092 [2021-07-07 14:34:55,118][root][INFO] - ===> Epoch[1170](5850/5): Loss 0.0201, Sem 0.0201 LR: 9.118e-02 Score 99.274 Data time: 0.2757, Forward time: 1.2602, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.6683 [2021-07-07 14:35:09,042][root][INFO] - ===> Epoch[1171](5855/5): Loss 0.0223, Sem 0.0223 LR: 9.117e-02 Score 99.244 Data time: 0.2589, Forward time: 1.2601, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7375 [2021-07-07 14:35:22,765][root][INFO] - ===> Epoch[1172](5860/5): Loss 0.0232, Sem 0.0232 LR: 9.117e-02 Score 99.206 Data time: 0.2077, Forward time: 1.2601, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7001 [2021-07-07 14:35:37,306][root][INFO] - ===> Epoch[1173](5865/5): Loss 0.0226, Sem 0.0226 LR: 9.116e-02 Score 99.200 Data time: 0.2828, Forward time: 1.2602, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.8693 [2021-07-07 14:35:51,692][root][INFO] - ===> Epoch[1174](5870/5): Loss 0.0219, Sem 0.0219 LR: 9.115e-02 Score 99.223 Data time: 0.2391, Forward time: 1.2603, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.8231 [2021-07-07 14:36:05,565][root][INFO] - ===> Epoch[1175](5875/5): Loss 0.0213, Sem 0.0213 LR: 9.114e-02 Score 99.233 Data time: 0.1766, Forward time: 1.2602, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7389 [2021-07-07 14:36:19,323][root][INFO] - ===> Epoch[1176](5880/5): Loss 0.0208, Sem 0.0208 LR: 9.114e-02 Score 99.251 Data time: 0.2568, Forward time: 1.2601, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7112 [2021-07-07 14:36:32,836][root][INFO] - ===> Epoch[1177](5885/5): Loss 0.0211, Sem 0.0211 LR: 9.113e-02 Score 99.245 Data time: 0.2478, Forward time: 1.2601, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.6531 [2021-07-07 14:36:45,733][root][INFO] - ===> Epoch[1178](5890/5): Loss 0.0218, Sem 0.0218 LR: 9.112e-02 Score 99.215 Data time: 0.2044, Forward time: 1.2600, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.5284 [2021-07-07 14:37:00,258][root][INFO] - ===> Epoch[1179](5895/5): Loss 0.0210, Sem 0.0210 LR: 9.111e-02 Score 99.252 Data time: 0.1845, Forward time: 1.2600, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.8522 [2021-07-07 14:37:14,110][root][INFO] - ===> Epoch[1180](5900/5): Loss 0.0199, Sem 0.0199 LR: 9.110e-02 Score 99.280 Data time: 0.2454, Forward time: 1.2599, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7248 [2021-07-07 14:37:28,224][root][INFO] - ===> Epoch[1181](5905/5): Loss 0.0204, Sem 0.0204 LR: 9.110e-02 Score 99.266 Data time: 0.2262, Forward time: 1.2598, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7809 [2021-07-07 14:37:41,883][root][INFO] - ===> Epoch[1182](5910/5): Loss 0.0194, Sem 0.0194 LR: 9.109e-02 Score 99.298 Data time: 0.2569, Forward time: 1.2598, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.6851 [2021-07-07 14:37:55,798][root][INFO] - ===> Epoch[1183](5915/5): Loss 0.0199, Sem 0.0199 LR: 9.108e-02 Score 99.275 Data time: 0.2409, Forward time: 1.2597, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7368 [2021-07-07 14:38:09,337][root][INFO] - ===> Epoch[1184](5920/5): Loss 0.0198, Sem 0.0198 LR: 9.107e-02 Score 99.280 Data time: 0.2077, Forward time: 1.2598, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.6477 [2021-07-07 14:38:23,268][root][INFO] - ===> Epoch[1185](5925/5): Loss 0.0193, Sem 0.0193 LR: 9.107e-02 Score 99.301 Data time: 0.2209, Forward time: 1.2597, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7225 [2021-07-07 14:38:37,083][root][INFO] - ===> Epoch[1186](5930/5): Loss 0.0189, Sem 0.0189 LR: 9.106e-02 Score 99.315 Data time: 0.2205, Forward time: 1.2597, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7272 [2021-07-07 14:38:51,409][root][INFO] - ===> Epoch[1187](5935/5): Loss 0.0199, Sem 0.0199 LR: 9.105e-02 Score 99.274 Data time: 0.1690, Forward time: 1.2596, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8085 [2021-07-07 14:39:05,167][root][INFO] - ===> Epoch[1188](5940/5): Loss 0.0190, Sem 0.0190 LR: 9.104e-02 Score 99.320 Data time: 0.2012, Forward time: 1.2596, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6993 [2021-07-07 14:39:18,510][root][INFO] - ===> Epoch[1189](5945/5): Loss 0.0204, Sem 0.0204 LR: 9.104e-02 Score 99.268 Data time: 0.2048, Forward time: 1.2595, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6318 [2021-07-07 14:39:32,870][root][INFO] - ===> Epoch[1190](5950/5): Loss 0.0197, Sem 0.0197 LR: 9.103e-02 Score 99.294 Data time: 0.2142, Forward time: 1.2595, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8201 [2021-07-07 14:39:46,636][root][INFO] - ===> Epoch[1191](5955/5): Loss 0.0207, Sem 0.0207 LR: 9.102e-02 Score 99.252 Data time: 0.2384, Forward time: 1.2595, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6996 [2021-07-07 14:40:00,935][root][INFO] - ===> Epoch[1192](5960/5): Loss 0.0209, Sem 0.0209 LR: 9.101e-02 Score 99.253 Data time: 0.2515, Forward time: 1.2594, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8159 [2021-07-07 14:40:14,865][root][INFO] - ===> Epoch[1193](5965/5): Loss 0.0234, Sem 0.0234 LR: 9.101e-02 Score 99.184 Data time: 0.2762, Forward time: 1.2595, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7394 [2021-07-07 14:40:28,938][root][INFO] - ===> Epoch[1194](5970/5): Loss 0.0238, Sem 0.0238 LR: 9.100e-02 Score 99.199 Data time: 0.2259, Forward time: 1.2596, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7603 [2021-07-07 14:40:42,869][root][INFO] - ===> Epoch[1195](5975/5): Loss 0.0239, Sem 0.0239 LR: 9.099e-02 Score 99.159 Data time: 0.2840, Forward time: 1.2597, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7357 [2021-07-07 14:40:56,958][root][INFO] - ===> Epoch[1196](5980/5): Loss 0.0224, Sem 0.0224 LR: 9.098e-02 Score 99.196 Data time: 0.2323, Forward time: 1.2598, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7707 [2021-07-07 14:41:11,196][root][INFO] - ===> Epoch[1197](5985/5): Loss 0.0230, Sem 0.0230 LR: 9.098e-02 Score 99.196 Data time: 0.2200, Forward time: 1.2597, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7956 [2021-07-07 14:41:25,189][root][INFO] - ===> Epoch[1198](5990/5): Loss 0.0224, Sem 0.0224 LR: 9.097e-02 Score 99.201 Data time: 0.2047, Forward time: 1.2596, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7527 [2021-07-07 14:41:38,465][root][INFO] - ===> Epoch[1199](5995/5): Loss 0.0211, Sem 0.0211 LR: 9.096e-02 Score 99.240 Data time: 0.2318, Forward time: 1.2596, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.6013 [2021-07-07 14:41:52,653][root][INFO] - ===> Epoch[1200](6000/5): Loss 0.0226, Sem 0.0226 LR: 9.095e-02 Score 99.192 Data time: 0.2233, Forward time: 1.2596, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7969 [2021-07-07 14:41:52,655][root][INFO] - ===> Start testing [2021-07-07 14:42:28,492][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1230 Loss 0.313 (AVG: 0.496) Score 91.837 (AVG: 90.988) mIOU 70.577 mAP 84.670 mAcc 77.544 IOU: 64.096 0.040 78.589 76.444 94.797 89.555 38.734 78.699 98.583 76.656 77.098 87.102 57.110 mAP: 85.171 13.368 91.576 88.916 98.924 98.632 65.310 96.839 99.836 82.012 92.728 95.304 92.095 mAcc: 80.228 0.273 88.174 83.194 96.036 96.388 43.807 95.097 99.321 79.015 89.062 96.211 61.261 [2021-07-07 14:42:28,493][root][INFO] - Finished test. Elapsed time: 35.8369 [2021-07-07 14:42:29,503][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 14:42:42,298][root][INFO] - ===> Epoch[1201](6005/5): Loss 0.0208, Sem 0.0208 LR: 9.095e-02 Score 99.253 Data time: 0.1723, Forward time: 1.2595, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.5202 [2021-07-07 14:42:55,623][root][INFO] - ===> Epoch[1202](6010/5): Loss 0.0220, Sem 0.0220 LR: 9.094e-02 Score 99.207 Data time: 0.1894, Forward time: 1.2594, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.6184 [2021-07-07 14:43:10,436][root][INFO] - ===> Epoch[1203](6015/5): Loss 0.0206, Sem 0.0206 LR: 9.093e-02 Score 99.252 Data time: 0.2052, Forward time: 1.2593, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.9062 [2021-07-07 14:43:24,572][root][INFO] - ===> Epoch[1204](6020/5): Loss 0.0214, Sem 0.0214 LR: 9.092e-02 Score 99.239 Data time: 0.1907, Forward time: 1.2592, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7841 [2021-07-07 14:43:38,200][root][INFO] - ===> Epoch[1205](6025/5): Loss 0.0219, Sem 0.0219 LR: 9.092e-02 Score 99.214 Data time: 0.2302, Forward time: 1.2593, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.6724 [2021-07-07 14:43:52,161][root][INFO] - ===> Epoch[1206](6030/5): Loss 0.0210, Sem 0.0210 LR: 9.091e-02 Score 99.242 Data time: 0.1835, Forward time: 1.2592, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7397 [2021-07-07 14:44:06,022][root][INFO] - ===> Epoch[1207](6035/5): Loss 0.0226, Sem 0.0226 LR: 9.090e-02 Score 99.204 Data time: 0.1988, Forward time: 1.2591, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7286 [2021-07-07 14:44:19,556][root][INFO] - ===> Epoch[1208](6040/5): Loss 0.0215, Sem 0.0215 LR: 9.089e-02 Score 99.223 Data time: 0.2439, Forward time: 1.2592, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.6637 [2021-07-07 14:44:33,138][root][INFO] - ===> Epoch[1209](6045/5): Loss 0.0222, Sem 0.0222 LR: 9.089e-02 Score 99.207 Data time: 0.2366, Forward time: 1.2593, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6659 [2021-07-07 14:44:47,335][root][INFO] - ===> Epoch[1210](6050/5): Loss 0.0223, Sem 0.0223 LR: 9.088e-02 Score 99.213 Data time: 0.2366, Forward time: 1.2592, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7894 [2021-07-07 14:45:01,192][root][INFO] - ===> Epoch[1211](6055/5): Loss 0.0235, Sem 0.0235 LR: 9.087e-02 Score 99.161 Data time: 0.1880, Forward time: 1.2591, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7329 [2021-07-07 14:45:14,853][root][INFO] - ===> Epoch[1212](6060/5): Loss 0.0253, Sem 0.0253 LR: 9.086e-02 Score 99.146 Data time: 0.2487, Forward time: 1.2591, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.6788 [2021-07-07 14:45:29,154][root][INFO] - ===> Epoch[1213](6065/5): Loss 0.0253, Sem 0.0253 LR: 9.085e-02 Score 99.108 Data time: 0.2085, Forward time: 1.2590, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.8106 [2021-07-07 14:45:42,257][root][INFO] - ===> Epoch[1214](6070/5): Loss 0.0260, Sem 0.0260 LR: 9.085e-02 Score 99.085 Data time: 0.2336, Forward time: 1.2589, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.5791 [2021-07-07 14:45:55,992][root][INFO] - ===> Epoch[1215](6075/5): Loss 0.0243, Sem 0.0243 LR: 9.084e-02 Score 99.130 Data time: 0.2244, Forward time: 1.2589, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7034 [2021-07-07 14:46:09,713][root][INFO] - ===> Epoch[1216](6080/5): Loss 0.0260, Sem 0.0260 LR: 9.083e-02 Score 99.099 Data time: 0.1690, Forward time: 1.2589, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7025 [2021-07-07 14:46:24,239][root][INFO] - ===> Epoch[1217](6085/5): Loss 0.0289, Sem 0.0289 LR: 9.082e-02 Score 98.969 Data time: 0.2507, Forward time: 1.2590, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8524 [2021-07-07 14:46:37,790][root][INFO] - ===> Epoch[1218](6090/5): Loss 0.0322, Sem 0.0322 LR: 9.082e-02 Score 98.866 Data time: 0.2651, Forward time: 1.2590, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.6561 [2021-07-07 14:46:51,229][root][INFO] - ===> Epoch[1219](6095/5): Loss 0.0333, Sem 0.0333 LR: 9.081e-02 Score 98.848 Data time: 0.1946, Forward time: 1.2588, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.6412 [2021-07-07 14:47:05,233][root][INFO] - ===> Epoch[1220](6100/5): Loss 0.0321, Sem 0.0321 LR: 9.080e-02 Score 98.890 Data time: 0.2572, Forward time: 1.2589, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7398 [2021-07-07 14:47:18,756][root][INFO] - ===> Epoch[1221](6105/5): Loss 0.0292, Sem 0.0292 LR: 9.079e-02 Score 98.963 Data time: 0.2624, Forward time: 1.2590, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.6478 [2021-07-07 14:47:32,562][root][INFO] - ===> Epoch[1222](6110/5): Loss 0.0272, Sem 0.0272 LR: 9.079e-02 Score 99.022 Data time: 0.2120, Forward time: 1.2589, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7143 [2021-07-07 14:47:46,507][root][INFO] - ===> Epoch[1223](6115/5): Loss 0.0271, Sem 0.0271 LR: 9.078e-02 Score 99.040 Data time: 0.2545, Forward time: 1.2589, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7343 [2021-07-07 14:48:00,810][root][INFO] - ===> Epoch[1224](6120/5): Loss 0.0279, Sem 0.0279 LR: 9.077e-02 Score 99.025 Data time: 0.2642, Forward time: 1.2590, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.8175 [2021-07-07 14:48:15,205][root][INFO] - ===> Epoch[1225](6125/5): Loss 0.0245, Sem 0.0245 LR: 9.076e-02 Score 99.132 Data time: 0.2641, Forward time: 1.2590, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.8274 [2021-07-07 14:48:28,205][root][INFO] - ===> Epoch[1226](6130/5): Loss 0.0246, Sem 0.0246 LR: 9.076e-02 Score 99.114 Data time: 0.2059, Forward time: 1.2590, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.5541 [2021-07-07 14:48:42,832][root][INFO] - ===> Epoch[1227](6135/5): Loss 0.0259, Sem 0.0259 LR: 9.075e-02 Score 99.084 Data time: 0.2244, Forward time: 1.2589, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.8820 [2021-07-07 14:48:56,470][root][INFO] - ===> Epoch[1228](6140/5): Loss 0.0264, Sem 0.0264 LR: 9.074e-02 Score 99.073 Data time: 0.2711, Forward time: 1.2589, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.6760 [2021-07-07 14:49:10,061][root][INFO] - ===> Epoch[1229](6145/5): Loss 0.0241, Sem 0.0241 LR: 9.073e-02 Score 99.153 Data time: 0.2166, Forward time: 1.2589, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.6749 [2021-07-07 14:49:24,182][root][INFO] - ===> Epoch[1230](6150/5): Loss 0.0242, Sem 0.0242 LR: 9.073e-02 Score 99.146 Data time: 0.2087, Forward time: 1.2588, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7646 [2021-07-07 14:49:37,667][root][INFO] - ===> Epoch[1231](6155/5): Loss 0.0248, Sem 0.0248 LR: 9.072e-02 Score 99.118 Data time: 0.2356, Forward time: 1.2587, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.6457 [2021-07-07 14:49:52,232][root][INFO] - ===> Epoch[1232](6160/5): Loss 0.0242, Sem 0.0242 LR: 9.071e-02 Score 99.155 Data time: 0.2515, Forward time: 1.2588, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.8645 [2021-07-07 14:50:07,802][root][INFO] - ===> Epoch[1233](6165/5): Loss 0.0247, Sem 0.0247 LR: 9.070e-02 Score 99.124 Data time: 0.2892, Forward time: 1.2590, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 3.0567 [2021-07-07 14:50:22,389][root][INFO] - ===> Epoch[1234](6170/5): Loss 0.0270, Sem 0.0270 LR: 9.070e-02 Score 99.063 Data time: 0.2776, Forward time: 1.2591, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.8629 [2021-07-07 14:50:36,338][root][INFO] - ===> Epoch[1235](6175/5): Loss 0.0254, Sem 0.0254 LR: 9.069e-02 Score 99.099 Data time: 0.2641, Forward time: 1.2591, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7468 [2021-07-07 14:50:49,679][root][INFO] - ===> Epoch[1236](6180/5): Loss 0.0284, Sem 0.0284 LR: 9.068e-02 Score 98.997 Data time: 0.2458, Forward time: 1.2592, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.6248 [2021-07-07 14:51:03,013][root][INFO] - ===> Epoch[1237](6185/5): Loss 0.0253, Sem 0.0253 LR: 9.067e-02 Score 99.093 Data time: 0.2827, Forward time: 1.2592, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.6158 [2021-07-07 14:51:17,242][root][INFO] - ===> Epoch[1238](6190/5): Loss 0.0253, Sem 0.0253 LR: 9.067e-02 Score 99.124 Data time: 0.2708, Forward time: 1.2592, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.7976 [2021-07-07 14:51:30,433][root][INFO] - ===> Epoch[1239](6195/5): Loss 0.0291, Sem 0.0291 LR: 9.066e-02 Score 98.978 Data time: 0.2520, Forward time: 1.2592, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.5944 [2021-07-07 14:51:44,478][root][INFO] - ===> Epoch[1240](6200/5): Loss 0.0322, Sem 0.0322 LR: 9.065e-02 Score 98.926 Data time: 0.1841, Forward time: 1.2592, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.7528 [2021-07-07 14:51:44,480][root][INFO] - ===> Start testing [2021-07-07 14:52:19,404][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1210 Loss 0.294 (AVG: 0.475) Score 93.199 (AVG: 90.762) mIOU 69.710 mAP 85.168 mAcc 75.743 IOU: 65.241 0.399 78.002 76.812 95.218 91.025 26.656 75.226 98.738 80.656 79.178 85.344 53.736 mAP: 84.469 20.765 90.412 90.837 99.159 98.682 60.774 93.410 99.818 90.497 92.713 94.281 91.362 mAcc: 81.647 2.005 81.455 82.824 96.884 95.000 27.839 87.971 99.580 85.852 89.658 96.902 57.042 [2021-07-07 14:52:19,404][root][INFO] - Finished test. Elapsed time: 34.9230 [2021-07-07 14:52:20,498][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 14:52:33,702][root][INFO] - ===> Epoch[1241](6205/5): Loss 0.0289, Sem 0.0289 LR: 9.064e-02 Score 98.987 Data time: 0.1774, Forward time: 1.2590, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.6049 [2021-07-07 14:52:47,775][root][INFO] - ===> Epoch[1242](6210/5): Loss 0.0309, Sem 0.0309 LR: 9.064e-02 Score 98.950 Data time: 0.2544, Forward time: 1.2590, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.7534 [2021-07-07 14:53:02,530][root][INFO] - ===> Epoch[1243](6215/5): Loss 0.0327, Sem 0.0327 LR: 9.063e-02 Score 98.879 Data time: 0.1724, Forward time: 1.2589, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.8983 [2021-07-07 14:53:17,074][root][INFO] - ===> Epoch[1244](6220/5): Loss 0.0306, Sem 0.0306 LR: 9.062e-02 Score 98.909 Data time: 0.2207, Forward time: 1.2590, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.8621 [2021-07-07 14:53:31,060][root][INFO] - ===> Epoch[1245](6225/5): Loss 0.0333, Sem 0.0333 LR: 9.061e-02 Score 98.883 Data time: 0.2678, Forward time: 1.2591, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.7503 [2021-07-07 14:53:46,097][root][INFO] - ===> Epoch[1246](6230/5): Loss 0.0330, Sem 0.0330 LR: 9.060e-02 Score 98.872 Data time: 0.2850, Forward time: 1.2592, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.9657 [2021-07-07 14:54:00,610][root][INFO] - ===> Epoch[1247](6235/5): Loss 0.0312, Sem 0.0312 LR: 9.060e-02 Score 98.910 Data time: 0.2260, Forward time: 1.2593, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.8597 [2021-07-07 14:54:14,382][root][INFO] - ===> Epoch[1248](6240/5): Loss 0.0310, Sem 0.0310 LR: 9.059e-02 Score 98.966 Data time: 0.2282, Forward time: 1.2594, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.7090 [2021-07-07 14:54:28,856][root][INFO] - ===> Epoch[1249](6245/5): Loss 0.0281, Sem 0.0281 LR: 9.058e-02 Score 99.016 Data time: 0.2235, Forward time: 1.2593, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.8534 [2021-07-07 14:54:42,821][root][INFO] - ===> Epoch[1250](6250/5): Loss 0.0285, Sem 0.0285 LR: 9.057e-02 Score 98.992 Data time: 0.2370, Forward time: 1.2594, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.7515 [2021-07-07 14:54:56,823][root][INFO] - ===> Epoch[1251](6255/5): Loss 0.0269, Sem 0.0269 LR: 9.057e-02 Score 99.044 Data time: 0.2520, Forward time: 1.2594, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.7582 [2021-07-07 14:55:11,421][root][INFO] - ===> Epoch[1252](6260/5): Loss 0.0257, Sem 0.0257 LR: 9.056e-02 Score 99.090 Data time: 0.3574, Forward time: 1.2596, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.8688 [2021-07-07 14:55:25,465][root][INFO] - ===> Epoch[1253](6265/5): Loss 0.0260, Sem 0.0260 LR: 9.055e-02 Score 99.075 Data time: 0.1865, Forward time: 1.2595, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.7741 [2021-07-07 14:55:39,801][root][INFO] - ===> Epoch[1254](6270/5): Loss 0.0249, Sem 0.0249 LR: 9.054e-02 Score 99.119 Data time: 0.2801, Forward time: 1.2595, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.8195 [2021-07-07 14:55:53,707][root][INFO] - ===> Epoch[1255](6275/5): Loss 0.0244, Sem 0.0244 LR: 9.054e-02 Score 99.125 Data time: 0.2447, Forward time: 1.2594, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.7309 [2021-07-07 14:56:08,351][root][INFO] - ===> Epoch[1256](6280/5): Loss 0.0234, Sem 0.0234 LR: 9.053e-02 Score 99.176 Data time: 0.2162, Forward time: 1.2595, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.8780 [2021-07-07 14:56:22,420][root][INFO] - ===> Epoch[1257](6285/5): Loss 0.0241, Sem 0.0241 LR: 9.052e-02 Score 99.149 Data time: 0.2124, Forward time: 1.2596, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.7735 [2021-07-07 14:56:36,776][root][INFO] - ===> Epoch[1258](6290/5): Loss 0.0294, Sem 0.0294 LR: 9.051e-02 Score 99.046 Data time: 0.2087, Forward time: 1.2596, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.8255 [2021-07-07 14:56:50,692][root][INFO] - ===> Epoch[1259](6295/5): Loss 0.0345, Sem 0.0345 LR: 9.051e-02 Score 98.787 Data time: 0.2440, Forward time: 1.2596, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.7396 [2021-07-07 14:57:05,790][root][INFO] - ===> Epoch[1260](6300/5): Loss 0.0376, Sem 0.0376 LR: 9.050e-02 Score 98.778 Data time: 0.2898, Forward time: 1.2597, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.9673 [2021-07-07 14:57:20,277][root][INFO] - ===> Epoch[1261](6305/5): Loss 0.0397, Sem 0.0397 LR: 9.049e-02 Score 98.665 Data time: 0.2372, Forward time: 1.2596, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.8549 [2021-07-07 14:57:34,291][root][INFO] - ===> Epoch[1262](6310/5): Loss 0.0366, Sem 0.0366 LR: 9.048e-02 Score 98.745 Data time: 0.2154, Forward time: 1.2595, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7613 [2021-07-07 14:57:48,649][root][INFO] - ===> Epoch[1263](6315/5): Loss 0.0337, Sem 0.0337 LR: 9.048e-02 Score 98.826 Data time: 0.2255, Forward time: 1.2595, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.8152 [2021-07-07 14:58:02,234][root][INFO] - ===> Epoch[1264](6320/5): Loss 0.0387, Sem 0.0387 LR: 9.047e-02 Score 98.676 Data time: 0.2187, Forward time: 1.2594, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.6730 [2021-07-07 14:58:16,477][root][INFO] - ===> Epoch[1265](6325/5): Loss 0.0409, Sem 0.0409 LR: 9.046e-02 Score 98.667 Data time: 0.1968, Forward time: 1.2594, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7976 [2021-07-07 14:58:30,905][root][INFO] - ===> Epoch[1266](6330/5): Loss 0.0364, Sem 0.0364 LR: 9.045e-02 Score 98.731 Data time: 0.1738, Forward time: 1.2595, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8520 [2021-07-07 14:58:44,404][root][INFO] - ===> Epoch[1267](6335/5): Loss 0.0364, Sem 0.0364 LR: 9.045e-02 Score 98.742 Data time: 0.1862, Forward time: 1.2594, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6505 [2021-07-07 14:58:58,297][root][INFO] - ===> Epoch[1268](6340/5): Loss 0.0381, Sem 0.0381 LR: 9.044e-02 Score 98.690 Data time: 0.2193, Forward time: 1.2594, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7371 [2021-07-07 14:59:13,141][root][INFO] - ===> Epoch[1269](6345/5): Loss 0.0348, Sem 0.0348 LR: 9.043e-02 Score 98.803 Data time: 0.2538, Forward time: 1.2595, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.9289 [2021-07-07 14:59:27,477][root][INFO] - ===> Epoch[1270](6350/5): Loss 0.0398, Sem 0.0398 LR: 9.042e-02 Score 98.635 Data time: 0.2524, Forward time: 1.2595, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8114 [2021-07-07 14:59:41,987][root][INFO] - ===> Epoch[1271](6355/5): Loss 0.0355, Sem 0.0355 LR: 9.042e-02 Score 98.756 Data time: 0.2044, Forward time: 1.2595, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8415 [2021-07-07 14:59:56,538][root][INFO] - ===> Epoch[1272](6360/5): Loss 0.0399, Sem 0.0399 LR: 9.041e-02 Score 98.642 Data time: 0.2546, Forward time: 1.2595, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.8618 [2021-07-07 15:00:10,733][root][INFO] - ===> Epoch[1273](6365/5): Loss 0.0344, Sem 0.0344 LR: 9.040e-02 Score 98.782 Data time: 0.2042, Forward time: 1.2596, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7990 [2021-07-07 15:00:24,914][root][INFO] - ===> Epoch[1274](6370/5): Loss 0.0326, Sem 0.0326 LR: 9.039e-02 Score 98.852 Data time: 0.2235, Forward time: 1.2595, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7861 [2021-07-07 15:00:39,510][root][INFO] - ===> Epoch[1275](6375/5): Loss 0.0342, Sem 0.0342 LR: 9.038e-02 Score 98.819 Data time: 0.1803, Forward time: 1.2596, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.8805 [2021-07-07 15:00:53,358][root][INFO] - ===> Epoch[1276](6380/5): Loss 0.0320, Sem 0.0320 LR: 9.038e-02 Score 98.891 Data time: 0.2168, Forward time: 1.2598, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7177 [2021-07-07 15:01:07,030][root][INFO] - ===> Epoch[1277](6385/5): Loss 0.0326, Sem 0.0326 LR: 9.037e-02 Score 98.873 Data time: 0.2672, Forward time: 1.2598, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6823 [2021-07-07 15:01:21,270][root][INFO] - ===> Epoch[1278](6390/5): Loss 0.0286, Sem 0.0286 LR: 9.036e-02 Score 98.998 Data time: 0.1860, Forward time: 1.2598, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8023 [2021-07-07 15:01:35,522][root][INFO] - ===> Epoch[1279](6395/5): Loss 0.0312, Sem 0.0312 LR: 9.035e-02 Score 98.916 Data time: 0.2233, Forward time: 1.2599, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8045 [2021-07-07 15:01:49,808][root][INFO] - ===> Epoch[1280](6400/5): Loss 0.0289, Sem 0.0289 LR: 9.035e-02 Score 98.989 Data time: 0.3182, Forward time: 1.2600, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.8081 [2021-07-07 15:01:49,811][root][INFO] - ===> Start testing [2021-07-07 15:02:24,467][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1317 Loss 0.245 (AVG: 0.497) Score 94.288 (AVG: 90.124) mIOU 68.048 mAP 82.949 mAcc 74.492 IOU: 61.867 0.000 78.983 74.609 93.974 91.265 26.358 73.888 98.656 63.397 78.401 85.858 57.372 mAP: 84.398 4.960 92.890 88.521 98.699 98.296 60.676 94.621 99.819 74.933 93.791 95.322 91.404 mAcc: 84.593 0.000 86.686 80.592 95.162 97.318 30.944 85.346 99.570 64.848 85.304 95.485 62.549 [2021-07-07 15:02:24,467][root][INFO] - Finished test. Elapsed time: 34.6550 [2021-07-07 15:02:25,577][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 15:02:38,999][root][INFO] - ===> Epoch[1281](6405/5): Loss 0.0294, Sem 0.0294 LR: 9.034e-02 Score 98.961 Data time: 0.1829, Forward time: 1.2600, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.6478 [2021-07-07 15:02:52,851][root][INFO] - ===> Epoch[1282](6410/5): Loss 0.0275, Sem 0.0275 LR: 9.033e-02 Score 99.027 Data time: 0.2561, Forward time: 1.2601, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7244 [2021-07-07 15:03:06,716][root][INFO] - ===> Epoch[1283](6415/5): Loss 0.0284, Sem 0.0284 LR: 9.032e-02 Score 99.001 Data time: 0.1383, Forward time: 1.2601, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7387 [2021-07-07 15:03:20,798][root][INFO] - ===> Epoch[1284](6420/5): Loss 0.0277, Sem 0.0277 LR: 9.032e-02 Score 99.019 Data time: 0.2041, Forward time: 1.2601, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7685 [2021-07-07 15:03:33,887][root][INFO] - ===> Epoch[1285](6425/5): Loss 0.0278, Sem 0.0278 LR: 9.031e-02 Score 99.013 Data time: 0.2161, Forward time: 1.2601, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.5748 [2021-07-07 15:03:48,281][root][INFO] - ===> Epoch[1286](6430/5): Loss 0.0291, Sem 0.0291 LR: 9.030e-02 Score 98.965 Data time: 0.2436, Forward time: 1.2602, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.8300 [2021-07-07 15:04:01,754][root][INFO] - ===> Epoch[1287](6435/5): Loss 0.0291, Sem 0.0291 LR: 9.029e-02 Score 98.976 Data time: 0.2187, Forward time: 1.2602, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.6483 [2021-07-07 15:04:16,210][root][INFO] - ===> Epoch[1288](6440/5): Loss 0.0315, Sem 0.0315 LR: 9.029e-02 Score 98.950 Data time: 0.2040, Forward time: 1.2601, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.8482 [2021-07-07 15:04:30,961][root][INFO] - ===> Epoch[1289](6445/5): Loss 0.0302, Sem 0.0302 LR: 9.028e-02 Score 98.975 Data time: 0.2507, Forward time: 1.2602, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.8975 [2021-07-07 15:04:44,958][root][INFO] - ===> Epoch[1290](6450/5): Loss 0.0276, Sem 0.0276 LR: 9.027e-02 Score 99.040 Data time: 0.2330, Forward time: 1.2601, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7671 [2021-07-07 15:04:59,029][root][INFO] - ===> Epoch[1291](6455/5): Loss 0.0290, Sem 0.0290 LR: 9.026e-02 Score 98.969 Data time: 0.2614, Forward time: 1.2601, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7607 [2021-07-07 15:05:12,960][root][INFO] - ===> Epoch[1292](6460/5): Loss 0.0265, Sem 0.0265 LR: 9.026e-02 Score 99.057 Data time: 0.2468, Forward time: 1.2602, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7422 [2021-07-07 15:05:27,113][root][INFO] - ===> Epoch[1293](6465/5): Loss 0.0265, Sem 0.0265 LR: 9.025e-02 Score 99.067 Data time: 0.2330, Forward time: 1.2602, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7862 [2021-07-07 15:05:42,072][root][INFO] - ===> Epoch[1294](6470/5): Loss 0.0251, Sem 0.0251 LR: 9.024e-02 Score 99.108 Data time: 0.2537, Forward time: 1.2603, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.9450 [2021-07-07 15:05:55,915][root][INFO] - ===> Epoch[1295](6475/5): Loss 0.0262, Sem 0.0262 LR: 9.023e-02 Score 99.072 Data time: 0.2601, Forward time: 1.2603, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7212 [2021-07-07 15:06:10,280][root][INFO] - ===> Epoch[1296](6480/5): Loss 0.0253, Sem 0.0253 LR: 9.023e-02 Score 99.101 Data time: 0.2266, Forward time: 1.2603, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.8232 [2021-07-07 15:06:23,830][root][INFO] - ===> Epoch[1297](6485/5): Loss 0.0261, Sem 0.0261 LR: 9.022e-02 Score 99.091 Data time: 0.2756, Forward time: 1.2604, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.6634 [2021-07-07 15:06:38,756][root][INFO] - ===> Epoch[1298](6490/5): Loss 0.0262, Sem 0.0262 LR: 9.021e-02 Score 99.092 Data time: 0.2267, Forward time: 1.2604, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.9351 [2021-07-07 15:06:52,588][root][INFO] - ===> Epoch[1299](6495/5): Loss 0.0238, Sem 0.0238 LR: 9.020e-02 Score 99.146 Data time: 0.1869, Forward time: 1.2603, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7209 [2021-07-07 15:07:07,991][root][INFO] - ===> Epoch[1300](6500/5): Loss 0.0245, Sem 0.0245 LR: 9.020e-02 Score 99.135 Data time: 0.2197, Forward time: 1.2604, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 3.0359 [2021-07-07 15:07:22,357][root][INFO] - ===> Epoch[1301](6505/5): Loss 0.0234, Sem 0.0234 LR: 9.019e-02 Score 99.165 Data time: 0.1920, Forward time: 1.2604, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8205 [2021-07-07 15:07:36,388][root][INFO] - ===> Epoch[1302](6510/5): Loss 0.0237, Sem 0.0237 LR: 9.018e-02 Score 99.154 Data time: 0.2648, Forward time: 1.2604, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7633 [2021-07-07 15:07:50,684][root][INFO] - ===> Epoch[1303](6515/5): Loss 0.0231, Sem 0.0231 LR: 9.017e-02 Score 99.183 Data time: 0.1870, Forward time: 1.2604, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8168 [2021-07-07 15:08:05,263][root][INFO] - ===> Epoch[1304](6520/5): Loss 0.0236, Sem 0.0236 LR: 9.016e-02 Score 99.164 Data time: 0.2328, Forward time: 1.2605, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8681 [2021-07-07 15:08:20,222][root][INFO] - ===> Epoch[1305](6525/5): Loss 0.0222, Sem 0.0222 LR: 9.016e-02 Score 99.208 Data time: 0.2381, Forward time: 1.2605, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.9552 [2021-07-07 15:08:34,458][root][INFO] - ===> Epoch[1306](6530/5): Loss 0.0230, Sem 0.0230 LR: 9.015e-02 Score 99.174 Data time: 0.1993, Forward time: 1.2606, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.8066 [2021-07-07 15:08:48,456][root][INFO] - ===> Epoch[1307](6535/5): Loss 0.0216, Sem 0.0216 LR: 9.014e-02 Score 99.220 Data time: 0.2157, Forward time: 1.2606, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7413 [2021-07-07 15:09:03,129][root][INFO] - ===> Epoch[1308](6540/5): Loss 0.0222, Sem 0.0222 LR: 9.013e-02 Score 99.203 Data time: 0.3284, Forward time: 1.2607, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8781 [2021-07-07 15:09:17,106][root][INFO] - ===> Epoch[1309](6545/5): Loss 0.0238, Sem 0.0238 LR: 9.013e-02 Score 99.153 Data time: 0.2489, Forward time: 1.2607, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7473 [2021-07-07 15:09:31,149][root][INFO] - ===> Epoch[1310](6550/5): Loss 0.0216, Sem 0.0216 LR: 9.012e-02 Score 99.215 Data time: 0.1402, Forward time: 1.2608, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7629 [2021-07-07 15:09:44,459][root][INFO] - ===> Epoch[1311](6555/5): Loss 0.0226, Sem 0.0226 LR: 9.011e-02 Score 99.188 Data time: 0.1997, Forward time: 1.2607, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6169 [2021-07-07 15:09:58,633][root][INFO] - ===> Epoch[1312](6560/5): Loss 0.0217, Sem 0.0217 LR: 9.010e-02 Score 99.226 Data time: 0.1553, Forward time: 1.2606, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.7953 [2021-07-07 15:10:12,458][root][INFO] - ===> Epoch[1313](6565/5): Loss 0.0222, Sem 0.0222 LR: 9.010e-02 Score 99.204 Data time: 0.2346, Forward time: 1.2607, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7193 [2021-07-07 15:10:26,457][root][INFO] - ===> Epoch[1314](6570/5): Loss 0.0221, Sem 0.0221 LR: 9.009e-02 Score 99.208 Data time: 0.2152, Forward time: 1.2606, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7514 [2021-07-07 15:10:40,672][root][INFO] - ===> Epoch[1315](6575/5): Loss 0.0216, Sem 0.0216 LR: 9.008e-02 Score 99.242 Data time: 0.2885, Forward time: 1.2607, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7932 [2021-07-07 15:10:55,267][root][INFO] - ===> Epoch[1316](6580/5): Loss 0.0225, Sem 0.0225 LR: 9.007e-02 Score 99.188 Data time: 0.2438, Forward time: 1.2608, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8619 [2021-07-07 15:11:09,227][root][INFO] - ===> Epoch[1317](6585/5): Loss 0.0224, Sem 0.0224 LR: 9.007e-02 Score 99.195 Data time: 0.2403, Forward time: 1.2607, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7483 [2021-07-07 15:11:23,202][root][INFO] - ===> Epoch[1318](6590/5): Loss 0.0215, Sem 0.0215 LR: 9.006e-02 Score 99.238 Data time: 0.2189, Forward time: 1.2607, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7382 [2021-07-07 15:11:37,188][root][INFO] - ===> Epoch[1319](6595/5): Loss 0.0224, Sem 0.0224 LR: 9.005e-02 Score 99.193 Data time: 0.2298, Forward time: 1.2608, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7443 [2021-07-07 15:11:50,795][root][INFO] - ===> Epoch[1320](6600/5): Loss 0.0247, Sem 0.0247 LR: 9.004e-02 Score 99.142 Data time: 0.2370, Forward time: 1.2606, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.6771 [2021-07-07 15:11:50,798][root][INFO] - ===> Start testing [2021-07-07 15:12:27,065][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1535 Loss 0.264 (AVG: 0.491) Score 92.823 (AVG: 91.001) mIOU 70.894 mAP 85.393 mAcc 77.125 IOU: 63.567 0.171 81.624 76.286 95.055 91.316 39.530 81.223 98.559 70.500 81.798 86.625 55.368 mAP: 84.547 20.696 91.884 89.694 98.740 98.734 67.907 97.449 99.835 80.483 93.308 95.029 91.809 mAcc: 81.793 1.550 90.431 82.664 96.547 96.432 43.378 92.569 99.368 72.218 89.295 95.922 60.456 [2021-07-07 15:12:27,065][root][INFO] - Finished test. Elapsed time: 36.2658 [2021-07-07 15:12:28,166][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 15:12:40,981][root][INFO] - ===> Epoch[1321](6605/5): Loss 0.0230, Sem 0.0230 LR: 9.004e-02 Score 99.173 Data time: 0.2339, Forward time: 1.2606, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.5116 [2021-07-07 15:12:54,796][root][INFO] - ===> Epoch[1322](6610/5): Loss 0.0227, Sem 0.0227 LR: 9.003e-02 Score 99.185 Data time: 0.2239, Forward time: 1.2606, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7131 [2021-07-07 15:13:09,035][root][INFO] - ===> Epoch[1323](6615/5): Loss 0.0225, Sem 0.0225 LR: 9.002e-02 Score 99.182 Data time: 0.2192, Forward time: 1.2606, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7908 [2021-07-07 15:13:22,894][root][INFO] - ===> Epoch[1324](6620/5): Loss 0.0229, Sem 0.0229 LR: 9.001e-02 Score 99.181 Data time: 0.2165, Forward time: 1.2606, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7238 [2021-07-07 15:13:35,865][root][INFO] - ===> Epoch[1325](6625/5): Loss 0.0217, Sem 0.0217 LR: 9.001e-02 Score 99.228 Data time: 0.2377, Forward time: 1.2606, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.5559 [2021-07-07 15:13:50,503][root][INFO] - ===> Epoch[1326](6630/5): Loss 0.0208, Sem 0.0208 LR: 9.000e-02 Score 99.248 Data time: 0.2351, Forward time: 1.2606, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.8810 [2021-07-07 15:14:04,704][root][INFO] - ===> Epoch[1327](6635/5): Loss 0.0207, Sem 0.0207 LR: 8.999e-02 Score 99.248 Data time: 0.1891, Forward time: 1.2605, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7977 [2021-07-07 15:14:18,659][root][INFO] - ===> Epoch[1328](6640/5): Loss 0.0206, Sem 0.0206 LR: 8.998e-02 Score 99.264 Data time: 0.1862, Forward time: 1.2605, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7473 [2021-07-07 15:14:32,642][root][INFO] - ===> Epoch[1329](6645/5): Loss 0.0209, Sem 0.0209 LR: 8.998e-02 Score 99.247 Data time: 0.1703, Forward time: 1.2605, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7568 [2021-07-07 15:14:46,645][root][INFO] - ===> Epoch[1330](6650/5): Loss 0.0204, Sem 0.0204 LR: 8.997e-02 Score 99.270 Data time: 0.2307, Forward time: 1.2606, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7512 [2021-07-07 15:14:59,755][root][INFO] - ===> Epoch[1331](6655/5): Loss 0.0203, Sem 0.0203 LR: 8.996e-02 Score 99.268 Data time: 0.2307, Forward time: 1.2605, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.5704 [2021-07-07 15:15:14,554][root][INFO] - ===> Epoch[1332](6660/5): Loss 0.0198, Sem 0.0198 LR: 8.995e-02 Score 99.291 Data time: 0.2392, Forward time: 1.2605, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.9108 [2021-07-07 15:15:28,838][root][INFO] - ===> Epoch[1333](6665/5): Loss 0.0203, Sem 0.0203 LR: 8.994e-02 Score 99.271 Data time: 0.2927, Forward time: 1.2606, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7957 [2021-07-07 15:15:43,743][root][INFO] - ===> Epoch[1334](6670/5): Loss 0.0205, Sem 0.0205 LR: 8.994e-02 Score 99.274 Data time: 0.2577, Forward time: 1.2607, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.9321 [2021-07-07 15:15:58,049][root][INFO] - ===> Epoch[1335](6675/5): Loss 0.0199, Sem 0.0199 LR: 8.993e-02 Score 99.289 Data time: 0.2469, Forward time: 1.2607, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8179 [2021-07-07 15:16:12,447][root][INFO] - ===> Epoch[1336](6680/5): Loss 0.0199, Sem 0.0199 LR: 8.992e-02 Score 99.282 Data time: 0.2307, Forward time: 1.2607, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8291 [2021-07-07 15:16:25,991][root][INFO] - ===> Epoch[1337](6685/5): Loss 0.0210, Sem 0.0210 LR: 8.991e-02 Score 99.245 Data time: 0.2339, Forward time: 1.2607, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6680 [2021-07-07 15:16:39,810][root][INFO] - ===> Epoch[1338](6690/5): Loss 0.0214, Sem 0.0214 LR: 8.991e-02 Score 99.230 Data time: 0.1987, Forward time: 1.2608, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7153 [2021-07-07 15:16:53,979][root][INFO] - ===> Epoch[1339](6695/5): Loss 0.0229, Sem 0.0229 LR: 8.990e-02 Score 99.178 Data time: 0.1926, Forward time: 1.2608, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7899 [2021-07-07 15:17:08,185][root][INFO] - ===> Epoch[1340](6700/5): Loss 0.0226, Sem 0.0226 LR: 8.989e-02 Score 99.200 Data time: 0.2402, Forward time: 1.2608, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7965 [2021-07-07 15:17:23,050][root][INFO] - ===> Epoch[1341](6705/5): Loss 0.0235, Sem 0.0235 LR: 8.988e-02 Score 99.182 Data time: 0.2564, Forward time: 1.2609, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.9194 [2021-07-07 15:17:36,199][root][INFO] - ===> Epoch[1342](6710/5): Loss 0.0244, Sem 0.0244 LR: 8.988e-02 Score 99.120 Data time: 0.2547, Forward time: 1.2608, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.5787 [2021-07-07 15:17:50,044][root][INFO] - ===> Epoch[1343](6715/5): Loss 0.0220, Sem 0.0220 LR: 8.987e-02 Score 99.216 Data time: 0.1760, Forward time: 1.2609, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7236 [2021-07-07 15:18:04,393][root][INFO] - ===> Epoch[1344](6720/5): Loss 0.0232, Sem 0.0232 LR: 8.986e-02 Score 99.164 Data time: 0.1975, Forward time: 1.2610, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.8301 [2021-07-07 15:18:18,082][root][INFO] - ===> Epoch[1345](6725/5): Loss 0.0222, Sem 0.0222 LR: 8.985e-02 Score 99.201 Data time: 0.2076, Forward time: 1.2609, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.6873 [2021-07-07 15:18:32,612][root][INFO] - ===> Epoch[1346](6730/5): Loss 0.0213, Sem 0.0213 LR: 8.985e-02 Score 99.230 Data time: 0.2457, Forward time: 1.2610, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8610 [2021-07-07 15:18:46,003][root][INFO] - ===> Epoch[1347](6735/5): Loss 0.0221, Sem 0.0221 LR: 8.984e-02 Score 99.202 Data time: 0.2160, Forward time: 1.2609, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6352 [2021-07-07 15:18:59,518][root][INFO] - ===> Epoch[1348](6740/5): Loss 0.0213, Sem 0.0213 LR: 8.983e-02 Score 99.235 Data time: 0.2191, Forward time: 1.2609, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.6431 [2021-07-07 15:19:13,649][root][INFO] - ===> Epoch[1349](6745/5): Loss 0.0213, Sem 0.0213 LR: 8.982e-02 Score 99.238 Data time: 0.2664, Forward time: 1.2609, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7745 [2021-07-07 15:19:28,160][root][INFO] - ===> Epoch[1350](6750/5): Loss 0.0214, Sem 0.0214 LR: 8.982e-02 Score 99.236 Data time: 0.2860, Forward time: 1.2610, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.8503 [2021-07-07 15:19:42,721][root][INFO] - ===> Epoch[1351](6755/5): Loss 0.0207, Sem 0.0207 LR: 8.981e-02 Score 99.257 Data time: 0.2122, Forward time: 1.2611, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.8759 [2021-07-07 15:19:56,668][root][INFO] - ===> Epoch[1352](6760/5): Loss 0.0213, Sem 0.0213 LR: 8.980e-02 Score 99.240 Data time: 0.2184, Forward time: 1.2611, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7502 [2021-07-07 15:20:10,840][root][INFO] - ===> Epoch[1353](6765/5): Loss 0.0214, Sem 0.0214 LR: 8.979e-02 Score 99.234 Data time: 0.2776, Forward time: 1.2611, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7866 [2021-07-07 15:20:24,645][root][INFO] - ===> Epoch[1354](6770/5): Loss 0.0217, Sem 0.0217 LR: 8.979e-02 Score 99.225 Data time: 0.2084, Forward time: 1.2612, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7142 [2021-07-07 15:20:38,677][root][INFO] - ===> Epoch[1355](6775/5): Loss 0.0216, Sem 0.0216 LR: 8.978e-02 Score 99.226 Data time: 0.2276, Forward time: 1.2613, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7656 [2021-07-07 15:20:52,296][root][INFO] - ===> Epoch[1356](6780/5): Loss 0.0214, Sem 0.0214 LR: 8.977e-02 Score 99.237 Data time: 0.2458, Forward time: 1.2612, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.6694 [2021-07-07 15:21:06,894][root][INFO] - ===> Epoch[1357](6785/5): Loss 0.0222, Sem 0.0222 LR: 8.976e-02 Score 99.225 Data time: 0.3245, Forward time: 1.2612, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.8592 [2021-07-07 15:21:21,034][root][INFO] - ===> Epoch[1358](6790/5): Loss 0.0207, Sem 0.0207 LR: 8.975e-02 Score 99.250 Data time: 0.2491, Forward time: 1.2613, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.7584 [2021-07-07 15:21:35,077][root][INFO] - ===> Epoch[1359](6795/5): Loss 0.0209, Sem 0.0209 LR: 8.975e-02 Score 99.253 Data time: 0.2374, Forward time: 1.2613, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7654 [2021-07-07 15:21:49,228][root][INFO] - ===> Epoch[1360](6800/5): Loss 0.0202, Sem 0.0202 LR: 8.974e-02 Score 99.268 Data time: 0.1236, Forward time: 1.2612, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7856 [2021-07-07 15:21:49,230][root][INFO] - ===> Start testing [2021-07-07 15:22:25,863][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1267 Loss 0.256 (AVG: 0.489) Score 93.610 (AVG: 91.049) mIOU 69.614 mAP 84.887 mAcc 76.144 IOU: 64.952 0.000 80.130 77.149 95.155 90.208 21.901 79.348 98.532 72.009 79.600 86.629 59.366 mAP: 84.530 12.329 95.479 90.365 99.088 98.257 55.976 97.965 99.832 89.564 92.430 94.891 92.823 mAcc: 81.784 0.000 90.050 84.274 96.434 96.448 22.838 93.502 99.469 76.066 88.246 96.223 64.541 [2021-07-07 15:22:25,864][root][INFO] - Finished test. Elapsed time: 36.6329 [2021-07-07 15:22:26,930][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 15:22:40,105][root][INFO] - ===> Epoch[1361](6805/5): Loss 0.0210, Sem 0.0210 LR: 8.973e-02 Score 99.249 Data time: 0.2225, Forward time: 1.2612, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.5857 [2021-07-07 15:22:54,438][root][INFO] - ===> Epoch[1362](6810/5): Loss 0.0207, Sem 0.0207 LR: 8.972e-02 Score 99.257 Data time: 0.2321, Forward time: 1.2613, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.8240 [2021-07-07 15:23:08,022][root][INFO] - ===> Epoch[1363](6815/5): Loss 0.0204, Sem 0.0204 LR: 8.972e-02 Score 99.274 Data time: 0.2158, Forward time: 1.2613, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.6609 [2021-07-07 15:23:21,409][root][INFO] - ===> Epoch[1364](6820/5): Loss 0.0196, Sem 0.0196 LR: 8.971e-02 Score 99.293 Data time: 0.2681, Forward time: 1.2613, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.6147 [2021-07-07 15:23:35,621][root][INFO] - ===> Epoch[1365](6825/5): Loss 0.0201, Sem 0.0201 LR: 8.970e-02 Score 99.276 Data time: 0.2298, Forward time: 1.2613, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.7968 [2021-07-07 15:23:49,154][root][INFO] - ===> Epoch[1366](6830/5): Loss 0.0199, Sem 0.0199 LR: 8.969e-02 Score 99.283 Data time: 0.2246, Forward time: 1.2612, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.6571 [2021-07-07 15:24:02,799][root][INFO] - ===> Epoch[1367](6835/5): Loss 0.0206, Sem 0.0206 LR: 8.969e-02 Score 99.260 Data time: 0.2152, Forward time: 1.2613, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.6777 [2021-07-07 15:24:16,958][root][INFO] - ===> Epoch[1368](6840/5): Loss 0.0197, Sem 0.0197 LR: 8.968e-02 Score 99.291 Data time: 0.2162, Forward time: 1.2613, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.7842 [2021-07-07 15:24:32,684][root][INFO] - ===> Epoch[1369](6845/5): Loss 0.0197, Sem 0.0197 LR: 8.967e-02 Score 99.289 Data time: 0.2077, Forward time: 1.2613, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 3.1036 [2021-07-07 15:24:47,259][root][INFO] - ===> Epoch[1370](6850/5): Loss 0.0200, Sem 0.0200 LR: 8.966e-02 Score 99.274 Data time: 0.2490, Forward time: 1.2613, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.8613 [2021-07-07 15:25:02,002][root][INFO] - ===> Epoch[1371](6855/5): Loss 0.0193, Sem 0.0193 LR: 8.966e-02 Score 99.301 Data time: 0.2636, Forward time: 1.2613, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.8922 [2021-07-07 15:25:16,435][root][INFO] - ===> Epoch[1372](6860/5): Loss 0.0204, Sem 0.0204 LR: 8.965e-02 Score 99.276 Data time: 0.2175, Forward time: 1.2613, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.8329 [2021-07-07 15:25:30,205][root][INFO] - ===> Epoch[1373](6865/5): Loss 0.0209, Sem 0.0209 LR: 8.964e-02 Score 99.254 Data time: 0.1789, Forward time: 1.2613, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7129 [2021-07-07 15:25:44,619][root][INFO] - ===> Epoch[1374](6870/5): Loss 0.0200, Sem 0.0200 LR: 8.963e-02 Score 99.275 Data time: 0.2666, Forward time: 1.2613, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.8215 [2021-07-07 15:25:58,419][root][INFO] - ===> Epoch[1375](6875/5): Loss 0.0191, Sem 0.0191 LR: 8.963e-02 Score 99.311 Data time: 0.1734, Forward time: 1.2613, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7141 [2021-07-07 15:26:12,497][root][INFO] - ===> Epoch[1376](6880/5): Loss 0.0199, Sem 0.0199 LR: 8.962e-02 Score 99.288 Data time: 0.2617, Forward time: 1.2613, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7560 [2021-07-07 15:26:26,310][root][INFO] - ===> Epoch[1377](6885/5): Loss 0.0196, Sem 0.0196 LR: 8.961e-02 Score 99.294 Data time: 0.2163, Forward time: 1.2613, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7038 [2021-07-07 15:26:40,126][root][INFO] - ===> Epoch[1378](6890/5): Loss 0.0202, Sem 0.0202 LR: 8.960e-02 Score 99.269 Data time: 0.2166, Forward time: 1.2614, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7177 [2021-07-07 15:26:54,436][root][INFO] - ===> Epoch[1379](6895/5): Loss 0.0186, Sem 0.0186 LR: 8.960e-02 Score 99.331 Data time: 0.2190, Forward time: 1.2614, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.8141 [2021-07-07 15:27:07,956][root][INFO] - ===> Epoch[1380](6900/5): Loss 0.0187, Sem 0.0187 LR: 8.959e-02 Score 99.327 Data time: 0.2521, Forward time: 1.2615, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.6529 [2021-07-07 15:27:22,664][root][INFO] - ===> Epoch[1381](6905/5): Loss 0.0193, Sem 0.0193 LR: 8.958e-02 Score 99.304 Data time: 0.2055, Forward time: 1.2616, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.8882 [2021-07-07 15:27:35,714][root][INFO] - ===> Epoch[1382](6910/5): Loss 0.0189, Sem 0.0189 LR: 8.957e-02 Score 99.319 Data time: 0.2294, Forward time: 1.2615, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.5621 [2021-07-07 15:27:50,312][root][INFO] - ===> Epoch[1383](6915/5): Loss 0.0195, Sem 0.0195 LR: 8.957e-02 Score 99.292 Data time: 0.2756, Forward time: 1.2615, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.8767 [2021-07-07 15:28:04,264][root][INFO] - ===> Epoch[1384](6920/5): Loss 0.0201, Sem 0.0201 LR: 8.956e-02 Score 99.278 Data time: 0.1909, Forward time: 1.2615, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7475 [2021-07-07 15:28:17,832][root][INFO] - ===> Epoch[1385](6925/5): Loss 0.0196, Sem 0.0196 LR: 8.955e-02 Score 99.300 Data time: 0.2406, Forward time: 1.2615, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.6671 [2021-07-07 15:28:32,456][root][INFO] - ===> Epoch[1386](6930/5): Loss 0.0195, Sem 0.0195 LR: 8.954e-02 Score 99.294 Data time: 0.2221, Forward time: 1.2615, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.8807 [2021-07-07 15:28:47,067][root][INFO] - ===> Epoch[1387](6935/5): Loss 0.0193, Sem 0.0193 LR: 8.953e-02 Score 99.304 Data time: 0.2404, Forward time: 1.2614, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.8835 [2021-07-07 15:29:00,785][root][INFO] - ===> Epoch[1388](6940/5): Loss 0.0198, Sem 0.0198 LR: 8.953e-02 Score 99.283 Data time: 0.2712, Forward time: 1.2614, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.6933 [2021-07-07 15:29:15,480][root][INFO] - ===> Epoch[1389](6945/5): Loss 0.0195, Sem 0.0195 LR: 8.952e-02 Score 99.297 Data time: 0.1684, Forward time: 1.2613, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.8881 [2021-07-07 15:29:29,180][root][INFO] - ===> Epoch[1390](6950/5): Loss 0.0190, Sem 0.0190 LR: 8.951e-02 Score 99.319 Data time: 0.2040, Forward time: 1.2613, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.6915 [2021-07-07 15:29:43,636][root][INFO] - ===> Epoch[1391](6955/5): Loss 0.0198, Sem 0.0198 LR: 8.950e-02 Score 99.292 Data time: 0.2412, Forward time: 1.2613, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.8431 [2021-07-07 15:29:57,276][root][INFO] - ===> Epoch[1392](6960/5): Loss 0.0198, Sem 0.0198 LR: 8.950e-02 Score 99.280 Data time: 0.2445, Forward time: 1.2614, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.6845 [2021-07-07 15:30:11,192][root][INFO] - ===> Epoch[1393](6965/5): Loss 0.0207, Sem 0.0207 LR: 8.949e-02 Score 99.254 Data time: 0.1795, Forward time: 1.2613, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7417 [2021-07-07 15:30:24,947][root][INFO] - ===> Epoch[1394](6970/5): Loss 0.0206, Sem 0.0206 LR: 8.948e-02 Score 99.256 Data time: 0.2610, Forward time: 1.2614, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7069 [2021-07-07 15:30:39,170][root][INFO] - ===> Epoch[1395](6975/5): Loss 0.0209, Sem 0.0209 LR: 8.947e-02 Score 99.262 Data time: 0.1975, Forward time: 1.2614, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.7994 [2021-07-07 15:30:52,312][root][INFO] - ===> Epoch[1396](6980/5): Loss 0.0192, Sem 0.0192 LR: 8.947e-02 Score 99.316 Data time: 0.1934, Forward time: 1.2614, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.5811 [2021-07-07 15:31:05,584][root][INFO] - ===> Epoch[1397](6985/5): Loss 0.0199, Sem 0.0199 LR: 8.946e-02 Score 99.286 Data time: 0.2292, Forward time: 1.2613, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.5994 [2021-07-07 15:31:19,698][root][INFO] - ===> Epoch[1398](6990/5): Loss 0.0190, Sem 0.0190 LR: 8.945e-02 Score 99.318 Data time: 0.2148, Forward time: 1.2613, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7772 [2021-07-07 15:31:33,799][root][INFO] - ===> Epoch[1399](6995/5): Loss 0.0184, Sem 0.0184 LR: 8.944e-02 Score 99.340 Data time: 0.2388, Forward time: 1.2613, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7711 [2021-07-07 15:31:48,812][root][INFO] - ===> Epoch[1400](7000/5): Loss 0.0187, Sem 0.0187 LR: 8.944e-02 Score 99.326 Data time: 0.2669, Forward time: 1.2614, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.9526 [2021-07-07 15:31:48,813][root][INFO] - ===> Start testing [2021-07-07 15:32:25,154][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1312 Loss 0.354 (AVG: 0.491) Score 91.053 (AVG: 91.256) mIOU 71.367 mAP 85.250 mAcc 77.844 IOU: 64.528 0.129 80.900 78.100 94.930 92.031 34.427 79.515 98.408 78.982 80.365 87.023 58.435 mAP: 84.606 13.529 92.226 91.072 98.917 98.751 66.101 97.329 99.833 85.000 93.081 95.539 92.267 mAcc: 79.657 1.185 90.063 85.012 96.541 96.821 37.446 91.754 98.984 84.086 91.038 96.621 62.765 [2021-07-07 15:32:25,155][root][INFO] - Finished test. Elapsed time: 36.3399 [2021-07-07 15:32:26,223][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 15:32:39,450][root][INFO] - ===> Epoch[1401](7005/5): Loss 0.0193, Sem 0.0193 LR: 8.943e-02 Score 99.304 Data time: 0.2038, Forward time: 1.2613, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.5957 [2021-07-07 15:32:53,326][root][INFO] - ===> Epoch[1402](7010/5): Loss 0.0187, Sem 0.0187 LR: 8.942e-02 Score 99.334 Data time: 0.2508, Forward time: 1.2614, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7325 [2021-07-07 15:33:06,838][root][INFO] - ===> Epoch[1403](7015/5): Loss 0.0198, Sem 0.0198 LR: 8.941e-02 Score 99.291 Data time: 0.1732, Forward time: 1.2614, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.6537 [2021-07-07 15:33:21,410][root][INFO] - ===> Epoch[1404](7020/5): Loss 0.0188, Sem 0.0188 LR: 8.941e-02 Score 99.322 Data time: 0.2302, Forward time: 1.2615, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.8631 [2021-07-07 15:33:36,218][root][INFO] - ===> Epoch[1405](7025/5): Loss 0.0195, Sem 0.0195 LR: 8.940e-02 Score 99.306 Data time: 0.2688, Forward time: 1.2617, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.9058 [2021-07-07 15:33:50,273][root][INFO] - ===> Epoch[1406](7030/5): Loss 0.0195, Sem 0.0195 LR: 8.939e-02 Score 99.289 Data time: 0.1931, Forward time: 1.2616, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.7725 [2021-07-07 15:34:04,226][root][INFO] - ===> Epoch[1407](7035/5): Loss 0.0199, Sem 0.0199 LR: 8.938e-02 Score 99.278 Data time: 0.2191, Forward time: 1.2617, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.7467 [2021-07-07 15:34:17,891][root][INFO] - ===> Epoch[1408](7040/5): Loss 0.0190, Sem 0.0190 LR: 8.938e-02 Score 99.312 Data time: 0.2584, Forward time: 1.2617, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.6841 [2021-07-07 15:34:32,522][root][INFO] - ===> Epoch[1409](7045/5): Loss 0.0198, Sem 0.0198 LR: 8.937e-02 Score 99.299 Data time: 0.2738, Forward time: 1.2618, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.8723 [2021-07-07 15:34:46,786][root][INFO] - ===> Epoch[1410](7050/5): Loss 0.0190, Sem 0.0190 LR: 8.936e-02 Score 99.313 Data time: 0.1899, Forward time: 1.2618, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.8112 [2021-07-07 15:35:00,757][root][INFO] - ===> Epoch[1411](7055/5): Loss 0.0193, Sem 0.0193 LR: 8.935e-02 Score 99.305 Data time: 0.2350, Forward time: 1.2618, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.7516 [2021-07-07 15:35:14,148][root][INFO] - ===> Epoch[1412](7060/5): Loss 0.0199, Sem 0.0199 LR: 8.935e-02 Score 99.294 Data time: 0.2364, Forward time: 1.2617, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.6377 [2021-07-07 15:35:28,529][root][INFO] - ===> Epoch[1413](7065/5): Loss 0.0197, Sem 0.0197 LR: 8.934e-02 Score 99.289 Data time: 0.2639, Forward time: 1.2618, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.8330 [2021-07-07 15:35:42,097][root][INFO] - ===> Epoch[1414](7070/5): Loss 0.0192, Sem 0.0192 LR: 8.933e-02 Score 99.313 Data time: 0.2224, Forward time: 1.2618, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.6632 [2021-07-07 15:35:56,625][root][INFO] - ===> Epoch[1415](7075/5): Loss 0.0193, Sem 0.0193 LR: 8.932e-02 Score 99.308 Data time: 0.2050, Forward time: 1.2618, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.8623 [2021-07-07 15:36:10,886][root][INFO] - ===> Epoch[1416](7080/5): Loss 0.0193, Sem 0.0193 LR: 8.931e-02 Score 99.309 Data time: 0.1962, Forward time: 1.2618, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.8001 [2021-07-07 15:36:24,627][root][INFO] - ===> Epoch[1417](7085/5): Loss 0.0192, Sem 0.0192 LR: 8.931e-02 Score 99.317 Data time: 0.2087, Forward time: 1.2618, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.7120 [2021-07-07 15:36:38,601][root][INFO] - ===> Epoch[1418](7090/5): Loss 0.0200, Sem 0.0200 LR: 8.930e-02 Score 99.298 Data time: 0.1720, Forward time: 1.2617, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.7526 [2021-07-07 15:36:52,121][root][INFO] - ===> Epoch[1419](7095/5): Loss 0.0203, Sem 0.0203 LR: 8.929e-02 Score 99.273 Data time: 0.1880, Forward time: 1.2617, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.6678 [2021-07-07 15:37:06,075][root][INFO] - ===> Epoch[1420](7100/5): Loss 0.0202, Sem 0.0202 LR: 8.928e-02 Score 99.286 Data time: 0.3036, Forward time: 1.2618, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.7402 [2021-07-07 15:37:20,076][root][INFO] - ===> Epoch[1421](7105/5): Loss 0.0207, Sem 0.0207 LR: 8.928e-02 Score 99.274 Data time: 0.2467, Forward time: 1.2618, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.7578 [2021-07-07 15:37:33,055][root][INFO] - ===> Epoch[1422](7110/5): Loss 0.0197, Sem 0.0197 LR: 8.927e-02 Score 99.297 Data time: 0.2788, Forward time: 1.2617, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.5419 [2021-07-07 15:37:47,074][root][INFO] - ===> Epoch[1423](7115/5): Loss 0.0197, Sem 0.0197 LR: 8.926e-02 Score 99.293 Data time: 0.1983, Forward time: 1.2617, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.7606 [2021-07-07 15:38:00,746][root][INFO] - ===> Epoch[1424](7120/5): Loss 0.0194, Sem 0.0194 LR: 8.925e-02 Score 99.308 Data time: 0.2241, Forward time: 1.2617, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.6846 [2021-07-07 15:38:14,631][root][INFO] - ===> Epoch[1425](7125/5): Loss 0.0186, Sem 0.0186 LR: 8.925e-02 Score 99.325 Data time: 0.2891, Forward time: 1.2617, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.7272 [2021-07-07 15:38:28,413][root][INFO] - ===> Epoch[1426](7130/5): Loss 0.0190, Sem 0.0190 LR: 8.924e-02 Score 99.314 Data time: 0.2838, Forward time: 1.2617, Backward time: 1.2288, DDP time: 0.0042, Total iter time: 2.7069 [2021-07-07 15:38:42,382][root][INFO] - ===> Epoch[1427](7135/5): Loss 0.0184, Sem 0.0184 LR: 8.923e-02 Score 99.334 Data time: 0.2098, Forward time: 1.2617, Backward time: 1.2288, DDP time: 0.0042, Total iter time: 2.7508 [2021-07-07 15:38:56,646][root][INFO] - ===> Epoch[1428](7140/5): Loss 0.0189, Sem 0.0189 LR: 8.922e-02 Score 99.318 Data time: 0.2556, Forward time: 1.2616, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.8082 [2021-07-07 15:39:10,130][root][INFO] - ===> Epoch[1429](7145/5): Loss 0.0191, Sem 0.0191 LR: 8.922e-02 Score 99.317 Data time: 0.1913, Forward time: 1.2616, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.6455 [2021-07-07 15:39:23,943][root][INFO] - ===> Epoch[1430](7150/5): Loss 0.0192, Sem 0.0192 LR: 8.921e-02 Score 99.303 Data time: 0.1924, Forward time: 1.2615, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.7091 [2021-07-07 15:39:37,850][root][INFO] - ===> Epoch[1431](7155/5): Loss 0.0190, Sem 0.0190 LR: 8.920e-02 Score 99.314 Data time: 0.2226, Forward time: 1.2616, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.7256 [2021-07-07 15:39:52,308][root][INFO] - ===> Epoch[1432](7160/5): Loss 0.0185, Sem 0.0185 LR: 8.919e-02 Score 99.331 Data time: 0.2405, Forward time: 1.2616, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.8359 [2021-07-07 15:40:06,982][root][INFO] - ===> Epoch[1433](7165/5): Loss 0.0180, Sem 0.0180 LR: 8.919e-02 Score 99.351 Data time: 0.2706, Forward time: 1.2617, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.8918 [2021-07-07 15:40:20,445][root][INFO] - ===> Epoch[1434](7170/5): Loss 0.0187, Sem 0.0187 LR: 8.918e-02 Score 99.324 Data time: 0.2792, Forward time: 1.2616, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.6454 [2021-07-07 15:40:34,259][root][INFO] - ===> Epoch[1435](7175/5): Loss 0.0191, Sem 0.0191 LR: 8.917e-02 Score 99.323 Data time: 0.2011, Forward time: 1.2616, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.7133 [2021-07-07 15:40:47,938][root][INFO] - ===> Epoch[1436](7180/5): Loss 0.0185, Sem 0.0185 LR: 8.916e-02 Score 99.329 Data time: 0.1859, Forward time: 1.2616, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.6962 [2021-07-07 15:41:01,980][root][INFO] - ===> Epoch[1437](7185/5): Loss 0.0188, Sem 0.0188 LR: 8.916e-02 Score 99.331 Data time: 0.1746, Forward time: 1.2616, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.7690 [2021-07-07 15:41:15,793][root][INFO] - ===> Epoch[1438](7190/5): Loss 0.0180, Sem 0.0180 LR: 8.915e-02 Score 99.346 Data time: 0.2926, Forward time: 1.2615, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.7225 [2021-07-07 15:41:29,110][root][INFO] - ===> Epoch[1439](7195/5): Loss 0.0185, Sem 0.0185 LR: 8.914e-02 Score 99.332 Data time: 0.1693, Forward time: 1.2615, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.6176 [2021-07-07 15:41:43,394][root][INFO] - ===> Epoch[1440](7200/5): Loss 0.0195, Sem 0.0195 LR: 8.913e-02 Score 99.298 Data time: 0.2167, Forward time: 1.2614, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.8111 [2021-07-07 15:41:43,395][root][INFO] - ===> Start testing [2021-07-07 15:42:18,787][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1310 Loss 0.250 (AVG: 0.498) Score 93.404 (AVG: 91.030) mIOU 70.326 mAP 84.404 mAcc 76.833 IOU: 63.958 0.000 78.661 75.999 94.915 91.769 36.054 79.706 98.526 68.905 80.657 87.144 57.940 mAP: 84.729 9.806 93.774 89.052 98.827 98.543 64.679 97.817 99.814 80.111 92.891 94.932 92.273 mAcc: 81.870 0.000 89.756 82.582 96.346 95.814 39.866 93.478 99.245 71.379 89.125 96.108 63.259 [2021-07-07 15:42:18,788][root][INFO] - Finished test. Elapsed time: 35.3917 [2021-07-07 15:42:19,879][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 15:42:32,919][root][INFO] - ===> Epoch[1441](7205/5): Loss 0.0185, Sem 0.0185 LR: 8.912e-02 Score 99.338 Data time: 0.2211, Forward time: 1.2614, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.5642 [2021-07-07 15:42:46,986][root][INFO] - ===> Epoch[1442](7210/5): Loss 0.0186, Sem 0.0186 LR: 8.912e-02 Score 99.323 Data time: 0.2564, Forward time: 1.2614, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.7630 [2021-07-07 15:43:00,894][root][INFO] - ===> Epoch[1443](7215/5): Loss 0.0178, Sem 0.0178 LR: 8.911e-02 Score 99.365 Data time: 0.2548, Forward time: 1.2615, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.7383 [2021-07-07 15:43:14,750][root][INFO] - ===> Epoch[1444](7220/5): Loss 0.0186, Sem 0.0186 LR: 8.910e-02 Score 99.332 Data time: 0.2592, Forward time: 1.2615, Backward time: 1.2288, DDP time: 0.0042, Total iter time: 2.7222 [2021-07-07 15:43:28,051][root][INFO] - ===> Epoch[1445](7225/5): Loss 0.0184, Sem 0.0184 LR: 8.909e-02 Score 99.337 Data time: 0.1941, Forward time: 1.2615, Backward time: 1.2287, DDP time: 0.0042, Total iter time: 2.6051 [2021-07-07 15:43:41,775][root][INFO] - ===> Epoch[1446](7230/5): Loss 0.0179, Sem 0.0179 LR: 8.909e-02 Score 99.358 Data time: 0.1752, Forward time: 1.2615, Backward time: 1.2287, DDP time: 0.0042, Total iter time: 2.7080 [2021-07-07 15:43:56,612][root][INFO] - ===> Epoch[1447](7235/5): Loss 0.0190, Sem 0.0190 LR: 8.908e-02 Score 99.331 Data time: 0.2570, Forward time: 1.2616, Backward time: 1.2287, DDP time: 0.0042, Total iter time: 2.9235 [2021-07-07 15:44:10,950][root][INFO] - ===> Epoch[1448](7240/5): Loss 0.0174, Sem 0.0174 LR: 8.907e-02 Score 99.368 Data time: 0.1865, Forward time: 1.2616, Backward time: 1.2288, DDP time: 0.0042, Total iter time: 2.8250 [2021-07-07 15:44:25,115][root][INFO] - ===> Epoch[1449](7245/5): Loss 0.0181, Sem 0.0181 LR: 8.906e-02 Score 99.342 Data time: 0.1627, Forward time: 1.2615, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.7930 [2021-07-07 15:44:38,905][root][INFO] - ===> Epoch[1450](7250/5): Loss 0.0178, Sem 0.0178 LR: 8.906e-02 Score 99.354 Data time: 0.2163, Forward time: 1.2615, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.7029 [2021-07-07 15:44:52,904][root][INFO] - ===> Epoch[1451](7255/5): Loss 0.0187, Sem 0.0187 LR: 8.905e-02 Score 99.323 Data time: 0.2567, Forward time: 1.2616, Backward time: 1.2288, DDP time: 0.0042, Total iter time: 2.7553 [2021-07-07 15:45:07,894][root][INFO] - ===> Epoch[1452](7260/5): Loss 0.0186, Sem 0.0186 LR: 8.904e-02 Score 99.335 Data time: 0.2849, Forward time: 1.2616, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.9516 [2021-07-07 15:45:22,540][root][INFO] - ===> Epoch[1453](7265/5): Loss 0.0178, Sem 0.0178 LR: 8.903e-02 Score 99.349 Data time: 0.2298, Forward time: 1.2616, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.8722 [2021-07-07 15:45:36,933][root][INFO] - ===> Epoch[1454](7270/5): Loss 0.0176, Sem 0.0176 LR: 8.903e-02 Score 99.372 Data time: 0.1680, Forward time: 1.2615, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.8415 [2021-07-07 15:45:51,932][root][INFO] - ===> Epoch[1455](7275/5): Loss 0.0180, Sem 0.0180 LR: 8.902e-02 Score 99.347 Data time: 0.2037, Forward time: 1.2616, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.9480 [2021-07-07 15:46:06,116][root][INFO] - ===> Epoch[1456](7280/5): Loss 0.0178, Sem 0.0178 LR: 8.901e-02 Score 99.356 Data time: 0.2429, Forward time: 1.2616, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.7973 [2021-07-07 15:46:20,027][root][INFO] - ===> Epoch[1457](7285/5): Loss 0.0180, Sem 0.0180 LR: 8.900e-02 Score 99.349 Data time: 0.2517, Forward time: 1.2616, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.7324 [2021-07-07 15:46:34,514][root][INFO] - ===> Epoch[1458](7290/5): Loss 0.0177, Sem 0.0177 LR: 8.900e-02 Score 99.358 Data time: 0.2351, Forward time: 1.2616, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.8525 [2021-07-07 15:46:48,935][root][INFO] - ===> Epoch[1459](7295/5): Loss 0.0177, Sem 0.0177 LR: 8.899e-02 Score 99.360 Data time: 0.2365, Forward time: 1.2617, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.8296 [2021-07-07 15:47:02,946][root][INFO] - ===> Epoch[1460](7300/5): Loss 0.0170, Sem 0.0170 LR: 8.898e-02 Score 99.381 Data time: 0.3219, Forward time: 1.2617, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.7521 [2021-07-07 15:47:17,756][root][INFO] - ===> Epoch[1461](7305/5): Loss 0.0167, Sem 0.0167 LR: 8.897e-02 Score 99.390 Data time: 0.2052, Forward time: 1.2617, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.9228 [2021-07-07 15:47:31,863][root][INFO] - ===> Epoch[1462](7310/5): Loss 0.0185, Sem 0.0185 LR: 8.897e-02 Score 99.325 Data time: 0.1901, Forward time: 1.2617, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7551 [2021-07-07 15:47:46,114][root][INFO] - ===> Epoch[1463](7315/5): Loss 0.0181, Sem 0.0181 LR: 8.896e-02 Score 99.349 Data time: 0.2573, Forward time: 1.2617, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.8015 [2021-07-07 15:47:59,847][root][INFO] - ===> Epoch[1464](7320/5): Loss 0.0169, Sem 0.0169 LR: 8.895e-02 Score 99.384 Data time: 0.1564, Forward time: 1.2617, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7088 [2021-07-07 15:48:14,800][root][INFO] - ===> Epoch[1465](7325/5): Loss 0.0168, Sem 0.0168 LR: 8.894e-02 Score 99.387 Data time: 0.2156, Forward time: 1.2616, Backward time: 1.2295, DDP time: 0.0042, Total iter time: 2.9442 [2021-07-07 15:48:28,923][root][INFO] - ===> Epoch[1466](7330/5): Loss 0.0188, Sem 0.0188 LR: 8.893e-02 Score 99.331 Data time: 0.2093, Forward time: 1.2616, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7786 [2021-07-07 15:48:43,115][root][INFO] - ===> Epoch[1467](7335/5): Loss 0.0180, Sem 0.0180 LR: 8.893e-02 Score 99.349 Data time: 0.1874, Forward time: 1.2616, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7942 [2021-07-07 15:48:57,492][root][INFO] - ===> Epoch[1468](7340/5): Loss 0.0180, Sem 0.0180 LR: 8.892e-02 Score 99.350 Data time: 0.1777, Forward time: 1.2616, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8303 [2021-07-07 15:49:10,644][root][INFO] - ===> Epoch[1469](7345/5): Loss 0.0186, Sem 0.0186 LR: 8.891e-02 Score 99.331 Data time: 0.2638, Forward time: 1.2615, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.5728 [2021-07-07 15:49:24,656][root][INFO] - ===> Epoch[1470](7350/5): Loss 0.0183, Sem 0.0183 LR: 8.890e-02 Score 99.343 Data time: 0.2074, Forward time: 1.2614, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7530 [2021-07-07 15:49:38,722][root][INFO] - ===> Epoch[1471](7355/5): Loss 0.0176, Sem 0.0176 LR: 8.890e-02 Score 99.363 Data time: 0.2235, Forward time: 1.2614, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7633 [2021-07-07 15:49:52,362][root][INFO] - ===> Epoch[1472](7360/5): Loss 0.0187, Sem 0.0187 LR: 8.889e-02 Score 99.321 Data time: 0.2598, Forward time: 1.2614, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6902 [2021-07-07 15:50:05,930][root][INFO] - ===> Epoch[1473](7365/5): Loss 0.0174, Sem 0.0174 LR: 8.888e-02 Score 99.371 Data time: 0.1948, Forward time: 1.2614, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6758 [2021-07-07 15:50:19,926][root][INFO] - ===> Epoch[1474](7370/5): Loss 0.0200, Sem 0.0200 LR: 8.887e-02 Score 99.298 Data time: 0.1788, Forward time: 1.2614, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7603 [2021-07-07 15:50:33,727][root][INFO] - ===> Epoch[1475](7375/5): Loss 0.0181, Sem 0.0181 LR: 8.887e-02 Score 99.351 Data time: 0.2067, Forward time: 1.2613, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7129 [2021-07-07 15:50:47,229][root][INFO] - ===> Epoch[1476](7380/5): Loss 0.0183, Sem 0.0183 LR: 8.886e-02 Score 99.338 Data time: 0.1913, Forward time: 1.2612, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.6585 [2021-07-07 15:51:02,077][root][INFO] - ===> Epoch[1477](7385/5): Loss 0.0192, Sem 0.0192 LR: 8.885e-02 Score 99.311 Data time: 0.2153, Forward time: 1.2612, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.9163 [2021-07-07 15:51:15,476][root][INFO] - ===> Epoch[1478](7390/5): Loss 0.0183, Sem 0.0183 LR: 8.884e-02 Score 99.344 Data time: 0.2239, Forward time: 1.2613, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.6388 [2021-07-07 15:51:29,506][root][INFO] - ===> Epoch[1479](7395/5): Loss 0.0187, Sem 0.0187 LR: 8.884e-02 Score 99.331 Data time: 0.2323, Forward time: 1.2612, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7662 [2021-07-07 15:51:43,113][root][INFO] - ===> Epoch[1480](7400/5): Loss 0.0181, Sem 0.0181 LR: 8.883e-02 Score 99.346 Data time: 0.1831, Forward time: 1.2612, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.6832 [2021-07-07 15:51:43,115][root][INFO] - ===> Start testing [2021-07-07 15:52:18,242][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1237 Loss 0.241 (AVG: 0.526) Score 93.829 (AVG: 90.854) mIOU 70.001 mAP 84.465 mAcc 76.382 IOU: 64.617 0.183 81.016 76.908 93.715 91.716 28.885 81.470 97.219 69.248 79.146 86.727 59.169 mAP: 85.235 10.753 93.739 88.918 98.851 98.459 58.297 97.800 99.742 87.225 92.191 94.808 92.021 mAcc: 80.122 1.823 88.565 84.444 95.205 95.538 30.570 93.715 99.169 74.822 87.606 96.775 64.604 [2021-07-07 15:52:18,243][root][INFO] - Finished test. Elapsed time: 35.1272 [2021-07-07 15:52:19,301][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 15:52:32,779][root][INFO] - ===> Epoch[1481](7405/5): Loss 0.0175, Sem 0.0175 LR: 8.882e-02 Score 99.363 Data time: 0.1864, Forward time: 1.2612, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.6581 [2021-07-07 15:52:46,687][root][INFO] - ===> Epoch[1482](7410/5): Loss 0.0168, Sem 0.0168 LR: 8.881e-02 Score 99.391 Data time: 0.2675, Forward time: 1.2611, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7336 [2021-07-07 15:53:00,285][root][INFO] - ===> Epoch[1483](7415/5): Loss 0.0184, Sem 0.0184 LR: 8.881e-02 Score 99.331 Data time: 0.2058, Forward time: 1.2611, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.6787 [2021-07-07 15:53:14,453][root][INFO] - ===> Epoch[1484](7420/5): Loss 0.0179, Sem 0.0179 LR: 8.880e-02 Score 99.357 Data time: 0.2315, Forward time: 1.2610, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7797 [2021-07-07 15:53:28,131][root][INFO] - ===> Epoch[1485](7425/5): Loss 0.0179, Sem 0.0179 LR: 8.879e-02 Score 99.359 Data time: 0.2290, Forward time: 1.2610, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.6818 [2021-07-07 15:53:42,734][root][INFO] - ===> Epoch[1486](7430/5): Loss 0.0177, Sem 0.0177 LR: 8.878e-02 Score 99.358 Data time: 0.2240, Forward time: 1.2611, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.8825 [2021-07-07 15:53:56,540][root][INFO] - ===> Epoch[1487](7435/5): Loss 0.0176, Sem 0.0176 LR: 8.878e-02 Score 99.362 Data time: 0.2575, Forward time: 1.2611, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7142 [2021-07-07 15:54:09,952][root][INFO] - ===> Epoch[1488](7440/5): Loss 0.0183, Sem 0.0183 LR: 8.877e-02 Score 99.338 Data time: 0.2977, Forward time: 1.2612, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6386 [2021-07-07 15:54:23,813][root][INFO] - ===> Epoch[1489](7445/5): Loss 0.0182, Sem 0.0182 LR: 8.876e-02 Score 99.350 Data time: 0.2126, Forward time: 1.2612, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7291 [2021-07-07 15:54:38,388][root][INFO] - ===> Epoch[1490](7450/5): Loss 0.0173, Sem 0.0173 LR: 8.875e-02 Score 99.376 Data time: 0.2056, Forward time: 1.2612, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8732 [2021-07-07 15:54:52,025][root][INFO] - ===> Epoch[1491](7455/5): Loss 0.0170, Sem 0.0170 LR: 8.874e-02 Score 99.384 Data time: 0.2193, Forward time: 1.2612, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6913 [2021-07-07 15:55:07,235][root][INFO] - ===> Epoch[1492](7460/5): Loss 0.0172, Sem 0.0172 LR: 8.874e-02 Score 99.375 Data time: 0.2746, Forward time: 1.2613, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 3.0010 [2021-07-07 15:55:21,037][root][INFO] - ===> Epoch[1493](7465/5): Loss 0.0171, Sem 0.0171 LR: 8.873e-02 Score 99.380 Data time: 0.2161, Forward time: 1.2612, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7215 [2021-07-07 15:55:34,321][root][INFO] - ===> Epoch[1494](7470/5): Loss 0.0181, Sem 0.0181 LR: 8.872e-02 Score 99.348 Data time: 0.2200, Forward time: 1.2612, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.6189 [2021-07-07 15:55:48,589][root][INFO] - ===> Epoch[1495](7475/5): Loss 0.0169, Sem 0.0169 LR: 8.871e-02 Score 99.387 Data time: 0.2258, Forward time: 1.2612, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7979 [2021-07-07 15:56:03,032][root][INFO] - ===> Epoch[1496](7480/5): Loss 0.0170, Sem 0.0170 LR: 8.871e-02 Score 99.390 Data time: 0.2134, Forward time: 1.2612, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8465 [2021-07-07 15:56:16,589][root][INFO] - ===> Epoch[1497](7485/5): Loss 0.0173, Sem 0.0173 LR: 8.870e-02 Score 99.374 Data time: 0.2138, Forward time: 1.2612, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.6733 [2021-07-07 15:56:30,041][root][INFO] - ===> Epoch[1498](7490/5): Loss 0.0174, Sem 0.0174 LR: 8.869e-02 Score 99.376 Data time: 0.1865, Forward time: 1.2611, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.6550 [2021-07-07 15:56:43,773][root][INFO] - ===> Epoch[1499](7495/5): Loss 0.0173, Sem 0.0173 LR: 8.868e-02 Score 99.379 Data time: 0.2442, Forward time: 1.2611, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7051 [2021-07-07 15:56:58,343][root][INFO] - ===> Epoch[1500](7500/5): Loss 0.0176, Sem 0.0176 LR: 8.868e-02 Score 99.365 Data time: 0.3262, Forward time: 1.2612, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8708 [2021-07-07 15:57:12,697][root][INFO] - ===> Epoch[1501](7505/5): Loss 0.0170, Sem 0.0170 LR: 8.867e-02 Score 99.381 Data time: 0.2322, Forward time: 1.2612, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8250 [2021-07-07 15:57:26,676][root][INFO] - ===> Epoch[1502](7510/5): Loss 0.0176, Sem 0.0176 LR: 8.866e-02 Score 99.376 Data time: 0.2751, Forward time: 1.2613, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7438 [2021-07-07 15:57:40,661][root][INFO] - ===> Epoch[1503](7515/5): Loss 0.0175, Sem 0.0175 LR: 8.865e-02 Score 99.372 Data time: 0.2077, Forward time: 1.2613, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7587 [2021-07-07 15:57:54,496][root][INFO] - ===> Epoch[1504](7520/5): Loss 0.0173, Sem 0.0173 LR: 8.865e-02 Score 99.374 Data time: 0.2170, Forward time: 1.2613, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7260 [2021-07-07 15:58:08,085][root][INFO] - ===> Epoch[1505](7525/5): Loss 0.0173, Sem 0.0173 LR: 8.864e-02 Score 99.376 Data time: 0.1846, Forward time: 1.2612, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6804 [2021-07-07 15:58:24,071][root][INFO] - ===> Epoch[1506](7530/5): Loss 0.0172, Sem 0.0172 LR: 8.863e-02 Score 99.381 Data time: 0.2388, Forward time: 1.2613, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 3.1603 [2021-07-07 15:58:37,434][root][INFO] - ===> Epoch[1507](7535/5): Loss 0.0186, Sem 0.0186 LR: 8.862e-02 Score 99.326 Data time: 0.2227, Forward time: 1.2613, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.6212 [2021-07-07 15:58:51,820][root][INFO] - ===> Epoch[1508](7540/5): Loss 0.0175, Sem 0.0175 LR: 8.862e-02 Score 99.366 Data time: 0.2616, Forward time: 1.2612, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.8329 [2021-07-07 15:59:05,330][root][INFO] - ===> Epoch[1509](7545/5): Loss 0.0187, Sem 0.0187 LR: 8.861e-02 Score 99.323 Data time: 0.2783, Forward time: 1.2612, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.6566 [2021-07-07 15:59:19,597][root][INFO] - ===> Epoch[1510](7550/5): Loss 0.0170, Sem 0.0170 LR: 8.860e-02 Score 99.382 Data time: 0.2592, Forward time: 1.2612, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.8062 [2021-07-07 15:59:33,596][root][INFO] - ===> Epoch[1511](7555/5): Loss 0.0186, Sem 0.0186 LR: 8.859e-02 Score 99.331 Data time: 0.2262, Forward time: 1.2612, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7497 [2021-07-07 15:59:47,960][root][INFO] - ===> Epoch[1512](7560/5): Loss 0.0179, Sem 0.0179 LR: 8.859e-02 Score 99.357 Data time: 0.2424, Forward time: 1.2612, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.8277 [2021-07-07 16:00:01,835][root][INFO] - ===> Epoch[1513](7565/5): Loss 0.0176, Sem 0.0176 LR: 8.858e-02 Score 99.360 Data time: 0.2211, Forward time: 1.2612, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7249 [2021-07-07 16:00:15,646][root][INFO] - ===> Epoch[1514](7570/5): Loss 0.0183, Sem 0.0183 LR: 8.857e-02 Score 99.345 Data time: 0.2666, Forward time: 1.2612, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7143 [2021-07-07 16:00:29,752][root][INFO] - ===> Epoch[1515](7575/5): Loss 0.0180, Sem 0.0180 LR: 8.856e-02 Score 99.346 Data time: 0.1754, Forward time: 1.2611, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.7784 [2021-07-07 16:00:43,340][root][INFO] - ===> Epoch[1516](7580/5): Loss 0.0181, Sem 0.0181 LR: 8.855e-02 Score 99.354 Data time: 0.2243, Forward time: 1.2611, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.6717 [2021-07-07 16:00:57,155][root][INFO] - ===> Epoch[1517](7585/5): Loss 0.0178, Sem 0.0178 LR: 8.855e-02 Score 99.353 Data time: 0.2081, Forward time: 1.2611, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7102 [2021-07-07 16:01:10,984][root][INFO] - ===> Epoch[1518](7590/5): Loss 0.0181, Sem 0.0181 LR: 8.854e-02 Score 99.342 Data time: 0.1614, Forward time: 1.2612, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7157 [2021-07-07 16:01:24,008][root][INFO] - ===> Epoch[1519](7595/5): Loss 0.0195, Sem 0.0195 LR: 8.853e-02 Score 99.301 Data time: 0.2343, Forward time: 1.2611, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.5613 [2021-07-07 16:01:38,087][root][INFO] - ===> Epoch[1520](7600/5): Loss 0.0186, Sem 0.0186 LR: 8.852e-02 Score 99.342 Data time: 0.2095, Forward time: 1.2612, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7685 [2021-07-07 16:01:38,088][root][INFO] - ===> Start testing [2021-07-07 16:02:13,358][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1299 Loss 0.323 (AVG: 0.502) Score 93.112 (AVG: 91.118) mIOU 70.509 mAP 84.672 mAcc 76.376 IOU: 64.472 0.162 80.136 77.158 94.105 91.676 36.513 82.414 98.612 67.091 79.614 87.015 57.650 mAP: 85.237 10.853 92.296 90.124 98.657 98.221 63.694 97.761 99.815 84.406 92.323 95.259 92.091 mAcc: 81.521 1.276 83.812 84.397 95.417 96.566 39.514 95.279 99.494 68.064 88.445 96.507 62.595 [2021-07-07 16:02:13,358][root][INFO] - Finished test. Elapsed time: 35.2689 [2021-07-07 16:02:14,409][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 16:02:28,518][root][INFO] - ===> Epoch[1521](7605/5): Loss 0.0186, Sem 0.0186 LR: 8.852e-02 Score 99.333 Data time: 0.2136, Forward time: 1.2612, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7826 [2021-07-07 16:02:42,908][root][INFO] - ===> Epoch[1522](7610/5): Loss 0.0191, Sem 0.0191 LR: 8.851e-02 Score 99.317 Data time: 0.1874, Forward time: 1.2612, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8314 [2021-07-07 16:02:57,233][root][INFO] - ===> Epoch[1523](7615/5): Loss 0.0186, Sem 0.0186 LR: 8.850e-02 Score 99.353 Data time: 0.2798, Forward time: 1.2612, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8164 [2021-07-07 16:03:11,904][root][INFO] - ===> Epoch[1524](7620/5): Loss 0.0193, Sem 0.0193 LR: 8.849e-02 Score 99.310 Data time: 0.2264, Forward time: 1.2613, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8807 [2021-07-07 16:03:25,740][root][INFO] - ===> Epoch[1525](7625/5): Loss 0.0177, Sem 0.0177 LR: 8.849e-02 Score 99.362 Data time: 0.2412, Forward time: 1.2614, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7258 [2021-07-07 16:03:39,879][root][INFO] - ===> Epoch[1526](7630/5): Loss 0.0190, Sem 0.0190 LR: 8.848e-02 Score 99.320 Data time: 0.3077, Forward time: 1.2614, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7737 [2021-07-07 16:03:53,681][root][INFO] - ===> Epoch[1527](7635/5): Loss 0.0176, Sem 0.0176 LR: 8.847e-02 Score 99.363 Data time: 0.2702, Forward time: 1.2614, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7093 [2021-07-07 16:04:07,774][root][INFO] - ===> Epoch[1528](7640/5): Loss 0.0191, Sem 0.0191 LR: 8.846e-02 Score 99.319 Data time: 0.2297, Forward time: 1.2614, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7712 [2021-07-07 16:04:22,880][root][INFO] - ===> Epoch[1529](7645/5): Loss 0.0182, Sem 0.0182 LR: 8.846e-02 Score 99.350 Data time: 0.2635, Forward time: 1.2614, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.9585 [2021-07-07 16:04:37,013][root][INFO] - ===> Epoch[1530](7650/5): Loss 0.0180, Sem 0.0180 LR: 8.845e-02 Score 99.352 Data time: 0.3004, Forward time: 1.2615, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7602 [2021-07-07 16:04:51,426][root][INFO] - ===> Epoch[1531](7655/5): Loss 0.0189, Sem 0.0189 LR: 8.844e-02 Score 99.328 Data time: 0.2091, Forward time: 1.2614, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8398 [2021-07-07 16:05:05,095][root][INFO] - ===> Epoch[1532](7660/5): Loss 0.0177, Sem 0.0177 LR: 8.843e-02 Score 99.362 Data time: 0.2431, Forward time: 1.2614, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6921 [2021-07-07 16:05:19,708][root][INFO] - ===> Epoch[1533](7665/5): Loss 0.0175, Sem 0.0175 LR: 8.843e-02 Score 99.367 Data time: 0.3310, Forward time: 1.2615, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.8733 [2021-07-07 16:05:33,302][root][INFO] - ===> Epoch[1534](7670/5): Loss 0.0176, Sem 0.0176 LR: 8.842e-02 Score 99.366 Data time: 0.1762, Forward time: 1.2615, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.6853 [2021-07-07 16:05:47,758][root][INFO] - ===> Epoch[1535](7675/5): Loss 0.0182, Sem 0.0182 LR: 8.841e-02 Score 99.341 Data time: 0.2035, Forward time: 1.2615, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8495 [2021-07-07 16:06:02,220][root][INFO] - ===> Epoch[1536](7680/5): Loss 0.0173, Sem 0.0173 LR: 8.840e-02 Score 99.377 Data time: 0.2989, Forward time: 1.2615, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8383 [2021-07-07 16:06:16,494][root][INFO] - ===> Epoch[1537](7685/5): Loss 0.0169, Sem 0.0169 LR: 8.840e-02 Score 99.386 Data time: 0.3111, Forward time: 1.2615, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.8038 [2021-07-07 16:06:30,685][root][INFO] - ===> Epoch[1538](7690/5): Loss 0.0177, Sem 0.0177 LR: 8.839e-02 Score 99.361 Data time: 0.1813, Forward time: 1.2615, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8020 [2021-07-07 16:06:44,154][root][INFO] - ===> Epoch[1539](7695/5): Loss 0.0173, Sem 0.0173 LR: 8.838e-02 Score 99.374 Data time: 0.2288, Forward time: 1.2615, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.6474 [2021-07-07 16:06:57,997][root][INFO] - ===> Epoch[1540](7700/5): Loss 0.0179, Sem 0.0179 LR: 8.837e-02 Score 99.356 Data time: 0.2247, Forward time: 1.2615, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7221 [2021-07-07 16:07:11,770][root][INFO] - ===> Epoch[1541](7705/5): Loss 0.0177, Sem 0.0177 LR: 8.836e-02 Score 99.365 Data time: 0.2091, Forward time: 1.2615, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7058 [2021-07-07 16:07:25,777][root][INFO] - ===> Epoch[1542](7710/5): Loss 0.0183, Sem 0.0183 LR: 8.836e-02 Score 99.339 Data time: 0.2191, Forward time: 1.2615, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 2.7483 [2021-07-07 16:07:38,886][root][INFO] - ===> Epoch[1543](7715/5): Loss 0.0168, Sem 0.0168 LR: 8.835e-02 Score 99.389 Data time: 0.2083, Forward time: 1.2616, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.5792 [2021-07-07 16:07:53,349][root][INFO] - ===> Epoch[1544](7720/5): Loss 0.0177, Sem 0.0177 LR: 8.834e-02 Score 99.360 Data time: 0.2381, Forward time: 1.2617, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.8366 [2021-07-07 16:08:06,773][root][INFO] - ===> Epoch[1545](7725/5): Loss 0.0178, Sem 0.0178 LR: 8.833e-02 Score 99.354 Data time: 0.2509, Forward time: 1.2616, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.6414 [2021-07-07 16:08:20,962][root][INFO] - ===> Epoch[1546](7730/5): Loss 0.0185, Sem 0.0185 LR: 8.833e-02 Score 99.328 Data time: 0.2138, Forward time: 1.2617, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7896 [2021-07-07 16:08:35,087][root][INFO] - ===> Epoch[1547](7735/5): Loss 0.0176, Sem 0.0176 LR: 8.832e-02 Score 99.365 Data time: 0.2585, Forward time: 1.2616, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7733 [2021-07-07 16:08:49,517][root][INFO] - ===> Epoch[1548](7740/5): Loss 0.0177, Sem 0.0177 LR: 8.831e-02 Score 99.364 Data time: 0.2552, Forward time: 1.2617, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.8374 [2021-07-07 16:09:02,869][root][INFO] - ===> Epoch[1549](7745/5): Loss 0.0185, Sem 0.0185 LR: 8.830e-02 Score 99.332 Data time: 0.1906, Forward time: 1.2617, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.6324 [2021-07-07 16:09:16,831][root][INFO] - ===> Epoch[1550](7750/5): Loss 0.0180, Sem 0.0180 LR: 8.830e-02 Score 99.351 Data time: 0.2566, Forward time: 1.2616, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7532 [2021-07-07 16:09:30,671][root][INFO] - ===> Epoch[1551](7755/5): Loss 0.0175, Sem 0.0175 LR: 8.829e-02 Score 99.370 Data time: 0.2664, Forward time: 1.2616, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7288 [2021-07-07 16:09:44,817][root][INFO] - ===> Epoch[1552](7760/5): Loss 0.0173, Sem 0.0173 LR: 8.828e-02 Score 99.371 Data time: 0.2285, Forward time: 1.2616, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7782 [2021-07-07 16:09:58,488][root][INFO] - ===> Epoch[1553](7765/5): Loss 0.0170, Sem 0.0170 LR: 8.827e-02 Score 99.383 Data time: 0.2687, Forward time: 1.2616, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.6897 [2021-07-07 16:10:12,446][root][INFO] - ===> Epoch[1554](7770/5): Loss 0.0178, Sem 0.0178 LR: 8.827e-02 Score 99.360 Data time: 0.2083, Forward time: 1.2616, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7501 [2021-07-07 16:10:25,881][root][INFO] - ===> Epoch[1555](7775/5): Loss 0.0186, Sem 0.0186 LR: 8.826e-02 Score 99.336 Data time: 0.2315, Forward time: 1.2615, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.6273 [2021-07-07 16:10:39,636][root][INFO] - ===> Epoch[1556](7780/5): Loss 0.0175, Sem 0.0175 LR: 8.825e-02 Score 99.364 Data time: 0.2828, Forward time: 1.2616, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.7063 [2021-07-07 16:10:53,175][root][INFO] - ===> Epoch[1557](7785/5): Loss 0.0175, Sem 0.0175 LR: 8.824e-02 Score 99.367 Data time: 0.2377, Forward time: 1.2616, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.6605 [2021-07-07 16:11:07,759][root][INFO] - ===> Epoch[1558](7790/5): Loss 0.0179, Sem 0.0179 LR: 8.824e-02 Score 99.356 Data time: 0.2449, Forward time: 1.2617, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.8720 [2021-07-07 16:11:21,813][root][INFO] - ===> Epoch[1559](7795/5): Loss 0.0180, Sem 0.0180 LR: 8.823e-02 Score 99.347 Data time: 0.1989, Forward time: 1.2616, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.7699 [2021-07-07 16:11:35,385][root][INFO] - ===> Epoch[1560](7800/5): Loss 0.0166, Sem 0.0166 LR: 8.822e-02 Score 99.398 Data time: 0.2470, Forward time: 1.2616, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.6746 [2021-07-07 16:11:35,386][root][INFO] - ===> Start testing [2021-07-07 16:12:09,346][root][INFO] - 68/68: Data time: 0.0017, Iter time: 0.1328 Loss 0.301 (AVG: 0.506) Score 92.232 (AVG: 91.196) mIOU 70.968 mAP 84.172 mAcc 77.086 IOU: 64.620 0.022 82.662 78.099 94.587 92.184 29.069 81.552 98.688 76.770 80.866 86.872 56.597 mAP: 84.946 9.185 94.393 90.977 98.661 98.758 58.366 97.868 99.846 82.276 92.599 94.603 91.755 mAcc: 80.187 0.182 90.080 85.838 96.068 96.180 30.645 95.285 99.400 79.390 89.804 96.265 62.798 [2021-07-07 16:12:09,346][root][INFO] - Finished test. Elapsed time: 33.9587 [2021-07-07 16:12:10,399][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 16:12:22,941][root][INFO] - ===> Epoch[1561](7805/5): Loss 0.0179, Sem 0.0179 LR: 8.821e-02 Score 99.356 Data time: 0.2185, Forward time: 1.2615, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.4615 [2021-07-07 16:12:36,096][root][INFO] - ===> Epoch[1562](7810/5): Loss 0.0174, Sem 0.0174 LR: 8.821e-02 Score 99.370 Data time: 0.2515, Forward time: 1.2616, Backward time: 1.2288, DDP time: 0.0042, Total iter time: 2.5819 [2021-07-07 16:12:49,651][root][INFO] - ===> Epoch[1563](7815/5): Loss 0.0200, Sem 0.0200 LR: 8.820e-02 Score 99.295 Data time: 0.1994, Forward time: 1.2615, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.6702 [2021-07-07 16:13:02,790][root][INFO] - ===> Epoch[1564](7820/5): Loss 0.0199, Sem 0.0199 LR: 8.819e-02 Score 99.285 Data time: 0.2603, Forward time: 1.2615, Backward time: 1.2288, DDP time: 0.0042, Total iter time: 2.5797 [2021-07-07 16:13:16,421][root][INFO] - ===> Epoch[1565](7825/5): Loss 0.0216, Sem 0.0216 LR: 8.818e-02 Score 99.229 Data time: 0.1672, Forward time: 1.2615, Backward time: 1.2288, DDP time: 0.0042, Total iter time: 2.6855 [2021-07-07 16:13:29,841][root][INFO] - ===> Epoch[1566](7830/5): Loss 0.0202, Sem 0.0202 LR: 8.817e-02 Score 99.278 Data time: 0.2212, Forward time: 1.2615, Backward time: 1.2287, DDP time: 0.0042, Total iter time: 2.6295 [2021-07-07 16:13:43,028][root][INFO] - ===> Epoch[1567](7835/5): Loss 0.0191, Sem 0.0191 LR: 8.817e-02 Score 99.316 Data time: 0.1684, Forward time: 1.2614, Backward time: 1.2287, DDP time: 0.0042, Total iter time: 2.6002 [2021-07-07 16:13:57,438][root][INFO] - ===> Epoch[1568](7840/5): Loss 0.0186, Sem 0.0186 LR: 8.816e-02 Score 99.328 Data time: 0.2252, Forward time: 1.2614, Backward time: 1.2287, DDP time: 0.0042, Total iter time: 2.8363 [2021-07-07 16:14:11,628][root][INFO] - ===> Epoch[1569](7845/5): Loss 0.0189, Sem 0.0189 LR: 8.815e-02 Score 99.324 Data time: 0.2997, Forward time: 1.2615, Backward time: 1.2286, DDP time: 0.0042, Total iter time: 2.7842 [2021-07-07 16:14:25,864][root][INFO] - ===> Epoch[1570](7850/5): Loss 0.0186, Sem 0.0186 LR: 8.814e-02 Score 99.348 Data time: 0.1778, Forward time: 1.2616, Backward time: 1.2286, DDP time: 0.0042, Total iter time: 2.8001 [2021-07-07 16:14:40,715][root][INFO] - ===> Epoch[1571](7855/5): Loss 0.0186, Sem 0.0186 LR: 8.814e-02 Score 99.330 Data time: 0.2424, Forward time: 1.2615, Backward time: 1.2288, DDP time: 0.0042, Total iter time: 2.9235 [2021-07-07 16:14:54,732][root][INFO] - ===> Epoch[1572](7860/5): Loss 0.0184, Sem 0.0184 LR: 8.813e-02 Score 99.340 Data time: 0.2296, Forward time: 1.2615, Backward time: 1.2288, DDP time: 0.0042, Total iter time: 2.7532 [2021-07-07 16:15:08,508][root][INFO] - ===> Epoch[1573](7865/5): Loss 0.0184, Sem 0.0184 LR: 8.812e-02 Score 99.335 Data time: 0.1992, Forward time: 1.2615, Backward time: 1.2288, DDP time: 0.0042, Total iter time: 2.7060 [2021-07-07 16:15:22,558][root][INFO] - ===> Epoch[1574](7870/5): Loss 0.0174, Sem 0.0174 LR: 8.811e-02 Score 99.367 Data time: 0.1866, Forward time: 1.2614, Backward time: 1.2289, DDP time: 0.0042, Total iter time: 2.7660 [2021-07-07 16:15:36,493][root][INFO] - ===> Epoch[1575](7875/5): Loss 0.0183, Sem 0.0183 LR: 8.811e-02 Score 99.336 Data time: 0.2158, Forward time: 1.2613, Backward time: 1.2290, DDP time: 0.0042, Total iter time: 2.7509 [2021-07-07 16:15:50,537][root][INFO] - ===> Epoch[1576](7880/5): Loss 0.0184, Sem 0.0184 LR: 8.810e-02 Score 99.334 Data time: 0.2474, Forward time: 1.2612, Backward time: 1.2291, DDP time: 0.0042, Total iter time: 2.7593 [2021-07-07 16:16:04,889][root][INFO] - ===> Epoch[1577](7885/5): Loss 0.0181, Sem 0.0181 LR: 8.809e-02 Score 99.355 Data time: 0.2128, Forward time: 1.2612, Backward time: 1.2292, DDP time: 0.0042, Total iter time: 2.8194 [2021-07-07 16:16:19,539][root][INFO] - ===> Epoch[1578](7890/5): Loss 0.0186, Sem 0.0186 LR: 8.808e-02 Score 99.331 Data time: 0.2451, Forward time: 1.2611, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.8785 [2021-07-07 16:16:33,668][root][INFO] - ===> Epoch[1579](7895/5): Loss 0.0182, Sem 0.0182 LR: 8.808e-02 Score 99.344 Data time: 0.2204, Forward time: 1.2611, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7843 [2021-07-07 16:16:47,594][root][INFO] - ===> Epoch[1580](7900/5): Loss 0.0178, Sem 0.0178 LR: 8.807e-02 Score 99.357 Data time: 0.2016, Forward time: 1.2611, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7364 [2021-07-07 16:17:01,798][root][INFO] - ===> Epoch[1581](7905/5): Loss 0.0183, Sem 0.0183 LR: 8.806e-02 Score 99.342 Data time: 0.2351, Forward time: 1.2611, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7990 [2021-07-07 16:17:15,795][root][INFO] - ===> Epoch[1582](7910/5): Loss 0.0176, Sem 0.0176 LR: 8.805e-02 Score 99.365 Data time: 0.2434, Forward time: 1.2611, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7541 [2021-07-07 16:17:30,003][root][INFO] - ===> Epoch[1583](7915/5): Loss 0.0176, Sem 0.0176 LR: 8.805e-02 Score 99.362 Data time: 0.1934, Forward time: 1.2611, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.8012 [2021-07-07 16:17:43,977][root][INFO] - ===> Epoch[1584](7920/5): Loss 0.0173, Sem 0.0173 LR: 8.804e-02 Score 99.380 Data time: 0.2189, Forward time: 1.2611, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7481 [2021-07-07 16:17:58,064][root][INFO] - ===> Epoch[1585](7925/5): Loss 0.0182, Sem 0.0182 LR: 8.803e-02 Score 99.360 Data time: 0.2259, Forward time: 1.2611, Backward time: 1.2294, DDP time: 0.0042, Total iter time: 2.7659 [2021-07-07 16:18:12,066][root][INFO] - ===> Epoch[1586](7930/5): Loss 0.0188, Sem 0.0188 LR: 8.802e-02 Score 99.317 Data time: 0.2243, Forward time: 1.2612, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7556 [2021-07-07 16:18:26,093][root][INFO] - ===> Epoch[1587](7935/5): Loss 0.0212, Sem 0.0212 LR: 8.801e-02 Score 99.246 Data time: 0.2550, Forward time: 1.2612, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7533 [2021-07-07 16:18:40,453][root][INFO] - ===> Epoch[1588](7940/5): Loss 0.0221, Sem 0.0221 LR: 8.801e-02 Score 99.210 Data time: 0.2399, Forward time: 1.2612, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.8178 [2021-07-07 16:18:54,389][root][INFO] - ===> Epoch[1589](7945/5): Loss 0.0229, Sem 0.0229 LR: 8.800e-02 Score 99.187 Data time: 0.2224, Forward time: 1.2612, Backward time: 1.2293, DDP time: 0.0042, Total iter time: 2.7423 [2021-07-07 16:19:09,957][root][INFO] - ===> Epoch[1590](7950/5): Loss 0.0226, Sem 0.0226 LR: 8.799e-02 Score 99.193 Data time: 0.2080, Forward time: 1.2612, Backward time: 1.2296, DDP time: 0.0042, Total iter time: 3.0637 [2021-07-07 16:19:24,558][root][INFO] - ===> Epoch[1591](7955/5): Loss 0.0210, Sem 0.0210 LR: 8.798e-02 Score 99.249 Data time: 0.1872, Forward time: 1.2611, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8794 [2021-07-07 16:19:39,262][root][INFO] - ===> Epoch[1592](7960/5): Loss 0.0202, Sem 0.0202 LR: 8.798e-02 Score 99.289 Data time: 0.2920, Forward time: 1.2612, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8749 [2021-07-07 16:19:53,917][root][INFO] - ===> Epoch[1593](7965/5): Loss 0.0205, Sem 0.0205 LR: 8.797e-02 Score 99.268 Data time: 0.2880, Forward time: 1.2612, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8870 [2021-07-07 16:20:08,792][root][INFO] - ===> Epoch[1594](7970/5): Loss 0.0206, Sem 0.0206 LR: 8.796e-02 Score 99.256 Data time: 0.2584, Forward time: 1.2612, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.9333 [2021-07-07 16:20:22,598][root][INFO] - ===> Epoch[1595](7975/5): Loss 0.0199, Sem 0.0199 LR: 8.795e-02 Score 99.284 Data time: 0.2303, Forward time: 1.2612, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7175 [2021-07-07 16:20:35,256][root][INFO] - ===> Epoch[1596](7980/5): Loss 0.0194, Sem 0.0194 LR: 8.795e-02 Score 99.299 Data time: 0.1722, Forward time: 1.2611, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.4928 [2021-07-07 16:20:49,533][root][INFO] - ===> Epoch[1597](7985/5): Loss 0.0194, Sem 0.0194 LR: 8.794e-02 Score 99.301 Data time: 0.2668, Forward time: 1.2612, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.8046 [2021-07-07 16:21:03,471][root][INFO] - ===> Epoch[1598](7990/5): Loss 0.0218, Sem 0.0218 LR: 8.793e-02 Score 99.227 Data time: 0.1793, Forward time: 1.2611, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.7461 [2021-07-07 16:21:17,788][root][INFO] - ===> Epoch[1599](7995/5): Loss 0.0210, Sem 0.0210 LR: 8.792e-02 Score 99.253 Data time: 0.1999, Forward time: 1.2611, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8211 [2021-07-07 16:21:32,198][root][INFO] - ===> Epoch[1600](8000/5): Loss 0.0213, Sem 0.0213 LR: 8.792e-02 Score 99.265 Data time: 0.2053, Forward time: 1.2611, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.8306 [2021-07-07 16:21:32,199][root][INFO] - ===> Start testing [2021-07-07 16:22:06,242][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1221 Loss 0.152 (AVG: 0.510) Score 95.971 (AVG: 90.804) mIOU 69.622 mAP 84.002 mAcc 76.158 IOU: 62.288 0.373 73.699 74.921 94.938 91.517 39.336 79.225 98.583 67.582 79.643 87.528 55.457 mAP: 84.042 15.698 89.009 88.821 98.926 98.656 59.561 97.098 99.796 83.782 92.973 95.045 88.624 mAcc: 81.624 3.008 76.248 80.992 96.456 96.115 42.665 94.218 99.354 71.636 91.508 96.245 59.991 [2021-07-07 16:22:06,242][root][INFO] - Finished test. Elapsed time: 34.0424 [2021-07-07 16:22:07,313][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 16:22:20,114][root][INFO] - ===> Epoch[1601](8005/5): Loss 0.0221, Sem 0.0221 LR: 8.791e-02 Score 99.218 Data time: 0.2620, Forward time: 1.2611, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.5117 [2021-07-07 16:22:34,252][root][INFO] - ===> Epoch[1602](8010/5): Loss 0.0199, Sem 0.0199 LR: 8.790e-02 Score 99.292 Data time: 0.2240, Forward time: 1.2610, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.7725 [2021-07-07 16:22:47,965][root][INFO] - ===> Epoch[1603](8015/5): Loss 0.0215, Sem 0.0215 LR: 8.789e-02 Score 99.226 Data time: 0.2456, Forward time: 1.2611, Backward time: 1.2297, DDP time: 0.0042, Total iter time: 2.6994 [2021-07-07 16:23:02,477][root][INFO] - ===> Epoch[1604](8020/5): Loss 0.0212, Sem 0.0212 LR: 8.789e-02 Score 99.247 Data time: 0.1901, Forward time: 1.2611, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.8530 [2021-07-07 16:23:17,386][root][INFO] - ===> Epoch[1605](8025/5): Loss 0.0212, Sem 0.0212 LR: 8.788e-02 Score 99.256 Data time: 0.1595, Forward time: 1.2611, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.9272 [2021-07-07 16:23:30,946][root][INFO] - ===> Epoch[1606](8030/5): Loss 0.0226, Sem 0.0226 LR: 8.787e-02 Score 99.194 Data time: 0.1859, Forward time: 1.2611, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.6705 [2021-07-07 16:23:44,390][root][INFO] - ===> Epoch[1607](8035/5): Loss 0.0225, Sem 0.0225 LR: 8.786e-02 Score 99.195 Data time: 0.2656, Forward time: 1.2611, Backward time: 1.2298, DDP time: 0.0042, Total iter time: 2.6459 [2021-07-07 16:23:58,643][root][INFO] - ===> Epoch[1608](8040/5): Loss 0.0237, Sem 0.0237 LR: 8.786e-02 Score 99.188 Data time: 0.1865, Forward time: 1.2611, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.8036 [2021-07-07 16:24:12,522][root][INFO] - ===> Epoch[1609](8045/5): Loss 0.0209, Sem 0.0209 LR: 8.785e-02 Score 99.248 Data time: 0.2351, Forward time: 1.2611, Backward time: 1.2299, DDP time: 0.0042, Total iter time: 2.7303 [2021-07-07 16:24:26,475][root][INFO] - ===> Epoch[1610](8050/5): Loss 0.0219, Sem 0.0219 LR: 8.784e-02 Score 99.217 Data time: 0.2000, Forward time: 1.2610, Backward time: 1.2300, DDP time: 0.0042, Total iter time: 2.7507 [2021-07-07 16:24:41,252][root][INFO] - ===> Epoch[1611](8055/5): Loss 0.0208, Sem 0.0208 LR: 8.783e-02 Score 99.258 Data time: 0.2031, Forward time: 1.2609, Backward time: 1.2301, DDP time: 0.0042, Total iter time: 2.9193 [2021-07-07 16:24:55,605][root][INFO] - ===> Epoch[1612](8060/5): Loss 0.0202, Sem 0.0202 LR: 8.782e-02 Score 99.273 Data time: 0.2190, Forward time: 1.2610, Backward time: 1.2301, DDP time: 0.0042, Total iter time: 2.8183 [2021-07-07 16:25:10,152][root][INFO] - ===> Epoch[1613](8065/5): Loss 0.0193, Sem 0.0193 LR: 8.782e-02 Score 99.307 Data time: 0.2657, Forward time: 1.2610, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.8676 [2021-07-07 16:25:23,124][root][INFO] - ===> Epoch[1614](8070/5): Loss 0.0205, Sem 0.0205 LR: 8.781e-02 Score 99.262 Data time: 0.1951, Forward time: 1.2610, Backward time: 1.2301, DDP time: 0.0042, Total iter time: 2.5606 [2021-07-07 16:25:37,595][root][INFO] - ===> Epoch[1615](8075/5): Loss 0.0197, Sem 0.0197 LR: 8.780e-02 Score 99.299 Data time: 0.2349, Forward time: 1.2610, Backward time: 1.2301, DDP time: 0.0042, Total iter time: 2.8327 [2021-07-07 16:25:51,451][root][INFO] - ===> Epoch[1616](8080/5): Loss 0.0213, Sem 0.0213 LR: 8.779e-02 Score 99.236 Data time: 0.1475, Forward time: 1.2610, Backward time: 1.2301, DDP time: 0.0042, Total iter time: 2.7295 [2021-07-07 16:26:04,513][root][INFO] - ===> Epoch[1617](8085/5): Loss 0.0193, Sem 0.0193 LR: 8.779e-02 Score 99.311 Data time: 0.2915, Forward time: 1.2609, Backward time: 1.2301, DDP time: 0.0042, Total iter time: 2.5694 [2021-07-07 16:26:19,013][root][INFO] - ===> Epoch[1618](8090/5): Loss 0.0202, Sem 0.0202 LR: 8.778e-02 Score 99.275 Data time: 0.1837, Forward time: 1.2609, Backward time: 1.2301, DDP time: 0.0042, Total iter time: 2.8467 [2021-07-07 16:26:33,092][root][INFO] - ===> Epoch[1619](8095/5): Loss 0.0209, Sem 0.0209 LR: 8.777e-02 Score 99.257 Data time: 0.2157, Forward time: 1.2609, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.7660 [2021-07-07 16:26:47,105][root][INFO] - ===> Epoch[1620](8100/5): Loss 0.0204, Sem 0.0204 LR: 8.776e-02 Score 99.269 Data time: 0.2084, Forward time: 1.2608, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.7544 [2021-07-07 16:27:01,179][root][INFO] - ===> Epoch[1621](8105/5): Loss 0.0200, Sem 0.0200 LR: 8.776e-02 Score 99.283 Data time: 0.2669, Forward time: 1.2608, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.7556 [2021-07-07 16:27:15,005][root][INFO] - ===> Epoch[1622](8110/5): Loss 0.0199, Sem 0.0199 LR: 8.775e-02 Score 99.277 Data time: 0.2070, Forward time: 1.2608, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.7226 [2021-07-07 16:27:29,075][root][INFO] - ===> Epoch[1623](8115/5): Loss 0.0196, Sem 0.0196 LR: 8.774e-02 Score 99.294 Data time: 0.1894, Forward time: 1.2607, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.7636 [2021-07-07 16:27:42,597][root][INFO] - ===> Epoch[1624](8120/5): Loss 0.0203, Sem 0.0203 LR: 8.773e-02 Score 99.283 Data time: 0.2614, Forward time: 1.2608, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.6542 [2021-07-07 16:27:56,176][root][INFO] - ===> Epoch[1625](8125/5): Loss 0.0231, Sem 0.0231 LR: 8.773e-02 Score 99.233 Data time: 0.1839, Forward time: 1.2608, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.6844 [2021-07-07 16:28:10,145][root][INFO] - ===> Epoch[1626](8130/5): Loss 0.0257, Sem 0.0257 LR: 8.772e-02 Score 99.081 Data time: 0.2135, Forward time: 1.2607, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.7470 [2021-07-07 16:28:24,536][root][INFO] - ===> Epoch[1627](8135/5): Loss 0.0246, Sem 0.0246 LR: 8.771e-02 Score 99.140 Data time: 0.2332, Forward time: 1.2607, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.8300 [2021-07-07 16:28:37,951][root][INFO] - ===> Epoch[1628](8140/5): Loss 0.0253, Sem 0.0253 LR: 8.770e-02 Score 99.118 Data time: 0.2145, Forward time: 1.2607, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.6365 [2021-07-07 16:28:52,971][root][INFO] - ===> Epoch[1629](8145/5): Loss 0.0265, Sem 0.0265 LR: 8.770e-02 Score 99.064 Data time: 0.2308, Forward time: 1.2608, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.9678 [2021-07-07 16:29:07,462][root][INFO] - ===> Epoch[1630](8150/5): Loss 0.0269, Sem 0.0269 LR: 8.769e-02 Score 99.061 Data time: 0.2803, Forward time: 1.2608, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.8429 [2021-07-07 16:29:20,888][root][INFO] - ===> Epoch[1631](8155/5): Loss 0.0340, Sem 0.0340 LR: 8.768e-02 Score 98.877 Data time: 0.2760, Forward time: 1.2608, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.6393 [2021-07-07 16:29:35,684][root][INFO] - ===> Epoch[1632](8160/5): Loss 0.0351, Sem 0.0351 LR: 8.767e-02 Score 98.817 Data time: 0.1693, Forward time: 1.2608, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.9182 [2021-07-07 16:29:49,470][root][INFO] - ===> Epoch[1633](8165/5): Loss 0.0346, Sem 0.0346 LR: 8.766e-02 Score 98.799 Data time: 0.1873, Forward time: 1.2608, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.7079 [2021-07-07 16:30:03,084][root][INFO] - ===> Epoch[1634](8170/5): Loss 0.0342, Sem 0.0342 LR: 8.766e-02 Score 98.817 Data time: 0.1958, Forward time: 1.2608, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.6783 [2021-07-07 16:30:17,201][root][INFO] - ===> Epoch[1635](8175/5): Loss 0.0319, Sem 0.0319 LR: 8.765e-02 Score 98.903 Data time: 0.2174, Forward time: 1.2608, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.7718 [2021-07-07 16:30:30,749][root][INFO] - ===> Epoch[1636](8180/5): Loss 0.0318, Sem 0.0318 LR: 8.764e-02 Score 98.881 Data time: 0.2189, Forward time: 1.2608, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.6671 [2021-07-07 16:30:45,164][root][INFO] - ===> Epoch[1637](8185/5): Loss 0.0320, Sem 0.0320 LR: 8.763e-02 Score 98.893 Data time: 0.2289, Forward time: 1.2608, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.8320 [2021-07-07 16:30:59,525][root][INFO] - ===> Epoch[1638](8190/5): Loss 0.0286, Sem 0.0286 LR: 8.763e-02 Score 99.015 Data time: 0.2400, Forward time: 1.2608, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.8318 [2021-07-07 16:31:13,619][root][INFO] - ===> Epoch[1639](8195/5): Loss 0.0300, Sem 0.0300 LR: 8.762e-02 Score 98.975 Data time: 0.2319, Forward time: 1.2609, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.7561 [2021-07-07 16:31:27,764][root][INFO] - ===> Epoch[1640](8200/5): Loss 0.0299, Sem 0.0299 LR: 8.761e-02 Score 98.989 Data time: 0.2012, Forward time: 1.2609, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.7869 [2021-07-07 16:31:27,766][root][INFO] - ===> Start testing [2021-07-07 16:32:03,682][root][INFO] - 68/68: Data time: 0.0017, Iter time: 0.1298 Loss 0.215 (AVG: 0.524) Score 93.703 (AVG: 90.266) mIOU 65.893 mAP 83.524 mAcc 72.879 IOU: 63.379 0.029 80.433 75.429 94.149 85.597 20.954 78.031 98.081 38.652 77.802 85.631 58.439 mAP: 85.252 13.283 93.217 89.543 98.854 98.486 55.994 96.705 99.838 76.887 93.187 94.283 90.287 mAcc: 80.984 0.273 86.983 80.561 96.077 98.486 21.418 95.511 98.770 39.789 89.215 96.072 63.289 [2021-07-07 16:32:03,683][root][INFO] - Finished test. Elapsed time: 35.9157 [2021-07-07 16:32:04,755][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 16:32:17,560][root][INFO] - ===> Epoch[1641](8205/5): Loss 0.0338, Sem 0.0338 LR: 8.760e-02 Score 98.897 Data time: 0.2371, Forward time: 1.2609, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.5180 [2021-07-07 16:32:32,075][root][INFO] - ===> Epoch[1642](8210/5): Loss 0.0332, Sem 0.0332 LR: 8.760e-02 Score 98.854 Data time: 0.2731, Forward time: 1.2608, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.8562 [2021-07-07 16:32:45,940][root][INFO] - ===> Epoch[1643](8215/5): Loss 0.0288, Sem 0.0288 LR: 8.759e-02 Score 98.991 Data time: 0.2480, Forward time: 1.2608, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.7107 [2021-07-07 16:33:00,718][root][INFO] - ===> Epoch[1644](8220/5): Loss 0.0309, Sem 0.0309 LR: 8.758e-02 Score 98.946 Data time: 0.2638, Forward time: 1.2609, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.9118 [2021-07-07 16:33:14,785][root][INFO] - ===> Epoch[1645](8225/5): Loss 0.0290, Sem 0.0290 LR: 8.757e-02 Score 98.992 Data time: 0.2081, Forward time: 1.2608, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.7750 [2021-07-07 16:33:28,551][root][INFO] - ===> Epoch[1646](8230/5): Loss 0.0315, Sem 0.0315 LR: 8.757e-02 Score 98.936 Data time: 0.2274, Forward time: 1.2609, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.7033 [2021-07-07 16:33:42,576][root][INFO] - ===> Epoch[1647](8235/5): Loss 0.0359, Sem 0.0359 LR: 8.756e-02 Score 98.811 Data time: 0.3145, Forward time: 1.2609, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.7594 [2021-07-07 16:33:56,396][root][INFO] - ===> Epoch[1648](8240/5): Loss 0.0368, Sem 0.0368 LR: 8.755e-02 Score 98.742 Data time: 0.2400, Forward time: 1.2609, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.7131 [2021-07-07 16:34:09,993][root][INFO] - ===> Epoch[1649](8245/5): Loss 0.0415, Sem 0.0415 LR: 8.754e-02 Score 98.614 Data time: 0.1860, Forward time: 1.2609, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.6711 [2021-07-07 16:34:24,140][root][INFO] - ===> Epoch[1650](8250/5): Loss 0.0386, Sem 0.0386 LR: 8.754e-02 Score 98.673 Data time: 0.2513, Forward time: 1.2609, Backward time: 1.2301, DDP time: 0.0042, Total iter time: 2.7720 [2021-07-07 16:34:38,120][root][INFO] - ===> Epoch[1651](8255/5): Loss 0.0382, Sem 0.0382 LR: 8.753e-02 Score 98.726 Data time: 0.2621, Forward time: 1.2610, Backward time: 1.2301, DDP time: 0.0042, Total iter time: 2.7516 [2021-07-07 16:34:52,649][root][INFO] - ===> Epoch[1652](8260/5): Loss 0.0329, Sem 0.0329 LR: 8.752e-02 Score 98.873 Data time: 0.2145, Forward time: 1.2610, Backward time: 1.2301, DDP time: 0.0042, Total iter time: 2.8647 [2021-07-07 16:35:07,157][root][INFO] - ===> Epoch[1653](8265/5): Loss 0.0389, Sem 0.0389 LR: 8.751e-02 Score 98.703 Data time: 0.2105, Forward time: 1.2610, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.8454 [2021-07-07 16:35:21,463][root][INFO] - ===> Epoch[1654](8270/5): Loss 0.0401, Sem 0.0401 LR: 8.751e-02 Score 98.728 Data time: 0.2920, Forward time: 1.2609, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.8067 [2021-07-07 16:35:35,123][root][INFO] - ===> Epoch[1655](8275/5): Loss 0.0489, Sem 0.0489 LR: 8.750e-02 Score 98.504 Data time: 0.2558, Forward time: 1.2609, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.6901 [2021-07-07 16:35:48,484][root][INFO] - ===> Epoch[1656](8280/5): Loss 0.0527, Sem 0.0527 LR: 8.749e-02 Score 98.258 Data time: 0.2103, Forward time: 1.2609, Backward time: 1.2301, DDP time: 0.0042, Total iter time: 2.6367 [2021-07-07 16:36:02,426][root][INFO] - ===> Epoch[1657](8285/5): Loss 0.0438, Sem 0.0438 LR: 8.748e-02 Score 98.523 Data time: 0.2286, Forward time: 1.2609, Backward time: 1.2302, DDP time: 0.0042, Total iter time: 2.7475 [2021-07-07 16:36:17,139][root][INFO] - ===> Epoch[1658](8290/5): Loss 0.0368, Sem 0.0368 LR: 8.747e-02 Score 98.729 Data time: 0.1961, Forward time: 1.2608, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.9074 [2021-07-07 16:36:30,858][root][INFO] - ===> Epoch[1659](8295/5): Loss 0.0376, Sem 0.0376 LR: 8.747e-02 Score 98.723 Data time: 0.2141, Forward time: 1.2608, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.6966 [2021-07-07 16:36:45,511][root][INFO] - ===> Epoch[1660](8300/5): Loss 0.0348, Sem 0.0348 LR: 8.746e-02 Score 98.816 Data time: 0.1942, Forward time: 1.2608, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.8839 [2021-07-07 16:36:59,227][root][INFO] - ===> Epoch[1661](8305/5): Loss 0.0356, Sem 0.0356 LR: 8.745e-02 Score 98.782 Data time: 0.2707, Forward time: 1.2608, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.6943 [2021-07-07 16:37:13,058][root][INFO] - ===> Epoch[1662](8310/5): Loss 0.0364, Sem 0.0364 LR: 8.744e-02 Score 98.769 Data time: 0.2329, Forward time: 1.2608, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.7139 [2021-07-07 16:37:27,208][root][INFO] - ===> Epoch[1663](8315/5): Loss 0.0341, Sem 0.0341 LR: 8.744e-02 Score 98.820 Data time: 0.1689, Forward time: 1.2608, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.7854 [2021-07-07 16:37:41,860][root][INFO] - ===> Epoch[1664](8320/5): Loss 0.0317, Sem 0.0317 LR: 8.743e-02 Score 98.895 Data time: 0.2936, Forward time: 1.2609, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.8822 [2021-07-07 16:37:56,238][root][INFO] - ===> Epoch[1665](8325/5): Loss 0.0312, Sem 0.0312 LR: 8.742e-02 Score 98.922 Data time: 0.2563, Forward time: 1.2609, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.8248 [2021-07-07 16:38:10,301][root][INFO] - ===> Epoch[1666](8330/5): Loss 0.0304, Sem 0.0304 LR: 8.741e-02 Score 98.930 Data time: 0.2120, Forward time: 1.2609, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.7716 [2021-07-07 16:38:24,150][root][INFO] - ===> Epoch[1667](8335/5): Loss 0.0287, Sem 0.0287 LR: 8.741e-02 Score 99.000 Data time: 0.2029, Forward time: 1.2609, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.7343 [2021-07-07 16:38:38,328][root][INFO] - ===> Epoch[1668](8340/5): Loss 0.0291, Sem 0.0291 LR: 8.740e-02 Score 98.972 Data time: 0.1816, Forward time: 1.2608, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.7967 [2021-07-07 16:38:51,689][root][INFO] - ===> Epoch[1669](8345/5): Loss 0.0309, Sem 0.0309 LR: 8.739e-02 Score 98.980 Data time: 0.1911, Forward time: 1.2608, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.6211 [2021-07-07 16:39:05,381][root][INFO] - ===> Epoch[1670](8350/5): Loss 0.0303, Sem 0.0303 LR: 8.738e-02 Score 98.958 Data time: 0.2541, Forward time: 1.2608, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.6772 [2021-07-07 16:39:18,938][root][INFO] - ===> Epoch[1671](8355/5): Loss 0.0302, Sem 0.0302 LR: 8.738e-02 Score 98.941 Data time: 0.2139, Forward time: 1.2607, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.6547 [2021-07-07 16:39:33,867][root][INFO] - ===> Epoch[1672](8360/5): Loss 0.0287, Sem 0.0287 LR: 8.737e-02 Score 99.015 Data time: 0.1961, Forward time: 1.2607, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.9391 [2021-07-07 16:39:47,040][root][INFO] - ===> Epoch[1673](8365/5): Loss 0.0306, Sem 0.0306 LR: 8.736e-02 Score 99.017 Data time: 0.2755, Forward time: 1.2606, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.5920 [2021-07-07 16:40:01,790][root][INFO] - ===> Epoch[1674](8370/5): Loss 0.0285, Sem 0.0285 LR: 8.735e-02 Score 99.001 Data time: 0.1569, Forward time: 1.2606, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.8966 [2021-07-07 16:40:15,014][root][INFO] - ===> Epoch[1675](8375/5): Loss 0.0297, Sem 0.0297 LR: 8.735e-02 Score 98.951 Data time: 0.2167, Forward time: 1.2605, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.5982 [2021-07-07 16:40:29,578][root][INFO] - ===> Epoch[1676](8380/5): Loss 0.0290, Sem 0.0290 LR: 8.734e-02 Score 98.986 Data time: 0.2301, Forward time: 1.2605, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.8670 [2021-07-07 16:40:44,543][root][INFO] - ===> Epoch[1677](8385/5): Loss 0.0264, Sem 0.0264 LR: 8.733e-02 Score 99.062 Data time: 0.2630, Forward time: 1.2606, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.9479 [2021-07-07 16:40:58,671][root][INFO] - ===> Epoch[1678](8390/5): Loss 0.0257, Sem 0.0257 LR: 8.732e-02 Score 99.093 Data time: 0.3043, Forward time: 1.2606, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.7669 [2021-07-07 16:41:12,453][root][INFO] - ===> Epoch[1679](8395/5): Loss 0.0241, Sem 0.0241 LR: 8.731e-02 Score 99.141 Data time: 0.2257, Forward time: 1.2606, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.7127 [2021-07-07 16:41:26,235][root][INFO] - ===> Epoch[1680](8400/5): Loss 0.0230, Sem 0.0230 LR: 8.731e-02 Score 99.192 Data time: 0.2308, Forward time: 1.2606, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.7122 [2021-07-07 16:41:26,237][root][INFO] - ===> Start testing [2021-07-07 16:42:02,231][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1292 Loss 0.295 (AVG: 0.441) Score 92.582 (AVG: 91.195) mIOU 71.883 mAP 85.481 mAcc 78.208 IOU: 64.149 0.000 82.582 77.207 94.733 92.049 36.132 83.272 98.551 79.149 81.587 86.644 58.426 mAP: 85.262 5.770 91.338 91.809 98.796 98.954 66.426 98.098 99.821 94.466 93.334 94.834 92.352 mAcc: 79.027 0.000 88.422 85.429 96.077 98.054 39.721 95.412 99.415 86.247 90.426 96.036 62.442 [2021-07-07 16:42:02,231][root][INFO] - Finished test. Elapsed time: 35.9933 [2021-07-07 16:42:03,253][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2021-07-07 16:42:03,253][root][INFO] - Current best mIoU: 71.883 at iter 1 [2021-07-07 16:42:04,180][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 16:42:16,478][root][INFO] - ===> Epoch[1681](8405/5): Loss 0.0241, Sem 0.0241 LR: 8.730e-02 Score 99.138 Data time: 0.1968, Forward time: 1.2605, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.4277 [2021-07-07 16:42:30,476][root][INFO] - ===> Epoch[1682](8410/5): Loss 0.0226, Sem 0.0226 LR: 8.729e-02 Score 99.190 Data time: 0.3115, Forward time: 1.2605, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.7538 [2021-07-07 16:42:44,627][root][INFO] - ===> Epoch[1683](8415/5): Loss 0.0214, Sem 0.0214 LR: 8.728e-02 Score 99.233 Data time: 0.2169, Forward time: 1.2605, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.7882 [2021-07-07 16:42:58,238][root][INFO] - ===> Epoch[1684](8420/5): Loss 0.0232, Sem 0.0232 LR: 8.728e-02 Score 99.162 Data time: 0.2278, Forward time: 1.2605, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.6846 [2021-07-07 16:43:11,863][root][INFO] - ===> Epoch[1685](8425/5): Loss 0.0215, Sem 0.0215 LR: 8.727e-02 Score 99.231 Data time: 0.2499, Forward time: 1.2605, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.6786 [2021-07-07 16:43:24,778][root][INFO] - ===> Epoch[1686](8430/5): Loss 0.0215, Sem 0.0215 LR: 8.726e-02 Score 99.221 Data time: 0.1639, Forward time: 1.2605, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.5388 [2021-07-07 16:43:39,537][root][INFO] - ===> Epoch[1687](8435/5): Loss 0.0212, Sem 0.0212 LR: 8.725e-02 Score 99.230 Data time: 0.3057, Forward time: 1.2605, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.8960 [2021-07-07 16:43:54,320][root][INFO] - ===> Epoch[1688](8440/5): Loss 0.0224, Sem 0.0224 LR: 8.725e-02 Score 99.202 Data time: 0.2437, Forward time: 1.2606, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.9232 [2021-07-07 16:44:08,605][root][INFO] - ===> Epoch[1689](8445/5): Loss 0.0218, Sem 0.0218 LR: 8.724e-02 Score 99.237 Data time: 0.2861, Forward time: 1.2606, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.8010 [2021-07-07 16:44:22,814][root][INFO] - ===> Epoch[1690](8450/5): Loss 0.0221, Sem 0.0221 LR: 8.723e-02 Score 99.205 Data time: 0.2446, Forward time: 1.2606, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.7953 [2021-07-07 16:44:36,254][root][INFO] - ===> Epoch[1691](8455/5): Loss 0.0209, Sem 0.0209 LR: 8.722e-02 Score 99.254 Data time: 0.2290, Forward time: 1.2607, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.6415 [2021-07-07 16:44:50,708][root][INFO] - ===> Epoch[1692](8460/5): Loss 0.0205, Sem 0.0205 LR: 8.722e-02 Score 99.272 Data time: 0.2300, Forward time: 1.2607, Backward time: 1.2303, DDP time: 0.0042, Total iter time: 2.8502 [2021-07-07 16:45:04,630][root][INFO] - ===> Epoch[1693](8465/5): Loss 0.0217, Sem 0.0217 LR: 8.721e-02 Score 99.227 Data time: 0.1666, Forward time: 1.2607, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.7493 [2021-07-07 16:45:19,143][root][INFO] - ===> Epoch[1694](8470/5): Loss 0.0205, Sem 0.0205 LR: 8.720e-02 Score 99.268 Data time: 0.2381, Forward time: 1.2607, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.8571 [2021-07-07 16:45:33,697][root][INFO] - ===> Epoch[1695](8475/5): Loss 0.0215, Sem 0.0215 LR: 8.719e-02 Score 99.237 Data time: 0.2051, Forward time: 1.2606, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.8696 [2021-07-07 16:45:47,600][root][INFO] - ===> Epoch[1696](8480/5): Loss 0.0216, Sem 0.0216 LR: 8.719e-02 Score 99.226 Data time: 0.2681, Forward time: 1.2606, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.7301 [2021-07-07 16:46:01,491][root][INFO] - ===> Epoch[1697](8485/5): Loss 0.0220, Sem 0.0220 LR: 8.718e-02 Score 99.216 Data time: 0.2109, Forward time: 1.2606, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.7252 [2021-07-07 16:46:16,358][root][INFO] - ===> Epoch[1698](8490/5): Loss 0.0210, Sem 0.0210 LR: 8.717e-02 Score 99.248 Data time: 0.2424, Forward time: 1.2607, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.9134 [2021-07-07 16:46:30,572][root][INFO] - ===> Epoch[1699](8495/5): Loss 0.0210, Sem 0.0210 LR: 8.716e-02 Score 99.249 Data time: 0.1979, Forward time: 1.2607, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.8078 [2021-07-07 16:46:43,609][root][INFO] - ===> Epoch[1700](8500/5): Loss 0.0207, Sem 0.0207 LR: 8.715e-02 Score 99.255 Data time: 0.2233, Forward time: 1.2607, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.5668 [2021-07-07 16:46:57,648][root][INFO] - ===> Epoch[1701](8505/5): Loss 0.0198, Sem 0.0198 LR: 8.715e-02 Score 99.286 Data time: 0.2253, Forward time: 1.2607, Backward time: 1.2304, DDP time: 0.0042, Total iter time: 2.7695 [2021-07-07 16:47:11,534][root][INFO] - ===> Epoch[1702](8510/5): Loss 0.0203, Sem 0.0203 LR: 8.714e-02 Score 99.278 Data time: 0.2047, Forward time: 1.2607, Backward time: 1.2305, DDP time: 0.0042, Total iter time: 2.7349 [2021-07-07 16:47:26,012][root][INFO] - ===> Epoch[1703](8515/5): Loss 0.0203, Sem 0.0203 LR: 8.713e-02 Score 99.294 Data time: 0.1864, Forward time: 1.2606, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.8516 [2021-07-07 16:47:40,550][root][INFO] - ===> Epoch[1704](8520/5): Loss 0.0195, Sem 0.0195 LR: 8.712e-02 Score 99.295 Data time: 0.2898, Forward time: 1.2607, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.8576 [2021-07-07 16:47:54,834][root][INFO] - ===> Epoch[1705](8525/5): Loss 0.0196, Sem 0.0196 LR: 8.712e-02 Score 99.294 Data time: 0.2269, Forward time: 1.2606, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.8146 [2021-07-07 16:48:08,393][root][INFO] - ===> Epoch[1706](8530/5): Loss 0.0201, Sem 0.0201 LR: 8.711e-02 Score 99.295 Data time: 0.2479, Forward time: 1.2606, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.6505 [2021-07-07 16:48:22,416][root][INFO] - ===> Epoch[1707](8535/5): Loss 0.0197, Sem 0.0197 LR: 8.710e-02 Score 99.292 Data time: 0.2188, Forward time: 1.2606, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.7510 [2021-07-07 16:48:36,664][root][INFO] - ===> Epoch[1708](8540/5): Loss 0.0205, Sem 0.0205 LR: 8.709e-02 Score 99.269 Data time: 0.2141, Forward time: 1.2606, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.8029 [2021-07-07 16:48:51,092][root][INFO] - ===> Epoch[1709](8545/5): Loss 0.0203, Sem 0.0203 LR: 8.709e-02 Score 99.271 Data time: 0.2123, Forward time: 1.2606, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.8396 [2021-07-07 16:49:04,546][root][INFO] - ===> Epoch[1710](8550/5): Loss 0.0203, Sem 0.0203 LR: 8.708e-02 Score 99.274 Data time: 0.1867, Forward time: 1.2606, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.6459 [2021-07-07 16:49:18,074][root][INFO] - ===> Epoch[1711](8555/5): Loss 0.0191, Sem 0.0191 LR: 8.707e-02 Score 99.308 Data time: 0.2345, Forward time: 1.2606, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.6593 [2021-07-07 16:49:32,153][root][INFO] - ===> Epoch[1712](8560/5): Loss 0.0205, Sem 0.0205 LR: 8.706e-02 Score 99.260 Data time: 0.2280, Forward time: 1.2606, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.7668 [2021-07-07 16:49:45,618][root][INFO] - ===> Epoch[1713](8565/5): Loss 0.0185, Sem 0.0185 LR: 8.706e-02 Score 99.332 Data time: 0.2238, Forward time: 1.2606, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.6372 [2021-07-07 16:50:00,065][root][INFO] - ===> Epoch[1714](8570/5): Loss 0.0197, Sem 0.0197 LR: 8.705e-02 Score 99.299 Data time: 0.2197, Forward time: 1.2606, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.8339 [2021-07-07 16:50:14,543][root][INFO] - ===> Epoch[1715](8575/5): Loss 0.0199, Sem 0.0199 LR: 8.704e-02 Score 99.278 Data time: 0.2777, Forward time: 1.2606, Backward time: 1.2306, DDP time: 0.0042, Total iter time: 2.8455 [2021-07-07 16:50:29,178][root][INFO] - ===> Epoch[1716](8580/5): Loss 0.0186, Sem 0.0186 LR: 8.703e-02 Score 99.333 Data time: 0.1894, Forward time: 1.2606, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.8883 [2021-07-07 16:50:43,427][root][INFO] - ===> Epoch[1717](8585/5): Loss 0.0196, Sem 0.0196 LR: 8.703e-02 Score 99.296 Data time: 0.2598, Forward time: 1.2606, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.8056 [2021-07-07 16:50:57,749][root][INFO] - ===> Epoch[1718](8590/5): Loss 0.0188, Sem 0.0188 LR: 8.702e-02 Score 99.328 Data time: 0.1966, Forward time: 1.2606, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.8211 [2021-07-07 16:51:11,549][root][INFO] - ===> Epoch[1719](8595/5): Loss 0.0181, Sem 0.0181 LR: 8.701e-02 Score 99.349 Data time: 0.1839, Forward time: 1.2605, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.7188 [2021-07-07 16:51:25,895][root][INFO] - ===> Epoch[1720](8600/5): Loss 0.0186, Sem 0.0186 LR: 8.700e-02 Score 99.333 Data time: 0.1948, Forward time: 1.2605, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.8273 [2021-07-07 16:51:25,896][root][INFO] - ===> Start testing [2021-07-07 16:52:01,500][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1298 Loss 0.287 (AVG: 0.497) Score 93.218 (AVG: 91.038) mIOU 71.059 mAP 84.724 mAcc 77.100 IOU: 64.972 0.000 82.967 76.602 95.159 91.999 29.088 82.030 98.591 78.670 80.528 86.183 56.976 mAP: 85.289 9.526 94.383 90.098 98.912 98.507 59.666 97.624 99.846 88.887 92.974 93.901 91.801 mAcc: 81.229 0.000 90.698 84.382 96.543 95.916 32.071 94.884 99.510 81.363 88.800 95.839 61.068 [2021-07-07 16:52:01,501][root][INFO] - Finished test. Elapsed time: 35.6033 [2021-07-07 16:52:02,582][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 16:52:14,945][root][INFO] - ===> Epoch[1721](8605/5): Loss 0.0183, Sem 0.0183 LR: 8.699e-02 Score 99.343 Data time: 0.2018, Forward time: 1.2605, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.4325 [2021-07-07 16:52:29,250][root][INFO] - ===> Epoch[1722](8610/5): Loss 0.0187, Sem 0.0187 LR: 8.699e-02 Score 99.326 Data time: 0.2189, Forward time: 1.2605, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.8242 [2021-07-07 16:52:43,581][root][INFO] - ===> Epoch[1723](8615/5): Loss 0.0185, Sem 0.0185 LR: 8.698e-02 Score 99.329 Data time: 0.1938, Forward time: 1.2606, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.8122 [2021-07-07 16:52:57,322][root][INFO] - ===> Epoch[1724](8620/5): Loss 0.0183, Sem 0.0183 LR: 8.697e-02 Score 99.339 Data time: 0.2607, Forward time: 1.2606, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.7061 [2021-07-07 16:53:11,322][root][INFO] - ===> Epoch[1725](8625/5): Loss 0.0179, Sem 0.0179 LR: 8.696e-02 Score 99.353 Data time: 0.2515, Forward time: 1.2606, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.7482 [2021-07-07 16:53:26,169][root][INFO] - ===> Epoch[1726](8630/5): Loss 0.0168, Sem 0.0168 LR: 8.696e-02 Score 99.398 Data time: 0.2346, Forward time: 1.2607, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.9157 [2021-07-07 16:53:40,043][root][INFO] - ===> Epoch[1727](8635/5): Loss 0.0170, Sem 0.0170 LR: 8.695e-02 Score 99.381 Data time: 0.2913, Forward time: 1.2606, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.7214 [2021-07-07 16:53:53,747][root][INFO] - ===> Epoch[1728](8640/5): Loss 0.0187, Sem 0.0187 LR: 8.694e-02 Score 99.327 Data time: 0.2085, Forward time: 1.2606, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.6987 [2021-07-07 16:54:06,918][root][INFO] - ===> Epoch[1729](8645/5): Loss 0.0186, Sem 0.0186 LR: 8.693e-02 Score 99.329 Data time: 0.2007, Forward time: 1.2605, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.5819 [2021-07-07 16:54:21,554][root][INFO] - ===> Epoch[1730](8650/5): Loss 0.0173, Sem 0.0173 LR: 8.693e-02 Score 99.371 Data time: 0.2591, Forward time: 1.2605, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.8721 [2021-07-07 16:54:35,599][root][INFO] - ===> Epoch[1731](8655/5): Loss 0.0168, Sem 0.0168 LR: 8.692e-02 Score 99.389 Data time: 0.2132, Forward time: 1.2604, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.7654 [2021-07-07 16:54:49,496][root][INFO] - ===> Epoch[1732](8660/5): Loss 0.0189, Sem 0.0189 LR: 8.691e-02 Score 99.333 Data time: 0.2718, Forward time: 1.2604, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.7296 [2021-07-07 16:55:03,277][root][INFO] - ===> Epoch[1733](8665/5): Loss 0.0179, Sem 0.0179 LR: 8.690e-02 Score 99.353 Data time: 0.2520, Forward time: 1.2604, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.7000 [2021-07-07 16:55:17,196][root][INFO] - ===> Epoch[1734](8670/5): Loss 0.0175, Sem 0.0175 LR: 8.690e-02 Score 99.367 Data time: 0.2264, Forward time: 1.2604, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.7256 [2021-07-07 16:55:31,409][root][INFO] - ===> Epoch[1735](8675/5): Loss 0.0187, Sem 0.0187 LR: 8.689e-02 Score 99.332 Data time: 0.2040, Forward time: 1.2605, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.7958 [2021-07-07 16:55:45,480][root][INFO] - ===> Epoch[1736](8680/5): Loss 0.0180, Sem 0.0180 LR: 8.688e-02 Score 99.344 Data time: 0.1967, Forward time: 1.2606, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.7731 [2021-07-07 16:55:59,670][root][INFO] - ===> Epoch[1737](8685/5): Loss 0.0178, Sem 0.0178 LR: 8.687e-02 Score 99.358 Data time: 0.2267, Forward time: 1.2606, Backward time: 1.2307, DDP time: 0.0042, Total iter time: 2.7922 [2021-07-07 16:56:14,895][root][INFO] - ===> Epoch[1738](8690/5): Loss 0.0183, Sem 0.0183 LR: 8.687e-02 Score 99.345 Data time: 0.2628, Forward time: 1.2607, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 3.0012 [2021-07-07 16:56:29,371][root][INFO] - ===> Epoch[1739](8695/5): Loss 0.0170, Sem 0.0170 LR: 8.686e-02 Score 99.386 Data time: 0.2452, Forward time: 1.2607, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.8499 [2021-07-07 16:56:44,257][root][INFO] - ===> Epoch[1740](8700/5): Loss 0.0191, Sem 0.0191 LR: 8.685e-02 Score 99.320 Data time: 0.2118, Forward time: 1.2607, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.9286 [2021-07-07 16:56:57,995][root][INFO] - ===> Epoch[1741](8705/5): Loss 0.0167, Sem 0.0167 LR: 8.684e-02 Score 99.399 Data time: 0.1958, Forward time: 1.2606, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.7062 [2021-07-07 16:57:11,371][root][INFO] - ===> Epoch[1742](8710/5): Loss 0.0174, Sem 0.0174 LR: 8.683e-02 Score 99.370 Data time: 0.2073, Forward time: 1.2606, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.6185 [2021-07-07 16:57:25,805][root][INFO] - ===> Epoch[1743](8715/5): Loss 0.0172, Sem 0.0172 LR: 8.683e-02 Score 99.375 Data time: 0.2862, Forward time: 1.2607, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.8444 [2021-07-07 16:57:39,827][root][INFO] - ===> Epoch[1744](8720/5): Loss 0.0176, Sem 0.0176 LR: 8.682e-02 Score 99.368 Data time: 0.2757, Forward time: 1.2606, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.7473 [2021-07-07 16:57:53,406][root][INFO] - ===> Epoch[1745](8725/5): Loss 0.0166, Sem 0.0166 LR: 8.681e-02 Score 99.397 Data time: 0.2360, Forward time: 1.2607, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.6752 [2021-07-07 16:58:08,746][root][INFO] - ===> Epoch[1746](8730/5): Loss 0.0172, Sem 0.0172 LR: 8.680e-02 Score 99.378 Data time: 0.2409, Forward time: 1.2608, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 3.0186 [2021-07-07 16:58:22,025][root][INFO] - ===> Epoch[1747](8735/5): Loss 0.0182, Sem 0.0182 LR: 8.680e-02 Score 99.349 Data time: 0.2128, Forward time: 1.2607, Backward time: 1.2308, DDP time: 0.0042, Total iter time: 2.6217 [2021-07-07 16:58:35,895][root][INFO] - ===> Epoch[1748](8740/5): Loss 0.0164, Sem 0.0164 LR: 8.679e-02 Score 99.410 Data time: 0.1986, Forward time: 1.2607, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.7271 [2021-07-07 16:58:50,091][root][INFO] - ===> Epoch[1749](8745/5): Loss 0.0171, Sem 0.0171 LR: 8.678e-02 Score 99.383 Data time: 0.2011, Forward time: 1.2606, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.7963 [2021-07-07 16:59:04,863][root][INFO] - ===> Epoch[1750](8750/5): Loss 0.0166, Sem 0.0166 LR: 8.677e-02 Score 99.398 Data time: 0.1466, Forward time: 1.2607, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.9174 [2021-07-07 16:59:19,638][root][INFO] - ===> Epoch[1751](8755/5): Loss 0.0173, Sem 0.0173 LR: 8.677e-02 Score 99.374 Data time: 0.2111, Forward time: 1.2607, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.9101 [2021-07-07 16:59:34,655][root][INFO] - ===> Epoch[1752](8760/5): Loss 0.0165, Sem 0.0165 LR: 8.676e-02 Score 99.411 Data time: 0.2902, Forward time: 1.2607, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.9501 [2021-07-07 16:59:48,494][root][INFO] - ===> Epoch[1753](8765/5): Loss 0.0169, Sem 0.0169 LR: 8.675e-02 Score 99.388 Data time: 0.2652, Forward time: 1.2607, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.7147 [2021-07-07 17:00:03,027][root][INFO] - ===> Epoch[1754](8770/5): Loss 0.0171, Sem 0.0171 LR: 8.674e-02 Score 99.383 Data time: 0.2660, Forward time: 1.2607, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.8630 [2021-07-07 17:00:16,728][root][INFO] - ===> Epoch[1755](8775/5): Loss 0.0177, Sem 0.0177 LR: 8.674e-02 Score 99.362 Data time: 0.3558, Forward time: 1.2607, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.6761 [2021-07-07 17:00:31,294][root][INFO] - ===> Epoch[1756](8780/5): Loss 0.0173, Sem 0.0173 LR: 8.673e-02 Score 99.376 Data time: 0.2242, Forward time: 1.2607, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.8611 [2021-07-07 17:00:44,508][root][INFO] - ===> Epoch[1757](8785/5): Loss 0.0165, Sem 0.0165 LR: 8.672e-02 Score 99.403 Data time: 0.1775, Forward time: 1.2606, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.5990 [2021-07-07 17:00:59,437][root][INFO] - ===> Epoch[1758](8790/5): Loss 0.0181, Sem 0.0181 LR: 8.671e-02 Score 99.369 Data time: 0.2039, Forward time: 1.2607, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.9445 [2021-07-07 17:01:13,209][root][INFO] - ===> Epoch[1759](8795/5): Loss 0.0173, Sem 0.0173 LR: 8.671e-02 Score 99.372 Data time: 0.1933, Forward time: 1.2607, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7013 [2021-07-07 17:01:27,138][root][INFO] - ===> Epoch[1760](8800/5): Loss 0.0181, Sem 0.0181 LR: 8.670e-02 Score 99.352 Data time: 0.2269, Forward time: 1.2607, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7422 [2021-07-07 17:01:27,140][root][INFO] - ===> Start testing [2021-07-07 17:02:02,174][root][INFO] - 68/68: Data time: 0.0021, Iter time: 0.1452 Loss 0.283 (AVG: 0.484) Score 93.074 (AVG: 91.287) mIOU 71.375 mAP 84.511 mAcc 77.780 IOU: 65.060 0.000 79.290 77.402 95.424 92.014 35.726 81.045 98.571 78.881 79.742 86.979 57.739 mAP: 85.154 7.711 93.076 90.866 98.802 98.825 63.392 97.586 99.840 85.021 92.244 94.351 91.779 mAcc: 80.147 0.000 89.664 84.888 96.903 96.975 37.949 96.184 99.407 81.768 87.417 95.999 63.845 [2021-07-07 17:02:02,174][root][INFO] - Finished test. Elapsed time: 35.0339 [2021-07-07 17:02:03,261][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 17:02:16,124][root][INFO] - ===> Epoch[1761](8805/5): Loss 0.0174, Sem 0.0174 LR: 8.669e-02 Score 99.374 Data time: 0.1392, Forward time: 1.2607, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.5332 [2021-07-07 17:02:30,709][root][INFO] - ===> Epoch[1762](8810/5): Loss 0.0168, Sem 0.0168 LR: 8.668e-02 Score 99.395 Data time: 0.1543, Forward time: 1.2607, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8808 [2021-07-07 17:02:44,264][root][INFO] - ===> Epoch[1763](8815/5): Loss 0.0179, Sem 0.0179 LR: 8.667e-02 Score 99.359 Data time: 0.1825, Forward time: 1.2606, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.6818 [2021-07-07 17:02:59,486][root][INFO] - ===> Epoch[1764](8820/5): Loss 0.0164, Sem 0.0164 LR: 8.667e-02 Score 99.398 Data time: 0.2613, Forward time: 1.2606, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.9932 [2021-07-07 17:03:14,474][root][INFO] - ===> Epoch[1765](8825/5): Loss 0.0184, Sem 0.0184 LR: 8.666e-02 Score 99.355 Data time: 0.2382, Forward time: 1.2607, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.9582 [2021-07-07 17:03:29,035][root][INFO] - ===> Epoch[1766](8830/5): Loss 0.0182, Sem 0.0182 LR: 8.665e-02 Score 99.347 Data time: 0.2255, Forward time: 1.2607, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8601 [2021-07-07 17:03:43,307][root][INFO] - ===> Epoch[1767](8835/5): Loss 0.0181, Sem 0.0181 LR: 8.664e-02 Score 99.347 Data time: 0.2247, Forward time: 1.2607, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8085 [2021-07-07 17:03:57,899][root][INFO] - ===> Epoch[1768](8840/5): Loss 0.0174, Sem 0.0174 LR: 8.664e-02 Score 99.368 Data time: 0.2348, Forward time: 1.2607, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8721 [2021-07-07 17:04:11,736][root][INFO] - ===> Epoch[1769](8845/5): Loss 0.0171, Sem 0.0171 LR: 8.663e-02 Score 99.380 Data time: 0.2586, Forward time: 1.2607, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7190 [2021-07-07 17:04:25,452][root][INFO] - ===> Epoch[1770](8850/5): Loss 0.0181, Sem 0.0181 LR: 8.662e-02 Score 99.347 Data time: 0.2403, Forward time: 1.2607, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6941 [2021-07-07 17:04:39,342][root][INFO] - ===> Epoch[1771](8855/5): Loss 0.0175, Sem 0.0175 LR: 8.661e-02 Score 99.366 Data time: 0.1788, Forward time: 1.2608, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7258 [2021-07-07 17:04:53,163][root][INFO] - ===> Epoch[1772](8860/5): Loss 0.0178, Sem 0.0178 LR: 8.661e-02 Score 99.356 Data time: 0.1784, Forward time: 1.2608, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7217 [2021-07-07 17:05:08,438][root][INFO] - ===> Epoch[1773](8865/5): Loss 0.0180, Sem 0.0180 LR: 8.660e-02 Score 99.355 Data time: 0.2552, Forward time: 1.2608, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 3.0107 [2021-07-07 17:05:22,622][root][INFO] - ===> Epoch[1774](8870/5): Loss 0.0179, Sem 0.0179 LR: 8.659e-02 Score 99.352 Data time: 0.3162, Forward time: 1.2608, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7868 [2021-07-07 17:05:36,912][root][INFO] - ===> Epoch[1775](8875/5): Loss 0.0175, Sem 0.0175 LR: 8.658e-02 Score 99.372 Data time: 0.2111, Forward time: 1.2608, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8189 [2021-07-07 17:05:50,912][root][INFO] - ===> Epoch[1776](8880/5): Loss 0.0176, Sem 0.0176 LR: 8.658e-02 Score 99.368 Data time: 0.2509, Forward time: 1.2608, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7490 [2021-07-07 17:06:05,358][root][INFO] - ===> Epoch[1777](8885/5): Loss 0.0175, Sem 0.0175 LR: 8.657e-02 Score 99.371 Data time: 0.1618, Forward time: 1.2607, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8497 [2021-07-07 17:06:20,003][root][INFO] - ===> Epoch[1778](8890/5): Loss 0.0174, Sem 0.0174 LR: 8.656e-02 Score 99.374 Data time: 0.2476, Forward time: 1.2607, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8809 [2021-07-07 17:06:33,122][root][INFO] - ===> Epoch[1779](8895/5): Loss 0.0178, Sem 0.0178 LR: 8.655e-02 Score 99.355 Data time: 0.1743, Forward time: 1.2607, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.5862 [2021-07-07 17:06:47,838][root][INFO] - ===> Epoch[1780](8900/5): Loss 0.0168, Sem 0.0168 LR: 8.655e-02 Score 99.392 Data time: 0.2263, Forward time: 1.2606, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8946 [2021-07-07 17:07:02,311][root][INFO] - ===> Epoch[1781](8905/5): Loss 0.0171, Sem 0.0171 LR: 8.654e-02 Score 99.383 Data time: 0.2441, Forward time: 1.2606, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8440 [2021-07-07 17:07:16,244][root][INFO] - ===> Epoch[1782](8910/5): Loss 0.0172, Sem 0.0172 LR: 8.653e-02 Score 99.379 Data time: 0.2449, Forward time: 1.2606, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7272 [2021-07-07 17:07:30,173][root][INFO] - ===> Epoch[1783](8915/5): Loss 0.0171, Sem 0.0171 LR: 8.652e-02 Score 99.384 Data time: 0.2880, Forward time: 1.2606, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7313 [2021-07-07 17:07:43,598][root][INFO] - ===> Epoch[1784](8920/5): Loss 0.0163, Sem 0.0163 LR: 8.651e-02 Score 99.407 Data time: 0.2119, Forward time: 1.2606, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.6419 [2021-07-07 17:07:58,565][root][INFO] - ===> Epoch[1785](8925/5): Loss 0.0162, Sem 0.0162 LR: 8.651e-02 Score 99.408 Data time: 0.1978, Forward time: 1.2607, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.9545 [2021-07-07 17:08:12,522][root][INFO] - ===> Epoch[1786](8930/5): Loss 0.0172, Sem 0.0172 LR: 8.650e-02 Score 99.373 Data time: 0.2173, Forward time: 1.2607, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7524 [2021-07-07 17:08:27,451][root][INFO] - ===> Epoch[1787](8935/5): Loss 0.0168, Sem 0.0168 LR: 8.649e-02 Score 99.394 Data time: 0.2201, Forward time: 1.2607, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.9424 [2021-07-07 17:08:42,238][root][INFO] - ===> Epoch[1788](8940/5): Loss 0.0167, Sem 0.0167 LR: 8.648e-02 Score 99.396 Data time: 0.2240, Forward time: 1.2607, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.9080 [2021-07-07 17:08:56,186][root][INFO] - ===> Epoch[1789](8945/5): Loss 0.0173, Sem 0.0173 LR: 8.648e-02 Score 99.375 Data time: 0.1759, Forward time: 1.2607, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7399 [2021-07-07 17:09:11,197][root][INFO] - ===> Epoch[1790](8950/5): Loss 0.0174, Sem 0.0174 LR: 8.647e-02 Score 99.372 Data time: 0.1602, Forward time: 1.2607, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.9703 [2021-07-07 17:09:24,955][root][INFO] - ===> Epoch[1791](8955/5): Loss 0.0157, Sem 0.0157 LR: 8.646e-02 Score 99.430 Data time: 0.2002, Forward time: 1.2607, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7158 [2021-07-07 17:09:38,852][root][INFO] - ===> Epoch[1792](8960/5): Loss 0.0166, Sem 0.0166 LR: 8.645e-02 Score 99.398 Data time: 0.2339, Forward time: 1.2608, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7322 [2021-07-07 17:09:53,597][root][INFO] - ===> Epoch[1793](8965/5): Loss 0.0169, Sem 0.0169 LR: 8.645e-02 Score 99.387 Data time: 0.1774, Forward time: 1.2608, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8942 [2021-07-07 17:10:07,754][root][INFO] - ===> Epoch[1794](8970/5): Loss 0.0167, Sem 0.0167 LR: 8.644e-02 Score 99.395 Data time: 0.2119, Forward time: 1.2608, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7886 [2021-07-07 17:10:21,739][root][INFO] - ===> Epoch[1795](8975/5): Loss 0.0170, Sem 0.0170 LR: 8.643e-02 Score 99.380 Data time: 0.2856, Forward time: 1.2608, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7493 [2021-07-07 17:10:35,133][root][INFO] - ===> Epoch[1796](8980/5): Loss 0.0172, Sem 0.0172 LR: 8.642e-02 Score 99.378 Data time: 0.2765, Forward time: 1.2608, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6288 [2021-07-07 17:10:48,984][root][INFO] - ===> Epoch[1797](8985/5): Loss 0.0167, Sem 0.0167 LR: 8.642e-02 Score 99.396 Data time: 0.2210, Forward time: 1.2609, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7259 [2021-07-07 17:11:02,945][root][INFO] - ===> Epoch[1798](8990/5): Loss 0.0171, Sem 0.0171 LR: 8.641e-02 Score 99.381 Data time: 0.2102, Forward time: 1.2608, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7427 [2021-07-07 17:11:17,482][root][INFO] - ===> Epoch[1799](8995/5): Loss 0.0164, Sem 0.0164 LR: 8.640e-02 Score 99.414 Data time: 0.1818, Forward time: 1.2608, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8504 [2021-07-07 17:11:31,492][root][INFO] - ===> Epoch[1800](9000/5): Loss 0.0163, Sem 0.0163 LR: 8.639e-02 Score 99.417 Data time: 0.3076, Forward time: 1.2608, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7522 [2021-07-07 17:11:31,494][root][INFO] - ===> Start testing [2021-07-07 17:12:06,160][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1234 Loss 0.236 (AVG: 0.490) Score 94.285 (AVG: 91.403) mIOU 71.246 mAP 84.348 mAcc 77.451 IOU: 65.642 0.000 81.487 77.883 95.435 91.825 26.881 82.964 98.575 78.439 80.220 87.076 59.772 mAP: 85.356 6.697 93.956 90.214 98.906 98.845 57.636 98.107 99.836 86.141 93.193 94.429 93.207 mAcc: 81.497 0.000 91.907 85.080 96.872 97.234 28.517 96.491 99.420 80.831 87.986 96.096 64.931 [2021-07-07 17:12:06,160][root][INFO] - Finished test. Elapsed time: 34.6651 [2021-07-07 17:12:07,235][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 17:12:20,031][root][INFO] - ===> Epoch[1801](9005/5): Loss 0.0161, Sem 0.0161 LR: 8.639e-02 Score 99.415 Data time: 0.2139, Forward time: 1.2608, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.5200 [2021-07-07 17:12:33,960][root][INFO] - ===> Epoch[1802](9010/5): Loss 0.0165, Sem 0.0165 LR: 8.638e-02 Score 99.408 Data time: 0.2081, Forward time: 1.2608, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7442 [2021-07-07 17:12:47,935][root][INFO] - ===> Epoch[1803](9015/5): Loss 0.0165, Sem 0.0165 LR: 8.637e-02 Score 99.396 Data time: 0.2023, Forward time: 1.2608, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7447 [2021-07-07 17:13:02,638][root][INFO] - ===> Epoch[1804](9020/5): Loss 0.0171, Sem 0.0171 LR: 8.636e-02 Score 99.388 Data time: 0.2306, Forward time: 1.2608, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.9031 [2021-07-07 17:13:16,870][root][INFO] - ===> Epoch[1805](9025/5): Loss 0.0189, Sem 0.0189 LR: 8.635e-02 Score 99.356 Data time: 0.2228, Forward time: 1.2609, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7974 [2021-07-07 17:13:29,731][root][INFO] - ===> Epoch[1806](9030/5): Loss 0.0182, Sem 0.0182 LR: 8.635e-02 Score 99.345 Data time: 0.2418, Forward time: 1.2608, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.5304 [2021-07-07 17:13:43,998][root][INFO] - ===> Epoch[1807](9035/5): Loss 0.0180, Sem 0.0180 LR: 8.634e-02 Score 99.363 Data time: 0.3093, Forward time: 1.2609, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8108 [2021-07-07 17:13:57,614][root][INFO] - ===> Epoch[1808](9040/5): Loss 0.0182, Sem 0.0182 LR: 8.633e-02 Score 99.347 Data time: 0.2280, Forward time: 1.2609, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.6727 [2021-07-07 17:14:12,229][root][INFO] - ===> Epoch[1809](9045/5): Loss 0.0181, Sem 0.0181 LR: 8.632e-02 Score 99.360 Data time: 0.2565, Forward time: 1.2609, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8756 [2021-07-07 17:14:26,290][root][INFO] - ===> Epoch[1810](9050/5): Loss 0.0179, Sem 0.0179 LR: 8.632e-02 Score 99.352 Data time: 0.2453, Forward time: 1.2609, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7594 [2021-07-07 17:14:39,999][root][INFO] - ===> Epoch[1811](9055/5): Loss 0.0199, Sem 0.0199 LR: 8.631e-02 Score 99.323 Data time: 0.2447, Forward time: 1.2610, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.6960 [2021-07-07 17:14:54,090][root][INFO] - ===> Epoch[1812](9060/5): Loss 0.0176, Sem 0.0176 LR: 8.630e-02 Score 99.370 Data time: 0.2697, Forward time: 1.2610, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7747 [2021-07-07 17:15:07,890][root][INFO] - ===> Epoch[1813](9065/5): Loss 0.0184, Sem 0.0184 LR: 8.629e-02 Score 99.351 Data time: 0.1862, Forward time: 1.2610, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7206 [2021-07-07 17:15:21,296][root][INFO] - ===> Epoch[1814](9070/5): Loss 0.0192, Sem 0.0192 LR: 8.629e-02 Score 99.309 Data time: 0.2448, Forward time: 1.2610, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6136 [2021-07-07 17:15:35,774][root][INFO] - ===> Epoch[1815](9075/5): Loss 0.0179, Sem 0.0179 LR: 8.628e-02 Score 99.350 Data time: 0.2663, Forward time: 1.2610, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8421 [2021-07-07 17:15:49,577][root][INFO] - ===> Epoch[1816](9080/5): Loss 0.0175, Sem 0.0175 LR: 8.627e-02 Score 99.367 Data time: 0.2348, Forward time: 1.2610, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7208 [2021-07-07 17:16:04,269][root][INFO] - ===> Epoch[1817](9085/5): Loss 0.0179, Sem 0.0179 LR: 8.626e-02 Score 99.360 Data time: 0.2755, Forward time: 1.2609, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8897 [2021-07-07 17:16:18,408][root][INFO] - ===> Epoch[1818](9090/5): Loss 0.0185, Sem 0.0185 LR: 8.626e-02 Score 99.350 Data time: 0.2734, Forward time: 1.2609, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7783 [2021-07-07 17:16:31,476][root][INFO] - ===> Epoch[1819](9095/5): Loss 0.0181, Sem 0.0181 LR: 8.625e-02 Score 99.350 Data time: 0.2005, Forward time: 1.2608, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.5804 [2021-07-07 17:16:45,275][root][INFO] - ===> Epoch[1820](9100/5): Loss 0.0182, Sem 0.0182 LR: 8.624e-02 Score 99.344 Data time: 0.1975, Forward time: 1.2608, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7184 [2021-07-07 17:16:59,046][root][INFO] - ===> Epoch[1821](9105/5): Loss 0.0185, Sem 0.0185 LR: 8.623e-02 Score 99.330 Data time: 0.2025, Forward time: 1.2607, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7126 [2021-07-07 17:17:13,223][root][INFO] - ===> Epoch[1822](9110/5): Loss 0.0174, Sem 0.0174 LR: 8.623e-02 Score 99.373 Data time: 0.2007, Forward time: 1.2608, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7941 [2021-07-07 17:17:27,138][root][INFO] - ===> Epoch[1823](9115/5): Loss 0.0166, Sem 0.0166 LR: 8.622e-02 Score 99.400 Data time: 0.2244, Forward time: 1.2608, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7450 [2021-07-07 17:17:40,686][root][INFO] - ===> Epoch[1824](9120/5): Loss 0.0177, Sem 0.0177 LR: 8.621e-02 Score 99.362 Data time: 0.2175, Forward time: 1.2609, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.6611 [2021-07-07 17:17:55,358][root][INFO] - ===> Epoch[1825](9125/5): Loss 0.0162, Sem 0.0162 LR: 8.620e-02 Score 99.411 Data time: 0.2449, Forward time: 1.2609, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8894 [2021-07-07 17:18:09,720][root][INFO] - ===> Epoch[1826](9130/5): Loss 0.0170, Sem 0.0170 LR: 8.619e-02 Score 99.388 Data time: 0.2381, Forward time: 1.2610, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8266 [2021-07-07 17:18:23,944][root][INFO] - ===> Epoch[1827](9135/5): Loss 0.0163, Sem 0.0163 LR: 8.619e-02 Score 99.414 Data time: 0.2643, Forward time: 1.2610, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7958 [2021-07-07 17:18:37,608][root][INFO] - ===> Epoch[1828](9140/5): Loss 0.0166, Sem 0.0166 LR: 8.618e-02 Score 99.399 Data time: 0.2304, Forward time: 1.2609, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6860 [2021-07-07 17:18:51,526][root][INFO] - ===> Epoch[1829](9145/5): Loss 0.0170, Sem 0.0170 LR: 8.617e-02 Score 99.385 Data time: 0.2332, Forward time: 1.2609, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7392 [2021-07-07 17:19:05,618][root][INFO] - ===> Epoch[1830](9150/5): Loss 0.0174, Sem 0.0174 LR: 8.616e-02 Score 99.368 Data time: 0.1749, Forward time: 1.2609, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7715 [2021-07-07 17:19:19,422][root][INFO] - ===> Epoch[1831](9155/5): Loss 0.0162, Sem 0.0162 LR: 8.616e-02 Score 99.414 Data time: 0.2641, Forward time: 1.2609, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7106 [2021-07-07 17:19:33,626][root][INFO] - ===> Epoch[1832](9160/5): Loss 0.0163, Sem 0.0163 LR: 8.615e-02 Score 99.411 Data time: 0.1830, Forward time: 1.2609, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8019 [2021-07-07 17:19:47,342][root][INFO] - ===> Epoch[1833](9165/5): Loss 0.0163, Sem 0.0163 LR: 8.614e-02 Score 99.408 Data time: 0.1856, Forward time: 1.2608, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7080 [2021-07-07 17:20:01,543][root][INFO] - ===> Epoch[1834](9170/5): Loss 0.0161, Sem 0.0161 LR: 8.613e-02 Score 99.408 Data time: 0.2557, Forward time: 1.2609, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7984 [2021-07-07 17:20:16,086][root][INFO] - ===> Epoch[1835](9175/5): Loss 0.0162, Sem 0.0162 LR: 8.613e-02 Score 99.414 Data time: 0.2624, Forward time: 1.2609, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8582 [2021-07-07 17:20:31,025][root][INFO] - ===> Epoch[1836](9180/5): Loss 0.0160, Sem 0.0160 LR: 8.612e-02 Score 99.427 Data time: 0.2201, Forward time: 1.2609, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.9384 [2021-07-07 17:20:44,839][root][INFO] - ===> Epoch[1837](9185/5): Loss 0.0165, Sem 0.0165 LR: 8.611e-02 Score 99.403 Data time: 0.2401, Forward time: 1.2610, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7102 [2021-07-07 17:20:58,660][root][INFO] - ===> Epoch[1838](9190/5): Loss 0.0169, Sem 0.0169 LR: 8.610e-02 Score 99.392 Data time: 0.2282, Forward time: 1.2610, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7281 [2021-07-07 17:21:13,885][root][INFO] - ===> Epoch[1839](9195/5): Loss 0.0162, Sem 0.0162 LR: 8.610e-02 Score 99.416 Data time: 0.2440, Forward time: 1.2610, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 3.0042 [2021-07-07 17:21:27,745][root][INFO] - ===> Epoch[1840](9200/5): Loss 0.0162, Sem 0.0162 LR: 8.609e-02 Score 99.412 Data time: 0.2658, Forward time: 1.2611, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7298 [2021-07-07 17:21:27,746][root][INFO] - ===> Start testing [2021-07-07 17:22:04,348][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1212 Loss 0.284 (AVG: 0.493) Score 93.559 (AVG: 91.397) mIOU 71.828 mAP 85.079 mAcc 77.777 IOU: 64.991 0.000 82.419 77.853 95.097 93.148 33.568 84.529 98.634 77.325 80.512 86.890 58.795 mAP: 83.985 10.249 94.268 90.623 98.642 98.831 61.528 98.332 99.808 89.990 92.913 94.670 92.194 mAcc: 79.245 0.000 91.051 85.689 96.590 97.037 36.004 96.530 99.467 80.377 89.070 96.386 63.658 [2021-07-07 17:22:04,348][root][INFO] - Finished test. Elapsed time: 36.6013 [2021-07-07 17:22:05,455][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 17:22:19,072][root][INFO] - ===> Epoch[1841](9205/5): Loss 0.0159, Sem 0.0159 LR: 8.608e-02 Score 99.421 Data time: 0.1823, Forward time: 1.2610, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.6813 [2021-07-07 17:22:34,182][root][INFO] - ===> Epoch[1842](9210/5): Loss 0.0162, Sem 0.0162 LR: 8.607e-02 Score 99.413 Data time: 0.2705, Forward time: 1.2611, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.9810 [2021-07-07 17:22:48,697][root][INFO] - ===> Epoch[1843](9215/5): Loss 0.0160, Sem 0.0160 LR: 8.607e-02 Score 99.420 Data time: 0.2104, Forward time: 1.2611, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8521 [2021-07-07 17:23:02,523][root][INFO] - ===> Epoch[1844](9220/5): Loss 0.0155, Sem 0.0155 LR: 8.606e-02 Score 99.438 Data time: 0.1667, Forward time: 1.2610, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7251 [2021-07-07 17:23:16,759][root][INFO] - ===> Epoch[1845](9225/5): Loss 0.0155, Sem 0.0155 LR: 8.605e-02 Score 99.442 Data time: 0.2051, Forward time: 1.2610, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7936 [2021-07-07 17:23:31,029][root][INFO] - ===> Epoch[1846](9230/5): Loss 0.0158, Sem 0.0158 LR: 8.604e-02 Score 99.425 Data time: 0.2173, Forward time: 1.2610, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8030 [2021-07-07 17:23:44,668][root][INFO] - ===> Epoch[1847](9235/5): Loss 0.0160, Sem 0.0160 LR: 8.603e-02 Score 99.420 Data time: 0.2545, Forward time: 1.2610, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6815 [2021-07-07 17:23:58,940][root][INFO] - ===> Epoch[1848](9240/5): Loss 0.0162, Sem 0.0162 LR: 8.603e-02 Score 99.410 Data time: 0.2368, Forward time: 1.2610, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8033 [2021-07-07 17:24:12,928][root][INFO] - ===> Epoch[1849](9245/5): Loss 0.0161, Sem 0.0161 LR: 8.602e-02 Score 99.419 Data time: 0.2437, Forward time: 1.2611, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7467 [2021-07-07 17:24:27,161][root][INFO] - ===> Epoch[1850](9250/5): Loss 0.0158, Sem 0.0158 LR: 8.601e-02 Score 99.431 Data time: 0.2465, Forward time: 1.2611, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8042 [2021-07-07 17:24:40,949][root][INFO] - ===> Epoch[1851](9255/5): Loss 0.0162, Sem 0.0162 LR: 8.600e-02 Score 99.412 Data time: 0.1958, Forward time: 1.2611, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7101 [2021-07-07 17:24:54,208][root][INFO] - ===> Epoch[1852](9260/5): Loss 0.0159, Sem 0.0159 LR: 8.600e-02 Score 99.421 Data time: 0.1744, Forward time: 1.2610, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6149 [2021-07-07 17:25:08,168][root][INFO] - ===> Epoch[1853](9265/5): Loss 0.0157, Sem 0.0157 LR: 8.599e-02 Score 99.434 Data time: 0.2090, Forward time: 1.2610, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7467 [2021-07-07 17:25:21,204][root][INFO] - ===> Epoch[1854](9270/5): Loss 0.0162, Sem 0.0162 LR: 8.598e-02 Score 99.414 Data time: 0.2731, Forward time: 1.2610, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.5561 [2021-07-07 17:25:34,838][root][INFO] - ===> Epoch[1855](9275/5): Loss 0.0158, Sem 0.0158 LR: 8.597e-02 Score 99.432 Data time: 0.2014, Forward time: 1.2609, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6867 [2021-07-07 17:25:49,695][root][INFO] - ===> Epoch[1856](9280/5): Loss 0.0153, Sem 0.0153 LR: 8.597e-02 Score 99.447 Data time: 0.1962, Forward time: 1.2609, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.9326 [2021-07-07 17:26:03,781][root][INFO] - ===> Epoch[1857](9285/5): Loss 0.0159, Sem 0.0159 LR: 8.596e-02 Score 99.422 Data time: 0.2242, Forward time: 1.2608, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7615 [2021-07-07 17:26:17,797][root][INFO] - ===> Epoch[1858](9290/5): Loss 0.0163, Sem 0.0163 LR: 8.595e-02 Score 99.413 Data time: 0.2837, Forward time: 1.2608, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7530 [2021-07-07 17:26:31,561][root][INFO] - ===> Epoch[1859](9295/5): Loss 0.0157, Sem 0.0157 LR: 8.594e-02 Score 99.432 Data time: 0.2259, Forward time: 1.2607, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6961 [2021-07-07 17:26:45,394][root][INFO] - ===> Epoch[1860](9300/5): Loss 0.0158, Sem 0.0158 LR: 8.594e-02 Score 99.430 Data time: 0.2424, Forward time: 1.2608, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7211 [2021-07-07 17:26:59,946][root][INFO] - ===> Epoch[1861](9305/5): Loss 0.0157, Sem 0.0157 LR: 8.593e-02 Score 99.434 Data time: 0.2122, Forward time: 1.2608, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8612 [2021-07-07 17:27:13,750][root][INFO] - ===> Epoch[1862](9310/5): Loss 0.0166, Sem 0.0166 LR: 8.592e-02 Score 99.401 Data time: 0.2729, Forward time: 1.2608, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7013 [2021-07-07 17:27:27,974][root][INFO] - ===> Epoch[1863](9315/5): Loss 0.0161, Sem 0.0161 LR: 8.591e-02 Score 99.416 Data time: 0.2684, Forward time: 1.2609, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7909 [2021-07-07 17:27:42,549][root][INFO] - ===> Epoch[1864](9320/5): Loss 0.0154, Sem 0.0154 LR: 8.590e-02 Score 99.439 Data time: 0.2508, Forward time: 1.2609, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8687 [2021-07-07 17:27:56,946][root][INFO] - ===> Epoch[1865](9325/5): Loss 0.0152, Sem 0.0152 LR: 8.590e-02 Score 99.451 Data time: 0.2333, Forward time: 1.2609, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8422 [2021-07-07 17:28:10,993][root][INFO] - ===> Epoch[1866](9330/5): Loss 0.0159, Sem 0.0159 LR: 8.589e-02 Score 99.426 Data time: 0.2576, Forward time: 1.2609, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7668 [2021-07-07 17:28:24,864][root][INFO] - ===> Epoch[1867](9335/5): Loss 0.0155, Sem 0.0155 LR: 8.588e-02 Score 99.436 Data time: 0.2081, Forward time: 1.2610, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7298 [2021-07-07 17:28:38,621][root][INFO] - ===> Epoch[1868](9340/5): Loss 0.0158, Sem 0.0158 LR: 8.587e-02 Score 99.428 Data time: 0.2496, Forward time: 1.2610, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7012 [2021-07-07 17:28:53,137][root][INFO] - ===> Epoch[1869](9345/5): Loss 0.0151, Sem 0.0151 LR: 8.587e-02 Score 99.454 Data time: 0.1639, Forward time: 1.2609, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8698 [2021-07-07 17:29:07,273][root][INFO] - ===> Epoch[1870](9350/5): Loss 0.0150, Sem 0.0150 LR: 8.586e-02 Score 99.454 Data time: 0.2995, Forward time: 1.2610, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7783 [2021-07-07 17:29:20,528][root][INFO] - ===> Epoch[1871](9355/5): Loss 0.0159, Sem 0.0159 LR: 8.585e-02 Score 99.430 Data time: 0.1935, Forward time: 1.2609, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6070 [2021-07-07 17:29:34,875][root][INFO] - ===> Epoch[1872](9360/5): Loss 0.0157, Sem 0.0157 LR: 8.584e-02 Score 99.433 Data time: 0.2271, Forward time: 1.2609, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8162 [2021-07-07 17:29:48,509][root][INFO] - ===> Epoch[1873](9365/5): Loss 0.0150, Sem 0.0150 LR: 8.584e-02 Score 99.458 Data time: 0.1933, Forward time: 1.2609, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6703 [2021-07-07 17:30:02,726][root][INFO] - ===> Epoch[1874](9370/5): Loss 0.0148, Sem 0.0148 LR: 8.583e-02 Score 99.464 Data time: 0.1956, Forward time: 1.2610, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8006 [2021-07-07 17:30:15,869][root][INFO] - ===> Epoch[1875](9375/5): Loss 0.0161, Sem 0.0161 LR: 8.582e-02 Score 99.411 Data time: 0.2372, Forward time: 1.2610, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.5791 [2021-07-07 17:30:30,316][root][INFO] - ===> Epoch[1876](9380/5): Loss 0.0160, Sem 0.0160 LR: 8.581e-02 Score 99.438 Data time: 0.1899, Forward time: 1.2609, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8446 [2021-07-07 17:30:44,216][root][INFO] - ===> Epoch[1877](9385/5): Loss 0.0157, Sem 0.0157 LR: 8.581e-02 Score 99.438 Data time: 0.2433, Forward time: 1.2610, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7391 [2021-07-07 17:30:58,497][root][INFO] - ===> Epoch[1878](9390/5): Loss 0.0168, Sem 0.0168 LR: 8.580e-02 Score 99.403 Data time: 0.2827, Forward time: 1.2611, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7991 [2021-07-07 17:31:12,150][root][INFO] - ===> Epoch[1879](9395/5): Loss 0.0161, Sem 0.0161 LR: 8.579e-02 Score 99.418 Data time: 0.2715, Forward time: 1.2611, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.6788 [2021-07-07 17:31:26,161][root][INFO] - ===> Epoch[1880](9400/5): Loss 0.0176, Sem 0.0176 LR: 8.578e-02 Score 99.386 Data time: 0.2783, Forward time: 1.2610, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7411 [2021-07-07 17:31:26,162][root][INFO] - ===> Start testing [2021-07-07 17:32:01,132][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1230 Loss 0.305 (AVG: 0.532) Score 92.534 (AVG: 90.853) mIOU 70.815 mAP 85.448 mAcc 77.687 IOU: 63.018 0.106 77.818 77.468 94.755 91.636 32.547 83.398 98.234 79.018 79.478 86.607 56.515 mAP: 84.032 27.344 94.342 90.722 98.972 98.698 57.104 97.774 99.818 84.288 93.011 94.354 90.363 mAcc: 76.792 2.097 91.697 84.535 96.151 97.394 34.675 94.988 99.109 85.270 89.946 96.688 60.591 [2021-07-07 17:32:01,132][root][INFO] - Finished test. Elapsed time: 34.9696 [2021-07-07 17:32:02,296][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 17:32:15,569][root][INFO] - ===> Epoch[1881](9405/5): Loss 0.0178, Sem 0.0178 LR: 8.578e-02 Score 99.357 Data time: 0.1828, Forward time: 1.2610, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.6078 [2021-07-07 17:32:29,487][root][INFO] - ===> Epoch[1882](9410/5): Loss 0.0172, Sem 0.0172 LR: 8.577e-02 Score 99.374 Data time: 0.2289, Forward time: 1.2610, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7388 [2021-07-07 17:32:43,468][root][INFO] - ===> Epoch[1883](9415/5): Loss 0.0187, Sem 0.0187 LR: 8.576e-02 Score 99.343 Data time: 0.2523, Forward time: 1.2609, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7529 [2021-07-07 17:32:57,069][root][INFO] - ===> Epoch[1884](9420/5): Loss 0.0192, Sem 0.0192 LR: 8.575e-02 Score 99.318 Data time: 0.1592, Forward time: 1.2610, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.6762 [2021-07-07 17:33:10,386][root][INFO] - ===> Epoch[1885](9425/5): Loss 0.0179, Sem 0.0179 LR: 8.574e-02 Score 99.359 Data time: 0.2762, Forward time: 1.2610, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6123 [2021-07-07 17:33:24,326][root][INFO] - ===> Epoch[1886](9430/5): Loss 0.0180, Sem 0.0180 LR: 8.574e-02 Score 99.355 Data time: 0.2403, Forward time: 1.2610, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7414 [2021-07-07 17:33:38,812][root][INFO] - ===> Epoch[1887](9435/5): Loss 0.0174, Sem 0.0174 LR: 8.573e-02 Score 99.378 Data time: 0.1739, Forward time: 1.2610, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8445 [2021-07-07 17:33:53,487][root][INFO] - ===> Epoch[1888](9440/5): Loss 0.0179, Sem 0.0179 LR: 8.572e-02 Score 99.357 Data time: 0.2788, Forward time: 1.2610, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8937 [2021-07-07 17:34:08,214][root][INFO] - ===> Epoch[1889](9445/5): Loss 0.0166, Sem 0.0166 LR: 8.571e-02 Score 99.399 Data time: 0.3152, Forward time: 1.2611, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8875 [2021-07-07 17:34:21,929][root][INFO] - ===> Epoch[1890](9450/5): Loss 0.0173, Sem 0.0173 LR: 8.571e-02 Score 99.375 Data time: 0.3111, Forward time: 1.2611, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6915 [2021-07-07 17:34:36,779][root][INFO] - ===> Epoch[1891](9455/5): Loss 0.0167, Sem 0.0167 LR: 8.570e-02 Score 99.402 Data time: 0.2963, Forward time: 1.2612, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.9270 [2021-07-07 17:34:50,674][root][INFO] - ===> Epoch[1892](9460/5): Loss 0.0173, Sem 0.0173 LR: 8.569e-02 Score 99.377 Data time: 0.1352, Forward time: 1.2611, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7390 [2021-07-07 17:35:05,272][root][INFO] - ===> Epoch[1893](9465/5): Loss 0.0173, Sem 0.0173 LR: 8.568e-02 Score 99.391 Data time: 0.2351, Forward time: 1.2611, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8795 [2021-07-07 17:35:19,291][root][INFO] - ===> Epoch[1894](9470/5): Loss 0.0166, Sem 0.0166 LR: 8.568e-02 Score 99.401 Data time: 0.2584, Forward time: 1.2611, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7644 [2021-07-07 17:35:33,270][root][INFO] - ===> Epoch[1895](9475/5): Loss 0.0166, Sem 0.0166 LR: 8.567e-02 Score 99.401 Data time: 0.2505, Forward time: 1.2611, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7399 [2021-07-07 17:35:48,259][root][INFO] - ===> Epoch[1896](9480/5): Loss 0.0161, Sem 0.0161 LR: 8.566e-02 Score 99.422 Data time: 0.1763, Forward time: 1.2610, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.9644 [2021-07-07 17:36:01,779][root][INFO] - ===> Epoch[1897](9485/5): Loss 0.0170, Sem 0.0170 LR: 8.565e-02 Score 99.387 Data time: 0.2140, Forward time: 1.2610, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6633 [2021-07-07 17:36:15,525][root][INFO] - ===> Epoch[1898](9490/5): Loss 0.0164, Sem 0.0164 LR: 8.565e-02 Score 99.412 Data time: 0.2082, Forward time: 1.2609, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7022 [2021-07-07 17:36:29,392][root][INFO] - ===> Epoch[1899](9495/5): Loss 0.0161, Sem 0.0161 LR: 8.564e-02 Score 99.419 Data time: 0.2451, Forward time: 1.2610, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7130 [2021-07-07 17:36:42,883][root][INFO] - ===> Epoch[1900](9500/5): Loss 0.0163, Sem 0.0163 LR: 8.563e-02 Score 99.407 Data time: 0.2470, Forward time: 1.2610, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.6550 [2021-07-07 17:36:57,574][root][INFO] - ===> Epoch[1901](9505/5): Loss 0.0167, Sem 0.0167 LR: 8.562e-02 Score 99.402 Data time: 0.2250, Forward time: 1.2609, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8854 [2021-07-07 17:37:11,987][root][INFO] - ===> Epoch[1902](9510/5): Loss 0.0170, Sem 0.0170 LR: 8.562e-02 Score 99.384 Data time: 0.2958, Forward time: 1.2610, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8271 [2021-07-07 17:37:26,882][root][INFO] - ===> Epoch[1903](9515/5): Loss 0.0169, Sem 0.0169 LR: 8.561e-02 Score 99.392 Data time: 0.3092, Forward time: 1.2611, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.9332 [2021-07-07 17:37:40,705][root][INFO] - ===> Epoch[1904](9520/5): Loss 0.0164, Sem 0.0164 LR: 8.560e-02 Score 99.406 Data time: 0.2526, Forward time: 1.2612, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7155 [2021-07-07 17:37:55,208][root][INFO] - ===> Epoch[1905](9525/5): Loss 0.0175, Sem 0.0175 LR: 8.559e-02 Score 99.376 Data time: 0.2115, Forward time: 1.2611, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8534 [2021-07-07 17:38:09,164][root][INFO] - ===> Epoch[1906](9530/5): Loss 0.0177, Sem 0.0177 LR: 8.558e-02 Score 99.363 Data time: 0.2230, Forward time: 1.2612, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7471 [2021-07-07 17:38:23,315][root][INFO] - ===> Epoch[1907](9535/5): Loss 0.0156, Sem 0.0156 LR: 8.558e-02 Score 99.436 Data time: 0.1739, Forward time: 1.2612, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7845 [2021-07-07 17:38:37,289][root][INFO] - ===> Epoch[1908](9540/5): Loss 0.0169, Sem 0.0169 LR: 8.557e-02 Score 99.393 Data time: 0.2074, Forward time: 1.2612, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7557 [2021-07-07 17:38:51,270][root][INFO] - ===> Epoch[1909](9545/5): Loss 0.0162, Sem 0.0162 LR: 8.556e-02 Score 99.414 Data time: 0.2354, Forward time: 1.2612, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7515 [2021-07-07 17:39:05,127][root][INFO] - ===> Epoch[1910](9550/5): Loss 0.0163, Sem 0.0163 LR: 8.555e-02 Score 99.419 Data time: 0.2479, Forward time: 1.2612, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7169 [2021-07-07 17:39:19,313][root][INFO] - ===> Epoch[1911](9555/5): Loss 0.0162, Sem 0.0162 LR: 8.555e-02 Score 99.413 Data time: 0.2240, Forward time: 1.2612, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7875 [2021-07-07 17:39:33,651][root][INFO] - ===> Epoch[1912](9560/5): Loss 0.0168, Sem 0.0168 LR: 8.554e-02 Score 99.396 Data time: 0.2090, Forward time: 1.2612, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8110 [2021-07-07 17:39:46,854][root][INFO] - ===> Epoch[1913](9565/5): Loss 0.0156, Sem 0.0156 LR: 8.553e-02 Score 99.438 Data time: 0.2451, Forward time: 1.2612, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.5916 [2021-07-07 17:40:01,930][root][INFO] - ===> Epoch[1914](9570/5): Loss 0.0176, Sem 0.0176 LR: 8.552e-02 Score 99.379 Data time: 0.2536, Forward time: 1.2613, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.9723 [2021-07-07 17:40:15,641][root][INFO] - ===> Epoch[1915](9575/5): Loss 0.0172, Sem 0.0172 LR: 8.552e-02 Score 99.380 Data time: 0.2391, Forward time: 1.2612, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.6917 [2021-07-07 17:40:30,017][root][INFO] - ===> Epoch[1916](9580/5): Loss 0.0202, Sem 0.0202 LR: 8.551e-02 Score 99.296 Data time: 0.1969, Forward time: 1.2612, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8252 [2021-07-07 17:40:44,308][root][INFO] - ===> Epoch[1917](9585/5): Loss 0.0189, Sem 0.0189 LR: 8.550e-02 Score 99.328 Data time: 0.1659, Forward time: 1.2612, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8131 [2021-07-07 17:40:58,723][root][INFO] - ===> Epoch[1918](9590/5): Loss 0.0180, Sem 0.0180 LR: 8.549e-02 Score 99.353 Data time: 0.2166, Forward time: 1.2613, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8200 [2021-07-07 17:41:12,577][root][INFO] - ===> Epoch[1919](9595/5): Loss 0.0183, Sem 0.0183 LR: 8.549e-02 Score 99.345 Data time: 0.2643, Forward time: 1.2613, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7128 [2021-07-07 17:41:27,009][root][INFO] - ===> Epoch[1920](9600/5): Loss 0.0172, Sem 0.0172 LR: 8.548e-02 Score 99.384 Data time: 0.2603, Forward time: 1.2614, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8436 [2021-07-07 17:41:27,011][root][INFO] - ===> Start testing [2021-07-07 17:42:02,012][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1250 Loss 0.266 (AVG: 0.470) Score 93.678 (AVG: 91.491) mIOU 72.329 mAP 85.671 mAcc 78.638 IOU: 65.365 0.030 78.352 77.638 94.812 92.446 42.018 83.334 98.533 80.114 80.321 87.582 59.735 mAP: 84.845 20.402 93.348 90.687 98.746 98.937 64.840 98.274 99.825 83.087 92.974 95.440 92.325 mAcc: 79.313 0.365 86.808 84.615 96.424 97.927 46.460 95.179 99.406 82.873 90.549 96.358 66.022 [2021-07-07 17:42:02,013][root][INFO] - Finished test. Elapsed time: 35.0011 [2021-07-07 17:42:03,107][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2021-07-07 17:42:03,108][root][INFO] - Current best mIoU: 72.329 at iter 1 [2021-07-07 17:42:04,038][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 17:42:17,021][root][INFO] - ===> Epoch[1921](9605/5): Loss 0.0163, Sem 0.0163 LR: 8.547e-02 Score 99.410 Data time: 0.2036, Forward time: 1.2614, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.5603 [2021-07-07 17:42:31,077][root][INFO] - ===> Epoch[1922](9610/5): Loss 0.0166, Sem 0.0166 LR: 8.546e-02 Score 99.402 Data time: 0.2322, Forward time: 1.2614, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7665 [2021-07-07 17:42:44,901][root][INFO] - ===> Epoch[1923](9615/5): Loss 0.0167, Sem 0.0167 LR: 8.545e-02 Score 99.399 Data time: 0.1769, Forward time: 1.2614, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7184 [2021-07-07 17:42:58,995][root][INFO] - ===> Epoch[1924](9620/5): Loss 0.0168, Sem 0.0168 LR: 8.545e-02 Score 99.398 Data time: 0.1879, Forward time: 1.2614, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7758 [2021-07-07 17:43:13,737][root][INFO] - ===> Epoch[1925](9625/5): Loss 0.0160, Sem 0.0160 LR: 8.544e-02 Score 99.420 Data time: 0.2307, Forward time: 1.2614, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.9008 [2021-07-07 17:43:27,948][root][INFO] - ===> Epoch[1926](9630/5): Loss 0.0161, Sem 0.0161 LR: 8.543e-02 Score 99.422 Data time: 0.2799, Forward time: 1.2614, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7938 [2021-07-07 17:43:41,500][root][INFO] - ===> Epoch[1927](9635/5): Loss 0.0170, Sem 0.0170 LR: 8.542e-02 Score 99.382 Data time: 0.2180, Forward time: 1.2614, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6631 [2021-07-07 17:43:55,696][root][INFO] - ===> Epoch[1928](9640/5): Loss 0.0161, Sem 0.0161 LR: 8.542e-02 Score 99.420 Data time: 0.2483, Forward time: 1.2615, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7893 [2021-07-07 17:44:08,869][root][INFO] - ===> Epoch[1929](9645/5): Loss 0.0160, Sem 0.0160 LR: 8.541e-02 Score 99.421 Data time: 0.2424, Forward time: 1.2615, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.5833 [2021-07-07 17:44:22,939][root][INFO] - ===> Epoch[1930](9650/5): Loss 0.0157, Sem 0.0157 LR: 8.540e-02 Score 99.434 Data time: 0.2243, Forward time: 1.2615, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7726 [2021-07-07 17:44:36,075][root][INFO] - ===> Epoch[1931](9655/5): Loss 0.0156, Sem 0.0156 LR: 8.539e-02 Score 99.437 Data time: 0.1969, Forward time: 1.2614, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.5867 [2021-07-07 17:44:50,653][root][INFO] - ===> Epoch[1932](9660/5): Loss 0.0158, Sem 0.0158 LR: 8.539e-02 Score 99.431 Data time: 0.2237, Forward time: 1.2615, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8651 [2021-07-07 17:45:04,394][root][INFO] - ===> Epoch[1933](9665/5): Loss 0.0164, Sem 0.0164 LR: 8.538e-02 Score 99.409 Data time: 0.2105, Forward time: 1.2615, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6964 [2021-07-07 17:45:18,711][root][INFO] - ===> Epoch[1934](9670/5): Loss 0.0167, Sem 0.0167 LR: 8.537e-02 Score 99.402 Data time: 0.2158, Forward time: 1.2615, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8069 [2021-07-07 17:45:32,515][root][INFO] - ===> Epoch[1935](9675/5): Loss 0.0166, Sem 0.0166 LR: 8.536e-02 Score 99.406 Data time: 0.2590, Forward time: 1.2615, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7160 [2021-07-07 17:45:48,058][root][INFO] - ===> Epoch[1936](9680/5): Loss 0.0220, Sem 0.0220 LR: 8.536e-02 Score 99.290 Data time: 0.2575, Forward time: 1.2615, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 3.0552 [2021-07-07 17:46:01,269][root][INFO] - ===> Epoch[1937](9685/5): Loss 0.0322, Sem 0.0322 LR: 8.535e-02 Score 98.968 Data time: 0.2030, Forward time: 1.2615, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.5817 [2021-07-07 17:46:14,694][root][INFO] - ===> Epoch[1938](9690/5): Loss 0.0428, Sem 0.0428 LR: 8.534e-02 Score 98.615 Data time: 0.2072, Forward time: 1.2615, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6446 [2021-07-07 17:46:28,969][root][INFO] - ===> Epoch[1939](9695/5): Loss 0.0416, Sem 0.0416 LR: 8.533e-02 Score 98.597 Data time: 0.2087, Forward time: 1.2615, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8082 [2021-07-07 17:46:42,593][root][INFO] - ===> Epoch[1940](9700/5): Loss 0.0413, Sem 0.0413 LR: 8.533e-02 Score 98.612 Data time: 0.2018, Forward time: 1.2615, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6817 [2021-07-07 17:46:55,501][root][INFO] - ===> Epoch[1941](9705/5): Loss 0.0419, Sem 0.0419 LR: 8.532e-02 Score 98.573 Data time: 0.2110, Forward time: 1.2615, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.5402 [2021-07-07 17:47:09,391][root][INFO] - ===> Epoch[1942](9710/5): Loss 0.0446, Sem 0.0446 LR: 8.531e-02 Score 98.522 Data time: 0.2663, Forward time: 1.2616, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7388 [2021-07-07 17:47:23,175][root][INFO] - ===> Epoch[1943](9715/5): Loss 0.0415, Sem 0.0415 LR: 8.530e-02 Score 98.589 Data time: 0.2446, Forward time: 1.2615, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7137 [2021-07-07 17:47:36,912][root][INFO] - ===> Epoch[1944](9720/5): Loss 0.0426, Sem 0.0426 LR: 8.529e-02 Score 98.605 Data time: 0.2097, Forward time: 1.2615, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7125 [2021-07-07 17:47:50,615][root][INFO] - ===> Epoch[1945](9725/5): Loss 0.0357, Sem 0.0357 LR: 8.529e-02 Score 98.771 Data time: 0.2369, Forward time: 1.2615, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6898 [2021-07-07 17:48:04,230][root][INFO] - ===> Epoch[1946](9730/5): Loss 0.0329, Sem 0.0329 LR: 8.528e-02 Score 98.867 Data time: 0.2086, Forward time: 1.2615, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.6787 [2021-07-07 17:48:19,434][root][INFO] - ===> Epoch[1947](9735/5): Loss 0.0302, Sem 0.0302 LR: 8.527e-02 Score 98.932 Data time: 0.2055, Forward time: 1.2614, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.9931 [2021-07-07 17:48:33,011][root][INFO] - ===> Epoch[1948](9740/5): Loss 0.0298, Sem 0.0298 LR: 8.526e-02 Score 98.985 Data time: 0.1954, Forward time: 1.2614, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6598 [2021-07-07 17:48:47,151][root][INFO] - ===> Epoch[1949](9745/5): Loss 0.0281, Sem 0.0281 LR: 8.526e-02 Score 99.028 Data time: 0.2222, Forward time: 1.2614, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7880 [2021-07-07 17:49:00,061][root][INFO] - ===> Epoch[1950](9750/5): Loss 0.0295, Sem 0.0295 LR: 8.525e-02 Score 98.993 Data time: 0.2069, Forward time: 1.2613, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.5247 [2021-07-07 17:49:13,572][root][INFO] - ===> Epoch[1951](9755/5): Loss 0.0274, Sem 0.0274 LR: 8.524e-02 Score 99.049 Data time: 0.2447, Forward time: 1.2613, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6608 [2021-07-07 17:49:27,244][root][INFO] - ===> Epoch[1952](9760/5): Loss 0.0259, Sem 0.0259 LR: 8.523e-02 Score 99.084 Data time: 0.1688, Forward time: 1.2612, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6975 [2021-07-07 17:49:42,184][root][INFO] - ===> Epoch[1953](9765/5): Loss 0.0252, Sem 0.0252 LR: 8.523e-02 Score 99.112 Data time: 0.2870, Forward time: 1.2613, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.9296 [2021-07-07 17:49:56,532][root][INFO] - ===> Epoch[1954](9770/5): Loss 0.0238, Sem 0.0238 LR: 8.522e-02 Score 99.165 Data time: 0.2620, Forward time: 1.2613, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8288 [2021-07-07 17:50:10,358][root][INFO] - ===> Epoch[1955](9775/5): Loss 0.0233, Sem 0.0233 LR: 8.521e-02 Score 99.169 Data time: 0.2521, Forward time: 1.2612, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7170 [2021-07-07 17:50:24,437][root][INFO] - ===> Epoch[1956](9780/5): Loss 0.0228, Sem 0.0228 LR: 8.520e-02 Score 99.198 Data time: 0.2245, Forward time: 1.2613, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7761 [2021-07-07 17:50:38,445][root][INFO] - ===> Epoch[1957](9785/5): Loss 0.0227, Sem 0.0227 LR: 8.520e-02 Score 99.185 Data time: 0.2578, Forward time: 1.2613, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7579 [2021-07-07 17:50:53,004][root][INFO] - ===> Epoch[1958](9790/5): Loss 0.0226, Sem 0.0226 LR: 8.519e-02 Score 99.194 Data time: 0.2449, Forward time: 1.2613, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8677 [2021-07-07 17:51:06,499][root][INFO] - ===> Epoch[1959](9795/5): Loss 0.0211, Sem 0.0211 LR: 8.518e-02 Score 99.258 Data time: 0.1959, Forward time: 1.2613, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6618 [2021-07-07 17:51:21,571][root][INFO] - ===> Epoch[1960](9800/5): Loss 0.0216, Sem 0.0216 LR: 8.517e-02 Score 99.225 Data time: 0.2284, Forward time: 1.2614, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.9675 [2021-07-07 17:51:21,573][root][INFO] - ===> Start testing [2021-07-07 17:51:56,640][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1265 Loss 0.213 (AVG: 0.506) Score 94.294 (AVG: 90.742) mIOU 69.096 mAP 83.806 mAcc 75.160 IOU: 62.854 0.000 81.236 74.630 94.456 91.038 32.096 83.050 98.435 54.298 80.929 86.738 58.487 mAP: 85.461 8.282 89.591 87.569 98.778 98.975 65.751 98.181 99.822 76.731 93.994 94.917 91.433 mAcc: 82.379 0.000 86.624 80.302 95.713 97.577 34.746 95.871 99.002 56.462 89.688 96.656 62.053 [2021-07-07 17:51:56,640][root][INFO] - Finished test. Elapsed time: 35.0664 [2021-07-07 17:51:57,767][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 17:52:10,079][root][INFO] - ===> Epoch[1961](9805/5): Loss 0.0210, Sem 0.0210 LR: 8.516e-02 Score 99.265 Data time: 0.2160, Forward time: 1.2614, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.4256 [2021-07-07 17:52:24,293][root][INFO] - ===> Epoch[1962](9810/5): Loss 0.0227, Sem 0.0227 LR: 8.516e-02 Score 99.206 Data time: 0.2060, Forward time: 1.2613, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8044 [2021-07-07 17:52:37,938][root][INFO] - ===> Epoch[1963](9815/5): Loss 0.0240, Sem 0.0240 LR: 8.515e-02 Score 99.151 Data time: 0.2524, Forward time: 1.2613, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6828 [2021-07-07 17:52:52,103][root][INFO] - ===> Epoch[1964](9820/5): Loss 0.0226, Sem 0.0226 LR: 8.514e-02 Score 99.191 Data time: 0.2214, Forward time: 1.2613, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7888 [2021-07-07 17:53:06,425][root][INFO] - ===> Epoch[1965](9825/5): Loss 0.0231, Sem 0.0231 LR: 8.513e-02 Score 99.196 Data time: 0.1722, Forward time: 1.2614, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8304 [2021-07-07 17:53:20,012][root][INFO] - ===> Epoch[1966](9830/5): Loss 0.0229, Sem 0.0229 LR: 8.513e-02 Score 99.194 Data time: 0.1880, Forward time: 1.2613, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6842 [2021-07-07 17:53:33,466][root][INFO] - ===> Epoch[1967](9835/5): Loss 0.0211, Sem 0.0211 LR: 8.512e-02 Score 99.252 Data time: 0.1796, Forward time: 1.2613, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6396 [2021-07-07 17:53:46,821][root][INFO] - ===> Epoch[1968](9840/5): Loss 0.0218, Sem 0.0218 LR: 8.511e-02 Score 99.219 Data time: 0.2185, Forward time: 1.2613, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6240 [2021-07-07 17:54:00,986][root][INFO] - ===> Epoch[1969](9845/5): Loss 0.0197, Sem 0.0197 LR: 8.510e-02 Score 99.294 Data time: 0.2769, Forward time: 1.2613, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7732 [2021-07-07 17:54:16,240][root][INFO] - ===> Epoch[1970](9850/5): Loss 0.0204, Sem 0.0204 LR: 8.510e-02 Score 99.273 Data time: 0.3005, Forward time: 1.2614, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.9939 [2021-07-07 17:54:30,081][root][INFO] - ===> Epoch[1971](9855/5): Loss 0.0197, Sem 0.0197 LR: 8.509e-02 Score 99.294 Data time: 0.2775, Forward time: 1.2614, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7186 [2021-07-07 17:54:44,573][root][INFO] - ===> Epoch[1972](9860/5): Loss 0.0183, Sem 0.0183 LR: 8.508e-02 Score 99.334 Data time: 0.2292, Forward time: 1.2615, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8497 [2021-07-07 17:54:58,755][root][INFO] - ===> Epoch[1973](9865/5): Loss 0.0197, Sem 0.0197 LR: 8.507e-02 Score 99.297 Data time: 0.2884, Forward time: 1.2615, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7907 [2021-07-07 17:55:12,490][root][INFO] - ===> Epoch[1974](9870/5): Loss 0.0184, Sem 0.0184 LR: 8.507e-02 Score 99.344 Data time: 0.2418, Forward time: 1.2615, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.7116 [2021-07-07 17:55:26,537][root][INFO] - ===> Epoch[1975](9875/5): Loss 0.0191, Sem 0.0191 LR: 8.506e-02 Score 99.313 Data time: 0.1837, Forward time: 1.2615, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7676 [2021-07-07 17:55:40,894][root][INFO] - ===> Epoch[1976](9880/5): Loss 0.0186, Sem 0.0186 LR: 8.505e-02 Score 99.335 Data time: 0.2289, Forward time: 1.2615, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8167 [2021-07-07 17:55:54,835][root][INFO] - ===> Epoch[1977](9885/5): Loss 0.0197, Sem 0.0197 LR: 8.504e-02 Score 99.308 Data time: 0.2150, Forward time: 1.2614, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7499 [2021-07-07 17:56:09,394][root][INFO] - ===> Epoch[1978](9890/5): Loss 0.0176, Sem 0.0176 LR: 8.503e-02 Score 99.364 Data time: 0.2447, Forward time: 1.2614, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8685 [2021-07-07 17:56:23,668][root][INFO] - ===> Epoch[1979](9895/5): Loss 0.0176, Sem 0.0176 LR: 8.503e-02 Score 99.367 Data time: 0.2264, Forward time: 1.2615, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8066 [2021-07-07 17:56:37,461][root][INFO] - ===> Epoch[1980](9900/5): Loss 0.0184, Sem 0.0184 LR: 8.502e-02 Score 99.338 Data time: 0.2444, Forward time: 1.2615, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7143 [2021-07-07 17:56:51,736][root][INFO] - ===> Epoch[1981](9905/5): Loss 0.0179, Sem 0.0179 LR: 8.501e-02 Score 99.352 Data time: 0.2250, Forward time: 1.2615, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8121 [2021-07-07 17:57:05,754][root][INFO] - ===> Epoch[1982](9910/5): Loss 0.0168, Sem 0.0168 LR: 8.500e-02 Score 99.394 Data time: 0.2934, Forward time: 1.2615, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7457 [2021-07-07 17:57:19,985][root][INFO] - ===> Epoch[1983](9915/5): Loss 0.0176, Sem 0.0176 LR: 8.500e-02 Score 99.370 Data time: 0.3129, Forward time: 1.2616, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7867 [2021-07-07 17:57:34,824][root][INFO] - ===> Epoch[1984](9920/5): Loss 0.0189, Sem 0.0189 LR: 8.499e-02 Score 99.343 Data time: 0.2554, Forward time: 1.2616, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.9162 [2021-07-07 17:57:48,905][root][INFO] - ===> Epoch[1985](9925/5): Loss 0.0191, Sem 0.0191 LR: 8.498e-02 Score 99.314 Data time: 0.2248, Forward time: 1.2616, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7718 [2021-07-07 17:58:02,005][root][INFO] - ===> Epoch[1986](9930/5): Loss 0.0191, Sem 0.0191 LR: 8.497e-02 Score 99.316 Data time: 0.2259, Forward time: 1.2615, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.5832 [2021-07-07 17:58:16,064][root][INFO] - ===> Epoch[1987](9935/5): Loss 0.0187, Sem 0.0187 LR: 8.497e-02 Score 99.335 Data time: 0.2714, Forward time: 1.2615, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7601 [2021-07-07 17:58:30,081][root][INFO] - ===> Epoch[1988](9940/5): Loss 0.0192, Sem 0.0192 LR: 8.496e-02 Score 99.311 Data time: 0.2552, Forward time: 1.2616, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.7586 [2021-07-07 17:58:43,879][root][INFO] - ===> Epoch[1989](9945/5): Loss 0.0186, Sem 0.0186 LR: 8.495e-02 Score 99.342 Data time: 0.2097, Forward time: 1.2616, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.7067 [2021-07-07 17:58:58,728][root][INFO] - ===> Epoch[1990](9950/5): Loss 0.0180, Sem 0.0180 LR: 8.494e-02 Score 99.352 Data time: 0.2036, Forward time: 1.2617, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.9213 [2021-07-07 17:59:13,156][root][INFO] - ===> Epoch[1991](9955/5): Loss 0.0174, Sem 0.0174 LR: 8.494e-02 Score 99.366 Data time: 0.1924, Forward time: 1.2616, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8466 [2021-07-07 17:59:27,662][root][INFO] - ===> Epoch[1992](9960/5): Loss 0.0186, Sem 0.0186 LR: 8.493e-02 Score 99.342 Data time: 0.2604, Forward time: 1.2616, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8648 [2021-07-07 17:59:41,248][root][INFO] - ===> Epoch[1993](9965/5): Loss 0.0178, Sem 0.0178 LR: 8.492e-02 Score 99.358 Data time: 0.2541, Forward time: 1.2617, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6544 [2021-07-07 17:59:56,004][root][INFO] - ===> Epoch[1994](9970/5): Loss 0.0181, Sem 0.0181 LR: 8.491e-02 Score 99.353 Data time: 0.2392, Forward time: 1.2617, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8978 [2021-07-07 18:00:09,673][root][INFO] - ===> Epoch[1995](9975/5): Loss 0.0175, Sem 0.0175 LR: 8.491e-02 Score 99.364 Data time: 0.1796, Forward time: 1.2616, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6826 [2021-07-07 18:00:24,049][root][INFO] - ===> Epoch[1996](9980/5): Loss 0.0175, Sem 0.0175 LR: 8.490e-02 Score 99.375 Data time: 0.2171, Forward time: 1.2616, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8188 [2021-07-07 18:00:38,138][root][INFO] - ===> Epoch[1997](9985/5): Loss 0.0177, Sem 0.0177 LR: 8.489e-02 Score 99.366 Data time: 0.2285, Forward time: 1.2616, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7711 [2021-07-07 18:00:52,101][root][INFO] - ===> Epoch[1998](9990/5): Loss 0.0176, Sem 0.0176 LR: 8.488e-02 Score 99.369 Data time: 0.2515, Forward time: 1.2616, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7464 [2021-07-07 18:01:07,154][root][INFO] - ===> Epoch[1999](9995/5): Loss 0.0179, Sem 0.0179 LR: 8.487e-02 Score 99.355 Data time: 0.2245, Forward time: 1.2617, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.9708 [2021-07-07 18:01:20,784][root][INFO] - ===> Epoch[2000](10000/5): Loss 0.0170, Sem 0.0170 LR: 8.487e-02 Score 99.380 Data time: 0.2381, Forward time: 1.2617, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.6779 [2021-07-07 18:01:20,785][root][INFO] - ===> Start testing [2021-07-07 18:01:56,824][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1648 Loss 0.215 (AVG: 0.509) Score 95.255 (AVG: 91.262) mIOU 70.646 mAP 84.244 mAcc 76.615 IOU: 63.999 0.066 81.261 77.520 95.321 91.703 28.252 85.109 98.547 72.451 78.899 87.245 58.028 mAP: 83.965 10.289 93.081 89.900 98.899 99.034 58.649 98.343 99.838 84.155 92.100 95.311 91.607 mAcc: 81.571 0.547 91.540 84.249 96.858 97.435 30.041 95.298 99.382 75.918 85.081 96.927 61.145 [2021-07-07 18:01:56,825][root][INFO] - Finished test. Elapsed time: 36.0381 [2021-07-07 18:01:57,900][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 18:02:11,004][root][INFO] - ===> Epoch[2001](10005/5): Loss 0.0187, Sem 0.0187 LR: 8.486e-02 Score 99.329 Data time: 0.1498, Forward time: 1.2616, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.5809 [2021-07-07 18:02:24,572][root][INFO] - ===> Epoch[2002](10010/5): Loss 0.0182, Sem 0.0182 LR: 8.485e-02 Score 99.346 Data time: 0.2436, Forward time: 1.2616, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.6678 [2021-07-07 18:02:39,333][root][INFO] - ===> Epoch[2003](10015/5): Loss 0.0171, Sem 0.0171 LR: 8.484e-02 Score 99.381 Data time: 0.1970, Forward time: 1.2615, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.9123 [2021-07-07 18:02:53,007][root][INFO] - ===> Epoch[2004](10020/5): Loss 0.0176, Sem 0.0176 LR: 8.484e-02 Score 99.364 Data time: 0.2628, Forward time: 1.2615, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6797 [2021-07-07 18:03:06,979][root][INFO] - ===> Epoch[2005](10025/5): Loss 0.0168, Sem 0.0168 LR: 8.483e-02 Score 99.395 Data time: 0.2985, Forward time: 1.2615, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7355 [2021-07-07 18:03:21,127][root][INFO] - ===> Epoch[2006](10030/5): Loss 0.0163, Sem 0.0163 LR: 8.482e-02 Score 99.411 Data time: 0.2257, Forward time: 1.2615, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7819 [2021-07-07 18:03:35,028][root][INFO] - ===> Epoch[2007](10035/5): Loss 0.0165, Sem 0.0165 LR: 8.481e-02 Score 99.411 Data time: 0.2776, Forward time: 1.2615, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7331 [2021-07-07 18:03:48,832][root][INFO] - ===> Epoch[2008](10040/5): Loss 0.0173, Sem 0.0173 LR: 8.481e-02 Score 99.376 Data time: 0.2223, Forward time: 1.2615, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7031 [2021-07-07 18:04:02,688][root][INFO] - ===> Epoch[2009](10045/5): Loss 0.0170, Sem 0.0170 LR: 8.480e-02 Score 99.393 Data time: 0.2224, Forward time: 1.2615, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7288 [2021-07-07 18:04:17,459][root][INFO] - ===> Epoch[2010](10050/5): Loss 0.0161, Sem 0.0161 LR: 8.479e-02 Score 99.417 Data time: 0.2479, Forward time: 1.2616, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.9022 [2021-07-07 18:04:31,097][root][INFO] - ===> Epoch[2011](10055/5): Loss 0.0165, Sem 0.0165 LR: 8.478e-02 Score 99.406 Data time: 0.2841, Forward time: 1.2616, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6849 [2021-07-07 18:04:44,825][root][INFO] - ===> Epoch[2012](10060/5): Loss 0.0162, Sem 0.0162 LR: 8.478e-02 Score 99.417 Data time: 0.1708, Forward time: 1.2616, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7086 [2021-07-07 18:04:59,902][root][INFO] - ===> Epoch[2013](10065/5): Loss 0.0164, Sem 0.0164 LR: 8.477e-02 Score 99.409 Data time: 0.2013, Forward time: 1.2615, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.9632 [2021-07-07 18:05:13,449][root][INFO] - ===> Epoch[2014](10070/5): Loss 0.0164, Sem 0.0164 LR: 8.476e-02 Score 99.414 Data time: 0.2566, Forward time: 1.2615, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.6549 [2021-07-07 18:05:28,208][root][INFO] - ===> Epoch[2015](10075/5): Loss 0.0161, Sem 0.0161 LR: 8.475e-02 Score 99.416 Data time: 0.2227, Forward time: 1.2615, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.9101 [2021-07-07 18:05:42,499][root][INFO] - ===> Epoch[2016](10080/5): Loss 0.0155, Sem 0.0155 LR: 8.474e-02 Score 99.436 Data time: 0.2377, Forward time: 1.2615, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8064 [2021-07-07 18:05:56,290][root][INFO] - ===> Epoch[2017](10085/5): Loss 0.0166, Sem 0.0166 LR: 8.474e-02 Score 99.398 Data time: 0.2491, Forward time: 1.2615, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7154 [2021-07-07 18:06:11,045][root][INFO] - ===> Epoch[2018](10090/5): Loss 0.0155, Sem 0.0155 LR: 8.473e-02 Score 99.438 Data time: 0.2077, Forward time: 1.2615, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8997 [2021-07-07 18:06:24,943][root][INFO] - ===> Epoch[2019](10095/5): Loss 0.0155, Sem 0.0155 LR: 8.472e-02 Score 99.440 Data time: 0.2144, Forward time: 1.2615, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7404 [2021-07-07 18:06:38,930][root][INFO] - ===> Epoch[2020](10100/5): Loss 0.0164, Sem 0.0164 LR: 8.471e-02 Score 99.413 Data time: 0.2107, Forward time: 1.2615, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7449 [2021-07-07 18:06:52,650][root][INFO] - ===> Epoch[2021](10105/5): Loss 0.0163, Sem 0.0163 LR: 8.471e-02 Score 99.414 Data time: 0.2337, Forward time: 1.2615, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7032 [2021-07-07 18:07:06,536][root][INFO] - ===> Epoch[2022](10110/5): Loss 0.0157, Sem 0.0157 LR: 8.470e-02 Score 99.432 Data time: 0.2674, Forward time: 1.2615, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7325 [2021-07-07 18:07:20,815][root][INFO] - ===> Epoch[2023](10115/5): Loss 0.0155, Sem 0.0155 LR: 8.469e-02 Score 99.440 Data time: 0.2201, Forward time: 1.2614, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8143 [2021-07-07 18:07:35,069][root][INFO] - ===> Epoch[2024](10120/5): Loss 0.0162, Sem 0.0162 LR: 8.468e-02 Score 99.418 Data time: 0.2257, Forward time: 1.2614, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8023 [2021-07-07 18:07:49,058][root][INFO] - ===> Epoch[2025](10125/5): Loss 0.0159, Sem 0.0159 LR: 8.468e-02 Score 99.424 Data time: 0.2671, Forward time: 1.2614, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7480 [2021-07-07 18:08:04,525][root][INFO] - ===> Epoch[2026](10130/5): Loss 0.0158, Sem 0.0158 LR: 8.467e-02 Score 99.429 Data time: 0.2495, Forward time: 1.2614, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 3.0501 [2021-07-07 18:08:18,965][root][INFO] - ===> Epoch[2027](10135/5): Loss 0.0158, Sem 0.0158 LR: 8.466e-02 Score 99.429 Data time: 0.2475, Forward time: 1.2615, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8480 [2021-07-07 18:08:33,007][root][INFO] - ===> Epoch[2028](10140/5): Loss 0.0154, Sem 0.0154 LR: 8.465e-02 Score 99.446 Data time: 0.2130, Forward time: 1.2615, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7588 [2021-07-07 18:08:46,697][root][INFO] - ===> Epoch[2029](10145/5): Loss 0.0159, Sem 0.0159 LR: 8.465e-02 Score 99.425 Data time: 0.1727, Forward time: 1.2615, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.6916 [2021-07-07 18:09:00,070][root][INFO] - ===> Epoch[2030](10150/5): Loss 0.0166, Sem 0.0166 LR: 8.464e-02 Score 99.407 Data time: 0.1651, Forward time: 1.2614, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.6273 [2021-07-07 18:09:13,864][root][INFO] - ===> Epoch[2031](10155/5): Loss 0.0163, Sem 0.0163 LR: 8.463e-02 Score 99.416 Data time: 0.2124, Forward time: 1.2615, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7177 [2021-07-07 18:09:28,706][root][INFO] - ===> Epoch[2032](10160/5): Loss 0.0153, Sem 0.0153 LR: 8.462e-02 Score 99.440 Data time: 0.2143, Forward time: 1.2615, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.9191 [2021-07-07 18:09:42,260][root][INFO] - ===> Epoch[2033](10165/5): Loss 0.0159, Sem 0.0159 LR: 8.461e-02 Score 99.424 Data time: 0.1535, Forward time: 1.2615, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6729 [2021-07-07 18:09:56,442][root][INFO] - ===> Epoch[2034](10170/5): Loss 0.0162, Sem 0.0162 LR: 8.461e-02 Score 99.414 Data time: 0.2027, Forward time: 1.2615, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7894 [2021-07-07 18:10:10,333][root][INFO] - ===> Epoch[2035](10175/5): Loss 0.0157, Sem 0.0157 LR: 8.460e-02 Score 99.430 Data time: 0.1762, Forward time: 1.2614, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7492 [2021-07-07 18:10:24,633][root][INFO] - ===> Epoch[2036](10180/5): Loss 0.0157, Sem 0.0157 LR: 8.459e-02 Score 99.435 Data time: 0.2033, Forward time: 1.2615, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8179 [2021-07-07 18:10:37,891][root][INFO] - ===> Epoch[2037](10185/5): Loss 0.0154, Sem 0.0154 LR: 8.458e-02 Score 99.442 Data time: 0.2425, Forward time: 1.2615, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.6068 [2021-07-07 18:10:52,941][root][INFO] - ===> Epoch[2038](10190/5): Loss 0.0158, Sem 0.0158 LR: 8.458e-02 Score 99.430 Data time: 0.2368, Forward time: 1.2615, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.9564 [2021-07-07 18:11:06,699][root][INFO] - ===> Epoch[2039](10195/5): Loss 0.0160, Sem 0.0160 LR: 8.457e-02 Score 99.419 Data time: 0.2430, Forward time: 1.2615, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7042 [2021-07-07 18:11:21,592][root][INFO] - ===> Epoch[2040](10200/5): Loss 0.0154, Sem 0.0154 LR: 8.456e-02 Score 99.442 Data time: 0.2667, Forward time: 1.2616, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.9324 [2021-07-07 18:11:21,594][root][INFO] - ===> Start testing [2021-07-07 18:11:57,738][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1271 Loss 0.274 (AVG: 0.479) Score 94.021 (AVG: 91.804) mIOU 72.495 mAP 84.352 mAcc 78.025 IOU: 66.090 0.000 84.602 78.768 95.049 92.500 38.998 85.656 98.533 73.435 80.187 87.873 60.742 mAP: 86.066 6.924 89.773 90.068 98.890 98.700 64.646 98.588 99.824 82.403 92.590 95.507 92.597 mAcc: 82.722 0.000 87.089 85.951 96.361 96.316 42.395 96.827 99.278 76.608 89.022 96.406 65.357 [2021-07-07 18:11:57,738][root][INFO] - Finished test. Elapsed time: 36.1434 [2021-07-07 18:11:58,810][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2021-07-07 18:11:58,811][root][INFO] - Current best mIoU: 72.495 at iter 1 [2021-07-07 18:11:59,714][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 18:12:12,334][root][INFO] - ===> Epoch[2041](10205/5): Loss 0.0156, Sem 0.0156 LR: 8.455e-02 Score 99.436 Data time: 0.1875, Forward time: 1.2615, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.4933 [2021-07-07 18:12:26,643][root][INFO] - ===> Epoch[2042](10210/5): Loss 0.0157, Sem 0.0157 LR: 8.455e-02 Score 99.436 Data time: 0.2869, Forward time: 1.2616, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8075 [2021-07-07 18:12:40,473][root][INFO] - ===> Epoch[2043](10215/5): Loss 0.0160, Sem 0.0160 LR: 8.454e-02 Score 99.422 Data time: 0.2204, Forward time: 1.2616, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7165 [2021-07-07 18:12:54,260][root][INFO] - ===> Epoch[2044](10220/5): Loss 0.0156, Sem 0.0156 LR: 8.453e-02 Score 99.434 Data time: 0.2936, Forward time: 1.2616, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6981 [2021-07-07 18:13:07,760][root][INFO] - ===> Epoch[2045](10225/5): Loss 0.0146, Sem 0.0146 LR: 8.452e-02 Score 99.471 Data time: 0.2437, Forward time: 1.2616, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6469 [2021-07-07 18:13:21,815][root][INFO] - ===> Epoch[2046](10230/5): Loss 0.0152, Sem 0.0152 LR: 8.452e-02 Score 99.448 Data time: 0.1470, Forward time: 1.2617, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7706 [2021-07-07 18:13:36,097][root][INFO] - ===> Epoch[2047](10235/5): Loss 0.0148, Sem 0.0148 LR: 8.451e-02 Score 99.462 Data time: 0.2613, Forward time: 1.2616, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8067 [2021-07-07 18:13:50,221][root][INFO] - ===> Epoch[2048](10240/5): Loss 0.0150, Sem 0.0150 LR: 8.450e-02 Score 99.451 Data time: 0.2001, Forward time: 1.2616, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7762 [2021-07-07 18:14:03,665][root][INFO] - ===> Epoch[2049](10245/5): Loss 0.0151, Sem 0.0151 LR: 8.449e-02 Score 99.455 Data time: 0.2095, Forward time: 1.2616, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6513 [2021-07-07 18:14:17,989][root][INFO] - ===> Epoch[2050](10250/5): Loss 0.0154, Sem 0.0154 LR: 8.448e-02 Score 99.443 Data time: 0.2292, Forward time: 1.2616, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8194 [2021-07-07 18:14:31,571][root][INFO] - ===> Epoch[2051](10255/5): Loss 0.0157, Sem 0.0157 LR: 8.448e-02 Score 99.436 Data time: 0.3030, Forward time: 1.2616, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6648 [2021-07-07 18:14:45,764][root][INFO] - ===> Epoch[2052](10260/5): Loss 0.0153, Sem 0.0153 LR: 8.447e-02 Score 99.442 Data time: 0.2934, Forward time: 1.2616, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7897 [2021-07-07 18:15:00,683][root][INFO] - ===> Epoch[2053](10265/5): Loss 0.0160, Sem 0.0160 LR: 8.446e-02 Score 99.419 Data time: 0.1907, Forward time: 1.2616, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.9350 [2021-07-07 18:15:15,044][root][INFO] - ===> Epoch[2054](10270/5): Loss 0.0152, Sem 0.0152 LR: 8.445e-02 Score 99.452 Data time: 0.2659, Forward time: 1.2615, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8220 [2021-07-07 18:15:28,652][root][INFO] - ===> Epoch[2055](10275/5): Loss 0.0146, Sem 0.0146 LR: 8.445e-02 Score 99.473 Data time: 0.2683, Forward time: 1.2615, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6774 [2021-07-07 18:15:42,865][root][INFO] - ===> Epoch[2056](10280/5): Loss 0.0147, Sem 0.0147 LR: 8.444e-02 Score 99.468 Data time: 0.2219, Forward time: 1.2615, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7914 [2021-07-07 18:15:57,408][root][INFO] - ===> Epoch[2057](10285/5): Loss 0.0157, Sem 0.0157 LR: 8.443e-02 Score 99.435 Data time: 0.2416, Forward time: 1.2615, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8585 [2021-07-07 18:16:10,360][root][INFO] - ===> Epoch[2058](10290/5): Loss 0.0158, Sem 0.0158 LR: 8.442e-02 Score 99.428 Data time: 0.2437, Forward time: 1.2614, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.5442 [2021-07-07 18:16:24,129][root][INFO] - ===> Epoch[2059](10295/5): Loss 0.0148, Sem 0.0148 LR: 8.442e-02 Score 99.464 Data time: 0.2244, Forward time: 1.2615, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7021 [2021-07-07 18:16:37,634][root][INFO] - ===> Epoch[2060](10300/5): Loss 0.0159, Sem 0.0159 LR: 8.441e-02 Score 99.429 Data time: 0.2570, Forward time: 1.2615, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6498 [2021-07-07 18:16:51,047][root][INFO] - ===> Epoch[2061](10305/5): Loss 0.0160, Sem 0.0160 LR: 8.440e-02 Score 99.423 Data time: 0.2275, Forward time: 1.2615, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6276 [2021-07-07 18:17:05,223][root][INFO] - ===> Epoch[2062](10310/5): Loss 0.0152, Sem 0.0152 LR: 8.439e-02 Score 99.458 Data time: 0.2737, Forward time: 1.2615, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7879 [2021-07-07 18:17:18,979][root][INFO] - ===> Epoch[2063](10315/5): Loss 0.0156, Sem 0.0156 LR: 8.439e-02 Score 99.439 Data time: 0.1827, Forward time: 1.2615, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7160 [2021-07-07 18:17:33,204][root][INFO] - ===> Epoch[2064](10320/5): Loss 0.0157, Sem 0.0157 LR: 8.438e-02 Score 99.437 Data time: 0.2037, Forward time: 1.2614, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7988 [2021-07-07 18:17:47,800][root][INFO] - ===> Epoch[2065](10325/5): Loss 0.0155, Sem 0.0155 LR: 8.437e-02 Score 99.438 Data time: 0.2249, Forward time: 1.2614, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8688 [2021-07-07 18:18:01,809][root][INFO] - ===> Epoch[2066](10330/5): Loss 0.0161, Sem 0.0161 LR: 8.436e-02 Score 99.418 Data time: 0.2223, Forward time: 1.2615, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7376 [2021-07-07 18:18:15,556][root][INFO] - ===> Epoch[2067](10335/5): Loss 0.0166, Sem 0.0166 LR: 8.435e-02 Score 99.402 Data time: 0.2490, Forward time: 1.2615, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6987 [2021-07-07 18:18:30,284][root][INFO] - ===> Epoch[2068](10340/5): Loss 0.0155, Sem 0.0155 LR: 8.435e-02 Score 99.443 Data time: 0.2850, Forward time: 1.2616, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8996 [2021-07-07 18:18:45,977][root][INFO] - ===> Epoch[2069](10345/5): Loss 0.0152, Sem 0.0152 LR: 8.434e-02 Score 99.450 Data time: 0.2175, Forward time: 1.2616, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 3.0766 [2021-07-07 18:19:00,345][root][INFO] - ===> Epoch[2070](10350/5): Loss 0.0157, Sem 0.0157 LR: 8.433e-02 Score 99.436 Data time: 0.2301, Forward time: 1.2616, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8312 [2021-07-07 18:19:14,499][root][INFO] - ===> Epoch[2071](10355/5): Loss 0.0155, Sem 0.0155 LR: 8.432e-02 Score 99.442 Data time: 0.2474, Forward time: 1.2616, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7809 [2021-07-07 18:19:27,958][root][INFO] - ===> Epoch[2072](10360/5): Loss 0.0162, Sem 0.0162 LR: 8.432e-02 Score 99.414 Data time: 0.1614, Forward time: 1.2616, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6483 [2021-07-07 18:19:42,064][root][INFO] - ===> Epoch[2073](10365/5): Loss 0.0147, Sem 0.0147 LR: 8.431e-02 Score 99.469 Data time: 0.2244, Forward time: 1.2616, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7759 [2021-07-07 18:19:57,243][root][INFO] - ===> Epoch[2074](10370/5): Loss 0.0154, Sem 0.0154 LR: 8.430e-02 Score 99.441 Data time: 0.2569, Forward time: 1.2616, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.9810 [2021-07-07 18:20:10,833][root][INFO] - ===> Epoch[2075](10375/5): Loss 0.0152, Sem 0.0152 LR: 8.429e-02 Score 99.452 Data time: 0.2971, Forward time: 1.2616, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6664 [2021-07-07 18:20:24,801][root][INFO] - ===> Epoch[2076](10380/5): Loss 0.0156, Sem 0.0156 LR: 8.429e-02 Score 99.435 Data time: 0.2243, Forward time: 1.2616, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7451 [2021-07-07 18:20:39,840][root][INFO] - ===> Epoch[2077](10385/5): Loss 0.0149, Sem 0.0149 LR: 8.428e-02 Score 99.461 Data time: 0.2409, Forward time: 1.2617, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.9477 [2021-07-07 18:20:53,292][root][INFO] - ===> Epoch[2078](10390/5): Loss 0.0151, Sem 0.0151 LR: 8.427e-02 Score 99.451 Data time: 0.2604, Forward time: 1.2617, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6439 [2021-07-07 18:21:07,049][root][INFO] - ===> Epoch[2079](10395/5): Loss 0.0154, Sem 0.0154 LR: 8.426e-02 Score 99.444 Data time: 0.2154, Forward time: 1.2617, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7132 [2021-07-07 18:21:21,613][root][INFO] - ===> Epoch[2080](10400/5): Loss 0.0142, Sem 0.0142 LR: 8.426e-02 Score 99.482 Data time: 0.2795, Forward time: 1.2617, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8664 [2021-07-07 18:21:21,616][root][INFO] - ===> Start testing [2021-07-07 18:21:55,840][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1497 Loss 0.327 (AVG: 0.503) Score 93.318 (AVG: 91.616) mIOU 72.093 mAP 85.003 mAcc 77.639 IOU: 66.203 0.045 85.913 78.295 95.062 92.176 34.679 85.060 98.625 74.449 79.932 87.330 59.445 mAP: 86.280 11.162 93.513 89.716 98.743 98.765 60.989 98.492 99.818 86.891 92.820 95.112 92.734 mAcc: 83.056 0.365 89.380 84.639 96.360 96.391 38.187 97.662 99.459 76.638 88.080 96.605 62.487 [2021-07-07 18:21:55,840][root][INFO] - Finished test. Elapsed time: 34.2233 [2021-07-07 18:21:57,023][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 18:22:10,061][root][INFO] - ===> Epoch[2081](10405/5): Loss 0.0153, Sem 0.0153 LR: 8.425e-02 Score 99.447 Data time: 0.1945, Forward time: 1.2617, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.5704 [2021-07-07 18:22:24,635][root][INFO] - ===> Epoch[2082](10410/5): Loss 0.0148, Sem 0.0148 LR: 8.424e-02 Score 99.468 Data time: 0.2043, Forward time: 1.2617, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8693 [2021-07-07 18:22:39,027][root][INFO] - ===> Epoch[2083](10415/5): Loss 0.0152, Sem 0.0152 LR: 8.423e-02 Score 99.449 Data time: 0.2561, Forward time: 1.2617, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8305 [2021-07-07 18:22:53,241][root][INFO] - ===> Epoch[2084](10420/5): Loss 0.0148, Sem 0.0148 LR: 8.423e-02 Score 99.461 Data time: 0.2288, Forward time: 1.2618, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8085 [2021-07-07 18:23:07,412][root][INFO] - ===> Epoch[2085](10425/5): Loss 0.0155, Sem 0.0155 LR: 8.422e-02 Score 99.435 Data time: 0.1889, Forward time: 1.2618, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7853 [2021-07-07 18:23:21,869][root][INFO] - ===> Epoch[2086](10430/5): Loss 0.0149, Sem 0.0149 LR: 8.421e-02 Score 99.461 Data time: 0.2255, Forward time: 1.2617, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8465 [2021-07-07 18:23:36,156][root][INFO] - ===> Epoch[2087](10435/5): Loss 0.0148, Sem 0.0148 LR: 8.420e-02 Score 99.461 Data time: 0.2245, Forward time: 1.2617, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7962 [2021-07-07 18:23:49,820][root][INFO] - ===> Epoch[2088](10440/5): Loss 0.0143, Sem 0.0143 LR: 8.419e-02 Score 99.481 Data time: 0.2133, Forward time: 1.2617, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6893 [2021-07-07 18:24:03,141][root][INFO] - ===> Epoch[2089](10445/5): Loss 0.0140, Sem 0.0140 LR: 8.419e-02 Score 99.498 Data time: 0.2348, Forward time: 1.2617, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6112 [2021-07-07 18:24:17,283][root][INFO] - ===> Epoch[2090](10450/5): Loss 0.0164, Sem 0.0164 LR: 8.418e-02 Score 99.434 Data time: 0.2691, Forward time: 1.2618, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7866 [2021-07-07 18:24:31,100][root][INFO] - ===> Epoch[2091](10455/5): Loss 0.0147, Sem 0.0147 LR: 8.417e-02 Score 99.470 Data time: 0.1867, Forward time: 1.2618, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7220 [2021-07-07 18:24:44,815][root][INFO] - ===> Epoch[2092](10460/5): Loss 0.0149, Sem 0.0149 LR: 8.416e-02 Score 99.465 Data time: 0.2414, Forward time: 1.2618, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6977 [2021-07-07 18:24:59,764][root][INFO] - ===> Epoch[2093](10465/5): Loss 0.0150, Sem 0.0150 LR: 8.416e-02 Score 99.460 Data time: 0.1660, Forward time: 1.2618, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.9511 [2021-07-07 18:25:13,818][root][INFO] - ===> Epoch[2094](10470/5): Loss 0.0145, Sem 0.0145 LR: 8.415e-02 Score 99.476 Data time: 0.2404, Forward time: 1.2618, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7599 [2021-07-07 18:25:28,089][root][INFO] - ===> Epoch[2095](10475/5): Loss 0.0145, Sem 0.0145 LR: 8.414e-02 Score 99.477 Data time: 0.2575, Forward time: 1.2618, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8001 [2021-07-07 18:25:42,005][root][INFO] - ===> Epoch[2096](10480/5): Loss 0.0147, Sem 0.0147 LR: 8.413e-02 Score 99.466 Data time: 0.2134, Forward time: 1.2618, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7318 [2021-07-07 18:25:55,827][root][INFO] - ===> Epoch[2097](10485/5): Loss 0.0157, Sem 0.0157 LR: 8.413e-02 Score 99.427 Data time: 0.2395, Forward time: 1.2617, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7094 [2021-07-07 18:26:10,084][root][INFO] - ===> Epoch[2098](10490/5): Loss 0.0156, Sem 0.0156 LR: 8.412e-02 Score 99.443 Data time: 0.3023, Forward time: 1.2618, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7930 [2021-07-07 18:26:24,218][root][INFO] - ===> Epoch[2099](10495/5): Loss 0.0148, Sem 0.0148 LR: 8.411e-02 Score 99.464 Data time: 0.2134, Forward time: 1.2617, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7777 [2021-07-07 18:26:38,048][root][INFO] - ===> Epoch[2100](10500/5): Loss 0.0148, Sem 0.0148 LR: 8.410e-02 Score 99.467 Data time: 0.1875, Forward time: 1.2617, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7115 [2021-07-07 18:26:51,957][root][INFO] - ===> Epoch[2101](10505/5): Loss 0.0155, Sem 0.0155 LR: 8.410e-02 Score 99.437 Data time: 0.2417, Forward time: 1.2618, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7360 [2021-07-07 18:27:07,161][root][INFO] - ===> Epoch[2102](10510/5): Loss 0.0152, Sem 0.0152 LR: 8.409e-02 Score 99.451 Data time: 0.2589, Forward time: 1.2619, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.9993 [2021-07-07 18:27:21,745][root][INFO] - ===> Epoch[2103](10515/5): Loss 0.0167, Sem 0.0167 LR: 8.408e-02 Score 99.396 Data time: 0.2188, Forward time: 1.2618, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8765 [2021-07-07 18:27:36,344][root][INFO] - ===> Epoch[2104](10520/5): Loss 0.0148, Sem 0.0148 LR: 8.407e-02 Score 99.463 Data time: 0.2844, Forward time: 1.2619, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8641 [2021-07-07 18:27:50,899][root][INFO] - ===> Epoch[2105](10525/5): Loss 0.0161, Sem 0.0161 LR: 8.406e-02 Score 99.426 Data time: 0.2075, Forward time: 1.2619, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8651 [2021-07-07 18:28:04,586][root][INFO] - ===> Epoch[2106](10530/5): Loss 0.0159, Sem 0.0159 LR: 8.406e-02 Score 99.429 Data time: 0.2471, Forward time: 1.2619, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6929 [2021-07-07 18:28:18,358][root][INFO] - ===> Epoch[2107](10535/5): Loss 0.0154, Sem 0.0154 LR: 8.405e-02 Score 99.443 Data time: 0.3154, Forward time: 1.2620, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7089 [2021-07-07 18:28:32,744][root][INFO] - ===> Epoch[2108](10540/5): Loss 0.0157, Sem 0.0157 LR: 8.404e-02 Score 99.432 Data time: 0.2793, Forward time: 1.2620, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8228 [2021-07-07 18:28:45,723][root][INFO] - ===> Epoch[2109](10545/5): Loss 0.0153, Sem 0.0153 LR: 8.403e-02 Score 99.446 Data time: 0.2362, Forward time: 1.2620, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.5498 [2021-07-07 18:28:59,925][root][INFO] - ===> Epoch[2110](10550/5): Loss 0.0155, Sem 0.0155 LR: 8.403e-02 Score 99.442 Data time: 0.1980, Forward time: 1.2619, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8028 [2021-07-07 18:29:13,969][root][INFO] - ===> Epoch[2111](10555/5): Loss 0.0149, Sem 0.0149 LR: 8.402e-02 Score 99.467 Data time: 0.2258, Forward time: 1.2619, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7556 [2021-07-07 18:29:27,634][root][INFO] - ===> Epoch[2112](10560/5): Loss 0.0151, Sem 0.0151 LR: 8.401e-02 Score 99.453 Data time: 0.2430, Forward time: 1.2619, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6778 [2021-07-07 18:29:41,558][root][INFO] - ===> Epoch[2113](10565/5): Loss 0.0156, Sem 0.0156 LR: 8.400e-02 Score 99.436 Data time: 0.2727, Forward time: 1.2619, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7316 [2021-07-07 18:29:54,891][root][INFO] - ===> Epoch[2114](10570/5): Loss 0.0151, Sem 0.0151 LR: 8.400e-02 Score 99.460 Data time: 0.2309, Forward time: 1.2619, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.6155 [2021-07-07 18:30:09,425][root][INFO] - ===> Epoch[2115](10575/5): Loss 0.0155, Sem 0.0155 LR: 8.399e-02 Score 99.445 Data time: 0.2902, Forward time: 1.2620, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8624 [2021-07-07 18:30:23,537][root][INFO] - ===> Epoch[2116](10580/5): Loss 0.0149, Sem 0.0149 LR: 8.398e-02 Score 99.463 Data time: 0.2753, Forward time: 1.2620, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7700 [2021-07-07 18:30:37,070][root][INFO] - ===> Epoch[2117](10585/5): Loss 0.0146, Sem 0.0146 LR: 8.397e-02 Score 99.476 Data time: 0.1829, Forward time: 1.2620, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6605 [2021-07-07 18:30:51,725][root][INFO] - ===> Epoch[2118](10590/5): Loss 0.0153, Sem 0.0153 LR: 8.397e-02 Score 99.447 Data time: 0.2443, Forward time: 1.2620, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8755 [2021-07-07 18:31:05,441][root][INFO] - ===> Epoch[2119](10595/5): Loss 0.0157, Sem 0.0157 LR: 8.396e-02 Score 99.434 Data time: 0.2101, Forward time: 1.2620, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7102 [2021-07-07 18:31:18,680][root][INFO] - ===> Epoch[2120](10600/5): Loss 0.0149, Sem 0.0149 LR: 8.395e-02 Score 99.460 Data time: 0.1586, Forward time: 1.2620, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.6092 [2021-07-07 18:31:18,682][root][INFO] - ===> Start testing [2021-07-07 18:31:53,280][root][INFO] - 68/68: Data time: 0.0017, Iter time: 0.1308 Loss 0.328 (AVG: 0.527) Score 92.781 (AVG: 91.191) mIOU 70.739 mAP 84.011 mAcc 76.763 IOU: 64.702 0.020 84.067 77.852 94.687 92.189 28.396 81.634 98.646 71.333 80.727 86.833 58.515 mAP: 84.748 8.690 93.311 89.538 98.829 98.482 52.387 98.047 99.809 89.363 92.599 94.275 92.063 mAcc: 81.839 0.182 89.345 84.289 95.935 96.963 31.283 96.011 99.459 74.852 90.149 96.397 61.220 [2021-07-07 18:31:53,281][root][INFO] - Finished test. Elapsed time: 34.5977 [2021-07-07 18:31:54,358][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 18:32:07,721][root][INFO] - ===> Epoch[2121](10605/5): Loss 0.0157, Sem 0.0157 LR: 8.394e-02 Score 99.440 Data time: 0.2485, Forward time: 1.2619, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6262 [2021-07-07 18:32:22,465][root][INFO] - ===> Epoch[2122](10610/5): Loss 0.0155, Sem 0.0155 LR: 8.393e-02 Score 99.443 Data time: 0.2355, Forward time: 1.2619, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.9077 [2021-07-07 18:32:36,687][root][INFO] - ===> Epoch[2123](10615/5): Loss 0.0156, Sem 0.0156 LR: 8.393e-02 Score 99.437 Data time: 0.2227, Forward time: 1.2619, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8022 [2021-07-07 18:32:50,532][root][INFO] - ===> Epoch[2124](10620/5): Loss 0.0161, Sem 0.0161 LR: 8.392e-02 Score 99.412 Data time: 0.2270, Forward time: 1.2619, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7262 [2021-07-07 18:33:04,919][root][INFO] - ===> Epoch[2125](10625/5): Loss 0.0159, Sem 0.0159 LR: 8.391e-02 Score 99.427 Data time: 0.3125, Forward time: 1.2619, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8417 [2021-07-07 18:33:18,719][root][INFO] - ===> Epoch[2126](10630/5): Loss 0.0149, Sem 0.0149 LR: 8.390e-02 Score 99.463 Data time: 0.2691, Forward time: 1.2618, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7080 [2021-07-07 18:33:33,320][root][INFO] - ===> Epoch[2127](10635/5): Loss 0.0156, Sem 0.0156 LR: 8.390e-02 Score 99.434 Data time: 0.2177, Forward time: 1.2618, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8718 [2021-07-07 18:33:46,866][root][INFO] - ===> Epoch[2128](10640/5): Loss 0.0159, Sem 0.0159 LR: 8.389e-02 Score 99.430 Data time: 0.1759, Forward time: 1.2617, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6730 [2021-07-07 18:34:01,082][root][INFO] - ===> Epoch[2129](10645/5): Loss 0.0153, Sem 0.0153 LR: 8.388e-02 Score 99.441 Data time: 0.2172, Forward time: 1.2618, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8079 [2021-07-07 18:34:14,977][root][INFO] - ===> Epoch[2130](10650/5): Loss 0.0151, Sem 0.0151 LR: 8.387e-02 Score 99.452 Data time: 0.2003, Forward time: 1.2617, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7357 [2021-07-07 18:34:28,316][root][INFO] - ===> Epoch[2131](10655/5): Loss 0.0156, Sem 0.0156 LR: 8.387e-02 Score 99.435 Data time: 0.2330, Forward time: 1.2617, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6035 [2021-07-07 18:34:42,102][root][INFO] - ===> Epoch[2132](10660/5): Loss 0.0148, Sem 0.0148 LR: 8.386e-02 Score 99.465 Data time: 0.2190, Forward time: 1.2616, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6960 [2021-07-07 18:34:56,175][root][INFO] - ===> Epoch[2133](10665/5): Loss 0.0149, Sem 0.0149 LR: 8.385e-02 Score 99.460 Data time: 0.2243, Forward time: 1.2616, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7623 [2021-07-07 18:35:10,557][root][INFO] - ===> Epoch[2134](10670/5): Loss 0.0146, Sem 0.0146 LR: 8.384e-02 Score 99.466 Data time: 0.2440, Forward time: 1.2616, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8202 [2021-07-07 18:35:25,547][root][INFO] - ===> Epoch[2135](10675/5): Loss 0.0142, Sem 0.0142 LR: 8.384e-02 Score 99.483 Data time: 0.2153, Forward time: 1.2616, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.9529 [2021-07-07 18:35:39,320][root][INFO] - ===> Epoch[2136](10680/5): Loss 0.0152, Sem 0.0152 LR: 8.383e-02 Score 99.449 Data time: 0.2309, Forward time: 1.2616, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7121 [2021-07-07 18:35:54,074][root][INFO] - ===> Epoch[2137](10685/5): Loss 0.0144, Sem 0.0144 LR: 8.382e-02 Score 99.479 Data time: 0.2980, Forward time: 1.2616, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.9021 [2021-07-07 18:36:07,621][root][INFO] - ===> Epoch[2138](10690/5): Loss 0.0156, Sem 0.0156 LR: 8.381e-02 Score 99.436 Data time: 0.2463, Forward time: 1.2616, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6651 [2021-07-07 18:36:20,830][root][INFO] - ===> Epoch[2139](10695/5): Loss 0.0157, Sem 0.0157 LR: 8.380e-02 Score 99.425 Data time: 0.2145, Forward time: 1.2616, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.5914 [2021-07-07 18:36:34,430][root][INFO] - ===> Epoch[2140](10700/5): Loss 0.0155, Sem 0.0155 LR: 8.380e-02 Score 99.439 Data time: 0.2518, Forward time: 1.2617, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6758 [2021-07-07 18:36:47,882][root][INFO] - ===> Epoch[2141](10705/5): Loss 0.0149, Sem 0.0149 LR: 8.379e-02 Score 99.463 Data time: 0.2172, Forward time: 1.2617, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6419 [2021-07-07 18:37:01,729][root][INFO] - ===> Epoch[2142](10710/5): Loss 0.0152, Sem 0.0152 LR: 8.378e-02 Score 99.443 Data time: 0.1544, Forward time: 1.2616, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7295 [2021-07-07 18:37:15,293][root][INFO] - ===> Epoch[2143](10715/5): Loss 0.0173, Sem 0.0173 LR: 8.377e-02 Score 99.389 Data time: 0.2162, Forward time: 1.2616, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6664 [2021-07-07 18:37:29,887][root][INFO] - ===> Epoch[2144](10720/5): Loss 0.0160, Sem 0.0160 LR: 8.377e-02 Score 99.429 Data time: 0.2628, Forward time: 1.2617, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8792 [2021-07-07 18:37:44,030][root][INFO] - ===> Epoch[2145](10725/5): Loss 0.0166, Sem 0.0166 LR: 8.376e-02 Score 99.411 Data time: 0.2029, Forward time: 1.2617, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7798 [2021-07-07 18:37:58,264][root][INFO] - ===> Epoch[2146](10730/5): Loss 0.0155, Sem 0.0155 LR: 8.375e-02 Score 99.444 Data time: 0.2299, Forward time: 1.2617, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7995 [2021-07-07 18:38:12,904][root][INFO] - ===> Epoch[2147](10735/5): Loss 0.0157, Sem 0.0157 LR: 8.374e-02 Score 99.435 Data time: 0.2376, Forward time: 1.2617, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8810 [2021-07-07 18:38:27,182][root][INFO] - ===> Epoch[2148](10740/5): Loss 0.0159, Sem 0.0159 LR: 8.374e-02 Score 99.421 Data time: 0.2571, Forward time: 1.2617, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8088 [2021-07-07 18:38:40,938][root][INFO] - ===> Epoch[2149](10745/5): Loss 0.0160, Sem 0.0160 LR: 8.373e-02 Score 99.423 Data time: 0.2652, Forward time: 1.2617, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6951 [2021-07-07 18:38:55,087][root][INFO] - ===> Epoch[2150](10750/5): Loss 0.0161, Sem 0.0161 LR: 8.372e-02 Score 99.413 Data time: 0.2391, Forward time: 1.2617, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7876 [2021-07-07 18:39:09,794][root][INFO] - ===> Epoch[2151](10755/5): Loss 0.0150, Sem 0.0150 LR: 8.371e-02 Score 99.458 Data time: 0.2759, Forward time: 1.2618, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8941 [2021-07-07 18:39:23,920][root][INFO] - ===> Epoch[2152](10760/5): Loss 0.0146, Sem 0.0146 LR: 8.371e-02 Score 99.469 Data time: 0.3081, Forward time: 1.2618, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7804 [2021-07-07 18:39:38,585][root][INFO] - ===> Epoch[2153](10765/5): Loss 0.0149, Sem 0.0149 LR: 8.370e-02 Score 99.459 Data time: 0.2308, Forward time: 1.2619, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8881 [2021-07-07 18:39:53,394][root][INFO] - ===> Epoch[2154](10770/5): Loss 0.0150, Sem 0.0150 LR: 8.369e-02 Score 99.461 Data time: 0.2522, Forward time: 1.2619, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.9100 [2021-07-07 18:40:07,842][root][INFO] - ===> Epoch[2155](10775/5): Loss 0.0146, Sem 0.0146 LR: 8.368e-02 Score 99.474 Data time: 0.2977, Forward time: 1.2620, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8362 [2021-07-07 18:40:21,909][root][INFO] - ===> Epoch[2156](10780/5): Loss 0.0151, Sem 0.0151 LR: 8.367e-02 Score 99.462 Data time: 0.2078, Forward time: 1.2619, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7683 [2021-07-07 18:40:36,008][root][INFO] - ===> Epoch[2157](10785/5): Loss 0.0142, Sem 0.0142 LR: 8.367e-02 Score 99.485 Data time: 0.2436, Forward time: 1.2620, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7772 [2021-07-07 18:40:50,056][root][INFO] - ===> Epoch[2158](10790/5): Loss 0.0161, Sem 0.0161 LR: 8.366e-02 Score 99.421 Data time: 0.2388, Forward time: 1.2621, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7509 [2021-07-07 18:41:03,998][root][INFO] - ===> Epoch[2159](10795/5): Loss 0.0150, Sem 0.0150 LR: 8.365e-02 Score 99.455 Data time: 0.2587, Forward time: 1.2620, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7457 [2021-07-07 18:41:18,318][root][INFO] - ===> Epoch[2160](10800/5): Loss 0.0148, Sem 0.0148 LR: 8.364e-02 Score 99.465 Data time: 0.2156, Forward time: 1.2620, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8145 [2021-07-07 18:41:18,321][root][INFO] - ===> Start testing [2021-07-07 18:41:52,771][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1314 Loss 0.219 (AVG: 0.512) Score 94.673 (AVG: 91.497) mIOU 71.693 mAP 84.528 mAcc 77.623 IOU: 65.454 0.012 84.267 78.643 94.554 92.626 34.004 81.435 98.556 75.720 79.813 87.333 59.593 mAP: 85.349 7.571 93.713 90.144 98.814 98.875 57.242 97.436 99.769 90.239 92.916 94.864 91.931 mAcc: 79.846 0.091 88.733 86.599 96.009 97.074 35.980 96.424 99.498 79.410 89.238 96.646 63.555 [2021-07-07 18:41:52,771][root][INFO] - Finished test. Elapsed time: 34.4489 [2021-07-07 18:41:53,855][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 18:42:06,519][root][INFO] - ===> Epoch[2161](10805/5): Loss 0.0156, Sem 0.0156 LR: 8.364e-02 Score 99.438 Data time: 0.2301, Forward time: 1.2619, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.4955 [2021-07-07 18:42:20,739][root][INFO] - ===> Epoch[2162](10810/5): Loss 0.0154, Sem 0.0154 LR: 8.363e-02 Score 99.443 Data time: 0.2207, Forward time: 1.2619, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8044 [2021-07-07 18:42:36,072][root][INFO] - ===> Epoch[2163](10815/5): Loss 0.0145, Sem 0.0145 LR: 8.362e-02 Score 99.477 Data time: 0.1980, Forward time: 1.2620, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 3.0195 [2021-07-07 18:42:50,626][root][INFO] - ===> Epoch[2164](10820/5): Loss 0.0147, Sem 0.0147 LR: 8.361e-02 Score 99.472 Data time: 0.2592, Forward time: 1.2620, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8582 [2021-07-07 18:43:04,780][root][INFO] - ===> Epoch[2165](10825/5): Loss 0.0147, Sem 0.0147 LR: 8.361e-02 Score 99.461 Data time: 0.1871, Forward time: 1.2620, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7820 [2021-07-07 18:43:18,327][root][INFO] - ===> Epoch[2166](10830/5): Loss 0.0141, Sem 0.0141 LR: 8.360e-02 Score 99.489 Data time: 0.1946, Forward time: 1.2620, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6456 [2021-07-07 18:43:32,650][root][INFO] - ===> Epoch[2167](10835/5): Loss 0.0147, Sem 0.0147 LR: 8.359e-02 Score 99.467 Data time: 0.1728, Forward time: 1.2620, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8324 [2021-07-07 18:43:45,919][root][INFO] - ===> Epoch[2168](10840/5): Loss 0.0139, Sem 0.0139 LR: 8.358e-02 Score 99.507 Data time: 0.1682, Forward time: 1.2620, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6173 [2021-07-07 18:43:59,609][root][INFO] - ===> Epoch[2169](10845/5): Loss 0.0152, Sem 0.0152 LR: 8.357e-02 Score 99.452 Data time: 0.2235, Forward time: 1.2620, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6929 [2021-07-07 18:44:14,553][root][INFO] - ===> Epoch[2170](10850/5): Loss 0.0145, Sem 0.0145 LR: 8.357e-02 Score 99.474 Data time: 0.1991, Forward time: 1.2619, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.9351 [2021-07-07 18:44:28,079][root][INFO] - ===> Epoch[2171](10855/5): Loss 0.0153, Sem 0.0153 LR: 8.356e-02 Score 99.446 Data time: 0.2295, Forward time: 1.2620, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6624 [2021-07-07 18:44:42,049][root][INFO] - ===> Epoch[2172](10860/5): Loss 0.0148, Sem 0.0148 LR: 8.355e-02 Score 99.464 Data time: 0.1586, Forward time: 1.2620, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7626 [2021-07-07 18:44:56,500][root][INFO] - ===> Epoch[2173](10865/5): Loss 0.0155, Sem 0.0155 LR: 8.354e-02 Score 99.441 Data time: 0.2163, Forward time: 1.2620, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8446 [2021-07-07 18:45:10,732][root][INFO] - ===> Epoch[2174](10870/5): Loss 0.0147, Sem 0.0147 LR: 8.354e-02 Score 99.476 Data time: 0.2199, Forward time: 1.2619, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8080 [2021-07-07 18:45:24,971][root][INFO] - ===> Epoch[2175](10875/5): Loss 0.0150, Sem 0.0150 LR: 8.353e-02 Score 99.458 Data time: 0.1899, Forward time: 1.2619, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7990 [2021-07-07 18:45:38,996][root][INFO] - ===> Epoch[2176](10880/5): Loss 0.0152, Sem 0.0152 LR: 8.352e-02 Score 99.450 Data time: 0.1940, Forward time: 1.2619, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7609 [2021-07-07 18:45:52,826][root][INFO] - ===> Epoch[2177](10885/5): Loss 0.0154, Sem 0.0154 LR: 8.351e-02 Score 99.444 Data time: 0.2411, Forward time: 1.2619, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7112 [2021-07-07 18:46:08,565][root][INFO] - ===> Epoch[2178](10890/5): Loss 0.0161, Sem 0.0161 LR: 8.351e-02 Score 99.428 Data time: 0.2192, Forward time: 1.2620, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 3.0883 [2021-07-07 18:46:22,908][root][INFO] - ===> Epoch[2179](10895/5): Loss 0.0155, Sem 0.0155 LR: 8.350e-02 Score 99.448 Data time: 0.1977, Forward time: 1.2620, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8235 [2021-07-07 18:46:36,899][root][INFO] - ===> Epoch[2180](10900/5): Loss 0.0164, Sem 0.0164 LR: 8.349e-02 Score 99.415 Data time: 0.1923, Forward time: 1.2620, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7580 [2021-07-07 18:46:50,763][root][INFO] - ===> Epoch[2181](10905/5): Loss 0.0158, Sem 0.0158 LR: 8.348e-02 Score 99.429 Data time: 0.1608, Forward time: 1.2619, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7209 [2021-07-07 18:47:04,651][root][INFO] - ===> Epoch[2182](10910/5): Loss 0.0162, Sem 0.0162 LR: 8.348e-02 Score 99.413 Data time: 0.2352, Forward time: 1.2619, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7335 [2021-07-07 18:47:18,917][root][INFO] - ===> Epoch[2183](10915/5): Loss 0.0155, Sem 0.0155 LR: 8.347e-02 Score 99.438 Data time: 0.2304, Forward time: 1.2619, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7939 [2021-07-07 18:47:33,159][root][INFO] - ===> Epoch[2184](10920/5): Loss 0.0165, Sem 0.0165 LR: 8.346e-02 Score 99.410 Data time: 0.2177, Forward time: 1.2619, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7965 [2021-07-07 18:47:47,607][root][INFO] - ===> Epoch[2185](10925/5): Loss 0.0161, Sem 0.0161 LR: 8.345e-02 Score 99.422 Data time: 0.2408, Forward time: 1.2619, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8471 [2021-07-07 18:48:01,994][root][INFO] - ===> Epoch[2186](10930/5): Loss 0.0157, Sem 0.0157 LR: 8.344e-02 Score 99.437 Data time: 0.2111, Forward time: 1.2618, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8298 [2021-07-07 18:48:16,287][root][INFO] - ===> Epoch[2187](10935/5): Loss 0.0159, Sem 0.0159 LR: 8.344e-02 Score 99.435 Data time: 0.1920, Forward time: 1.2618, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8175 [2021-07-07 18:48:29,867][root][INFO] - ===> Epoch[2188](10940/5): Loss 0.0161, Sem 0.0161 LR: 8.343e-02 Score 99.423 Data time: 0.1528, Forward time: 1.2618, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.6776 [2021-07-07 18:48:44,522][root][INFO] - ===> Epoch[2189](10945/5): Loss 0.0159, Sem 0.0159 LR: 8.342e-02 Score 99.424 Data time: 0.2010, Forward time: 1.2617, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8870 [2021-07-07 18:48:58,468][root][INFO] - ===> Epoch[2190](10950/5): Loss 0.0150, Sem 0.0150 LR: 8.341e-02 Score 99.459 Data time: 0.2438, Forward time: 1.2617, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7444 [2021-07-07 18:49:12,052][root][INFO] - ===> Epoch[2191](10955/5): Loss 0.0148, Sem 0.0148 LR: 8.341e-02 Score 99.463 Data time: 0.2483, Forward time: 1.2616, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6736 [2021-07-07 18:49:26,745][root][INFO] - ===> Epoch[2192](10960/5): Loss 0.0158, Sem 0.0158 LR: 8.340e-02 Score 99.447 Data time: 0.2195, Forward time: 1.2616, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8950 [2021-07-07 18:49:40,967][root][INFO] - ===> Epoch[2193](10965/5): Loss 0.0160, Sem 0.0160 LR: 8.339e-02 Score 99.420 Data time: 0.2884, Forward time: 1.2616, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8009 [2021-07-07 18:49:55,767][root][INFO] - ===> Epoch[2194](10970/5): Loss 0.0149, Sem 0.0149 LR: 8.338e-02 Score 99.457 Data time: 0.2994, Forward time: 1.2617, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8994 [2021-07-07 18:50:10,207][root][INFO] - ===> Epoch[2195](10975/5): Loss 0.0151, Sem 0.0151 LR: 8.338e-02 Score 99.450 Data time: 0.2042, Forward time: 1.2617, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8498 [2021-07-07 18:50:24,206][root][INFO] - ===> Epoch[2196](10980/5): Loss 0.0142, Sem 0.0142 LR: 8.337e-02 Score 99.484 Data time: 0.2180, Forward time: 1.2617, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7641 [2021-07-07 18:50:38,620][root][INFO] - ===> Epoch[2197](10985/5): Loss 0.0149, Sem 0.0149 LR: 8.336e-02 Score 99.461 Data time: 0.2004, Forward time: 1.2617, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8414 [2021-07-07 18:50:52,488][root][INFO] - ===> Epoch[2198](10990/5): Loss 0.0151, Sem 0.0151 LR: 8.335e-02 Score 99.459 Data time: 0.2149, Forward time: 1.2617, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7345 [2021-07-07 18:51:07,139][root][INFO] - ===> Epoch[2199](10995/5): Loss 0.0145, Sem 0.0145 LR: 8.335e-02 Score 99.471 Data time: 0.2409, Forward time: 1.2617, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8895 [2021-07-07 18:51:21,960][root][INFO] - ===> Epoch[2200](11000/5): Loss 0.0150, Sem 0.0150 LR: 8.334e-02 Score 99.458 Data time: 0.2475, Forward time: 1.2617, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9233 [2021-07-07 18:51:21,961][root][INFO] - ===> Start testing [2021-07-07 18:51:57,444][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.2229 Loss 0.247 (AVG: 0.526) Score 94.346 (AVG: 91.079) mIOU 70.116 mAP 83.218 mAcc 75.699 IOU: 65.006 0.103 83.236 77.344 94.520 92.646 20.997 85.521 98.617 68.321 79.536 86.194 59.465 mAP: 85.510 9.033 91.528 89.432 98.583 98.942 50.380 98.387 99.820 81.776 91.888 94.245 92.312 mAcc: 81.953 0.912 88.325 84.390 96.260 96.953 21.744 95.743 99.392 70.916 87.404 96.530 63.568 [2021-07-07 18:51:57,444][root][INFO] - Finished test. Elapsed time: 35.4825 [2021-07-07 18:51:58,599][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 18:52:11,972][root][INFO] - ===> Epoch[2201](11005/5): Loss 0.0149, Sem 0.0149 LR: 8.333e-02 Score 99.462 Data time: 0.1750, Forward time: 1.2616, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6363 [2021-07-07 18:52:25,221][root][INFO] - ===> Epoch[2202](11010/5): Loss 0.0144, Sem 0.0144 LR: 8.332e-02 Score 99.475 Data time: 0.2025, Forward time: 1.2615, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6032 [2021-07-07 18:52:39,161][root][INFO] - ===> Epoch[2203](11015/5): Loss 0.0148, Sem 0.0148 LR: 8.331e-02 Score 99.474 Data time: 0.2244, Forward time: 1.2615, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7368 [2021-07-07 18:52:52,914][root][INFO] - ===> Epoch[2204](11020/5): Loss 0.0147, Sem 0.0147 LR: 8.331e-02 Score 99.473 Data time: 0.1690, Forward time: 1.2615, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7012 [2021-07-07 18:53:07,900][root][INFO] - ===> Epoch[2205](11025/5): Loss 0.0152, Sem 0.0152 LR: 8.330e-02 Score 99.457 Data time: 0.2147, Forward time: 1.2616, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9549 [2021-07-07 18:53:23,178][root][INFO] - ===> Epoch[2206](11030/5): Loss 0.0150, Sem 0.0150 LR: 8.329e-02 Score 99.459 Data time: 0.1894, Forward time: 1.2616, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 3.0040 [2021-07-07 18:53:37,302][root][INFO] - ===> Epoch[2207](11035/5): Loss 0.0147, Sem 0.0147 LR: 8.328e-02 Score 99.468 Data time: 0.2699, Forward time: 1.2617, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7806 [2021-07-07 18:53:51,655][root][INFO] - ===> Epoch[2208](11040/5): Loss 0.0148, Sem 0.0148 LR: 8.328e-02 Score 99.463 Data time: 0.1674, Forward time: 1.2616, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8224 [2021-07-07 18:54:05,007][root][INFO] - ===> Epoch[2209](11045/5): Loss 0.0144, Sem 0.0144 LR: 8.327e-02 Score 99.482 Data time: 0.2383, Forward time: 1.2616, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6119 [2021-07-07 18:54:20,196][root][INFO] - ===> Epoch[2210](11050/5): Loss 0.0142, Sem 0.0142 LR: 8.326e-02 Score 99.483 Data time: 0.2518, Forward time: 1.2616, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.9958 [2021-07-07 18:54:33,991][root][INFO] - ===> Epoch[2211](11055/5): Loss 0.0145, Sem 0.0145 LR: 8.325e-02 Score 99.472 Data time: 0.1875, Forward time: 1.2616, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6943 [2021-07-07 18:54:47,666][root][INFO] - ===> Epoch[2212](11060/5): Loss 0.0138, Sem 0.0138 LR: 8.325e-02 Score 99.499 Data time: 0.2044, Forward time: 1.2615, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6829 [2021-07-07 18:55:02,717][root][INFO] - ===> Epoch[2213](11065/5): Loss 0.0144, Sem 0.0144 LR: 8.324e-02 Score 99.481 Data time: 0.2220, Forward time: 1.2615, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.9540 [2021-07-07 18:55:16,551][root][INFO] - ===> Epoch[2214](11070/5): Loss 0.0149, Sem 0.0149 LR: 8.323e-02 Score 99.462 Data time: 0.1929, Forward time: 1.2615, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7158 [2021-07-07 18:55:29,617][root][INFO] - ===> Epoch[2215](11075/5): Loss 0.0146, Sem 0.0146 LR: 8.322e-02 Score 99.468 Data time: 0.2202, Forward time: 1.2615, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.5626 [2021-07-07 18:55:43,253][root][INFO] - ===> Epoch[2216](11080/5): Loss 0.0146, Sem 0.0146 LR: 8.322e-02 Score 99.472 Data time: 0.1472, Forward time: 1.2615, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6833 [2021-07-07 18:55:57,529][root][INFO] - ===> Epoch[2217](11085/5): Loss 0.0142, Sem 0.0142 LR: 8.321e-02 Score 99.483 Data time: 0.2176, Forward time: 1.2615, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8084 [2021-07-07 18:56:11,981][root][INFO] - ===> Epoch[2218](11090/5): Loss 0.0145, Sem 0.0145 LR: 8.320e-02 Score 99.482 Data time: 0.2458, Forward time: 1.2615, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8263 [2021-07-07 18:56:25,957][root][INFO] - ===> Epoch[2219](11095/5): Loss 0.0146, Sem 0.0146 LR: 8.319e-02 Score 99.479 Data time: 0.2115, Forward time: 1.2615, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7371 [2021-07-07 18:56:40,026][root][INFO] - ===> Epoch[2220](11100/5): Loss 0.0149, Sem 0.0149 LR: 8.318e-02 Score 99.464 Data time: 0.2351, Forward time: 1.2615, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7693 [2021-07-07 18:56:53,941][root][INFO] - ===> Epoch[2221](11105/5): Loss 0.0140, Sem 0.0140 LR: 8.318e-02 Score 99.491 Data time: 0.2440, Forward time: 1.2615, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7261 [2021-07-07 18:57:07,684][root][INFO] - ===> Epoch[2222](11110/5): Loss 0.0147, Sem 0.0147 LR: 8.317e-02 Score 99.469 Data time: 0.2612, Forward time: 1.2615, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6954 [2021-07-07 18:57:22,302][root][INFO] - ===> Epoch[2223](11115/5): Loss 0.0141, Sem 0.0141 LR: 8.316e-02 Score 99.491 Data time: 0.2037, Forward time: 1.2615, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8695 [2021-07-07 18:57:36,026][root][INFO] - ===> Epoch[2224](11120/5): Loss 0.0147, Sem 0.0147 LR: 8.315e-02 Score 99.471 Data time: 0.2073, Forward time: 1.2615, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7018 [2021-07-07 18:57:49,758][root][INFO] - ===> Epoch[2225](11125/5): Loss 0.0144, Sem 0.0144 LR: 8.315e-02 Score 99.480 Data time: 0.2326, Forward time: 1.2614, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6923 [2021-07-07 18:58:03,706][root][INFO] - ===> Epoch[2226](11130/5): Loss 0.0146, Sem 0.0146 LR: 8.314e-02 Score 99.474 Data time: 0.2343, Forward time: 1.2614, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7431 [2021-07-07 18:58:16,914][root][INFO] - ===> Epoch[2227](11135/5): Loss 0.0156, Sem 0.0156 LR: 8.313e-02 Score 99.439 Data time: 0.1617, Forward time: 1.2614, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.5958 [2021-07-07 18:58:31,105][root][INFO] - ===> Epoch[2228](11140/5): Loss 0.0148, Sem 0.0148 LR: 8.312e-02 Score 99.468 Data time: 0.2334, Forward time: 1.2614, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7928 [2021-07-07 18:58:45,141][root][INFO] - ===> Epoch[2229](11145/5): Loss 0.0154, Sem 0.0154 LR: 8.312e-02 Score 99.447 Data time: 0.2402, Forward time: 1.2614, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7649 [2021-07-07 18:58:59,063][root][INFO] - ===> Epoch[2230](11150/5): Loss 0.0149, Sem 0.0149 LR: 8.311e-02 Score 99.463 Data time: 0.2486, Forward time: 1.2613, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7268 [2021-07-07 18:59:13,115][root][INFO] - ===> Epoch[2231](11155/5): Loss 0.0152, Sem 0.0152 LR: 8.310e-02 Score 99.450 Data time: 0.2417, Forward time: 1.2613, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7663 [2021-07-07 18:59:27,123][root][INFO] - ===> Epoch[2232](11160/5): Loss 0.0157, Sem 0.0157 LR: 8.309e-02 Score 99.432 Data time: 0.2408, Forward time: 1.2613, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7555 [2021-07-07 18:59:42,462][root][INFO] - ===> Epoch[2233](11165/5): Loss 0.0158, Sem 0.0158 LR: 8.309e-02 Score 99.432 Data time: 0.2085, Forward time: 1.2613, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 3.0244 [2021-07-07 18:59:56,696][root][INFO] - ===> Epoch[2234](11170/5): Loss 0.0148, Sem 0.0148 LR: 8.308e-02 Score 99.463 Data time: 0.2014, Forward time: 1.2612, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7901 [2021-07-07 19:00:10,276][root][INFO] - ===> Epoch[2235](11175/5): Loss 0.0156, Sem 0.0156 LR: 8.307e-02 Score 99.437 Data time: 0.1921, Forward time: 1.2613, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.6692 [2021-07-07 19:00:24,680][root][INFO] - ===> Epoch[2236](11180/5): Loss 0.0156, Sem 0.0156 LR: 8.306e-02 Score 99.450 Data time: 0.2047, Forward time: 1.2613, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.8410 [2021-07-07 19:00:39,190][root][INFO] - ===> Epoch[2237](11185/5): Loss 0.0152, Sem 0.0152 LR: 8.305e-02 Score 99.448 Data time: 0.2244, Forward time: 1.2613, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.8519 [2021-07-07 19:00:54,069][root][INFO] - ===> Epoch[2238](11190/5): Loss 0.0158, Sem 0.0158 LR: 8.305e-02 Score 99.440 Data time: 0.2370, Forward time: 1.2613, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.9305 [2021-07-07 19:01:07,495][root][INFO] - ===> Epoch[2239](11195/5): Loss 0.0172, Sem 0.0172 LR: 8.304e-02 Score 99.389 Data time: 0.2247, Forward time: 1.2613, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6316 [2021-07-07 19:01:22,156][root][INFO] - ===> Epoch[2240](11200/5): Loss 0.0154, Sem 0.0154 LR: 8.303e-02 Score 99.443 Data time: 0.2225, Forward time: 1.2613, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8787 [2021-07-07 19:01:22,157][root][INFO] - ===> Start testing [2021-07-07 19:01:57,828][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1266 Loss 0.377 (AVG: 0.524) Score 91.541 (AVG: 91.002) mIOU 71.030 mAP 83.982 mAcc 77.588 IOU: 63.584 0.073 77.509 76.624 94.785 91.513 32.343 85.534 98.601 76.979 78.354 86.537 60.952 mAP: 84.075 11.446 94.339 89.594 98.824 98.806 55.467 98.320 99.835 82.683 91.744 94.082 92.558 mAcc: 78.278 1.094 92.461 85.387 96.585 98.178 34.489 97.042 99.350 79.045 85.573 95.652 65.506 [2021-07-07 19:01:57,828][root][INFO] - Finished test. Elapsed time: 35.6697 [2021-07-07 19:01:58,892][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 19:02:11,285][root][INFO] - ===> Epoch[2241](11205/5): Loss 0.0157, Sem 0.0157 LR: 8.302e-02 Score 99.437 Data time: 0.1617, Forward time: 1.2612, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.4330 [2021-07-07 19:02:24,961][root][INFO] - ===> Epoch[2242](11210/5): Loss 0.0154, Sem 0.0154 LR: 8.302e-02 Score 99.447 Data time: 0.1905, Forward time: 1.2612, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6944 [2021-07-07 19:02:39,267][root][INFO] - ===> Epoch[2243](11215/5): Loss 0.0155, Sem 0.0155 LR: 8.301e-02 Score 99.440 Data time: 0.2297, Forward time: 1.2612, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8106 [2021-07-07 19:02:53,055][root][INFO] - ===> Epoch[2244](11220/5): Loss 0.0150, Sem 0.0150 LR: 8.300e-02 Score 99.466 Data time: 0.1636, Forward time: 1.2611, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7131 [2021-07-07 19:03:06,052][root][INFO] - ===> Epoch[2245](11225/5): Loss 0.0162, Sem 0.0162 LR: 8.299e-02 Score 99.416 Data time: 0.1430, Forward time: 1.2611, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.5500 [2021-07-07 19:03:19,537][root][INFO] - ===> Epoch[2246](11230/5): Loss 0.0154, Sem 0.0154 LR: 8.299e-02 Score 99.440 Data time: 0.1875, Forward time: 1.2611, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.6553 [2021-07-07 19:03:34,746][root][INFO] - ===> Epoch[2247](11235/5): Loss 0.0156, Sem 0.0156 LR: 8.298e-02 Score 99.439 Data time: 0.2883, Forward time: 1.2611, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.9881 [2021-07-07 19:03:48,851][root][INFO] - ===> Epoch[2248](11240/5): Loss 0.0157, Sem 0.0157 LR: 8.297e-02 Score 99.444 Data time: 0.1792, Forward time: 1.2611, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7656 [2021-07-07 19:04:02,805][root][INFO] - ===> Epoch[2249](11245/5): Loss 0.0162, Sem 0.0162 LR: 8.296e-02 Score 99.421 Data time: 0.2842, Forward time: 1.2611, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7491 [2021-07-07 19:04:17,770][root][INFO] - ===> Epoch[2250](11250/5): Loss 0.0157, Sem 0.0157 LR: 8.295e-02 Score 99.430 Data time: 0.2183, Forward time: 1.2612, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.9523 [2021-07-07 19:04:31,398][root][INFO] - ===> Epoch[2251](11255/5): Loss 0.0163, Sem 0.0163 LR: 8.295e-02 Score 99.418 Data time: 0.2488, Forward time: 1.2611, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.6826 [2021-07-07 19:04:45,753][root][INFO] - ===> Epoch[2252](11260/5): Loss 0.0156, Sem 0.0156 LR: 8.294e-02 Score 99.442 Data time: 0.2187, Forward time: 1.2611, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.8175 [2021-07-07 19:04:59,236][root][INFO] - ===> Epoch[2253](11265/5): Loss 0.0170, Sem 0.0170 LR: 8.293e-02 Score 99.390 Data time: 0.2440, Forward time: 1.2611, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.6460 [2021-07-07 19:05:12,627][root][INFO] - ===> Epoch[2254](11270/5): Loss 0.0155, Sem 0.0155 LR: 8.292e-02 Score 99.445 Data time: 0.2530, Forward time: 1.2612, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6336 [2021-07-07 19:05:26,901][root][INFO] - ===> Epoch[2255](11275/5): Loss 0.0147, Sem 0.0147 LR: 8.292e-02 Score 99.470 Data time: 0.2618, Forward time: 1.2612, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7991 [2021-07-07 19:05:40,797][root][INFO] - ===> Epoch[2256](11280/5): Loss 0.0155, Sem 0.0155 LR: 8.291e-02 Score 99.438 Data time: 0.2532, Forward time: 1.2612, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7288 [2021-07-07 19:05:54,872][root][INFO] - ===> Epoch[2257](11285/5): Loss 0.0159, Sem 0.0159 LR: 8.290e-02 Score 99.445 Data time: 0.2426, Forward time: 1.2612, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7720 [2021-07-07 19:06:09,184][root][INFO] - ===> Epoch[2258](11290/5): Loss 0.0149, Sem 0.0149 LR: 8.289e-02 Score 99.465 Data time: 0.2280, Forward time: 1.2613, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.8173 [2021-07-07 19:06:23,138][root][INFO] - ===> Epoch[2259](11295/5): Loss 0.0153, Sem 0.0153 LR: 8.289e-02 Score 99.453 Data time: 0.1825, Forward time: 1.2613, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7448 [2021-07-07 19:06:36,979][root][INFO] - ===> Epoch[2260](11300/5): Loss 0.0150, Sem 0.0150 LR: 8.288e-02 Score 99.458 Data time: 0.2269, Forward time: 1.2612, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7236 [2021-07-07 19:06:50,475][root][INFO] - ===> Epoch[2261](11305/5): Loss 0.0147, Sem 0.0147 LR: 8.287e-02 Score 99.469 Data time: 0.2077, Forward time: 1.2612, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.6638 [2021-07-07 19:07:04,559][root][INFO] - ===> Epoch[2262](11310/5): Loss 0.0157, Sem 0.0157 LR: 8.286e-02 Score 99.434 Data time: 0.1870, Forward time: 1.2613, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7841 [2021-07-07 19:07:18,705][root][INFO] - ===> Epoch[2263](11315/5): Loss 0.0154, Sem 0.0154 LR: 8.286e-02 Score 99.444 Data time: 0.2715, Forward time: 1.2613, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7852 [2021-07-07 19:07:32,896][root][INFO] - ===> Epoch[2264](11320/5): Loss 0.0152, Sem 0.0152 LR: 8.285e-02 Score 99.452 Data time: 0.2354, Forward time: 1.2613, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7884 [2021-07-07 19:07:46,209][root][INFO] - ===> Epoch[2265](11325/5): Loss 0.0153, Sem 0.0153 LR: 8.284e-02 Score 99.448 Data time: 0.2530, Forward time: 1.2613, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.6174 [2021-07-07 19:08:01,880][root][INFO] - ===> Epoch[2266](11330/5): Loss 0.0150, Sem 0.0150 LR: 8.283e-02 Score 99.463 Data time: 0.2497, Forward time: 1.2614, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 3.0849 [2021-07-07 19:08:15,945][root][INFO] - ===> Epoch[2267](11335/5): Loss 0.0157, Sem 0.0157 LR: 8.282e-02 Score 99.446 Data time: 0.2120, Forward time: 1.2615, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7662 [2021-07-07 19:08:30,362][root][INFO] - ===> Epoch[2268](11340/5): Loss 0.0154, Sem 0.0154 LR: 8.282e-02 Score 99.446 Data time: 0.2809, Forward time: 1.2615, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8242 [2021-07-07 19:08:44,419][root][INFO] - ===> Epoch[2269](11345/5): Loss 0.0154, Sem 0.0154 LR: 8.281e-02 Score 99.444 Data time: 0.2197, Forward time: 1.2615, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7614 [2021-07-07 19:08:58,091][root][INFO] - ===> Epoch[2270](11350/5): Loss 0.0162, Sem 0.0162 LR: 8.280e-02 Score 99.425 Data time: 0.1520, Forward time: 1.2615, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.6905 [2021-07-07 19:09:11,676][root][INFO] - ===> Epoch[2271](11355/5): Loss 0.0175, Sem 0.0175 LR: 8.279e-02 Score 99.378 Data time: 0.2151, Forward time: 1.2615, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6712 [2021-07-07 19:09:25,453][root][INFO] - ===> Epoch[2272](11360/5): Loss 0.0170, Sem 0.0170 LR: 8.279e-02 Score 99.384 Data time: 0.1731, Forward time: 1.2614, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7137 [2021-07-07 19:09:39,295][root][INFO] - ===> Epoch[2273](11365/5): Loss 0.0178, Sem 0.0178 LR: 8.278e-02 Score 99.372 Data time: 0.2316, Forward time: 1.2614, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7159 [2021-07-07 19:09:53,707][root][INFO] - ===> Epoch[2274](11370/5): Loss 0.0192, Sem 0.0192 LR: 8.277e-02 Score 99.354 Data time: 0.1786, Forward time: 1.2614, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.8430 [2021-07-07 19:10:07,800][root][INFO] - ===> Epoch[2275](11375/5): Loss 0.0177, Sem 0.0177 LR: 8.276e-02 Score 99.367 Data time: 0.2139, Forward time: 1.2614, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7756 [2021-07-07 19:10:22,388][root][INFO] - ===> Epoch[2276](11380/5): Loss 0.0179, Sem 0.0179 LR: 8.276e-02 Score 99.364 Data time: 0.2205, Forward time: 1.2614, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.8654 [2021-07-07 19:10:36,511][root][INFO] - ===> Epoch[2277](11385/5): Loss 0.0188, Sem 0.0188 LR: 8.275e-02 Score 99.350 Data time: 0.2371, Forward time: 1.2614, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7826 [2021-07-07 19:10:50,636][root][INFO] - ===> Epoch[2278](11390/5): Loss 0.0184, Sem 0.0184 LR: 8.274e-02 Score 99.351 Data time: 0.2331, Forward time: 1.2615, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7837 [2021-07-07 19:11:04,500][root][INFO] - ===> Epoch[2279](11395/5): Loss 0.0187, Sem 0.0187 LR: 8.273e-02 Score 99.358 Data time: 0.1863, Forward time: 1.2615, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7293 [2021-07-07 19:11:18,530][root][INFO] - ===> Epoch[2280](11400/5): Loss 0.0184, Sem 0.0184 LR: 8.273e-02 Score 99.355 Data time: 0.2445, Forward time: 1.2615, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7627 [2021-07-07 19:11:18,533][root][INFO] - ===> Start testing [2021-07-07 19:11:53,515][root][INFO] - 68/68: Data time: 0.0017, Iter time: 0.1267 Loss 0.450 (AVG: 0.547) Score 88.300 (AVG: 90.482) mIOU 69.704 mAP 82.788 mAcc 76.845 IOU: 62.273 0.000 77.352 75.356 93.691 91.988 31.272 79.044 98.401 76.884 78.002 86.596 55.288 mAP: 82.977 4.636 88.185 88.108 98.389 98.643 58.939 97.350 99.807 84.044 91.410 93.971 89.789 mAcc: 76.525 0.000 83.104 83.647 95.231 96.363 34.066 96.023 99.302 81.146 90.519 95.787 67.264 [2021-07-07 19:11:53,516][root][INFO] - Finished test. Elapsed time: 34.9816 [2021-07-07 19:11:54,638][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 19:12:08,169][root][INFO] - ===> Epoch[2281](11405/5): Loss 0.0192, Sem 0.0192 LR: 8.272e-02 Score 99.314 Data time: 0.2047, Forward time: 1.2615, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.6691 [2021-07-07 19:12:22,125][root][INFO] - ===> Epoch[2282](11410/5): Loss 0.0182, Sem 0.0182 LR: 8.271e-02 Score 99.353 Data time: 0.2676, Forward time: 1.2615, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7343 [2021-07-07 19:12:36,858][root][INFO] - ===> Epoch[2283](11415/5): Loss 0.0180, Sem 0.0180 LR: 8.270e-02 Score 99.351 Data time: 0.2859, Forward time: 1.2615, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.8932 [2021-07-07 19:12:51,392][root][INFO] - ===> Epoch[2284](11420/5): Loss 0.0185, Sem 0.0185 LR: 8.269e-02 Score 99.345 Data time: 0.1845, Forward time: 1.2616, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.8640 [2021-07-07 19:13:05,482][root][INFO] - ===> Epoch[2285](11425/5): Loss 0.0216, Sem 0.0216 LR: 8.269e-02 Score 99.264 Data time: 0.2614, Forward time: 1.2616, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7796 [2021-07-07 19:13:19,754][root][INFO] - ===> Epoch[2286](11430/5): Loss 0.0229, Sem 0.0229 LR: 8.268e-02 Score 99.206 Data time: 0.2460, Forward time: 1.2616, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.8154 [2021-07-07 19:13:33,520][root][INFO] - ===> Epoch[2287](11435/5): Loss 0.0232, Sem 0.0232 LR: 8.267e-02 Score 99.180 Data time: 0.1972, Forward time: 1.2616, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7186 [2021-07-07 19:13:47,502][root][INFO] - ===> Epoch[2288](11440/5): Loss 0.0229, Sem 0.0229 LR: 8.266e-02 Score 99.192 Data time: 0.2391, Forward time: 1.2616, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7553 [2021-07-07 19:14:01,530][root][INFO] - ===> Epoch[2289](11445/5): Loss 0.0248, Sem 0.0248 LR: 8.266e-02 Score 99.145 Data time: 0.2134, Forward time: 1.2616, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7599 [2021-07-07 19:14:15,779][root][INFO] - ===> Epoch[2290](11450/5): Loss 0.0248, Sem 0.0248 LR: 8.265e-02 Score 99.150 Data time: 0.2519, Forward time: 1.2617, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8042 [2021-07-07 19:14:30,621][root][INFO] - ===> Epoch[2291](11455/5): Loss 0.0255, Sem 0.0255 LR: 8.264e-02 Score 99.111 Data time: 0.1760, Forward time: 1.2617, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.9266 [2021-07-07 19:14:45,249][root][INFO] - ===> Epoch[2292](11460/5): Loss 0.0252, Sem 0.0252 LR: 8.263e-02 Score 99.122 Data time: 0.3360, Forward time: 1.2617, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.8677 [2021-07-07 19:14:59,977][root][INFO] - ===> Epoch[2293](11465/5): Loss 0.0229, Sem 0.0229 LR: 8.263e-02 Score 99.199 Data time: 0.1811, Forward time: 1.2617, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.8988 [2021-07-07 19:15:14,437][root][INFO] - ===> Epoch[2294](11470/5): Loss 0.0228, Sem 0.0228 LR: 8.262e-02 Score 99.195 Data time: 0.3299, Forward time: 1.2617, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.8388 [2021-07-07 19:15:28,540][root][INFO] - ===> Epoch[2295](11475/5): Loss 0.0214, Sem 0.0214 LR: 8.261e-02 Score 99.245 Data time: 0.2999, Forward time: 1.2617, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7672 [2021-07-07 19:15:42,552][root][INFO] - ===> Epoch[2296](11480/5): Loss 0.0232, Sem 0.0232 LR: 8.260e-02 Score 99.205 Data time: 0.2012, Forward time: 1.2618, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7529 [2021-07-07 19:15:56,528][root][INFO] - ===> Epoch[2297](11485/5): Loss 0.0265, Sem 0.0265 LR: 8.260e-02 Score 99.123 Data time: 0.2324, Forward time: 1.2618, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7527 [2021-07-07 19:16:09,852][root][INFO] - ===> Epoch[2298](11490/5): Loss 0.0360, Sem 0.0360 LR: 8.259e-02 Score 98.820 Data time: 0.2329, Forward time: 1.2618, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6215 [2021-07-07 19:16:24,074][root][INFO] - ===> Epoch[2299](11495/5): Loss 0.0339, Sem 0.0339 LR: 8.258e-02 Score 98.838 Data time: 0.2455, Forward time: 1.2618, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7963 [2021-07-07 19:16:37,663][root][INFO] - ===> Epoch[2300](11500/5): Loss 0.0333, Sem 0.0333 LR: 8.257e-02 Score 98.855 Data time: 0.2752, Forward time: 1.2618, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6631 [2021-07-07 19:16:52,090][root][INFO] - ===> Epoch[2301](11505/5): Loss 0.0370, Sem 0.0370 LR: 8.256e-02 Score 98.762 Data time: 0.2331, Forward time: 1.2618, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8460 [2021-07-07 19:17:05,845][root][INFO] - ===> Epoch[2302](11510/5): Loss 0.0412, Sem 0.0412 LR: 8.256e-02 Score 98.688 Data time: 0.2404, Forward time: 1.2618, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6984 [2021-07-07 19:17:21,107][root][INFO] - ===> Epoch[2303](11515/5): Loss 0.0613, Sem 0.0613 LR: 8.255e-02 Score 98.029 Data time: 0.2552, Forward time: 1.2619, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 3.0096 [2021-07-07 19:17:35,065][root][INFO] - ===> Epoch[2304](11520/5): Loss 0.0618, Sem 0.0618 LR: 8.254e-02 Score 98.050 Data time: 0.1940, Forward time: 1.2619, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7480 [2021-07-07 19:17:49,020][root][INFO] - ===> Epoch[2305](11525/5): Loss 0.0739, Sem 0.0739 LR: 8.253e-02 Score 97.734 Data time: 0.2204, Forward time: 1.2619, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7484 [2021-07-07 19:18:03,729][root][INFO] - ===> Epoch[2306](11530/5): Loss 0.0683, Sem 0.0683 LR: 8.253e-02 Score 97.840 Data time: 0.2168, Forward time: 1.2619, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8886 [2021-07-07 19:18:17,200][root][INFO] - ===> Epoch[2307](11535/5): Loss 0.0747, Sem 0.0747 LR: 8.252e-02 Score 97.667 Data time: 0.2237, Forward time: 1.2619, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6473 [2021-07-07 19:18:31,440][root][INFO] - ===> Epoch[2308](11540/5): Loss 0.0824, Sem 0.0824 LR: 8.251e-02 Score 97.464 Data time: 0.2303, Forward time: 1.2619, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8075 [2021-07-07 19:18:45,361][root][INFO] - ===> Epoch[2309](11545/5): Loss 0.0824, Sem 0.0824 LR: 8.250e-02 Score 97.493 Data time: 0.2274, Forward time: 1.2619, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7451 [2021-07-07 19:19:00,095][root][INFO] - ===> Epoch[2310](11550/5): Loss 0.0877, Sem 0.0877 LR: 8.250e-02 Score 97.204 Data time: 0.2091, Forward time: 1.2619, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.8992 [2021-07-07 19:19:13,577][root][INFO] - ===> Epoch[2311](11555/5): Loss 0.0678, Sem 0.0678 LR: 8.249e-02 Score 97.815 Data time: 0.3317, Forward time: 1.2619, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6548 [2021-07-07 19:19:27,509][root][INFO] - ===> Epoch[2312](11560/5): Loss 0.0683, Sem 0.0683 LR: 8.248e-02 Score 97.809 Data time: 0.1807, Forward time: 1.2619, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7470 [2021-07-07 19:19:41,846][root][INFO] - ===> Epoch[2313](11565/5): Loss 0.0647, Sem 0.0647 LR: 8.247e-02 Score 97.882 Data time: 0.2489, Forward time: 1.2619, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8194 [2021-07-07 19:19:55,411][root][INFO] - ===> Epoch[2314](11570/5): Loss 0.0634, Sem 0.0634 LR: 8.246e-02 Score 97.891 Data time: 0.2720, Forward time: 1.2619, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6660 [2021-07-07 19:20:08,573][root][INFO] - ===> Epoch[2315](11575/5): Loss 0.0531, Sem 0.0531 LR: 8.246e-02 Score 98.198 Data time: 0.1572, Forward time: 1.2618, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.5953 [2021-07-07 19:20:22,138][root][INFO] - ===> Epoch[2316](11580/5): Loss 0.0538, Sem 0.0538 LR: 8.245e-02 Score 98.219 Data time: 0.1869, Forward time: 1.2617, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6768 [2021-07-07 19:20:37,132][root][INFO] - ===> Epoch[2317](11585/5): Loss 0.0482, Sem 0.0482 LR: 8.244e-02 Score 98.360 Data time: 0.2343, Forward time: 1.2617, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.9481 [2021-07-07 19:20:51,365][root][INFO] - ===> Epoch[2318](11590/5): Loss 0.0468, Sem 0.0468 LR: 8.243e-02 Score 98.415 Data time: 0.2223, Forward time: 1.2617, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7962 [2021-07-07 19:21:04,558][root][INFO] - ===> Epoch[2319](11595/5): Loss 0.0455, Sem 0.0455 LR: 8.243e-02 Score 98.478 Data time: 0.2620, Forward time: 1.2617, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.5880 [2021-07-07 19:21:18,441][root][INFO] - ===> Epoch[2320](11600/5): Loss 0.0459, Sem 0.0459 LR: 8.242e-02 Score 98.447 Data time: 0.2302, Forward time: 1.2617, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7291 [2021-07-07 19:21:18,443][root][INFO] - ===> Start testing [2021-07-07 19:21:53,774][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1194 Loss 0.269 (AVG: 0.472) Score 92.110 (AVG: 89.784) mIOU 66.090 mAP 83.251 mAcc 73.009 IOU: 60.774 0.000 72.840 72.623 94.019 88.335 15.646 78.700 98.703 62.092 71.776 85.720 57.947 mAP: 83.246 7.419 84.705 89.890 98.762 98.123 56.306 96.883 99.807 91.051 91.042 94.185 90.844 mAcc: 75.869 0.000 77.319 83.428 95.156 93.350 16.343 90.624 99.615 68.923 89.958 96.569 61.962 [2021-07-07 19:21:53,774][root][INFO] - Finished test. Elapsed time: 35.3302 [2021-07-07 19:21:54,823][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 19:22:07,664][root][INFO] - ===> Epoch[2321](11605/5): Loss 0.0430, Sem 0.0430 LR: 8.241e-02 Score 98.557 Data time: 0.2344, Forward time: 1.2617, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.5283 [2021-07-07 19:22:21,931][root][INFO] - ===> Epoch[2322](11610/5): Loss 0.0406, Sem 0.0406 LR: 8.240e-02 Score 98.622 Data time: 0.2175, Forward time: 1.2617, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8004 [2021-07-07 19:22:35,434][root][INFO] - ===> Epoch[2323](11615/5): Loss 0.0408, Sem 0.0408 LR: 8.240e-02 Score 98.602 Data time: 0.2121, Forward time: 1.2617, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6636 [2021-07-07 19:22:50,023][root][INFO] - ===> Epoch[2324](11620/5): Loss 0.0380, Sem 0.0380 LR: 8.239e-02 Score 98.717 Data time: 0.1809, Forward time: 1.2617, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.8856 [2021-07-07 19:23:04,035][root][INFO] - ===> Epoch[2325](11625/5): Loss 0.0359, Sem 0.0359 LR: 8.238e-02 Score 98.765 Data time: 0.3135, Forward time: 1.2617, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7461 [2021-07-07 19:23:18,189][root][INFO] - ===> Epoch[2326](11630/5): Loss 0.0314, Sem 0.0314 LR: 8.237e-02 Score 98.906 Data time: 0.1988, Forward time: 1.2617, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7902 [2021-07-07 19:23:32,068][root][INFO] - ===> Epoch[2327](11635/5): Loss 0.0302, Sem 0.0302 LR: 8.237e-02 Score 98.948 Data time: 0.1682, Forward time: 1.2617, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7276 [2021-07-07 19:23:46,243][root][INFO] - ===> Epoch[2328](11640/5): Loss 0.0310, Sem 0.0310 LR: 8.236e-02 Score 98.925 Data time: 0.2112, Forward time: 1.2617, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7894 [2021-07-07 19:23:59,937][root][INFO] - ===> Epoch[2329](11645/5): Loss 0.0313, Sem 0.0313 LR: 8.235e-02 Score 98.922 Data time: 0.2026, Forward time: 1.2616, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.6960 [2021-07-07 19:24:14,774][root][INFO] - ===> Epoch[2330](11650/5): Loss 0.0311, Sem 0.0311 LR: 8.234e-02 Score 98.915 Data time: 0.2186, Forward time: 1.2616, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.9193 [2021-07-07 19:24:28,457][root][INFO] - ===> Epoch[2331](11655/5): Loss 0.0276, Sem 0.0276 LR: 8.233e-02 Score 99.032 Data time: 0.2671, Forward time: 1.2617, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.6775 [2021-07-07 19:24:42,164][root][INFO] - ===> Epoch[2332](11660/5): Loss 0.0262, Sem 0.0262 LR: 8.233e-02 Score 99.070 Data time: 0.2833, Forward time: 1.2617, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6922 [2021-07-07 19:24:56,160][root][INFO] - ===> Epoch[2333](11665/5): Loss 0.0250, Sem 0.0250 LR: 8.232e-02 Score 99.102 Data time: 0.2171, Forward time: 1.2617, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7603 [2021-07-07 19:25:10,941][root][INFO] - ===> Epoch[2334](11670/5): Loss 0.0245, Sem 0.0245 LR: 8.231e-02 Score 99.145 Data time: 0.2851, Forward time: 1.2618, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.9089 [2021-07-07 19:25:24,094][root][INFO] - ===> Epoch[2335](11675/5): Loss 0.0251, Sem 0.0251 LR: 8.230e-02 Score 99.116 Data time: 0.1967, Forward time: 1.2617, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.5926 [2021-07-07 19:25:37,420][root][INFO] - ===> Epoch[2336](11680/5): Loss 0.0259, Sem 0.0259 LR: 8.230e-02 Score 99.094 Data time: 0.2923, Forward time: 1.2618, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.6196 [2021-07-07 19:25:51,369][root][INFO] - ===> Epoch[2337](11685/5): Loss 0.0257, Sem 0.0257 LR: 8.229e-02 Score 99.110 Data time: 0.2542, Forward time: 1.2618, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7404 [2021-07-07 19:26:06,027][root][INFO] - ===> Epoch[2338](11690/5): Loss 0.0255, Sem 0.0255 LR: 8.228e-02 Score 99.118 Data time: 0.2124, Forward time: 1.2618, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8792 [2021-07-07 19:26:19,994][root][INFO] - ===> Epoch[2339](11695/5): Loss 0.0262, Sem 0.0262 LR: 8.227e-02 Score 99.091 Data time: 0.2584, Forward time: 1.2618, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7412 [2021-07-07 19:26:33,932][root][INFO] - ===> Epoch[2340](11700/5): Loss 0.0248, Sem 0.0248 LR: 8.227e-02 Score 99.131 Data time: 0.2645, Forward time: 1.2618, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7431 [2021-07-07 19:26:48,490][root][INFO] - ===> Epoch[2341](11705/5): Loss 0.0230, Sem 0.0230 LR: 8.226e-02 Score 99.182 Data time: 0.2670, Forward time: 1.2618, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8621 [2021-07-07 19:27:02,861][root][INFO] - ===> Epoch[2342](11710/5): Loss 0.0246, Sem 0.0246 LR: 8.225e-02 Score 99.173 Data time: 0.2804, Forward time: 1.2619, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8316 [2021-07-07 19:27:16,725][root][INFO] - ===> Epoch[2343](11715/5): Loss 0.0246, Sem 0.0246 LR: 8.224e-02 Score 99.133 Data time: 0.2309, Forward time: 1.2619, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7368 [2021-07-07 19:27:30,025][root][INFO] - ===> Epoch[2344](11720/5): Loss 0.0249, Sem 0.0249 LR: 8.223e-02 Score 99.112 Data time: 0.2117, Forward time: 1.2619, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6264 [2021-07-07 19:27:44,411][root][INFO] - ===> Epoch[2345](11725/5): Loss 0.0244, Sem 0.0244 LR: 8.223e-02 Score 99.135 Data time: 0.2806, Forward time: 1.2619, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8258 [2021-07-07 19:27:58,656][root][INFO] - ===> Epoch[2346](11730/5): Loss 0.0240, Sem 0.0240 LR: 8.222e-02 Score 99.164 Data time: 0.2780, Forward time: 1.2619, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7987 [2021-07-07 19:28:12,954][root][INFO] - ===> Epoch[2347](11735/5): Loss 0.0271, Sem 0.0271 LR: 8.221e-02 Score 99.111 Data time: 0.2730, Forward time: 1.2620, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7995 [2021-07-07 19:28:26,997][root][INFO] - ===> Epoch[2348](11740/5): Loss 0.0252, Sem 0.0252 LR: 8.220e-02 Score 99.113 Data time: 0.2663, Forward time: 1.2619, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7627 [2021-07-07 19:28:41,062][root][INFO] - ===> Epoch[2349](11745/5): Loss 0.0264, Sem 0.0264 LR: 8.220e-02 Score 99.100 Data time: 0.2867, Forward time: 1.2620, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7662 [2021-07-07 19:28:54,768][root][INFO] - ===> Epoch[2350](11750/5): Loss 0.0234, Sem 0.0234 LR: 8.219e-02 Score 99.191 Data time: 0.2936, Forward time: 1.2620, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6965 [2021-07-07 19:29:08,765][root][INFO] - ===> Epoch[2351](11755/5): Loss 0.0224, Sem 0.0224 LR: 8.218e-02 Score 99.198 Data time: 0.2132, Forward time: 1.2620, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7546 [2021-07-07 19:29:22,888][root][INFO] - ===> Epoch[2352](11760/5): Loss 0.0243, Sem 0.0243 LR: 8.217e-02 Score 99.156 Data time: 0.2238, Forward time: 1.2620, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7775 [2021-07-07 19:29:37,494][root][INFO] - ===> Epoch[2353](11765/5): Loss 0.0247, Sem 0.0247 LR: 8.217e-02 Score 99.167 Data time: 0.2042, Forward time: 1.2620, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8881 [2021-07-07 19:29:51,077][root][INFO] - ===> Epoch[2354](11770/5): Loss 0.0239, Sem 0.0239 LR: 8.216e-02 Score 99.170 Data time: 0.1828, Forward time: 1.2620, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6793 [2021-07-07 19:30:05,129][root][INFO] - ===> Epoch[2355](11775/5): Loss 0.0267, Sem 0.0267 LR: 8.215e-02 Score 99.090 Data time: 0.2251, Forward time: 1.2619, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7677 [2021-07-07 19:30:19,297][root][INFO] - ===> Epoch[2356](11780/5): Loss 0.0256, Sem 0.0256 LR: 8.214e-02 Score 99.102 Data time: 0.1246, Forward time: 1.2619, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8038 [2021-07-07 19:30:34,284][root][INFO] - ===> Epoch[2357](11785/5): Loss 0.0235, Sem 0.0235 LR: 8.214e-02 Score 99.165 Data time: 0.2423, Forward time: 1.2619, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.9543 [2021-07-07 19:30:47,681][root][INFO] - ===> Epoch[2358](11790/5): Loss 0.0252, Sem 0.0252 LR: 8.213e-02 Score 99.110 Data time: 0.2328, Forward time: 1.2619, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6266 [2021-07-07 19:31:01,834][root][INFO] - ===> Epoch[2359](11795/5): Loss 0.0285, Sem 0.0285 LR: 8.212e-02 Score 99.073 Data time: 0.2322, Forward time: 1.2619, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7783 [2021-07-07 19:31:15,585][root][INFO] - ===> Epoch[2360](11800/5): Loss 0.0257, Sem 0.0257 LR: 8.211e-02 Score 99.103 Data time: 0.2434, Forward time: 1.2619, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7015 [2021-07-07 19:31:15,586][root][INFO] - ===> Start testing [2021-07-07 19:31:50,960][root][INFO] - 68/68: Data time: 0.0377, Iter time: 0.1359 Loss 0.301 (AVG: 0.476) Score 89.790 (AVG: 90.492) mIOU 68.898 mAP 83.453 mAcc 76.438 IOU: 63.949 0.074 70.686 74.221 93.911 91.907 29.456 78.654 98.565 70.650 78.320 86.330 58.950 mAP: 83.784 5.507 93.645 87.655 98.878 98.934 61.759 97.275 99.843 81.127 92.454 93.534 90.502 mAcc: 84.926 0.365 89.626 81.786 95.065 96.389 31.589 95.287 99.331 73.382 87.884 94.633 63.433 [2021-07-07 19:31:50,960][root][INFO] - Finished test. Elapsed time: 35.3734 [2021-07-07 19:31:52,054][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 19:32:05,010][root][INFO] - ===> Epoch[2361](11805/5): Loss 0.0268, Sem 0.0268 LR: 8.210e-02 Score 99.071 Data time: 0.2199, Forward time: 1.2619, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.5461 [2021-07-07 19:32:19,513][root][INFO] - ===> Epoch[2362](11810/5): Loss 0.0246, Sem 0.0246 LR: 8.210e-02 Score 99.132 Data time: 0.2062, Forward time: 1.2619, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8626 [2021-07-07 19:32:32,833][root][INFO] - ===> Epoch[2363](11815/5): Loss 0.0244, Sem 0.0244 LR: 8.209e-02 Score 99.135 Data time: 0.2340, Forward time: 1.2620, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6261 [2021-07-07 19:32:47,056][root][INFO] - ===> Epoch[2364](11820/5): Loss 0.0221, Sem 0.0221 LR: 8.208e-02 Score 99.215 Data time: 0.1944, Forward time: 1.2619, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8141 [2021-07-07 19:33:01,035][root][INFO] - ===> Epoch[2365](11825/5): Loss 0.0216, Sem 0.0216 LR: 8.207e-02 Score 99.231 Data time: 0.1741, Forward time: 1.2619, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7575 [2021-07-07 19:33:14,163][root][INFO] - ===> Epoch[2366](11830/5): Loss 0.0218, Sem 0.0218 LR: 8.207e-02 Score 99.224 Data time: 0.2821, Forward time: 1.2619, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.5823 [2021-07-07 19:33:27,451][root][INFO] - ===> Epoch[2367](11835/5): Loss 0.0214, Sem 0.0214 LR: 8.206e-02 Score 99.251 Data time: 0.2200, Forward time: 1.2619, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6160 [2021-07-07 19:33:41,730][root][INFO] - ===> Epoch[2368](11840/5): Loss 0.0217, Sem 0.0217 LR: 8.205e-02 Score 99.228 Data time: 0.2444, Forward time: 1.2619, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8076 [2021-07-07 19:33:56,043][root][INFO] - ===> Epoch[2369](11845/5): Loss 0.0252, Sem 0.0252 LR: 8.204e-02 Score 99.169 Data time: 0.2796, Forward time: 1.2618, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8186 [2021-07-07 19:34:09,879][root][INFO] - ===> Epoch[2370](11850/5): Loss 0.0213, Sem 0.0213 LR: 8.204e-02 Score 99.256 Data time: 0.2485, Forward time: 1.2619, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7180 [2021-07-07 19:34:23,919][root][INFO] - ===> Epoch[2371](11855/5): Loss 0.0208, Sem 0.0208 LR: 8.203e-02 Score 99.265 Data time: 0.2148, Forward time: 1.2618, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7701 [2021-07-07 19:34:37,893][root][INFO] - ===> Epoch[2372](11860/5): Loss 0.0201, Sem 0.0201 LR: 8.202e-02 Score 99.275 Data time: 0.3099, Forward time: 1.2619, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7483 [2021-07-07 19:34:52,230][root][INFO] - ===> Epoch[2373](11865/5): Loss 0.0213, Sem 0.0213 LR: 8.201e-02 Score 99.256 Data time: 0.2521, Forward time: 1.2619, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8224 [2021-07-07 19:35:05,897][root][INFO] - ===> Epoch[2374](11870/5): Loss 0.0205, Sem 0.0205 LR: 8.200e-02 Score 99.263 Data time: 0.1918, Forward time: 1.2619, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6798 [2021-07-07 19:35:20,126][root][INFO] - ===> Epoch[2375](11875/5): Loss 0.0189, Sem 0.0189 LR: 8.200e-02 Score 99.331 Data time: 0.2305, Forward time: 1.2619, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7957 [2021-07-07 19:35:35,055][root][INFO] - ===> Epoch[2376](11880/5): Loss 0.0201, Sem 0.0201 LR: 8.199e-02 Score 99.280 Data time: 0.2156, Forward time: 1.2619, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9405 [2021-07-07 19:35:48,435][root][INFO] - ===> Epoch[2377](11885/5): Loss 0.0189, Sem 0.0189 LR: 8.198e-02 Score 99.311 Data time: 0.2063, Forward time: 1.2619, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6253 [2021-07-07 19:36:03,042][root][INFO] - ===> Epoch[2378](11890/5): Loss 0.0187, Sem 0.0187 LR: 8.197e-02 Score 99.330 Data time: 0.2363, Forward time: 1.2619, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8657 [2021-07-07 19:36:17,172][root][INFO] - ===> Epoch[2379](11895/5): Loss 0.0182, Sem 0.0182 LR: 8.197e-02 Score 99.350 Data time: 0.2330, Forward time: 1.2620, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7664 [2021-07-07 19:36:30,757][root][INFO] - ===> Epoch[2380](11900/5): Loss 0.0187, Sem 0.0187 LR: 8.196e-02 Score 99.335 Data time: 0.3049, Forward time: 1.2619, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6759 [2021-07-07 19:36:44,891][root][INFO] - ===> Epoch[2381](11905/5): Loss 0.0176, Sem 0.0176 LR: 8.195e-02 Score 99.362 Data time: 0.2138, Forward time: 1.2620, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7818 [2021-07-07 19:36:58,502][root][INFO] - ===> Epoch[2382](11910/5): Loss 0.0182, Sem 0.0182 LR: 8.194e-02 Score 99.347 Data time: 0.2070, Forward time: 1.2620, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6703 [2021-07-07 19:37:12,638][root][INFO] - ===> Epoch[2383](11915/5): Loss 0.0180, Sem 0.0180 LR: 8.194e-02 Score 99.349 Data time: 0.2320, Forward time: 1.2620, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7847 [2021-07-07 19:37:27,484][root][INFO] - ===> Epoch[2384](11920/5): Loss 0.0184, Sem 0.0184 LR: 8.193e-02 Score 99.339 Data time: 0.2574, Forward time: 1.2620, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.9259 [2021-07-07 19:37:41,638][root][INFO] - ===> Epoch[2385](11925/5): Loss 0.0172, Sem 0.0172 LR: 8.192e-02 Score 99.379 Data time: 0.2426, Forward time: 1.2621, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7858 [2021-07-07 19:37:55,679][root][INFO] - ===> Epoch[2386](11930/5): Loss 0.0170, Sem 0.0170 LR: 8.191e-02 Score 99.388 Data time: 0.2225, Forward time: 1.2621, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7611 [2021-07-07 19:38:09,864][root][INFO] - ===> Epoch[2387](11935/5): Loss 0.0175, Sem 0.0175 LR: 8.191e-02 Score 99.365 Data time: 0.2489, Forward time: 1.2622, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7756 [2021-07-07 19:38:24,689][root][INFO] - ===> Epoch[2388](11940/5): Loss 0.0166, Sem 0.0166 LR: 8.190e-02 Score 99.401 Data time: 0.2116, Forward time: 1.2622, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.9229 [2021-07-07 19:38:38,789][root][INFO] - ===> Epoch[2389](11945/5): Loss 0.0169, Sem 0.0169 LR: 8.189e-02 Score 99.391 Data time: 0.1843, Forward time: 1.2622, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7647 [2021-07-07 19:38:53,110][root][INFO] - ===> Epoch[2390](11950/5): Loss 0.0164, Sem 0.0164 LR: 8.188e-02 Score 99.407 Data time: 0.2623, Forward time: 1.2622, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8106 [2021-07-07 19:39:06,532][root][INFO] - ===> Epoch[2391](11955/5): Loss 0.0180, Sem 0.0180 LR: 8.187e-02 Score 99.350 Data time: 0.1927, Forward time: 1.2622, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6496 [2021-07-07 19:39:20,481][root][INFO] - ===> Epoch[2392](11960/5): Loss 0.0169, Sem 0.0169 LR: 8.187e-02 Score 99.395 Data time: 0.2620, Forward time: 1.2622, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7425 [2021-07-07 19:39:35,293][root][INFO] - ===> Epoch[2393](11965/5): Loss 0.0171, Sem 0.0171 LR: 8.186e-02 Score 99.385 Data time: 0.1753, Forward time: 1.2622, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9235 [2021-07-07 19:39:49,657][root][INFO] - ===> Epoch[2394](11970/5): Loss 0.0170, Sem 0.0170 LR: 8.185e-02 Score 99.390 Data time: 0.2457, Forward time: 1.2622, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8223 [2021-07-07 19:40:03,652][root][INFO] - ===> Epoch[2395](11975/5): Loss 0.0170, Sem 0.0170 LR: 8.184e-02 Score 99.387 Data time: 0.1978, Forward time: 1.2622, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7522 [2021-07-07 19:40:17,063][root][INFO] - ===> Epoch[2396](11980/5): Loss 0.0165, Sem 0.0165 LR: 8.184e-02 Score 99.403 Data time: 0.2374, Forward time: 1.2622, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6273 [2021-07-07 19:40:30,385][root][INFO] - ===> Epoch[2397](11985/5): Loss 0.0166, Sem 0.0166 LR: 8.183e-02 Score 99.394 Data time: 0.2647, Forward time: 1.2622, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6109 [2021-07-07 19:40:44,775][root][INFO] - ===> Epoch[2398](11990/5): Loss 0.0178, Sem 0.0178 LR: 8.182e-02 Score 99.358 Data time: 0.1863, Forward time: 1.2622, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8309 [2021-07-07 19:40:58,677][root][INFO] - ===> Epoch[2399](11995/5): Loss 0.0168, Sem 0.0168 LR: 8.181e-02 Score 99.393 Data time: 0.2303, Forward time: 1.2622, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7359 [2021-07-07 19:41:13,116][root][INFO] - ===> Epoch[2400](12000/5): Loss 0.0166, Sem 0.0166 LR: 8.181e-02 Score 99.404 Data time: 0.2934, Forward time: 1.2623, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8196 [2021-07-07 19:41:13,117][root][INFO] - ===> Start testing [2021-07-07 19:41:49,680][root][INFO] - 68/68: Data time: 0.0016, Iter time: 0.1406 Loss 0.296 (AVG: 0.496) Score 92.598 (AVG: 91.301) mIOU 70.465 mAP 83.893 mAcc 76.239 IOU: 64.916 0.000 82.894 76.842 95.177 92.161 28.212 81.763 98.570 68.504 80.559 87.164 59.288 mAP: 84.740 6.842 90.370 89.367 98.994 98.915 60.733 98.057 99.840 82.580 93.184 95.346 91.644 mAcc: 81.392 0.000 86.481 83.343 96.389 97.405 29.827 96.534 99.333 71.004 89.640 97.156 62.604 [2021-07-07 19:41:49,680][root][INFO] - Finished test. Elapsed time: 36.5619 [2021-07-07 19:41:50,727][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 19:42:03,852][root][INFO] - ===> Epoch[2401](12005/5): Loss 0.0187, Sem 0.0187 LR: 8.180e-02 Score 99.358 Data time: 0.2120, Forward time: 1.2622, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.5907 [2021-07-07 19:42:18,084][root][INFO] - ===> Epoch[2402](12010/5): Loss 0.0164, Sem 0.0164 LR: 8.179e-02 Score 99.415 Data time: 0.2804, Forward time: 1.2623, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7863 [2021-07-07 19:42:31,645][root][INFO] - ===> Epoch[2403](12015/5): Loss 0.0177, Sem 0.0177 LR: 8.178e-02 Score 99.363 Data time: 0.2359, Forward time: 1.2623, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.6720 [2021-07-07 19:42:45,747][root][INFO] - ===> Epoch[2404](12020/5): Loss 0.0170, Sem 0.0170 LR: 8.177e-02 Score 99.395 Data time: 0.2368, Forward time: 1.2623, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7726 [2021-07-07 19:42:59,737][root][INFO] - ===> Epoch[2405](12025/5): Loss 0.0176, Sem 0.0176 LR: 8.177e-02 Score 99.371 Data time: 0.2216, Forward time: 1.2622, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7523 [2021-07-07 19:43:13,249][root][INFO] - ===> Epoch[2406](12030/5): Loss 0.0172, Sem 0.0172 LR: 8.176e-02 Score 99.378 Data time: 0.1996, Forward time: 1.2622, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.6439 [2021-07-07 19:43:28,072][root][INFO] - ===> Epoch[2407](12035/5): Loss 0.0172, Sem 0.0172 LR: 8.175e-02 Score 99.381 Data time: 0.2276, Forward time: 1.2622, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.9117 [2021-07-07 19:43:41,815][root][INFO] - ===> Epoch[2408](12040/5): Loss 0.0158, Sem 0.0158 LR: 8.174e-02 Score 99.424 Data time: 0.2383, Forward time: 1.2622, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6914 [2021-07-07 19:43:56,744][root][INFO] - ===> Epoch[2409](12045/5): Loss 0.0172, Sem 0.0172 LR: 8.174e-02 Score 99.383 Data time: 0.1537, Forward time: 1.2622, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.9389 [2021-07-07 19:44:10,257][root][INFO] - ===> Epoch[2410](12050/5): Loss 0.0161, Sem 0.0161 LR: 8.173e-02 Score 99.415 Data time: 0.2335, Forward time: 1.2621, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6525 [2021-07-07 19:44:24,609][root][INFO] - ===> Epoch[2411](12055/5): Loss 0.0163, Sem 0.0163 LR: 8.172e-02 Score 99.409 Data time: 0.2263, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8231 [2021-07-07 19:44:39,052][root][INFO] - ===> Epoch[2412](12060/5): Loss 0.0158, Sem 0.0158 LR: 8.171e-02 Score 99.428 Data time: 0.2302, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8478 [2021-07-07 19:44:53,031][root][INFO] - ===> Epoch[2413](12065/5): Loss 0.0160, Sem 0.0160 LR: 8.171e-02 Score 99.421 Data time: 0.2030, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7490 [2021-07-07 19:45:06,602][root][INFO] - ===> Epoch[2414](12070/5): Loss 0.0163, Sem 0.0163 LR: 8.170e-02 Score 99.413 Data time: 0.2017, Forward time: 1.2621, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6661 [2021-07-07 19:45:20,658][root][INFO] - ===> Epoch[2415](12075/5): Loss 0.0159, Sem 0.0159 LR: 8.169e-02 Score 99.426 Data time: 0.2311, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7587 [2021-07-07 19:45:34,745][root][INFO] - ===> Epoch[2416](12080/5): Loss 0.0161, Sem 0.0161 LR: 8.168e-02 Score 99.413 Data time: 0.2019, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7720 [2021-07-07 19:45:49,573][root][INFO] - ===> Epoch[2417](12085/5): Loss 0.0157, Sem 0.0157 LR: 8.168e-02 Score 99.431 Data time: 0.2766, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9119 [2021-07-07 19:46:03,596][root][INFO] - ===> Epoch[2418](12090/5): Loss 0.0155, Sem 0.0155 LR: 8.167e-02 Score 99.440 Data time: 0.2151, Forward time: 1.2622, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7620 [2021-07-07 19:46:18,030][root][INFO] - ===> Epoch[2419](12095/5): Loss 0.0157, Sem 0.0157 LR: 8.166e-02 Score 99.430 Data time: 0.2341, Forward time: 1.2622, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8431 [2021-07-07 19:46:31,699][root][INFO] - ===> Epoch[2420](12100/5): Loss 0.0155, Sem 0.0155 LR: 8.165e-02 Score 99.438 Data time: 0.2173, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6856 [2021-07-07 19:46:45,790][root][INFO] - ===> Epoch[2421](12105/5): Loss 0.0162, Sem 0.0162 LR: 8.164e-02 Score 99.423 Data time: 0.2803, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7750 [2021-07-07 19:46:59,516][root][INFO] - ===> Epoch[2422](12110/5): Loss 0.0156, Sem 0.0156 LR: 8.164e-02 Score 99.443 Data time: 0.2516, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6986 [2021-07-07 19:47:13,970][root][INFO] - ===> Epoch[2423](12115/5): Loss 0.0158, Sem 0.0158 LR: 8.163e-02 Score 99.429 Data time: 0.2173, Forward time: 1.2622, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8388 [2021-07-07 19:47:27,801][root][INFO] - ===> Epoch[2424](12120/5): Loss 0.0161, Sem 0.0161 LR: 8.162e-02 Score 99.420 Data time: 0.1966, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7160 [2021-07-07 19:47:41,994][root][INFO] - ===> Epoch[2425](12125/5): Loss 0.0156, Sem 0.0156 LR: 8.161e-02 Score 99.432 Data time: 0.2367, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7859 [2021-07-07 19:47:55,440][root][INFO] - ===> Epoch[2426](12130/5): Loss 0.0158, Sem 0.0158 LR: 8.161e-02 Score 99.434 Data time: 0.1826, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6478 [2021-07-07 19:48:09,628][root][INFO] - ===> Epoch[2427](12135/5): Loss 0.0154, Sem 0.0154 LR: 8.160e-02 Score 99.442 Data time: 0.2316, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7855 [2021-07-07 19:48:23,079][root][INFO] - ===> Epoch[2428](12140/5): Loss 0.0163, Sem 0.0163 LR: 8.159e-02 Score 99.416 Data time: 0.1798, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6475 [2021-07-07 19:48:37,417][root][INFO] - ===> Epoch[2429](12145/5): Loss 0.0158, Sem 0.0158 LR: 8.158e-02 Score 99.431 Data time: 0.2124, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8159 [2021-07-07 19:48:51,056][root][INFO] - ===> Epoch[2430](12150/5): Loss 0.0156, Sem 0.0156 LR: 8.158e-02 Score 99.440 Data time: 0.2119, Forward time: 1.2620, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6652 [2021-07-07 19:49:04,364][root][INFO] - ===> Epoch[2431](12155/5): Loss 0.0156, Sem 0.0156 LR: 8.157e-02 Score 99.438 Data time: 0.2396, Forward time: 1.2620, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6154 [2021-07-07 19:49:17,705][root][INFO] - ===> Epoch[2432](12160/5): Loss 0.0153, Sem 0.0153 LR: 8.156e-02 Score 99.440 Data time: 0.2394, Forward time: 1.2620, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6207 [2021-07-07 19:49:31,844][root][INFO] - ===> Epoch[2433](12165/5): Loss 0.0152, Sem 0.0152 LR: 8.155e-02 Score 99.448 Data time: 0.3021, Forward time: 1.2621, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7718 [2021-07-07 19:49:45,545][root][INFO] - ===> Epoch[2434](12170/5): Loss 0.0158, Sem 0.0158 LR: 8.154e-02 Score 99.434 Data time: 0.2122, Forward time: 1.2621, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6873 [2021-07-07 19:50:00,729][root][INFO] - ===> Epoch[2435](12175/5): Loss 0.0148, Sem 0.0148 LR: 8.154e-02 Score 99.461 Data time: 0.2019, Forward time: 1.2621, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.9966 [2021-07-07 19:50:14,591][root][INFO] - ===> Epoch[2436](12180/5): Loss 0.0153, Sem 0.0153 LR: 8.153e-02 Score 99.440 Data time: 0.1500, Forward time: 1.2620, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7390 [2021-07-07 19:50:28,080][root][INFO] - ===> Epoch[2437](12185/5): Loss 0.0152, Sem 0.0152 LR: 8.152e-02 Score 99.450 Data time: 0.1888, Forward time: 1.2621, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6549 [2021-07-07 19:50:42,254][root][INFO] - ===> Epoch[2438](12190/5): Loss 0.0154, Sem 0.0154 LR: 8.151e-02 Score 99.441 Data time: 0.2189, Forward time: 1.2621, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7913 [2021-07-07 19:50:56,458][root][INFO] - ===> Epoch[2439](12195/5): Loss 0.0155, Sem 0.0155 LR: 8.151e-02 Score 99.442 Data time: 0.1897, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7921 [2021-07-07 19:51:10,322][root][INFO] - ===> Epoch[2440](12200/5): Loss 0.0155, Sem 0.0155 LR: 8.150e-02 Score 99.442 Data time: 0.1911, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7313 [2021-07-07 19:51:10,324][root][INFO] - ===> Start testing [2021-07-07 19:51:45,860][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1358 Loss 0.331 (AVG: 0.502) Score 92.566 (AVG: 91.506) mIOU 71.347 mAP 84.638 mAcc 77.480 IOU: 65.347 0.000 83.392 76.456 95.272 92.505 34.280 80.122 98.586 72.827 80.143 87.919 60.661 mAP: 85.279 6.117 94.591 89.715 98.885 98.975 64.006 97.866 99.810 85.023 92.514 95.412 92.101 mAcc: 80.365 0.000 90.447 84.784 96.430 97.069 35.698 96.362 99.531 75.730 89.879 96.750 64.201 [2021-07-07 19:51:45,860][root][INFO] - Finished test. Elapsed time: 35.5353 [2021-07-07 19:51:46,942][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 19:51:59,841][root][INFO] - ===> Epoch[2441](12205/5): Loss 0.0152, Sem 0.0152 LR: 8.149e-02 Score 99.450 Data time: 0.1828, Forward time: 1.2620, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.5403 [2021-07-07 19:52:14,574][root][INFO] - ===> Epoch[2442](12210/5): Loss 0.0152, Sem 0.0152 LR: 8.148e-02 Score 99.455 Data time: 0.2127, Forward time: 1.2620, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9063 [2021-07-07 19:52:28,982][root][INFO] - ===> Epoch[2443](12215/5): Loss 0.0149, Sem 0.0149 LR: 8.148e-02 Score 99.459 Data time: 0.2710, Forward time: 1.2620, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8396 [2021-07-07 19:52:42,246][root][INFO] - ===> Epoch[2444](12220/5): Loss 0.0146, Sem 0.0146 LR: 8.147e-02 Score 99.472 Data time: 0.2415, Forward time: 1.2620, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6066 [2021-07-07 19:52:56,375][root][INFO] - ===> Epoch[2445](12225/5): Loss 0.0150, Sem 0.0150 LR: 8.146e-02 Score 99.456 Data time: 0.2415, Forward time: 1.2620, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7763 [2021-07-07 19:53:11,230][root][INFO] - ===> Epoch[2446](12230/5): Loss 0.0152, Sem 0.0152 LR: 8.145e-02 Score 99.447 Data time: 0.2888, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9153 [2021-07-07 19:53:25,040][root][INFO] - ===> Epoch[2447](12235/5): Loss 0.0152, Sem 0.0152 LR: 8.144e-02 Score 99.451 Data time: 0.2309, Forward time: 1.2621, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7210 [2021-07-07 19:53:40,002][root][INFO] - ===> Epoch[2448](12240/5): Loss 0.0156, Sem 0.0156 LR: 8.144e-02 Score 99.435 Data time: 0.2655, Forward time: 1.2622, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.9470 [2021-07-07 19:53:53,966][root][INFO] - ===> Epoch[2449](12245/5): Loss 0.0153, Sem 0.0153 LR: 8.143e-02 Score 99.442 Data time: 0.2690, Forward time: 1.2621, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7453 [2021-07-07 19:54:08,025][root][INFO] - ===> Epoch[2450](12250/5): Loss 0.0154, Sem 0.0154 LR: 8.142e-02 Score 99.442 Data time: 0.2096, Forward time: 1.2621, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7671 [2021-07-07 19:54:22,767][root][INFO] - ===> Epoch[2451](12255/5): Loss 0.0147, Sem 0.0147 LR: 8.141e-02 Score 99.464 Data time: 0.1968, Forward time: 1.2622, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9067 [2021-07-07 19:54:36,883][root][INFO] - ===> Epoch[2452](12260/5): Loss 0.0153, Sem 0.0153 LR: 8.141e-02 Score 99.449 Data time: 0.2118, Forward time: 1.2622, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7836 [2021-07-07 19:54:50,769][root][INFO] - ===> Epoch[2453](12265/5): Loss 0.0148, Sem 0.0148 LR: 8.140e-02 Score 99.464 Data time: 0.2937, Forward time: 1.2622, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7340 [2021-07-07 19:55:04,378][root][INFO] - ===> Epoch[2454](12270/5): Loss 0.0147, Sem 0.0147 LR: 8.139e-02 Score 99.469 Data time: 0.2420, Forward time: 1.2623, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6770 [2021-07-07 19:55:18,094][root][INFO] - ===> Epoch[2455](12275/5): Loss 0.0148, Sem 0.0148 LR: 8.138e-02 Score 99.467 Data time: 0.2101, Forward time: 1.2622, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7091 [2021-07-07 19:55:31,992][root][INFO] - ===> Epoch[2456](12280/5): Loss 0.0140, Sem 0.0140 LR: 8.138e-02 Score 99.487 Data time: 0.2254, Forward time: 1.2623, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7342 [2021-07-07 19:55:45,971][root][INFO] - ===> Epoch[2457](12285/5): Loss 0.0152, Sem 0.0152 LR: 8.137e-02 Score 99.463 Data time: 0.2232, Forward time: 1.2622, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7503 [2021-07-07 19:56:00,505][root][INFO] - ===> Epoch[2458](12290/5): Loss 0.0149, Sem 0.0149 LR: 8.136e-02 Score 99.462 Data time: 0.2379, Forward time: 1.2623, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8523 [2021-07-07 19:56:14,913][root][INFO] - ===> Epoch[2459](12295/5): Loss 0.0146, Sem 0.0146 LR: 8.135e-02 Score 99.471 Data time: 0.1624, Forward time: 1.2623, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8349 [2021-07-07 19:56:29,839][root][INFO] - ===> Epoch[2460](12300/5): Loss 0.0154, Sem 0.0154 LR: 8.135e-02 Score 99.452 Data time: 0.2740, Forward time: 1.2623, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.9304 [2021-07-07 19:56:44,067][root][INFO] - ===> Epoch[2461](12305/5): Loss 0.0147, Sem 0.0147 LR: 8.134e-02 Score 99.467 Data time: 0.2693, Forward time: 1.2623, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7957 [2021-07-07 19:56:59,094][root][INFO] - ===> Epoch[2462](12310/5): Loss 0.0147, Sem 0.0147 LR: 8.133e-02 Score 99.475 Data time: 0.2016, Forward time: 1.2623, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9532 [2021-07-07 19:57:13,961][root][INFO] - ===> Epoch[2463](12315/5): Loss 0.0147, Sem 0.0147 LR: 8.132e-02 Score 99.470 Data time: 0.2880, Forward time: 1.2623, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9266 [2021-07-07 19:57:27,961][root][INFO] - ===> Epoch[2464](12320/5): Loss 0.0148, Sem 0.0148 LR: 8.131e-02 Score 99.463 Data time: 0.2366, Forward time: 1.2623, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7587 [2021-07-07 19:57:42,232][root][INFO] - ===> Epoch[2465](12325/5): Loss 0.0145, Sem 0.0145 LR: 8.131e-02 Score 99.473 Data time: 0.2137, Forward time: 1.2623, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8085 [2021-07-07 19:57:56,618][root][INFO] - ===> Epoch[2466](12330/5): Loss 0.0149, Sem 0.0149 LR: 8.130e-02 Score 99.461 Data time: 0.1976, Forward time: 1.2623, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8279 [2021-07-07 19:58:10,824][root][INFO] - ===> Epoch[2467](12335/5): Loss 0.0151, Sem 0.0151 LR: 8.129e-02 Score 99.453 Data time: 0.2843, Forward time: 1.2623, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7995 [2021-07-07 19:58:25,021][root][INFO] - ===> Epoch[2468](12340/5): Loss 0.0147, Sem 0.0147 LR: 8.128e-02 Score 99.465 Data time: 0.2426, Forward time: 1.2622, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7934 [2021-07-07 19:58:39,008][root][INFO] - ===> Epoch[2469](12345/5): Loss 0.0158, Sem 0.0158 LR: 8.128e-02 Score 99.441 Data time: 0.2029, Forward time: 1.2622, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7566 [2021-07-07 19:58:53,714][root][INFO] - ===> Epoch[2470](12350/5): Loss 0.0151, Sem 0.0151 LR: 8.127e-02 Score 99.457 Data time: 0.2920, Forward time: 1.2622, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8794 [2021-07-07 19:59:07,589][root][INFO] - ===> Epoch[2471](12355/5): Loss 0.0147, Sem 0.0147 LR: 8.126e-02 Score 99.463 Data time: 0.2135, Forward time: 1.2623, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7376 [2021-07-07 19:59:21,198][root][INFO] - ===> Epoch[2472](12360/5): Loss 0.0145, Sem 0.0145 LR: 8.125e-02 Score 99.476 Data time: 0.2228, Forward time: 1.2623, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6883 [2021-07-07 19:59:35,597][root][INFO] - ===> Epoch[2473](12365/5): Loss 0.0150, Sem 0.0150 LR: 8.125e-02 Score 99.457 Data time: 0.2341, Forward time: 1.2623, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8196 [2021-07-07 19:59:49,952][root][INFO] - ===> Epoch[2474](12370/5): Loss 0.0148, Sem 0.0148 LR: 8.124e-02 Score 99.465 Data time: 0.2416, Forward time: 1.2624, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8225 [2021-07-07 20:00:03,703][root][INFO] - ===> Epoch[2475](12375/5): Loss 0.0142, Sem 0.0142 LR: 8.123e-02 Score 99.489 Data time: 0.2535, Forward time: 1.2624, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6979 [2021-07-07 20:00:18,261][root][INFO] - ===> Epoch[2476](12380/5): Loss 0.0142, Sem 0.0142 LR: 8.122e-02 Score 99.499 Data time: 0.2525, Forward time: 1.2625, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8541 [2021-07-07 20:00:31,874][root][INFO] - ===> Epoch[2477](12385/5): Loss 0.0150, Sem 0.0150 LR: 8.121e-02 Score 99.464 Data time: 0.1980, Forward time: 1.2625, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6882 [2021-07-07 20:00:45,810][root][INFO] - ===> Epoch[2478](12390/5): Loss 0.0151, Sem 0.0151 LR: 8.121e-02 Score 99.457 Data time: 0.1924, Forward time: 1.2625, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7416 [2021-07-07 20:01:00,020][root][INFO] - ===> Epoch[2479](12395/5): Loss 0.0146, Sem 0.0146 LR: 8.120e-02 Score 99.469 Data time: 0.1687, Forward time: 1.2625, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7995 [2021-07-07 20:01:14,523][root][INFO] - ===> Epoch[2480](12400/5): Loss 0.0139, Sem 0.0139 LR: 8.119e-02 Score 99.495 Data time: 0.2650, Forward time: 1.2625, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8549 [2021-07-07 20:01:14,525][root][INFO] - ===> Start testing [2021-07-07 20:01:50,706][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1243 Loss 0.291 (AVG: 0.503) Score 92.415 (AVG: 91.267) mIOU 71.021 mAP 84.855 mAcc 77.403 IOU: 64.743 0.000 81.938 77.285 95.067 92.655 30.025 81.331 98.573 75.446 79.573 87.308 59.327 mAP: 84.152 14.277 94.313 89.641 98.846 98.762 58.322 98.357 99.786 87.330 92.267 95.108 91.959 mAcc: 80.397 0.000 90.460 84.216 96.189 96.664 33.200 96.523 99.452 79.242 89.530 96.551 63.819 [2021-07-07 20:01:50,706][root][INFO] - Finished test. Elapsed time: 36.1801 [2021-07-07 20:01:51,827][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 20:02:05,163][root][INFO] - ===> Epoch[2481](12405/5): Loss 0.0145, Sem 0.0145 LR: 8.118e-02 Score 99.477 Data time: 0.2222, Forward time: 1.2625, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6231 [2021-07-07 20:02:19,166][root][INFO] - ===> Epoch[2482](12410/5): Loss 0.0148, Sem 0.0148 LR: 8.118e-02 Score 99.465 Data time: 0.1688, Forward time: 1.2625, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7636 [2021-07-07 20:02:33,874][root][INFO] - ===> Epoch[2483](12415/5): Loss 0.0142, Sem 0.0142 LR: 8.117e-02 Score 99.488 Data time: 0.2507, Forward time: 1.2625, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8965 [2021-07-07 20:02:48,648][root][INFO] - ===> Epoch[2484](12420/5): Loss 0.0152, Sem 0.0152 LR: 8.116e-02 Score 99.464 Data time: 0.2502, Forward time: 1.2625, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8964 [2021-07-07 20:03:02,197][root][INFO] - ===> Epoch[2485](12425/5): Loss 0.0150, Sem 0.0150 LR: 8.115e-02 Score 99.454 Data time: 0.2441, Forward time: 1.2626, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6603 [2021-07-07 20:03:16,244][root][INFO] - ===> Epoch[2486](12430/5): Loss 0.0144, Sem 0.0144 LR: 8.115e-02 Score 99.478 Data time: 0.1694, Forward time: 1.2626, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7625 [2021-07-07 20:03:30,193][root][INFO] - ===> Epoch[2487](12435/5): Loss 0.0152, Sem 0.0152 LR: 8.114e-02 Score 99.455 Data time: 0.1768, Forward time: 1.2626, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7482 [2021-07-07 20:03:43,887][root][INFO] - ===> Epoch[2488](12440/5): Loss 0.0145, Sem 0.0145 LR: 8.113e-02 Score 99.473 Data time: 0.1836, Forward time: 1.2626, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6950 [2021-07-07 20:03:57,584][root][INFO] - ===> Epoch[2489](12445/5): Loss 0.0149, Sem 0.0149 LR: 8.112e-02 Score 99.463 Data time: 0.2085, Forward time: 1.2625, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6964 [2021-07-07 20:04:11,511][root][INFO] - ===> Epoch[2490](12450/5): Loss 0.0153, Sem 0.0153 LR: 8.111e-02 Score 99.449 Data time: 0.2279, Forward time: 1.2625, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7415 [2021-07-07 20:04:24,831][root][INFO] - ===> Epoch[2491](12455/5): Loss 0.0166, Sem 0.0166 LR: 8.111e-02 Score 99.411 Data time: 0.2226, Forward time: 1.2625, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6203 [2021-07-07 20:04:38,615][root][INFO] - ===> Epoch[2492](12460/5): Loss 0.0161, Sem 0.0161 LR: 8.110e-02 Score 99.420 Data time: 0.2033, Forward time: 1.2625, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7100 [2021-07-07 20:04:53,443][root][INFO] - ===> Epoch[2493](12465/5): Loss 0.0155, Sem 0.0155 LR: 8.109e-02 Score 99.442 Data time: 0.2325, Forward time: 1.2625, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9111 [2021-07-07 20:05:07,701][root][INFO] - ===> Epoch[2494](12470/5): Loss 0.0154, Sem 0.0154 LR: 8.108e-02 Score 99.446 Data time: 0.2899, Forward time: 1.2625, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7988 [2021-07-07 20:05:21,183][root][INFO] - ===> Epoch[2495](12475/5): Loss 0.0153, Sem 0.0153 LR: 8.108e-02 Score 99.445 Data time: 0.1995, Forward time: 1.2626, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6524 [2021-07-07 20:05:36,065][root][INFO] - ===> Epoch[2496](12480/5): Loss 0.0152, Sem 0.0152 LR: 8.107e-02 Score 99.450 Data time: 0.2527, Forward time: 1.2626, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9190 [2021-07-07 20:05:50,078][root][INFO] - ===> Epoch[2497](12485/5): Loss 0.0152, Sem 0.0152 LR: 8.106e-02 Score 99.453 Data time: 0.2254, Forward time: 1.2626, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7428 [2021-07-07 20:06:04,765][root][INFO] - ===> Epoch[2498](12490/5): Loss 0.0151, Sem 0.0151 LR: 8.105e-02 Score 99.458 Data time: 0.2234, Forward time: 1.2626, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8824 [2021-07-07 20:06:18,544][root][INFO] - ===> Epoch[2499](12495/5): Loss 0.0152, Sem 0.0152 LR: 8.105e-02 Score 99.451 Data time: 0.2786, Forward time: 1.2626, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7087 [2021-07-07 20:06:32,645][root][INFO] - ===> Epoch[2500](12500/5): Loss 0.0154, Sem 0.0154 LR: 8.104e-02 Score 99.440 Data time: 0.2194, Forward time: 1.2626, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7726 [2021-07-07 20:06:46,537][root][INFO] - ===> Epoch[2501](12505/5): Loss 0.0149, Sem 0.0149 LR: 8.103e-02 Score 99.467 Data time: 0.2685, Forward time: 1.2627, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7301 [2021-07-07 20:07:01,044][root][INFO] - ===> Epoch[2502](12510/5): Loss 0.0140, Sem 0.0140 LR: 8.102e-02 Score 99.495 Data time: 0.2193, Forward time: 1.2627, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8611 [2021-07-07 20:07:15,565][root][INFO] - ===> Epoch[2503](12515/5): Loss 0.0145, Sem 0.0145 LR: 8.102e-02 Score 99.474 Data time: 0.2225, Forward time: 1.2627, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8647 [2021-07-07 20:07:29,770][root][INFO] - ===> Epoch[2504](12520/5): Loss 0.0145, Sem 0.0145 LR: 8.101e-02 Score 99.475 Data time: 0.2193, Forward time: 1.2627, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7908 [2021-07-07 20:07:43,267][root][INFO] - ===> Epoch[2505](12525/5): Loss 0.0149, Sem 0.0149 LR: 8.100e-02 Score 99.467 Data time: 0.2550, Forward time: 1.2627, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6427 [2021-07-07 20:07:56,909][root][INFO] - ===> Epoch[2506](12530/5): Loss 0.0149, Sem 0.0149 LR: 8.099e-02 Score 99.465 Data time: 0.2451, Forward time: 1.2627, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6929 [2021-07-07 20:08:11,645][root][INFO] - ===> Epoch[2507](12535/5): Loss 0.0142, Sem 0.0142 LR: 8.098e-02 Score 99.485 Data time: 0.1945, Forward time: 1.2626, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9026 [2021-07-07 20:08:24,941][root][INFO] - ===> Epoch[2508](12540/5): Loss 0.0153, Sem 0.0153 LR: 8.098e-02 Score 99.460 Data time: 0.1650, Forward time: 1.2626, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6109 [2021-07-07 20:08:38,531][root][INFO] - ===> Epoch[2509](12545/5): Loss 0.0156, Sem 0.0156 LR: 8.097e-02 Score 99.437 Data time: 0.2084, Forward time: 1.2626, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6672 [2021-07-07 20:08:53,354][root][INFO] - ===> Epoch[2510](12550/5): Loss 0.0148, Sem 0.0148 LR: 8.096e-02 Score 99.462 Data time: 0.2055, Forward time: 1.2626, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9118 [2021-07-07 20:09:07,278][root][INFO] - ===> Epoch[2511](12555/5): Loss 0.0153, Sem 0.0153 LR: 8.095e-02 Score 99.452 Data time: 0.2425, Forward time: 1.2626, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7326 [2021-07-07 20:09:21,180][root][INFO] - ===> Epoch[2512](12560/5): Loss 0.0154, Sem 0.0154 LR: 8.095e-02 Score 99.448 Data time: 0.1963, Forward time: 1.2626, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7425 [2021-07-07 20:09:34,889][root][INFO] - ===> Epoch[2513](12565/5): Loss 0.0152, Sem 0.0152 LR: 8.094e-02 Score 99.459 Data time: 0.2482, Forward time: 1.2627, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6927 [2021-07-07 20:09:49,149][root][INFO] - ===> Epoch[2514](12570/5): Loss 0.0158, Sem 0.0158 LR: 8.093e-02 Score 99.432 Data time: 0.2415, Forward time: 1.2626, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8080 [2021-07-07 20:10:02,912][root][INFO] - ===> Epoch[2515](12575/5): Loss 0.0147, Sem 0.0147 LR: 8.092e-02 Score 99.468 Data time: 0.2690, Forward time: 1.2626, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6941 [2021-07-07 20:10:16,303][root][INFO] - ===> Epoch[2516](12580/5): Loss 0.0145, Sem 0.0145 LR: 8.092e-02 Score 99.479 Data time: 0.2534, Forward time: 1.2626, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6259 [2021-07-07 20:10:30,019][root][INFO] - ===> Epoch[2517](12585/5): Loss 0.0147, Sem 0.0147 LR: 8.091e-02 Score 99.467 Data time: 0.2320, Forward time: 1.2627, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6902 [2021-07-07 20:10:44,487][root][INFO] - ===> Epoch[2518](12590/5): Loss 0.0151, Sem 0.0151 LR: 8.090e-02 Score 99.463 Data time: 0.2488, Forward time: 1.2626, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8494 [2021-07-07 20:10:58,279][root][INFO] - ===> Epoch[2519](12595/5): Loss 0.0146, Sem 0.0146 LR: 8.089e-02 Score 99.467 Data time: 0.1788, Forward time: 1.2627, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7167 [2021-07-07 20:11:11,956][root][INFO] - ===> Epoch[2520](12600/5): Loss 0.0139, Sem 0.0139 LR: 8.088e-02 Score 99.494 Data time: 0.2536, Forward time: 1.2626, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6778 [2021-07-07 20:11:11,957][root][INFO] - ===> Start testing [2021-07-07 20:11:46,955][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1252 Loss 0.299 (AVG: 0.516) Score 93.090 (AVG: 91.326) mIOU 70.923 mAP 84.052 mAcc 77.129 IOU: 64.478 0.000 83.139 77.078 95.476 92.086 29.043 81.857 98.581 72.530 79.867 87.548 60.312 mAP: 84.227 7.588 94.695 88.735 98.918 98.671 59.164 97.776 99.822 83.894 92.562 94.982 91.640 mAcc: 79.442 0.000 89.219 84.720 96.828 97.314 31.906 96.006 99.307 76.559 90.118 96.507 64.748 [2021-07-07 20:11:46,956][root][INFO] - Finished test. Elapsed time: 34.9982 [2021-07-07 20:11:48,007][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 20:12:00,888][root][INFO] - ===> Epoch[2521](12605/5): Loss 0.0144, Sem 0.0144 LR: 8.088e-02 Score 99.473 Data time: 0.1817, Forward time: 1.2626, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.5316 [2021-07-07 20:12:15,783][root][INFO] - ===> Epoch[2522](12610/5): Loss 0.0143, Sem 0.0143 LR: 8.087e-02 Score 99.485 Data time: 0.2507, Forward time: 1.2626, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.9300 [2021-07-07 20:12:30,481][root][INFO] - ===> Epoch[2523](12615/5): Loss 0.0153, Sem 0.0153 LR: 8.086e-02 Score 99.453 Data time: 0.2818, Forward time: 1.2626, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.9020 [2021-07-07 20:12:44,779][root][INFO] - ===> Epoch[2524](12620/5): Loss 0.0142, Sem 0.0142 LR: 8.085e-02 Score 99.490 Data time: 0.2094, Forward time: 1.2626, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8086 [2021-07-07 20:12:59,532][root][INFO] - ===> Epoch[2525](12625/5): Loss 0.0147, Sem 0.0147 LR: 8.085e-02 Score 99.464 Data time: 0.2386, Forward time: 1.2626, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9117 [2021-07-07 20:13:14,015][root][INFO] - ===> Epoch[2526](12630/5): Loss 0.0153, Sem 0.0153 LR: 8.084e-02 Score 99.447 Data time: 0.2405, Forward time: 1.2626, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8505 [2021-07-07 20:13:28,192][root][INFO] - ===> Epoch[2527](12635/5): Loss 0.0155, Sem 0.0155 LR: 8.083e-02 Score 99.443 Data time: 0.2459, Forward time: 1.2626, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7925 [2021-07-07 20:13:42,291][root][INFO] - ===> Epoch[2528](12640/5): Loss 0.0157, Sem 0.0157 LR: 8.082e-02 Score 99.433 Data time: 0.2811, Forward time: 1.2626, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7658 [2021-07-07 20:13:56,292][root][INFO] - ===> Epoch[2529](12645/5): Loss 0.0157, Sem 0.0157 LR: 8.082e-02 Score 99.434 Data time: 0.2088, Forward time: 1.2626, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7508 [2021-07-07 20:14:10,065][root][INFO] - ===> Epoch[2530](12650/5): Loss 0.0152, Sem 0.0152 LR: 8.081e-02 Score 99.456 Data time: 0.2742, Forward time: 1.2626, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7058 [2021-07-07 20:14:24,617][root][INFO] - ===> Epoch[2531](12655/5): Loss 0.0165, Sem 0.0165 LR: 8.080e-02 Score 99.406 Data time: 0.2450, Forward time: 1.2626, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8593 [2021-07-07 20:14:38,480][root][INFO] - ===> Epoch[2532](12660/5): Loss 0.0157, Sem 0.0157 LR: 8.079e-02 Score 99.433 Data time: 0.1703, Forward time: 1.2626, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7324 [2021-07-07 20:14:52,398][root][INFO] - ===> Epoch[2533](12665/5): Loss 0.0154, Sem 0.0154 LR: 8.078e-02 Score 99.440 Data time: 0.2156, Forward time: 1.2626, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7269 [2021-07-07 20:15:06,075][root][INFO] - ===> Epoch[2534](12670/5): Loss 0.0152, Sem 0.0152 LR: 8.078e-02 Score 99.451 Data time: 0.2218, Forward time: 1.2627, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6861 [2021-07-07 20:15:19,472][root][INFO] - ===> Epoch[2535](12675/5): Loss 0.0151, Sem 0.0151 LR: 8.077e-02 Score 99.460 Data time: 0.2044, Forward time: 1.2627, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6341 [2021-07-07 20:15:32,758][root][INFO] - ===> Epoch[2536](12680/5): Loss 0.0148, Sem 0.0148 LR: 8.076e-02 Score 99.459 Data time: 0.1811, Forward time: 1.2627, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6118 [2021-07-07 20:15:47,349][root][INFO] - ===> Epoch[2537](12685/5): Loss 0.0153, Sem 0.0153 LR: 8.075e-02 Score 99.446 Data time: 0.2453, Forward time: 1.2627, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8788 [2021-07-07 20:16:00,489][root][INFO] - ===> Epoch[2538](12690/5): Loss 0.0143, Sem 0.0143 LR: 8.075e-02 Score 99.490 Data time: 0.2551, Forward time: 1.2627, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.5784 [2021-07-07 20:16:15,490][root][INFO] - ===> Epoch[2539](12695/5): Loss 0.0153, Sem 0.0153 LR: 8.074e-02 Score 99.444 Data time: 0.2533, Forward time: 1.2627, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.9299 [2021-07-07 20:16:29,364][root][INFO] - ===> Epoch[2540](12700/5): Loss 0.0147, Sem 0.0147 LR: 8.073e-02 Score 99.468 Data time: 0.1932, Forward time: 1.2627, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7202 [2021-07-07 20:16:43,782][root][INFO] - ===> Epoch[2541](12705/5): Loss 0.0150, Sem 0.0150 LR: 8.072e-02 Score 99.459 Data time: 0.1721, Forward time: 1.2627, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8347 [2021-07-07 20:16:57,455][root][INFO] - ===> Epoch[2542](12710/5): Loss 0.0145, Sem 0.0145 LR: 8.072e-02 Score 99.480 Data time: 0.2579, Forward time: 1.2627, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6840 [2021-07-07 20:17:12,437][root][INFO] - ===> Epoch[2543](12715/5): Loss 0.0151, Sem 0.0151 LR: 8.071e-02 Score 99.458 Data time: 0.1972, Forward time: 1.2628, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.9496 [2021-07-07 20:17:26,733][root][INFO] - ===> Epoch[2544](12720/5): Loss 0.0148, Sem 0.0148 LR: 8.070e-02 Score 99.465 Data time: 0.2289, Forward time: 1.2628, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8138 [2021-07-07 20:17:40,652][root][INFO] - ===> Epoch[2545](12725/5): Loss 0.0149, Sem 0.0149 LR: 8.069e-02 Score 99.463 Data time: 0.2268, Forward time: 1.2628, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7380 [2021-07-07 20:17:53,884][root][INFO] - ===> Epoch[2546](12730/5): Loss 0.0151, Sem 0.0151 LR: 8.068e-02 Score 99.457 Data time: 0.2740, Forward time: 1.2628, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6002 [2021-07-07 20:18:08,170][root][INFO] - ===> Epoch[2547](12735/5): Loss 0.0143, Sem 0.0143 LR: 8.068e-02 Score 99.490 Data time: 0.1904, Forward time: 1.2627, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8182 [2021-07-07 20:18:21,820][root][INFO] - ===> Epoch[2548](12740/5): Loss 0.0144, Sem 0.0144 LR: 8.067e-02 Score 99.478 Data time: 0.2346, Forward time: 1.2627, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6891 [2021-07-07 20:18:35,680][root][INFO] - ===> Epoch[2549](12745/5): Loss 0.0143, Sem 0.0143 LR: 8.066e-02 Score 99.483 Data time: 0.2320, Forward time: 1.2627, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7251 [2021-07-07 20:18:48,505][root][INFO] - ===> Epoch[2550](12750/5): Loss 0.0148, Sem 0.0148 LR: 8.065e-02 Score 99.476 Data time: 0.2514, Forward time: 1.2627, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.5126 [2021-07-07 20:19:02,305][root][INFO] - ===> Epoch[2551](12755/5): Loss 0.0140, Sem 0.0140 LR: 8.065e-02 Score 99.492 Data time: 0.2309, Forward time: 1.2627, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7204 [2021-07-07 20:19:16,506][root][INFO] - ===> Epoch[2552](12760/5): Loss 0.0142, Sem 0.0142 LR: 8.064e-02 Score 99.486 Data time: 0.1940, Forward time: 1.2627, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7866 [2021-07-07 20:19:30,395][root][INFO] - ===> Epoch[2553](12765/5): Loss 0.0149, Sem 0.0149 LR: 8.063e-02 Score 99.465 Data time: 0.1601, Forward time: 1.2628, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7367 [2021-07-07 20:19:44,150][root][INFO] - ===> Epoch[2554](12770/5): Loss 0.0141, Sem 0.0141 LR: 8.062e-02 Score 99.489 Data time: 0.1949, Forward time: 1.2628, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.6943 [2021-07-07 20:19:57,871][root][INFO] - ===> Epoch[2555](12775/5): Loss 0.0146, Sem 0.0146 LR: 8.062e-02 Score 99.468 Data time: 0.2810, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6987 [2021-07-07 20:20:12,484][root][INFO] - ===> Epoch[2556](12780/5): Loss 0.0144, Sem 0.0144 LR: 8.061e-02 Score 99.480 Data time: 0.2499, Forward time: 1.2627, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.8834 [2021-07-07 20:20:25,454][root][INFO] - ===> Epoch[2557](12785/5): Loss 0.0146, Sem 0.0146 LR: 8.060e-02 Score 99.469 Data time: 0.2206, Forward time: 1.2627, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.5508 [2021-07-07 20:20:39,751][root][INFO] - ===> Epoch[2558](12790/5): Loss 0.0150, Sem 0.0150 LR: 8.059e-02 Score 99.455 Data time: 0.1879, Forward time: 1.2627, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.8225 [2021-07-07 20:20:54,655][root][INFO] - ===> Epoch[2559](12795/5): Loss 0.0147, Sem 0.0147 LR: 8.059e-02 Score 99.470 Data time: 0.2347, Forward time: 1.2627, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.9382 [2021-07-07 20:21:09,178][root][INFO] - ===> Epoch[2560](12800/5): Loss 0.0140, Sem 0.0140 LR: 8.058e-02 Score 99.496 Data time: 0.2242, Forward time: 1.2627, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8595 [2021-07-07 20:21:09,179][root][INFO] - ===> Start testing [2021-07-07 20:21:44,013][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1255 Loss 0.305 (AVG: 0.508) Score 92.887 (AVG: 91.400) mIOU 71.885 mAP 84.742 mAcc 77.914 IOU: 65.081 0.021 84.087 76.338 95.169 92.477 36.538 82.223 98.637 76.056 80.155 87.537 60.184 mAP: 84.207 7.212 92.487 88.825 98.871 98.765 63.086 98.002 99.802 90.714 92.519 95.236 91.924 mAcc: 81.914 0.182 88.468 83.278 96.234 96.943 40.299 96.088 99.441 80.101 89.076 96.507 64.346 [2021-07-07 20:21:44,013][root][INFO] - Finished test. Elapsed time: 34.8328 [2021-07-07 20:21:45,100][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 20:21:59,046][root][INFO] - ===> Epoch[2561](12805/5): Loss 0.0141, Sem 0.0141 LR: 8.057e-02 Score 99.486 Data time: 0.1996, Forward time: 1.2627, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7390 [2021-07-07 20:22:12,866][root][INFO] - ===> Epoch[2562](12810/5): Loss 0.0142, Sem 0.0142 LR: 8.056e-02 Score 99.486 Data time: 0.2715, Forward time: 1.2626, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7154 [2021-07-07 20:22:27,565][root][INFO] - ===> Epoch[2563](12815/5): Loss 0.0139, Sem 0.0139 LR: 8.055e-02 Score 99.496 Data time: 0.2521, Forward time: 1.2627, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8899 [2021-07-07 20:22:40,981][root][INFO] - ===> Epoch[2564](12820/5): Loss 0.0147, Sem 0.0147 LR: 8.055e-02 Score 99.461 Data time: 0.2659, Forward time: 1.2626, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6092 [2021-07-07 20:22:55,061][root][INFO] - ===> Epoch[2565](12825/5): Loss 0.0141, Sem 0.0141 LR: 8.054e-02 Score 99.488 Data time: 0.1999, Forward time: 1.2626, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7732 [2021-07-07 20:23:08,097][root][INFO] - ===> Epoch[2566](12830/5): Loss 0.0135, Sem 0.0135 LR: 8.053e-02 Score 99.513 Data time: 0.2108, Forward time: 1.2626, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.5648 [2021-07-07 20:23:22,102][root][INFO] - ===> Epoch[2567](12835/5): Loss 0.0141, Sem 0.0141 LR: 8.052e-02 Score 99.489 Data time: 0.1799, Forward time: 1.2626, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7490 [2021-07-07 20:23:35,540][root][INFO] - ===> Epoch[2568](12840/5): Loss 0.0138, Sem 0.0138 LR: 8.052e-02 Score 99.499 Data time: 0.2730, Forward time: 1.2626, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.6392 [2021-07-07 20:23:49,592][root][INFO] - ===> Epoch[2569](12845/5): Loss 0.0136, Sem 0.0136 LR: 8.051e-02 Score 99.507 Data time: 0.2402, Forward time: 1.2627, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7657 [2021-07-07 20:24:03,710][root][INFO] - ===> Epoch[2570](12850/5): Loss 0.0139, Sem 0.0139 LR: 8.050e-02 Score 99.500 Data time: 0.2068, Forward time: 1.2626, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7815 [2021-07-07 20:24:18,165][root][INFO] - ===> Epoch[2571](12855/5): Loss 0.0133, Sem 0.0133 LR: 8.049e-02 Score 99.520 Data time: 0.2097, Forward time: 1.2627, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.8434 [2021-07-07 20:24:32,095][root][INFO] - ===> Epoch[2572](12860/5): Loss 0.0137, Sem 0.0137 LR: 8.049e-02 Score 99.514 Data time: 0.2025, Forward time: 1.2627, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7362 [2021-07-07 20:24:46,849][root][INFO] - ===> Epoch[2573](12865/5): Loss 0.0132, Sem 0.0132 LR: 8.048e-02 Score 99.518 Data time: 0.2554, Forward time: 1.2627, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.8904 [2021-07-07 20:25:00,670][root][INFO] - ===> Epoch[2574](12870/5): Loss 0.0138, Sem 0.0138 LR: 8.047e-02 Score 99.498 Data time: 0.2641, Forward time: 1.2627, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7210 [2021-07-07 20:25:15,049][root][INFO] - ===> Epoch[2575](12875/5): Loss 0.0138, Sem 0.0138 LR: 8.046e-02 Score 99.499 Data time: 0.2985, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8275 [2021-07-07 20:25:29,360][root][INFO] - ===> Epoch[2576](12880/5): Loss 0.0138, Sem 0.0138 LR: 8.045e-02 Score 99.500 Data time: 0.1824, Forward time: 1.2627, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.8225 [2021-07-07 20:25:42,602][root][INFO] - ===> Epoch[2577](12885/5): Loss 0.0141, Sem 0.0141 LR: 8.045e-02 Score 99.489 Data time: 0.2180, Forward time: 1.2627, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.6043 [2021-07-07 20:25:56,881][root][INFO] - ===> Epoch[2578](12890/5): Loss 0.0138, Sem 0.0138 LR: 8.044e-02 Score 99.498 Data time: 0.2143, Forward time: 1.2627, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.8020 [2021-07-07 20:26:10,541][root][INFO] - ===> Epoch[2579](12895/5): Loss 0.0140, Sem 0.0140 LR: 8.043e-02 Score 99.494 Data time: 0.2301, Forward time: 1.2627, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6951 [2021-07-07 20:26:24,282][root][INFO] - ===> Epoch[2580](12900/5): Loss 0.0136, Sem 0.0136 LR: 8.042e-02 Score 99.504 Data time: 0.2120, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7083 [2021-07-07 20:26:37,935][root][INFO] - ===> Epoch[2581](12905/5): Loss 0.0139, Sem 0.0139 LR: 8.042e-02 Score 99.494 Data time: 0.2197, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6753 [2021-07-07 20:26:52,353][root][INFO] - ===> Epoch[2582](12910/5): Loss 0.0136, Sem 0.0136 LR: 8.041e-02 Score 99.506 Data time: 0.2230, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8409 [2021-07-07 20:27:06,403][root][INFO] - ===> Epoch[2583](12915/5): Loss 0.0143, Sem 0.0143 LR: 8.040e-02 Score 99.486 Data time: 0.2595, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7581 [2021-07-07 20:27:19,906][root][INFO] - ===> Epoch[2584](12920/5): Loss 0.0134, Sem 0.0134 LR: 8.039e-02 Score 99.514 Data time: 0.2287, Forward time: 1.2628, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6544 [2021-07-07 20:27:34,105][root][INFO] - ===> Epoch[2585](12925/5): Loss 0.0144, Sem 0.0144 LR: 8.039e-02 Score 99.489 Data time: 0.2057, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7985 [2021-07-07 20:27:47,723][root][INFO] - ===> Epoch[2586](12930/5): Loss 0.0140, Sem 0.0140 LR: 8.038e-02 Score 99.499 Data time: 0.1982, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6733 [2021-07-07 20:28:02,300][root][INFO] - ===> Epoch[2587](12935/5): Loss 0.0134, Sem 0.0134 LR: 8.037e-02 Score 99.516 Data time: 0.2042, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8691 [2021-07-07 20:28:16,314][root][INFO] - ===> Epoch[2588](12940/5): Loss 0.0137, Sem 0.0137 LR: 8.036e-02 Score 99.505 Data time: 0.2247, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7612 [2021-07-07 20:28:29,680][root][INFO] - ===> Epoch[2589](12945/5): Loss 0.0139, Sem 0.0139 LR: 8.035e-02 Score 99.492 Data time: 0.1978, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6348 [2021-07-07 20:28:44,084][root][INFO] - ===> Epoch[2590](12950/5): Loss 0.0131, Sem 0.0131 LR: 8.035e-02 Score 99.526 Data time: 0.2981, Forward time: 1.2628, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8403 [2021-07-07 20:28:58,128][root][INFO] - ===> Epoch[2591](12955/5): Loss 0.0134, Sem 0.0134 LR: 8.034e-02 Score 99.514 Data time: 0.2061, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7655 [2021-07-07 20:29:12,030][root][INFO] - ===> Epoch[2592](12960/5): Loss 0.0141, Sem 0.0141 LR: 8.033e-02 Score 99.488 Data time: 0.2549, Forward time: 1.2628, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7335 [2021-07-07 20:29:26,215][root][INFO] - ===> Epoch[2593](12965/5): Loss 0.0128, Sem 0.0128 LR: 8.032e-02 Score 99.533 Data time: 0.2426, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7818 [2021-07-07 20:29:40,611][root][INFO] - ===> Epoch[2594](12970/5): Loss 0.0132, Sem 0.0132 LR: 8.032e-02 Score 99.519 Data time: 0.2235, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8263 [2021-07-07 20:29:55,083][root][INFO] - ===> Epoch[2595](12975/5): Loss 0.0129, Sem 0.0129 LR: 8.031e-02 Score 99.532 Data time: 0.2432, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8492 [2021-07-07 20:30:08,976][root][INFO] - ===> Epoch[2596](12980/5): Loss 0.0130, Sem 0.0130 LR: 8.030e-02 Score 99.529 Data time: 0.2257, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7282 [2021-07-07 20:30:23,158][root][INFO] - ===> Epoch[2597](12985/5): Loss 0.0129, Sem 0.0129 LR: 8.029e-02 Score 99.534 Data time: 0.1830, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7839 [2021-07-07 20:30:37,074][root][INFO] - ===> Epoch[2598](12990/5): Loss 0.0134, Sem 0.0134 LR: 8.029e-02 Score 99.511 Data time: 0.2028, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7426 [2021-07-07 20:30:51,010][root][INFO] - ===> Epoch[2599](12995/5): Loss 0.0131, Sem 0.0131 LR: 8.028e-02 Score 99.524 Data time: 0.2811, Forward time: 1.2629, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7310 [2021-07-07 20:31:05,129][root][INFO] - ===> Epoch[2600](13000/5): Loss 0.0136, Sem 0.0136 LR: 8.027e-02 Score 99.511 Data time: 0.2793, Forward time: 1.2629, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7698 [2021-07-07 20:31:05,131][root][INFO] - ===> Start testing [2021-07-07 20:31:40,504][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1282 Loss 0.255 (AVG: 0.511) Score 93.979 (AVG: 91.475) mIOU 71.587 mAP 84.317 mAcc 77.517 IOU: 65.123 0.048 83.611 76.376 95.243 92.943 38.482 82.995 98.699 69.683 79.538 87.748 60.143 mAP: 84.449 9.783 92.600 89.264 99.039 98.831 63.844 98.249 99.758 82.001 92.281 94.681 91.338 mAcc: 81.228 0.456 89.515 82.971 96.426 97.196 42.282 96.683 99.586 71.024 89.553 96.750 64.057 [2021-07-07 20:31:40,504][root][INFO] - Finished test. Elapsed time: 35.3720 [2021-07-07 20:31:41,544][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 20:31:54,433][root][INFO] - ===> Epoch[2601](13005/5): Loss 0.0134, Sem 0.0134 LR: 8.026e-02 Score 99.515 Data time: 0.2101, Forward time: 1.2628, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.5464 [2021-07-07 20:32:08,120][root][INFO] - ===> Epoch[2602](13010/5): Loss 0.0132, Sem 0.0132 LR: 8.025e-02 Score 99.518 Data time: 0.2970, Forward time: 1.2628, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6885 [2021-07-07 20:32:22,443][root][INFO] - ===> Epoch[2603](13015/5): Loss 0.0134, Sem 0.0134 LR: 8.025e-02 Score 99.513 Data time: 0.2183, Forward time: 1.2628, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8138 [2021-07-07 20:32:35,924][root][INFO] - ===> Epoch[2604](13020/5): Loss 0.0129, Sem 0.0129 LR: 8.024e-02 Score 99.532 Data time: 0.2841, Forward time: 1.2627, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6467 [2021-07-07 20:32:50,531][root][INFO] - ===> Epoch[2605](13025/5): Loss 0.0131, Sem 0.0131 LR: 8.023e-02 Score 99.524 Data time: 0.2304, Forward time: 1.2628, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8711 [2021-07-07 20:33:05,477][root][INFO] - ===> Epoch[2606](13030/5): Loss 0.0139, Sem 0.0139 LR: 8.022e-02 Score 99.499 Data time: 0.3006, Forward time: 1.2628, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.9395 [2021-07-07 20:33:19,014][root][INFO] - ===> Epoch[2607](13035/5): Loss 0.0139, Sem 0.0139 LR: 8.022e-02 Score 99.500 Data time: 0.2206, Forward time: 1.2629, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6582 [2021-07-07 20:33:32,707][root][INFO] - ===> Epoch[2608](13040/5): Loss 0.0134, Sem 0.0134 LR: 8.021e-02 Score 99.516 Data time: 0.2517, Forward time: 1.2628, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6901 [2021-07-07 20:33:47,539][root][INFO] - ===> Epoch[2609](13045/5): Loss 0.0137, Sem 0.0137 LR: 8.020e-02 Score 99.509 Data time: 0.2354, Forward time: 1.2629, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.9221 [2021-07-07 20:34:01,136][root][INFO] - ===> Epoch[2610](13050/5): Loss 0.0133, Sem 0.0133 LR: 8.019e-02 Score 99.520 Data time: 0.2445, Forward time: 1.2629, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6765 [2021-07-07 20:34:15,091][root][INFO] - ===> Epoch[2611](13055/5): Loss 0.0133, Sem 0.0133 LR: 8.019e-02 Score 99.520 Data time: 0.2157, Forward time: 1.2629, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7499 [2021-07-07 20:34:29,053][root][INFO] - ===> Epoch[2612](13060/5): Loss 0.0138, Sem 0.0138 LR: 8.018e-02 Score 99.501 Data time: 0.2262, Forward time: 1.2629, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7548 [2021-07-07 20:34:42,714][root][INFO] - ===> Epoch[2613](13065/5): Loss 0.0139, Sem 0.0139 LR: 8.017e-02 Score 99.497 Data time: 0.2178, Forward time: 1.2629, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.6903 [2021-07-07 20:34:56,164][root][INFO] - ===> Epoch[2614](13070/5): Loss 0.0140, Sem 0.0140 LR: 8.016e-02 Score 99.494 Data time: 0.2113, Forward time: 1.2629, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.6431 [2021-07-07 20:35:09,853][root][INFO] - ===> Epoch[2615](13075/5): Loss 0.0139, Sem 0.0139 LR: 8.015e-02 Score 99.501 Data time: 0.2271, Forward time: 1.2629, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.6945 [2021-07-07 20:35:23,774][root][INFO] - ===> Epoch[2616](13080/5): Loss 0.0134, Sem 0.0134 LR: 8.015e-02 Score 99.519 Data time: 0.2301, Forward time: 1.2629, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7391 [2021-07-07 20:35:36,743][root][INFO] - ===> Epoch[2617](13085/5): Loss 0.0143, Sem 0.0143 LR: 8.014e-02 Score 99.478 Data time: 0.1722, Forward time: 1.2629, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.5565 [2021-07-07 20:35:51,132][root][INFO] - ===> Epoch[2618](13090/5): Loss 0.0142, Sem 0.0142 LR: 8.013e-02 Score 99.495 Data time: 0.2363, Forward time: 1.2629, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8205 [2021-07-07 20:36:05,256][root][INFO] - ===> Epoch[2619](13095/5): Loss 0.0143, Sem 0.0143 LR: 8.012e-02 Score 99.485 Data time: 0.1813, Forward time: 1.2629, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7773 [2021-07-07 20:36:18,967][root][INFO] - ===> Epoch[2620](13100/5): Loss 0.0145, Sem 0.0145 LR: 8.012e-02 Score 99.480 Data time: 0.2417, Forward time: 1.2629, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.6871 [2021-07-07 20:36:32,915][root][INFO] - ===> Epoch[2621](13105/5): Loss 0.0137, Sem 0.0137 LR: 8.011e-02 Score 99.507 Data time: 0.2260, Forward time: 1.2629, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7438 [2021-07-07 20:36:46,580][root][INFO] - ===> Epoch[2622](13110/5): Loss 0.0141, Sem 0.0141 LR: 8.010e-02 Score 99.489 Data time: 0.2366, Forward time: 1.2629, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.6829 [2021-07-07 20:37:00,615][root][INFO] - ===> Epoch[2623](13115/5): Loss 0.0134, Sem 0.0134 LR: 8.009e-02 Score 99.523 Data time: 0.1811, Forward time: 1.2629, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7524 [2021-07-07 20:37:15,483][root][INFO] - ===> Epoch[2624](13120/5): Loss 0.0139, Sem 0.0139 LR: 8.009e-02 Score 99.501 Data time: 0.2748, Forward time: 1.2629, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.9268 [2021-07-07 20:37:30,608][root][INFO] - ===> Epoch[2625](13125/5): Loss 0.0132, Sem 0.0132 LR: 8.008e-02 Score 99.522 Data time: 0.3217, Forward time: 1.2630, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.9595 [2021-07-07 20:37:45,085][root][INFO] - ===> Epoch[2626](13130/5): Loss 0.0134, Sem 0.0134 LR: 8.007e-02 Score 99.512 Data time: 0.2297, Forward time: 1.2630, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8482 [2021-07-07 20:37:59,694][root][INFO] - ===> Epoch[2627](13135/5): Loss 0.0134, Sem 0.0134 LR: 8.006e-02 Score 99.521 Data time: 0.3142, Forward time: 1.2630, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8730 [2021-07-07 20:38:13,591][root][INFO] - ===> Epoch[2628](13140/5): Loss 0.0135, Sem 0.0135 LR: 8.005e-02 Score 99.511 Data time: 0.2311, Forward time: 1.2631, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7451 [2021-07-07 20:38:28,195][root][INFO] - ===> Epoch[2629](13145/5): Loss 0.0131, Sem 0.0131 LR: 8.005e-02 Score 99.524 Data time: 0.1969, Forward time: 1.2631, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8779 [2021-07-07 20:38:42,313][root][INFO] - ===> Epoch[2630](13150/5): Loss 0.0134, Sem 0.0134 LR: 8.004e-02 Score 99.512 Data time: 0.2798, Forward time: 1.2631, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7774 [2021-07-07 20:38:56,051][root][INFO] - ===> Epoch[2631](13155/5): Loss 0.0136, Sem 0.0136 LR: 8.003e-02 Score 99.506 Data time: 0.2262, Forward time: 1.2631, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.6985 [2021-07-07 20:39:09,375][root][INFO] - ===> Epoch[2632](13160/5): Loss 0.0138, Sem 0.0138 LR: 8.002e-02 Score 99.497 Data time: 0.2707, Forward time: 1.2631, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6203 [2021-07-07 20:39:23,571][root][INFO] - ===> Epoch[2633](13165/5): Loss 0.0131, Sem 0.0131 LR: 8.002e-02 Score 99.526 Data time: 0.2187, Forward time: 1.2631, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7687 [2021-07-07 20:39:37,128][root][INFO] - ===> Epoch[2634](13170/5): Loss 0.0130, Sem 0.0130 LR: 8.001e-02 Score 99.530 Data time: 0.2180, Forward time: 1.2631, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6637 [2021-07-07 20:39:51,087][root][INFO] - ===> Epoch[2635](13175/5): Loss 0.0132, Sem 0.0132 LR: 8.000e-02 Score 99.523 Data time: 0.2407, Forward time: 1.2631, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7413 [2021-07-07 20:40:05,782][root][INFO] - ===> Epoch[2636](13180/5): Loss 0.0130, Sem 0.0130 LR: 7.999e-02 Score 99.529 Data time: 0.2742, Forward time: 1.2631, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8899 [2021-07-07 20:40:19,951][root][INFO] - ===> Epoch[2637](13185/5): Loss 0.0131, Sem 0.0131 LR: 7.999e-02 Score 99.529 Data time: 0.2155, Forward time: 1.2630, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7772 [2021-07-07 20:40:33,894][root][INFO] - ===> Epoch[2638](13190/5): Loss 0.0129, Sem 0.0129 LR: 7.998e-02 Score 99.528 Data time: 0.1917, Forward time: 1.2630, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7501 [2021-07-07 20:40:48,803][root][INFO] - ===> Epoch[2639](13195/5): Loss 0.0128, Sem 0.0128 LR: 7.997e-02 Score 99.537 Data time: 0.2087, Forward time: 1.2630, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.9309 [2021-07-07 20:41:02,217][root][INFO] - ===> Epoch[2640](13200/5): Loss 0.0138, Sem 0.0138 LR: 7.996e-02 Score 99.504 Data time: 0.2728, Forward time: 1.2630, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.6340 [2021-07-07 20:41:02,219][root][INFO] - ===> Start testing [2021-07-07 20:41:39,520][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1213 Loss 0.285 (AVG: 0.544) Score 93.703 (AVG: 91.202) mIOU 70.943 mAP 84.093 mAcc 77.053 IOU: 64.676 0.021 83.154 76.248 94.774 92.635 33.605 82.916 98.575 70.513 80.035 87.091 58.010 mAP: 84.002 6.258 93.813 88.053 98.896 98.816 61.314 98.422 99.833 86.818 92.007 94.396 90.577 mAcc: 81.583 0.182 89.942 82.639 95.993 97.310 36.096 97.133 99.316 73.727 90.099 96.841 60.826 [2021-07-07 20:41:39,520][root][INFO] - Finished test. Elapsed time: 37.3008 [2021-07-07 20:41:40,561][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 20:41:53,017][root][INFO] - ===> Epoch[2641](13205/5): Loss 0.0133, Sem 0.0133 LR: 7.995e-02 Score 99.518 Data time: 0.2105, Forward time: 1.2630, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.4510 [2021-07-07 20:42:06,075][root][INFO] - ===> Epoch[2642](13210/5): Loss 0.0136, Sem 0.0136 LR: 7.995e-02 Score 99.509 Data time: 0.2414, Forward time: 1.2629, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.5636 [2021-07-07 20:42:19,624][root][INFO] - ===> Epoch[2643](13215/5): Loss 0.0137, Sem 0.0137 LR: 7.994e-02 Score 99.505 Data time: 0.2266, Forward time: 1.2629, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6624 [2021-07-07 20:42:33,530][root][INFO] - ===> Epoch[2644](13220/5): Loss 0.0131, Sem 0.0131 LR: 7.993e-02 Score 99.527 Data time: 0.2385, Forward time: 1.2629, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7368 [2021-07-07 20:42:47,322][root][INFO] - ===> Epoch[2645](13225/5): Loss 0.0130, Sem 0.0130 LR: 7.992e-02 Score 99.530 Data time: 0.2332, Forward time: 1.2629, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7189 [2021-07-07 20:43:01,198][root][INFO] - ===> Epoch[2646](13230/5): Loss 0.0138, Sem 0.0138 LR: 7.992e-02 Score 99.503 Data time: 0.2586, Forward time: 1.2630, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7363 [2021-07-07 20:43:15,335][root][INFO] - ===> Epoch[2647](13235/5): Loss 0.0137, Sem 0.0137 LR: 7.991e-02 Score 99.507 Data time: 0.1698, Forward time: 1.2629, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7880 [2021-07-07 20:43:29,006][root][INFO] - ===> Epoch[2648](13240/5): Loss 0.0134, Sem 0.0134 LR: 7.990e-02 Score 99.516 Data time: 0.2203, Forward time: 1.2629, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6892 [2021-07-07 20:43:43,481][root][INFO] - ===> Epoch[2649](13245/5): Loss 0.0132, Sem 0.0132 LR: 7.989e-02 Score 99.520 Data time: 0.2487, Forward time: 1.2629, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8336 [2021-07-07 20:43:56,631][root][INFO] - ===> Epoch[2650](13250/5): Loss 0.0135, Sem 0.0135 LR: 7.989e-02 Score 99.508 Data time: 0.2165, Forward time: 1.2629, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.5847 [2021-07-07 20:44:10,398][root][INFO] - ===> Epoch[2651](13255/5): Loss 0.0134, Sem 0.0134 LR: 7.988e-02 Score 99.518 Data time: 0.2369, Forward time: 1.2629, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7063 [2021-07-07 20:44:24,998][root][INFO] - ===> Epoch[2652](13260/5): Loss 0.0133, Sem 0.0133 LR: 7.987e-02 Score 99.514 Data time: 0.2148, Forward time: 1.2629, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8851 [2021-07-07 20:44:39,782][root][INFO] - ===> Epoch[2653](13265/5): Loss 0.0131, Sem 0.0131 LR: 7.986e-02 Score 99.529 Data time: 0.2206, Forward time: 1.2629, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.9031 [2021-07-07 20:44:54,124][root][INFO] - ===> Epoch[2654](13270/5): Loss 0.0134, Sem 0.0134 LR: 7.985e-02 Score 99.517 Data time: 0.2395, Forward time: 1.2629, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8227 [2021-07-07 20:45:08,782][root][INFO] - ===> Epoch[2655](13275/5): Loss 0.0127, Sem 0.0127 LR: 7.985e-02 Score 99.543 Data time: 0.2438, Forward time: 1.2629, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8932 [2021-07-07 20:45:23,430][root][INFO] - ===> Epoch[2656](13280/5): Loss 0.0135, Sem 0.0135 LR: 7.984e-02 Score 99.510 Data time: 0.2935, Forward time: 1.2629, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8745 [2021-07-07 20:45:37,306][root][INFO] - ===> Epoch[2657](13285/5): Loss 0.0133, Sem 0.0133 LR: 7.983e-02 Score 99.518 Data time: 0.2178, Forward time: 1.2629, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7333 [2021-07-07 20:45:51,868][root][INFO] - ===> Epoch[2658](13290/5): Loss 0.0131, Sem 0.0131 LR: 7.982e-02 Score 99.528 Data time: 0.2302, Forward time: 1.2630, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8537 [2021-07-07 20:46:05,060][root][INFO] - ===> Epoch[2659](13295/5): Loss 0.0130, Sem 0.0130 LR: 7.982e-02 Score 99.529 Data time: 0.2541, Forward time: 1.2630, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.5937 [2021-07-07 20:46:19,242][root][INFO] - ===> Epoch[2660](13300/5): Loss 0.0129, Sem 0.0129 LR: 7.981e-02 Score 99.530 Data time: 0.2120, Forward time: 1.2630, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7967 [2021-07-07 20:46:33,942][root][INFO] - ===> Epoch[2661](13305/5): Loss 0.0133, Sem 0.0133 LR: 7.980e-02 Score 99.514 Data time: 0.1866, Forward time: 1.2630, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.9010 [2021-07-07 20:46:47,306][root][INFO] - ===> Epoch[2662](13310/5): Loss 0.0129, Sem 0.0129 LR: 7.979e-02 Score 99.537 Data time: 0.2459, Forward time: 1.2630, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6240 [2021-07-07 20:47:01,684][root][INFO] - ===> Epoch[2663](13315/5): Loss 0.0125, Sem 0.0125 LR: 7.979e-02 Score 99.549 Data time: 0.2496, Forward time: 1.2630, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8254 [2021-07-07 20:47:16,383][root][INFO] - ===> Epoch[2664](13320/5): Loss 0.0125, Sem 0.0125 LR: 7.978e-02 Score 99.550 Data time: 0.2257, Forward time: 1.2631, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.9012 [2021-07-07 20:47:30,230][root][INFO] - ===> Epoch[2665](13325/5): Loss 0.0133, Sem 0.0133 LR: 7.977e-02 Score 99.521 Data time: 0.2285, Forward time: 1.2631, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7283 [2021-07-07 20:47:44,407][root][INFO] - ===> Epoch[2666](13330/5): Loss 0.0137, Sem 0.0137 LR: 7.976e-02 Score 99.509 Data time: 0.2688, Forward time: 1.2630, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7721 [2021-07-07 20:47:58,836][root][INFO] - ===> Epoch[2667](13335/5): Loss 0.0131, Sem 0.0131 LR: 7.975e-02 Score 99.521 Data time: 0.2213, Forward time: 1.2630, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8383 [2021-07-07 20:48:13,293][root][INFO] - ===> Epoch[2668](13340/5): Loss 0.0124, Sem 0.0124 LR: 7.975e-02 Score 99.549 Data time: 0.2058, Forward time: 1.2630, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8394 [2021-07-07 20:48:26,775][root][INFO] - ===> Epoch[2669](13345/5): Loss 0.0135, Sem 0.0135 LR: 7.974e-02 Score 99.511 Data time: 0.2028, Forward time: 1.2630, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.6434 [2021-07-07 20:48:41,238][root][INFO] - ===> Epoch[2670](13350/5): Loss 0.0128, Sem 0.0128 LR: 7.973e-02 Score 99.536 Data time: 0.1336, Forward time: 1.2629, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8512 [2021-07-07 20:48:54,477][root][INFO] - ===> Epoch[2671](13355/5): Loss 0.0141, Sem 0.0141 LR: 7.972e-02 Score 99.488 Data time: 0.2154, Forward time: 1.2629, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.5912 [2021-07-07 20:49:09,135][root][INFO] - ===> Epoch[2672](13360/5): Loss 0.0131, Sem 0.0131 LR: 7.972e-02 Score 99.521 Data time: 0.2227, Forward time: 1.2628, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8852 [2021-07-07 20:49:23,448][root][INFO] - ===> Epoch[2673](13365/5): Loss 0.0131, Sem 0.0131 LR: 7.971e-02 Score 99.524 Data time: 0.2788, Forward time: 1.2628, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8060 [2021-07-07 20:49:37,089][root][INFO] - ===> Epoch[2674](13370/5): Loss 0.0130, Sem 0.0130 LR: 7.970e-02 Score 99.527 Data time: 0.2534, Forward time: 1.2628, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6802 [2021-07-07 20:49:50,266][root][INFO] - ===> Epoch[2675](13375/5): Loss 0.0123, Sem 0.0123 LR: 7.969e-02 Score 99.557 Data time: 0.2051, Forward time: 1.2628, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.5812 [2021-07-07 20:50:05,336][root][INFO] - ===> Epoch[2676](13380/5): Loss 0.0135, Sem 0.0135 LR: 7.969e-02 Score 99.513 Data time: 0.2359, Forward time: 1.2628, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.9540 [2021-07-07 20:50:19,445][root][INFO] - ===> Epoch[2677](13385/5): Loss 0.0126, Sem 0.0126 LR: 7.968e-02 Score 99.544 Data time: 0.2484, Forward time: 1.2628, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7735 [2021-07-07 20:50:33,682][root][INFO] - ===> Epoch[2678](13390/5): Loss 0.0128, Sem 0.0128 LR: 7.967e-02 Score 99.533 Data time: 0.2114, Forward time: 1.2628, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8063 [2021-07-07 20:50:47,612][root][INFO] - ===> Epoch[2679](13395/5): Loss 0.0133, Sem 0.0133 LR: 7.966e-02 Score 99.514 Data time: 0.1982, Forward time: 1.2627, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7424 [2021-07-07 20:51:01,393][root][INFO] - ===> Epoch[2680](13400/5): Loss 0.0131, Sem 0.0131 LR: 7.966e-02 Score 99.528 Data time: 0.2511, Forward time: 1.2627, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6962 [2021-07-07 20:51:01,395][root][INFO] - ===> Start testing [2021-07-07 20:51:36,998][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1204 Loss 0.270 (AVG: 0.543) Score 93.954 (AVG: 91.294) mIOU 70.898 mAP 83.773 mAcc 77.006 IOU: 65.499 0.149 84.135 76.642 95.113 92.556 25.060 82.993 98.685 73.520 80.212 87.130 59.972 mAP: 83.950 12.251 93.864 89.211 98.829 98.621 50.977 98.148 99.810 84.761 92.471 94.188 91.960 mAcc: 81.192 1.641 91.146 84.207 96.418 97.426 25.922 96.577 99.489 76.233 90.114 96.446 64.267 [2021-07-07 20:51:36,998][root][INFO] - Finished test. Elapsed time: 35.6017 [2021-07-07 20:51:38,177][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 20:51:51,346][root][INFO] - ===> Epoch[2681](13405/5): Loss 0.0140, Sem 0.0140 LR: 7.965e-02 Score 99.497 Data time: 0.2059, Forward time: 1.2627, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.5905 [2021-07-07 20:52:05,152][root][INFO] - ===> Epoch[2682](13410/5): Loss 0.0132, Sem 0.0132 LR: 7.964e-02 Score 99.524 Data time: 0.2367, Forward time: 1.2627, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7086 [2021-07-07 20:52:19,450][root][INFO] - ===> Epoch[2683](13415/5): Loss 0.0129, Sem 0.0129 LR: 7.963e-02 Score 99.528 Data time: 0.1921, Forward time: 1.2628, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8054 [2021-07-07 20:52:33,312][root][INFO] - ===> Epoch[2684](13420/5): Loss 0.0139, Sem 0.0139 LR: 7.962e-02 Score 99.506 Data time: 0.2087, Forward time: 1.2628, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7259 [2021-07-07 20:52:46,897][root][INFO] - ===> Epoch[2685](13425/5): Loss 0.0143, Sem 0.0143 LR: 7.962e-02 Score 99.483 Data time: 0.2763, Forward time: 1.2628, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.6580 [2021-07-07 20:53:01,305][root][INFO] - ===> Epoch[2686](13430/5): Loss 0.0133, Sem 0.0133 LR: 7.961e-02 Score 99.520 Data time: 0.2310, Forward time: 1.2628, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8359 [2021-07-07 20:53:15,519][root][INFO] - ===> Epoch[2687](13435/5): Loss 0.0133, Sem 0.0133 LR: 7.960e-02 Score 99.524 Data time: 0.2370, Forward time: 1.2628, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7895 [2021-07-07 20:53:30,558][root][INFO] - ===> Epoch[2688](13440/5): Loss 0.0128, Sem 0.0128 LR: 7.959e-02 Score 99.531 Data time: 0.2341, Forward time: 1.2628, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.9562 [2021-07-07 20:53:45,673][root][INFO] - ===> Epoch[2689](13445/5): Loss 0.0139, Sem 0.0139 LR: 7.959e-02 Score 99.495 Data time: 0.2705, Forward time: 1.2629, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.9756 [2021-07-07 20:53:59,802][root][INFO] - ===> Epoch[2690](13450/5): Loss 0.0136, Sem 0.0136 LR: 7.958e-02 Score 99.510 Data time: 0.2272, Forward time: 1.2628, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7776 [2021-07-07 20:54:14,308][root][INFO] - ===> Epoch[2691](13455/5): Loss 0.0138, Sem 0.0138 LR: 7.957e-02 Score 99.508 Data time: 0.2342, Forward time: 1.2629, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8567 [2021-07-07 20:54:28,848][root][INFO] - ===> Epoch[2692](13460/5): Loss 0.0134, Sem 0.0134 LR: 7.956e-02 Score 99.513 Data time: 0.2261, Forward time: 1.2628, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8646 [2021-07-07 20:54:43,501][root][INFO] - ===> Epoch[2693](13465/5): Loss 0.0138, Sem 0.0138 LR: 7.956e-02 Score 99.500 Data time: 0.2836, Forward time: 1.2628, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8669 [2021-07-07 20:54:57,412][root][INFO] - ===> Epoch[2694](13470/5): Loss 0.0140, Sem 0.0140 LR: 7.955e-02 Score 99.494 Data time: 0.2120, Forward time: 1.2629, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7395 [2021-07-07 20:55:11,567][root][INFO] - ===> Epoch[2695](13475/5): Loss 0.0129, Sem 0.0129 LR: 7.954e-02 Score 99.536 Data time: 0.2249, Forward time: 1.2628, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7893 [2021-07-07 20:55:25,309][root][INFO] - ===> Epoch[2696](13480/5): Loss 0.0134, Sem 0.0134 LR: 7.953e-02 Score 99.512 Data time: 0.3062, Forward time: 1.2628, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6879 [2021-07-07 20:55:40,328][root][INFO] - ===> Epoch[2697](13485/5): Loss 0.0126, Sem 0.0126 LR: 7.952e-02 Score 99.543 Data time: 0.2701, Forward time: 1.2628, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.9554 [2021-07-07 20:55:55,149][root][INFO] - ===> Epoch[2698](13490/5): Loss 0.0135, Sem 0.0135 LR: 7.952e-02 Score 99.510 Data time: 0.2353, Forward time: 1.2629, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.9194 [2021-07-07 20:56:09,969][root][INFO] - ===> Epoch[2699](13495/5): Loss 0.0130, Sem 0.0130 LR: 7.951e-02 Score 99.530 Data time: 0.2436, Forward time: 1.2629, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.9200 [2021-07-07 20:56:23,860][root][INFO] - ===> Epoch[2700](13500/5): Loss 0.0136, Sem 0.0136 LR: 7.950e-02 Score 99.515 Data time: 0.2490, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7307 [2021-07-07 20:56:37,954][root][INFO] - ===> Epoch[2701](13505/5): Loss 0.0131, Sem 0.0131 LR: 7.949e-02 Score 99.528 Data time: 0.1934, Forward time: 1.2628, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7642 [2021-07-07 20:56:52,817][root][INFO] - ===> Epoch[2702](13510/5): Loss 0.0137, Sem 0.0137 LR: 7.949e-02 Score 99.507 Data time: 0.2849, Forward time: 1.2629, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.9114 [2021-07-07 20:57:07,289][root][INFO] - ===> Epoch[2703](13515/5): Loss 0.0131, Sem 0.0131 LR: 7.948e-02 Score 99.528 Data time: 0.2518, Forward time: 1.2629, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8454 [2021-07-07 20:57:21,297][root][INFO] - ===> Epoch[2704](13520/5): Loss 0.0137, Sem 0.0137 LR: 7.947e-02 Score 99.507 Data time: 0.2264, Forward time: 1.2630, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7597 [2021-07-07 20:57:35,665][root][INFO] - ===> Epoch[2705](13525/5): Loss 0.0137, Sem 0.0137 LR: 7.946e-02 Score 99.507 Data time: 0.2670, Forward time: 1.2630, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8212 [2021-07-07 20:57:49,886][root][INFO] - ===> Epoch[2706](13530/5): Loss 0.0140, Sem 0.0140 LR: 7.945e-02 Score 99.499 Data time: 0.2767, Forward time: 1.2630, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7950 [2021-07-07 20:58:03,810][root][INFO] - ===> Epoch[2707](13535/5): Loss 0.0139, Sem 0.0139 LR: 7.945e-02 Score 99.503 Data time: 0.1608, Forward time: 1.2630, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7382 [2021-07-07 20:58:18,586][root][INFO] - ===> Epoch[2708](13540/5): Loss 0.0134, Sem 0.0134 LR: 7.944e-02 Score 99.515 Data time: 0.2381, Forward time: 1.2630, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.9013 [2021-07-07 20:58:33,478][root][INFO] - ===> Epoch[2709](13545/5): Loss 0.0136, Sem 0.0136 LR: 7.943e-02 Score 99.511 Data time: 0.1603, Forward time: 1.2630, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.9429 [2021-07-07 20:58:47,020][root][INFO] - ===> Epoch[2710](13550/5): Loss 0.0137, Sem 0.0137 LR: 7.942e-02 Score 99.506 Data time: 0.1829, Forward time: 1.2630, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.6600 [2021-07-07 20:59:01,588][root][INFO] - ===> Epoch[2711](13555/5): Loss 0.0130, Sem 0.0130 LR: 7.942e-02 Score 99.530 Data time: 0.2017, Forward time: 1.2629, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8657 [2021-07-07 20:59:15,324][root][INFO] - ===> Epoch[2712](13560/5): Loss 0.0138, Sem 0.0138 LR: 7.941e-02 Score 99.498 Data time: 0.2622, Forward time: 1.2629, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6991 [2021-07-07 20:59:29,050][root][INFO] - ===> Epoch[2713](13565/5): Loss 0.0149, Sem 0.0149 LR: 7.940e-02 Score 99.467 Data time: 0.2201, Forward time: 1.2629, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7006 [2021-07-07 20:59:43,862][root][INFO] - ===> Epoch[2714](13570/5): Loss 0.0137, Sem 0.0137 LR: 7.939e-02 Score 99.513 Data time: 0.1954, Forward time: 1.2629, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.9171 [2021-07-07 20:59:57,151][root][INFO] - ===> Epoch[2715](13575/5): Loss 0.0136, Sem 0.0136 LR: 7.939e-02 Score 99.506 Data time: 0.2485, Forward time: 1.2629, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6142 [2021-07-07 21:00:11,364][root][INFO] - ===> Epoch[2716](13580/5): Loss 0.0136, Sem 0.0136 LR: 7.938e-02 Score 99.505 Data time: 0.1821, Forward time: 1.2629, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7904 [2021-07-07 21:00:25,890][root][INFO] - ===> Epoch[2717](13585/5): Loss 0.0133, Sem 0.0133 LR: 7.937e-02 Score 99.522 Data time: 0.2068, Forward time: 1.2629, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8609 [2021-07-07 21:00:40,191][root][INFO] - ===> Epoch[2718](13590/5): Loss 0.0140, Sem 0.0140 LR: 7.936e-02 Score 99.495 Data time: 0.2456, Forward time: 1.2629, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7986 [2021-07-07 21:00:53,748][root][INFO] - ===> Epoch[2719](13595/5): Loss 0.0130, Sem 0.0130 LR: 7.935e-02 Score 99.529 Data time: 0.2042, Forward time: 1.2629, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6549 [2021-07-07 21:01:08,237][root][INFO] - ===> Epoch[2720](13600/5): Loss 0.0128, Sem 0.0128 LR: 7.935e-02 Score 99.532 Data time: 0.2198, Forward time: 1.2628, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8554 [2021-07-07 21:01:08,239][root][INFO] - ===> Start testing [2021-07-07 21:01:43,803][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1267 Loss 0.323 (AVG: 0.533) Score 92.296 (AVG: 91.383) mIOU 71.189 mAP 84.180 mAcc 77.534 IOU: 65.780 0.008 82.052 76.881 95.411 92.704 30.230 81.919 98.660 76.150 79.208 87.446 59.011 mAP: 84.775 8.265 94.822 89.167 98.923 98.646 58.781 97.408 99.796 85.671 92.741 94.048 91.292 mAcc: 80.565 0.091 92.307 84.109 96.748 97.160 32.603 96.740 99.381 79.351 90.182 96.643 62.057 [2021-07-07 21:01:43,803][root][INFO] - Finished test. Elapsed time: 35.5638 [2021-07-07 21:01:44,861][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 21:01:58,292][root][INFO] - ===> Epoch[2721](13605/5): Loss 0.0134, Sem 0.0134 LR: 7.934e-02 Score 99.516 Data time: 0.2413, Forward time: 1.2628, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6464 [2021-07-07 21:02:12,247][root][INFO] - ===> Epoch[2722](13610/5): Loss 0.0135, Sem 0.0135 LR: 7.933e-02 Score 99.509 Data time: 0.2283, Forward time: 1.2628, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7318 [2021-07-07 21:02:25,612][root][INFO] - ===> Epoch[2723](13615/5): Loss 0.0131, Sem 0.0131 LR: 7.932e-02 Score 99.523 Data time: 0.1913, Forward time: 1.2628, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6323 [2021-07-07 21:02:39,992][root][INFO] - ===> Epoch[2724](13620/5): Loss 0.0130, Sem 0.0130 LR: 7.932e-02 Score 99.532 Data time: 0.2451, Forward time: 1.2629, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8268 [2021-07-07 21:02:55,051][root][INFO] - ===> Epoch[2725](13625/5): Loss 0.0129, Sem 0.0129 LR: 7.931e-02 Score 99.537 Data time: 0.2026, Forward time: 1.2628, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.9737 [2021-07-07 21:03:09,202][root][INFO] - ===> Epoch[2726](13630/5): Loss 0.0126, Sem 0.0126 LR: 7.930e-02 Score 99.544 Data time: 0.2138, Forward time: 1.2628, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7896 [2021-07-07 21:03:23,236][root][INFO] - ===> Epoch[2727](13635/5): Loss 0.0129, Sem 0.0129 LR: 7.929e-02 Score 99.532 Data time: 0.2471, Forward time: 1.2629, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7555 [2021-07-07 21:03:37,353][root][INFO] - ===> Epoch[2728](13640/5): Loss 0.0128, Sem 0.0128 LR: 7.929e-02 Score 99.539 Data time: 0.2889, Forward time: 1.2629, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7752 [2021-07-07 21:03:50,457][root][INFO] - ===> Epoch[2729](13645/5): Loss 0.0134, Sem 0.0134 LR: 7.928e-02 Score 99.519 Data time: 0.1152, Forward time: 1.2628, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.5859 [2021-07-07 21:04:04,747][root][INFO] - ===> Epoch[2730](13650/5): Loss 0.0126, Sem 0.0126 LR: 7.927e-02 Score 99.546 Data time: 0.2098, Forward time: 1.2629, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8065 [2021-07-07 21:04:18,799][root][INFO] - ===> Epoch[2731](13655/5): Loss 0.0131, Sem 0.0131 LR: 7.926e-02 Score 99.526 Data time: 0.2104, Forward time: 1.2629, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7530 [2021-07-07 21:04:32,923][root][INFO] - ===> Epoch[2732](13660/5): Loss 0.0126, Sem 0.0126 LR: 7.925e-02 Score 99.543 Data time: 0.2292, Forward time: 1.2629, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7689 [2021-07-07 21:04:47,341][root][INFO] - ===> Epoch[2733](13665/5): Loss 0.0132, Sem 0.0132 LR: 7.925e-02 Score 99.528 Data time: 0.1852, Forward time: 1.2629, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8354 [2021-07-07 21:05:01,382][root][INFO] - ===> Epoch[2734](13670/5): Loss 0.0128, Sem 0.0128 LR: 7.924e-02 Score 99.540 Data time: 0.1816, Forward time: 1.2629, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7600 [2021-07-07 21:05:14,998][root][INFO] - ===> Epoch[2735](13675/5): Loss 0.0133, Sem 0.0133 LR: 7.923e-02 Score 99.520 Data time: 0.2308, Forward time: 1.2629, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6715 [2021-07-07 21:05:29,279][root][INFO] - ===> Epoch[2736](13680/5): Loss 0.0128, Sem 0.0128 LR: 7.922e-02 Score 99.541 Data time: 0.2282, Forward time: 1.2629, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8114 [2021-07-07 21:05:43,621][root][INFO] - ===> Epoch[2737](13685/5): Loss 0.0131, Sem 0.0131 LR: 7.922e-02 Score 99.530 Data time: 0.2773, Forward time: 1.2629, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8179 [2021-07-07 21:05:58,666][root][INFO] - ===> Epoch[2738](13690/5): Loss 0.0128, Sem 0.0128 LR: 7.921e-02 Score 99.534 Data time: 0.2187, Forward time: 1.2630, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.9599 [2021-07-07 21:06:13,081][root][INFO] - ===> Epoch[2739](13695/5): Loss 0.0128, Sem 0.0128 LR: 7.920e-02 Score 99.538 Data time: 0.2574, Forward time: 1.2630, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8361 [2021-07-07 21:06:27,101][root][INFO] - ===> Epoch[2740](13700/5): Loss 0.0130, Sem 0.0130 LR: 7.919e-02 Score 99.530 Data time: 0.2850, Forward time: 1.2630, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7554 [2021-07-07 21:06:41,652][root][INFO] - ===> Epoch[2741](13705/5): Loss 0.0133, Sem 0.0133 LR: 7.919e-02 Score 99.522 Data time: 0.2066, Forward time: 1.2630, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8633 [2021-07-07 21:06:55,448][root][INFO] - ===> Epoch[2742](13710/5): Loss 0.0132, Sem 0.0132 LR: 7.918e-02 Score 99.521 Data time: 0.1714, Forward time: 1.2630, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7174 [2021-07-07 21:07:10,099][root][INFO] - ===> Epoch[2743](13715/5): Loss 0.0129, Sem 0.0129 LR: 7.917e-02 Score 99.534 Data time: 0.2616, Forward time: 1.2630, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8895 [2021-07-07 21:07:22,843][root][INFO] - ===> Epoch[2744](13720/5): Loss 0.0148, Sem 0.0148 LR: 7.916e-02 Score 99.492 Data time: 0.1930, Forward time: 1.2630, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.5040 [2021-07-07 21:07:36,591][root][INFO] - ===> Epoch[2745](13725/5): Loss 0.0141, Sem 0.0141 LR: 7.915e-02 Score 99.494 Data time: 0.1950, Forward time: 1.2631, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7103 [2021-07-07 21:07:50,544][root][INFO] - ===> Epoch[2746](13730/5): Loss 0.0144, Sem 0.0144 LR: 7.915e-02 Score 99.488 Data time: 0.2460, Forward time: 1.2631, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7473 [2021-07-07 21:08:04,816][root][INFO] - ===> Epoch[2747](13735/5): Loss 0.0144, Sem 0.0144 LR: 7.914e-02 Score 99.484 Data time: 0.2080, Forward time: 1.2631, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8078 [2021-07-07 21:08:19,032][root][INFO] - ===> Epoch[2748](13740/5): Loss 0.0145, Sem 0.0145 LR: 7.913e-02 Score 99.484 Data time: 0.1597, Forward time: 1.2630, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8028 [2021-07-07 21:08:32,626][root][INFO] - ===> Epoch[2749](13745/5): Loss 0.0138, Sem 0.0138 LR: 7.912e-02 Score 99.503 Data time: 0.1563, Forward time: 1.2630, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6869 [2021-07-07 21:08:47,077][root][INFO] - ===> Epoch[2750](13750/5): Loss 0.0138, Sem 0.0138 LR: 7.912e-02 Score 99.496 Data time: 0.2130, Forward time: 1.2630, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8375 [2021-07-07 21:09:00,961][root][INFO] - ===> Epoch[2751](13755/5): Loss 0.0137, Sem 0.0137 LR: 7.911e-02 Score 99.504 Data time: 0.1652, Forward time: 1.2630, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7419 [2021-07-07 21:09:14,322][root][INFO] - ===> Epoch[2752](13760/5): Loss 0.0135, Sem 0.0135 LR: 7.910e-02 Score 99.516 Data time: 0.2081, Forward time: 1.2629, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6357 [2021-07-07 21:09:27,721][root][INFO] - ===> Epoch[2753](13765/5): Loss 0.0139, Sem 0.0139 LR: 7.909e-02 Score 99.501 Data time: 0.2321, Forward time: 1.2629, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6306 [2021-07-07 21:09:42,501][root][INFO] - ===> Epoch[2754](13770/5): Loss 0.0126, Sem 0.0126 LR: 7.909e-02 Score 99.543 Data time: 0.3281, Forward time: 1.2629, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9019 [2021-07-07 21:09:56,445][root][INFO] - ===> Epoch[2755](13775/5): Loss 0.0133, Sem 0.0133 LR: 7.908e-02 Score 99.517 Data time: 0.2779, Forward time: 1.2629, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7448 [2021-07-07 21:10:11,173][root][INFO] - ===> Epoch[2756](13780/5): Loss 0.0133, Sem 0.0133 LR: 7.907e-02 Score 99.520 Data time: 0.2357, Forward time: 1.2630, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8890 [2021-07-07 21:10:25,132][root][INFO] - ===> Epoch[2757](13785/5): Loss 0.0129, Sem 0.0129 LR: 7.906e-02 Score 99.539 Data time: 0.2432, Forward time: 1.2630, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7479 [2021-07-07 21:10:40,416][root][INFO] - ===> Epoch[2758](13790/5): Loss 0.0130, Sem 0.0130 LR: 7.905e-02 Score 99.532 Data time: 0.2101, Forward time: 1.2630, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 3.0066 [2021-07-07 21:10:54,218][root][INFO] - ===> Epoch[2759](13795/5): Loss 0.0126, Sem 0.0126 LR: 7.905e-02 Score 99.539 Data time: 0.1673, Forward time: 1.2629, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7267 [2021-07-07 21:11:07,289][root][INFO] - ===> Epoch[2760](13800/5): Loss 0.0134, Sem 0.0134 LR: 7.904e-02 Score 99.522 Data time: 0.2207, Forward time: 1.2629, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.5706 [2021-07-07 21:11:07,291][root][INFO] - ===> Start testing [2021-07-07 21:11:43,692][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1273 Loss 0.453 (AVG: 0.537) Score 89.774 (AVG: 91.407) mIOU 71.719 mAP 84.670 mAcc 77.875 IOU: 65.009 0.000 84.405 76.865 95.461 92.186 31.283 82.518 98.533 78.249 80.941 87.184 59.715 mAP: 83.644 5.565 94.837 90.417 98.838 98.504 59.420 98.003 99.769 93.277 92.713 94.516 91.206 mAcc: 79.997 0.000 90.906 83.892 96.952 95.981 33.598 96.607 99.312 84.718 90.601 96.790 63.025 [2021-07-07 21:11:43,692][root][INFO] - Finished test. Elapsed time: 36.4001 [2021-07-07 21:11:44,783][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 21:11:57,100][root][INFO] - ===> Epoch[2761](13805/5): Loss 0.0131, Sem 0.0131 LR: 7.903e-02 Score 99.527 Data time: 0.1874, Forward time: 1.2628, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.4232 [2021-07-07 21:12:10,653][root][INFO] - ===> Epoch[2762](13810/5): Loss 0.0132, Sem 0.0132 LR: 7.902e-02 Score 99.522 Data time: 0.2314, Forward time: 1.2628, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6636 [2021-07-07 21:12:25,069][root][INFO] - ===> Epoch[2763](13815/5): Loss 0.0132, Sem 0.0132 LR: 7.902e-02 Score 99.522 Data time: 0.2126, Forward time: 1.2628, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8383 [2021-07-07 21:12:38,476][root][INFO] - ===> Epoch[2764](13820/5): Loss 0.0142, Sem 0.0142 LR: 7.901e-02 Score 99.488 Data time: 0.2047, Forward time: 1.2628, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6292 [2021-07-07 21:12:53,052][root][INFO] - ===> Epoch[2765](13825/5): Loss 0.0128, Sem 0.0128 LR: 7.900e-02 Score 99.531 Data time: 0.1840, Forward time: 1.2628, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8708 [2021-07-07 21:13:07,274][root][INFO] - ===> Epoch[2766](13830/5): Loss 0.0127, Sem 0.0127 LR: 7.899e-02 Score 99.539 Data time: 0.2528, Forward time: 1.2628, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8058 [2021-07-07 21:13:21,966][root][INFO] - ===> Epoch[2767](13835/5): Loss 0.0141, Sem 0.0141 LR: 7.899e-02 Score 99.499 Data time: 0.2297, Forward time: 1.2628, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8886 [2021-07-07 21:13:35,433][root][INFO] - ===> Epoch[2768](13840/5): Loss 0.0137, Sem 0.0137 LR: 7.898e-02 Score 99.505 Data time: 0.2222, Forward time: 1.2628, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6516 [2021-07-07 21:13:49,087][root][INFO] - ===> Epoch[2769](13845/5): Loss 0.0142, Sem 0.0142 LR: 7.897e-02 Score 99.491 Data time: 0.3031, Forward time: 1.2628, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6839 [2021-07-07 21:14:03,239][root][INFO] - ===> Epoch[2770](13850/5): Loss 0.0135, Sem 0.0135 LR: 7.896e-02 Score 99.508 Data time: 0.2727, Forward time: 1.2628, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7804 [2021-07-07 21:14:17,568][root][INFO] - ===> Epoch[2771](13855/5): Loss 0.0136, Sem 0.0136 LR: 7.895e-02 Score 99.512 Data time: 0.2251, Forward time: 1.2629, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8182 [2021-07-07 21:14:31,360][root][INFO] - ===> Epoch[2772](13860/5): Loss 0.0136, Sem 0.0136 LR: 7.895e-02 Score 99.510 Data time: 0.2800, Forward time: 1.2629, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7122 [2021-07-07 21:14:45,868][root][INFO] - ===> Epoch[2773](13865/5): Loss 0.0132, Sem 0.0132 LR: 7.894e-02 Score 99.524 Data time: 0.2605, Forward time: 1.2629, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8470 [2021-07-07 21:14:59,610][root][INFO] - ===> Epoch[2774](13870/5): Loss 0.0134, Sem 0.0134 LR: 7.893e-02 Score 99.514 Data time: 0.3174, Forward time: 1.2629, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7021 [2021-07-07 21:15:14,228][root][INFO] - ===> Epoch[2775](13875/5): Loss 0.0132, Sem 0.0132 LR: 7.892e-02 Score 99.523 Data time: 0.1966, Forward time: 1.2629, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8818 [2021-07-07 21:15:28,315][root][INFO] - ===> Epoch[2776](13880/5): Loss 0.0132, Sem 0.0132 LR: 7.892e-02 Score 99.518 Data time: 0.1916, Forward time: 1.2629, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7792 [2021-07-07 21:15:42,373][root][INFO] - ===> Epoch[2777](13885/5): Loss 0.0134, Sem 0.0134 LR: 7.891e-02 Score 99.518 Data time: 0.2759, Forward time: 1.2628, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7703 [2021-07-07 21:15:56,898][root][INFO] - ===> Epoch[2778](13890/5): Loss 0.0133, Sem 0.0133 LR: 7.890e-02 Score 99.525 Data time: 0.1915, Forward time: 1.2629, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8580 [2021-07-07 21:16:10,694][root][INFO] - ===> Epoch[2779](13895/5): Loss 0.0131, Sem 0.0131 LR: 7.889e-02 Score 99.527 Data time: 0.2296, Forward time: 1.2628, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7075 [2021-07-07 21:16:24,333][root][INFO] - ===> Epoch[2780](13900/5): Loss 0.0128, Sem 0.0128 LR: 7.889e-02 Score 99.535 Data time: 0.1633, Forward time: 1.2628, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6842 [2021-07-07 21:16:39,210][root][INFO] - ===> Epoch[2781](13905/5): Loss 0.0128, Sem 0.0128 LR: 7.888e-02 Score 99.533 Data time: 0.1984, Forward time: 1.2628, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9307 [2021-07-07 21:16:53,079][root][INFO] - ===> Epoch[2782](13910/5): Loss 0.0133, Sem 0.0133 LR: 7.887e-02 Score 99.515 Data time: 0.2097, Forward time: 1.2628, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7340 [2021-07-07 21:17:08,028][root][INFO] - ===> Epoch[2783](13915/5): Loss 0.0130, Sem 0.0130 LR: 7.886e-02 Score 99.532 Data time: 0.2193, Forward time: 1.2628, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9520 [2021-07-07 21:17:21,635][root][INFO] - ===> Epoch[2784](13920/5): Loss 0.0131, Sem 0.0131 LR: 7.885e-02 Score 99.522 Data time: 0.2339, Forward time: 1.2628, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6784 [2021-07-07 21:17:36,396][root][INFO] - ===> Epoch[2785](13925/5): Loss 0.0130, Sem 0.0130 LR: 7.885e-02 Score 99.532 Data time: 0.2330, Forward time: 1.2628, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.9043 [2021-07-07 21:17:49,937][root][INFO] - ===> Epoch[2786](13930/5): Loss 0.0125, Sem 0.0125 LR: 7.884e-02 Score 99.548 Data time: 0.2101, Forward time: 1.2627, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.6561 [2021-07-07 21:18:04,191][root][INFO] - ===> Epoch[2787](13935/5): Loss 0.0127, Sem 0.0127 LR: 7.883e-02 Score 99.541 Data time: 0.1869, Forward time: 1.2628, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8075 [2021-07-07 21:18:18,521][root][INFO] - ===> Epoch[2788](13940/5): Loss 0.0125, Sem 0.0125 LR: 7.882e-02 Score 99.546 Data time: 0.2399, Forward time: 1.2628, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8113 [2021-07-07 21:18:33,032][root][INFO] - ===> Epoch[2789](13945/5): Loss 0.0136, Sem 0.0136 LR: 7.882e-02 Score 99.510 Data time: 0.2952, Forward time: 1.2628, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8503 [2021-07-07 21:18:47,277][root][INFO] - ===> Epoch[2790](13950/5): Loss 0.0127, Sem 0.0127 LR: 7.881e-02 Score 99.541 Data time: 0.2902, Forward time: 1.2628, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8055 [2021-07-07 21:19:00,742][root][INFO] - ===> Epoch[2791](13955/5): Loss 0.0125, Sem 0.0125 LR: 7.880e-02 Score 99.553 Data time: 0.2558, Forward time: 1.2628, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6479 [2021-07-07 21:19:15,382][root][INFO] - ===> Epoch[2792](13960/5): Loss 0.0124, Sem 0.0124 LR: 7.879e-02 Score 99.547 Data time: 0.1535, Forward time: 1.2627, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8974 [2021-07-07 21:19:29,796][root][INFO] - ===> Epoch[2793](13965/5): Loss 0.0123, Sem 0.0123 LR: 7.879e-02 Score 99.554 Data time: 0.2343, Forward time: 1.2628, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8298 [2021-07-07 21:19:44,122][root][INFO] - ===> Epoch[2794](13970/5): Loss 0.0123, Sem 0.0123 LR: 7.878e-02 Score 99.552 Data time: 0.2450, Forward time: 1.2628, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8160 [2021-07-07 21:19:58,792][root][INFO] - ===> Epoch[2795](13975/5): Loss 0.0127, Sem 0.0127 LR: 7.877e-02 Score 99.539 Data time: 0.2667, Forward time: 1.2628, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8837 [2021-07-07 21:20:12,820][root][INFO] - ===> Epoch[2796](13980/5): Loss 0.0130, Sem 0.0130 LR: 7.876e-02 Score 99.530 Data time: 0.2162, Forward time: 1.2628, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7651 [2021-07-07 21:20:28,063][root][INFO] - ===> Epoch[2797](13985/5): Loss 0.0126, Sem 0.0126 LR: 7.875e-02 Score 99.540 Data time: 0.2171, Forward time: 1.2628, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 3.0079 [2021-07-07 21:20:42,839][root][INFO] - ===> Epoch[2798](13990/5): Loss 0.0125, Sem 0.0125 LR: 7.875e-02 Score 99.553 Data time: 0.1776, Forward time: 1.2628, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.9003 [2021-07-07 21:20:55,989][root][INFO] - ===> Epoch[2799](13995/5): Loss 0.0118, Sem 0.0118 LR: 7.874e-02 Score 99.575 Data time: 0.2039, Forward time: 1.2628, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.5936 [2021-07-07 21:21:09,901][root][INFO] - ===> Epoch[2800](14000/5): Loss 0.0130, Sem 0.0130 LR: 7.873e-02 Score 99.532 Data time: 0.2727, Forward time: 1.2628, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7279 [2021-07-07 21:21:09,903][root][INFO] - ===> Start testing [2021-07-07 21:21:46,290][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1310 Loss 0.342 (AVG: 0.552) Score 93.077 (AVG: 91.221) mIOU 70.934 mAP 83.676 mAcc 76.776 IOU: 65.382 0.033 84.349 75.788 95.293 91.338 24.823 83.089 98.439 76.922 80.447 86.980 59.265 mAP: 84.191 6.077 93.169 88.480 98.737 98.525 52.980 98.017 99.791 89.701 92.118 94.493 91.503 mAcc: 82.160 0.273 90.139 82.180 96.626 96.621 26.035 96.851 99.123 79.578 89.310 97.177 62.014 [2021-07-07 21:21:46,290][root][INFO] - Finished test. Elapsed time: 36.3861 [2021-07-07 21:21:47,384][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 21:22:00,550][root][INFO] - ===> Epoch[2801](14005/5): Loss 0.0123, Sem 0.0123 LR: 7.872e-02 Score 99.556 Data time: 0.2116, Forward time: 1.2627, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.5881 [2021-07-07 21:22:14,408][root][INFO] - ===> Epoch[2802](14010/5): Loss 0.0127, Sem 0.0127 LR: 7.872e-02 Score 99.543 Data time: 0.2877, Forward time: 1.2628, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7154 [2021-07-07 21:22:28,216][root][INFO] - ===> Epoch[2803](14015/5): Loss 0.0129, Sem 0.0129 LR: 7.871e-02 Score 99.532 Data time: 0.2195, Forward time: 1.2628, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7185 [2021-07-07 21:22:43,113][root][INFO] - ===> Epoch[2804](14020/5): Loss 0.0127, Sem 0.0127 LR: 7.870e-02 Score 99.538 Data time: 0.2221, Forward time: 1.2628, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.9320 [2021-07-07 21:22:57,846][root][INFO] - ===> Epoch[2805](14025/5): Loss 0.0125, Sem 0.0125 LR: 7.869e-02 Score 99.552 Data time: 0.2354, Forward time: 1.2629, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.9051 [2021-07-07 21:23:11,817][root][INFO] - ===> Epoch[2806](14030/5): Loss 0.0126, Sem 0.0126 LR: 7.869e-02 Score 99.546 Data time: 0.2212, Forward time: 1.2629, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7422 [2021-07-07 21:23:25,707][root][INFO] - ===> Epoch[2807](14035/5): Loss 0.0129, Sem 0.0129 LR: 7.868e-02 Score 99.535 Data time: 0.1700, Forward time: 1.2629, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7390 [2021-07-07 21:23:39,929][root][INFO] - ===> Epoch[2808](14040/5): Loss 0.0120, Sem 0.0120 LR: 7.867e-02 Score 99.560 Data time: 0.1998, Forward time: 1.2629, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8065 [2021-07-07 21:23:53,533][root][INFO] - ===> Epoch[2809](14045/5): Loss 0.0130, Sem 0.0130 LR: 7.866e-02 Score 99.528 Data time: 0.2068, Forward time: 1.2629, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6772 [2021-07-07 21:24:07,405][root][INFO] - ===> Epoch[2810](14050/5): Loss 0.0135, Sem 0.0135 LR: 7.865e-02 Score 99.515 Data time: 0.2134, Forward time: 1.2629, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7293 [2021-07-07 21:24:22,233][root][INFO] - ===> Epoch[2811](14055/5): Loss 0.0126, Sem 0.0126 LR: 7.865e-02 Score 99.553 Data time: 0.2866, Forward time: 1.2629, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.9130 [2021-07-07 21:24:36,241][root][INFO] - ===> Epoch[2812](14060/5): Loss 0.0129, Sem 0.0129 LR: 7.864e-02 Score 99.531 Data time: 0.2281, Forward time: 1.2629, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7560 [2021-07-07 21:24:50,057][root][INFO] - ===> Epoch[2813](14065/5): Loss 0.0124, Sem 0.0124 LR: 7.863e-02 Score 99.551 Data time: 0.2210, Forward time: 1.2629, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7136 [2021-07-07 21:25:04,298][root][INFO] - ===> Epoch[2814](14070/5): Loss 0.0121, Sem 0.0121 LR: 7.862e-02 Score 99.563 Data time: 0.1853, Forward time: 1.2629, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8043 [2021-07-07 21:25:18,102][root][INFO] - ===> Epoch[2815](14075/5): Loss 0.0121, Sem 0.0121 LR: 7.862e-02 Score 99.563 Data time: 0.2670, Forward time: 1.2629, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7126 [2021-07-07 21:25:32,741][root][INFO] - ===> Epoch[2816](14080/5): Loss 0.0120, Sem 0.0120 LR: 7.861e-02 Score 99.563 Data time: 0.2300, Forward time: 1.2629, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8815 [2021-07-07 21:25:47,281][root][INFO] - ===> Epoch[2817](14085/5): Loss 0.0126, Sem 0.0126 LR: 7.860e-02 Score 99.544 Data time: 0.2555, Forward time: 1.2628, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8679 [2021-07-07 21:26:01,049][root][INFO] - ===> Epoch[2818](14090/5): Loss 0.0124, Sem 0.0124 LR: 7.859e-02 Score 99.550 Data time: 0.2196, Forward time: 1.2628, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7085 [2021-07-07 21:26:14,632][root][INFO] - ===> Epoch[2819](14095/5): Loss 0.0122, Sem 0.0122 LR: 7.859e-02 Score 99.557 Data time: 0.2139, Forward time: 1.2628, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6641 [2021-07-07 21:26:28,785][root][INFO] - ===> Epoch[2820](14100/5): Loss 0.0120, Sem 0.0120 LR: 7.858e-02 Score 99.568 Data time: 0.2724, Forward time: 1.2629, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7676 [2021-07-07 21:26:42,645][root][INFO] - ===> Epoch[2821](14105/5): Loss 0.0123, Sem 0.0123 LR: 7.857e-02 Score 99.555 Data time: 0.2604, Forward time: 1.2629, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7187 [2021-07-07 21:26:57,911][root][INFO] - ===> Epoch[2822](14110/5): Loss 0.0123, Sem 0.0123 LR: 7.856e-02 Score 99.553 Data time: 0.3051, Forward time: 1.2629, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 3.0066 [2021-07-07 21:27:12,620][root][INFO] - ===> Epoch[2823](14115/5): Loss 0.0121, Sem 0.0121 LR: 7.855e-02 Score 99.560 Data time: 0.2064, Forward time: 1.2630, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.9010 [2021-07-07 21:27:26,963][root][INFO] - ===> Epoch[2824](14120/5): Loss 0.0128, Sem 0.0128 LR: 7.855e-02 Score 99.540 Data time: 0.2559, Forward time: 1.2630, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8146 [2021-07-07 21:27:40,741][root][INFO] - ===> Epoch[2825](14125/5): Loss 0.0124, Sem 0.0124 LR: 7.854e-02 Score 99.549 Data time: 0.2960, Forward time: 1.2630, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7048 [2021-07-07 21:27:54,509][root][INFO] - ===> Epoch[2826](14130/5): Loss 0.0125, Sem 0.0125 LR: 7.853e-02 Score 99.550 Data time: 0.2657, Forward time: 1.2630, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7119 [2021-07-07 21:28:08,543][root][INFO] - ===> Epoch[2827](14135/5): Loss 0.0127, Sem 0.0127 LR: 7.852e-02 Score 99.544 Data time: 0.2413, Forward time: 1.2630, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7550 [2021-07-07 21:28:23,064][root][INFO] - ===> Epoch[2828](14140/5): Loss 0.0119, Sem 0.0119 LR: 7.852e-02 Score 99.567 Data time: 0.2307, Forward time: 1.2630, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8607 [2021-07-07 21:28:36,588][root][INFO] - ===> Epoch[2829](14145/5): Loss 0.0121, Sem 0.0121 LR: 7.851e-02 Score 99.563 Data time: 0.2005, Forward time: 1.2631, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6606 [2021-07-07 21:28:50,866][root][INFO] - ===> Epoch[2830](14150/5): Loss 0.0121, Sem 0.0121 LR: 7.850e-02 Score 99.558 Data time: 0.2181, Forward time: 1.2630, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8069 [2021-07-07 21:29:04,902][root][INFO] - ===> Epoch[2831](14155/5): Loss 0.0124, Sem 0.0124 LR: 7.849e-02 Score 99.552 Data time: 0.2145, Forward time: 1.2630, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7556 [2021-07-07 21:29:19,018][root][INFO] - ===> Epoch[2832](14160/5): Loss 0.0129, Sem 0.0129 LR: 7.848e-02 Score 99.533 Data time: 0.2612, Forward time: 1.2630, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7644 [2021-07-07 21:29:32,738][root][INFO] - ===> Epoch[2833](14165/5): Loss 0.0131, Sem 0.0131 LR: 7.848e-02 Score 99.533 Data time: 0.2207, Forward time: 1.2630, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.6952 [2021-07-07 21:29:47,294][root][INFO] - ===> Epoch[2834](14170/5): Loss 0.0144, Sem 0.0144 LR: 7.847e-02 Score 99.501 Data time: 0.1857, Forward time: 1.2630, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8653 [2021-07-07 21:30:01,531][root][INFO] - ===> Epoch[2835](14175/5): Loss 0.0140, Sem 0.0140 LR: 7.846e-02 Score 99.504 Data time: 0.2405, Forward time: 1.2630, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7979 [2021-07-07 21:30:14,696][root][INFO] - ===> Epoch[2836](14180/5): Loss 0.0145, Sem 0.0145 LR: 7.845e-02 Score 99.492 Data time: 0.2217, Forward time: 1.2630, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.5815 [2021-07-07 21:30:28,556][root][INFO] - ===> Epoch[2837](14185/5): Loss 0.0147, Sem 0.0147 LR: 7.845e-02 Score 99.473 Data time: 0.2322, Forward time: 1.2630, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7277 [2021-07-07 21:30:42,215][root][INFO] - ===> Epoch[2838](14190/5): Loss 0.0140, Sem 0.0140 LR: 7.844e-02 Score 99.496 Data time: 0.1761, Forward time: 1.2630, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.6919 [2021-07-07 21:30:55,412][root][INFO] - ===> Epoch[2839](14195/5): Loss 0.0152, Sem 0.0152 LR: 7.843e-02 Score 99.469 Data time: 0.1958, Forward time: 1.2630, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6008 [2021-07-07 21:31:10,229][root][INFO] - ===> Epoch[2840](14200/5): Loss 0.0143, Sem 0.0143 LR: 7.842e-02 Score 99.493 Data time: 0.2549, Forward time: 1.2630, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9201 [2021-07-07 21:31:10,231][root][INFO] - ===> Start testing [2021-07-07 21:31:44,763][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1332 Loss 0.420 (AVG: 0.524) Score 90.680 (AVG: 91.241) mIOU 71.797 mAP 85.439 mAcc 78.351 IOU: 64.077 0.170 83.510 76.158 94.668 91.501 39.356 86.005 98.575 73.604 79.288 86.898 59.558 mAP: 83.978 12.180 94.078 89.326 98.839 98.794 63.405 98.351 99.793 93.808 92.441 93.830 91.886 mAcc: 78.641 1.459 90.663 84.122 96.114 96.838 44.362 95.828 99.512 81.511 90.325 96.508 62.686 [2021-07-07 21:31:44,764][root][INFO] - Finished test. Elapsed time: 34.5322 [2021-07-07 21:31:45,848][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 21:31:58,331][root][INFO] - ===> Epoch[2841](14205/5): Loss 0.0139, Sem 0.0139 LR: 7.842e-02 Score 99.500 Data time: 0.2067, Forward time: 1.2630, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.4492 [2021-07-07 21:32:13,242][root][INFO] - ===> Epoch[2842](14210/5): Loss 0.0144, Sem 0.0144 LR: 7.841e-02 Score 99.485 Data time: 0.2275, Forward time: 1.2630, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9332 [2021-07-07 21:32:27,567][root][INFO] - ===> Epoch[2843](14215/5): Loss 0.0142, Sem 0.0142 LR: 7.840e-02 Score 99.487 Data time: 0.1825, Forward time: 1.2630, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8280 [2021-07-07 21:32:41,575][root][INFO] - ===> Epoch[2844](14220/5): Loss 0.0138, Sem 0.0138 LR: 7.839e-02 Score 99.497 Data time: 0.2149, Forward time: 1.2630, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7604 [2021-07-07 21:32:55,225][root][INFO] - ===> Epoch[2845](14225/5): Loss 0.0143, Sem 0.0143 LR: 7.838e-02 Score 99.491 Data time: 0.2528, Forward time: 1.2630, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6880 [2021-07-07 21:33:09,832][root][INFO] - ===> Epoch[2846](14230/5): Loss 0.0131, Sem 0.0131 LR: 7.838e-02 Score 99.531 Data time: 0.2675, Forward time: 1.2630, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8705 [2021-07-07 21:33:23,014][root][INFO] - ===> Epoch[2847](14235/5): Loss 0.0141, Sem 0.0141 LR: 7.837e-02 Score 99.504 Data time: 0.2536, Forward time: 1.2630, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.5937 [2021-07-07 21:33:37,665][root][INFO] - ===> Epoch[2848](14240/5): Loss 0.0130, Sem 0.0130 LR: 7.836e-02 Score 99.531 Data time: 0.2586, Forward time: 1.2631, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8779 [2021-07-07 21:33:51,718][root][INFO] - ===> Epoch[2849](14245/5): Loss 0.0139, Sem 0.0139 LR: 7.835e-02 Score 99.498 Data time: 0.1630, Forward time: 1.2630, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7559 [2021-07-07 21:34:05,899][root][INFO] - ===> Epoch[2850](14250/5): Loss 0.0129, Sem 0.0129 LR: 7.835e-02 Score 99.534 Data time: 0.2219, Forward time: 1.2630, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7873 [2021-07-07 21:34:20,029][root][INFO] - ===> Epoch[2851](14255/5): Loss 0.0133, Sem 0.0133 LR: 7.834e-02 Score 99.532 Data time: 0.2487, Forward time: 1.2630, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7706 [2021-07-07 21:34:34,477][root][INFO] - ===> Epoch[2852](14260/5): Loss 0.0142, Sem 0.0142 LR: 7.833e-02 Score 99.500 Data time: 0.2850, Forward time: 1.2631, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8489 [2021-07-07 21:34:48,189][root][INFO] - ===> Epoch[2853](14265/5): Loss 0.0132, Sem 0.0132 LR: 7.832e-02 Score 99.525 Data time: 0.2658, Forward time: 1.2631, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6892 [2021-07-07 21:35:01,650][root][INFO] - ===> Epoch[2854](14270/5): Loss 0.0141, Sem 0.0141 LR: 7.832e-02 Score 99.494 Data time: 0.2446, Forward time: 1.2631, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6310 [2021-07-07 21:35:15,946][root][INFO] - ===> Epoch[2855](14275/5): Loss 0.0135, Sem 0.0135 LR: 7.831e-02 Score 99.510 Data time: 0.2074, Forward time: 1.2631, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8149 [2021-07-07 21:35:29,911][root][INFO] - ===> Epoch[2856](14280/5): Loss 0.0134, Sem 0.0134 LR: 7.830e-02 Score 99.521 Data time: 0.2267, Forward time: 1.2631, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7395 [2021-07-07 21:35:43,369][root][INFO] - ===> Epoch[2857](14285/5): Loss 0.0138, Sem 0.0138 LR: 7.829e-02 Score 99.504 Data time: 0.2699, Forward time: 1.2631, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6492 [2021-07-07 21:35:57,481][root][INFO] - ===> Epoch[2858](14290/5): Loss 0.0139, Sem 0.0139 LR: 7.828e-02 Score 99.498 Data time: 0.2036, Forward time: 1.2631, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7831 [2021-07-07 21:36:11,305][root][INFO] - ===> Epoch[2859](14295/5): Loss 0.0132, Sem 0.0132 LR: 7.828e-02 Score 99.526 Data time: 0.2036, Forward time: 1.2631, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7200 [2021-07-07 21:36:25,194][root][INFO] - ===> Epoch[2860](14300/5): Loss 0.0129, Sem 0.0129 LR: 7.827e-02 Score 99.536 Data time: 0.1994, Forward time: 1.2631, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7372 [2021-07-07 21:36:39,579][root][INFO] - ===> Epoch[2861](14305/5): Loss 0.0133, Sem 0.0133 LR: 7.826e-02 Score 99.522 Data time: 0.2421, Forward time: 1.2631, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8335 [2021-07-07 21:36:53,268][root][INFO] - ===> Epoch[2862](14310/5): Loss 0.0135, Sem 0.0135 LR: 7.825e-02 Score 99.510 Data time: 0.1591, Forward time: 1.2631, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7006 [2021-07-07 21:37:08,134][root][INFO] - ===> Epoch[2863](14315/5): Loss 0.0127, Sem 0.0127 LR: 7.825e-02 Score 99.541 Data time: 0.2314, Forward time: 1.2631, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9284 [2021-07-07 21:37:22,451][root][INFO] - ===> Epoch[2864](14320/5): Loss 0.0124, Sem 0.0124 LR: 7.824e-02 Score 99.554 Data time: 0.1911, Forward time: 1.2631, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8246 [2021-07-07 21:37:36,504][root][INFO] - ===> Epoch[2865](14325/5): Loss 0.0129, Sem 0.0129 LR: 7.823e-02 Score 99.536 Data time: 0.2510, Forward time: 1.2631, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7652 [2021-07-07 21:37:50,621][root][INFO] - ===> Epoch[2866](14330/5): Loss 0.0127, Sem 0.0127 LR: 7.822e-02 Score 99.537 Data time: 0.2315, Forward time: 1.2631, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7744 [2021-07-07 21:38:04,003][root][INFO] - ===> Epoch[2867](14335/5): Loss 0.0140, Sem 0.0140 LR: 7.822e-02 Score 99.493 Data time: 0.1655, Forward time: 1.2631, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6310 [2021-07-07 21:38:18,293][root][INFO] - ===> Epoch[2868](14340/5): Loss 0.0137, Sem 0.0137 LR: 7.821e-02 Score 99.508 Data time: 0.2233, Forward time: 1.2631, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8139 [2021-07-07 21:38:32,013][root][INFO] - ===> Epoch[2869](14345/5): Loss 0.0130, Sem 0.0130 LR: 7.820e-02 Score 99.529 Data time: 0.2687, Forward time: 1.2631, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6991 [2021-07-07 21:38:46,129][root][INFO] - ===> Epoch[2870](14350/5): Loss 0.0140, Sem 0.0140 LR: 7.819e-02 Score 99.497 Data time: 0.1626, Forward time: 1.2631, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7757 [2021-07-07 21:39:00,066][root][INFO] - ===> Epoch[2871](14355/5): Loss 0.0132, Sem 0.0132 LR: 7.818e-02 Score 99.521 Data time: 0.2670, Forward time: 1.2631, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7190 [2021-07-07 21:39:14,133][root][INFO] - ===> Epoch[2872](14360/5): Loss 0.0135, Sem 0.0135 LR: 7.818e-02 Score 99.513 Data time: 0.2596, Forward time: 1.2632, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7633 [2021-07-07 21:39:28,069][root][INFO] - ===> Epoch[2873](14365/5): Loss 0.0142, Sem 0.0142 LR: 7.817e-02 Score 99.489 Data time: 0.2370, Forward time: 1.2632, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7398 [2021-07-07 21:39:42,431][root][INFO] - ===> Epoch[2874](14370/5): Loss 0.0139, Sem 0.0139 LR: 7.816e-02 Score 99.500 Data time: 0.2713, Forward time: 1.2632, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8305 [2021-07-07 21:39:57,396][root][INFO] - ===> Epoch[2875](14375/5): Loss 0.0132, Sem 0.0132 LR: 7.815e-02 Score 99.524 Data time: 0.2814, Forward time: 1.2633, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9456 [2021-07-07 21:40:12,014][root][INFO] - ===> Epoch[2876](14380/5): Loss 0.0132, Sem 0.0132 LR: 7.815e-02 Score 99.523 Data time: 0.2529, Forward time: 1.2633, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8814 [2021-07-07 21:40:26,338][root][INFO] - ===> Epoch[2877](14385/5): Loss 0.0132, Sem 0.0132 LR: 7.814e-02 Score 99.524 Data time: 0.1752, Forward time: 1.2633, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8204 [2021-07-07 21:40:40,554][root][INFO] - ===> Epoch[2878](14390/5): Loss 0.0129, Sem 0.0129 LR: 7.813e-02 Score 99.535 Data time: 0.2778, Forward time: 1.2633, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7944 [2021-07-07 21:40:54,620][root][INFO] - ===> Epoch[2879](14395/5): Loss 0.0128, Sem 0.0128 LR: 7.812e-02 Score 99.536 Data time: 0.2540, Forward time: 1.2633, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7666 [2021-07-07 21:41:08,318][root][INFO] - ===> Epoch[2880](14400/5): Loss 0.0134, Sem 0.0134 LR: 7.811e-02 Score 99.514 Data time: 0.2075, Forward time: 1.2633, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6988 [2021-07-07 21:41:08,320][root][INFO] - ===> Start testing [2021-07-07 21:41:44,107][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1232 Loss 0.281 (AVG: 0.535) Score 93.507 (AVG: 91.343) mIOU 71.874 mAP 83.914 mAcc 77.506 IOU: 64.299 0.000 83.592 76.231 95.244 92.674 31.979 86.346 98.518 78.791 80.686 86.986 59.012 mAP: 83.166 4.887 94.061 89.603 98.777 98.519 54.719 98.140 99.789 91.702 92.238 94.307 90.977 mAcc: 80.087 0.000 90.009 83.507 96.826 97.160 33.724 96.582 99.325 80.890 89.839 96.882 62.754 [2021-07-07 21:41:44,107][root][INFO] - Finished test. Elapsed time: 35.7863 [2021-07-07 21:41:45,175][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 21:41:57,951][root][INFO] - ===> Epoch[2881](14405/5): Loss 0.0134, Sem 0.0134 LR: 7.811e-02 Score 99.516 Data time: 0.2271, Forward time: 1.2632, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.5155 [2021-07-07 21:42:11,778][root][INFO] - ===> Epoch[2882](14410/5): Loss 0.0137, Sem 0.0137 LR: 7.810e-02 Score 99.506 Data time: 0.2083, Forward time: 1.2632, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7177 [2021-07-07 21:42:26,017][root][INFO] - ===> Epoch[2883](14415/5): Loss 0.0135, Sem 0.0135 LR: 7.809e-02 Score 99.515 Data time: 0.2038, Forward time: 1.2633, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8017 [2021-07-07 21:42:40,670][root][INFO] - ===> Epoch[2884](14420/5): Loss 0.0129, Sem 0.0129 LR: 7.808e-02 Score 99.531 Data time: 0.2799, Forward time: 1.2633, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8805 [2021-07-07 21:42:54,589][root][INFO] - ===> Epoch[2885](14425/5): Loss 0.0131, Sem 0.0131 LR: 7.808e-02 Score 99.522 Data time: 0.1805, Forward time: 1.2633, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7238 [2021-07-07 21:43:09,059][root][INFO] - ===> Epoch[2886](14430/5): Loss 0.0125, Sem 0.0125 LR: 7.807e-02 Score 99.541 Data time: 0.2386, Forward time: 1.2633, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8375 [2021-07-07 21:43:23,178][root][INFO] - ===> Epoch[2887](14435/5): Loss 0.0134, Sem 0.0134 LR: 7.806e-02 Score 99.516 Data time: 0.1946, Forward time: 1.2632, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7793 [2021-07-07 21:43:36,735][root][INFO] - ===> Epoch[2888](14440/5): Loss 0.0133, Sem 0.0133 LR: 7.805e-02 Score 99.518 Data time: 0.2387, Forward time: 1.2633, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6690 [2021-07-07 21:43:50,294][root][INFO] - ===> Epoch[2889](14445/5): Loss 0.0134, Sem 0.0134 LR: 7.805e-02 Score 99.516 Data time: 0.2168, Forward time: 1.2633, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6624 [2021-07-07 21:44:04,220][root][INFO] - ===> Epoch[2890](14450/5): Loss 0.0135, Sem 0.0135 LR: 7.804e-02 Score 99.524 Data time: 0.2184, Forward time: 1.2633, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7375 [2021-07-07 21:44:18,435][root][INFO] - ===> Epoch[2891](14455/5): Loss 0.0127, Sem 0.0127 LR: 7.803e-02 Score 99.541 Data time: 0.2051, Forward time: 1.2633, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7980 [2021-07-07 21:44:32,789][root][INFO] - ===> Epoch[2892](14460/5): Loss 0.0137, Sem 0.0137 LR: 7.802e-02 Score 99.504 Data time: 0.3201, Forward time: 1.2633, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8151 [2021-07-07 21:44:47,216][root][INFO] - ===> Epoch[2893](14465/5): Loss 0.0126, Sem 0.0126 LR: 7.801e-02 Score 99.546 Data time: 0.2251, Forward time: 1.2633, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8468 [2021-07-07 21:45:01,850][root][INFO] - ===> Epoch[2894](14470/5): Loss 0.0130, Sem 0.0130 LR: 7.801e-02 Score 99.534 Data time: 0.2246, Forward time: 1.2634, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8801 [2021-07-07 21:45:15,410][root][INFO] - ===> Epoch[2895](14475/5): Loss 0.0123, Sem 0.0123 LR: 7.800e-02 Score 99.554 Data time: 0.2251, Forward time: 1.2634, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6699 [2021-07-07 21:45:29,281][root][INFO] - ===> Epoch[2896](14480/5): Loss 0.0124, Sem 0.0124 LR: 7.799e-02 Score 99.550 Data time: 0.3275, Forward time: 1.2634, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7291 [2021-07-07 21:45:43,658][root][INFO] - ===> Epoch[2897](14485/5): Loss 0.0128, Sem 0.0128 LR: 7.798e-02 Score 99.544 Data time: 0.2288, Forward time: 1.2634, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8306 [2021-07-07 21:45:57,470][root][INFO] - ===> Epoch[2898](14490/5): Loss 0.0125, Sem 0.0125 LR: 7.798e-02 Score 99.549 Data time: 0.2101, Forward time: 1.2634, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7187 [2021-07-07 21:46:11,968][root][INFO] - ===> Epoch[2899](14495/5): Loss 0.0128, Sem 0.0128 LR: 7.797e-02 Score 99.539 Data time: 0.2000, Forward time: 1.2635, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8533 [2021-07-07 21:46:26,335][root][INFO] - ===> Epoch[2900](14500/5): Loss 0.0128, Sem 0.0128 LR: 7.796e-02 Score 99.541 Data time: 0.2800, Forward time: 1.2635, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8268 [2021-07-07 21:46:40,691][root][INFO] - ===> Epoch[2901](14505/5): Loss 0.0127, Sem 0.0127 LR: 7.795e-02 Score 99.543 Data time: 0.1980, Forward time: 1.2634, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8281 [2021-07-07 21:46:54,769][root][INFO] - ===> Epoch[2902](14510/5): Loss 0.0131, Sem 0.0131 LR: 7.795e-02 Score 99.532 Data time: 0.3050, Forward time: 1.2635, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7514 [2021-07-07 21:47:09,041][root][INFO] - ===> Epoch[2903](14515/5): Loss 0.0128, Sem 0.0128 LR: 7.794e-02 Score 99.545 Data time: 0.2414, Forward time: 1.2635, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8112 [2021-07-07 21:47:24,278][root][INFO] - ===> Epoch[2904](14520/5): Loss 0.0130, Sem 0.0130 LR: 7.793e-02 Score 99.531 Data time: 0.2865, Forward time: 1.2635, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 3.0006 [2021-07-07 21:47:38,856][root][INFO] - ===> Epoch[2905](14525/5): Loss 0.0124, Sem 0.0124 LR: 7.792e-02 Score 99.553 Data time: 0.2057, Forward time: 1.2635, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8769 [2021-07-07 21:47:52,828][root][INFO] - ===> Epoch[2906](14530/5): Loss 0.0125, Sem 0.0125 LR: 7.791e-02 Score 99.558 Data time: 0.2923, Forward time: 1.2635, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7534 [2021-07-07 21:48:07,827][root][INFO] - ===> Epoch[2907](14535/5): Loss 0.0126, Sem 0.0126 LR: 7.791e-02 Score 99.545 Data time: 0.2490, Forward time: 1.2635, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9539 [2021-07-07 21:48:21,546][root][INFO] - ===> Epoch[2908](14540/5): Loss 0.0126, Sem 0.0126 LR: 7.790e-02 Score 99.538 Data time: 0.2328, Forward time: 1.2635, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6947 [2021-07-07 21:48:36,234][root][INFO] - ===> Epoch[2909](14545/5): Loss 0.0127, Sem 0.0127 LR: 7.789e-02 Score 99.542 Data time: 0.2244, Forward time: 1.2636, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8907 [2021-07-07 21:48:51,697][root][INFO] - ===> Epoch[2910](14550/5): Loss 0.0129, Sem 0.0129 LR: 7.788e-02 Score 99.535 Data time: 0.1707, Forward time: 1.2635, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 3.0496 [2021-07-07 21:49:05,627][root][INFO] - ===> Epoch[2911](14555/5): Loss 0.0127, Sem 0.0127 LR: 7.788e-02 Score 99.542 Data time: 0.2294, Forward time: 1.2635, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7323 [2021-07-07 21:49:20,233][root][INFO] - ===> Epoch[2912](14560/5): Loss 0.0130, Sem 0.0130 LR: 7.787e-02 Score 99.536 Data time: 0.2797, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8791 [2021-07-07 21:49:33,631][root][INFO] - ===> Epoch[2913](14565/5): Loss 0.0124, Sem 0.0124 LR: 7.786e-02 Score 99.554 Data time: 0.1764, Forward time: 1.2635, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6304 [2021-07-07 21:49:47,699][root][INFO] - ===> Epoch[2914](14570/5): Loss 0.0121, Sem 0.0121 LR: 7.785e-02 Score 99.566 Data time: 0.2514, Forward time: 1.2635, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7670 [2021-07-07 21:50:01,629][root][INFO] - ===> Epoch[2915](14575/5): Loss 0.0128, Sem 0.0128 LR: 7.785e-02 Score 99.535 Data time: 0.2312, Forward time: 1.2635, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7345 [2021-07-07 21:50:16,469][root][INFO] - ===> Epoch[2916](14580/5): Loss 0.0122, Sem 0.0122 LR: 7.784e-02 Score 99.557 Data time: 0.2001, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9269 [2021-07-07 21:50:30,165][root][INFO] - ===> Epoch[2917](14585/5): Loss 0.0126, Sem 0.0126 LR: 7.783e-02 Score 99.551 Data time: 0.2766, Forward time: 1.2635, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6995 [2021-07-07 21:50:44,620][root][INFO] - ===> Epoch[2918](14590/5): Loss 0.0124, Sem 0.0124 LR: 7.782e-02 Score 99.549 Data time: 0.2331, Forward time: 1.2635, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8331 [2021-07-07 21:50:58,963][root][INFO] - ===> Epoch[2919](14595/5): Loss 0.0129, Sem 0.0129 LR: 7.781e-02 Score 99.540 Data time: 0.2129, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8245 [2021-07-07 21:51:14,134][root][INFO] - ===> Epoch[2920](14600/5): Loss 0.0128, Sem 0.0128 LR: 7.781e-02 Score 99.537 Data time: 0.2350, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9868 [2021-07-07 21:51:14,135][root][INFO] - ===> Start testing [2021-07-07 21:51:49,902][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1252 Loss 0.351 (AVG: 0.538) Score 91.939 (AVG: 91.340) mIOU 71.398 mAP 83.557 mAcc 76.986 IOU: 65.541 0.000 85.022 76.658 95.271 92.465 25.408 84.029 98.635 78.022 81.136 86.795 59.195 mAP: 85.197 4.331 92.488 88.389 98.792 98.868 51.777 98.208 99.756 90.868 92.371 93.716 91.483 mAcc: 81.005 0.000 89.273 83.907 96.517 96.399 26.693 96.499 99.451 81.709 89.702 97.096 62.570 [2021-07-07 21:51:49,902][root][INFO] - Finished test. Elapsed time: 35.7662 [2021-07-07 21:51:50,994][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 21:52:03,157][root][INFO] - ===> Epoch[2921](14605/5): Loss 0.0134, Sem 0.0134 LR: 7.780e-02 Score 99.520 Data time: 0.1609, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.3914 [2021-07-07 21:52:17,061][root][INFO] - ===> Epoch[2922](14610/5): Loss 0.0125, Sem 0.0125 LR: 7.779e-02 Score 99.550 Data time: 0.1993, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7401 [2021-07-07 21:52:29,912][root][INFO] - ===> Epoch[2923](14615/5): Loss 0.0128, Sem 0.0128 LR: 7.778e-02 Score 99.534 Data time: 0.1847, Forward time: 1.2634, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.5221 [2021-07-07 21:52:43,660][root][INFO] - ===> Epoch[2924](14620/5): Loss 0.0123, Sem 0.0123 LR: 7.778e-02 Score 99.556 Data time: 0.2690, Forward time: 1.2635, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7048 [2021-07-07 21:52:57,508][root][INFO] - ===> Epoch[2925](14625/5): Loss 0.0128, Sem 0.0128 LR: 7.777e-02 Score 99.538 Data time: 0.2622, Forward time: 1.2635, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7284 [2021-07-07 21:53:11,820][root][INFO] - ===> Epoch[2926](14630/5): Loss 0.0124, Sem 0.0124 LR: 7.776e-02 Score 99.552 Data time: 0.1894, Forward time: 1.2635, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8176 [2021-07-07 21:53:26,110][root][INFO] - ===> Epoch[2927](14635/5): Loss 0.0120, Sem 0.0120 LR: 7.775e-02 Score 99.562 Data time: 0.2206, Forward time: 1.2635, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8172 [2021-07-07 21:53:40,799][root][INFO] - ===> Epoch[2928](14640/5): Loss 0.0130, Sem 0.0130 LR: 7.774e-02 Score 99.530 Data time: 0.2197, Forward time: 1.2635, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8999 [2021-07-07 21:53:54,608][root][INFO] - ===> Epoch[2929](14645/5): Loss 0.0123, Sem 0.0123 LR: 7.774e-02 Score 99.557 Data time: 0.1831, Forward time: 1.2635, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7239 [2021-07-07 21:54:08,347][root][INFO] - ===> Epoch[2930](14650/5): Loss 0.0127, Sem 0.0127 LR: 7.773e-02 Score 99.539 Data time: 0.2884, Forward time: 1.2636, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6952 [2021-07-07 21:54:22,089][root][INFO] - ===> Epoch[2931](14655/5): Loss 0.0130, Sem 0.0130 LR: 7.772e-02 Score 99.533 Data time: 0.1597, Forward time: 1.2636, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7088 [2021-07-07 21:54:35,707][root][INFO] - ===> Epoch[2932](14660/5): Loss 0.0127, Sem 0.0127 LR: 7.771e-02 Score 99.545 Data time: 0.2093, Forward time: 1.2636, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6727 [2021-07-07 21:54:49,680][root][INFO] - ===> Epoch[2933](14665/5): Loss 0.0126, Sem 0.0126 LR: 7.771e-02 Score 99.541 Data time: 0.2109, Forward time: 1.2636, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7495 [2021-07-07 21:55:03,920][root][INFO] - ===> Epoch[2934](14670/5): Loss 0.0125, Sem 0.0125 LR: 7.770e-02 Score 99.547 Data time: 0.1957, Forward time: 1.2636, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7979 [2021-07-07 21:55:18,138][root][INFO] - ===> Epoch[2935](14675/5): Loss 0.0123, Sem 0.0123 LR: 7.769e-02 Score 99.557 Data time: 0.2371, Forward time: 1.2636, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8043 [2021-07-07 21:55:32,653][root][INFO] - ===> Epoch[2936](14680/5): Loss 0.0128, Sem 0.0128 LR: 7.768e-02 Score 99.538 Data time: 0.2167, Forward time: 1.2636, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8625 [2021-07-07 21:55:47,123][root][INFO] - ===> Epoch[2937](14685/5): Loss 0.0128, Sem 0.0128 LR: 7.768e-02 Score 99.538 Data time: 0.1508, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8486 [2021-07-07 21:56:01,018][root][INFO] - ===> Epoch[2938](14690/5): Loss 0.0127, Sem 0.0127 LR: 7.767e-02 Score 99.542 Data time: 0.2296, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7277 [2021-07-07 21:56:14,989][root][INFO] - ===> Epoch[2939](14695/5): Loss 0.0126, Sem 0.0126 LR: 7.766e-02 Score 99.543 Data time: 0.1861, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7543 [2021-07-07 21:56:28,507][root][INFO] - ===> Epoch[2940](14700/5): Loss 0.0130, Sem 0.0130 LR: 7.765e-02 Score 99.532 Data time: 0.2205, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6565 [2021-07-07 21:56:43,073][root][INFO] - ===> Epoch[2941](14705/5): Loss 0.0130, Sem 0.0130 LR: 7.764e-02 Score 99.536 Data time: 0.2678, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8551 [2021-07-07 21:56:57,258][root][INFO] - ===> Epoch[2942](14710/5): Loss 0.0137, Sem 0.0137 LR: 7.764e-02 Score 99.503 Data time: 0.2358, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7905 [2021-07-07 21:57:10,998][root][INFO] - ===> Epoch[2943](14715/5): Loss 0.0129, Sem 0.0129 LR: 7.763e-02 Score 99.530 Data time: 0.1914, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6983 [2021-07-07 21:57:24,690][root][INFO] - ===> Epoch[2944](14720/5): Loss 0.0129, Sem 0.0129 LR: 7.762e-02 Score 99.533 Data time: 0.1578, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6992 [2021-07-07 21:57:38,160][root][INFO] - ===> Epoch[2945](14725/5): Loss 0.0127, Sem 0.0127 LR: 7.761e-02 Score 99.540 Data time: 0.2134, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6532 [2021-07-07 21:57:52,940][root][INFO] - ===> Epoch[2946](14730/5): Loss 0.0125, Sem 0.0125 LR: 7.761e-02 Score 99.551 Data time: 0.2414, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9066 [2021-07-07 21:58:07,105][root][INFO] - ===> Epoch[2947](14735/5): Loss 0.0124, Sem 0.0124 LR: 7.760e-02 Score 99.554 Data time: 0.2975, Forward time: 1.2636, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7848 [2021-07-07 21:58:21,198][root][INFO] - ===> Epoch[2948](14740/5): Loss 0.0127, Sem 0.0127 LR: 7.759e-02 Score 99.541 Data time: 0.2420, Forward time: 1.2636, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7718 [2021-07-07 21:58:35,266][root][INFO] - ===> Epoch[2949](14745/5): Loss 0.0126, Sem 0.0126 LR: 7.758e-02 Score 99.546 Data time: 0.1793, Forward time: 1.2636, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7730 [2021-07-07 21:58:50,015][root][INFO] - ===> Epoch[2950](14750/5): Loss 0.0121, Sem 0.0121 LR: 7.758e-02 Score 99.561 Data time: 0.2293, Forward time: 1.2637, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9083 [2021-07-07 21:59:04,522][root][INFO] - ===> Epoch[2951](14755/5): Loss 0.0130, Sem 0.0130 LR: 7.757e-02 Score 99.532 Data time: 0.2093, Forward time: 1.2637, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8632 [2021-07-07 21:59:19,008][root][INFO] - ===> Epoch[2952](14760/5): Loss 0.0119, Sem 0.0119 LR: 7.756e-02 Score 99.571 Data time: 0.2248, Forward time: 1.2637, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8435 [2021-07-07 21:59:32,667][root][INFO] - ===> Epoch[2953](14765/5): Loss 0.0116, Sem 0.0116 LR: 7.755e-02 Score 99.580 Data time: 0.2362, Forward time: 1.2637, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6871 [2021-07-07 21:59:47,025][root][INFO] - ===> Epoch[2954](14770/5): Loss 0.0119, Sem 0.0119 LR: 7.754e-02 Score 99.568 Data time: 0.2637, Forward time: 1.2637, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8250 [2021-07-07 22:00:00,581][root][INFO] - ===> Epoch[2955](14775/5): Loss 0.0120, Sem 0.0120 LR: 7.754e-02 Score 99.567 Data time: 0.2899, Forward time: 1.2637, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6436 [2021-07-07 22:00:14,702][root][INFO] - ===> Epoch[2956](14780/5): Loss 0.0124, Sem 0.0124 LR: 7.753e-02 Score 99.552 Data time: 0.1826, Forward time: 1.2637, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7696 [2021-07-07 22:00:28,825][root][INFO] - ===> Epoch[2957](14785/5): Loss 0.0124, Sem 0.0124 LR: 7.752e-02 Score 99.550 Data time: 0.2372, Forward time: 1.2636, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7847 [2021-07-07 22:00:42,809][root][INFO] - ===> Epoch[2958](14790/5): Loss 0.0127, Sem 0.0127 LR: 7.751e-02 Score 99.548 Data time: 0.2098, Forward time: 1.2637, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7529 [2021-07-07 22:00:56,890][root][INFO] - ===> Epoch[2959](14795/5): Loss 0.0133, Sem 0.0133 LR: 7.751e-02 Score 99.528 Data time: 0.2207, Forward time: 1.2637, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7749 [2021-07-07 22:01:10,879][root][INFO] - ===> Epoch[2960](14800/5): Loss 0.0129, Sem 0.0129 LR: 7.750e-02 Score 99.539 Data time: 0.1900, Forward time: 1.2637, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7511 [2021-07-07 22:01:10,880][root][INFO] - ===> Start testing [2021-07-07 22:01:46,593][root][INFO] - 68/68: Data time: 0.0016, Iter time: 0.1280 Loss 0.242 (AVG: 0.542) Score 94.555 (AVG: 91.251) mIOU 70.693 mAP 83.392 mAcc 77.090 IOU: 65.714 0.074 84.155 75.755 95.252 90.795 30.493 79.403 98.575 72.209 80.385 87.223 58.982 mAP: 84.200 6.453 92.866 88.190 98.767 98.496 54.824 97.781 99.784 84.761 92.012 94.663 91.302 mAcc: 81.567 0.547 90.077 82.503 96.397 97.228 32.883 96.840 99.412 75.365 90.548 96.734 62.066 [2021-07-07 22:01:46,593][root][INFO] - Finished test. Elapsed time: 35.7119 [2021-07-07 22:01:47,695][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 22:02:01,487][root][INFO] - ===> Epoch[2961](14805/5): Loss 0.0133, Sem 0.0133 LR: 7.749e-02 Score 99.522 Data time: 0.2412, Forward time: 1.2637, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7235 [2021-07-07 22:02:14,997][root][INFO] - ===> Epoch[2962](14810/5): Loss 0.0130, Sem 0.0130 LR: 7.748e-02 Score 99.534 Data time: 0.1739, Forward time: 1.2637, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6585 [2021-07-07 22:02:29,875][root][INFO] - ===> Epoch[2963](14815/5): Loss 0.0128, Sem 0.0128 LR: 7.747e-02 Score 99.536 Data time: 0.2619, Forward time: 1.2637, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9158 [2021-07-07 22:02:44,201][root][INFO] - ===> Epoch[2964](14820/5): Loss 0.0135, Sem 0.0135 LR: 7.747e-02 Score 99.516 Data time: 0.2360, Forward time: 1.2637, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8218 [2021-07-07 22:02:58,163][root][INFO] - ===> Epoch[2965](14825/5): Loss 0.0132, Sem 0.0132 LR: 7.746e-02 Score 99.522 Data time: 0.2579, Forward time: 1.2637, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7365 [2021-07-07 22:03:12,874][root][INFO] - ===> Epoch[2966](14830/5): Loss 0.0127, Sem 0.0127 LR: 7.745e-02 Score 99.541 Data time: 0.2208, Forward time: 1.2637, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8940 [2021-07-07 22:03:26,223][root][INFO] - ===> Epoch[2967](14835/5): Loss 0.0129, Sem 0.0129 LR: 7.744e-02 Score 99.539 Data time: 0.2393, Forward time: 1.2637, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6090 [2021-07-07 22:03:40,958][root][INFO] - ===> Epoch[2968](14840/5): Loss 0.0127, Sem 0.0127 LR: 7.744e-02 Score 99.542 Data time: 0.1896, Forward time: 1.2637, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9035 [2021-07-07 22:03:55,480][root][INFO] - ===> Epoch[2969](14845/5): Loss 0.0125, Sem 0.0125 LR: 7.743e-02 Score 99.547 Data time: 0.2337, Forward time: 1.2637, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8574 [2021-07-07 22:04:08,893][root][INFO] - ===> Epoch[2970](14850/5): Loss 0.0125, Sem 0.0125 LR: 7.742e-02 Score 99.545 Data time: 0.2347, Forward time: 1.2637, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6415 [2021-07-07 22:04:22,082][root][INFO] - ===> Epoch[2971](14855/5): Loss 0.0126, Sem 0.0126 LR: 7.741e-02 Score 99.545 Data time: 0.2629, Forward time: 1.2637, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.5954 [2021-07-07 22:04:35,644][root][INFO] - ===> Epoch[2972](14860/5): Loss 0.0124, Sem 0.0124 LR: 7.741e-02 Score 99.551 Data time: 0.2270, Forward time: 1.2636, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6647 [2021-07-07 22:04:49,313][root][INFO] - ===> Epoch[2973](14865/5): Loss 0.0119, Sem 0.0119 LR: 7.740e-02 Score 99.571 Data time: 0.1708, Forward time: 1.2636, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6906 [2021-07-07 22:05:03,094][root][INFO] - ===> Epoch[2974](14870/5): Loss 0.0123, Sem 0.0123 LR: 7.739e-02 Score 99.554 Data time: 0.2287, Forward time: 1.2636, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7049 [2021-07-07 22:05:16,734][root][INFO] - ===> Epoch[2975](14875/5): Loss 0.0123, Sem 0.0123 LR: 7.738e-02 Score 99.557 Data time: 0.1801, Forward time: 1.2636, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6874 [2021-07-07 22:05:30,972][root][INFO] - ===> Epoch[2976](14880/5): Loss 0.0123, Sem 0.0123 LR: 7.737e-02 Score 99.551 Data time: 0.3117, Forward time: 1.2635, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8018 [2021-07-07 22:05:45,512][root][INFO] - ===> Epoch[2977](14885/5): Loss 0.0119, Sem 0.0119 LR: 7.737e-02 Score 99.571 Data time: 0.2562, Forward time: 1.2636, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8640 [2021-07-07 22:05:59,925][root][INFO] - ===> Epoch[2978](14890/5): Loss 0.0123, Sem 0.0123 LR: 7.736e-02 Score 99.561 Data time: 0.2586, Forward time: 1.2636, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8230 [2021-07-07 22:06:13,174][root][INFO] - ===> Epoch[2979](14895/5): Loss 0.0133, Sem 0.0133 LR: 7.735e-02 Score 99.522 Data time: 0.2167, Forward time: 1.2636, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.5863 [2021-07-07 22:06:27,144][root][INFO] - ===> Epoch[2980](14900/5): Loss 0.0134, Sem 0.0134 LR: 7.734e-02 Score 99.515 Data time: 0.2169, Forward time: 1.2636, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7498 [2021-07-07 22:06:40,805][root][INFO] - ===> Epoch[2981](14905/5): Loss 0.0128, Sem 0.0128 LR: 7.734e-02 Score 99.539 Data time: 0.1938, Forward time: 1.2636, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6883 [2021-07-07 22:06:55,343][root][INFO] - ===> Epoch[2982](14910/5): Loss 0.0134, Sem 0.0134 LR: 7.733e-02 Score 99.521 Data time: 0.2730, Forward time: 1.2636, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8508 [2021-07-07 22:07:10,479][root][INFO] - ===> Epoch[2983](14915/5): Loss 0.0149, Sem 0.0149 LR: 7.732e-02 Score 99.483 Data time: 0.2522, Forward time: 1.2636, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9856 [2021-07-07 22:07:24,778][root][INFO] - ===> Epoch[2984](14920/5): Loss 0.0147, Sem 0.0147 LR: 7.731e-02 Score 99.474 Data time: 0.2771, Forward time: 1.2636, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8120 [2021-07-07 22:07:38,891][root][INFO] - ===> Epoch[2985](14925/5): Loss 0.0146, Sem 0.0146 LR: 7.731e-02 Score 99.480 Data time: 0.2105, Forward time: 1.2636, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7803 [2021-07-07 22:07:53,264][root][INFO] - ===> Epoch[2986](14930/5): Loss 0.0143, Sem 0.0143 LR: 7.730e-02 Score 99.484 Data time: 0.1380, Forward time: 1.2635, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8352 [2021-07-07 22:08:06,407][root][INFO] - ===> Epoch[2987](14935/5): Loss 0.0152, Sem 0.0152 LR: 7.729e-02 Score 99.456 Data time: 0.2730, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.5811 [2021-07-07 22:08:20,182][root][INFO] - ===> Epoch[2988](14940/5): Loss 0.0156, Sem 0.0156 LR: 7.728e-02 Score 99.442 Data time: 0.2242, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7072 [2021-07-07 22:08:34,779][root][INFO] - ===> Epoch[2989](14945/5): Loss 0.0156, Sem 0.0156 LR: 7.727e-02 Score 99.438 Data time: 0.2056, Forward time: 1.2635, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8676 [2021-07-07 22:08:48,657][root][INFO] - ===> Epoch[2990](14950/5): Loss 0.0157, Sem 0.0157 LR: 7.727e-02 Score 99.456 Data time: 0.2857, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7186 [2021-07-07 22:09:02,343][root][INFO] - ===> Epoch[2991](14955/5): Loss 0.0159, Sem 0.0159 LR: 7.726e-02 Score 99.433 Data time: 0.2452, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6993 [2021-07-07 22:09:16,721][root][INFO] - ===> Epoch[2992](14960/5): Loss 0.0185, Sem 0.0185 LR: 7.725e-02 Score 99.361 Data time: 0.2506, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8094 [2021-07-07 22:09:31,753][root][INFO] - ===> Epoch[2993](14965/5): Loss 0.0211, Sem 0.0211 LR: 7.724e-02 Score 99.283 Data time: 0.1947, Forward time: 1.2634, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.9613 [2021-07-07 22:09:45,721][root][INFO] - ===> Epoch[2994](14970/5): Loss 0.0220, Sem 0.0220 LR: 7.724e-02 Score 99.238 Data time: 0.2562, Forward time: 1.2634, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7452 [2021-07-07 22:09:59,603][root][INFO] - ===> Epoch[2995](14975/5): Loss 0.0215, Sem 0.0215 LR: 7.723e-02 Score 99.236 Data time: 0.2163, Forward time: 1.2634, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7301 [2021-07-07 22:10:14,153][root][INFO] - ===> Epoch[2996](14980/5): Loss 0.0216, Sem 0.0216 LR: 7.722e-02 Score 99.254 Data time: 0.2336, Forward time: 1.2634, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8649 [2021-07-07 22:10:27,809][root][INFO] - ===> Epoch[2997](14985/5): Loss 0.0226, Sem 0.0226 LR: 7.721e-02 Score 99.208 Data time: 0.1673, Forward time: 1.2634, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6864 [2021-07-07 22:10:42,125][root][INFO] - ===> Epoch[2998](14990/5): Loss 0.0227, Sem 0.0227 LR: 7.720e-02 Score 99.219 Data time: 0.2310, Forward time: 1.2634, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8080 [2021-07-07 22:10:55,827][root][INFO] - ===> Epoch[2999](14995/5): Loss 0.0238, Sem 0.0238 LR: 7.720e-02 Score 99.171 Data time: 0.2408, Forward time: 1.2634, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6828 [2021-07-07 22:11:10,250][root][INFO] - ===> Epoch[3000](15000/5): Loss 0.0245, Sem 0.0245 LR: 7.719e-02 Score 99.154 Data time: 0.2297, Forward time: 1.2634, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8314 [2021-07-07 22:11:10,251][root][INFO] - ===> Start testing [2021-07-07 22:11:48,901][root][INFO] - 68/68: Data time: 0.0018, Iter time: 0.1276 Loss 0.613 (AVG: 0.551) Score 87.082 (AVG: 90.050) mIOU 67.739 mAP 83.565 mAcc 76.782 IOU: 62.692 0.000 68.730 73.088 94.203 88.194 35.678 80.240 98.363 61.139 76.464 86.148 55.669 mAP: 81.738 3.650 93.840 86.630 98.693 97.951 64.332 96.760 99.704 87.476 91.310 94.229 90.033 mAcc: 83.054 0.000 92.628 80.766 95.221 91.425 43.106 92.376 99.388 76.707 86.935 94.132 62.424 [2021-07-07 22:11:48,901][root][INFO] - Finished test. Elapsed time: 38.6491 [2021-07-07 22:11:49,971][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 22:12:03,552][root][INFO] - ===> Epoch[3001](15005/5): Loss 0.0265, Sem 0.0265 LR: 7.718e-02 Score 99.108 Data time: 0.2078, Forward time: 1.2634, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6792 [2021-07-07 22:12:17,126][root][INFO] - ===> Epoch[3002](15010/5): Loss 0.0265, Sem 0.0265 LR: 7.717e-02 Score 99.095 Data time: 0.2773, Forward time: 1.2634, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.6671 [2021-07-07 22:12:31,973][root][INFO] - ===> Epoch[3003](15015/5): Loss 0.0292, Sem 0.0292 LR: 7.717e-02 Score 99.027 Data time: 0.2057, Forward time: 1.2634, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.9233 [2021-07-07 22:12:46,440][root][INFO] - ===> Epoch[3004](15020/5): Loss 0.0346, Sem 0.0346 LR: 7.716e-02 Score 98.842 Data time: 0.2390, Forward time: 1.2634, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8427 [2021-07-07 22:13:00,925][root][INFO] - ===> Epoch[3005](15025/5): Loss 0.0437, Sem 0.0437 LR: 7.715e-02 Score 98.594 Data time: 0.2283, Forward time: 1.2634, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8581 [2021-07-07 22:13:14,892][root][INFO] - ===> Epoch[3006](15030/5): Loss 0.0614, Sem 0.0614 LR: 7.714e-02 Score 98.118 Data time: 0.2399, Forward time: 1.2634, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7502 [2021-07-07 22:13:29,280][root][INFO] - ===> Epoch[3007](15035/5): Loss 0.0585, Sem 0.0585 LR: 7.714e-02 Score 98.081 Data time: 0.2084, Forward time: 1.2634, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8360 [2021-07-07 22:13:43,991][root][INFO] - ===> Epoch[3008](15040/5): Loss 0.0636, Sem 0.0636 LR: 7.713e-02 Score 97.959 Data time: 0.2182, Forward time: 1.2635, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.9055 [2021-07-07 22:13:58,155][root][INFO] - ===> Epoch[3009](15045/5): Loss 0.0719, Sem 0.0719 LR: 7.712e-02 Score 97.805 Data time: 0.2216, Forward time: 1.2635, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7862 [2021-07-07 22:14:12,219][root][INFO] - ===> Epoch[3010](15050/5): Loss 0.0733, Sem 0.0733 LR: 7.711e-02 Score 97.645 Data time: 0.2171, Forward time: 1.2635, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7735 [2021-07-07 22:14:25,806][root][INFO] - ===> Epoch[3011](15055/5): Loss 0.0772, Sem 0.0772 LR: 7.710e-02 Score 97.604 Data time: 0.2276, Forward time: 1.2635, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6756 [2021-07-07 22:14:39,252][root][INFO] - ===> Epoch[3012](15060/5): Loss 0.0787, Sem 0.0787 LR: 7.710e-02 Score 97.475 Data time: 0.1859, Forward time: 1.2635, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6547 [2021-07-07 22:14:53,376][root][INFO] - ===> Epoch[3013](15065/5): Loss 0.0816, Sem 0.0816 LR: 7.709e-02 Score 97.489 Data time: 0.1989, Forward time: 1.2635, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7695 [2021-07-07 22:15:06,683][root][INFO] - ===> Epoch[3014](15070/5): Loss 0.0836, Sem 0.0836 LR: 7.708e-02 Score 97.386 Data time: 0.1964, Forward time: 1.2635, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6185 [2021-07-07 22:15:20,189][root][INFO] - ===> Epoch[3015](15075/5): Loss 0.0860, Sem 0.0860 LR: 7.707e-02 Score 97.250 Data time: 0.2649, Forward time: 1.2635, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.6384 [2021-07-07 22:15:33,852][root][INFO] - ===> Epoch[3016](15080/5): Loss 0.0813, Sem 0.0813 LR: 7.707e-02 Score 97.353 Data time: 0.2549, Forward time: 1.2636, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6790 [2021-07-07 22:15:48,290][root][INFO] - ===> Epoch[3017](15085/5): Loss 0.0751, Sem 0.0751 LR: 7.706e-02 Score 97.601 Data time: 0.2702, Forward time: 1.2636, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8359 [2021-07-07 22:16:01,866][root][INFO] - ===> Epoch[3018](15090/5): Loss 0.0752, Sem 0.0752 LR: 7.705e-02 Score 97.564 Data time: 0.2443, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6739 [2021-07-07 22:16:16,063][root][INFO] - ===> Epoch[3019](15095/5): Loss 0.0644, Sem 0.0644 LR: 7.704e-02 Score 97.910 Data time: 0.2974, Forward time: 1.2636, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7853 [2021-07-07 22:16:30,376][root][INFO] - ===> Epoch[3020](15100/5): Loss 0.0694, Sem 0.0694 LR: 7.703e-02 Score 97.711 Data time: 0.2422, Forward time: 1.2636, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8166 [2021-07-07 22:16:43,487][root][INFO] - ===> Epoch[3021](15105/5): Loss 0.0646, Sem 0.0646 LR: 7.703e-02 Score 97.906 Data time: 0.2008, Forward time: 1.2636, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.5795 [2021-07-07 22:16:57,878][root][INFO] - ===> Epoch[3022](15110/5): Loss 0.0708, Sem 0.0708 LR: 7.702e-02 Score 97.764 Data time: 0.2277, Forward time: 1.2636, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8430 [2021-07-07 22:17:12,094][root][INFO] - ===> Epoch[3023](15115/5): Loss 0.0664, Sem 0.0664 LR: 7.701e-02 Score 97.813 Data time: 0.2079, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7861 [2021-07-07 22:17:25,533][root][INFO] - ===> Epoch[3024](15120/5): Loss 0.0549, Sem 0.0549 LR: 7.700e-02 Score 98.171 Data time: 0.1945, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6472 [2021-07-07 22:17:39,721][root][INFO] - ===> Epoch[3025](15125/5): Loss 0.0543, Sem 0.0543 LR: 7.700e-02 Score 98.218 Data time: 0.2359, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7975 [2021-07-07 22:17:53,710][root][INFO] - ===> Epoch[3026](15130/5): Loss 0.0552, Sem 0.0552 LR: 7.699e-02 Score 98.190 Data time: 0.2304, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7483 [2021-07-07 22:18:07,497][root][INFO] - ===> Epoch[3027](15135/5): Loss 0.0561, Sem 0.0561 LR: 7.698e-02 Score 98.203 Data time: 0.2737, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7044 [2021-07-07 22:18:21,273][root][INFO] - ===> Epoch[3028](15140/5): Loss 0.0490, Sem 0.0490 LR: 7.697e-02 Score 98.363 Data time: 0.2078, Forward time: 1.2635, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7174 [2021-07-07 22:18:34,941][root][INFO] - ===> Epoch[3029](15145/5): Loss 0.0515, Sem 0.0515 LR: 7.697e-02 Score 98.294 Data time: 0.2340, Forward time: 1.2634, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6809 [2021-07-07 22:18:48,911][root][INFO] - ===> Epoch[3030](15150/5): Loss 0.0503, Sem 0.0503 LR: 7.696e-02 Score 98.345 Data time: 0.2150, Forward time: 1.2634, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7414 [2021-07-07 22:19:02,844][root][INFO] - ===> Epoch[3031](15155/5): Loss 0.0546, Sem 0.0546 LR: 7.695e-02 Score 98.198 Data time: 0.1979, Forward time: 1.2634, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7371 [2021-07-07 22:19:17,209][root][INFO] - ===> Epoch[3032](15160/5): Loss 0.0486, Sem 0.0486 LR: 7.694e-02 Score 98.354 Data time: 0.2429, Forward time: 1.2634, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8139 [2021-07-07 22:19:31,101][root][INFO] - ===> Epoch[3033](15165/5): Loss 0.0524, Sem 0.0524 LR: 7.693e-02 Score 98.267 Data time: 0.2429, Forward time: 1.2634, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7279 [2021-07-07 22:19:44,775][root][INFO] - ===> Epoch[3034](15170/5): Loss 0.0517, Sem 0.0517 LR: 7.693e-02 Score 98.273 Data time: 0.2110, Forward time: 1.2634, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6951 [2021-07-07 22:19:58,138][root][INFO] - ===> Epoch[3035](15175/5): Loss 0.0595, Sem 0.0595 LR: 7.692e-02 Score 98.066 Data time: 0.2473, Forward time: 1.2634, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6225 [2021-07-07 22:20:12,348][root][INFO] - ===> Epoch[3036](15180/5): Loss 0.0512, Sem 0.0512 LR: 7.691e-02 Score 98.294 Data time: 0.2264, Forward time: 1.2634, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7997 [2021-07-07 22:20:26,289][root][INFO] - ===> Epoch[3037](15185/5): Loss 0.0473, Sem 0.0473 LR: 7.690e-02 Score 98.384 Data time: 0.2639, Forward time: 1.2634, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7409 [2021-07-07 22:20:40,646][root][INFO] - ===> Epoch[3038](15190/5): Loss 0.0486, Sem 0.0486 LR: 7.690e-02 Score 98.397 Data time: 0.2448, Forward time: 1.2635, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8232 [2021-07-07 22:20:54,820][root][INFO] - ===> Epoch[3039](15195/5): Loss 0.0478, Sem 0.0478 LR: 7.689e-02 Score 98.362 Data time: 0.2834, Forward time: 1.2635, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7892 [2021-07-07 22:21:08,671][root][INFO] - ===> Epoch[3040](15200/5): Loss 0.0471, Sem 0.0471 LR: 7.688e-02 Score 98.433 Data time: 0.2042, Forward time: 1.2635, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7182 [2021-07-07 22:21:08,674][root][INFO] - ===> Start testing [2021-07-07 22:21:43,892][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1198 Loss 0.299 (AVG: 0.467) Score 91.255 (AVG: 89.834) mIOU 68.303 mAP 82.688 mAcc 74.675 IOU: 61.060 0.000 78.772 73.219 93.864 91.118 29.033 82.911 98.463 65.765 73.370 84.248 56.120 mAP: 83.324 2.984 90.735 90.617 98.740 98.908 57.022 95.146 99.830 82.806 91.101 94.846 88.890 mAcc: 83.000 0.000 85.231 78.296 94.570 97.534 31.434 85.940 99.552 76.490 81.409 97.032 60.290 [2021-07-07 22:21:43,892][root][INFO] - Finished test. Elapsed time: 35.2177 [2021-07-07 22:21:44,934][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 22:21:58,571][root][INFO] - ===> Epoch[3041](15205/5): Loss 0.0430, Sem 0.0430 LR: 7.687e-02 Score 98.543 Data time: 0.1980, Forward time: 1.2635, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6766 [2021-07-07 22:22:12,980][root][INFO] - ===> Epoch[3042](15210/5): Loss 0.0452, Sem 0.0452 LR: 7.687e-02 Score 98.443 Data time: 0.1907, Forward time: 1.2634, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8343 [2021-07-07 22:22:26,444][root][INFO] - ===> Epoch[3043](15215/5): Loss 0.0431, Sem 0.0431 LR: 7.686e-02 Score 98.546 Data time: 0.2198, Forward time: 1.2634, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6472 [2021-07-07 22:22:40,805][root][INFO] - ===> Epoch[3044](15220/5): Loss 0.0420, Sem 0.0420 LR: 7.685e-02 Score 98.618 Data time: 0.1965, Forward time: 1.2633, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8208 [2021-07-07 22:22:54,118][root][INFO] - ===> Epoch[3045](15225/5): Loss 0.0382, Sem 0.0382 LR: 7.684e-02 Score 98.677 Data time: 0.2308, Forward time: 1.2633, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6057 [2021-07-07 22:23:08,586][root][INFO] - ===> Epoch[3046](15230/5): Loss 0.0397, Sem 0.0397 LR: 7.683e-02 Score 98.643 Data time: 0.2027, Forward time: 1.2633, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8483 [2021-07-07 22:23:22,637][root][INFO] - ===> Epoch[3047](15235/5): Loss 0.0356, Sem 0.0356 LR: 7.683e-02 Score 98.824 Data time: 0.1742, Forward time: 1.2633, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7672 [2021-07-07 22:23:37,067][root][INFO] - ===> Epoch[3048](15240/5): Loss 0.0363, Sem 0.0363 LR: 7.682e-02 Score 98.787 Data time: 0.1939, Forward time: 1.2633, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8415 [2021-07-07 22:23:51,981][root][INFO] - ===> Epoch[3049](15245/5): Loss 0.0357, Sem 0.0357 LR: 7.681e-02 Score 98.774 Data time: 0.2573, Forward time: 1.2633, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.9301 [2021-07-07 22:24:06,355][root][INFO] - ===> Epoch[3050](15250/5): Loss 0.0344, Sem 0.0344 LR: 7.680e-02 Score 98.816 Data time: 0.1998, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8348 [2021-07-07 22:24:20,312][root][INFO] - ===> Epoch[3051](15255/5): Loss 0.0330, Sem 0.0330 LR: 7.680e-02 Score 98.868 Data time: 0.2462, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7504 [2021-07-07 22:24:34,430][root][INFO] - ===> Epoch[3052](15260/5): Loss 0.0296, Sem 0.0296 LR: 7.679e-02 Score 98.953 Data time: 0.2410, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7688 [2021-07-07 22:24:48,487][root][INFO] - ===> Epoch[3053](15265/5): Loss 0.0273, Sem 0.0273 LR: 7.678e-02 Score 99.033 Data time: 0.2391, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7650 [2021-07-07 22:25:03,289][root][INFO] - ===> Epoch[3054](15270/5): Loss 0.0274, Sem 0.0274 LR: 7.677e-02 Score 99.029 Data time: 0.2644, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.9110 [2021-07-07 22:25:17,955][root][INFO] - ===> Epoch[3055](15275/5): Loss 0.0269, Sem 0.0269 LR: 7.676e-02 Score 99.064 Data time: 0.2716, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8916 [2021-07-07 22:25:32,516][root][INFO] - ===> Epoch[3056](15280/5): Loss 0.0271, Sem 0.0271 LR: 7.676e-02 Score 99.063 Data time: 0.2314, Forward time: 1.2633, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8612 [2021-07-07 22:25:45,794][root][INFO] - ===> Epoch[3057](15285/5): Loss 0.0277, Sem 0.0277 LR: 7.675e-02 Score 99.016 Data time: 0.2586, Forward time: 1.2633, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6074 [2021-07-07 22:26:00,514][root][INFO] - ===> Epoch[3058](15290/5): Loss 0.0247, Sem 0.0247 LR: 7.674e-02 Score 99.141 Data time: 0.2188, Forward time: 1.2633, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.9065 [2021-07-07 22:26:13,811][root][INFO] - ===> Epoch[3059](15295/5): Loss 0.0240, Sem 0.0240 LR: 7.673e-02 Score 99.150 Data time: 0.2073, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6064 [2021-07-07 22:26:27,915][root][INFO] - ===> Epoch[3060](15300/5): Loss 0.0228, Sem 0.0228 LR: 7.673e-02 Score 99.195 Data time: 0.2356, Forward time: 1.2633, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7811 [2021-07-07 22:26:41,884][root][INFO] - ===> Epoch[3061](15305/5): Loss 0.0227, Sem 0.0227 LR: 7.672e-02 Score 99.190 Data time: 0.2048, Forward time: 1.2633, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7510 [2021-07-07 22:26:55,973][root][INFO] - ===> Epoch[3062](15310/5): Loss 0.0230, Sem 0.0230 LR: 7.671e-02 Score 99.175 Data time: 0.1898, Forward time: 1.2633, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7826 [2021-07-07 22:27:09,168][root][INFO] - ===> Epoch[3063](15315/5): Loss 0.0218, Sem 0.0218 LR: 7.670e-02 Score 99.222 Data time: 0.2161, Forward time: 1.2633, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.5955 [2021-07-07 22:27:23,346][root][INFO] - ===> Epoch[3064](15320/5): Loss 0.0203, Sem 0.0203 LR: 7.670e-02 Score 99.276 Data time: 0.2089, Forward time: 1.2633, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7934 [2021-07-07 22:27:37,040][root][INFO] - ===> Epoch[3065](15325/5): Loss 0.0216, Sem 0.0216 LR: 7.669e-02 Score 99.235 Data time: 0.2128, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7024 [2021-07-07 22:27:50,770][root][INFO] - ===> Epoch[3066](15330/5): Loss 0.0223, Sem 0.0223 LR: 7.668e-02 Score 99.230 Data time: 0.3015, Forward time: 1.2632, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6971 [2021-07-07 22:28:05,524][root][INFO] - ===> Epoch[3067](15335/5): Loss 0.0224, Sem 0.0224 LR: 7.667e-02 Score 99.213 Data time: 0.2556, Forward time: 1.2633, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.9092 [2021-07-07 22:28:19,328][root][INFO] - ===> Epoch[3068](15340/5): Loss 0.0227, Sem 0.0227 LR: 7.666e-02 Score 99.193 Data time: 0.1992, Forward time: 1.2633, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7214 [2021-07-07 22:28:33,382][root][INFO] - ===> Epoch[3069](15345/5): Loss 0.0212, Sem 0.0212 LR: 7.666e-02 Score 99.241 Data time: 0.1983, Forward time: 1.2633, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7608 [2021-07-07 22:28:47,454][root][INFO] - ===> Epoch[3070](15350/5): Loss 0.0214, Sem 0.0214 LR: 7.665e-02 Score 99.241 Data time: 0.2469, Forward time: 1.2633, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7615 [2021-07-07 22:29:02,965][root][INFO] - ===> Epoch[3071](15355/5): Loss 0.0205, Sem 0.0205 LR: 7.664e-02 Score 99.270 Data time: 0.1989, Forward time: 1.2633, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 3.0432 [2021-07-07 22:29:16,515][root][INFO] - ===> Epoch[3072](15360/5): Loss 0.0195, Sem 0.0195 LR: 7.663e-02 Score 99.301 Data time: 0.1786, Forward time: 1.2633, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6662 [2021-07-07 22:29:30,414][root][INFO] - ===> Epoch[3073](15365/5): Loss 0.0210, Sem 0.0210 LR: 7.663e-02 Score 99.248 Data time: 0.2583, Forward time: 1.2633, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7331 [2021-07-07 22:29:43,978][root][INFO] - ===> Epoch[3074](15370/5): Loss 0.0206, Sem 0.0206 LR: 7.662e-02 Score 99.267 Data time: 0.2507, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6694 [2021-07-07 22:29:58,660][root][INFO] - ===> Epoch[3075](15375/5): Loss 0.0202, Sem 0.0202 LR: 7.661e-02 Score 99.285 Data time: 0.2275, Forward time: 1.2633, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8896 [2021-07-07 22:30:12,788][root][INFO] - ===> Epoch[3076](15380/5): Loss 0.0199, Sem 0.0199 LR: 7.660e-02 Score 99.292 Data time: 0.2819, Forward time: 1.2633, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7797 [2021-07-07 22:30:26,977][root][INFO] - ===> Epoch[3077](15385/5): Loss 0.0192, Sem 0.0192 LR: 7.659e-02 Score 99.305 Data time: 0.2095, Forward time: 1.2633, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7970 [2021-07-07 22:30:40,974][root][INFO] - ===> Epoch[3078](15390/5): Loss 0.0197, Sem 0.0197 LR: 7.659e-02 Score 99.301 Data time: 0.2081, Forward time: 1.2633, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7501 [2021-07-07 22:30:54,594][root][INFO] - ===> Epoch[3079](15395/5): Loss 0.0203, Sem 0.0203 LR: 7.658e-02 Score 99.276 Data time: 0.2899, Forward time: 1.2633, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6724 [2021-07-07 22:31:09,157][root][INFO] - ===> Epoch[3080](15400/5): Loss 0.0196, Sem 0.0196 LR: 7.657e-02 Score 99.300 Data time: 0.1881, Forward time: 1.2633, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8723 [2021-07-07 22:31:09,158][root][INFO] - ===> Start testing [2021-07-07 22:31:44,946][root][INFO] - 68/68: Data time: 0.0078, Iter time: 0.1245 Loss 0.242 (AVG: 0.472) Score 93.369 (AVG: 91.189) mIOU 69.712 mAP 84.613 mAcc 75.827 IOU: 63.456 0.000 83.561 76.192 95.437 90.582 25.996 83.504 98.468 61.620 78.882 87.268 61.286 mAP: 85.125 6.451 93.961 90.615 98.874 98.943 63.509 97.414 99.790 83.420 93.592 95.688 92.583 mAcc: 78.660 0.000 90.930 85.093 96.713 97.133 28.512 96.542 99.527 62.362 89.687 96.493 64.097 [2021-07-07 22:31:44,946][root][INFO] - Finished test. Elapsed time: 35.7865 [2021-07-07 22:31:45,992][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 22:31:59,664][root][INFO] - ===> Epoch[3081](15405/5): Loss 0.0212, Sem 0.0212 LR: 7.656e-02 Score 99.247 Data time: 0.2094, Forward time: 1.2632, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7033 [2021-07-07 22:32:12,819][root][INFO] - ===> Epoch[3082](15410/5): Loss 0.0207, Sem 0.0207 LR: 7.656e-02 Score 99.257 Data time: 0.1870, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.5883 [2021-07-07 22:32:26,887][root][INFO] - ===> Epoch[3083](15415/5): Loss 0.0195, Sem 0.0195 LR: 7.655e-02 Score 99.303 Data time: 0.2615, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7616 [2021-07-07 22:32:40,283][root][INFO] - ===> Epoch[3084](15420/5): Loss 0.0203, Sem 0.0203 LR: 7.654e-02 Score 99.294 Data time: 0.2744, Forward time: 1.2632, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6325 [2021-07-07 22:32:54,137][root][INFO] - ===> Epoch[3085](15425/5): Loss 0.0209, Sem 0.0209 LR: 7.653e-02 Score 99.283 Data time: 0.1955, Forward time: 1.2632, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7275 [2021-07-07 22:33:07,185][root][INFO] - ===> Epoch[3086](15430/5): Loss 0.0245, Sem 0.0245 LR: 7.653e-02 Score 99.200 Data time: 0.1918, Forward time: 1.2632, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.5687 [2021-07-07 22:33:21,678][root][INFO] - ===> Epoch[3087](15435/5): Loss 0.0226, Sem 0.0226 LR: 7.652e-02 Score 99.196 Data time: 0.2042, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8464 [2021-07-07 22:33:35,906][root][INFO] - ===> Epoch[3088](15440/5): Loss 0.0221, Sem 0.0221 LR: 7.651e-02 Score 99.221 Data time: 0.2652, Forward time: 1.2632, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7826 [2021-07-07 22:33:49,973][root][INFO] - ===> Epoch[3089](15445/5): Loss 0.0246, Sem 0.0246 LR: 7.650e-02 Score 99.158 Data time: 0.2179, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7713 [2021-07-07 22:34:04,122][root][INFO] - ===> Epoch[3090](15450/5): Loss 0.0267, Sem 0.0267 LR: 7.649e-02 Score 99.088 Data time: 0.2047, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7755 [2021-07-07 22:34:18,797][root][INFO] - ===> Epoch[3091](15455/5): Loss 0.0259, Sem 0.0259 LR: 7.649e-02 Score 99.091 Data time: 0.2114, Forward time: 1.2632, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.8896 [2021-07-07 22:34:33,628][root][INFO] - ===> Epoch[3092](15460/5): Loss 0.0263, Sem 0.0263 LR: 7.648e-02 Score 99.072 Data time: 0.2414, Forward time: 1.2632, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.9191 [2021-07-07 22:34:47,640][root][INFO] - ===> Epoch[3093](15465/5): Loss 0.0259, Sem 0.0259 LR: 7.647e-02 Score 99.084 Data time: 0.1974, Forward time: 1.2632, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7599 [2021-07-07 22:35:02,184][root][INFO] - ===> Epoch[3094](15470/5): Loss 0.0253, Sem 0.0253 LR: 7.646e-02 Score 99.114 Data time: 0.2327, Forward time: 1.2632, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.8606 [2021-07-07 22:35:16,196][root][INFO] - ===> Epoch[3095](15475/5): Loss 0.0226, Sem 0.0226 LR: 7.646e-02 Score 99.202 Data time: 0.1809, Forward time: 1.2632, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7577 [2021-07-07 22:35:29,859][root][INFO] - ===> Epoch[3096](15480/5): Loss 0.0227, Sem 0.0227 LR: 7.645e-02 Score 99.205 Data time: 0.2284, Forward time: 1.2632, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.6866 [2021-07-07 22:35:43,189][root][INFO] - ===> Epoch[3097](15485/5): Loss 0.0221, Sem 0.0221 LR: 7.644e-02 Score 99.224 Data time: 0.2852, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6124 [2021-07-07 22:35:56,863][root][INFO] - ===> Epoch[3098](15490/5): Loss 0.0217, Sem 0.0217 LR: 7.643e-02 Score 99.227 Data time: 0.2467, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6877 [2021-07-07 22:36:10,522][root][INFO] - ===> Epoch[3099](15495/5): Loss 0.0211, Sem 0.0211 LR: 7.642e-02 Score 99.245 Data time: 0.2229, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6916 [2021-07-07 22:36:24,191][root][INFO] - ===> Epoch[3100](15500/5): Loss 0.0216, Sem 0.0216 LR: 7.642e-02 Score 99.227 Data time: 0.2395, Forward time: 1.2632, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6896 [2021-07-07 22:36:38,229][root][INFO] - ===> Epoch[3101](15505/5): Loss 0.0204, Sem 0.0204 LR: 7.641e-02 Score 99.269 Data time: 0.2085, Forward time: 1.2631, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7625 [2021-07-07 22:36:52,683][root][INFO] - ===> Epoch[3102](15510/5): Loss 0.0212, Sem 0.0212 LR: 7.640e-02 Score 99.259 Data time: 0.2037, Forward time: 1.2631, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.8466 [2021-07-07 22:37:05,779][root][INFO] - ===> Epoch[3103](15515/5): Loss 0.0247, Sem 0.0247 LR: 7.639e-02 Score 99.149 Data time: 0.2420, Forward time: 1.2631, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.5796 [2021-07-07 22:37:19,972][root][INFO] - ===> Epoch[3104](15520/5): Loss 0.0214, Sem 0.0214 LR: 7.639e-02 Score 99.245 Data time: 0.2263, Forward time: 1.2631, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7912 [2021-07-07 22:37:34,612][root][INFO] - ===> Epoch[3105](15525/5): Loss 0.0211, Sem 0.0211 LR: 7.638e-02 Score 99.252 Data time: 0.2179, Forward time: 1.2631, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.8712 [2021-07-07 22:37:49,433][root][INFO] - ===> Epoch[3106](15530/5): Loss 0.0212, Sem 0.0212 LR: 7.637e-02 Score 99.270 Data time: 0.2150, Forward time: 1.2631, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.9190 [2021-07-07 22:38:03,810][root][INFO] - ===> Epoch[3107](15535/5): Loss 0.0205, Sem 0.0205 LR: 7.636e-02 Score 99.264 Data time: 0.2125, Forward time: 1.2631, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.8337 [2021-07-07 22:38:17,408][root][INFO] - ===> Epoch[3108](15540/5): Loss 0.0195, Sem 0.0195 LR: 7.636e-02 Score 99.301 Data time: 0.1997, Forward time: 1.2630, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.6818 [2021-07-07 22:38:31,645][root][INFO] - ===> Epoch[3109](15545/5): Loss 0.0199, Sem 0.0199 LR: 7.635e-02 Score 99.294 Data time: 0.1579, Forward time: 1.2630, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8095 [2021-07-07 22:38:44,922][root][INFO] - ===> Epoch[3110](15550/5): Loss 0.0193, Sem 0.0193 LR: 7.634e-02 Score 99.308 Data time: 0.1997, Forward time: 1.2630, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6042 [2021-07-07 22:38:58,891][root][INFO] - ===> Epoch[3111](15555/5): Loss 0.0189, Sem 0.0189 LR: 7.633e-02 Score 99.320 Data time: 0.1982, Forward time: 1.2630, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7478 [2021-07-07 22:39:13,176][root][INFO] - ===> Epoch[3112](15560/5): Loss 0.0181, Sem 0.0181 LR: 7.632e-02 Score 99.359 Data time: 0.2213, Forward time: 1.2630, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8157 [2021-07-07 22:39:27,225][root][INFO] - ===> Epoch[3113](15565/5): Loss 0.0187, Sem 0.0187 LR: 7.632e-02 Score 99.328 Data time: 0.2824, Forward time: 1.2629, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7554 [2021-07-07 22:39:41,127][root][INFO] - ===> Epoch[3114](15570/5): Loss 0.0175, Sem 0.0175 LR: 7.631e-02 Score 99.370 Data time: 0.3132, Forward time: 1.2629, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7229 [2021-07-07 22:39:56,770][root][INFO] - ===> Epoch[3115](15575/5): Loss 0.0181, Sem 0.0181 LR: 7.630e-02 Score 99.349 Data time: 0.2373, Forward time: 1.2629, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 3.0776 [2021-07-07 22:40:10,880][root][INFO] - ===> Epoch[3116](15580/5): Loss 0.0175, Sem 0.0175 LR: 7.629e-02 Score 99.377 Data time: 0.2599, Forward time: 1.2629, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7723 [2021-07-07 22:40:23,879][root][INFO] - ===> Epoch[3117](15585/5): Loss 0.0172, Sem 0.0172 LR: 7.629e-02 Score 99.387 Data time: 0.2473, Forward time: 1.2630, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.5569 [2021-07-07 22:40:38,460][root][INFO] - ===> Epoch[3118](15590/5): Loss 0.0172, Sem 0.0172 LR: 7.628e-02 Score 99.388 Data time: 0.2260, Forward time: 1.2629, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8718 [2021-07-07 22:40:52,218][root][INFO] - ===> Epoch[3119](15595/5): Loss 0.0175, Sem 0.0175 LR: 7.627e-02 Score 99.381 Data time: 0.1804, Forward time: 1.2629, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7052 [2021-07-07 22:41:05,912][root][INFO] - ===> Epoch[3120](15600/5): Loss 0.0189, Sem 0.0189 LR: 7.626e-02 Score 99.324 Data time: 0.2018, Forward time: 1.2629, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.6931 [2021-07-07 22:41:05,913][root][INFO] - ===> Start testing [2021-07-07 22:41:41,319][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1239 Loss 0.195 (AVG: 0.494) Score 93.951 (AVG: 91.245) mIOU 70.739 mAP 84.434 mAcc 76.715 IOU: 65.941 0.000 80.923 76.377 95.665 92.007 26.777 79.560 98.570 78.748 80.556 86.634 57.848 mAP: 85.893 6.125 91.582 89.749 98.928 98.760 56.112 96.341 99.752 95.131 93.269 95.125 90.871 mAcc: 80.077 0.000 85.329 84.104 97.397 97.590 29.241 91.050 99.553 83.060 91.054 96.714 62.126 [2021-07-07 22:41:41,320][root][INFO] - Finished test. Elapsed time: 35.4057 [2021-07-07 22:41:42,415][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 22:41:55,403][root][INFO] - ===> Epoch[3121](15605/5): Loss 0.0179, Sem 0.0179 LR: 7.625e-02 Score 99.361 Data time: 0.2324, Forward time: 1.2629, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.5552 [2021-07-07 22:42:08,899][root][INFO] - ===> Epoch[3122](15610/5): Loss 0.0178, Sem 0.0178 LR: 7.625e-02 Score 99.364 Data time: 0.1945, Forward time: 1.2629, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6555 [2021-07-07 22:42:23,625][root][INFO] - ===> Epoch[3123](15615/5): Loss 0.0167, Sem 0.0167 LR: 7.624e-02 Score 99.397 Data time: 0.1873, Forward time: 1.2628, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.8984 [2021-07-07 22:42:37,603][root][INFO] - ===> Epoch[3124](15620/5): Loss 0.0176, Sem 0.0176 LR: 7.623e-02 Score 99.359 Data time: 0.2184, Forward time: 1.2628, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7521 [2021-07-07 22:42:51,499][root][INFO] - ===> Epoch[3125](15625/5): Loss 0.0171, Sem 0.0171 LR: 7.622e-02 Score 99.381 Data time: 0.2208, Forward time: 1.2627, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7340 [2021-07-07 22:43:05,681][root][INFO] - ===> Epoch[3126](15630/5): Loss 0.0166, Sem 0.0166 LR: 7.622e-02 Score 99.401 Data time: 0.2206, Forward time: 1.2627, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7749 [2021-07-07 22:43:19,369][root][INFO] - ===> Epoch[3127](15635/5): Loss 0.0159, Sem 0.0159 LR: 7.621e-02 Score 99.428 Data time: 0.2619, Forward time: 1.2627, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.6814 [2021-07-07 22:43:33,244][root][INFO] - ===> Epoch[3128](15640/5): Loss 0.0167, Sem 0.0167 LR: 7.620e-02 Score 99.392 Data time: 0.2704, Forward time: 1.2627, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7185 [2021-07-07 22:43:47,789][root][INFO] - ===> Epoch[3129](15645/5): Loss 0.0163, Sem 0.0163 LR: 7.619e-02 Score 99.420 Data time: 0.2844, Forward time: 1.2627, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.8583 [2021-07-07 22:44:02,502][root][INFO] - ===> Epoch[3130](15650/5): Loss 0.0165, Sem 0.0165 LR: 7.619e-02 Score 99.407 Data time: 0.2151, Forward time: 1.2628, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.9001 [2021-07-07 22:44:16,576][root][INFO] - ===> Epoch[3131](15655/5): Loss 0.0167, Sem 0.0167 LR: 7.618e-02 Score 99.396 Data time: 0.1853, Forward time: 1.2627, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7710 [2021-07-07 22:44:30,485][root][INFO] - ===> Epoch[3132](15660/5): Loss 0.0163, Sem 0.0163 LR: 7.617e-02 Score 99.410 Data time: 0.2586, Forward time: 1.2628, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7241 [2021-07-07 22:44:45,099][root][INFO] - ===> Epoch[3133](15665/5): Loss 0.0163, Sem 0.0163 LR: 7.616e-02 Score 99.410 Data time: 0.2527, Forward time: 1.2628, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8745 [2021-07-07 22:44:59,354][root][INFO] - ===> Epoch[3134](15670/5): Loss 0.0156, Sem 0.0156 LR: 7.615e-02 Score 99.436 Data time: 0.2654, Forward time: 1.2628, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.8083 [2021-07-07 22:45:12,430][root][INFO] - ===> Epoch[3135](15675/5): Loss 0.0167, Sem 0.0167 LR: 7.615e-02 Score 99.404 Data time: 0.1799, Forward time: 1.2628, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.5743 [2021-07-07 22:45:26,598][root][INFO] - ===> Epoch[3136](15680/5): Loss 0.0158, Sem 0.0158 LR: 7.614e-02 Score 99.427 Data time: 0.1798, Forward time: 1.2627, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7917 [2021-07-07 22:45:39,571][root][INFO] - ===> Epoch[3137](15685/5): Loss 0.0171, Sem 0.0171 LR: 7.613e-02 Score 99.383 Data time: 0.2506, Forward time: 1.2627, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.5436 [2021-07-07 22:45:53,592][root][INFO] - ===> Epoch[3138](15690/5): Loss 0.0156, Sem 0.0156 LR: 7.612e-02 Score 99.431 Data time: 0.1573, Forward time: 1.2627, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7663 [2021-07-07 22:46:07,387][root][INFO] - ===> Epoch[3139](15695/5): Loss 0.0159, Sem 0.0159 LR: 7.612e-02 Score 99.424 Data time: 0.2164, Forward time: 1.2627, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7192 [2021-07-07 22:46:21,768][root][INFO] - ===> Epoch[3140](15700/5): Loss 0.0155, Sem 0.0155 LR: 7.611e-02 Score 99.443 Data time: 0.2739, Forward time: 1.2627, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8230 [2021-07-07 22:46:36,029][root][INFO] - ===> Epoch[3141](15705/5): Loss 0.0156, Sem 0.0156 LR: 7.610e-02 Score 99.435 Data time: 0.2850, Forward time: 1.2627, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8028 [2021-07-07 22:46:50,469][root][INFO] - ===> Epoch[3142](15710/5): Loss 0.0151, Sem 0.0151 LR: 7.609e-02 Score 99.452 Data time: 0.1818, Forward time: 1.2626, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8388 [2021-07-07 22:47:04,542][root][INFO] - ===> Epoch[3143](15715/5): Loss 0.0159, Sem 0.0159 LR: 7.608e-02 Score 99.424 Data time: 0.2482, Forward time: 1.2627, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7541 [2021-07-07 22:47:18,016][root][INFO] - ===> Epoch[3144](15720/5): Loss 0.0161, Sem 0.0161 LR: 7.608e-02 Score 99.427 Data time: 0.2291, Forward time: 1.2627, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6500 [2021-07-07 22:47:31,540][root][INFO] - ===> Epoch[3145](15725/5): Loss 0.0168, Sem 0.0168 LR: 7.607e-02 Score 99.402 Data time: 0.1957, Forward time: 1.2627, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6577 [2021-07-07 22:47:45,145][root][INFO] - ===> Epoch[3146](15730/5): Loss 0.0167, Sem 0.0167 LR: 7.606e-02 Score 99.399 Data time: 0.1839, Forward time: 1.2626, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6897 [2021-07-07 22:47:59,267][root][INFO] - ===> Epoch[3147](15735/5): Loss 0.0179, Sem 0.0179 LR: 7.605e-02 Score 99.386 Data time: 0.2055, Forward time: 1.2626, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7759 [2021-07-07 22:48:13,831][root][INFO] - ===> Epoch[3148](15740/5): Loss 0.0172, Sem 0.0172 LR: 7.605e-02 Score 99.380 Data time: 0.1640, Forward time: 1.2625, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8718 [2021-07-07 22:48:27,855][root][INFO] - ===> Epoch[3149](15745/5): Loss 0.0174, Sem 0.0174 LR: 7.604e-02 Score 99.381 Data time: 0.2796, Forward time: 1.2625, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7438 [2021-07-07 22:48:42,763][root][INFO] - ===> Epoch[3150](15750/5): Loss 0.0166, Sem 0.0166 LR: 7.603e-02 Score 99.410 Data time: 0.2643, Forward time: 1.2625, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.9229 [2021-07-07 22:48:56,122][root][INFO] - ===> Epoch[3151](15755/5): Loss 0.0161, Sem 0.0161 LR: 7.602e-02 Score 99.419 Data time: 0.2879, Forward time: 1.2626, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6205 [2021-07-07 22:49:09,927][root][INFO] - ===> Epoch[3152](15760/5): Loss 0.0155, Sem 0.0155 LR: 7.602e-02 Score 99.438 Data time: 0.2704, Forward time: 1.2626, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7156 [2021-07-07 22:49:24,805][root][INFO] - ===> Epoch[3153](15765/5): Loss 0.0156, Sem 0.0156 LR: 7.601e-02 Score 99.436 Data time: 0.2040, Forward time: 1.2626, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.9367 [2021-07-07 22:49:38,392][root][INFO] - ===> Epoch[3154](15770/5): Loss 0.0151, Sem 0.0151 LR: 7.600e-02 Score 99.454 Data time: 0.2315, Forward time: 1.2626, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6691 [2021-07-07 22:49:52,079][root][INFO] - ===> Epoch[3155](15775/5): Loss 0.0155, Sem 0.0155 LR: 7.599e-02 Score 99.441 Data time: 0.2011, Forward time: 1.2625, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6904 [2021-07-07 22:50:06,122][root][INFO] - ===> Epoch[3156](15780/5): Loss 0.0146, Sem 0.0146 LR: 7.598e-02 Score 99.472 Data time: 0.2211, Forward time: 1.2625, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7637 [2021-07-07 22:50:19,738][root][INFO] - ===> Epoch[3157](15785/5): Loss 0.0148, Sem 0.0148 LR: 7.598e-02 Score 99.469 Data time: 0.2069, Forward time: 1.2625, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6858 [2021-07-07 22:50:33,496][root][INFO] - ===> Epoch[3158](15790/5): Loss 0.0147, Sem 0.0147 LR: 7.597e-02 Score 99.469 Data time: 0.2197, Forward time: 1.2625, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7049 [2021-07-07 22:50:47,852][root][INFO] - ===> Epoch[3159](15795/5): Loss 0.0147, Sem 0.0147 LR: 7.596e-02 Score 99.467 Data time: 0.2500, Forward time: 1.2625, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8311 [2021-07-07 22:51:01,271][root][INFO] - ===> Epoch[3160](15800/5): Loss 0.0154, Sem 0.0154 LR: 7.595e-02 Score 99.440 Data time: 0.2441, Forward time: 1.2626, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6408 [2021-07-07 22:51:01,272][root][INFO] - ===> Start testing [2021-07-07 22:51:35,901][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1593 Loss 0.208 (AVG: 0.489) Score 94.770 (AVG: 91.539) mIOU 71.342 mAP 84.952 mAcc 77.246 IOU: 67.289 0.000 81.904 78.284 94.706 92.357 26.230 80.589 98.461 78.654 81.978 87.139 59.859 mAP: 85.356 7.114 92.643 90.803 98.819 98.827 58.201 97.468 99.802 94.518 93.191 95.182 92.445 mAcc: 83.782 0.000 88.568 83.586 96.020 96.564 27.696 95.229 99.515 81.028 91.159 96.963 64.090 [2021-07-07 22:51:35,901][root][INFO] - Finished test. Elapsed time: 34.6283 [2021-07-07 22:51:37,162][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 22:51:50,480][root][INFO] - ===> Epoch[3161](15805/5): Loss 0.0149, Sem 0.0149 LR: 7.595e-02 Score 99.461 Data time: 0.2226, Forward time: 1.2625, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6313 [2021-07-07 22:52:04,426][root][INFO] - ===> Epoch[3162](15810/5): Loss 0.0148, Sem 0.0148 LR: 7.594e-02 Score 99.460 Data time: 0.2090, Forward time: 1.2625, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7373 [2021-07-07 22:52:18,523][root][INFO] - ===> Epoch[3163](15815/5): Loss 0.0156, Sem 0.0156 LR: 7.593e-02 Score 99.439 Data time: 0.2017, Forward time: 1.2625, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7749 [2021-07-07 22:52:32,065][root][INFO] - ===> Epoch[3164](15820/5): Loss 0.0151, Sem 0.0151 LR: 7.592e-02 Score 99.459 Data time: 0.2114, Forward time: 1.2625, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6594 [2021-07-07 22:52:45,352][root][INFO] - ===> Epoch[3165](15825/5): Loss 0.0150, Sem 0.0150 LR: 7.591e-02 Score 99.454 Data time: 0.2807, Forward time: 1.2625, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6007 [2021-07-07 22:53:00,624][root][INFO] - ===> Epoch[3166](15830/5): Loss 0.0145, Sem 0.0145 LR: 7.591e-02 Score 99.473 Data time: 0.2936, Forward time: 1.2625, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 3.0020 [2021-07-07 22:53:15,177][root][INFO] - ===> Epoch[3167](15835/5): Loss 0.0143, Sem 0.0143 LR: 7.590e-02 Score 99.485 Data time: 0.2469, Forward time: 1.2625, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8549 [2021-07-07 22:53:29,620][root][INFO] - ===> Epoch[3168](15840/5): Loss 0.0145, Sem 0.0145 LR: 7.589e-02 Score 99.476 Data time: 0.2379, Forward time: 1.2625, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8337 [2021-07-07 22:53:43,929][root][INFO] - ===> Epoch[3169](15845/5): Loss 0.0150, Sem 0.0150 LR: 7.588e-02 Score 99.456 Data time: 0.2041, Forward time: 1.2625, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8050 [2021-07-07 22:53:57,051][root][INFO] - ===> Epoch[3170](15850/5): Loss 0.0152, Sem 0.0152 LR: 7.588e-02 Score 99.452 Data time: 0.2126, Forward time: 1.2625, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.5849 [2021-07-07 22:54:11,451][root][INFO] - ===> Epoch[3171](15855/5): Loss 0.0144, Sem 0.0144 LR: 7.587e-02 Score 99.477 Data time: 0.1867, Forward time: 1.2625, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8320 [2021-07-07 22:54:25,106][root][INFO] - ===> Epoch[3172](15860/5): Loss 0.0147, Sem 0.0147 LR: 7.586e-02 Score 99.471 Data time: 0.1913, Forward time: 1.2625, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6893 [2021-07-07 22:54:39,153][root][INFO] - ===> Epoch[3173](15865/5): Loss 0.0139, Sem 0.0139 LR: 7.585e-02 Score 99.496 Data time: 0.2599, Forward time: 1.2625, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7442 [2021-07-07 22:54:52,844][root][INFO] - ===> Epoch[3174](15870/5): Loss 0.0144, Sem 0.0144 LR: 7.584e-02 Score 99.474 Data time: 0.2846, Forward time: 1.2625, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6927 [2021-07-07 22:55:08,093][root][INFO] - ===> Epoch[3175](15875/5): Loss 0.0148, Sem 0.0148 LR: 7.584e-02 Score 99.472 Data time: 0.2371, Forward time: 1.2625, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 3.0044 [2021-07-07 22:55:22,807][root][INFO] - ===> Epoch[3176](15880/5): Loss 0.0156, Sem 0.0156 LR: 7.583e-02 Score 99.447 Data time: 0.1924, Forward time: 1.2625, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.9019 [2021-07-07 22:55:36,410][root][INFO] - ===> Epoch[3177](15885/5): Loss 0.0164, Sem 0.0164 LR: 7.582e-02 Score 99.417 Data time: 0.2708, Forward time: 1.2625, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6700 [2021-07-07 22:55:50,506][root][INFO] - ===> Epoch[3178](15890/5): Loss 0.0165, Sem 0.0165 LR: 7.581e-02 Score 99.423 Data time: 0.2061, Forward time: 1.2625, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7699 [2021-07-07 22:56:04,437][root][INFO] - ===> Epoch[3179](15895/5): Loss 0.0165, Sem 0.0165 LR: 7.581e-02 Score 99.413 Data time: 0.1836, Forward time: 1.2625, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7517 [2021-07-07 22:56:18,436][root][INFO] - ===> Epoch[3180](15900/5): Loss 0.0156, Sem 0.0156 LR: 7.580e-02 Score 99.439 Data time: 0.2557, Forward time: 1.2625, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7601 [2021-07-07 22:56:32,523][root][INFO] - ===> Epoch[3181](15905/5): Loss 0.0152, Sem 0.0152 LR: 7.579e-02 Score 99.448 Data time: 0.2411, Forward time: 1.2625, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7708 [2021-07-07 22:56:46,245][root][INFO] - ===> Epoch[3182](15910/5): Loss 0.0153, Sem 0.0153 LR: 7.578e-02 Score 99.439 Data time: 0.1571, Forward time: 1.2624, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6984 [2021-07-07 22:57:01,122][root][INFO] - ===> Epoch[3183](15915/5): Loss 0.0152, Sem 0.0152 LR: 7.578e-02 Score 99.451 Data time: 0.2594, Forward time: 1.2624, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.9348 [2021-07-07 22:57:14,750][root][INFO] - ===> Epoch[3184](15920/5): Loss 0.0150, Sem 0.0150 LR: 7.577e-02 Score 99.457 Data time: 0.2157, Forward time: 1.2624, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6876 [2021-07-07 22:57:28,860][root][INFO] - ===> Epoch[3185](15925/5): Loss 0.0145, Sem 0.0145 LR: 7.576e-02 Score 99.476 Data time: 0.1870, Forward time: 1.2623, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7820 [2021-07-07 22:57:43,182][root][INFO] - ===> Epoch[3186](15930/5): Loss 0.0149, Sem 0.0149 LR: 7.575e-02 Score 99.463 Data time: 0.2318, Forward time: 1.2623, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8073 [2021-07-07 22:57:57,319][root][INFO] - ===> Epoch[3187](15935/5): Loss 0.0151, Sem 0.0151 LR: 7.574e-02 Score 99.452 Data time: 0.2421, Forward time: 1.2624, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7789 [2021-07-07 22:58:12,616][root][INFO] - ===> Epoch[3188](15940/5): Loss 0.0149, Sem 0.0149 LR: 7.574e-02 Score 99.464 Data time: 0.2276, Forward time: 1.2624, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 3.0132 [2021-07-07 22:58:27,113][root][INFO] - ===> Epoch[3189](15945/5): Loss 0.0148, Sem 0.0148 LR: 7.573e-02 Score 99.463 Data time: 0.3053, Forward time: 1.2624, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8528 [2021-07-07 22:58:41,012][root][INFO] - ===> Epoch[3190](15950/5): Loss 0.0150, Sem 0.0150 LR: 7.572e-02 Score 99.478 Data time: 0.2406, Forward time: 1.2624, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7294 [2021-07-07 22:58:54,753][root][INFO] - ===> Epoch[3191](15955/5): Loss 0.0153, Sem 0.0153 LR: 7.571e-02 Score 99.448 Data time: 0.2198, Forward time: 1.2624, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7053 [2021-07-07 22:59:08,871][root][INFO] - ===> Epoch[3192](15960/5): Loss 0.0147, Sem 0.0147 LR: 7.571e-02 Score 99.470 Data time: 0.2077, Forward time: 1.2624, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7807 [2021-07-07 22:59:23,220][root][INFO] - ===> Epoch[3193](15965/5): Loss 0.0140, Sem 0.0140 LR: 7.570e-02 Score 99.496 Data time: 0.1797, Forward time: 1.2624, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8036 [2021-07-07 22:59:37,039][root][INFO] - ===> Epoch[3194](15970/5): Loss 0.0146, Sem 0.0146 LR: 7.569e-02 Score 99.468 Data time: 0.2091, Forward time: 1.2624, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7126 [2021-07-07 22:59:51,051][root][INFO] - ===> Epoch[3195](15975/5): Loss 0.0147, Sem 0.0147 LR: 7.568e-02 Score 99.473 Data time: 0.2803, Forward time: 1.2624, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7523 [2021-07-07 23:00:05,882][root][INFO] - ===> Epoch[3196](15980/5): Loss 0.0147, Sem 0.0147 LR: 7.567e-02 Score 99.473 Data time: 0.2130, Forward time: 1.2624, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.9167 [2021-07-07 23:00:20,513][root][INFO] - ===> Epoch[3197](15985/5): Loss 0.0147, Sem 0.0147 LR: 7.567e-02 Score 99.472 Data time: 0.1603, Forward time: 1.2624, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8850 [2021-07-07 23:00:34,493][root][INFO] - ===> Epoch[3198](15990/5): Loss 0.0143, Sem 0.0143 LR: 7.566e-02 Score 99.484 Data time: 0.2768, Forward time: 1.2624, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7439 [2021-07-07 23:00:47,939][root][INFO] - ===> Epoch[3199](15995/5): Loss 0.0140, Sem 0.0140 LR: 7.565e-02 Score 99.493 Data time: 0.2146, Forward time: 1.2624, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.6434 [2021-07-07 23:01:01,593][root][INFO] - ===> Epoch[3200](16000/5): Loss 0.0138, Sem 0.0138 LR: 7.564e-02 Score 99.504 Data time: 0.2135, Forward time: 1.2624, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.6886 [2021-07-07 23:01:01,594][root][INFO] - ===> Start testing [2021-07-07 23:01:37,114][root][INFO] - 68/68: Data time: 0.0016, Iter time: 0.1248 Loss 0.191 (AVG: 0.526) Score 95.274 (AVG: 91.430) mIOU 71.258 mAP 84.367 mAcc 77.364 IOU: 66.334 0.000 80.720 78.112 95.282 92.186 23.346 83.921 98.520 82.520 80.798 86.559 58.060 mAP: 84.620 4.824 93.260 90.448 98.780 98.824 53.498 97.985 99.804 95.671 92.437 94.839 91.779 mAcc: 82.988 0.000 87.326 84.153 96.612 97.254 24.382 96.144 99.476 89.049 91.016 96.699 60.628 [2021-07-07 23:01:37,114][root][INFO] - Finished test. Elapsed time: 35.5197 [2021-07-07 23:01:38,180][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 23:01:51,049][root][INFO] - ===> Epoch[3201](16005/5): Loss 0.0136, Sem 0.0136 LR: 7.564e-02 Score 99.512 Data time: 0.1674, Forward time: 1.2624, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.5382 [2021-07-07 23:02:05,640][root][INFO] - ===> Epoch[3202](16010/5): Loss 0.0138, Sem 0.0138 LR: 7.563e-02 Score 99.504 Data time: 0.2142, Forward time: 1.2623, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8722 [2021-07-07 23:02:19,040][root][INFO] - ===> Epoch[3203](16015/5): Loss 0.0140, Sem 0.0140 LR: 7.562e-02 Score 99.495 Data time: 0.1872, Forward time: 1.2623, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.6383 [2021-07-07 23:02:33,867][root][INFO] - ===> Epoch[3204](16020/5): Loss 0.0137, Sem 0.0137 LR: 7.561e-02 Score 99.504 Data time: 0.2869, Forward time: 1.2623, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.9140 [2021-07-07 23:02:47,954][root][INFO] - ===> Epoch[3205](16025/5): Loss 0.0142, Sem 0.0142 LR: 7.561e-02 Score 99.489 Data time: 0.2632, Forward time: 1.2623, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7724 [2021-07-07 23:03:02,374][root][INFO] - ===> Epoch[3206](16030/5): Loss 0.0141, Sem 0.0141 LR: 7.560e-02 Score 99.490 Data time: 0.2850, Forward time: 1.2624, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8375 [2021-07-07 23:03:16,359][root][INFO] - ===> Epoch[3207](16035/5): Loss 0.0140, Sem 0.0140 LR: 7.559e-02 Score 99.501 Data time: 0.2539, Forward time: 1.2624, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7518 [2021-07-07 23:03:30,671][root][INFO] - ===> Epoch[3208](16040/5): Loss 0.0139, Sem 0.0139 LR: 7.558e-02 Score 99.493 Data time: 0.2452, Forward time: 1.2624, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8033 [2021-07-07 23:03:45,438][root][INFO] - ===> Epoch[3209](16045/5): Loss 0.0134, Sem 0.0134 LR: 7.557e-02 Score 99.515 Data time: 0.2170, Forward time: 1.2625, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.9084 [2021-07-07 23:03:59,041][root][INFO] - ===> Epoch[3210](16050/5): Loss 0.0141, Sem 0.0141 LR: 7.557e-02 Score 99.491 Data time: 0.2701, Forward time: 1.2624, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.6742 [2021-07-07 23:04:13,681][root][INFO] - ===> Epoch[3211](16055/5): Loss 0.0146, Sem 0.0146 LR: 7.556e-02 Score 99.475 Data time: 0.1736, Forward time: 1.2624, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8851 [2021-07-07 23:04:29,222][root][INFO] - ===> Epoch[3212](16060/5): Loss 0.0144, Sem 0.0144 LR: 7.555e-02 Score 99.478 Data time: 0.2854, Forward time: 1.2624, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 3.0634 [2021-07-07 23:04:43,211][root][INFO] - ===> Epoch[3213](16065/5): Loss 0.0137, Sem 0.0137 LR: 7.554e-02 Score 99.508 Data time: 0.2106, Forward time: 1.2624, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.7535 [2021-07-07 23:04:56,616][root][INFO] - ===> Epoch[3214](16070/5): Loss 0.0143, Sem 0.0143 LR: 7.554e-02 Score 99.480 Data time: 0.1995, Forward time: 1.2624, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.6363 [2021-07-07 23:05:10,002][root][INFO] - ===> Epoch[3215](16075/5): Loss 0.0161, Sem 0.0161 LR: 7.553e-02 Score 99.434 Data time: 0.2436, Forward time: 1.2624, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.6322 [2021-07-07 23:05:24,971][root][INFO] - ===> Epoch[3216](16080/5): Loss 0.0154, Sem 0.0154 LR: 7.552e-02 Score 99.449 Data time: 0.2065, Forward time: 1.2623, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.9539 [2021-07-07 23:05:39,449][root][INFO] - ===> Epoch[3217](16085/5): Loss 0.0159, Sem 0.0159 LR: 7.551e-02 Score 99.436 Data time: 0.1942, Forward time: 1.2623, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8464 [2021-07-07 23:05:53,532][root][INFO] - ===> Epoch[3218](16090/5): Loss 0.0150, Sem 0.0150 LR: 7.550e-02 Score 99.461 Data time: 0.2310, Forward time: 1.2623, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7691 [2021-07-07 23:06:07,311][root][INFO] - ===> Epoch[3219](16095/5): Loss 0.0143, Sem 0.0143 LR: 7.550e-02 Score 99.482 Data time: 0.2585, Forward time: 1.2623, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7062 [2021-07-07 23:06:21,455][root][INFO] - ===> Epoch[3220](16100/5): Loss 0.0151, Sem 0.0151 LR: 7.549e-02 Score 99.463 Data time: 0.2297, Forward time: 1.2623, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7789 [2021-07-07 23:06:35,093][root][INFO] - ===> Epoch[3221](16105/5): Loss 0.0153, Sem 0.0153 LR: 7.548e-02 Score 99.452 Data time: 0.3194, Forward time: 1.2623, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.6814 [2021-07-07 23:06:49,798][root][INFO] - ===> Epoch[3222](16110/5): Loss 0.0149, Sem 0.0149 LR: 7.547e-02 Score 99.465 Data time: 0.1570, Forward time: 1.2624, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.8995 [2021-07-07 23:07:03,197][root][INFO] - ===> Epoch[3223](16115/5): Loss 0.0150, Sem 0.0150 LR: 7.547e-02 Score 99.455 Data time: 0.2672, Forward time: 1.2623, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.6236 [2021-07-07 23:07:17,626][root][INFO] - ===> Epoch[3224](16120/5): Loss 0.0148, Sem 0.0148 LR: 7.546e-02 Score 99.466 Data time: 0.2179, Forward time: 1.2623, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.8417 [2021-07-07 23:07:31,454][root][INFO] - ===> Epoch[3225](16125/5): Loss 0.0145, Sem 0.0145 LR: 7.545e-02 Score 99.478 Data time: 0.2778, Forward time: 1.2624, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.7101 [2021-07-07 23:07:45,291][root][INFO] - ===> Epoch[3226](16130/5): Loss 0.0141, Sem 0.0141 LR: 7.544e-02 Score 99.491 Data time: 0.2199, Forward time: 1.2624, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7261 [2021-07-07 23:07:59,951][root][INFO] - ===> Epoch[3227](16135/5): Loss 0.0139, Sem 0.0139 LR: 7.543e-02 Score 99.494 Data time: 0.2684, Forward time: 1.2624, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8780 [2021-07-07 23:08:14,804][root][INFO] - ===> Epoch[3228](16140/5): Loss 0.0133, Sem 0.0133 LR: 7.543e-02 Score 99.520 Data time: 0.2160, Forward time: 1.2624, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.9239 [2021-07-07 23:08:29,894][root][INFO] - ===> Epoch[3229](16145/5): Loss 0.0135, Sem 0.0135 LR: 7.542e-02 Score 99.512 Data time: 0.2574, Forward time: 1.2625, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.9717 [2021-07-07 23:08:43,922][root][INFO] - ===> Epoch[3230](16150/5): Loss 0.0141, Sem 0.0141 LR: 7.541e-02 Score 99.491 Data time: 0.2507, Forward time: 1.2624, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.7600 [2021-07-07 23:08:58,303][root][INFO] - ===> Epoch[3231](16155/5): Loss 0.0142, Sem 0.0142 LR: 7.540e-02 Score 99.485 Data time: 0.2139, Forward time: 1.2624, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8271 [2021-07-07 23:09:13,219][root][INFO] - ===> Epoch[3232](16160/5): Loss 0.0138, Sem 0.0138 LR: 7.540e-02 Score 99.501 Data time: 0.2322, Forward time: 1.2624, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.9116 [2021-07-07 23:09:27,426][root][INFO] - ===> Epoch[3233](16165/5): Loss 0.0139, Sem 0.0139 LR: 7.539e-02 Score 99.505 Data time: 0.2346, Forward time: 1.2625, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7938 [2021-07-07 23:09:40,733][root][INFO] - ===> Epoch[3234](16170/5): Loss 0.0142, Sem 0.0142 LR: 7.538e-02 Score 99.491 Data time: 0.2002, Forward time: 1.2624, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.6152 [2021-07-07 23:09:54,688][root][INFO] - ===> Epoch[3235](16175/5): Loss 0.0135, Sem 0.0135 LR: 7.537e-02 Score 99.508 Data time: 0.2556, Forward time: 1.2625, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.7539 [2021-07-07 23:10:09,045][root][INFO] - ===> Epoch[3236](16180/5): Loss 0.0135, Sem 0.0135 LR: 7.537e-02 Score 99.511 Data time: 0.2577, Forward time: 1.2625, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.8148 [2021-07-07 23:10:23,106][root][INFO] - ===> Epoch[3237](16185/5): Loss 0.0137, Sem 0.0137 LR: 7.536e-02 Score 99.503 Data time: 0.2575, Forward time: 1.2624, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.7642 [2021-07-07 23:10:37,557][root][INFO] - ===> Epoch[3238](16190/5): Loss 0.0135, Sem 0.0135 LR: 7.535e-02 Score 99.514 Data time: 0.2308, Forward time: 1.2624, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8461 [2021-07-07 23:10:51,869][root][INFO] - ===> Epoch[3239](16195/5): Loss 0.0137, Sem 0.0137 LR: 7.534e-02 Score 99.508 Data time: 0.2429, Forward time: 1.2624, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8038 [2021-07-07 23:11:05,994][root][INFO] - ===> Epoch[3240](16200/5): Loss 0.0131, Sem 0.0131 LR: 7.533e-02 Score 99.531 Data time: 0.2083, Forward time: 1.2624, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.7845 [2021-07-07 23:11:05,996][root][INFO] - ===> Start testing [2021-07-07 23:11:40,628][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1220 Loss 0.205 (AVG: 0.543) Score 94.648 (AVG: 91.119) mIOU 70.534 mAP 84.368 mAcc 77.045 IOU: 65.436 0.000 82.142 76.872 94.876 91.487 21.822 82.287 98.060 78.884 80.758 86.836 57.487 mAP: 84.590 6.743 94.199 90.741 98.821 98.937 52.808 98.022 99.710 94.099 92.070 94.586 91.453 mAcc: 80.776 0.000 90.668 83.898 96.057 97.825 22.893 95.191 99.401 85.912 90.062 96.910 61.987 [2021-07-07 23:11:40,628][root][INFO] - Finished test. Elapsed time: 34.6320 [2021-07-07 23:11:41,681][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 23:11:55,559][root][INFO] - ===> Epoch[3241](16205/5): Loss 0.0136, Sem 0.0136 LR: 7.533e-02 Score 99.511 Data time: 0.2274, Forward time: 1.2624, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.7290 [2021-07-07 23:12:10,046][root][INFO] - ===> Epoch[3242](16210/5): Loss 0.0135, Sem 0.0135 LR: 7.532e-02 Score 99.512 Data time: 0.2856, Forward time: 1.2624, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8541 [2021-07-07 23:12:24,382][root][INFO] - ===> Epoch[3243](16215/5): Loss 0.0132, Sem 0.0132 LR: 7.531e-02 Score 99.520 Data time: 0.2896, Forward time: 1.2624, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8142 [2021-07-07 23:12:39,470][root][INFO] - ===> Epoch[3244](16220/5): Loss 0.0134, Sem 0.0134 LR: 7.530e-02 Score 99.519 Data time: 0.2713, Forward time: 1.2625, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.9564 [2021-07-07 23:12:53,805][root][INFO] - ===> Epoch[3245](16225/5): Loss 0.0124, Sem 0.0124 LR: 7.530e-02 Score 99.554 Data time: 0.2544, Forward time: 1.2625, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8200 [2021-07-07 23:13:07,698][root][INFO] - ===> Epoch[3246](16230/5): Loss 0.0143, Sem 0.0143 LR: 7.529e-02 Score 99.484 Data time: 0.2333, Forward time: 1.2625, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7358 [2021-07-07 23:13:21,564][root][INFO] - ===> Epoch[3247](16235/5): Loss 0.0133, Sem 0.0133 LR: 7.528e-02 Score 99.517 Data time: 0.3134, Forward time: 1.2625, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.7262 [2021-07-07 23:13:35,904][root][INFO] - ===> Epoch[3248](16240/5): Loss 0.0133, Sem 0.0133 LR: 7.527e-02 Score 99.518 Data time: 0.1871, Forward time: 1.2625, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8220 [2021-07-07 23:13:50,217][root][INFO] - ===> Epoch[3249](16245/5): Loss 0.0133, Sem 0.0133 LR: 7.526e-02 Score 99.518 Data time: 0.2022, Forward time: 1.2625, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8287 [2021-07-07 23:14:04,182][root][INFO] - ===> Epoch[3250](16250/5): Loss 0.0133, Sem 0.0133 LR: 7.526e-02 Score 99.520 Data time: 0.2297, Forward time: 1.2625, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7418 [2021-07-07 23:14:18,247][root][INFO] - ===> Epoch[3251](16255/5): Loss 0.0129, Sem 0.0129 LR: 7.525e-02 Score 99.531 Data time: 0.2174, Forward time: 1.2625, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7699 [2021-07-07 23:14:32,633][root][INFO] - ===> Epoch[3252](16260/5): Loss 0.0137, Sem 0.0137 LR: 7.524e-02 Score 99.503 Data time: 0.2152, Forward time: 1.2625, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8327 [2021-07-07 23:14:46,122][root][INFO] - ===> Epoch[3253](16265/5): Loss 0.0135, Sem 0.0135 LR: 7.523e-02 Score 99.514 Data time: 0.2353, Forward time: 1.2625, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.6478 [2021-07-07 23:15:00,067][root][INFO] - ===> Epoch[3254](16270/5): Loss 0.0126, Sem 0.0126 LR: 7.523e-02 Score 99.545 Data time: 0.2555, Forward time: 1.2625, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7391 [2021-07-07 23:15:14,746][root][INFO] - ===> Epoch[3255](16275/5): Loss 0.0138, Sem 0.0138 LR: 7.522e-02 Score 99.498 Data time: 0.1856, Forward time: 1.2625, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8878 [2021-07-07 23:15:28,884][root][INFO] - ===> Epoch[3256](16280/5): Loss 0.0125, Sem 0.0125 LR: 7.521e-02 Score 99.547 Data time: 0.1981, Forward time: 1.2625, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7871 [2021-07-07 23:15:42,884][root][INFO] - ===> Epoch[3257](16285/5): Loss 0.0138, Sem 0.0138 LR: 7.520e-02 Score 99.500 Data time: 0.2288, Forward time: 1.2625, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7594 [2021-07-07 23:15:56,503][root][INFO] - ===> Epoch[3258](16290/5): Loss 0.0133, Sem 0.0133 LR: 7.520e-02 Score 99.515 Data time: 0.2671, Forward time: 1.2626, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.6793 [2021-07-07 23:16:11,637][root][INFO] - ===> Epoch[3259](16295/5): Loss 0.0133, Sem 0.0133 LR: 7.519e-02 Score 99.516 Data time: 0.2516, Forward time: 1.2626, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.9872 [2021-07-07 23:16:25,318][root][INFO] - ===> Epoch[3260](16300/5): Loss 0.0140, Sem 0.0140 LR: 7.518e-02 Score 99.490 Data time: 0.1846, Forward time: 1.2626, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.6963 [2021-07-07 23:16:39,479][root][INFO] - ===> Epoch[3261](16305/5): Loss 0.0138, Sem 0.0138 LR: 7.517e-02 Score 99.498 Data time: 0.1717, Forward time: 1.2626, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7957 [2021-07-07 23:16:53,792][root][INFO] - ===> Epoch[3262](16310/5): Loss 0.0140, Sem 0.0140 LR: 7.516e-02 Score 99.496 Data time: 0.2198, Forward time: 1.2626, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8179 [2021-07-07 23:17:09,293][root][INFO] - ===> Epoch[3263](16315/5): Loss 0.0133, Sem 0.0133 LR: 7.516e-02 Score 99.518 Data time: 0.2586, Forward time: 1.2626, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 3.0496 [2021-07-07 23:17:23,075][root][INFO] - ===> Epoch[3264](16320/5): Loss 0.0143, Sem 0.0143 LR: 7.515e-02 Score 99.492 Data time: 0.2584, Forward time: 1.2627, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.6987 [2021-07-07 23:17:36,602][root][INFO] - ===> Epoch[3265](16325/5): Loss 0.0139, Sem 0.0139 LR: 7.514e-02 Score 99.500 Data time: 0.2768, Forward time: 1.2627, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.6475 [2021-07-07 23:17:50,962][root][INFO] - ===> Epoch[3266](16330/5): Loss 0.0148, Sem 0.0148 LR: 7.513e-02 Score 99.469 Data time: 0.1814, Forward time: 1.2626, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8315 [2021-07-07 23:18:05,270][root][INFO] - ===> Epoch[3267](16335/5): Loss 0.0136, Sem 0.0136 LR: 7.513e-02 Score 99.513 Data time: 0.2438, Forward time: 1.2626, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8184 [2021-07-07 23:18:18,897][root][INFO] - ===> Epoch[3268](16340/5): Loss 0.0140, Sem 0.0140 LR: 7.512e-02 Score 99.493 Data time: 0.2230, Forward time: 1.2626, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.6852 [2021-07-07 23:18:32,699][root][INFO] - ===> Epoch[3269](16345/5): Loss 0.0135, Sem 0.0135 LR: 7.511e-02 Score 99.514 Data time: 0.2218, Forward time: 1.2625, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.7184 [2021-07-07 23:18:47,557][root][INFO] - ===> Epoch[3270](16350/5): Loss 0.0138, Sem 0.0138 LR: 7.510e-02 Score 99.505 Data time: 0.2307, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.9225 [2021-07-07 23:19:01,172][root][INFO] - ===> Epoch[3271](16355/5): Loss 0.0129, Sem 0.0129 LR: 7.509e-02 Score 99.534 Data time: 0.2550, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.6737 [2021-07-07 23:19:15,193][root][INFO] - ===> Epoch[3272](16360/5): Loss 0.0134, Sem 0.0134 LR: 7.509e-02 Score 99.516 Data time: 0.1913, Forward time: 1.2624, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7497 [2021-07-07 23:19:29,377][root][INFO] - ===> Epoch[3273](16365/5): Loss 0.0138, Sem 0.0138 LR: 7.508e-02 Score 99.507 Data time: 0.2218, Forward time: 1.2624, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7878 [2021-07-07 23:19:42,575][root][INFO] - ===> Epoch[3274](16370/5): Loss 0.0132, Sem 0.0132 LR: 7.507e-02 Score 99.523 Data time: 0.2633, Forward time: 1.2624, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.5775 [2021-07-07 23:19:56,313][root][INFO] - ===> Epoch[3275](16375/5): Loss 0.0143, Sem 0.0143 LR: 7.506e-02 Score 99.490 Data time: 0.1881, Forward time: 1.2624, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.6999 [2021-07-07 23:20:10,980][root][INFO] - ===> Epoch[3276](16380/5): Loss 0.0137, Sem 0.0137 LR: 7.506e-02 Score 99.503 Data time: 0.2064, Forward time: 1.2624, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.8717 [2021-07-07 23:20:25,300][root][INFO] - ===> Epoch[3277](16385/5): Loss 0.0139, Sem 0.0139 LR: 7.505e-02 Score 99.499 Data time: 0.2317, Forward time: 1.2624, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.8068 [2021-07-07 23:20:39,800][root][INFO] - ===> Epoch[3278](16390/5): Loss 0.0131, Sem 0.0131 LR: 7.504e-02 Score 99.528 Data time: 0.2818, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.8636 [2021-07-07 23:20:54,635][root][INFO] - ===> Epoch[3279](16395/5): Loss 0.0133, Sem 0.0133 LR: 7.503e-02 Score 99.518 Data time: 0.2241, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.9264 [2021-07-07 23:21:08,435][root][INFO] - ===> Epoch[3280](16400/5): Loss 0.0133, Sem 0.0133 LR: 7.502e-02 Score 99.525 Data time: 0.1885, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7208 [2021-07-07 23:21:08,437][root][INFO] - ===> Start testing [2021-07-07 23:21:44,461][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1562 Loss 0.330 (AVG: 0.522) Score 92.155 (AVG: 91.363) mIOU 71.121 mAP 84.013 mAcc 77.671 IOU: 66.196 0.000 83.165 78.193 94.739 91.199 21.571 83.405 98.414 80.309 80.845 86.599 59.934 mAP: 84.944 3.967 94.947 89.920 98.644 98.708 48.464 98.174 99.773 95.282 92.428 94.327 92.586 mAcc: 80.647 0.000 89.734 85.090 96.132 97.654 22.852 97.133 99.466 89.651 90.732 96.704 63.930 [2021-07-07 23:21:44,461][root][INFO] - Finished test. Elapsed time: 36.0235 [2021-07-07 23:21:45,667][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 23:21:58,257][root][INFO] - ===> Epoch[3281](16405/5): Loss 0.0138, Sem 0.0138 LR: 7.502e-02 Score 99.505 Data time: 0.2155, Forward time: 1.2625, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.4780 [2021-07-07 23:22:12,378][root][INFO] - ===> Epoch[3282](16410/5): Loss 0.0136, Sem 0.0136 LR: 7.501e-02 Score 99.513 Data time: 0.2354, Forward time: 1.2625, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.7781 [2021-07-07 23:22:26,897][root][INFO] - ===> Epoch[3283](16415/5): Loss 0.0132, Sem 0.0132 LR: 7.500e-02 Score 99.529 Data time: 0.2943, Forward time: 1.2625, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.8533 [2021-07-07 23:22:41,022][root][INFO] - ===> Epoch[3284](16420/5): Loss 0.0136, Sem 0.0136 LR: 7.499e-02 Score 99.509 Data time: 0.2305, Forward time: 1.2625, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.7730 [2021-07-07 23:22:54,682][root][INFO] - ===> Epoch[3285](16425/5): Loss 0.0134, Sem 0.0134 LR: 7.499e-02 Score 99.512 Data time: 0.2278, Forward time: 1.2625, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.6914 [2021-07-07 23:23:09,244][root][INFO] - ===> Epoch[3286](16430/5): Loss 0.0136, Sem 0.0136 LR: 7.498e-02 Score 99.509 Data time: 0.2291, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.8710 [2021-07-07 23:23:23,301][root][INFO] - ===> Epoch[3287](16435/5): Loss 0.0129, Sem 0.0129 LR: 7.497e-02 Score 99.534 Data time: 0.1858, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7693 [2021-07-07 23:23:38,015][root][INFO] - ===> Epoch[3288](16440/5): Loss 0.0129, Sem 0.0129 LR: 7.496e-02 Score 99.537 Data time: 0.2506, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.8982 [2021-07-07 23:23:52,205][root][INFO] - ===> Epoch[3289](16445/5): Loss 0.0135, Sem 0.0135 LR: 7.495e-02 Score 99.513 Data time: 0.2684, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7886 [2021-07-07 23:24:06,366][root][INFO] - ===> Epoch[3290](16450/5): Loss 0.0126, Sem 0.0126 LR: 7.495e-02 Score 99.544 Data time: 0.2664, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7827 [2021-07-07 23:24:19,885][root][INFO] - ===> Epoch[3291](16455/5): Loss 0.0135, Sem 0.0135 LR: 7.494e-02 Score 99.515 Data time: 0.2128, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.6643 [2021-07-07 23:24:33,129][root][INFO] - ===> Epoch[3292](16460/5): Loss 0.0132, Sem 0.0132 LR: 7.493e-02 Score 99.518 Data time: 0.2063, Forward time: 1.2624, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.6091 [2021-07-07 23:24:47,332][root][INFO] - ===> Epoch[3293](16465/5): Loss 0.0127, Sem 0.0127 LR: 7.492e-02 Score 99.540 Data time: 0.1857, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7974 [2021-07-07 23:25:01,088][root][INFO] - ===> Epoch[3294](16470/5): Loss 0.0134, Sem 0.0134 LR: 7.492e-02 Score 99.514 Data time: 0.2357, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7000 [2021-07-07 23:25:15,926][root][INFO] - ===> Epoch[3295](16475/5): Loss 0.0127, Sem 0.0127 LR: 7.491e-02 Score 99.539 Data time: 0.1650, Forward time: 1.2624, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.9298 [2021-07-07 23:25:29,653][root][INFO] - ===> Epoch[3296](16480/5): Loss 0.0126, Sem 0.0126 LR: 7.490e-02 Score 99.542 Data time: 0.2423, Forward time: 1.2624, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.6837 [2021-07-07 23:25:43,675][root][INFO] - ===> Epoch[3297](16485/5): Loss 0.0127, Sem 0.0127 LR: 7.489e-02 Score 99.543 Data time: 0.1777, Forward time: 1.2624, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.7497 [2021-07-07 23:25:57,260][root][INFO] - ===> Epoch[3298](16490/5): Loss 0.0129, Sem 0.0129 LR: 7.489e-02 Score 99.535 Data time: 0.2097, Forward time: 1.2624, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.6675 [2021-07-07 23:26:12,484][root][INFO] - ===> Epoch[3299](16495/5): Loss 0.0123, Sem 0.0123 LR: 7.488e-02 Score 99.556 Data time: 0.2525, Forward time: 1.2624, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 3.0000 [2021-07-07 23:26:26,931][root][INFO] - ===> Epoch[3300](16500/5): Loss 0.0129, Sem 0.0129 LR: 7.487e-02 Score 99.532 Data time: 0.2089, Forward time: 1.2624, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.8380 [2021-07-07 23:26:40,602][root][INFO] - ===> Epoch[3301](16505/5): Loss 0.0129, Sem 0.0129 LR: 7.486e-02 Score 99.538 Data time: 0.2188, Forward time: 1.2624, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.7008 [2021-07-07 23:26:54,428][root][INFO] - ===> Epoch[3302](16510/5): Loss 0.0130, Sem 0.0130 LR: 7.485e-02 Score 99.533 Data time: 0.1687, Forward time: 1.2624, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.7226 [2021-07-07 23:27:08,862][root][INFO] - ===> Epoch[3303](16515/5): Loss 0.0123, Sem 0.0123 LR: 7.485e-02 Score 99.557 Data time: 0.2300, Forward time: 1.2625, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.8427 [2021-07-07 23:27:23,079][root][INFO] - ===> Epoch[3304](16520/5): Loss 0.0129, Sem 0.0129 LR: 7.484e-02 Score 99.531 Data time: 0.2689, Forward time: 1.2625, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.7969 [2021-07-07 23:27:36,492][root][INFO] - ===> Epoch[3305](16525/5): Loss 0.0129, Sem 0.0129 LR: 7.483e-02 Score 99.531 Data time: 0.2725, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.6343 [2021-07-07 23:27:50,548][root][INFO] - ===> Epoch[3306](16530/5): Loss 0.0124, Sem 0.0124 LR: 7.482e-02 Score 99.551 Data time: 0.2162, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7585 [2021-07-07 23:28:05,827][root][INFO] - ===> Epoch[3307](16535/5): Loss 0.0126, Sem 0.0126 LR: 7.482e-02 Score 99.545 Data time: 0.2028, Forward time: 1.2625, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 3.0086 [2021-07-07 23:28:19,522][root][INFO] - ===> Epoch[3308](16540/5): Loss 0.0129, Sem 0.0129 LR: 7.481e-02 Score 99.534 Data time: 0.1984, Forward time: 1.2625, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.6978 [2021-07-07 23:28:33,386][root][INFO] - ===> Epoch[3309](16545/5): Loss 0.0123, Sem 0.0123 LR: 7.480e-02 Score 99.548 Data time: 0.1881, Forward time: 1.2625, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.7215 [2021-07-07 23:28:47,452][root][INFO] - ===> Epoch[3310](16550/5): Loss 0.0122, Sem 0.0122 LR: 7.479e-02 Score 99.558 Data time: 0.2767, Forward time: 1.2625, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.7705 [2021-07-07 23:29:00,943][root][INFO] - ===> Epoch[3311](16555/5): Loss 0.0135, Sem 0.0135 LR: 7.478e-02 Score 99.511 Data time: 0.2665, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.6388 [2021-07-07 23:29:14,682][root][INFO] - ===> Epoch[3312](16560/5): Loss 0.0131, Sem 0.0131 LR: 7.478e-02 Score 99.527 Data time: 0.2625, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.6952 [2021-07-07 23:29:28,936][root][INFO] - ===> Epoch[3313](16565/5): Loss 0.0129, Sem 0.0129 LR: 7.477e-02 Score 99.529 Data time: 0.2606, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.8011 [2021-07-07 23:29:42,478][root][INFO] - ===> Epoch[3314](16570/5): Loss 0.0132, Sem 0.0132 LR: 7.476e-02 Score 99.521 Data time: 0.2202, Forward time: 1.2625, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.6598 [2021-07-07 23:29:56,958][root][INFO] - ===> Epoch[3315](16575/5): Loss 0.0124, Sem 0.0124 LR: 7.475e-02 Score 99.552 Data time: 0.1870, Forward time: 1.2624, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.8600 [2021-07-07 23:30:10,477][root][INFO] - ===> Epoch[3316](16580/5): Loss 0.0129, Sem 0.0129 LR: 7.475e-02 Score 99.534 Data time: 0.2479, Forward time: 1.2624, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.6647 [2021-07-07 23:30:24,131][root][INFO] - ===> Epoch[3317](16585/5): Loss 0.0129, Sem 0.0129 LR: 7.474e-02 Score 99.537 Data time: 0.2332, Forward time: 1.2624, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.6824 [2021-07-07 23:30:37,836][root][INFO] - ===> Epoch[3318](16590/5): Loss 0.0124, Sem 0.0124 LR: 7.473e-02 Score 99.551 Data time: 0.2405, Forward time: 1.2624, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.6990 [2021-07-07 23:30:51,621][root][INFO] - ===> Epoch[3319](16595/5): Loss 0.0125, Sem 0.0125 LR: 7.472e-02 Score 99.554 Data time: 0.2188, Forward time: 1.2624, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7233 [2021-07-07 23:31:06,155][root][INFO] - ===> Epoch[3320](16600/5): Loss 0.0125, Sem 0.0125 LR: 7.471e-02 Score 99.544 Data time: 0.2822, Forward time: 1.2624, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.8692 [2021-07-07 23:31:06,156][root][INFO] - ===> Start testing [2021-07-07 23:31:42,071][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1241 Loss 0.238 (AVG: 0.520) Score 93.983 (AVG: 91.377) mIOU 71.786 mAP 84.610 mAcc 78.039 IOU: 65.819 0.065 84.254 77.730 94.782 92.353 26.541 85.586 98.619 80.461 81.858 86.455 58.690 mAP: 84.654 10.233 94.325 89.881 98.603 98.845 51.727 98.390 99.773 95.454 92.337 94.165 91.542 mAcc: 80.438 0.456 90.263 84.243 96.657 96.545 29.617 95.917 99.620 89.907 91.606 96.556 62.676 [2021-07-07 23:31:42,071][root][INFO] - Finished test. Elapsed time: 35.9133 [2021-07-07 23:31:43,154][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 23:31:56,350][root][INFO] - ===> Epoch[3321](16605/5): Loss 0.0135, Sem 0.0135 LR: 7.471e-02 Score 99.513 Data time: 0.2155, Forward time: 1.2624, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.5990 [2021-07-07 23:32:10,448][root][INFO] - ===> Epoch[3322](16610/5): Loss 0.0124, Sem 0.0124 LR: 7.470e-02 Score 99.552 Data time: 0.1487, Forward time: 1.2624, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7881 [2021-07-07 23:32:24,402][root][INFO] - ===> Epoch[3323](16615/5): Loss 0.0128, Sem 0.0128 LR: 7.469e-02 Score 99.539 Data time: 0.2581, Forward time: 1.2624, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7421 [2021-07-07 23:32:38,286][root][INFO] - ===> Epoch[3324](16620/5): Loss 0.0122, Sem 0.0122 LR: 7.468e-02 Score 99.561 Data time: 0.2205, Forward time: 1.2625, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.7335 [2021-07-07 23:32:52,039][root][INFO] - ===> Epoch[3325](16625/5): Loss 0.0121, Sem 0.0121 LR: 7.468e-02 Score 99.560 Data time: 0.2145, Forward time: 1.2625, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.6934 [2021-07-07 23:33:06,630][root][INFO] - ===> Epoch[3326](16630/5): Loss 0.0126, Sem 0.0126 LR: 7.467e-02 Score 99.544 Data time: 0.2720, Forward time: 1.2625, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.8694 [2021-07-07 23:33:20,323][root][INFO] - ===> Epoch[3327](16635/5): Loss 0.0126, Sem 0.0126 LR: 7.466e-02 Score 99.543 Data time: 0.2830, Forward time: 1.2625, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.6905 [2021-07-07 23:33:34,430][root][INFO] - ===> Epoch[3328](16640/5): Loss 0.0134, Sem 0.0134 LR: 7.465e-02 Score 99.525 Data time: 0.2321, Forward time: 1.2625, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.7696 [2021-07-07 23:33:48,534][root][INFO] - ===> Epoch[3329](16645/5): Loss 0.0125, Sem 0.0125 LR: 7.465e-02 Score 99.552 Data time: 0.2254, Forward time: 1.2625, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.7632 [2021-07-07 23:34:02,612][root][INFO] - ===> Epoch[3330](16650/5): Loss 0.0128, Sem 0.0128 LR: 7.464e-02 Score 99.534 Data time: 0.2195, Forward time: 1.2625, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.7751 [2021-07-07 23:34:16,539][root][INFO] - ===> Epoch[3331](16655/5): Loss 0.0134, Sem 0.0134 LR: 7.463e-02 Score 99.517 Data time: 0.1648, Forward time: 1.2625, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.7441 [2021-07-07 23:34:30,547][root][INFO] - ===> Epoch[3332](16660/5): Loss 0.0126, Sem 0.0126 LR: 7.462e-02 Score 99.543 Data time: 0.2792, Forward time: 1.2625, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7461 [2021-07-07 23:34:44,979][root][INFO] - ===> Epoch[3333](16665/5): Loss 0.0129, Sem 0.0129 LR: 7.461e-02 Score 99.537 Data time: 0.2460, Forward time: 1.2626, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8335 [2021-07-07 23:34:58,702][root][INFO] - ===> Epoch[3334](16670/5): Loss 0.0127, Sem 0.0127 LR: 7.461e-02 Score 99.537 Data time: 0.2844, Forward time: 1.2626, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.6853 [2021-07-07 23:35:11,922][root][INFO] - ===> Epoch[3335](16675/5): Loss 0.0135, Sem 0.0135 LR: 7.460e-02 Score 99.517 Data time: 0.2243, Forward time: 1.2626, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.5969 [2021-07-07 23:35:26,145][root][INFO] - ===> Epoch[3336](16680/5): Loss 0.0132, Sem 0.0132 LR: 7.459e-02 Score 99.521 Data time: 0.2162, Forward time: 1.2626, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.8008 [2021-07-07 23:35:40,257][root][INFO] - ===> Epoch[3337](16685/5): Loss 0.0127, Sem 0.0127 LR: 7.458e-02 Score 99.538 Data time: 0.2198, Forward time: 1.2626, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.7750 [2021-07-07 23:35:54,422][root][INFO] - ===> Epoch[3338](16690/5): Loss 0.0127, Sem 0.0127 LR: 7.458e-02 Score 99.541 Data time: 0.3405, Forward time: 1.2626, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.7699 [2021-07-07 23:36:08,376][root][INFO] - ===> Epoch[3339](16695/5): Loss 0.0126, Sem 0.0126 LR: 7.457e-02 Score 99.543 Data time: 0.1991, Forward time: 1.2627, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.7542 [2021-07-07 23:36:21,820][root][INFO] - ===> Epoch[3340](16700/5): Loss 0.0129, Sem 0.0129 LR: 7.456e-02 Score 99.532 Data time: 0.1982, Forward time: 1.2627, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.6426 [2021-07-07 23:36:35,563][root][INFO] - ===> Epoch[3341](16705/5): Loss 0.0126, Sem 0.0126 LR: 7.455e-02 Score 99.541 Data time: 0.2470, Forward time: 1.2627, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.6994 [2021-07-07 23:36:49,393][root][INFO] - ===> Epoch[3342](16710/5): Loss 0.0127, Sem 0.0127 LR: 7.454e-02 Score 99.544 Data time: 0.2054, Forward time: 1.2627, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7176 [2021-07-07 23:37:03,843][root][INFO] - ===> Epoch[3343](16715/5): Loss 0.0123, Sem 0.0123 LR: 7.454e-02 Score 99.555 Data time: 0.2987, Forward time: 1.2627, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8354 [2021-07-07 23:37:18,216][root][INFO] - ===> Epoch[3344](16720/5): Loss 0.0126, Sem 0.0126 LR: 7.453e-02 Score 99.545 Data time: 0.2595, Forward time: 1.2627, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8231 [2021-07-07 23:37:32,841][root][INFO] - ===> Epoch[3345](16725/5): Loss 0.0121, Sem 0.0121 LR: 7.452e-02 Score 99.564 Data time: 0.2160, Forward time: 1.2627, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8758 [2021-07-07 23:37:47,399][root][INFO] - ===> Epoch[3346](16730/5): Loss 0.0127, Sem 0.0127 LR: 7.451e-02 Score 99.540 Data time: 0.2892, Forward time: 1.2628, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8673 [2021-07-07 23:38:01,381][root][INFO] - ===> Epoch[3347](16735/5): Loss 0.0128, Sem 0.0128 LR: 7.451e-02 Score 99.534 Data time: 0.2354, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7581 [2021-07-07 23:38:15,242][root][INFO] - ===> Epoch[3348](16740/5): Loss 0.0125, Sem 0.0125 LR: 7.450e-02 Score 99.547 Data time: 0.2343, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7202 [2021-07-07 23:38:29,146][root][INFO] - ===> Epoch[3349](16745/5): Loss 0.0121, Sem 0.0121 LR: 7.449e-02 Score 99.562 Data time: 0.2250, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7418 [2021-07-07 23:38:43,536][root][INFO] - ===> Epoch[3350](16750/5): Loss 0.0121, Sem 0.0121 LR: 7.448e-02 Score 99.562 Data time: 0.2006, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8351 [2021-07-07 23:38:57,847][root][INFO] - ===> Epoch[3351](16755/5): Loss 0.0119, Sem 0.0119 LR: 7.447e-02 Score 99.569 Data time: 0.2244, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8190 [2021-07-07 23:39:12,169][root][INFO] - ===> Epoch[3352](16760/5): Loss 0.0124, Sem 0.0124 LR: 7.447e-02 Score 99.552 Data time: 0.2513, Forward time: 1.2628, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8280 [2021-07-07 23:39:26,872][root][INFO] - ===> Epoch[3353](16765/5): Loss 0.0120, Sem 0.0120 LR: 7.446e-02 Score 99.563 Data time: 0.2020, Forward time: 1.2628, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8975 [2021-07-07 23:39:41,333][root][INFO] - ===> Epoch[3354](16770/5): Loss 0.0123, Sem 0.0123 LR: 7.445e-02 Score 99.550 Data time: 0.1995, Forward time: 1.2628, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8409 [2021-07-07 23:39:55,336][root][INFO] - ===> Epoch[3355](16775/5): Loss 0.0124, Sem 0.0124 LR: 7.444e-02 Score 99.549 Data time: 0.3009, Forward time: 1.2628, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7512 [2021-07-07 23:40:09,589][root][INFO] - ===> Epoch[3356](16780/5): Loss 0.0123, Sem 0.0123 LR: 7.444e-02 Score 99.556 Data time: 0.2366, Forward time: 1.2628, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7996 [2021-07-07 23:40:23,282][root][INFO] - ===> Epoch[3357](16785/5): Loss 0.0126, Sem 0.0126 LR: 7.443e-02 Score 99.550 Data time: 0.1533, Forward time: 1.2628, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.6979 [2021-07-07 23:40:37,286][root][INFO] - ===> Epoch[3358](16790/5): Loss 0.0123, Sem 0.0123 LR: 7.442e-02 Score 99.556 Data time: 0.1970, Forward time: 1.2628, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7524 [2021-07-07 23:40:51,711][root][INFO] - ===> Epoch[3359](16795/5): Loss 0.0123, Sem 0.0123 LR: 7.441e-02 Score 99.552 Data time: 0.2360, Forward time: 1.2628, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8294 [2021-07-07 23:41:05,326][root][INFO] - ===> Epoch[3360](16800/5): Loss 0.0126, Sem 0.0126 LR: 7.440e-02 Score 99.544 Data time: 0.2345, Forward time: 1.2628, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.6839 [2021-07-07 23:41:05,328][root][INFO] - ===> Start testing [2021-07-07 23:41:40,205][root][INFO] - 68/68: Data time: 0.0018, Iter time: 0.1338 Loss 0.257 (AVG: 0.538) Score 94.204 (AVG: 91.287) mIOU 70.540 mAP 83.831 mAcc 76.178 IOU: 66.233 0.037 84.688 77.587 94.474 92.305 20.423 86.485 98.176 70.283 81.446 86.583 58.298 mAP: 84.560 8.967 94.163 89.956 98.555 98.831 48.261 98.510 99.728 90.013 92.247 94.307 91.700 mAcc: 81.288 0.273 90.579 85.293 96.011 97.492 21.276 96.612 99.398 72.356 89.851 96.804 63.080 [2021-07-07 23:41:40,205][root][INFO] - Finished test. Elapsed time: 34.8760 [2021-07-07 23:41:41,204][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 23:41:54,542][root][INFO] - ===> Epoch[3361](16805/5): Loss 0.0128, Sem 0.0128 LR: 7.440e-02 Score 99.548 Data time: 0.1799, Forward time: 1.2628, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.6302 [2021-07-07 23:42:08,534][root][INFO] - ===> Epoch[3362](16810/5): Loss 0.0124, Sem 0.0124 LR: 7.439e-02 Score 99.550 Data time: 0.2050, Forward time: 1.2628, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7461 [2021-07-07 23:42:22,349][root][INFO] - ===> Epoch[3363](16815/5): Loss 0.0123, Sem 0.0123 LR: 7.438e-02 Score 99.559 Data time: 0.2100, Forward time: 1.2628, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7185 [2021-07-07 23:42:35,651][root][INFO] - ===> Epoch[3364](16820/5): Loss 0.0126, Sem 0.0126 LR: 7.437e-02 Score 99.539 Data time: 0.2152, Forward time: 1.2628, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.6107 [2021-07-07 23:42:49,582][root][INFO] - ===> Epoch[3365](16825/5): Loss 0.0124, Sem 0.0124 LR: 7.437e-02 Score 99.554 Data time: 0.2335, Forward time: 1.2628, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7321 [2021-07-07 23:43:03,102][root][INFO] - ===> Epoch[3366](16830/5): Loss 0.0124, Sem 0.0124 LR: 7.436e-02 Score 99.552 Data time: 0.2415, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.6502 [2021-07-07 23:43:17,218][root][INFO] - ===> Epoch[3367](16835/5): Loss 0.0118, Sem 0.0118 LR: 7.435e-02 Score 99.572 Data time: 0.2059, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7779 [2021-07-07 23:43:31,652][root][INFO] - ===> Epoch[3368](16840/5): Loss 0.0118, Sem 0.0118 LR: 7.434e-02 Score 99.573 Data time: 0.2303, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8463 [2021-07-07 23:43:45,463][root][INFO] - ===> Epoch[3369](16845/5): Loss 0.0125, Sem 0.0125 LR: 7.434e-02 Score 99.548 Data time: 0.2075, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7112 [2021-07-07 23:43:58,645][root][INFO] - ===> Epoch[3370](16850/5): Loss 0.0128, Sem 0.0128 LR: 7.433e-02 Score 99.543 Data time: 0.2908, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.5888 [2021-07-07 23:44:12,165][root][INFO] - ===> Epoch[3371](16855/5): Loss 0.0123, Sem 0.0123 LR: 7.432e-02 Score 99.559 Data time: 0.1851, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.6692 [2021-07-07 23:44:26,020][root][INFO] - ===> Epoch[3372](16860/5): Loss 0.0120, Sem 0.0120 LR: 7.431e-02 Score 99.560 Data time: 0.2149, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7307 [2021-07-07 23:44:39,519][root][INFO] - ===> Epoch[3373](16865/5): Loss 0.0124, Sem 0.0124 LR: 7.430e-02 Score 99.551 Data time: 0.1794, Forward time: 1.2628, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6540 [2021-07-07 23:44:54,079][root][INFO] - ===> Epoch[3374](16870/5): Loss 0.0119, Sem 0.0119 LR: 7.430e-02 Score 99.570 Data time: 0.2149, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8711 [2021-07-07 23:45:07,785][root][INFO] - ===> Epoch[3375](16875/5): Loss 0.0121, Sem 0.0121 LR: 7.429e-02 Score 99.565 Data time: 0.2757, Forward time: 1.2628, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6833 [2021-07-07 23:45:21,706][root][INFO] - ===> Epoch[3376](16880/5): Loss 0.0120, Sem 0.0120 LR: 7.428e-02 Score 99.567 Data time: 0.2469, Forward time: 1.2628, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7274 [2021-07-07 23:45:35,443][root][INFO] - ===> Epoch[3377](16885/5): Loss 0.0126, Sem 0.0126 LR: 7.427e-02 Score 99.546 Data time: 0.2235, Forward time: 1.2628, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6971 [2021-07-07 23:45:48,682][root][INFO] - ===> Epoch[3378](16890/5): Loss 0.0125, Sem 0.0125 LR: 7.427e-02 Score 99.553 Data time: 0.2016, Forward time: 1.2628, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6062 [2021-07-07 23:46:03,021][root][INFO] - ===> Epoch[3379](16895/5): Loss 0.0121, Sem 0.0121 LR: 7.426e-02 Score 99.565 Data time: 0.2036, Forward time: 1.2627, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8256 [2021-07-07 23:46:16,361][root][INFO] - ===> Epoch[3380](16900/5): Loss 0.0131, Sem 0.0131 LR: 7.425e-02 Score 99.529 Data time: 0.1856, Forward time: 1.2628, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6151 [2021-07-07 23:46:30,809][root][INFO] - ===> Epoch[3381](16905/5): Loss 0.0127, Sem 0.0127 LR: 7.424e-02 Score 99.537 Data time: 0.2644, Forward time: 1.2628, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8427 [2021-07-07 23:46:44,763][root][INFO] - ===> Epoch[3382](16910/5): Loss 0.0119, Sem 0.0119 LR: 7.423e-02 Score 99.568 Data time: 0.2956, Forward time: 1.2628, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7435 [2021-07-07 23:47:00,083][root][INFO] - ===> Epoch[3383](16915/5): Loss 0.0127, Sem 0.0127 LR: 7.423e-02 Score 99.537 Data time: 0.2778, Forward time: 1.2628, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 3.0137 [2021-07-07 23:47:15,135][root][INFO] - ===> Epoch[3384](16920/5): Loss 0.0124, Sem 0.0124 LR: 7.422e-02 Score 99.547 Data time: 0.2107, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.9645 [2021-07-07 23:47:28,512][root][INFO] - ===> Epoch[3385](16925/5): Loss 0.0125, Sem 0.0125 LR: 7.421e-02 Score 99.547 Data time: 0.1777, Forward time: 1.2627, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.6365 [2021-07-07 23:47:42,576][root][INFO] - ===> Epoch[3386](16930/5): Loss 0.0127, Sem 0.0127 LR: 7.420e-02 Score 99.541 Data time: 0.2566, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7692 [2021-07-07 23:47:56,776][root][INFO] - ===> Epoch[3387](16935/5): Loss 0.0127, Sem 0.0127 LR: 7.420e-02 Score 99.538 Data time: 0.2141, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7882 [2021-07-07 23:48:11,764][root][INFO] - ===> Epoch[3388](16940/5): Loss 0.0123, Sem 0.0123 LR: 7.419e-02 Score 99.555 Data time: 0.2187, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.9548 [2021-07-07 23:48:26,173][root][INFO] - ===> Epoch[3389](16945/5): Loss 0.0125, Sem 0.0125 LR: 7.418e-02 Score 99.547 Data time: 0.3295, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8368 [2021-07-07 23:48:40,236][root][INFO] - ===> Epoch[3390](16950/5): Loss 0.0120, Sem 0.0120 LR: 7.417e-02 Score 99.566 Data time: 0.2388, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7684 [2021-07-07 23:48:54,160][root][INFO] - ===> Epoch[3391](16955/5): Loss 0.0121, Sem 0.0121 LR: 7.416e-02 Score 99.562 Data time: 0.1979, Forward time: 1.2628, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7419 [2021-07-07 23:49:08,544][root][INFO] - ===> Epoch[3392](16960/5): Loss 0.0116, Sem 0.0116 LR: 7.416e-02 Score 99.583 Data time: 0.3047, Forward time: 1.2629, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8269 [2021-07-07 23:49:22,007][root][INFO] - ===> Epoch[3393](16965/5): Loss 0.0130, Sem 0.0130 LR: 7.415e-02 Score 99.529 Data time: 0.1994, Forward time: 1.2629, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6499 [2021-07-07 23:49:35,862][root][INFO] - ===> Epoch[3394](16970/5): Loss 0.0120, Sem 0.0120 LR: 7.414e-02 Score 99.566 Data time: 0.2465, Forward time: 1.2629, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7205 [2021-07-07 23:49:49,812][root][INFO] - ===> Epoch[3395](16975/5): Loss 0.0122, Sem 0.0122 LR: 7.413e-02 Score 99.559 Data time: 0.3041, Forward time: 1.2629, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7314 [2021-07-07 23:50:04,541][root][INFO] - ===> Epoch[3396](16980/5): Loss 0.0122, Sem 0.0122 LR: 7.413e-02 Score 99.564 Data time: 0.2659, Forward time: 1.2629, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.9095 [2021-07-07 23:50:18,856][root][INFO] - ===> Epoch[3397](16985/5): Loss 0.0125, Sem 0.0125 LR: 7.412e-02 Score 99.550 Data time: 0.2145, Forward time: 1.2629, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8153 [2021-07-07 23:50:33,113][root][INFO] - ===> Epoch[3398](16990/5): Loss 0.0120, Sem 0.0120 LR: 7.411e-02 Score 99.565 Data time: 0.2478, Forward time: 1.2629, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8055 [2021-07-07 23:50:46,823][root][INFO] - ===> Epoch[3399](16995/5): Loss 0.0121, Sem 0.0121 LR: 7.410e-02 Score 99.562 Data time: 0.3210, Forward time: 1.2629, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6985 [2021-07-07 23:51:00,473][root][INFO] - ===> Epoch[3400](17000/5): Loss 0.0119, Sem 0.0119 LR: 7.409e-02 Score 99.571 Data time: 0.2501, Forward time: 1.2629, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6895 [2021-07-07 23:51:00,475][root][INFO] - ===> Start testing [2021-07-07 23:51:36,833][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1244 Loss 0.193 (AVG: 0.542) Score 95.531 (AVG: 91.265) mIOU 71.065 mAP 84.652 mAcc 77.239 IOU: 66.452 0.084 84.192 77.617 94.634 92.074 25.457 84.597 98.398 75.543 80.295 86.577 57.925 mAP: 84.048 15.012 94.923 89.741 98.758 98.713 51.290 98.297 99.797 93.450 91.854 93.580 91.014 mAcc: 81.782 0.820 91.119 83.728 96.062 97.536 27.331 96.624 99.433 79.963 90.292 96.620 62.799 [2021-07-07 23:51:36,833][root][INFO] - Finished test. Elapsed time: 36.3572 [2021-07-07 23:51:37,935][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-07 23:51:52,360][root][INFO] - ===> Epoch[3401](17005/5): Loss 0.0117, Sem 0.0117 LR: 7.409e-02 Score 99.573 Data time: 0.2865, Forward time: 1.2629, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8399 [2021-07-07 23:52:06,176][root][INFO] - ===> Epoch[3402](17010/5): Loss 0.0120, Sem 0.0120 LR: 7.408e-02 Score 99.571 Data time: 0.1957, Forward time: 1.2629, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7209 [2021-07-07 23:52:20,218][root][INFO] - ===> Epoch[3403](17015/5): Loss 0.0122, Sem 0.0122 LR: 7.407e-02 Score 99.556 Data time: 0.2720, Forward time: 1.2630, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7613 [2021-07-07 23:52:33,980][root][INFO] - ===> Epoch[3404](17020/5): Loss 0.0122, Sem 0.0122 LR: 7.406e-02 Score 99.557 Data time: 0.2034, Forward time: 1.2630, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7136 [2021-07-07 23:52:48,838][root][INFO] - ===> Epoch[3405](17025/5): Loss 0.0116, Sem 0.0116 LR: 7.406e-02 Score 99.579 Data time: 0.1982, Forward time: 1.2630, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.9181 [2021-07-07 23:53:02,477][root][INFO] - ===> Epoch[3406](17030/5): Loss 0.0123, Sem 0.0123 LR: 7.405e-02 Score 99.559 Data time: 0.2439, Forward time: 1.2630, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6877 [2021-07-07 23:53:16,762][root][INFO] - ===> Epoch[3407](17035/5): Loss 0.0120, Sem 0.0120 LR: 7.404e-02 Score 99.567 Data time: 0.2003, Forward time: 1.2630, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8046 [2021-07-07 23:53:30,855][root][INFO] - ===> Epoch[3408](17040/5): Loss 0.0120, Sem 0.0120 LR: 7.403e-02 Score 99.567 Data time: 0.2678, Forward time: 1.2630, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7608 [2021-07-07 23:53:44,365][root][INFO] - ===> Epoch[3409](17045/5): Loss 0.0121, Sem 0.0121 LR: 7.403e-02 Score 99.562 Data time: 0.2369, Forward time: 1.2631, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6570 [2021-07-07 23:53:58,768][root][INFO] - ===> Epoch[3410](17050/5): Loss 0.0114, Sem 0.0114 LR: 7.402e-02 Score 99.591 Data time: 0.2315, Forward time: 1.2631, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8369 [2021-07-07 23:54:12,762][root][INFO] - ===> Epoch[3411](17055/5): Loss 0.0122, Sem 0.0122 LR: 7.401e-02 Score 99.561 Data time: 0.2081, Forward time: 1.2631, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7641 [2021-07-07 23:54:26,506][root][INFO] - ===> Epoch[3412](17060/5): Loss 0.0120, Sem 0.0120 LR: 7.400e-02 Score 99.568 Data time: 0.2407, Forward time: 1.2631, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7037 [2021-07-07 23:54:40,894][root][INFO] - ===> Epoch[3413](17065/5): Loss 0.0118, Sem 0.0118 LR: 7.399e-02 Score 99.573 Data time: 0.2206, Forward time: 1.2631, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8320 [2021-07-07 23:54:55,533][root][INFO] - ===> Epoch[3414](17070/5): Loss 0.0116, Sem 0.0116 LR: 7.399e-02 Score 99.576 Data time: 0.2414, Forward time: 1.2631, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8878 [2021-07-07 23:55:09,724][root][INFO] - ===> Epoch[3415](17075/5): Loss 0.0116, Sem 0.0116 LR: 7.398e-02 Score 99.581 Data time: 0.2592, Forward time: 1.2631, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7894 [2021-07-07 23:55:23,493][root][INFO] - ===> Epoch[3416](17080/5): Loss 0.0115, Sem 0.0115 LR: 7.397e-02 Score 99.585 Data time: 0.2445, Forward time: 1.2632, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7049 [2021-07-07 23:55:37,620][root][INFO] - ===> Epoch[3417](17085/5): Loss 0.0119, Sem 0.0119 LR: 7.396e-02 Score 99.573 Data time: 0.2221, Forward time: 1.2632, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7857 [2021-07-07 23:55:51,435][root][INFO] - ===> Epoch[3418](17090/5): Loss 0.0114, Sem 0.0114 LR: 7.396e-02 Score 99.588 Data time: 0.2692, Forward time: 1.2632, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7137 [2021-07-07 23:56:06,122][root][INFO] - ===> Epoch[3419](17095/5): Loss 0.0119, Sem 0.0119 LR: 7.395e-02 Score 99.563 Data time: 0.2105, Forward time: 1.2632, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.8983 [2021-07-07 23:56:20,419][root][INFO] - ===> Epoch[3420](17100/5): Loss 0.0117, Sem 0.0117 LR: 7.394e-02 Score 99.577 Data time: 0.1356, Forward time: 1.2632, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8321 [2021-07-07 23:56:35,712][root][INFO] - ===> Epoch[3421](17105/5): Loss 0.0115, Sem 0.0115 LR: 7.393e-02 Score 99.581 Data time: 0.2992, Forward time: 1.2632, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 3.0168 [2021-07-07 23:56:49,843][root][INFO] - ===> Epoch[3422](17110/5): Loss 0.0130, Sem 0.0130 LR: 7.392e-02 Score 99.535 Data time: 0.2757, Forward time: 1.2632, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7801 [2021-07-07 23:57:03,370][root][INFO] - ===> Epoch[3423](17115/5): Loss 0.0192, Sem 0.0192 LR: 7.392e-02 Score 99.354 Data time: 0.2557, Forward time: 1.2632, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.6602 [2021-07-07 23:57:17,771][root][INFO] - ===> Epoch[3424](17120/5): Loss 0.0256, Sem 0.0256 LR: 7.391e-02 Score 99.200 Data time: 0.2518, Forward time: 1.2632, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.8403 [2021-07-07 23:57:31,143][root][INFO] - ===> Epoch[3425](17125/5): Loss 0.0268, Sem 0.0268 LR: 7.390e-02 Score 99.114 Data time: 0.2164, Forward time: 1.2632, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.6350 [2021-07-07 23:57:44,979][root][INFO] - ===> Epoch[3426](17130/5): Loss 0.0210, Sem 0.0210 LR: 7.389e-02 Score 99.255 Data time: 0.2645, Forward time: 1.2633, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7003 [2021-07-07 23:57:58,411][root][INFO] - ===> Epoch[3427](17135/5): Loss 0.0236, Sem 0.0236 LR: 7.389e-02 Score 99.203 Data time: 0.2159, Forward time: 1.2633, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6416 [2021-07-07 23:58:13,073][root][INFO] - ===> Epoch[3428](17140/5): Loss 0.0210, Sem 0.0210 LR: 7.388e-02 Score 99.264 Data time: 0.2533, Forward time: 1.2633, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8772 [2021-07-07 23:58:27,885][root][INFO] - ===> Epoch[3429](17145/5): Loss 0.0191, Sem 0.0191 LR: 7.387e-02 Score 99.326 Data time: 0.1972, Forward time: 1.2633, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.9180 [2021-07-07 23:58:42,386][root][INFO] - ===> Epoch[3430](17150/5): Loss 0.0193, Sem 0.0193 LR: 7.386e-02 Score 99.331 Data time: 0.2680, Forward time: 1.2634, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8559 [2021-07-07 23:58:56,321][root][INFO] - ===> Epoch[3431](17155/5): Loss 0.0183, Sem 0.0183 LR: 7.385e-02 Score 99.361 Data time: 0.2044, Forward time: 1.2634, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7419 [2021-07-07 23:59:09,742][root][INFO] - ===> Epoch[3432](17160/5): Loss 0.0199, Sem 0.0199 LR: 7.385e-02 Score 99.313 Data time: 0.2796, Forward time: 1.2633, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6258 [2021-07-07 23:59:23,677][root][INFO] - ===> Epoch[3433](17165/5): Loss 0.0184, Sem 0.0184 LR: 7.384e-02 Score 99.349 Data time: 0.2659, Forward time: 1.2633, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7435 [2021-07-07 23:59:37,142][root][INFO] - ===> Epoch[3434](17170/5): Loss 0.0187, Sem 0.0187 LR: 7.383e-02 Score 99.342 Data time: 0.2304, Forward time: 1.2633, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6413 [2021-07-07 23:59:51,757][root][INFO] - ===> Epoch[3435](17175/5): Loss 0.0196, Sem 0.0196 LR: 7.382e-02 Score 99.338 Data time: 0.2333, Forward time: 1.2633, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8748 [2021-07-08 00:00:05,376][root][INFO] - ===> Epoch[3436](17180/5): Loss 0.0184, Sem 0.0184 LR: 7.382e-02 Score 99.345 Data time: 0.2359, Forward time: 1.2633, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.6789 [2021-07-08 00:00:19,607][root][INFO] - ===> Epoch[3437](17185/5): Loss 0.0196, Sem 0.0196 LR: 7.381e-02 Score 99.313 Data time: 0.2304, Forward time: 1.2634, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7922 [2021-07-08 00:00:33,468][root][INFO] - ===> Epoch[3438](17190/5): Loss 0.0195, Sem 0.0195 LR: 7.380e-02 Score 99.311 Data time: 0.2463, Forward time: 1.2634, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7213 [2021-07-08 00:00:47,524][root][INFO] - ===> Epoch[3439](17195/5): Loss 0.0197, Sem 0.0197 LR: 7.379e-02 Score 99.312 Data time: 0.2259, Forward time: 1.2634, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7640 [2021-07-08 00:01:01,663][root][INFO] - ===> Epoch[3440](17200/5): Loss 0.0190, Sem 0.0190 LR: 7.378e-02 Score 99.319 Data time: 0.2181, Forward time: 1.2634, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7789 [2021-07-08 00:01:01,665][root][INFO] - ===> Start testing [2021-07-08 00:01:36,046][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1252 Loss 0.213 (AVG: 0.526) Score 94.330 (AVG: 90.772) mIOU 69.152 mAP 84.029 mAcc 77.136 IOU: 64.614 0.000 77.020 75.338 94.549 91.964 20.809 81.243 98.163 71.039 78.064 86.304 59.877 mAP: 82.634 10.002 93.913 89.396 98.880 98.748 51.655 96.558 99.713 92.503 92.248 94.725 91.402 mAcc: 81.243 0.000 91.486 80.876 96.134 97.646 21.629 93.356 99.339 89.177 90.066 96.532 65.281 [2021-07-08 00:01:36,047][root][INFO] - Finished test. Elapsed time: 34.3809 [2021-07-08 00:01:37,062][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 00:01:49,902][root][INFO] - ===> Epoch[3441](17205/5): Loss 0.0221, Sem 0.0221 LR: 7.378e-02 Score 99.266 Data time: 0.2098, Forward time: 1.2634, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.5364 [2021-07-08 00:02:05,281][root][INFO] - ===> Epoch[3442](17210/5): Loss 0.0231, Sem 0.0231 LR: 7.377e-02 Score 99.202 Data time: 0.2490, Forward time: 1.2635, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 3.0237 [2021-07-08 00:02:19,425][root][INFO] - ===> Epoch[3443](17215/5): Loss 0.0271, Sem 0.0271 LR: 7.376e-02 Score 99.106 Data time: 0.1590, Forward time: 1.2634, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7836 [2021-07-08 00:02:33,717][root][INFO] - ===> Epoch[3444](17220/5): Loss 0.0290, Sem 0.0290 LR: 7.375e-02 Score 99.033 Data time: 0.2209, Forward time: 1.2634, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.8012 [2021-07-08 00:02:48,532][root][INFO] - ===> Epoch[3445](17225/5): Loss 0.0250, Sem 0.0250 LR: 7.375e-02 Score 99.140 Data time: 0.2940, Forward time: 1.2635, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.9080 [2021-07-08 00:03:02,334][root][INFO] - ===> Epoch[3446](17230/5): Loss 0.0239, Sem 0.0239 LR: 7.374e-02 Score 99.192 Data time: 0.1767, Forward time: 1.2635, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7238 [2021-07-08 00:03:16,380][root][INFO] - ===> Epoch[3447](17235/5): Loss 0.0266, Sem 0.0266 LR: 7.373e-02 Score 99.107 Data time: 0.2604, Forward time: 1.2635, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7528 [2021-07-08 00:03:30,396][root][INFO] - ===> Epoch[3448](17240/5): Loss 0.0267, Sem 0.0267 LR: 7.372e-02 Score 99.091 Data time: 0.2730, Forward time: 1.2636, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7640 [2021-07-08 00:03:43,577][root][INFO] - ===> Epoch[3449](17245/5): Loss 0.0316, Sem 0.0316 LR: 7.371e-02 Score 98.988 Data time: 0.1895, Forward time: 1.2635, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.6023 [2021-07-08 00:03:57,324][root][INFO] - ===> Epoch[3450](17250/5): Loss 0.0266, Sem 0.0266 LR: 7.371e-02 Score 99.077 Data time: 0.2609, Forward time: 1.2635, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7024 [2021-07-08 00:04:10,670][root][INFO] - ===> Epoch[3451](17255/5): Loss 0.0271, Sem 0.0271 LR: 7.370e-02 Score 99.064 Data time: 0.1916, Forward time: 1.2636, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6294 [2021-07-08 00:04:25,042][root][INFO] - ===> Epoch[3452](17260/5): Loss 0.0285, Sem 0.0285 LR: 7.369e-02 Score 99.029 Data time: 0.2401, Forward time: 1.2635, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8236 [2021-07-08 00:04:38,956][root][INFO] - ===> Epoch[3453](17265/5): Loss 0.0289, Sem 0.0289 LR: 7.368e-02 Score 99.012 Data time: 0.1976, Forward time: 1.2636, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7334 [2021-07-08 00:04:52,249][root][INFO] - ===> Epoch[3454](17270/5): Loss 0.0277, Sem 0.0277 LR: 7.368e-02 Score 99.044 Data time: 0.2026, Forward time: 1.2636, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.5980 [2021-07-08 00:05:06,154][root][INFO] - ===> Epoch[3455](17275/5): Loss 0.0250, Sem 0.0250 LR: 7.367e-02 Score 99.126 Data time: 0.2611, Forward time: 1.2636, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7248 [2021-07-08 00:05:20,094][root][INFO] - ===> Epoch[3456](17280/5): Loss 0.0241, Sem 0.0241 LR: 7.366e-02 Score 99.168 Data time: 0.2246, Forward time: 1.2636, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7412 [2021-07-08 00:05:33,999][root][INFO] - ===> Epoch[3457](17285/5): Loss 0.0296, Sem 0.0296 LR: 7.365e-02 Score 99.060 Data time: 0.2112, Forward time: 1.2636, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7381 [2021-07-08 00:05:48,719][root][INFO] - ===> Epoch[3458](17290/5): Loss 0.0293, Sem 0.0293 LR: 7.364e-02 Score 99.099 Data time: 0.2185, Forward time: 1.2636, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8964 [2021-07-08 00:06:03,460][root][INFO] - ===> Epoch[3459](17295/5): Loss 0.0265, Sem 0.0265 LR: 7.364e-02 Score 99.112 Data time: 0.2421, Forward time: 1.2636, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.9023 [2021-07-08 00:06:17,124][root][INFO] - ===> Epoch[3460](17300/5): Loss 0.0289, Sem 0.0289 LR: 7.363e-02 Score 99.067 Data time: 0.2212, Forward time: 1.2637, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6893 [2021-07-08 00:06:31,338][root][INFO] - ===> Epoch[3461](17305/5): Loss 0.0275, Sem 0.0275 LR: 7.362e-02 Score 99.061 Data time: 0.2232, Forward time: 1.2637, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7908 [2021-07-08 00:06:45,864][root][INFO] - ===> Epoch[3462](17310/5): Loss 0.0310, Sem 0.0310 LR: 7.361e-02 Score 98.971 Data time: 0.2945, Forward time: 1.2637, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8572 [2021-07-08 00:07:00,489][root][INFO] - ===> Epoch[3463](17315/5): Loss 0.0289, Sem 0.0289 LR: 7.361e-02 Score 99.017 Data time: 0.2420, Forward time: 1.2638, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8841 [2021-07-08 00:07:14,827][root][INFO] - ===> Epoch[3464](17320/5): Loss 0.0273, Sem 0.0273 LR: 7.360e-02 Score 99.063 Data time: 0.2251, Forward time: 1.2638, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8113 [2021-07-08 00:07:28,462][root][INFO] - ===> Epoch[3465](17325/5): Loss 0.0305, Sem 0.0305 LR: 7.359e-02 Score 99.003 Data time: 0.2688, Forward time: 1.2639, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6624 [2021-07-08 00:07:42,606][root][INFO] - ===> Epoch[3466](17330/5): Loss 0.0355, Sem 0.0355 LR: 7.358e-02 Score 98.876 Data time: 0.2093, Forward time: 1.2639, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7808 [2021-07-08 00:07:57,009][root][INFO] - ===> Epoch[3467](17335/5): Loss 0.0352, Sem 0.0352 LR: 7.358e-02 Score 98.866 Data time: 0.3040, Forward time: 1.2639, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8320 [2021-07-08 00:08:11,226][root][INFO] - ===> Epoch[3468](17340/5): Loss 0.0332, Sem 0.0332 LR: 7.357e-02 Score 98.885 Data time: 0.2588, Forward time: 1.2640, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7964 [2021-07-08 00:08:25,850][root][INFO] - ===> Epoch[3469](17345/5): Loss 0.0330, Sem 0.0330 LR: 7.356e-02 Score 98.874 Data time: 0.2281, Forward time: 1.2640, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8802 [2021-07-08 00:08:40,648][root][INFO] - ===> Epoch[3470](17350/5): Loss 0.0325, Sem 0.0325 LR: 7.355e-02 Score 98.892 Data time: 0.1958, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9211 [2021-07-08 00:08:54,922][root][INFO] - ===> Epoch[3471](17355/5): Loss 0.0357, Sem 0.0357 LR: 7.354e-02 Score 98.816 Data time: 0.1866, Forward time: 1.2640, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8100 [2021-07-08 00:09:09,459][root][INFO] - ===> Epoch[3472](17360/5): Loss 0.0313, Sem 0.0313 LR: 7.354e-02 Score 98.953 Data time: 0.2241, Forward time: 1.2640, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8591 [2021-07-08 00:09:23,717][root][INFO] - ===> Epoch[3473](17365/5): Loss 0.0329, Sem 0.0329 LR: 7.353e-02 Score 98.884 Data time: 0.2472, Forward time: 1.2640, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8105 [2021-07-08 00:09:37,162][root][INFO] - ===> Epoch[3474](17370/5): Loss 0.0303, Sem 0.0303 LR: 7.352e-02 Score 98.956 Data time: 0.2426, Forward time: 1.2640, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6363 [2021-07-08 00:09:51,330][root][INFO] - ===> Epoch[3475](17375/5): Loss 0.0295, Sem 0.0295 LR: 7.351e-02 Score 98.990 Data time: 0.2646, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7852 [2021-07-08 00:10:05,623][root][INFO] - ===> Epoch[3476](17380/5): Loss 0.0298, Sem 0.0298 LR: 7.351e-02 Score 99.001 Data time: 0.2676, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8077 [2021-07-08 00:10:19,803][root][INFO] - ===> Epoch[3477](17385/5): Loss 0.0290, Sem 0.0290 LR: 7.350e-02 Score 99.001 Data time: 0.2434, Forward time: 1.2640, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8022 [2021-07-08 00:10:34,483][root][INFO] - ===> Epoch[3478](17390/5): Loss 0.0273, Sem 0.0273 LR: 7.349e-02 Score 99.076 Data time: 0.2848, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8831 [2021-07-08 00:10:49,192][root][INFO] - ===> Epoch[3479](17395/5): Loss 0.0251, Sem 0.0251 LR: 7.348e-02 Score 99.111 Data time: 0.2123, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8942 [2021-07-08 00:11:02,911][root][INFO] - ===> Epoch[3480](17400/5): Loss 0.0232, Sem 0.0232 LR: 7.347e-02 Score 99.186 Data time: 0.2653, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6887 [2021-07-08 00:11:02,914][root][INFO] - ===> Start testing [2021-07-08 00:11:36,959][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1241 Loss 0.366 (AVG: 0.536) Score 91.348 (AVG: 90.255) mIOU 68.519 mAP 82.669 mAcc 75.451 IOU: 62.873 0.000 76.674 74.316 94.244 91.051 19.886 79.973 97.989 69.631 78.509 85.175 60.428 mAP: 84.453 2.328 88.415 90.321 98.550 98.921 47.467 95.713 99.759 89.480 93.114 94.458 91.722 mAcc: 79.046 0.000 81.085 78.924 95.831 97.500 21.438 92.107 99.534 80.485 92.683 96.617 65.616 [2021-07-08 00:11:36,959][root][INFO] - Finished test. Elapsed time: 34.0444 [2021-07-08 00:11:37,986][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 00:11:50,978][root][INFO] - ===> Epoch[3481](17405/5): Loss 0.0227, Sem 0.0227 LR: 7.347e-02 Score 99.197 Data time: 0.2525, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.5626 [2021-07-08 00:12:05,195][root][INFO] - ===> Epoch[3482](17410/5): Loss 0.0226, Sem 0.0226 LR: 7.346e-02 Score 99.221 Data time: 0.2482, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7997 [2021-07-08 00:12:19,262][root][INFO] - ===> Epoch[3483](17415/5): Loss 0.0218, Sem 0.0218 LR: 7.345e-02 Score 99.234 Data time: 0.2884, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7580 [2021-07-08 00:12:33,338][root][INFO] - ===> Epoch[3484](17420/5): Loss 0.0204, Sem 0.0204 LR: 7.344e-02 Score 99.273 Data time: 0.2379, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7656 [2021-07-08 00:12:47,909][root][INFO] - ===> Epoch[3485](17425/5): Loss 0.0207, Sem 0.0207 LR: 7.344e-02 Score 99.288 Data time: 0.2381, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8650 [2021-07-08 00:13:02,187][root][INFO] - ===> Epoch[3486](17430/5): Loss 0.0187, Sem 0.0187 LR: 7.343e-02 Score 99.338 Data time: 0.2438, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8153 [2021-07-08 00:13:16,229][root][INFO] - ===> Epoch[3487](17435/5): Loss 0.0193, Sem 0.0193 LR: 7.342e-02 Score 99.320 Data time: 0.2846, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7574 [2021-07-08 00:13:30,985][root][INFO] - ===> Epoch[3488](17440/5): Loss 0.0194, Sem 0.0194 LR: 7.341e-02 Score 99.319 Data time: 0.2542, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8961 [2021-07-08 00:13:45,387][root][INFO] - ===> Epoch[3489](17445/5): Loss 0.0217, Sem 0.0217 LR: 7.340e-02 Score 99.277 Data time: 0.2181, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8347 [2021-07-08 00:14:00,722][root][INFO] - ===> Epoch[3490](17450/5): Loss 0.0189, Sem 0.0189 LR: 7.340e-02 Score 99.330 Data time: 0.3228, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 3.0140 [2021-07-08 00:14:15,058][root][INFO] - ===> Epoch[3491](17455/5): Loss 0.0193, Sem 0.0193 LR: 7.339e-02 Score 99.307 Data time: 0.2466, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8229 [2021-07-08 00:14:29,429][root][INFO] - ===> Epoch[3492](17460/5): Loss 0.0188, Sem 0.0188 LR: 7.338e-02 Score 99.344 Data time: 0.2597, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8188 [2021-07-08 00:14:43,880][root][INFO] - ===> Epoch[3493](17465/5): Loss 0.0179, Sem 0.0179 LR: 7.337e-02 Score 99.367 Data time: 0.2748, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8516 [2021-07-08 00:14:58,574][root][INFO] - ===> Epoch[3494](17470/5): Loss 0.0188, Sem 0.0188 LR: 7.337e-02 Score 99.334 Data time: 0.2001, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8957 [2021-07-08 00:15:12,631][root][INFO] - ===> Epoch[3495](17475/5): Loss 0.0181, Sem 0.0181 LR: 7.336e-02 Score 99.357 Data time: 0.2919, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7443 [2021-07-08 00:15:26,997][root][INFO] - ===> Epoch[3496](17480/5): Loss 0.0189, Sem 0.0189 LR: 7.335e-02 Score 99.328 Data time: 0.2072, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8340 [2021-07-08 00:15:40,734][root][INFO] - ===> Epoch[3497](17485/5): Loss 0.0207, Sem 0.0207 LR: 7.334e-02 Score 99.296 Data time: 0.2380, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7108 [2021-07-08 00:15:55,114][root][INFO] - ===> Epoch[3498](17490/5): Loss 0.0191, Sem 0.0191 LR: 7.333e-02 Score 99.324 Data time: 0.2256, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8282 [2021-07-08 00:16:08,790][root][INFO] - ===> Epoch[3499](17495/5): Loss 0.0203, Sem 0.0203 LR: 7.333e-02 Score 99.265 Data time: 0.1964, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6955 [2021-07-08 00:16:23,017][root][INFO] - ===> Epoch[3500](17500/5): Loss 0.0221, Sem 0.0221 LR: 7.332e-02 Score 99.246 Data time: 0.2047, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7954 [2021-07-08 00:16:37,468][root][INFO] - ===> Epoch[3501](17505/5): Loss 0.0200, Sem 0.0200 LR: 7.331e-02 Score 99.290 Data time: 0.2355, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8447 [2021-07-08 00:16:51,527][root][INFO] - ===> Epoch[3502](17510/5): Loss 0.0208, Sem 0.0208 LR: 7.330e-02 Score 99.275 Data time: 0.1878, Forward time: 1.2642, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7630 [2021-07-08 00:17:05,485][root][INFO] - ===> Epoch[3503](17515/5): Loss 0.0208, Sem 0.0208 LR: 7.330e-02 Score 99.289 Data time: 0.2483, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7510 [2021-07-08 00:17:20,046][root][INFO] - ===> Epoch[3504](17520/5): Loss 0.0193, Sem 0.0193 LR: 7.329e-02 Score 99.316 Data time: 0.3017, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8535 [2021-07-08 00:17:33,987][root][INFO] - ===> Epoch[3505](17525/5): Loss 0.0199, Sem 0.0199 LR: 7.328e-02 Score 99.306 Data time: 0.3613, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7437 [2021-07-08 00:17:47,810][root][INFO] - ===> Epoch[3506](17530/5): Loss 0.0193, Sem 0.0193 LR: 7.327e-02 Score 99.313 Data time: 0.1773, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7260 [2021-07-08 00:18:02,229][root][INFO] - ===> Epoch[3507](17535/5): Loss 0.0180, Sem 0.0180 LR: 7.326e-02 Score 99.365 Data time: 0.2835, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8402 [2021-07-08 00:18:16,755][root][INFO] - ===> Epoch[3508](17540/5): Loss 0.0188, Sem 0.0188 LR: 7.326e-02 Score 99.334 Data time: 0.1994, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8631 [2021-07-08 00:18:31,185][root][INFO] - ===> Epoch[3509](17545/5): Loss 0.0187, Sem 0.0187 LR: 7.325e-02 Score 99.346 Data time: 0.2594, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8350 [2021-07-08 00:18:45,042][root][INFO] - ===> Epoch[3510](17550/5): Loss 0.0234, Sem 0.0234 LR: 7.324e-02 Score 99.243 Data time: 0.2465, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7308 [2021-07-08 00:18:58,708][root][INFO] - ===> Epoch[3511](17555/5): Loss 0.0242, Sem 0.0242 LR: 7.323e-02 Score 99.165 Data time: 0.2399, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6875 [2021-07-08 00:19:13,773][root][INFO] - ===> Epoch[3512](17560/5): Loss 0.0226, Sem 0.0226 LR: 7.323e-02 Score 99.218 Data time: 0.2505, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9611 [2021-07-08 00:19:27,880][root][INFO] - ===> Epoch[3513](17565/5): Loss 0.0237, Sem 0.0237 LR: 7.322e-02 Score 99.186 Data time: 0.2164, Forward time: 1.2643, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7735 [2021-07-08 00:19:42,816][root][INFO] - ===> Epoch[3514](17570/5): Loss 0.0304, Sem 0.0304 LR: 7.321e-02 Score 98.927 Data time: 0.2560, Forward time: 1.2643, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9409 [2021-07-08 00:19:55,510][root][INFO] - ===> Epoch[3515](17575/5): Loss 0.0334, Sem 0.0334 LR: 7.320e-02 Score 98.921 Data time: 0.2069, Forward time: 1.2643, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.4895 [2021-07-08 00:20:09,607][root][INFO] - ===> Epoch[3516](17580/5): Loss 0.0301, Sem 0.0301 LR: 7.319e-02 Score 99.020 Data time: 0.1823, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7814 [2021-07-08 00:20:23,677][root][INFO] - ===> Epoch[3517](17585/5): Loss 0.0282, Sem 0.0282 LR: 7.319e-02 Score 99.031 Data time: 0.1891, Forward time: 1.2642, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7715 [2021-07-08 00:20:37,977][root][INFO] - ===> Epoch[3518](17590/5): Loss 0.0277, Sem 0.0277 LR: 7.318e-02 Score 99.047 Data time: 0.2456, Forward time: 1.2642, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8159 [2021-07-08 00:20:52,256][root][INFO] - ===> Epoch[3519](17595/5): Loss 0.0253, Sem 0.0253 LR: 7.317e-02 Score 99.119 Data time: 0.1550, Forward time: 1.2642, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8170 [2021-07-08 00:21:06,305][root][INFO] - ===> Epoch[3520](17600/5): Loss 0.0307, Sem 0.0307 LR: 7.316e-02 Score 98.979 Data time: 0.2976, Forward time: 1.2642, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7575 [2021-07-08 00:21:06,306][root][INFO] - ===> Start testing [2021-07-08 00:21:42,710][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1615 Loss 0.231 (AVG: 0.518) Score 94.024 (AVG: 89.742) mIOU 67.844 mAP 81.604 mAcc 75.660 IOU: 63.269 0.000 77.413 74.160 94.018 90.574 32.575 72.484 98.361 64.868 77.167 83.890 53.195 mAP: 82.367 3.926 90.521 90.505 98.991 97.831 55.878 92.364 99.826 79.671 91.259 90.601 87.116 mAcc: 79.368 0.000 86.098 79.968 95.712 94.198 47.570 86.626 99.507 72.336 91.583 94.919 55.694 [2021-07-08 00:21:42,710][root][INFO] - Finished test. Elapsed time: 36.4035 [2021-07-08 00:21:43,785][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 00:21:56,969][root][INFO] - ===> Epoch[3521](17605/5): Loss 0.0282, Sem 0.0282 LR: 7.316e-02 Score 99.029 Data time: 0.2019, Forward time: 1.2642, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.5923 [2021-07-08 00:22:10,590][root][INFO] - ===> Epoch[3522](17610/5): Loss 0.0269, Sem 0.0269 LR: 7.315e-02 Score 99.061 Data time: 0.1694, Forward time: 1.2642, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.6835 [2021-07-08 00:22:26,020][root][INFO] - ===> Epoch[3523](17615/5): Loss 0.0278, Sem 0.0278 LR: 7.314e-02 Score 99.035 Data time: 0.2827, Forward time: 1.2642, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 3.0315 [2021-07-08 00:22:40,817][root][INFO] - ===> Epoch[3524](17620/5): Loss 0.0283, Sem 0.0283 LR: 7.313e-02 Score 99.021 Data time: 0.2490, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.9153 [2021-07-08 00:22:56,154][root][INFO] - ===> Epoch[3525](17625/5): Loss 0.0252, Sem 0.0252 LR: 7.312e-02 Score 99.119 Data time: 0.2623, Forward time: 1.2642, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 3.0194 [2021-07-08 00:23:09,383][root][INFO] - ===> Epoch[3526](17630/5): Loss 0.0250, Sem 0.0250 LR: 7.312e-02 Score 99.131 Data time: 0.2589, Forward time: 1.2642, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.5952 [2021-07-08 00:23:23,180][root][INFO] - ===> Epoch[3527](17635/5): Loss 0.0239, Sem 0.0239 LR: 7.311e-02 Score 99.161 Data time: 0.2025, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7138 [2021-07-08 00:23:37,161][root][INFO] - ===> Epoch[3528](17640/5): Loss 0.0224, Sem 0.0224 LR: 7.310e-02 Score 99.204 Data time: 0.2437, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7494 [2021-07-08 00:23:51,313][root][INFO] - ===> Epoch[3529](17645/5): Loss 0.0208, Sem 0.0208 LR: 7.309e-02 Score 99.263 Data time: 0.2264, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7856 [2021-07-08 00:24:04,418][root][INFO] - ===> Epoch[3530](17650/5): Loss 0.0218, Sem 0.0218 LR: 7.309e-02 Score 99.235 Data time: 0.2291, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.5821 [2021-07-08 00:24:18,440][root][INFO] - ===> Epoch[3531](17655/5): Loss 0.0218, Sem 0.0218 LR: 7.308e-02 Score 99.242 Data time: 0.1849, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7651 [2021-07-08 00:24:31,995][root][INFO] - ===> Epoch[3532](17660/5): Loss 0.0218, Sem 0.0218 LR: 7.307e-02 Score 99.259 Data time: 0.2005, Forward time: 1.2641, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.6704 [2021-07-08 00:24:45,932][root][INFO] - ===> Epoch[3533](17665/5): Loss 0.0206, Sem 0.0206 LR: 7.306e-02 Score 99.284 Data time: 0.1751, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7465 [2021-07-08 00:25:00,156][root][INFO] - ===> Epoch[3534](17670/5): Loss 0.0209, Sem 0.0209 LR: 7.306e-02 Score 99.276 Data time: 0.2560, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8029 [2021-07-08 00:25:14,231][root][INFO] - ===> Epoch[3535](17675/5): Loss 0.0193, Sem 0.0193 LR: 7.305e-02 Score 99.320 Data time: 0.2180, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7676 [2021-07-08 00:25:28,635][root][INFO] - ===> Epoch[3536](17680/5): Loss 0.0213, Sem 0.0213 LR: 7.304e-02 Score 99.266 Data time: 0.2090, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8304 [2021-07-08 00:25:42,249][root][INFO] - ===> Epoch[3537](17685/5): Loss 0.0210, Sem 0.0210 LR: 7.303e-02 Score 99.293 Data time: 0.2386, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6702 [2021-07-08 00:25:56,378][root][INFO] - ===> Epoch[3538](17690/5): Loss 0.0186, Sem 0.0186 LR: 7.302e-02 Score 99.344 Data time: 0.1581, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7785 [2021-07-08 00:26:10,232][root][INFO] - ===> Epoch[3539](17695/5): Loss 0.0185, Sem 0.0185 LR: 7.302e-02 Score 99.343 Data time: 0.2334, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7135 [2021-07-08 00:26:24,187][root][INFO] - ===> Epoch[3540](17700/5): Loss 0.0187, Sem 0.0187 LR: 7.301e-02 Score 99.330 Data time: 0.2369, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7359 [2021-07-08 00:26:37,215][root][INFO] - ===> Epoch[3541](17705/5): Loss 0.0193, Sem 0.0193 LR: 7.300e-02 Score 99.316 Data time: 0.2037, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.5659 [2021-07-08 00:26:51,614][root][INFO] - ===> Epoch[3542](17710/5): Loss 0.0186, Sem 0.0186 LR: 7.299e-02 Score 99.334 Data time: 0.2135, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8380 [2021-07-08 00:27:05,060][root][INFO] - ===> Epoch[3543](17715/5): Loss 0.0221, Sem 0.0221 LR: 7.299e-02 Score 99.250 Data time: 0.1792, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6554 [2021-07-08 00:27:18,572][root][INFO] - ===> Epoch[3544](17720/5): Loss 0.0220, Sem 0.0220 LR: 7.298e-02 Score 99.256 Data time: 0.2010, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6565 [2021-07-08 00:27:33,637][root][INFO] - ===> Epoch[3545](17725/5): Loss 0.0228, Sem 0.0228 LR: 7.297e-02 Score 99.216 Data time: 0.2767, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.9606 [2021-07-08 00:27:48,039][root][INFO] - ===> Epoch[3546](17730/5): Loss 0.0211, Sem 0.0211 LR: 7.296e-02 Score 99.266 Data time: 0.2993, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8372 [2021-07-08 00:28:03,467][root][INFO] - ===> Epoch[3547](17735/5): Loss 0.0242, Sem 0.0242 LR: 7.295e-02 Score 99.207 Data time: 0.3203, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 3.0482 [2021-07-08 00:28:17,241][root][INFO] - ===> Epoch[3548](17740/5): Loss 0.0250, Sem 0.0250 LR: 7.295e-02 Score 99.182 Data time: 0.2375, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7002 [2021-07-08 00:28:31,351][root][INFO] - ===> Epoch[3549](17745/5): Loss 0.0256, Sem 0.0256 LR: 7.294e-02 Score 99.142 Data time: 0.2468, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7693 [2021-07-08 00:28:45,099][root][INFO] - ===> Epoch[3550](17750/5): Loss 0.0221, Sem 0.0221 LR: 7.293e-02 Score 99.223 Data time: 0.2241, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7110 [2021-07-08 00:28:59,143][root][INFO] - ===> Epoch[3551](17755/5): Loss 0.0242, Sem 0.0242 LR: 7.292e-02 Score 99.154 Data time: 0.1992, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7630 [2021-07-08 00:29:13,162][root][INFO] - ===> Epoch[3552](17760/5): Loss 0.0222, Sem 0.0222 LR: 7.292e-02 Score 99.217 Data time: 0.1786, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7615 [2021-07-08 00:29:26,977][root][INFO] - ===> Epoch[3553](17765/5): Loss 0.0225, Sem 0.0225 LR: 7.291e-02 Score 99.203 Data time: 0.2643, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7167 [2021-07-08 00:29:40,810][root][INFO] - ===> Epoch[3554](17770/5): Loss 0.0207, Sem 0.0207 LR: 7.290e-02 Score 99.266 Data time: 0.2339, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7210 [2021-07-08 00:29:54,959][root][INFO] - ===> Epoch[3555](17775/5): Loss 0.0213, Sem 0.0213 LR: 7.289e-02 Score 99.254 Data time: 0.2573, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7835 [2021-07-08 00:30:08,943][root][INFO] - ===> Epoch[3556](17780/5): Loss 0.0199, Sem 0.0199 LR: 7.288e-02 Score 99.298 Data time: 0.1979, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7497 [2021-07-08 00:30:22,441][root][INFO] - ===> Epoch[3557](17785/5): Loss 0.0195, Sem 0.0195 LR: 7.288e-02 Score 99.324 Data time: 0.2273, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6545 [2021-07-08 00:30:37,184][root][INFO] - ===> Epoch[3558](17790/5): Loss 0.0181, Sem 0.0181 LR: 7.287e-02 Score 99.354 Data time: 0.2526, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8860 [2021-07-08 00:30:50,550][root][INFO] - ===> Epoch[3559](17795/5): Loss 0.0179, Sem 0.0179 LR: 7.286e-02 Score 99.373 Data time: 0.2716, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6300 [2021-07-08 00:31:04,975][root][INFO] - ===> Epoch[3560](17800/5): Loss 0.0185, Sem 0.0185 LR: 7.285e-02 Score 99.349 Data time: 0.1632, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8498 [2021-07-08 00:31:04,976][root][INFO] - ===> Start testing [2021-07-08 00:31:39,547][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1421 Loss 0.305 (AVG: 0.503) Score 91.872 (AVG: 90.743) mIOU 69.045 mAP 84.001 mAcc 77.130 IOU: 62.765 0.000 85.129 75.753 94.505 91.317 22.637 81.245 98.353 63.410 78.650 86.435 57.390 mAP: 83.635 7.079 91.636 90.191 99.054 98.754 57.569 97.751 99.842 87.765 92.927 95.273 90.543 mAcc: 76.562 0.000 90.447 84.714 96.092 97.008 24.581 95.575 99.429 89.453 91.985 96.513 60.327 [2021-07-08 00:31:39,547][root][INFO] - Finished test. Elapsed time: 34.5694 [2021-07-08 00:31:40,589][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 00:31:54,112][root][INFO] - ===> Epoch[3561](17805/5): Loss 0.0185, Sem 0.0185 LR: 7.285e-02 Score 99.345 Data time: 0.1839, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6577 [2021-07-08 00:32:08,411][root][INFO] - ===> Epoch[3562](17810/5): Loss 0.0202, Sem 0.0202 LR: 7.284e-02 Score 99.294 Data time: 0.2244, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8049 [2021-07-08 00:32:22,418][root][INFO] - ===> Epoch[3563](17815/5): Loss 0.0235, Sem 0.0235 LR: 7.283e-02 Score 99.183 Data time: 0.2182, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7557 [2021-07-08 00:32:36,569][root][INFO] - ===> Epoch[3564](17820/5): Loss 0.0248, Sem 0.0248 LR: 7.282e-02 Score 99.157 Data time: 0.2515, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7813 [2021-07-08 00:32:50,328][root][INFO] - ===> Epoch[3565](17825/5): Loss 0.0253, Sem 0.0253 LR: 7.281e-02 Score 99.108 Data time: 0.2620, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7090 [2021-07-08 00:33:03,573][root][INFO] - ===> Epoch[3566](17830/5): Loss 0.0286, Sem 0.0286 LR: 7.281e-02 Score 99.016 Data time: 0.2128, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6030 [2021-07-08 00:33:17,696][root][INFO] - ===> Epoch[3567](17835/5): Loss 0.0297, Sem 0.0297 LR: 7.280e-02 Score 99.022 Data time: 0.1900, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7744 [2021-07-08 00:33:31,294][root][INFO] - ===> Epoch[3568](17840/5): Loss 0.0295, Sem 0.0295 LR: 7.279e-02 Score 99.012 Data time: 0.2331, Forward time: 1.2641, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.6681 [2021-07-08 00:33:44,965][root][INFO] - ===> Epoch[3569](17845/5): Loss 0.0283, Sem 0.0283 LR: 7.278e-02 Score 99.042 Data time: 0.2538, Forward time: 1.2641, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.6911 [2021-07-08 00:33:59,442][root][INFO] - ===> Epoch[3570](17850/5): Loss 0.0278, Sem 0.0278 LR: 7.278e-02 Score 99.044 Data time: 0.2418, Forward time: 1.2641, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8448 [2021-07-08 00:34:13,285][root][INFO] - ===> Epoch[3571](17855/5): Loss 0.0239, Sem 0.0239 LR: 7.277e-02 Score 99.167 Data time: 0.2620, Forward time: 1.2641, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7231 [2021-07-08 00:34:27,673][root][INFO] - ===> Epoch[3572](17860/5): Loss 0.0239, Sem 0.0239 LR: 7.276e-02 Score 99.166 Data time: 0.1669, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8325 [2021-07-08 00:34:42,974][root][INFO] - ===> Epoch[3573](17865/5): Loss 0.0254, Sem 0.0254 LR: 7.275e-02 Score 99.125 Data time: 0.2269, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 3.0132 [2021-07-08 00:34:56,248][root][INFO] - ===> Epoch[3574](17870/5): Loss 0.0275, Sem 0.0275 LR: 7.274e-02 Score 99.097 Data time: 0.2698, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.5984 [2021-07-08 00:35:10,632][root][INFO] - ===> Epoch[3575](17875/5): Loss 0.0320, Sem 0.0320 LR: 7.274e-02 Score 99.002 Data time: 0.2244, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8248 [2021-07-08 00:35:24,488][root][INFO] - ===> Epoch[3576](17880/5): Loss 0.0287, Sem 0.0287 LR: 7.273e-02 Score 99.034 Data time: 0.2538, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7145 [2021-07-08 00:35:38,012][root][INFO] - ===> Epoch[3577](17885/5): Loss 0.0289, Sem 0.0289 LR: 7.272e-02 Score 99.009 Data time: 0.2142, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6465 [2021-07-08 00:35:52,734][root][INFO] - ===> Epoch[3578](17890/5): Loss 0.0256, Sem 0.0256 LR: 7.271e-02 Score 99.114 Data time: 0.2401, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8882 [2021-07-08 00:36:06,526][root][INFO] - ===> Epoch[3579](17895/5): Loss 0.0246, Sem 0.0246 LR: 7.271e-02 Score 99.130 Data time: 0.2207, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7182 [2021-07-08 00:36:20,464][root][INFO] - ===> Epoch[3580](17900/5): Loss 0.0243, Sem 0.0243 LR: 7.270e-02 Score 99.166 Data time: 0.2300, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7426 [2021-07-08 00:36:34,467][root][INFO] - ===> Epoch[3581](17905/5): Loss 0.0232, Sem 0.0232 LR: 7.269e-02 Score 99.183 Data time: 0.2376, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7564 [2021-07-08 00:36:48,373][root][INFO] - ===> Epoch[3582](17910/5): Loss 0.0215, Sem 0.0215 LR: 7.268e-02 Score 99.243 Data time: 0.2169, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7324 [2021-07-08 00:37:02,628][root][INFO] - ===> Epoch[3583](17915/5): Loss 0.0209, Sem 0.0209 LR: 7.267e-02 Score 99.261 Data time: 0.2157, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7990 [2021-07-08 00:37:15,879][root][INFO] - ===> Epoch[3584](17920/5): Loss 0.0214, Sem 0.0214 LR: 7.267e-02 Score 99.239 Data time: 0.2232, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6098 [2021-07-08 00:37:30,372][root][INFO] - ===> Epoch[3585](17925/5): Loss 0.0190, Sem 0.0190 LR: 7.266e-02 Score 99.325 Data time: 0.2148, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8417 [2021-07-08 00:37:43,367][root][INFO] - ===> Epoch[3586](17930/5): Loss 0.0186, Sem 0.0186 LR: 7.265e-02 Score 99.330 Data time: 0.1922, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.5476 [2021-07-08 00:37:58,383][root][INFO] - ===> Epoch[3587](17935/5): Loss 0.0181, Sem 0.0181 LR: 7.264e-02 Score 99.348 Data time: 0.2441, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.9583 [2021-07-08 00:38:12,842][root][INFO] - ===> Epoch[3588](17940/5): Loss 0.0176, Sem 0.0176 LR: 7.264e-02 Score 99.379 Data time: 0.2266, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8500 [2021-07-08 00:38:27,729][root][INFO] - ===> Epoch[3589](17945/5): Loss 0.0178, Sem 0.0178 LR: 7.263e-02 Score 99.360 Data time: 0.2529, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.9358 [2021-07-08 00:38:41,831][root][INFO] - ===> Epoch[3590](17950/5): Loss 0.0174, Sem 0.0174 LR: 7.262e-02 Score 99.384 Data time: 0.2642, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7717 [2021-07-08 00:38:56,165][root][INFO] - ===> Epoch[3591](17955/5): Loss 0.0179, Sem 0.0179 LR: 7.261e-02 Score 99.363 Data time: 0.1911, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8255 [2021-07-08 00:39:10,868][root][INFO] - ===> Epoch[3592](17960/5): Loss 0.0184, Sem 0.0184 LR: 7.260e-02 Score 99.357 Data time: 0.2164, Forward time: 1.2641, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8953 [2021-07-08 00:39:24,537][root][INFO] - ===> Epoch[3593](17965/5): Loss 0.0192, Sem 0.0192 LR: 7.260e-02 Score 99.321 Data time: 0.2301, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6941 [2021-07-08 00:39:38,456][root][INFO] - ===> Epoch[3594](17970/5): Loss 0.0194, Sem 0.0194 LR: 7.259e-02 Score 99.315 Data time: 0.1516, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7417 [2021-07-08 00:39:52,352][root][INFO] - ===> Epoch[3595](17975/5): Loss 0.0203, Sem 0.0203 LR: 7.258e-02 Score 99.310 Data time: 0.2864, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7302 [2021-07-08 00:40:06,531][root][INFO] - ===> Epoch[3596](17980/5): Loss 0.0201, Sem 0.0201 LR: 7.257e-02 Score 99.294 Data time: 0.2222, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7922 [2021-07-08 00:40:20,950][root][INFO] - ===> Epoch[3597](17985/5): Loss 0.0197, Sem 0.0197 LR: 7.257e-02 Score 99.322 Data time: 0.2086, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8362 [2021-07-08 00:40:35,004][root][INFO] - ===> Epoch[3598](17990/5): Loss 0.0187, Sem 0.0187 LR: 7.256e-02 Score 99.338 Data time: 0.2337, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7632 [2021-07-08 00:40:48,346][root][INFO] - ===> Epoch[3599](17995/5): Loss 0.0180, Sem 0.0180 LR: 7.255e-02 Score 99.363 Data time: 0.2318, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6276 [2021-07-08 00:41:03,423][root][INFO] - ===> Epoch[3600](18000/5): Loss 0.0178, Sem 0.0178 LR: 7.254e-02 Score 99.365 Data time: 0.2720, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.9528 [2021-07-08 00:41:03,426][root][INFO] - ===> Start testing [2021-07-08 00:41:38,972][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1321 Loss 0.195 (AVG: 0.507) Score 95.162 (AVG: 91.020) mIOU 70.676 mAP 84.788 mAcc 76.587 IOU: 65.099 0.000 81.221 75.636 94.653 92.330 29.102 81.190 98.519 77.150 80.072 86.461 57.353 mAP: 85.326 3.208 93.224 89.432 98.984 98.978 63.334 97.153 99.832 93.834 92.630 94.720 91.588 mAcc: 82.710 0.000 83.992 81.835 96.050 97.388 31.554 93.669 99.528 81.876 89.965 96.879 60.179 [2021-07-08 00:41:38,973][root][INFO] - Finished test. Elapsed time: 35.5454 [2021-07-08 00:41:40,043][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 00:41:53,207][root][INFO] - ===> Epoch[3601](18005/5): Loss 0.0170, Sem 0.0170 LR: 7.253e-02 Score 99.386 Data time: 0.1844, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.5950 [2021-07-08 00:42:06,865][root][INFO] - ===> Epoch[3602](18010/5): Loss 0.0163, Sem 0.0163 LR: 7.253e-02 Score 99.413 Data time: 0.2123, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6857 [2021-07-08 00:42:20,194][root][INFO] - ===> Epoch[3603](18015/5): Loss 0.0176, Sem 0.0176 LR: 7.252e-02 Score 99.364 Data time: 0.2329, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6206 [2021-07-08 00:42:34,016][root][INFO] - ===> Epoch[3604](18020/5): Loss 0.0178, Sem 0.0178 LR: 7.251e-02 Score 99.393 Data time: 0.1768, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7202 [2021-07-08 00:42:48,209][root][INFO] - ===> Epoch[3605](18025/5): Loss 0.0170, Sem 0.0170 LR: 7.250e-02 Score 99.387 Data time: 0.2319, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7964 [2021-07-08 00:43:01,933][root][INFO] - ===> Epoch[3606](18030/5): Loss 0.0160, Sem 0.0160 LR: 7.250e-02 Score 99.424 Data time: 0.2372, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6970 [2021-07-08 00:43:15,759][root][INFO] - ===> Epoch[3607](18035/5): Loss 0.0165, Sem 0.0165 LR: 7.249e-02 Score 99.410 Data time: 0.1972, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7141 [2021-07-08 00:43:30,230][root][INFO] - ===> Epoch[3608](18040/5): Loss 0.0161, Sem 0.0161 LR: 7.248e-02 Score 99.415 Data time: 0.2761, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8494 [2021-07-08 00:43:44,641][root][INFO] - ===> Epoch[3609](18045/5): Loss 0.0159, Sem 0.0159 LR: 7.247e-02 Score 99.428 Data time: 0.3197, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8290 [2021-07-08 00:43:59,090][root][INFO] - ===> Epoch[3610](18050/5): Loss 0.0168, Sem 0.0168 LR: 7.246e-02 Score 99.412 Data time: 0.2635, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8436 [2021-07-08 00:44:13,085][root][INFO] - ===> Epoch[3611](18055/5): Loss 0.0171, Sem 0.0171 LR: 7.246e-02 Score 99.387 Data time: 0.2089, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7619 [2021-07-08 00:44:26,775][root][INFO] - ===> Epoch[3612](18060/5): Loss 0.0169, Sem 0.0169 LR: 7.245e-02 Score 99.398 Data time: 0.2627, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6921 [2021-07-08 00:44:40,195][root][INFO] - ===> Epoch[3613](18065/5): Loss 0.0168, Sem 0.0168 LR: 7.244e-02 Score 99.404 Data time: 0.2214, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6328 [2021-07-08 00:44:54,178][root][INFO] - ===> Epoch[3614](18070/5): Loss 0.0163, Sem 0.0163 LR: 7.243e-02 Score 99.416 Data time: 0.2122, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7483 [2021-07-08 00:45:08,037][root][INFO] - ===> Epoch[3615](18075/5): Loss 0.0169, Sem 0.0169 LR: 7.243e-02 Score 99.391 Data time: 0.2170, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7108 [2021-07-08 00:45:22,523][root][INFO] - ===> Epoch[3616](18080/5): Loss 0.0163, Sem 0.0163 LR: 7.242e-02 Score 99.418 Data time: 0.2464, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8520 [2021-07-08 00:45:36,614][root][INFO] - ===> Epoch[3617](18085/5): Loss 0.0157, Sem 0.0157 LR: 7.241e-02 Score 99.436 Data time: 0.2453, Forward time: 1.2639, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7749 [2021-07-08 00:45:50,691][root][INFO] - ===> Epoch[3618](18090/5): Loss 0.0157, Sem 0.0157 LR: 7.240e-02 Score 99.428 Data time: 0.2447, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7615 [2021-07-08 00:46:04,272][root][INFO] - ===> Epoch[3619](18095/5): Loss 0.0151, Sem 0.0151 LR: 7.239e-02 Score 99.458 Data time: 0.2361, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6602 [2021-07-08 00:46:18,917][root][INFO] - ===> Epoch[3620](18100/5): Loss 0.0152, Sem 0.0152 LR: 7.239e-02 Score 99.451 Data time: 0.1981, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8764 [2021-07-08 00:46:32,493][root][INFO] - ===> Epoch[3621](18105/5): Loss 0.0163, Sem 0.0163 LR: 7.238e-02 Score 99.429 Data time: 0.1907, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6666 [2021-07-08 00:46:45,988][root][INFO] - ===> Epoch[3622](18110/5): Loss 0.0155, Sem 0.0155 LR: 7.237e-02 Score 99.446 Data time: 0.2242, Forward time: 1.2639, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6504 [2021-07-08 00:46:59,899][root][INFO] - ===> Epoch[3623](18115/5): Loss 0.0159, Sem 0.0159 LR: 7.236e-02 Score 99.436 Data time: 0.2572, Forward time: 1.2639, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7310 [2021-07-08 00:47:14,096][root][INFO] - ===> Epoch[3624](18120/5): Loss 0.0153, Sem 0.0153 LR: 7.236e-02 Score 99.448 Data time: 0.2040, Forward time: 1.2639, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7911 [2021-07-08 00:47:28,194][root][INFO] - ===> Epoch[3625](18125/5): Loss 0.0156, Sem 0.0156 LR: 7.235e-02 Score 99.440 Data time: 0.1799, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7744 [2021-07-08 00:47:41,788][root][INFO] - ===> Epoch[3626](18130/5): Loss 0.0142, Sem 0.0142 LR: 7.234e-02 Score 99.490 Data time: 0.2329, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6638 [2021-07-08 00:47:56,326][root][INFO] - ===> Epoch[3627](18135/5): Loss 0.0145, Sem 0.0145 LR: 7.233e-02 Score 99.472 Data time: 0.1709, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8519 [2021-07-08 00:48:10,582][root][INFO] - ===> Epoch[3628](18140/5): Loss 0.0145, Sem 0.0145 LR: 7.232e-02 Score 99.483 Data time: 0.2136, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7999 [2021-07-08 00:48:24,378][root][INFO] - ===> Epoch[3629](18145/5): Loss 0.0140, Sem 0.0140 LR: 7.232e-02 Score 99.493 Data time: 0.2497, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6997 [2021-07-08 00:48:37,781][root][INFO] - ===> Epoch[3630](18150/5): Loss 0.0141, Sem 0.0141 LR: 7.231e-02 Score 99.497 Data time: 0.2478, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6388 [2021-07-08 00:48:52,002][root][INFO] - ===> Epoch[3631](18155/5): Loss 0.0150, Sem 0.0150 LR: 7.230e-02 Score 99.454 Data time: 0.2923, Forward time: 1.2639, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7853 [2021-07-08 00:49:06,594][root][INFO] - ===> Epoch[3632](18160/5): Loss 0.0143, Sem 0.0143 LR: 7.229e-02 Score 99.484 Data time: 0.2444, Forward time: 1.2639, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8763 [2021-07-08 00:49:20,837][root][INFO] - ===> Epoch[3633](18165/5): Loss 0.0144, Sem 0.0144 LR: 7.229e-02 Score 99.479 Data time: 0.2202, Forward time: 1.2639, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8042 [2021-07-08 00:49:35,422][root][INFO] - ===> Epoch[3634](18170/5): Loss 0.0143, Sem 0.0143 LR: 7.228e-02 Score 99.478 Data time: 0.2595, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8659 [2021-07-08 00:49:49,967][root][INFO] - ===> Epoch[3635](18175/5): Loss 0.0138, Sem 0.0138 LR: 7.227e-02 Score 99.499 Data time: 0.2578, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8490 [2021-07-08 00:50:04,580][root][INFO] - ===> Epoch[3636](18180/5): Loss 0.0140, Sem 0.0140 LR: 7.226e-02 Score 99.491 Data time: 0.1673, Forward time: 1.2640, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8925 [2021-07-08 00:50:18,703][root][INFO] - ===> Epoch[3637](18185/5): Loss 0.0142, Sem 0.0142 LR: 7.225e-02 Score 99.488 Data time: 0.2972, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7848 [2021-07-08 00:50:32,110][root][INFO] - ===> Epoch[3638](18190/5): Loss 0.0169, Sem 0.0169 LR: 7.225e-02 Score 99.432 Data time: 0.2097, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6386 [2021-07-08 00:50:45,191][root][INFO] - ===> Epoch[3639](18195/5): Loss 0.0166, Sem 0.0166 LR: 7.224e-02 Score 99.414 Data time: 0.2644, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.5698 [2021-07-08 00:50:59,584][root][INFO] - ===> Epoch[3640](18200/5): Loss 0.0178, Sem 0.0178 LR: 7.223e-02 Score 99.407 Data time: 0.1276, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8461 [2021-07-08 00:50:59,585][root][INFO] - ===> Start testing [2021-07-08 00:51:35,790][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1437 Loss 0.328 (AVG: 0.623) Score 92.238 (AVG: 89.516) mIOU 66.790 mAP 81.694 mAcc 74.012 IOU: 61.957 0.000 81.908 67.003 94.391 90.236 23.820 78.837 97.724 57.972 70.017 85.768 58.631 mAP: 82.656 1.824 94.455 85.564 98.885 98.409 54.644 95.151 99.633 75.365 89.839 94.243 91.355 mAcc: 84.059 0.000 90.190 70.041 95.343 96.903 25.548 91.643 99.110 58.652 92.658 96.688 61.317 [2021-07-08 00:51:35,791][root][INFO] - Finished test. Elapsed time: 36.2048 [2021-07-08 00:51:36,920][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 00:51:50,390][root][INFO] - ===> Epoch[3641](18205/5): Loss 0.0192, Sem 0.0192 LR: 7.222e-02 Score 99.337 Data time: 0.1906, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6392 [2021-07-08 00:52:03,890][root][INFO] - ===> Epoch[3642](18210/5): Loss 0.0177, Sem 0.0177 LR: 7.222e-02 Score 99.369 Data time: 0.1934, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6612 [2021-07-08 00:52:18,179][root][INFO] - ===> Epoch[3643](18215/5): Loss 0.0173, Sem 0.0173 LR: 7.221e-02 Score 99.374 Data time: 0.2144, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8153 [2021-07-08 00:52:31,616][root][INFO] - ===> Epoch[3644](18220/5): Loss 0.0185, Sem 0.0185 LR: 7.220e-02 Score 99.377 Data time: 0.1695, Forward time: 1.2638, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6571 [2021-07-08 00:52:46,450][root][INFO] - ===> Epoch[3645](18225/5): Loss 0.0169, Sem 0.0169 LR: 7.219e-02 Score 99.396 Data time: 0.1956, Forward time: 1.2639, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.9250 [2021-07-08 00:53:00,423][root][INFO] - ===> Epoch[3646](18230/5): Loss 0.0163, Sem 0.0163 LR: 7.218e-02 Score 99.410 Data time: 0.2960, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7448 [2021-07-08 00:53:13,606][root][INFO] - ===> Epoch[3647](18235/5): Loss 0.0161, Sem 0.0161 LR: 7.218e-02 Score 99.423 Data time: 0.2242, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.5937 [2021-07-08 00:53:27,406][root][INFO] - ===> Epoch[3648](18240/5): Loss 0.0156, Sem 0.0156 LR: 7.217e-02 Score 99.437 Data time: 0.2533, Forward time: 1.2639, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7137 [2021-07-08 00:53:40,809][root][INFO] - ===> Epoch[3649](18245/5): Loss 0.0161, Sem 0.0161 LR: 7.216e-02 Score 99.433 Data time: 0.2440, Forward time: 1.2638, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6408 [2021-07-08 00:53:54,776][root][INFO] - ===> Epoch[3650](18250/5): Loss 0.0145, Sem 0.0145 LR: 7.215e-02 Score 99.481 Data time: 0.2988, Forward time: 1.2639, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7499 [2021-07-08 00:54:08,939][root][INFO] - ===> Epoch[3651](18255/5): Loss 0.0151, Sem 0.0151 LR: 7.215e-02 Score 99.460 Data time: 0.2324, Forward time: 1.2639, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7835 [2021-07-08 00:54:23,191][root][INFO] - ===> Epoch[3652](18260/5): Loss 0.0149, Sem 0.0149 LR: 7.214e-02 Score 99.468 Data time: 0.1889, Forward time: 1.2639, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8127 [2021-07-08 00:54:36,877][root][INFO] - ===> Epoch[3653](18265/5): Loss 0.0144, Sem 0.0144 LR: 7.213e-02 Score 99.482 Data time: 0.2594, Forward time: 1.2639, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6873 [2021-07-08 00:54:51,072][root][INFO] - ===> Epoch[3654](18270/5): Loss 0.0143, Sem 0.0143 LR: 7.212e-02 Score 99.482 Data time: 0.2167, Forward time: 1.2639, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7866 [2021-07-08 00:55:05,600][root][INFO] - ===> Epoch[3655](18275/5): Loss 0.0138, Sem 0.0138 LR: 7.211e-02 Score 99.501 Data time: 0.1784, Forward time: 1.2639, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8542 [2021-07-08 00:55:20,226][root][INFO] - ===> Epoch[3656](18280/5): Loss 0.0145, Sem 0.0145 LR: 7.211e-02 Score 99.477 Data time: 0.2183, Forward time: 1.2639, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8806 [2021-07-08 00:55:34,408][root][INFO] - ===> Epoch[3657](18285/5): Loss 0.0134, Sem 0.0134 LR: 7.210e-02 Score 99.514 Data time: 0.2318, Forward time: 1.2640, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7893 [2021-07-08 00:55:48,005][root][INFO] - ===> Epoch[3658](18290/5): Loss 0.0151, Sem 0.0151 LR: 7.209e-02 Score 99.465 Data time: 0.2680, Forward time: 1.2640, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.6753 [2021-07-08 00:56:02,363][root][INFO] - ===> Epoch[3659](18295/5): Loss 0.0137, Sem 0.0137 LR: 7.208e-02 Score 99.508 Data time: 0.2969, Forward time: 1.2640, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8163 [2021-07-08 00:56:16,724][root][INFO] - ===> Epoch[3660](18300/5): Loss 0.0146, Sem 0.0146 LR: 7.208e-02 Score 99.473 Data time: 0.2245, Forward time: 1.2640, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8313 [2021-07-08 00:56:31,065][root][INFO] - ===> Epoch[3661](18305/5): Loss 0.0135, Sem 0.0135 LR: 7.207e-02 Score 99.513 Data time: 0.2056, Forward time: 1.2641, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8260 [2021-07-08 00:56:45,231][root][INFO] - ===> Epoch[3662](18310/5): Loss 0.0142, Sem 0.0142 LR: 7.206e-02 Score 99.483 Data time: 0.2293, Forward time: 1.2641, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7828 [2021-07-08 00:56:59,805][root][INFO] - ===> Epoch[3663](18315/5): Loss 0.0133, Sem 0.0133 LR: 7.205e-02 Score 99.522 Data time: 0.2045, Forward time: 1.2641, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8545 [2021-07-08 00:57:14,349][root][INFO] - ===> Epoch[3664](18320/5): Loss 0.0134, Sem 0.0134 LR: 7.204e-02 Score 99.514 Data time: 0.2313, Forward time: 1.2641, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8608 [2021-07-08 00:57:28,204][root][INFO] - ===> Epoch[3665](18325/5): Loss 0.0142, Sem 0.0142 LR: 7.204e-02 Score 99.485 Data time: 0.2898, Forward time: 1.2641, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7052 [2021-07-08 00:57:42,159][root][INFO] - ===> Epoch[3666](18330/5): Loss 0.0135, Sem 0.0135 LR: 7.203e-02 Score 99.512 Data time: 0.2382, Forward time: 1.2641, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7472 [2021-07-08 00:57:56,603][root][INFO] - ===> Epoch[3667](18335/5): Loss 0.0135, Sem 0.0135 LR: 7.202e-02 Score 99.512 Data time: 0.2978, Forward time: 1.2642, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.8394 [2021-07-08 00:58:10,110][root][INFO] - ===> Epoch[3668](18340/5): Loss 0.0137, Sem 0.0137 LR: 7.201e-02 Score 99.505 Data time: 0.1331, Forward time: 1.2641, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.6606 [2021-07-08 00:58:24,127][root][INFO] - ===> Epoch[3669](18345/5): Loss 0.0138, Sem 0.0138 LR: 7.201e-02 Score 99.499 Data time: 0.2667, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7605 [2021-07-08 00:58:38,895][root][INFO] - ===> Epoch[3670](18350/5): Loss 0.0136, Sem 0.0136 LR: 7.200e-02 Score 99.513 Data time: 0.3364, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8931 [2021-07-08 00:58:52,493][root][INFO] - ===> Epoch[3671](18355/5): Loss 0.0135, Sem 0.0135 LR: 7.199e-02 Score 99.510 Data time: 0.2618, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6729 [2021-07-08 00:59:06,667][root][INFO] - ===> Epoch[3672](18360/5): Loss 0.0143, Sem 0.0143 LR: 7.198e-02 Score 99.485 Data time: 0.2330, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7983 [2021-07-08 00:59:20,456][root][INFO] - ===> Epoch[3673](18365/5): Loss 0.0135, Sem 0.0135 LR: 7.197e-02 Score 99.512 Data time: 0.2393, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7079 [2021-07-08 00:59:35,035][root][INFO] - ===> Epoch[3674](18370/5): Loss 0.0141, Sem 0.0141 LR: 7.197e-02 Score 99.497 Data time: 0.2922, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8673 [2021-07-08 00:59:49,504][root][INFO] - ===> Epoch[3675](18375/5): Loss 0.0130, Sem 0.0130 LR: 7.196e-02 Score 99.529 Data time: 0.2335, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8419 [2021-07-08 01:00:03,918][root][INFO] - ===> Epoch[3676](18380/5): Loss 0.0140, Sem 0.0140 LR: 7.195e-02 Score 99.503 Data time: 0.2985, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8273 [2021-07-08 01:00:18,114][root][INFO] - ===> Epoch[3677](18385/5): Loss 0.0134, Sem 0.0134 LR: 7.194e-02 Score 99.517 Data time: 0.2755, Forward time: 1.2643, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7949 [2021-07-08 01:00:31,300][root][INFO] - ===> Epoch[3678](18390/5): Loss 0.0133, Sem 0.0133 LR: 7.194e-02 Score 99.518 Data time: 0.2305, Forward time: 1.2643, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.5908 [2021-07-08 01:00:45,240][root][INFO] - ===> Epoch[3679](18395/5): Loss 0.0131, Sem 0.0131 LR: 7.193e-02 Score 99.528 Data time: 0.2194, Forward time: 1.2643, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7462 [2021-07-08 01:00:58,791][root][INFO] - ===> Epoch[3680](18400/5): Loss 0.0124, Sem 0.0124 LR: 7.192e-02 Score 99.550 Data time: 0.3143, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6459 [2021-07-08 01:00:58,793][root][INFO] - ===> Start testing [2021-07-08 01:01:34,962][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1227 Loss 0.276 (AVG: 0.544) Score 93.796 (AVG: 91.067) mIOU 70.269 mAP 83.808 mAcc 76.028 IOU: 64.943 0.000 82.297 76.352 95.067 92.466 25.251 81.021 97.951 71.709 81.125 86.674 58.636 mAP: 84.458 7.450 94.632 90.045 98.890 98.917 57.048 96.993 99.756 81.705 93.249 94.827 91.534 mAcc: 81.000 0.000 86.049 83.629 96.081 96.914 26.833 95.037 99.533 73.520 90.966 96.690 62.110 [2021-07-08 01:01:34,962][root][INFO] - Finished test. Elapsed time: 36.1677 [2021-07-08 01:01:36,108][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 01:01:49,390][root][INFO] - ===> Epoch[3681](18405/5): Loss 0.0134, Sem 0.0134 LR: 7.191e-02 Score 99.511 Data time: 0.2085, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6253 [2021-07-08 01:02:04,178][root][INFO] - ===> Epoch[3682](18410/5): Loss 0.0124, Sem 0.0124 LR: 7.190e-02 Score 99.551 Data time: 0.2198, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9158 [2021-07-08 01:02:18,241][root][INFO] - ===> Epoch[3683](18415/5): Loss 0.0129, Sem 0.0129 LR: 7.190e-02 Score 99.538 Data time: 0.2762, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7718 [2021-07-08 01:02:31,265][root][INFO] - ===> Epoch[3684](18420/5): Loss 0.0130, Sem 0.0130 LR: 7.189e-02 Score 99.529 Data time: 0.1642, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.5597 [2021-07-08 01:02:44,932][root][INFO] - ===> Epoch[3685](18425/5): Loss 0.0135, Sem 0.0135 LR: 7.188e-02 Score 99.521 Data time: 0.2284, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6779 [2021-07-08 01:02:58,990][root][INFO] - ===> Epoch[3686](18430/5): Loss 0.0129, Sem 0.0129 LR: 7.187e-02 Score 99.533 Data time: 0.2784, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7665 [2021-07-08 01:03:12,986][root][INFO] - ===> Epoch[3687](18435/5): Loss 0.0130, Sem 0.0130 LR: 7.187e-02 Score 99.532 Data time: 0.2107, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7603 [2021-07-08 01:03:27,025][root][INFO] - ===> Epoch[3688](18440/5): Loss 0.0132, Sem 0.0132 LR: 7.186e-02 Score 99.517 Data time: 0.2775, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7585 [2021-07-08 01:03:40,584][root][INFO] - ===> Epoch[3689](18445/5): Loss 0.0131, Sem 0.0131 LR: 7.185e-02 Score 99.527 Data time: 0.2004, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6625 [2021-07-08 01:03:54,933][root][INFO] - ===> Epoch[3690](18450/5): Loss 0.0133, Sem 0.0133 LR: 7.184e-02 Score 99.515 Data time: 0.2379, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8167 [2021-07-08 01:04:09,159][root][INFO] - ===> Epoch[3691](18455/5): Loss 0.0129, Sem 0.0129 LR: 7.183e-02 Score 99.532 Data time: 0.2125, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8037 [2021-07-08 01:04:22,900][root][INFO] - ===> Epoch[3692](18460/5): Loss 0.0131, Sem 0.0131 LR: 7.183e-02 Score 99.527 Data time: 0.2018, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6981 [2021-07-08 01:04:37,584][root][INFO] - ===> Epoch[3693](18465/5): Loss 0.0133, Sem 0.0133 LR: 7.182e-02 Score 99.520 Data time: 0.2461, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8877 [2021-07-08 01:04:52,079][root][INFO] - ===> Epoch[3694](18470/5): Loss 0.0127, Sem 0.0127 LR: 7.181e-02 Score 99.534 Data time: 0.2447, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8448 [2021-07-08 01:05:06,567][root][INFO] - ===> Epoch[3695](18475/5): Loss 0.0128, Sem 0.0128 LR: 7.180e-02 Score 99.536 Data time: 0.3121, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8446 [2021-07-08 01:05:21,099][root][INFO] - ===> Epoch[3696](18480/5): Loss 0.0126, Sem 0.0126 LR: 7.180e-02 Score 99.544 Data time: 0.3121, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8534 [2021-07-08 01:05:36,168][root][INFO] - ===> Epoch[3697](18485/5): Loss 0.0125, Sem 0.0125 LR: 7.179e-02 Score 99.546 Data time: 0.3120, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9652 [2021-07-08 01:05:50,024][root][INFO] - ===> Epoch[3698](18490/5): Loss 0.0135, Sem 0.0135 LR: 7.178e-02 Score 99.510 Data time: 0.2160, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7219 [2021-07-08 01:06:03,409][root][INFO] - ===> Epoch[3699](18495/5): Loss 0.0134, Sem 0.0134 LR: 7.177e-02 Score 99.516 Data time: 0.2587, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6303 [2021-07-08 01:06:17,471][root][INFO] - ===> Epoch[3700](18500/5): Loss 0.0127, Sem 0.0127 LR: 7.176e-02 Score 99.539 Data time: 0.2613, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7637 [2021-07-08 01:06:31,747][root][INFO] - ===> Epoch[3701](18505/5): Loss 0.0132, Sem 0.0132 LR: 7.176e-02 Score 99.526 Data time: 0.2138, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8046 [2021-07-08 01:06:46,093][root][INFO] - ===> Epoch[3702](18510/5): Loss 0.0129, Sem 0.0129 LR: 7.175e-02 Score 99.538 Data time: 0.2392, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8154 [2021-07-08 01:06:59,720][root][INFO] - ===> Epoch[3703](18515/5): Loss 0.0138, Sem 0.0138 LR: 7.174e-02 Score 99.500 Data time: 0.1761, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6833 [2021-07-08 01:07:13,417][root][INFO] - ===> Epoch[3704](18520/5): Loss 0.0129, Sem 0.0129 LR: 7.173e-02 Score 99.540 Data time: 0.2685, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6924 [2021-07-08 01:07:26,880][root][INFO] - ===> Epoch[3705](18525/5): Loss 0.0135, Sem 0.0135 LR: 7.173e-02 Score 99.511 Data time: 0.2829, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6463 [2021-07-08 01:07:41,267][root][INFO] - ===> Epoch[3706](18530/5): Loss 0.0136, Sem 0.0136 LR: 7.172e-02 Score 99.510 Data time: 0.1884, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8361 [2021-07-08 01:07:55,139][root][INFO] - ===> Epoch[3707](18535/5): Loss 0.0121, Sem 0.0121 LR: 7.171e-02 Score 99.562 Data time: 0.2478, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7302 [2021-07-08 01:08:09,232][root][INFO] - ===> Epoch[3708](18540/5): Loss 0.0127, Sem 0.0127 LR: 7.170e-02 Score 99.538 Data time: 0.2257, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7734 [2021-07-08 01:08:22,982][root][INFO] - ===> Epoch[3709](18545/5): Loss 0.0134, Sem 0.0134 LR: 7.169e-02 Score 99.515 Data time: 0.2126, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6976 [2021-07-08 01:08:37,188][root][INFO] - ===> Epoch[3710](18550/5): Loss 0.0126, Sem 0.0126 LR: 7.169e-02 Score 99.545 Data time: 0.2094, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7989 [2021-07-08 01:08:51,189][root][INFO] - ===> Epoch[3711](18555/5): Loss 0.0126, Sem 0.0126 LR: 7.168e-02 Score 99.546 Data time: 0.2795, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7528 [2021-07-08 01:09:05,337][root][INFO] - ===> Epoch[3712](18560/5): Loss 0.0123, Sem 0.0123 LR: 7.167e-02 Score 99.558 Data time: 0.1870, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7741 [2021-07-08 01:09:18,877][root][INFO] - ===> Epoch[3713](18565/5): Loss 0.0133, Sem 0.0133 LR: 7.166e-02 Score 99.518 Data time: 0.2210, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6589 [2021-07-08 01:09:32,638][root][INFO] - ===> Epoch[3714](18570/5): Loss 0.0126, Sem 0.0126 LR: 7.166e-02 Score 99.544 Data time: 0.1453, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7031 [2021-07-08 01:09:47,565][root][INFO] - ===> Epoch[3715](18575/5): Loss 0.0130, Sem 0.0130 LR: 7.165e-02 Score 99.535 Data time: 0.2490, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9371 [2021-07-08 01:10:01,440][root][INFO] - ===> Epoch[3716](18580/5): Loss 0.0126, Sem 0.0126 LR: 7.164e-02 Score 99.544 Data time: 0.2233, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7291 [2021-07-08 01:10:14,788][root][INFO] - ===> Epoch[3717](18585/5): Loss 0.0124, Sem 0.0124 LR: 7.163e-02 Score 99.552 Data time: 0.2151, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6324 [2021-07-08 01:10:29,248][root][INFO] - ===> Epoch[3718](18590/5): Loss 0.0126, Sem 0.0126 LR: 7.162e-02 Score 99.543 Data time: 0.2473, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8475 [2021-07-08 01:10:43,807][root][INFO] - ===> Epoch[3719](18595/5): Loss 0.0123, Sem 0.0123 LR: 7.162e-02 Score 99.558 Data time: 0.2536, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8625 [2021-07-08 01:10:58,302][root][INFO] - ===> Epoch[3720](18600/5): Loss 0.0125, Sem 0.0125 LR: 7.161e-02 Score 99.543 Data time: 0.2801, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8480 [2021-07-08 01:10:58,303][root][INFO] - ===> Start testing [2021-07-08 01:11:32,274][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1286 Loss 0.252 (AVG: 0.535) Score 94.098 (AVG: 91.279) mIOU 71.444 mAP 84.119 mAcc 77.144 IOU: 66.052 0.000 85.516 76.826 95.351 92.610 28.361 82.915 98.411 75.762 81.026 86.412 59.526 mAP: 84.593 6.166 93.798 89.625 98.906 98.872 56.687 97.839 99.795 87.441 92.727 94.523 92.575 mAcc: 82.503 0.000 89.699 83.384 96.442 97.095 31.920 95.311 99.470 78.453 89.624 96.371 62.598 [2021-07-08 01:11:32,275][root][INFO] - Finished test. Elapsed time: 33.9704 [2021-07-08 01:11:33,372][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 01:11:46,347][root][INFO] - ===> Epoch[3721](18605/5): Loss 0.0125, Sem 0.0125 LR: 7.160e-02 Score 99.546 Data time: 0.1815, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.5539 [2021-07-08 01:12:00,880][root][INFO] - ===> Epoch[3722](18610/5): Loss 0.0126, Sem 0.0126 LR: 7.159e-02 Score 99.542 Data time: 0.2548, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8625 [2021-07-08 01:12:14,582][root][INFO] - ===> Epoch[3723](18615/5): Loss 0.0120, Sem 0.0120 LR: 7.159e-02 Score 99.564 Data time: 0.2351, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6967 [2021-07-08 01:12:28,494][root][INFO] - ===> Epoch[3724](18620/5): Loss 0.0130, Sem 0.0130 LR: 7.158e-02 Score 99.527 Data time: 0.2924, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7307 [2021-07-08 01:12:42,096][root][INFO] - ===> Epoch[3725](18625/5): Loss 0.0130, Sem 0.0130 LR: 7.157e-02 Score 99.537 Data time: 0.2062, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6793 [2021-07-08 01:12:56,558][root][INFO] - ===> Epoch[3726](18630/5): Loss 0.0122, Sem 0.0122 LR: 7.156e-02 Score 99.559 Data time: 0.3012, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8437 [2021-07-08 01:13:09,931][root][INFO] - ===> Epoch[3727](18635/5): Loss 0.0126, Sem 0.0126 LR: 7.155e-02 Score 99.535 Data time: 0.2025, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6270 [2021-07-08 01:13:23,910][root][INFO] - ===> Epoch[3728](18640/5): Loss 0.0125, Sem 0.0125 LR: 7.155e-02 Score 99.548 Data time: 0.2829, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7448 [2021-07-08 01:13:38,116][root][INFO] - ===> Epoch[3729](18645/5): Loss 0.0120, Sem 0.0120 LR: 7.154e-02 Score 99.566 Data time: 0.1906, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8047 [2021-07-08 01:13:52,358][root][INFO] - ===> Epoch[3730](18650/5): Loss 0.0121, Sem 0.0121 LR: 7.153e-02 Score 99.563 Data time: 0.2519, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7992 [2021-07-08 01:14:06,779][root][INFO] - ===> Epoch[3731](18655/5): Loss 0.0123, Sem 0.0123 LR: 7.152e-02 Score 99.554 Data time: 0.2176, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8377 [2021-07-08 01:14:20,552][root][INFO] - ===> Epoch[3732](18660/5): Loss 0.0132, Sem 0.0132 LR: 7.152e-02 Score 99.538 Data time: 0.1714, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7175 [2021-07-08 01:14:35,005][root][INFO] - ===> Epoch[3733](18665/5): Loss 0.0124, Sem 0.0124 LR: 7.151e-02 Score 99.548 Data time: 0.2234, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8457 [2021-07-08 01:14:49,073][root][INFO] - ===> Epoch[3734](18670/5): Loss 0.0125, Sem 0.0125 LR: 7.150e-02 Score 99.547 Data time: 0.2561, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7588 [2021-07-08 01:15:02,500][root][INFO] - ===> Epoch[3735](18675/5): Loss 0.0125, Sem 0.0125 LR: 7.149e-02 Score 99.546 Data time: 0.2610, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6351 [2021-07-08 01:15:16,803][root][INFO] - ===> Epoch[3736](18680/5): Loss 0.0123, Sem 0.0123 LR: 7.148e-02 Score 99.552 Data time: 0.1968, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8102 [2021-07-08 01:15:30,659][root][INFO] - ===> Epoch[3737](18685/5): Loss 0.0118, Sem 0.0118 LR: 7.148e-02 Score 99.570 Data time: 0.2406, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7169 [2021-07-08 01:15:45,004][root][INFO] - ===> Epoch[3738](18690/5): Loss 0.0123, Sem 0.0123 LR: 7.147e-02 Score 99.553 Data time: 0.2529, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7916 [2021-07-08 01:15:59,862][root][INFO] - ===> Epoch[3739](18695/5): Loss 0.0118, Sem 0.0118 LR: 7.146e-02 Score 99.570 Data time: 0.2126, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9239 [2021-07-08 01:16:13,910][root][INFO] - ===> Epoch[3740](18700/5): Loss 0.0125, Sem 0.0125 LR: 7.145e-02 Score 99.539 Data time: 0.2620, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7531 [2021-07-08 01:16:27,726][root][INFO] - ===> Epoch[3741](18705/5): Loss 0.0127, Sem 0.0127 LR: 7.145e-02 Score 99.539 Data time: 0.2329, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7046 [2021-07-08 01:16:41,586][root][INFO] - ===> Epoch[3742](18710/5): Loss 0.0124, Sem 0.0124 LR: 7.144e-02 Score 99.554 Data time: 0.2797, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7248 [2021-07-08 01:16:55,700][root][INFO] - ===> Epoch[3743](18715/5): Loss 0.0119, Sem 0.0119 LR: 7.143e-02 Score 99.570 Data time: 0.2831, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7738 [2021-07-08 01:17:09,663][root][INFO] - ===> Epoch[3744](18720/5): Loss 0.0120, Sem 0.0120 LR: 7.142e-02 Score 99.563 Data time: 0.2259, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7503 [2021-07-08 01:17:23,510][root][INFO] - ===> Epoch[3745](18725/5): Loss 0.0120, Sem 0.0120 LR: 7.141e-02 Score 99.566 Data time: 0.2352, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7223 [2021-07-08 01:17:38,245][root][INFO] - ===> Epoch[3746](18730/5): Loss 0.0119, Sem 0.0119 LR: 7.141e-02 Score 99.568 Data time: 0.2277, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.9108 [2021-07-08 01:17:51,490][root][INFO] - ===> Epoch[3747](18735/5): Loss 0.0117, Sem 0.0117 LR: 7.140e-02 Score 99.580 Data time: 0.2525, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6038 [2021-07-08 01:18:06,232][root][INFO] - ===> Epoch[3748](18740/5): Loss 0.0121, Sem 0.0121 LR: 7.139e-02 Score 99.566 Data time: 0.2547, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8945 [2021-07-08 01:18:20,540][root][INFO] - ===> Epoch[3749](18745/5): Loss 0.0116, Sem 0.0116 LR: 7.138e-02 Score 99.582 Data time: 0.2612, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8009 [2021-07-08 01:18:35,853][root][INFO] - ===> Epoch[3750](18750/5): Loss 0.0122, Sem 0.0122 LR: 7.138e-02 Score 99.561 Data time: 0.2408, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 3.0101 [2021-07-08 01:18:50,226][root][INFO] - ===> Epoch[3751](18755/5): Loss 0.0120, Sem 0.0120 LR: 7.137e-02 Score 99.568 Data time: 0.2119, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8356 [2021-07-08 01:19:04,448][root][INFO] - ===> Epoch[3752](18760/5): Loss 0.0119, Sem 0.0119 LR: 7.136e-02 Score 99.566 Data time: 0.1816, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7977 [2021-07-08 01:19:18,696][root][INFO] - ===> Epoch[3753](18765/5): Loss 0.0118, Sem 0.0118 LR: 7.135e-02 Score 99.570 Data time: 0.1521, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8043 [2021-07-08 01:19:33,679][root][INFO] - ===> Epoch[3754](18770/5): Loss 0.0118, Sem 0.0118 LR: 7.134e-02 Score 99.567 Data time: 0.1942, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9442 [2021-07-08 01:19:47,537][root][INFO] - ===> Epoch[3755](18775/5): Loss 0.0125, Sem 0.0125 LR: 7.134e-02 Score 99.552 Data time: 0.2114, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7279 [2021-07-08 01:20:01,074][root][INFO] - ===> Epoch[3756](18780/5): Loss 0.0128, Sem 0.0128 LR: 7.133e-02 Score 99.535 Data time: 0.2020, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6660 [2021-07-08 01:20:14,748][root][INFO] - ===> Epoch[3757](18785/5): Loss 0.0120, Sem 0.0120 LR: 7.132e-02 Score 99.564 Data time: 0.2928, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6850 [2021-07-08 01:20:28,408][root][INFO] - ===> Epoch[3758](18790/5): Loss 0.0120, Sem 0.0120 LR: 7.131e-02 Score 99.564 Data time: 0.2227, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6836 [2021-07-08 01:20:41,341][root][INFO] - ===> Epoch[3759](18795/5): Loss 0.0128, Sem 0.0128 LR: 7.131e-02 Score 99.539 Data time: 0.1484, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.5515 [2021-07-08 01:20:55,181][root][INFO] - ===> Epoch[3760](18800/5): Loss 0.0123, Sem 0.0123 LR: 7.130e-02 Score 99.564 Data time: 0.2164, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7199 [2021-07-08 01:20:55,182][root][INFO] - ===> Start testing [2021-07-08 01:21:29,255][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1240 Loss 0.284 (AVG: 0.541) Score 93.324 (AVG: 91.175) mIOU 70.879 mAP 83.440 mAcc 76.733 IOU: 65.737 0.000 82.609 76.221 95.545 91.839 24.662 83.364 98.330 77.586 80.540 86.099 58.891 mAP: 84.106 5.473 92.464 89.133 98.956 98.706 53.798 97.759 99.802 86.566 92.590 93.376 91.989 mAcc: 81.682 0.000 89.335 82.740 96.729 96.984 26.359 95.889 99.264 79.538 89.353 96.773 62.887 [2021-07-08 01:21:29,255][root][INFO] - Finished test. Elapsed time: 34.0724 [2021-07-08 01:21:30,275][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 01:21:43,245][root][INFO] - ===> Epoch[3761](18805/5): Loss 0.0128, Sem 0.0128 LR: 7.129e-02 Score 99.533 Data time: 0.1147, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.5543 [2021-07-08 01:21:57,488][root][INFO] - ===> Epoch[3762](18810/5): Loss 0.0129, Sem 0.0129 LR: 7.128e-02 Score 99.547 Data time: 0.2496, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7958 [2021-07-08 01:22:11,395][root][INFO] - ===> Epoch[3763](18815/5): Loss 0.0124, Sem 0.0124 LR: 7.127e-02 Score 99.555 Data time: 0.2312, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7336 [2021-07-08 01:22:26,671][root][INFO] - ===> Epoch[3764](18820/5): Loss 0.0128, Sem 0.0128 LR: 7.127e-02 Score 99.534 Data time: 0.2627, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 3.0162 [2021-07-08 01:22:40,468][root][INFO] - ===> Epoch[3765](18825/5): Loss 0.0117, Sem 0.0117 LR: 7.126e-02 Score 99.575 Data time: 0.2176, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7139 [2021-07-08 01:22:54,133][root][INFO] - ===> Epoch[3766](18830/5): Loss 0.0124, Sem 0.0124 LR: 7.125e-02 Score 99.559 Data time: 0.1669, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6942 [2021-07-08 01:23:08,025][root][INFO] - ===> Epoch[3767](18835/5): Loss 0.0124, Sem 0.0124 LR: 7.124e-02 Score 99.548 Data time: 0.2637, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7320 [2021-07-08 01:23:21,986][root][INFO] - ===> Epoch[3768](18840/5): Loss 0.0130, Sem 0.0130 LR: 7.124e-02 Score 99.528 Data time: 0.2597, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7394 [2021-07-08 01:23:35,504][root][INFO] - ===> Epoch[3769](18845/5): Loss 0.0133, Sem 0.0133 LR: 7.123e-02 Score 99.529 Data time: 0.2348, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6524 [2021-07-08 01:23:49,664][root][INFO] - ===> Epoch[3770](18850/5): Loss 0.0129, Sem 0.0129 LR: 7.122e-02 Score 99.529 Data time: 0.2367, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7857 [2021-07-08 01:24:02,888][root][INFO] - ===> Epoch[3771](18855/5): Loss 0.0132, Sem 0.0132 LR: 7.121e-02 Score 99.527 Data time: 0.2341, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.5947 [2021-07-08 01:24:17,492][root][INFO] - ===> Epoch[3772](18860/5): Loss 0.0128, Sem 0.0128 LR: 7.120e-02 Score 99.538 Data time: 0.2165, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8753 [2021-07-08 01:24:32,300][root][INFO] - ===> Epoch[3773](18865/5): Loss 0.0127, Sem 0.0127 LR: 7.120e-02 Score 99.543 Data time: 0.2473, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9144 [2021-07-08 01:24:45,758][root][INFO] - ===> Epoch[3774](18870/5): Loss 0.0133, Sem 0.0133 LR: 7.119e-02 Score 99.530 Data time: 0.2294, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6323 [2021-07-08 01:25:00,936][root][INFO] - ===> Epoch[3775](18875/5): Loss 0.0146, Sem 0.0146 LR: 7.118e-02 Score 99.490 Data time: 0.2352, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9946 [2021-07-08 01:25:14,564][root][INFO] - ===> Epoch[3776](18880/5): Loss 0.0162, Sem 0.0162 LR: 7.117e-02 Score 99.431 Data time: 0.2103, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6708 [2021-07-08 01:25:28,262][root][INFO] - ===> Epoch[3777](18885/5): Loss 0.0168, Sem 0.0168 LR: 7.117e-02 Score 99.399 Data time: 0.2013, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6941 [2021-07-08 01:25:41,445][root][INFO] - ===> Epoch[3778](18890/5): Loss 0.0156, Sem 0.0156 LR: 7.116e-02 Score 99.442 Data time: 0.2339, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.5922 [2021-07-08 01:25:56,025][root][INFO] - ===> Epoch[3779](18895/5): Loss 0.0134, Sem 0.0134 LR: 7.115e-02 Score 99.512 Data time: 0.1902, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8637 [2021-07-08 01:26:11,041][root][INFO] - ===> Epoch[3780](18900/5): Loss 0.0146, Sem 0.0146 LR: 7.114e-02 Score 99.478 Data time: 0.2517, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9557 [2021-07-08 01:26:25,036][root][INFO] - ===> Epoch[3781](18905/5): Loss 0.0131, Sem 0.0131 LR: 7.113e-02 Score 99.526 Data time: 0.2016, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7558 [2021-07-08 01:26:38,847][root][INFO] - ===> Epoch[3782](18910/5): Loss 0.0136, Sem 0.0136 LR: 7.113e-02 Score 99.507 Data time: 0.1695, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7177 [2021-07-08 01:26:52,542][root][INFO] - ===> Epoch[3783](18915/5): Loss 0.0128, Sem 0.0128 LR: 7.112e-02 Score 99.534 Data time: 0.2073, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6991 [2021-07-08 01:27:06,491][root][INFO] - ===> Epoch[3784](18920/5): Loss 0.0130, Sem 0.0130 LR: 7.111e-02 Score 99.531 Data time: 0.2122, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7464 [2021-07-08 01:27:21,512][root][INFO] - ===> Epoch[3785](18925/5): Loss 0.0128, Sem 0.0128 LR: 7.110e-02 Score 99.541 Data time: 0.2160, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9588 [2021-07-08 01:27:35,611][root][INFO] - ===> Epoch[3786](18930/5): Loss 0.0121, Sem 0.0121 LR: 7.110e-02 Score 99.560 Data time: 0.2836, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7623 [2021-07-08 01:27:50,497][root][INFO] - ===> Epoch[3787](18935/5): Loss 0.0127, Sem 0.0127 LR: 7.109e-02 Score 99.547 Data time: 0.2422, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9134 [2021-07-08 01:28:04,054][root][INFO] - ===> Epoch[3788](18940/5): Loss 0.0126, Sem 0.0126 LR: 7.108e-02 Score 99.543 Data time: 0.1961, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6690 [2021-07-08 01:28:18,619][root][INFO] - ===> Epoch[3789](18945/5): Loss 0.0128, Sem 0.0128 LR: 7.107e-02 Score 99.534 Data time: 0.3204, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8589 [2021-07-08 01:28:32,119][root][INFO] - ===> Epoch[3790](18950/5): Loss 0.0129, Sem 0.0129 LR: 7.106e-02 Score 99.529 Data time: 0.2134, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6597 [2021-07-08 01:28:45,982][root][INFO] - ===> Epoch[3791](18955/5): Loss 0.0129, Sem 0.0129 LR: 7.106e-02 Score 99.532 Data time: 0.2259, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7270 [2021-07-08 01:28:59,882][root][INFO] - ===> Epoch[3792](18960/5): Loss 0.0118, Sem 0.0118 LR: 7.105e-02 Score 99.579 Data time: 0.1911, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7430 [2021-07-08 01:29:14,784][root][INFO] - ===> Epoch[3793](18965/5): Loss 0.0124, Sem 0.0124 LR: 7.104e-02 Score 99.551 Data time: 0.2447, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9326 [2021-07-08 01:29:28,889][root][INFO] - ===> Epoch[3794](18970/5): Loss 0.0126, Sem 0.0126 LR: 7.103e-02 Score 99.548 Data time: 0.2629, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7778 [2021-07-08 01:29:43,278][root][INFO] - ===> Epoch[3795](18975/5): Loss 0.0119, Sem 0.0119 LR: 7.102e-02 Score 99.570 Data time: 0.2560, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8188 [2021-07-08 01:29:56,958][root][INFO] - ===> Epoch[3796](18980/5): Loss 0.0126, Sem 0.0126 LR: 7.102e-02 Score 99.543 Data time: 0.2178, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6923 [2021-07-08 01:30:10,452][root][INFO] - ===> Epoch[3797](18985/5): Loss 0.0126, Sem 0.0126 LR: 7.101e-02 Score 99.543 Data time: 0.2271, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6462 [2021-07-08 01:30:25,471][root][INFO] - ===> Epoch[3798](18990/5): Loss 0.0122, Sem 0.0122 LR: 7.100e-02 Score 99.563 Data time: 0.2461, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9367 [2021-07-08 01:30:39,932][root][INFO] - ===> Epoch[3799](18995/5): Loss 0.0113, Sem 0.0113 LR: 7.099e-02 Score 99.588 Data time: 0.2261, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8467 [2021-07-08 01:30:54,753][root][INFO] - ===> Epoch[3800](19000/5): Loss 0.0124, Sem 0.0124 LR: 7.099e-02 Score 99.560 Data time: 0.2207, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9179 [2021-07-08 01:30:54,755][root][INFO] - ===> Start testing [2021-07-08 01:31:30,404][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1258 Loss 0.298 (AVG: 0.540) Score 92.675 (AVG: 91.205) mIOU 70.910 mAP 84.185 mAcc 77.005 IOU: 65.593 0.015 84.058 75.745 95.814 92.355 23.621 82.454 98.555 78.190 80.940 86.362 58.129 mAP: 84.704 11.490 93.889 88.979 98.855 98.690 56.079 97.876 99.782 84.848 93.065 94.507 91.636 mAcc: 81.530 0.091 88.889 82.313 97.124 96.973 24.777 96.402 99.557 82.833 92.112 96.408 62.052 [2021-07-08 01:31:30,405][root][INFO] - Finished test. Elapsed time: 35.6495 [2021-07-08 01:31:31,457][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 01:31:44,640][root][INFO] - ===> Epoch[3801](19005/5): Loss 0.0120, Sem 0.0120 LR: 7.098e-02 Score 99.563 Data time: 0.1731, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6077 [2021-07-08 01:31:58,503][root][INFO] - ===> Epoch[3802](19010/5): Loss 0.0119, Sem 0.0119 LR: 7.097e-02 Score 99.571 Data time: 0.2726, Forward time: 1.2642, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7215 [2021-07-08 01:32:12,321][root][INFO] - ===> Epoch[3803](19015/5): Loss 0.0121, Sem 0.0121 LR: 7.096e-02 Score 99.561 Data time: 0.2553, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7070 [2021-07-08 01:32:26,272][root][INFO] - ===> Epoch[3804](19020/5): Loss 0.0118, Sem 0.0118 LR: 7.095e-02 Score 99.575 Data time: 0.2669, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7473 [2021-07-08 01:32:41,042][root][INFO] - ===> Epoch[3805](19025/5): Loss 0.0118, Sem 0.0118 LR: 7.095e-02 Score 99.576 Data time: 0.2784, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8965 [2021-07-08 01:32:55,258][root][INFO] - ===> Epoch[3806](19030/5): Loss 0.0122, Sem 0.0122 LR: 7.094e-02 Score 99.564 Data time: 0.3172, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7926 [2021-07-08 01:33:09,559][root][INFO] - ===> Epoch[3807](19035/5): Loss 0.0123, Sem 0.0123 LR: 7.093e-02 Score 99.554 Data time: 0.2613, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8153 [2021-07-08 01:33:23,157][root][INFO] - ===> Epoch[3808](19040/5): Loss 0.0122, Sem 0.0122 LR: 7.092e-02 Score 99.556 Data time: 0.1692, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6774 [2021-07-08 01:33:36,760][root][INFO] - ===> Epoch[3809](19045/5): Loss 0.0120, Sem 0.0120 LR: 7.092e-02 Score 99.569 Data time: 0.2492, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6747 [2021-07-08 01:33:51,219][root][INFO] - ===> Epoch[3810](19050/5): Loss 0.0119, Sem 0.0119 LR: 7.091e-02 Score 99.570 Data time: 0.2132, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8330 [2021-07-08 01:34:04,588][root][INFO] - ===> Epoch[3811](19055/5): Loss 0.0124, Sem 0.0124 LR: 7.090e-02 Score 99.555 Data time: 0.2146, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6360 [2021-07-08 01:34:17,996][root][INFO] - ===> Epoch[3812](19060/5): Loss 0.0124, Sem 0.0124 LR: 7.089e-02 Score 99.557 Data time: 0.2476, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6405 [2021-07-08 01:34:31,981][root][INFO] - ===> Epoch[3813](19065/5): Loss 0.0125, Sem 0.0125 LR: 7.088e-02 Score 99.549 Data time: 0.2109, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7534 [2021-07-08 01:34:46,356][root][INFO] - ===> Epoch[3814](19070/5): Loss 0.0112, Sem 0.0112 LR: 7.088e-02 Score 99.593 Data time: 0.2338, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8339 [2021-07-08 01:35:00,521][root][INFO] - ===> Epoch[3815](19075/5): Loss 0.0121, Sem 0.0121 LR: 7.087e-02 Score 99.562 Data time: 0.2164, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7820 [2021-07-08 01:35:14,300][root][INFO] - ===> Epoch[3816](19080/5): Loss 0.0119, Sem 0.0119 LR: 7.086e-02 Score 99.568 Data time: 0.2136, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7066 [2021-07-08 01:35:28,032][root][INFO] - ===> Epoch[3817](19085/5): Loss 0.0118, Sem 0.0118 LR: 7.085e-02 Score 99.569 Data time: 0.2448, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7062 [2021-07-08 01:35:41,626][root][INFO] - ===> Epoch[3818](19090/5): Loss 0.0115, Sem 0.0115 LR: 7.085e-02 Score 99.584 Data time: 0.1777, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6783 [2021-07-08 01:35:55,895][root][INFO] - ===> Epoch[3819](19095/5): Loss 0.0122, Sem 0.0122 LR: 7.084e-02 Score 99.560 Data time: 0.2078, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8004 [2021-07-08 01:36:09,945][root][INFO] - ===> Epoch[3820](19100/5): Loss 0.0115, Sem 0.0115 LR: 7.083e-02 Score 99.581 Data time: 0.2615, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7619 [2021-07-08 01:36:23,212][root][INFO] - ===> Epoch[3821](19105/5): Loss 0.0121, Sem 0.0121 LR: 7.082e-02 Score 99.565 Data time: 0.2373, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.5968 [2021-07-08 01:36:37,886][root][INFO] - ===> Epoch[3822](19110/5): Loss 0.0112, Sem 0.0112 LR: 7.081e-02 Score 99.594 Data time: 0.2085, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8826 [2021-07-08 01:36:51,537][root][INFO] - ===> Epoch[3823](19115/5): Loss 0.0117, Sem 0.0117 LR: 7.081e-02 Score 99.581 Data time: 0.2825, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6805 [2021-07-08 01:37:05,612][root][INFO] - ===> Epoch[3824](19120/5): Loss 0.0123, Sem 0.0123 LR: 7.080e-02 Score 99.558 Data time: 0.2244, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7679 [2021-07-08 01:37:19,797][root][INFO] - ===> Epoch[3825](19125/5): Loss 0.0120, Sem 0.0120 LR: 7.079e-02 Score 99.566 Data time: 0.2181, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7896 [2021-07-08 01:37:34,470][root][INFO] - ===> Epoch[3826](19130/5): Loss 0.0116, Sem 0.0116 LR: 7.078e-02 Score 99.582 Data time: 0.2241, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8797 [2021-07-08 01:37:48,677][root][INFO] - ===> Epoch[3827](19135/5): Loss 0.0118, Sem 0.0118 LR: 7.078e-02 Score 99.574 Data time: 0.2037, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7934 [2021-07-08 01:38:02,509][root][INFO] - ===> Epoch[3828](19140/5): Loss 0.0121, Sem 0.0121 LR: 7.077e-02 Score 99.559 Data time: 0.2363, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7189 [2021-07-08 01:38:16,766][root][INFO] - ===> Epoch[3829](19145/5): Loss 0.0118, Sem 0.0118 LR: 7.076e-02 Score 99.571 Data time: 0.2954, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8017 [2021-07-08 01:38:31,605][root][INFO] - ===> Epoch[3830](19150/5): Loss 0.0115, Sem 0.0115 LR: 7.075e-02 Score 99.580 Data time: 0.2859, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.9204 [2021-07-08 01:38:45,910][root][INFO] - ===> Epoch[3831](19155/5): Loss 0.0116, Sem 0.0116 LR: 7.074e-02 Score 99.580 Data time: 0.3001, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8046 [2021-07-08 01:39:00,256][root][INFO] - ===> Epoch[3832](19160/5): Loss 0.0121, Sem 0.0121 LR: 7.074e-02 Score 99.562 Data time: 0.2405, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8244 [2021-07-08 01:39:14,381][root][INFO] - ===> Epoch[3833](19165/5): Loss 0.0115, Sem 0.0115 LR: 7.073e-02 Score 99.581 Data time: 0.1972, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7739 [2021-07-08 01:39:28,289][root][INFO] - ===> Epoch[3834](19170/5): Loss 0.0121, Sem 0.0121 LR: 7.072e-02 Score 99.563 Data time: 0.1518, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7356 [2021-07-08 01:39:42,298][root][INFO] - ===> Epoch[3835](19175/5): Loss 0.0120, Sem 0.0120 LR: 7.071e-02 Score 99.566 Data time: 0.2049, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7646 [2021-07-08 01:39:55,834][root][INFO] - ===> Epoch[3836](19180/5): Loss 0.0117, Sem 0.0117 LR: 7.071e-02 Score 99.577 Data time: 0.1929, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6548 [2021-07-08 01:40:09,560][root][INFO] - ===> Epoch[3837](19185/5): Loss 0.0111, Sem 0.0111 LR: 7.070e-02 Score 99.599 Data time: 0.2472, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6865 [2021-07-08 01:40:24,091][root][INFO] - ===> Epoch[3838](19190/5): Loss 0.0111, Sem 0.0111 LR: 7.069e-02 Score 99.596 Data time: 0.2097, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8679 [2021-07-08 01:40:38,372][root][INFO] - ===> Epoch[3839](19195/5): Loss 0.0117, Sem 0.0117 LR: 7.068e-02 Score 99.577 Data time: 0.2176, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8096 [2021-07-08 01:40:51,453][root][INFO] - ===> Epoch[3840](19200/5): Loss 0.0115, Sem 0.0115 LR: 7.067e-02 Score 99.582 Data time: 0.2444, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.5786 [2021-07-08 01:40:51,454][root][INFO] - ===> Start testing [2021-07-08 01:41:26,661][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1197 Loss 0.285 (AVG: 0.568) Score 93.141 (AVG: 90.996) mIOU 71.018 mAP 83.402 mAcc 76.846 IOU: 64.616 0.000 85.283 76.204 95.019 92.573 25.122 82.873 98.594 79.427 80.366 86.044 57.111 mAP: 83.877 4.180 94.426 89.278 98.855 98.728 53.028 98.218 99.776 85.710 92.486 93.909 91.754 mAcc: 81.549 0.000 89.885 82.763 96.437 97.081 27.023 96.841 99.544 80.979 90.050 96.191 60.655 [2021-07-08 01:41:26,661][root][INFO] - Finished test. Elapsed time: 35.2061 [2021-07-08 01:41:27,662][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 01:41:40,741][root][INFO] - ===> Epoch[3841](19205/5): Loss 0.0114, Sem 0.0114 LR: 7.067e-02 Score 99.582 Data time: 0.1722, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.5784 [2021-07-08 01:41:54,663][root][INFO] - ===> Epoch[3842](19210/5): Loss 0.0120, Sem 0.0120 LR: 7.066e-02 Score 99.567 Data time: 0.2152, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7403 [2021-07-08 01:42:08,260][root][INFO] - ===> Epoch[3843](19215/5): Loss 0.0118, Sem 0.0118 LR: 7.065e-02 Score 99.567 Data time: 0.2808, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6664 [2021-07-08 01:42:22,462][root][INFO] - ===> Epoch[3844](19220/5): Loss 0.0117, Sem 0.0117 LR: 7.064e-02 Score 99.576 Data time: 0.2197, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7995 [2021-07-08 01:42:37,634][root][INFO] - ===> Epoch[3845](19225/5): Loss 0.0113, Sem 0.0113 LR: 7.064e-02 Score 99.592 Data time: 0.2328, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9819 [2021-07-08 01:42:51,132][root][INFO] - ===> Epoch[3846](19230/5): Loss 0.0115, Sem 0.0115 LR: 7.063e-02 Score 99.584 Data time: 0.1923, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6549 [2021-07-08 01:43:05,908][root][INFO] - ===> Epoch[3847](19235/5): Loss 0.0117, Sem 0.0117 LR: 7.062e-02 Score 99.573 Data time: 0.2277, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9085 [2021-07-08 01:43:20,092][root][INFO] - ===> Epoch[3848](19240/5): Loss 0.0113, Sem 0.0113 LR: 7.061e-02 Score 99.594 Data time: 0.2449, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7923 [2021-07-08 01:43:34,632][root][INFO] - ===> Epoch[3849](19245/5): Loss 0.0113, Sem 0.0113 LR: 7.060e-02 Score 99.589 Data time: 0.2025, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8683 [2021-07-08 01:43:48,498][root][INFO] - ===> Epoch[3850](19250/5): Loss 0.0119, Sem 0.0119 LR: 7.060e-02 Score 99.572 Data time: 0.2186, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7319 [2021-07-08 01:44:02,861][root][INFO] - ===> Epoch[3851](19255/5): Loss 0.0120, Sem 0.0120 LR: 7.059e-02 Score 99.571 Data time: 0.2757, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8217 [2021-07-08 01:44:17,349][root][INFO] - ===> Epoch[3852](19260/5): Loss 0.0121, Sem 0.0121 LR: 7.058e-02 Score 99.565 Data time: 0.2560, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8471 [2021-07-08 01:44:31,308][root][INFO] - ===> Epoch[3853](19265/5): Loss 0.0117, Sem 0.0117 LR: 7.057e-02 Score 99.575 Data time: 0.2235, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7411 [2021-07-08 01:44:45,191][root][INFO] - ===> Epoch[3854](19270/5): Loss 0.0120, Sem 0.0120 LR: 7.057e-02 Score 99.561 Data time: 0.2227, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7228 [2021-07-08 01:44:59,102][root][INFO] - ===> Epoch[3855](19275/5): Loss 0.0123, Sem 0.0123 LR: 7.056e-02 Score 99.552 Data time: 0.2505, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7296 [2021-07-08 01:45:13,036][root][INFO] - ===> Epoch[3856](19280/5): Loss 0.0118, Sem 0.0118 LR: 7.055e-02 Score 99.570 Data time: 0.2338, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7359 [2021-07-08 01:45:27,062][root][INFO] - ===> Epoch[3857](19285/5): Loss 0.0114, Sem 0.0114 LR: 7.054e-02 Score 99.582 Data time: 0.2059, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7617 [2021-07-08 01:45:40,928][root][INFO] - ===> Epoch[3858](19290/5): Loss 0.0113, Sem 0.0113 LR: 7.053e-02 Score 99.593 Data time: 0.2068, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7214 [2021-07-08 01:45:55,139][root][INFO] - ===> Epoch[3859](19295/5): Loss 0.0112, Sem 0.0112 LR: 7.053e-02 Score 99.594 Data time: 0.2223, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7907 [2021-07-08 01:46:08,920][root][INFO] - ===> Epoch[3860](19300/5): Loss 0.0118, Sem 0.0118 LR: 7.052e-02 Score 99.574 Data time: 0.2617, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7071 [2021-07-08 01:46:22,835][root][INFO] - ===> Epoch[3861](19305/5): Loss 0.0118, Sem 0.0118 LR: 7.051e-02 Score 99.574 Data time: 0.1877, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7309 [2021-07-08 01:46:37,008][root][INFO] - ===> Epoch[3862](19310/5): Loss 0.0109, Sem 0.0109 LR: 7.050e-02 Score 99.606 Data time: 0.2132, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7897 [2021-07-08 01:46:51,113][root][INFO] - ===> Epoch[3863](19315/5): Loss 0.0118, Sem 0.0118 LR: 7.049e-02 Score 99.568 Data time: 0.1998, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7737 [2021-07-08 01:47:04,528][root][INFO] - ===> Epoch[3864](19320/5): Loss 0.0118, Sem 0.0118 LR: 7.049e-02 Score 99.583 Data time: 0.1802, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6475 [2021-07-08 01:47:18,403][root][INFO] - ===> Epoch[3865](19325/5): Loss 0.0118, Sem 0.0118 LR: 7.048e-02 Score 99.573 Data time: 0.2521, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7332 [2021-07-08 01:47:32,677][root][INFO] - ===> Epoch[3866](19330/5): Loss 0.0122, Sem 0.0122 LR: 7.047e-02 Score 99.560 Data time: 0.2343, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8075 [2021-07-08 01:47:46,450][root][INFO] - ===> Epoch[3867](19335/5): Loss 0.0120, Sem 0.0120 LR: 7.046e-02 Score 99.568 Data time: 0.1997, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7026 [2021-07-08 01:48:00,115][root][INFO] - ===> Epoch[3868](19340/5): Loss 0.0119, Sem 0.0119 LR: 7.046e-02 Score 99.574 Data time: 0.2172, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6789 [2021-07-08 01:48:14,043][root][INFO] - ===> Epoch[3869](19345/5): Loss 0.0114, Sem 0.0114 LR: 7.045e-02 Score 99.588 Data time: 0.2150, Forward time: 1.2642, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7178 [2021-07-08 01:48:27,647][root][INFO] - ===> Epoch[3870](19350/5): Loss 0.0120, Sem 0.0120 LR: 7.044e-02 Score 99.567 Data time: 0.2048, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6806 [2021-07-08 01:48:41,308][root][INFO] - ===> Epoch[3871](19355/5): Loss 0.0119, Sem 0.0119 LR: 7.043e-02 Score 99.568 Data time: 0.2593, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6759 [2021-07-08 01:48:55,152][root][INFO] - ===> Epoch[3872](19360/5): Loss 0.0117, Sem 0.0117 LR: 7.042e-02 Score 99.579 Data time: 0.2182, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7206 [2021-07-08 01:49:09,781][root][INFO] - ===> Epoch[3873](19365/5): Loss 0.0120, Sem 0.0120 LR: 7.042e-02 Score 99.567 Data time: 0.2257, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8807 [2021-07-08 01:49:23,914][root][INFO] - ===> Epoch[3874](19370/5): Loss 0.0116, Sem 0.0116 LR: 7.041e-02 Score 99.578 Data time: 0.2176, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7803 [2021-07-08 01:49:37,829][root][INFO] - ===> Epoch[3875](19375/5): Loss 0.0115, Sem 0.0115 LR: 7.040e-02 Score 99.584 Data time: 0.2631, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7238 [2021-07-08 01:49:51,379][root][INFO] - ===> Epoch[3876](19380/5): Loss 0.0118, Sem 0.0118 LR: 7.039e-02 Score 99.571 Data time: 0.3140, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6586 [2021-07-08 01:50:05,526][root][INFO] - ===> Epoch[3877](19385/5): Loss 0.0112, Sem 0.0112 LR: 7.039e-02 Score 99.601 Data time: 0.2455, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7795 [2021-07-08 01:50:18,898][root][INFO] - ===> Epoch[3878](19390/5): Loss 0.0119, Sem 0.0119 LR: 7.038e-02 Score 99.568 Data time: 0.2928, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6318 [2021-07-08 01:50:31,970][root][INFO] - ===> Epoch[3879](19395/5): Loss 0.0118, Sem 0.0118 LR: 7.037e-02 Score 99.572 Data time: 0.1834, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.5698 [2021-07-08 01:50:45,994][root][INFO] - ===> Epoch[3880](19400/5): Loss 0.0112, Sem 0.0112 LR: 7.036e-02 Score 99.598 Data time: 0.2484, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7579 [2021-07-08 01:50:45,996][root][INFO] - ===> Start testing [2021-07-08 01:51:21,597][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1496 Loss 0.332 (AVG: 0.525) Score 92.736 (AVG: 91.530) mIOU 72.220 mAP 84.544 mAcc 78.059 IOU: 66.084 0.000 83.893 76.352 95.583 91.764 38.639 87.552 98.611 75.791 79.613 87.074 57.901 mAP: 84.755 6.478 95.430 88.252 98.870 98.681 61.995 98.575 99.792 88.812 92.254 94.750 90.433 mAcc: 82.997 0.000 91.856 82.411 96.763 96.689 42.071 96.494 99.469 78.266 89.295 96.610 61.852 [2021-07-08 01:51:21,597][root][INFO] - Finished test. Elapsed time: 35.6002 [2021-07-08 01:51:22,572][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 01:51:35,752][root][INFO] - ===> Epoch[3881](19405/5): Loss 0.0115, Sem 0.0115 LR: 7.035e-02 Score 99.585 Data time: 0.1568, Forward time: 1.2640, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.5943 [2021-07-08 01:51:49,839][root][INFO] - ===> Epoch[3882](19410/5): Loss 0.0111, Sem 0.0111 LR: 7.035e-02 Score 99.597 Data time: 0.1914, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7754 [2021-07-08 01:52:03,811][root][INFO] - ===> Epoch[3883](19415/5): Loss 0.0117, Sem 0.0117 LR: 7.034e-02 Score 99.580 Data time: 0.2268, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7627 [2021-07-08 01:52:18,210][root][INFO] - ===> Epoch[3884](19420/5): Loss 0.0109, Sem 0.0109 LR: 7.033e-02 Score 99.605 Data time: 0.1742, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8369 [2021-07-08 01:52:33,245][root][INFO] - ===> Epoch[3885](19425/5): Loss 0.0108, Sem 0.0108 LR: 7.032e-02 Score 99.604 Data time: 0.2649, Forward time: 1.2640, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9627 [2021-07-08 01:52:47,451][root][INFO] - ===> Epoch[3886](19430/5): Loss 0.0109, Sem 0.0109 LR: 7.032e-02 Score 99.611 Data time: 0.1763, Forward time: 1.2640, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8027 [2021-07-08 01:53:02,280][root][INFO] - ===> Epoch[3887](19435/5): Loss 0.0121, Sem 0.0121 LR: 7.031e-02 Score 99.562 Data time: 0.2081, Forward time: 1.2640, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9141 [2021-07-08 01:53:16,121][root][INFO] - ===> Epoch[3888](19440/5): Loss 0.0115, Sem 0.0115 LR: 7.030e-02 Score 99.579 Data time: 0.2062, Forward time: 1.2640, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7256 [2021-07-08 01:53:30,129][root][INFO] - ===> Epoch[3889](19445/5): Loss 0.0114, Sem 0.0114 LR: 7.029e-02 Score 99.590 Data time: 0.2839, Forward time: 1.2640, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7575 [2021-07-08 01:53:43,232][root][INFO] - ===> Epoch[3890](19450/5): Loss 0.0118, Sem 0.0118 LR: 7.028e-02 Score 99.575 Data time: 0.2650, Forward time: 1.2640, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.5593 [2021-07-08 01:53:57,373][root][INFO] - ===> Epoch[3891](19455/5): Loss 0.0116, Sem 0.0116 LR: 7.028e-02 Score 99.578 Data time: 0.2045, Forward time: 1.2640, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7814 [2021-07-08 01:54:10,913][root][INFO] - ===> Epoch[3892](19460/5): Loss 0.0115, Sem 0.0115 LR: 7.027e-02 Score 99.584 Data time: 0.1738, Forward time: 1.2639, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6695 [2021-07-08 01:54:24,987][root][INFO] - ===> Epoch[3893](19465/5): Loss 0.0114, Sem 0.0114 LR: 7.026e-02 Score 99.589 Data time: 0.2094, Forward time: 1.2639, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7701 [2021-07-08 01:54:39,386][root][INFO] - ===> Epoch[3894](19470/5): Loss 0.0113, Sem 0.0113 LR: 7.025e-02 Score 99.593 Data time: 0.2282, Forward time: 1.2639, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8304 [2021-07-08 01:54:55,684][root][INFO] - ===> Epoch[3895](19475/5): Loss 0.0111, Sem 0.0111 LR: 7.025e-02 Score 99.596 Data time: 0.3453, Forward time: 1.2640, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 3.1893 [2021-07-08 01:55:09,930][root][INFO] - ===> Epoch[3896](19480/5): Loss 0.0113, Sem 0.0113 LR: 7.024e-02 Score 99.592 Data time: 0.2654, Forward time: 1.2640, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7941 [2021-07-08 01:55:24,811][root][INFO] - ===> Epoch[3897](19485/5): Loss 0.0115, Sem 0.0115 LR: 7.023e-02 Score 99.582 Data time: 0.3198, Forward time: 1.2641, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9194 [2021-07-08 01:55:38,450][root][INFO] - ===> Epoch[3898](19490/5): Loss 0.0114, Sem 0.0114 LR: 7.022e-02 Score 99.589 Data time: 0.3024, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.6739 [2021-07-08 01:55:52,359][root][INFO] - ===> Epoch[3899](19495/5): Loss 0.0115, Sem 0.0115 LR: 7.021e-02 Score 99.583 Data time: 0.2251, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7344 [2021-07-08 01:56:06,535][root][INFO] - ===> Epoch[3900](19500/5): Loss 0.0112, Sem 0.0112 LR: 7.021e-02 Score 99.592 Data time: 0.2312, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7961 [2021-07-08 01:56:20,708][root][INFO] - ===> Epoch[3901](19505/5): Loss 0.0107, Sem 0.0107 LR: 7.020e-02 Score 99.611 Data time: 0.2385, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7780 [2021-07-08 01:56:33,790][root][INFO] - ===> Epoch[3902](19510/5): Loss 0.0117, Sem 0.0117 LR: 7.019e-02 Score 99.576 Data time: 0.1765, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.5759 [2021-07-08 01:56:47,879][root][INFO] - ===> Epoch[3903](19515/5): Loss 0.0113, Sem 0.0113 LR: 7.018e-02 Score 99.590 Data time: 0.2310, Forward time: 1.2641, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7730 [2021-07-08 01:57:01,549][root][INFO] - ===> Epoch[3904](19520/5): Loss 0.0112, Sem 0.0112 LR: 7.018e-02 Score 99.593 Data time: 0.2433, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6887 [2021-07-08 01:57:15,086][root][INFO] - ===> Epoch[3905](19525/5): Loss 0.0117, Sem 0.0117 LR: 7.017e-02 Score 99.581 Data time: 0.2183, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6686 [2021-07-08 01:57:28,763][root][INFO] - ===> Epoch[3906](19530/5): Loss 0.0116, Sem 0.0116 LR: 7.016e-02 Score 99.576 Data time: 0.2244, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6962 [2021-07-08 01:57:43,303][root][INFO] - ===> Epoch[3907](19535/5): Loss 0.0113, Sem 0.0113 LR: 7.015e-02 Score 99.592 Data time: 0.2212, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8644 [2021-07-08 01:57:57,232][root][INFO] - ===> Epoch[3908](19540/5): Loss 0.0115, Sem 0.0115 LR: 7.014e-02 Score 99.587 Data time: 0.2603, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7324 [2021-07-08 01:58:11,344][root][INFO] - ===> Epoch[3909](19545/5): Loss 0.0113, Sem 0.0113 LR: 7.014e-02 Score 99.588 Data time: 0.2746, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7772 [2021-07-08 01:58:25,361][root][INFO] - ===> Epoch[3910](19550/5): Loss 0.0116, Sem 0.0116 LR: 7.013e-02 Score 99.592 Data time: 0.2441, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7635 [2021-07-08 01:58:38,904][root][INFO] - ===> Epoch[3911](19555/5): Loss 0.0112, Sem 0.0112 LR: 7.012e-02 Score 99.588 Data time: 0.2049, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6728 [2021-07-08 01:58:52,974][root][INFO] - ===> Epoch[3912](19560/5): Loss 0.0112, Sem 0.0112 LR: 7.011e-02 Score 99.593 Data time: 0.2977, Forward time: 1.2641, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7663 [2021-07-08 01:59:06,585][root][INFO] - ===> Epoch[3913](19565/5): Loss 0.0114, Sem 0.0114 LR: 7.010e-02 Score 99.587 Data time: 0.2132, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6686 [2021-07-08 01:59:19,911][root][INFO] - ===> Epoch[3914](19570/5): Loss 0.0117, Sem 0.0117 LR: 7.010e-02 Score 99.581 Data time: 0.1997, Forward time: 1.2641, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6248 [2021-07-08 01:59:34,014][root][INFO] - ===> Epoch[3915](19575/5): Loss 0.0117, Sem 0.0117 LR: 7.009e-02 Score 99.578 Data time: 0.1863, Forward time: 1.2641, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7760 [2021-07-08 01:59:48,984][root][INFO] - ===> Epoch[3916](19580/5): Loss 0.0112, Sem 0.0112 LR: 7.008e-02 Score 99.595 Data time: 0.2324, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.9441 [2021-07-08 02:00:02,844][root][INFO] - ===> Epoch[3917](19585/5): Loss 0.0115, Sem 0.0115 LR: 7.007e-02 Score 99.584 Data time: 0.2148, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7338 [2021-07-08 02:00:17,222][root][INFO] - ===> Epoch[3918](19590/5): Loss 0.0114, Sem 0.0114 LR: 7.007e-02 Score 99.589 Data time: 0.2417, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8243 [2021-07-08 02:00:32,246][root][INFO] - ===> Epoch[3919](19595/5): Loss 0.0108, Sem 0.0108 LR: 7.006e-02 Score 99.609 Data time: 0.2571, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9555 [2021-07-08 02:00:47,251][root][INFO] - ===> Epoch[3920](19600/5): Loss 0.0111, Sem 0.0111 LR: 7.005e-02 Score 99.600 Data time: 0.2598, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9557 [2021-07-08 02:00:47,253][root][INFO] - ===> Start testing [2021-07-08 02:01:23,068][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1210 Loss 0.338 (AVG: 0.540) Score 92.996 (AVG: 91.418) mIOU 71.782 mAP 84.198 mAcc 77.881 IOU: 65.678 0.000 85.687 77.349 95.528 92.213 31.208 83.474 98.532 78.206 79.331 86.944 59.015 mAP: 83.904 5.408 95.300 88.825 98.897 98.581 57.560 98.176 99.771 89.853 92.251 94.589 91.461 mAcc: 82.077 0.000 91.923 83.526 96.779 97.281 33.953 97.285 99.500 80.929 90.297 96.113 62.793 [2021-07-08 02:01:23,069][root][INFO] - Finished test. Elapsed time: 35.8148 [2021-07-08 02:01:24,195][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 02:01:36,529][root][INFO] - ===> Epoch[3921](19605/5): Loss 0.0113, Sem 0.0113 LR: 7.004e-02 Score 99.593 Data time: 0.2517, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.4299 [2021-07-08 02:01:51,447][root][INFO] - ===> Epoch[3922](19610/5): Loss 0.0113, Sem 0.0113 LR: 7.003e-02 Score 99.594 Data time: 0.2070, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9419 [2021-07-08 02:02:05,503][root][INFO] - ===> Epoch[3923](19615/5): Loss 0.0119, Sem 0.0119 LR: 7.003e-02 Score 99.566 Data time: 0.2502, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7558 [2021-07-08 02:02:18,606][root][INFO] - ===> Epoch[3924](19620/5): Loss 0.0115, Sem 0.0115 LR: 7.002e-02 Score 99.587 Data time: 0.1731, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.5783 [2021-07-08 02:02:31,898][root][INFO] - ===> Epoch[3925](19625/5): Loss 0.0112, Sem 0.0112 LR: 7.001e-02 Score 99.594 Data time: 0.2143, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6098 [2021-07-08 02:02:46,117][root][INFO] - ===> Epoch[3926](19630/5): Loss 0.0111, Sem 0.0111 LR: 7.000e-02 Score 99.598 Data time: 0.2298, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7971 [2021-07-08 02:03:00,277][root][INFO] - ===> Epoch[3927](19635/5): Loss 0.0110, Sem 0.0110 LR: 7.000e-02 Score 99.598 Data time: 0.2597, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7896 [2021-07-08 02:03:14,282][root][INFO] - ===> Epoch[3928](19640/5): Loss 0.0113, Sem 0.0113 LR: 6.999e-02 Score 99.590 Data time: 0.2679, Forward time: 1.2641, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7562 [2021-07-08 02:03:28,858][root][INFO] - ===> Epoch[3929](19645/5): Loss 0.0108, Sem 0.0108 LR: 6.998e-02 Score 99.606 Data time: 0.2060, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8792 [2021-07-08 02:03:42,469][root][INFO] - ===> Epoch[3930](19650/5): Loss 0.0110, Sem 0.0110 LR: 6.997e-02 Score 99.603 Data time: 0.2236, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6741 [2021-07-08 02:03:56,512][root][INFO] - ===> Epoch[3931](19655/5): Loss 0.0114, Sem 0.0114 LR: 6.996e-02 Score 99.588 Data time: 0.1857, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7604 [2021-07-08 02:04:10,647][root][INFO] - ===> Epoch[3932](19660/5): Loss 0.0109, Sem 0.0109 LR: 6.996e-02 Score 99.609 Data time: 0.2385, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7722 [2021-07-08 02:04:25,409][root][INFO] - ===> Epoch[3933](19665/5): Loss 0.0113, Sem 0.0113 LR: 6.995e-02 Score 99.592 Data time: 0.2151, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9021 [2021-07-08 02:04:39,390][root][INFO] - ===> Epoch[3934](19670/5): Loss 0.0113, Sem 0.0113 LR: 6.994e-02 Score 99.593 Data time: 0.2451, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7419 [2021-07-08 02:04:53,049][root][INFO] - ===> Epoch[3935](19675/5): Loss 0.0115, Sem 0.0115 LR: 6.993e-02 Score 99.585 Data time: 0.2230, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6831 [2021-07-08 02:05:07,128][root][INFO] - ===> Epoch[3936](19680/5): Loss 0.0111, Sem 0.0111 LR: 6.993e-02 Score 99.607 Data time: 0.2400, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7698 [2021-07-08 02:05:21,664][root][INFO] - ===> Epoch[3937](19685/5): Loss 0.0109, Sem 0.0109 LR: 6.992e-02 Score 99.603 Data time: 0.2922, Forward time: 1.2641, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8597 [2021-07-08 02:05:35,788][root][INFO] - ===> Epoch[3938](19690/5): Loss 0.0113, Sem 0.0113 LR: 6.991e-02 Score 99.593 Data time: 0.1525, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7811 [2021-07-08 02:05:49,811][root][INFO] - ===> Epoch[3939](19695/5): Loss 0.0114, Sem 0.0114 LR: 6.990e-02 Score 99.585 Data time: 0.2429, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7549 [2021-07-08 02:06:03,510][root][INFO] - ===> Epoch[3940](19700/5): Loss 0.0114, Sem 0.0114 LR: 6.989e-02 Score 99.588 Data time: 0.2021, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6874 [2021-07-08 02:06:17,317][root][INFO] - ===> Epoch[3941](19705/5): Loss 0.0115, Sem 0.0115 LR: 6.989e-02 Score 99.587 Data time: 0.2249, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7182 [2021-07-08 02:06:31,783][root][INFO] - ===> Epoch[3942](19710/5): Loss 0.0111, Sem 0.0111 LR: 6.988e-02 Score 99.598 Data time: 0.2960, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8469 [2021-07-08 02:06:46,088][root][INFO] - ===> Epoch[3943](19715/5): Loss 0.0115, Sem 0.0115 LR: 6.987e-02 Score 99.584 Data time: 0.3638, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7983 [2021-07-08 02:07:00,584][root][INFO] - ===> Epoch[3944](19720/5): Loss 0.0109, Sem 0.0109 LR: 6.986e-02 Score 99.611 Data time: 0.1820, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8574 [2021-07-08 02:07:14,980][root][INFO] - ===> Epoch[3945](19725/5): Loss 0.0110, Sem 0.0110 LR: 6.986e-02 Score 99.607 Data time: 0.2016, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8345 [2021-07-08 02:07:29,162][root][INFO] - ===> Epoch[3946](19730/5): Loss 0.0112, Sem 0.0112 LR: 6.985e-02 Score 99.591 Data time: 0.2009, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7858 [2021-07-08 02:07:42,846][root][INFO] - ===> Epoch[3947](19735/5): Loss 0.0113, Sem 0.0113 LR: 6.984e-02 Score 99.592 Data time: 0.1694, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7026 [2021-07-08 02:07:56,894][root][INFO] - ===> Epoch[3948](19740/5): Loss 0.0108, Sem 0.0108 LR: 6.983e-02 Score 99.610 Data time: 0.2544, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7625 [2021-07-08 02:08:11,491][root][INFO] - ===> Epoch[3949](19745/5): Loss 0.0108, Sem 0.0108 LR: 6.982e-02 Score 99.612 Data time: 0.2403, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8735 [2021-07-08 02:08:24,690][root][INFO] - ===> Epoch[3950](19750/5): Loss 0.0112, Sem 0.0112 LR: 6.982e-02 Score 99.595 Data time: 0.2199, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.5973 [2021-07-08 02:08:39,039][root][INFO] - ===> Epoch[3951](19755/5): Loss 0.0105, Sem 0.0105 LR: 6.981e-02 Score 99.621 Data time: 0.2253, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8185 [2021-07-08 02:08:53,870][root][INFO] - ===> Epoch[3952](19760/5): Loss 0.0108, Sem 0.0108 LR: 6.980e-02 Score 99.607 Data time: 0.2526, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.9184 [2021-07-08 02:09:08,290][root][INFO] - ===> Epoch[3953](19765/5): Loss 0.0110, Sem 0.0110 LR: 6.979e-02 Score 99.601 Data time: 0.1987, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8390 [2021-07-08 02:09:22,876][root][INFO] - ===> Epoch[3954](19770/5): Loss 0.0106, Sem 0.0106 LR: 6.979e-02 Score 99.620 Data time: 0.2963, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8685 [2021-07-08 02:09:36,964][root][INFO] - ===> Epoch[3955](19775/5): Loss 0.0108, Sem 0.0108 LR: 6.978e-02 Score 99.612 Data time: 0.2708, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7628 [2021-07-08 02:09:51,538][root][INFO] - ===> Epoch[3956](19780/5): Loss 0.0110, Sem 0.0110 LR: 6.977e-02 Score 99.602 Data time: 0.1734, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8742 [2021-07-08 02:10:05,606][root][INFO] - ===> Epoch[3957](19785/5): Loss 0.0112, Sem 0.0112 LR: 6.976e-02 Score 99.592 Data time: 0.2059, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7633 [2021-07-08 02:10:18,845][root][INFO] - ===> Epoch[3958](19790/5): Loss 0.0114, Sem 0.0114 LR: 6.975e-02 Score 99.589 Data time: 0.2343, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6041 [2021-07-08 02:10:33,217][root][INFO] - ===> Epoch[3959](19795/5): Loss 0.0110, Sem 0.0110 LR: 6.975e-02 Score 99.599 Data time: 0.2004, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8260 [2021-07-08 02:10:47,340][root][INFO] - ===> Epoch[3960](19800/5): Loss 0.0113, Sem 0.0113 LR: 6.974e-02 Score 99.589 Data time: 0.2265, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7800 [2021-07-08 02:10:47,341][root][INFO] - ===> Start testing [2021-07-08 02:11:21,010][root][INFO] - 68/68: Data time: 0.0017, Iter time: 0.1213 Loss 0.397 (AVG: 0.569) Score 91.239 (AVG: 91.229) mIOU 70.620 mAP 82.767 mAcc 76.652 IOU: 65.746 0.000 83.781 76.767 95.460 92.447 19.384 82.589 98.620 77.311 80.874 86.324 58.751 mAP: 84.227 4.377 93.925 88.833 98.807 98.633 44.894 98.005 99.744 88.511 92.253 93.390 90.369 mAcc: 81.763 0.000 90.417 84.262 96.804 97.005 20.157 96.226 99.592 80.781 89.923 96.225 63.323 [2021-07-08 02:11:21,010][root][INFO] - Finished test. Elapsed time: 33.6684 [2021-07-08 02:11:22,114][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 02:11:35,019][root][INFO] - ===> Epoch[3961](19805/5): Loss 0.0111, Sem 0.0111 LR: 6.973e-02 Score 99.597 Data time: 0.2275, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.5255 [2021-07-08 02:11:49,941][root][INFO] - ===> Epoch[3962](19810/5): Loss 0.0116, Sem 0.0116 LR: 6.972e-02 Score 99.583 Data time: 0.2608, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9290 [2021-07-08 02:12:04,151][root][INFO] - ===> Epoch[3963](19815/5): Loss 0.0113, Sem 0.0113 LR: 6.971e-02 Score 99.595 Data time: 0.2322, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7919 [2021-07-08 02:12:17,833][root][INFO] - ===> Epoch[3964](19820/5): Loss 0.0110, Sem 0.0110 LR: 6.971e-02 Score 99.602 Data time: 0.2742, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6973 [2021-07-08 02:12:31,970][root][INFO] - ===> Epoch[3965](19825/5): Loss 0.0110, Sem 0.0110 LR: 6.970e-02 Score 99.601 Data time: 0.2824, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7757 [2021-07-08 02:12:45,850][root][INFO] - ===> Epoch[3966](19830/5): Loss 0.0115, Sem 0.0115 LR: 6.969e-02 Score 99.581 Data time: 0.1982, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7186 [2021-07-08 02:12:59,665][root][INFO] - ===> Epoch[3967](19835/5): Loss 0.0110, Sem 0.0110 LR: 6.968e-02 Score 99.605 Data time: 0.2770, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7106 [2021-07-08 02:13:13,913][root][INFO] - ===> Epoch[3968](19840/5): Loss 0.0110, Sem 0.0110 LR: 6.968e-02 Score 99.602 Data time: 0.1958, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8021 [2021-07-08 02:13:28,973][root][INFO] - ===> Epoch[3969](19845/5): Loss 0.0109, Sem 0.0109 LR: 6.967e-02 Score 99.607 Data time: 0.2802, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9618 [2021-07-08 02:13:42,606][root][INFO] - ===> Epoch[3970](19850/5): Loss 0.0111, Sem 0.0111 LR: 6.966e-02 Score 99.599 Data time: 0.3156, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6892 [2021-07-08 02:13:56,400][root][INFO] - ===> Epoch[3971](19855/5): Loss 0.0110, Sem 0.0110 LR: 6.965e-02 Score 99.602 Data time: 0.2700, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7115 [2021-07-08 02:14:10,625][root][INFO] - ===> Epoch[3972](19860/5): Loss 0.0110, Sem 0.0110 LR: 6.964e-02 Score 99.603 Data time: 0.2153, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8050 [2021-07-08 02:14:24,273][root][INFO] - ===> Epoch[3973](19865/5): Loss 0.0119, Sem 0.0119 LR: 6.964e-02 Score 99.572 Data time: 0.2128, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6861 [2021-07-08 02:14:39,988][root][INFO] - ===> Epoch[3974](19870/5): Loss 0.0113, Sem 0.0113 LR: 6.963e-02 Score 99.592 Data time: 0.2419, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 3.0962 [2021-07-08 02:14:53,815][root][INFO] - ===> Epoch[3975](19875/5): Loss 0.0107, Sem 0.0107 LR: 6.962e-02 Score 99.613 Data time: 0.2690, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7108 [2021-07-08 02:15:07,002][root][INFO] - ===> Epoch[3976](19880/5): Loss 0.0110, Sem 0.0110 LR: 6.961e-02 Score 99.601 Data time: 0.1359, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.5970 [2021-07-08 02:15:20,231][root][INFO] - ===> Epoch[3977](19885/5): Loss 0.0108, Sem 0.0108 LR: 6.961e-02 Score 99.608 Data time: 0.2344, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6018 [2021-07-08 02:15:34,079][root][INFO] - ===> Epoch[3978](19890/5): Loss 0.0111, Sem 0.0111 LR: 6.960e-02 Score 99.598 Data time: 0.2223, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7133 [2021-07-08 02:15:48,827][root][INFO] - ===> Epoch[3979](19895/5): Loss 0.0111, Sem 0.0111 LR: 6.959e-02 Score 99.595 Data time: 0.2418, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.9096 [2021-07-08 02:16:02,312][root][INFO] - ===> Epoch[3980](19900/5): Loss 0.0110, Sem 0.0110 LR: 6.958e-02 Score 99.602 Data time: 0.2068, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6531 [2021-07-08 02:16:16,595][root][INFO] - ===> Epoch[3981](19905/5): Loss 0.0106, Sem 0.0106 LR: 6.957e-02 Score 99.618 Data time: 0.2206, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8168 [2021-07-08 02:16:31,278][root][INFO] - ===> Epoch[3982](19910/5): Loss 0.0108, Sem 0.0108 LR: 6.957e-02 Score 99.605 Data time: 0.1882, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8891 [2021-07-08 02:16:46,011][root][INFO] - ===> Epoch[3983](19915/5): Loss 0.0106, Sem 0.0106 LR: 6.956e-02 Score 99.616 Data time: 0.2336, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8915 [2021-07-08 02:16:59,866][root][INFO] - ===> Epoch[3984](19920/5): Loss 0.0115, Sem 0.0115 LR: 6.955e-02 Score 99.592 Data time: 0.2221, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7231 [2021-07-08 02:17:14,158][root][INFO] - ===> Epoch[3985](19925/5): Loss 0.0110, Sem 0.0110 LR: 6.954e-02 Score 99.604 Data time: 0.2489, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8080 [2021-07-08 02:17:28,711][root][INFO] - ===> Epoch[3986](19930/5): Loss 0.0106, Sem 0.0106 LR: 6.954e-02 Score 99.611 Data time: 0.2762, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8571 [2021-07-08 02:17:42,729][root][INFO] - ===> Epoch[3987](19935/5): Loss 0.0114, Sem 0.0114 LR: 6.953e-02 Score 99.591 Data time: 0.3183, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7506 [2021-07-08 02:17:56,906][root][INFO] - ===> Epoch[3988](19940/5): Loss 0.0111, Sem 0.0111 LR: 6.952e-02 Score 99.601 Data time: 0.2787, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7975 [2021-07-08 02:18:10,998][root][INFO] - ===> Epoch[3989](19945/5): Loss 0.0113, Sem 0.0113 LR: 6.951e-02 Score 99.592 Data time: 0.2711, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7748 [2021-07-08 02:18:25,277][root][INFO] - ===> Epoch[3990](19950/5): Loss 0.0111, Sem 0.0111 LR: 6.950e-02 Score 99.596 Data time: 0.2298, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8090 [2021-07-08 02:18:39,541][root][INFO] - ===> Epoch[3991](19955/5): Loss 0.0110, Sem 0.0110 LR: 6.950e-02 Score 99.605 Data time: 0.1892, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8092 [2021-07-08 02:18:53,001][root][INFO] - ===> Epoch[3992](19960/5): Loss 0.0115, Sem 0.0115 LR: 6.949e-02 Score 99.585 Data time: 0.2669, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6408 [2021-07-08 02:19:07,468][root][INFO] - ===> Epoch[3993](19965/5): Loss 0.0107, Sem 0.0107 LR: 6.948e-02 Score 99.613 Data time: 0.1994, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8443 [2021-07-08 02:19:21,147][root][INFO] - ===> Epoch[3994](19970/5): Loss 0.0110, Sem 0.0110 LR: 6.947e-02 Score 99.604 Data time: 0.2449, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6895 [2021-07-08 02:19:35,100][root][INFO] - ===> Epoch[3995](19975/5): Loss 0.0114, Sem 0.0114 LR: 6.946e-02 Score 99.595 Data time: 0.2482, Forward time: 1.2643, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7495 [2021-07-08 02:19:48,989][root][INFO] - ===> Epoch[3996](19980/5): Loss 0.0111, Sem 0.0111 LR: 6.946e-02 Score 99.600 Data time: 0.2174, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7356 [2021-07-08 02:20:02,579][root][INFO] - ===> Epoch[3997](19985/5): Loss 0.0105, Sem 0.0105 LR: 6.945e-02 Score 99.617 Data time: 0.2340, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6785 [2021-07-08 02:20:16,251][root][INFO] - ===> Epoch[3998](19990/5): Loss 0.0116, Sem 0.0116 LR: 6.944e-02 Score 99.581 Data time: 0.1899, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6982 [2021-07-08 02:20:30,061][root][INFO] - ===> Epoch[3999](19995/5): Loss 0.0116, Sem 0.0116 LR: 6.943e-02 Score 99.581 Data time: 0.1887, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7145 [2021-07-08 02:20:44,027][root][INFO] - ===> Epoch[4000](20000/5): Loss 0.0110, Sem 0.0110 LR: 6.943e-02 Score 99.602 Data time: 0.2689, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7438 [2021-07-08 02:20:44,028][root][INFO] - ===> Start testing [2021-07-08 02:21:19,438][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1268 Loss 0.344 (AVG: 0.556) Score 92.206 (AVG: 91.390) mIOU 71.974 mAP 83.698 mAcc 78.046 IOU: 65.560 0.000 86.675 76.810 95.338 91.130 37.184 85.739 98.630 75.735 79.857 86.867 56.138 mAP: 84.130 7.334 94.523 88.167 98.808 98.528 59.637 98.142 99.801 82.918 92.432 93.927 89.724 mAcc: 80.634 0.000 91.937 83.961 96.758 97.188 41.746 96.516 99.467 79.509 91.143 96.325 59.412 [2021-07-08 02:21:19,438][root][INFO] - Finished test. Elapsed time: 35.4094 [2021-07-08 02:21:20,483][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 02:21:34,188][root][INFO] - ===> Epoch[4001](20005/5): Loss 0.0110, Sem 0.0110 LR: 6.942e-02 Score 99.603 Data time: 0.1928, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7087 [2021-07-08 02:21:48,674][root][INFO] - ===> Epoch[4002](20010/5): Loss 0.0111, Sem 0.0111 LR: 6.941e-02 Score 99.592 Data time: 0.2321, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8504 [2021-07-08 02:22:02,914][root][INFO] - ===> Epoch[4003](20015/5): Loss 0.0117, Sem 0.0117 LR: 6.940e-02 Score 99.578 Data time: 0.2371, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7957 [2021-07-08 02:22:17,251][root][INFO] - ===> Epoch[4004](20020/5): Loss 0.0111, Sem 0.0111 LR: 6.939e-02 Score 99.596 Data time: 0.2311, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8156 [2021-07-08 02:22:30,125][root][INFO] - ===> Epoch[4005](20025/5): Loss 0.0117, Sem 0.0117 LR: 6.939e-02 Score 99.578 Data time: 0.3071, Forward time: 1.2642, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.5239 [2021-07-08 02:22:43,668][root][INFO] - ===> Epoch[4006](20030/5): Loss 0.0107, Sem 0.0107 LR: 6.938e-02 Score 99.610 Data time: 0.2372, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6684 [2021-07-08 02:22:58,312][root][INFO] - ===> Epoch[4007](20035/5): Loss 0.0111, Sem 0.0111 LR: 6.937e-02 Score 99.598 Data time: 0.1993, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8860 [2021-07-08 02:23:12,525][root][INFO] - ===> Epoch[4008](20040/5): Loss 0.0111, Sem 0.0111 LR: 6.936e-02 Score 99.599 Data time: 0.2614, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7862 [2021-07-08 02:23:26,177][root][INFO] - ===> Epoch[4009](20045/5): Loss 0.0106, Sem 0.0106 LR: 6.936e-02 Score 99.614 Data time: 0.2490, Forward time: 1.2642, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6876 [2021-07-08 02:23:40,536][root][INFO] - ===> Epoch[4010](20050/5): Loss 0.0108, Sem 0.0108 LR: 6.935e-02 Score 99.608 Data time: 0.2998, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8187 [2021-07-08 02:23:54,608][root][INFO] - ===> Epoch[4011](20055/5): Loss 0.0105, Sem 0.0105 LR: 6.934e-02 Score 99.618 Data time: 0.3273, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7586 [2021-07-08 02:24:08,253][root][INFO] - ===> Epoch[4012](20060/5): Loss 0.0107, Sem 0.0107 LR: 6.933e-02 Score 99.609 Data time: 0.2091, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6893 [2021-07-08 02:24:22,021][root][INFO] - ===> Epoch[4013](20065/5): Loss 0.0112, Sem 0.0112 LR: 6.932e-02 Score 99.599 Data time: 0.2180, Forward time: 1.2642, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7166 [2021-07-08 02:24:36,350][root][INFO] - ===> Epoch[4014](20070/5): Loss 0.0112, Sem 0.0112 LR: 6.932e-02 Score 99.594 Data time: 0.2389, Forward time: 1.2642, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8270 [2021-07-08 02:24:51,413][root][INFO] - ===> Epoch[4015](20075/5): Loss 0.0111, Sem 0.0111 LR: 6.931e-02 Score 99.599 Data time: 0.2651, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.9618 [2021-07-08 02:25:05,434][root][INFO] - ===> Epoch[4016](20080/5): Loss 0.0116, Sem 0.0116 LR: 6.930e-02 Score 99.581 Data time: 0.1816, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7599 [2021-07-08 02:25:19,433][root][INFO] - ===> Epoch[4017](20085/5): Loss 0.0113, Sem 0.0113 LR: 6.929e-02 Score 99.595 Data time: 0.1554, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7595 [2021-07-08 02:25:34,103][root][INFO] - ===> Epoch[4018](20090/5): Loss 0.0120, Sem 0.0120 LR: 6.929e-02 Score 99.577 Data time: 0.2222, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8908 [2021-07-08 02:25:47,863][root][INFO] - ===> Epoch[4019](20095/5): Loss 0.0117, Sem 0.0117 LR: 6.928e-02 Score 99.578 Data time: 0.2522, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7169 [2021-07-08 02:26:01,349][root][INFO] - ===> Epoch[4020](20100/5): Loss 0.0118, Sem 0.0118 LR: 6.927e-02 Score 99.585 Data time: 0.1911, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6581 [2021-07-08 02:26:14,828][root][INFO] - ===> Epoch[4021](20105/5): Loss 0.0122, Sem 0.0122 LR: 6.926e-02 Score 99.561 Data time: 0.1939, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6476 [2021-07-08 02:26:29,259][root][INFO] - ===> Epoch[4022](20110/5): Loss 0.0116, Sem 0.0116 LR: 6.925e-02 Score 99.584 Data time: 0.2671, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8474 [2021-07-08 02:26:43,307][root][INFO] - ===> Epoch[4023](20115/5): Loss 0.0110, Sem 0.0110 LR: 6.925e-02 Score 99.603 Data time: 0.2046, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7613 [2021-07-08 02:26:57,349][root][INFO] - ===> Epoch[4024](20120/5): Loss 0.0115, Sem 0.0115 LR: 6.924e-02 Score 99.584 Data time: 0.2331, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7583 [2021-07-08 02:27:11,467][root][INFO] - ===> Epoch[4025](20125/5): Loss 0.0117, Sem 0.0117 LR: 6.923e-02 Score 99.581 Data time: 0.2223, Forward time: 1.2643, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7714 [2021-07-08 02:27:25,100][root][INFO] - ===> Epoch[4026](20130/5): Loss 0.0111, Sem 0.0111 LR: 6.922e-02 Score 99.600 Data time: 0.2684, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6834 [2021-07-08 02:27:39,663][root][INFO] - ===> Epoch[4027](20135/5): Loss 0.0110, Sem 0.0110 LR: 6.921e-02 Score 99.597 Data time: 0.2717, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8594 [2021-07-08 02:27:54,177][root][INFO] - ===> Epoch[4028](20140/5): Loss 0.0109, Sem 0.0109 LR: 6.921e-02 Score 99.607 Data time: 0.2638, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8595 [2021-07-08 02:28:08,221][root][INFO] - ===> Epoch[4029](20145/5): Loss 0.0111, Sem 0.0111 LR: 6.920e-02 Score 99.600 Data time: 0.2362, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7634 [2021-07-08 02:28:21,882][root][INFO] - ===> Epoch[4030](20150/5): Loss 0.0108, Sem 0.0108 LR: 6.919e-02 Score 99.607 Data time: 0.2250, Forward time: 1.2644, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6837 [2021-07-08 02:28:35,456][root][INFO] - ===> Epoch[4031](20155/5): Loss 0.0118, Sem 0.0118 LR: 6.918e-02 Score 99.575 Data time: 0.2680, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6656 [2021-07-08 02:28:50,263][root][INFO] - ===> Epoch[4032](20160/5): Loss 0.0109, Sem 0.0109 LR: 6.918e-02 Score 99.601 Data time: 0.3094, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.9121 [2021-07-08 02:29:04,550][root][INFO] - ===> Epoch[4033](20165/5): Loss 0.0106, Sem 0.0106 LR: 6.917e-02 Score 99.616 Data time: 0.3032, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8116 [2021-07-08 02:29:19,270][root][INFO] - ===> Epoch[4034](20170/5): Loss 0.0108, Sem 0.0108 LR: 6.916e-02 Score 99.608 Data time: 0.1802, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8990 [2021-07-08 02:29:33,490][root][INFO] - ===> Epoch[4035](20175/5): Loss 0.0115, Sem 0.0115 LR: 6.915e-02 Score 99.585 Data time: 0.3020, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7901 [2021-07-08 02:29:47,106][root][INFO] - ===> Epoch[4036](20180/5): Loss 0.0111, Sem 0.0111 LR: 6.914e-02 Score 99.594 Data time: 0.2683, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6678 [2021-07-08 02:30:01,498][root][INFO] - ===> Epoch[4037](20185/5): Loss 0.0106, Sem 0.0106 LR: 6.914e-02 Score 99.617 Data time: 0.1733, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8287 [2021-07-08 02:30:15,877][root][INFO] - ===> Epoch[4038](20190/5): Loss 0.0106, Sem 0.0106 LR: 6.913e-02 Score 99.617 Data time: 0.2592, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8314 [2021-07-08 02:30:30,297][root][INFO] - ===> Epoch[4039](20195/5): Loss 0.0103, Sem 0.0103 LR: 6.912e-02 Score 99.629 Data time: 0.2454, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8203 [2021-07-08 02:30:44,366][root][INFO] - ===> Epoch[4040](20200/5): Loss 0.0109, Sem 0.0109 LR: 6.911e-02 Score 99.609 Data time: 0.2195, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7699 [2021-07-08 02:30:44,368][root][INFO] - ===> Start testing [2021-07-08 02:31:19,127][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1200 Loss 0.276 (AVG: 0.554) Score 93.314 (AVG: 91.430) mIOU 71.351 mAP 83.503 mAcc 77.272 IOU: 66.812 0.000 85.914 77.006 95.783 92.125 27.843 81.704 98.498 76.941 79.850 86.691 58.393 mAP: 83.910 6.444 94.949 87.983 98.806 98.494 55.660 97.850 99.779 84.947 91.830 93.705 91.184 mAcc: 82.210 0.000 91.281 83.751 97.130 96.914 30.258 95.496 99.345 79.893 90.006 96.554 61.696 [2021-07-08 02:31:19,127][root][INFO] - Finished test. Elapsed time: 34.7583 [2021-07-08 02:31:20,268][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 02:31:33,158][root][INFO] - ===> Epoch[4041](20205/5): Loss 0.0106, Sem 0.0106 LR: 6.911e-02 Score 99.617 Data time: 0.1489, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.5489 [2021-07-08 02:31:47,854][root][INFO] - ===> Epoch[4042](20210/5): Loss 0.0107, Sem 0.0107 LR: 6.910e-02 Score 99.610 Data time: 0.2182, Forward time: 1.2643, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8898 [2021-07-08 02:32:02,063][root][INFO] - ===> Epoch[4043](20215/5): Loss 0.0106, Sem 0.0106 LR: 6.909e-02 Score 99.618 Data time: 0.3597, Forward time: 1.2644, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7900 [2021-07-08 02:32:16,966][root][INFO] - ===> Epoch[4044](20220/5): Loss 0.0106, Sem 0.0106 LR: 6.908e-02 Score 99.616 Data time: 0.2531, Forward time: 1.2644, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.9251 [2021-07-08 02:32:30,700][root][INFO] - ===> Epoch[4045](20225/5): Loss 0.0109, Sem 0.0109 LR: 6.907e-02 Score 99.610 Data time: 0.2705, Forward time: 1.2644, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7048 [2021-07-08 02:32:45,003][root][INFO] - ===> Epoch[4046](20230/5): Loss 0.0112, Sem 0.0112 LR: 6.907e-02 Score 99.593 Data time: 0.3149, Forward time: 1.2644, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7984 [2021-07-08 02:32:58,141][root][INFO] - ===> Epoch[4047](20235/5): Loss 0.0114, Sem 0.0114 LR: 6.906e-02 Score 99.587 Data time: 0.2630, Forward time: 1.2644, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.5820 [2021-07-08 02:33:12,728][root][INFO] - ===> Epoch[4048](20240/5): Loss 0.0109, Sem 0.0109 LR: 6.905e-02 Score 99.604 Data time: 0.2831, Forward time: 1.2644, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.8787 [2021-07-08 02:33:26,696][root][INFO] - ===> Epoch[4049](20245/5): Loss 0.0111, Sem 0.0111 LR: 6.904e-02 Score 99.597 Data time: 0.2443, Forward time: 1.2644, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7440 [2021-07-08 02:33:40,140][root][INFO] - ===> Epoch[4050](20250/5): Loss 0.0111, Sem 0.0111 LR: 6.904e-02 Score 99.601 Data time: 0.2248, Forward time: 1.2644, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.6355 [2021-07-08 02:33:53,393][root][INFO] - ===> Epoch[4051](20255/5): Loss 0.0117, Sem 0.0117 LR: 6.903e-02 Score 99.592 Data time: 0.1999, Forward time: 1.2644, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6058 [2021-07-08 02:34:08,368][root][INFO] - ===> Epoch[4052](20260/5): Loss 0.0109, Sem 0.0109 LR: 6.902e-02 Score 99.607 Data time: 0.2063, Forward time: 1.2644, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.9498 [2021-07-08 02:34:22,840][root][INFO] - ===> Epoch[4053](20265/5): Loss 0.0112, Sem 0.0112 LR: 6.901e-02 Score 99.598 Data time: 0.2348, Forward time: 1.2644, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.8418 [2021-07-08 02:34:36,833][root][INFO] - ===> Epoch[4054](20270/5): Loss 0.0111, Sem 0.0111 LR: 6.900e-02 Score 99.606 Data time: 0.2824, Forward time: 1.2644, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7514 [2021-07-08 02:34:50,542][root][INFO] - ===> Epoch[4055](20275/5): Loss 0.0117, Sem 0.0117 LR: 6.900e-02 Score 99.578 Data time: 0.1892, Forward time: 1.2644, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.6984 [2021-07-08 02:35:04,902][root][INFO] - ===> Epoch[4056](20280/5): Loss 0.0111, Sem 0.0111 LR: 6.899e-02 Score 99.600 Data time: 0.2370, Forward time: 1.2644, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.8329 [2021-07-08 02:35:18,729][root][INFO] - ===> Epoch[4057](20285/5): Loss 0.0103, Sem 0.0103 LR: 6.898e-02 Score 99.628 Data time: 0.2796, Forward time: 1.2644, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7068 [2021-07-08 02:35:31,902][root][INFO] - ===> Epoch[4058](20290/5): Loss 0.0115, Sem 0.0115 LR: 6.897e-02 Score 99.581 Data time: 0.2486, Forward time: 1.2644, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.5926 [2021-07-08 02:35:46,257][root][INFO] - ===> Epoch[4059](20295/5): Loss 0.0107, Sem 0.0107 LR: 6.896e-02 Score 99.610 Data time: 0.2140, Forward time: 1.2644, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8197 [2021-07-08 02:36:00,243][root][INFO] - ===> Epoch[4060](20300/5): Loss 0.0110, Sem 0.0110 LR: 6.896e-02 Score 99.599 Data time: 0.1767, Forward time: 1.2644, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7480 [2021-07-08 02:36:14,908][root][INFO] - ===> Epoch[4061](20305/5): Loss 0.0106, Sem 0.0106 LR: 6.895e-02 Score 99.617 Data time: 0.3054, Forward time: 1.2644, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8855 [2021-07-08 02:36:28,803][root][INFO] - ===> Epoch[4062](20310/5): Loss 0.0108, Sem 0.0108 LR: 6.894e-02 Score 99.610 Data time: 0.2321, Forward time: 1.2644, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7289 [2021-07-08 02:36:42,959][root][INFO] - ===> Epoch[4063](20315/5): Loss 0.0105, Sem 0.0105 LR: 6.893e-02 Score 99.618 Data time: 0.2706, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7789 [2021-07-08 02:36:56,735][root][INFO] - ===> Epoch[4064](20320/5): Loss 0.0105, Sem 0.0105 LR: 6.893e-02 Score 99.620 Data time: 0.2437, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6992 [2021-07-08 02:37:12,394][root][INFO] - ===> Epoch[4065](20325/5): Loss 0.0107, Sem 0.0107 LR: 6.892e-02 Score 99.616 Data time: 0.1973, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 3.0779 [2021-07-08 02:37:26,043][root][INFO] - ===> Epoch[4066](20330/5): Loss 0.0110, Sem 0.0110 LR: 6.891e-02 Score 99.604 Data time: 0.2517, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6798 [2021-07-08 02:37:39,485][root][INFO] - ===> Epoch[4067](20335/5): Loss 0.0111, Sem 0.0111 LR: 6.890e-02 Score 99.601 Data time: 0.1737, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6509 [2021-07-08 02:37:53,566][root][INFO] - ===> Epoch[4068](20340/5): Loss 0.0107, Sem 0.0107 LR: 6.889e-02 Score 99.614 Data time: 0.2060, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7754 [2021-07-08 02:38:06,793][root][INFO] - ===> Epoch[4069](20345/5): Loss 0.0108, Sem 0.0108 LR: 6.889e-02 Score 99.608 Data time: 0.2133, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.5998 [2021-07-08 02:38:20,480][root][INFO] - ===> Epoch[4070](20350/5): Loss 0.0105, Sem 0.0105 LR: 6.888e-02 Score 99.618 Data time: 0.2217, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6905 [2021-07-08 02:38:34,724][root][INFO] - ===> Epoch[4071](20355/5): Loss 0.0105, Sem 0.0105 LR: 6.887e-02 Score 99.619 Data time: 0.2393, Forward time: 1.2645, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7993 [2021-07-08 02:38:48,609][root][INFO] - ===> Epoch[4072](20360/5): Loss 0.0106, Sem 0.0106 LR: 6.886e-02 Score 99.621 Data time: 0.2057, Forward time: 1.2645, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7388 [2021-07-08 02:39:02,673][root][INFO] - ===> Epoch[4073](20365/5): Loss 0.0111, Sem 0.0111 LR: 6.886e-02 Score 99.595 Data time: 0.2489, Forward time: 1.2645, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7723 [2021-07-08 02:39:16,937][root][INFO] - ===> Epoch[4074](20370/5): Loss 0.0112, Sem 0.0112 LR: 6.885e-02 Score 99.600 Data time: 0.2807, Forward time: 1.2645, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8101 [2021-07-08 02:39:31,402][root][INFO] - ===> Epoch[4075](20375/5): Loss 0.0107, Sem 0.0107 LR: 6.884e-02 Score 99.611 Data time: 0.2306, Forward time: 1.2645, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8383 [2021-07-08 02:39:45,528][root][INFO] - ===> Epoch[4076](20380/5): Loss 0.0105, Sem 0.0105 LR: 6.883e-02 Score 99.621 Data time: 0.1657, Forward time: 1.2645, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7862 [2021-07-08 02:40:00,674][root][INFO] - ===> Epoch[4077](20385/5): Loss 0.0104, Sem 0.0104 LR: 6.882e-02 Score 99.624 Data time: 0.2438, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.9881 [2021-07-08 02:40:14,754][root][INFO] - ===> Epoch[4078](20390/5): Loss 0.0109, Sem 0.0109 LR: 6.882e-02 Score 99.607 Data time: 0.2399, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7666 [2021-07-08 02:40:29,125][root][INFO] - ===> Epoch[4079](20395/5): Loss 0.0112, Sem 0.0112 LR: 6.881e-02 Score 99.595 Data time: 0.2071, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8291 [2021-07-08 02:40:43,193][root][INFO] - ===> Epoch[4080](20400/5): Loss 0.0100, Sem 0.0100 LR: 6.880e-02 Score 99.638 Data time: 0.2577, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7424 [2021-07-08 02:40:43,194][root][INFO] - ===> Start testing [2021-07-08 02:41:16,970][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1258 Loss 0.367 (AVG: 0.552) Score 91.699 (AVG: 91.369) mIOU 71.360 mAP 83.987 mAcc 77.500 IOU: 66.543 0.000 83.905 76.607 95.522 92.600 28.435 82.235 98.561 78.822 80.596 86.661 57.185 mAP: 84.338 6.287 94.571 88.343 98.650 98.668 57.373 97.857 99.774 88.231 92.494 94.021 91.227 mAcc: 80.473 0.000 91.556 84.300 96.764 97.547 30.592 96.720 99.348 81.699 91.248 96.698 60.552 [2021-07-08 02:41:16,970][root][INFO] - Finished test. Elapsed time: 33.7753 [2021-07-08 02:41:18,136][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 02:41:31,373][root][INFO] - ===> Epoch[4081](20405/5): Loss 0.0114, Sem 0.0114 LR: 6.879e-02 Score 99.583 Data time: 0.1648, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6131 [2021-07-08 02:41:45,287][root][INFO] - ===> Epoch[4082](20410/5): Loss 0.0105, Sem 0.0105 LR: 6.879e-02 Score 99.617 Data time: 0.1641, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7399 [2021-07-08 02:41:59,677][root][INFO] - ===> Epoch[4083](20415/5): Loss 0.0107, Sem 0.0107 LR: 6.878e-02 Score 99.611 Data time: 0.2054, Forward time: 1.2645, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.8356 [2021-07-08 02:42:13,463][root][INFO] - ===> Epoch[4084](20420/5): Loss 0.0107, Sem 0.0107 LR: 6.877e-02 Score 99.611 Data time: 0.2072, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7085 [2021-07-08 02:42:28,379][root][INFO] - ===> Epoch[4085](20425/5): Loss 0.0107, Sem 0.0107 LR: 6.876e-02 Score 99.613 Data time: 0.3072, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.9273 [2021-07-08 02:42:41,772][root][INFO] - ===> Epoch[4086](20430/5): Loss 0.0107, Sem 0.0107 LR: 6.875e-02 Score 99.609 Data time: 0.2204, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6389 [2021-07-08 02:42:57,090][root][INFO] - ===> Epoch[4087](20435/5): Loss 0.0105, Sem 0.0105 LR: 6.875e-02 Score 99.621 Data time: 0.2805, Forward time: 1.2646, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 3.0157 [2021-07-08 02:43:10,837][root][INFO] - ===> Epoch[4088](20440/5): Loss 0.0106, Sem 0.0106 LR: 6.874e-02 Score 99.614 Data time: 0.2394, Forward time: 1.2646, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7105 [2021-07-08 02:43:24,968][root][INFO] - ===> Epoch[4089](20445/5): Loss 0.0107, Sem 0.0107 LR: 6.873e-02 Score 99.611 Data time: 0.2794, Forward time: 1.2646, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7750 [2021-07-08 02:43:38,456][root][INFO] - ===> Epoch[4090](20450/5): Loss 0.0104, Sem 0.0104 LR: 6.872e-02 Score 99.623 Data time: 0.2676, Forward time: 1.2646, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6553 [2021-07-08 02:43:52,478][root][INFO] - ===> Epoch[4091](20455/5): Loss 0.0107, Sem 0.0107 LR: 6.871e-02 Score 99.616 Data time: 0.2408, Forward time: 1.2646, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7592 [2021-07-08 02:44:06,643][root][INFO] - ===> Epoch[4092](20460/5): Loss 0.0106, Sem 0.0106 LR: 6.871e-02 Score 99.617 Data time: 0.2561, Forward time: 1.2646, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7869 [2021-07-08 02:44:20,257][root][INFO] - ===> Epoch[4093](20465/5): Loss 0.0112, Sem 0.0112 LR: 6.870e-02 Score 99.594 Data time: 0.2245, Forward time: 1.2646, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6739 [2021-07-08 02:44:34,893][root][INFO] - ===> Epoch[4094](20470/5): Loss 0.0108, Sem 0.0108 LR: 6.869e-02 Score 99.610 Data time: 0.2634, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8738 [2021-07-08 02:44:48,607][root][INFO] - ===> Epoch[4095](20475/5): Loss 0.0108, Sem 0.0108 LR: 6.868e-02 Score 99.613 Data time: 0.2291, Forward time: 1.2645, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6927 [2021-07-08 02:45:02,843][root][INFO] - ===> Epoch[4096](20480/5): Loss 0.0105, Sem 0.0105 LR: 6.868e-02 Score 99.630 Data time: 0.2350, Forward time: 1.2646, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8002 [2021-07-08 02:45:16,799][root][INFO] - ===> Epoch[4097](20485/5): Loss 0.0102, Sem 0.0102 LR: 6.867e-02 Score 99.632 Data time: 0.2147, Forward time: 1.2646, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7500 [2021-07-08 02:45:32,126][root][INFO] - ===> Epoch[4098](20490/5): Loss 0.0110, Sem 0.0110 LR: 6.866e-02 Score 99.601 Data time: 0.1979, Forward time: 1.2646, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 3.0245 [2021-07-08 02:45:46,297][root][INFO] - ===> Epoch[4099](20495/5): Loss 0.0107, Sem 0.0107 LR: 6.865e-02 Score 99.612 Data time: 0.2467, Forward time: 1.2646, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7849 [2021-07-08 02:46:00,398][root][INFO] - ===> Epoch[4100](20500/5): Loss 0.0105, Sem 0.0105 LR: 6.864e-02 Score 99.618 Data time: 0.2744, Forward time: 1.2646, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7593 [2021-07-08 02:46:15,289][root][INFO] - ===> Epoch[4101](20505/5): Loss 0.0109, Sem 0.0109 LR: 6.864e-02 Score 99.603 Data time: 0.1706, Forward time: 1.2646, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.9401 [2021-07-08 02:46:29,610][root][INFO] - ===> Epoch[4102](20510/5): Loss 0.0107, Sem 0.0107 LR: 6.863e-02 Score 99.610 Data time: 0.1336, Forward time: 1.2646, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8184 [2021-07-08 02:46:43,475][root][INFO] - ===> Epoch[4103](20515/5): Loss 0.0103, Sem 0.0103 LR: 6.862e-02 Score 99.623 Data time: 0.2318, Forward time: 1.2646, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7288 [2021-07-08 02:46:58,066][root][INFO] - ===> Epoch[4104](20520/5): Loss 0.0110, Sem 0.0110 LR: 6.861e-02 Score 99.601 Data time: 0.2265, Forward time: 1.2646, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8771 [2021-07-08 02:47:11,780][root][INFO] - ===> Epoch[4105](20525/5): Loss 0.0108, Sem 0.0108 LR: 6.861e-02 Score 99.611 Data time: 0.2035, Forward time: 1.2647, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6926 [2021-07-08 02:47:25,901][root][INFO] - ===> Epoch[4106](20530/5): Loss 0.0109, Sem 0.0109 LR: 6.860e-02 Score 99.607 Data time: 0.1835, Forward time: 1.2647, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7776 [2021-07-08 02:47:39,510][root][INFO] - ===> Epoch[4107](20535/5): Loss 0.0107, Sem 0.0107 LR: 6.859e-02 Score 99.614 Data time: 0.2633, Forward time: 1.2647, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6752 [2021-07-08 02:47:53,963][root][INFO] - ===> Epoch[4108](20540/5): Loss 0.0107, Sem 0.0107 LR: 6.858e-02 Score 99.616 Data time: 0.2376, Forward time: 1.2647, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8439 [2021-07-08 02:48:09,126][root][INFO] - ===> Epoch[4109](20545/5): Loss 0.0109, Sem 0.0109 LR: 6.857e-02 Score 99.604 Data time: 0.2829, Forward time: 1.2647, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.9848 [2021-07-08 02:48:23,466][root][INFO] - ===> Epoch[4110](20550/5): Loss 0.0105, Sem 0.0105 LR: 6.857e-02 Score 99.615 Data time: 0.3246, Forward time: 1.2647, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8082 [2021-07-08 02:48:37,005][root][INFO] - ===> Epoch[4111](20555/5): Loss 0.0107, Sem 0.0107 LR: 6.856e-02 Score 99.613 Data time: 0.2170, Forward time: 1.2647, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6623 [2021-07-08 02:48:50,436][root][INFO] - ===> Epoch[4112](20560/5): Loss 0.0103, Sem 0.0103 LR: 6.855e-02 Score 99.627 Data time: 0.2178, Forward time: 1.2646, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6356 [2021-07-08 02:49:04,961][root][INFO] - ===> Epoch[4113](20565/5): Loss 0.0106, Sem 0.0106 LR: 6.854e-02 Score 99.616 Data time: 0.2799, Forward time: 1.2647, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8596 [2021-07-08 02:49:18,521][root][INFO] - ===> Epoch[4114](20570/5): Loss 0.0103, Sem 0.0103 LR: 6.853e-02 Score 99.625 Data time: 0.2460, Forward time: 1.2647, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6633 [2021-07-08 02:49:32,666][root][INFO] - ===> Epoch[4115](20575/5): Loss 0.0108, Sem 0.0108 LR: 6.853e-02 Score 99.612 Data time: 0.2757, Forward time: 1.2647, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7830 [2021-07-08 02:49:47,714][root][INFO] - ===> Epoch[4116](20580/5): Loss 0.0110, Sem 0.0110 LR: 6.852e-02 Score 99.604 Data time: 0.2390, Forward time: 1.2647, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.9619 [2021-07-08 02:50:01,801][root][INFO] - ===> Epoch[4117](20585/5): Loss 0.0113, Sem 0.0113 LR: 6.851e-02 Score 99.594 Data time: 0.2679, Forward time: 1.2647, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7684 [2021-07-08 02:50:16,744][root][INFO] - ===> Epoch[4118](20590/5): Loss 0.0109, Sem 0.0109 LR: 6.850e-02 Score 99.603 Data time: 0.2077, Forward time: 1.2647, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.9400 [2021-07-08 02:50:30,907][root][INFO] - ===> Epoch[4119](20595/5): Loss 0.0110, Sem 0.0110 LR: 6.850e-02 Score 99.604 Data time: 0.2134, Forward time: 1.2647, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7948 [2021-07-08 02:50:45,311][root][INFO] - ===> Epoch[4120](20600/5): Loss 0.0111, Sem 0.0111 LR: 6.849e-02 Score 99.598 Data time: 0.2785, Forward time: 1.2647, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8252 [2021-07-08 02:50:45,313][root][INFO] - ===> Start testing [2021-07-08 02:51:19,554][root][INFO] - 68/68: Data time: 0.0016, Iter time: 0.1289 Loss 0.303 (AVG: 0.588) Score 93.697 (AVG: 90.961) mIOU 70.560 mAP 83.821 mAcc 76.476 IOU: 65.624 0.034 84.697 76.250 95.300 92.467 20.077 81.782 98.618 79.052 80.493 85.887 57.004 mAP: 83.694 12.311 95.080 87.511 98.664 98.558 48.370 98.045 99.796 92.389 91.928 93.417 89.908 mAcc: 81.828 0.273 89.807 83.154 96.491 96.989 22.290 96.066 99.486 81.087 89.182 96.247 61.281 [2021-07-08 02:51:19,555][root][INFO] - Finished test. Elapsed time: 34.2413 [2021-07-08 02:51:20,564][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 02:51:33,858][root][INFO] - ===> Epoch[4121](20605/5): Loss 0.0109, Sem 0.0109 LR: 6.848e-02 Score 99.604 Data time: 0.2130, Forward time: 1.2647, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6154 [2021-07-08 02:51:48,113][root][INFO] - ===> Epoch[4122](20610/5): Loss 0.0110, Sem 0.0110 LR: 6.847e-02 Score 99.603 Data time: 0.2009, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8095 [2021-07-08 02:52:02,094][root][INFO] - ===> Epoch[4123](20615/5): Loss 0.0108, Sem 0.0108 LR: 6.846e-02 Score 99.609 Data time: 0.3267, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7484 [2021-07-08 02:52:16,931][root][INFO] - ===> Epoch[4124](20620/5): Loss 0.0115, Sem 0.0115 LR: 6.846e-02 Score 99.590 Data time: 0.2498, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.9143 [2021-07-08 02:52:30,569][root][INFO] - ===> Epoch[4125](20625/5): Loss 0.0111, Sem 0.0111 LR: 6.845e-02 Score 99.597 Data time: 0.1806, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6867 [2021-07-08 02:52:43,973][root][INFO] - ===> Epoch[4126](20630/5): Loss 0.0110, Sem 0.0110 LR: 6.844e-02 Score 99.606 Data time: 0.2420, Forward time: 1.2648, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6352 [2021-07-08 02:52:57,628][root][INFO] - ===> Epoch[4127](20635/5): Loss 0.0108, Sem 0.0108 LR: 6.843e-02 Score 99.611 Data time: 0.2235, Forward time: 1.2648, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6881 [2021-07-08 02:53:11,679][root][INFO] - ===> Epoch[4128](20640/5): Loss 0.0103, Sem 0.0103 LR: 6.843e-02 Score 99.623 Data time: 0.2651, Forward time: 1.2648, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7666 [2021-07-08 02:53:26,730][root][INFO] - ===> Epoch[4129](20645/5): Loss 0.0106, Sem 0.0106 LR: 6.842e-02 Score 99.612 Data time: 0.2348, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.9691 [2021-07-08 02:53:40,269][root][INFO] - ===> Epoch[4130](20650/5): Loss 0.0110, Sem 0.0110 LR: 6.841e-02 Score 99.605 Data time: 0.2472, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6616 [2021-07-08 02:53:54,060][root][INFO] - ===> Epoch[4131](20655/5): Loss 0.0106, Sem 0.0106 LR: 6.840e-02 Score 99.618 Data time: 0.2326, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7064 [2021-07-08 02:54:08,263][root][INFO] - ===> Epoch[4132](20660/5): Loss 0.0108, Sem 0.0108 LR: 6.839e-02 Score 99.607 Data time: 0.2461, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7987 [2021-07-08 02:54:22,592][root][INFO] - ===> Epoch[4133](20665/5): Loss 0.0102, Sem 0.0102 LR: 6.839e-02 Score 99.634 Data time: 0.2431, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8196 [2021-07-08 02:54:37,268][root][INFO] - ===> Epoch[4134](20670/5): Loss 0.0108, Sem 0.0108 LR: 6.838e-02 Score 99.611 Data time: 0.2099, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8906 [2021-07-08 02:54:51,921][root][INFO] - ===> Epoch[4135](20675/5): Loss 0.0104, Sem 0.0104 LR: 6.837e-02 Score 99.624 Data time: 0.2348, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8799 [2021-07-08 02:55:06,096][root][INFO] - ===> Epoch[4136](20680/5): Loss 0.0104, Sem 0.0104 LR: 6.836e-02 Score 99.626 Data time: 0.2873, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7649 [2021-07-08 02:55:20,886][root][INFO] - ===> Epoch[4137](20685/5): Loss 0.0110, Sem 0.0110 LR: 6.835e-02 Score 99.601 Data time: 0.2552, Forward time: 1.2649, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8992 [2021-07-08 02:55:35,195][root][INFO] - ===> Epoch[4138](20690/5): Loss 0.0102, Sem 0.0102 LR: 6.835e-02 Score 99.628 Data time: 0.2657, Forward time: 1.2649, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8168 [2021-07-08 02:55:48,965][root][INFO] - ===> Epoch[4139](20695/5): Loss 0.0108, Sem 0.0108 LR: 6.834e-02 Score 99.610 Data time: 0.2492, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7142 [2021-07-08 02:56:02,526][root][INFO] - ===> Epoch[4140](20700/5): Loss 0.0104, Sem 0.0104 LR: 6.833e-02 Score 99.623 Data time: 0.2448, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6663 [2021-07-08 02:56:16,217][root][INFO] - ===> Epoch[4141](20705/5): Loss 0.0106, Sem 0.0106 LR: 6.832e-02 Score 99.617 Data time: 0.2400, Forward time: 1.2648, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6899 [2021-07-08 02:56:29,769][root][INFO] - ===> Epoch[4142](20710/5): Loss 0.0105, Sem 0.0105 LR: 6.832e-02 Score 99.618 Data time: 0.1626, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6652 [2021-07-08 02:56:44,211][root][INFO] - ===> Epoch[4143](20715/5): Loss 0.0101, Sem 0.0101 LR: 6.831e-02 Score 99.630 Data time: 0.2669, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8460 [2021-07-08 02:56:58,584][root][INFO] - ===> Epoch[4144](20720/5): Loss 0.0103, Sem 0.0103 LR: 6.830e-02 Score 99.629 Data time: 0.2470, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8272 [2021-07-08 02:57:12,547][root][INFO] - ===> Epoch[4145](20725/5): Loss 0.0104, Sem 0.0104 LR: 6.829e-02 Score 99.625 Data time: 0.2324, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7424 [2021-07-08 02:57:26,499][root][INFO] - ===> Epoch[4146](20730/5): Loss 0.0106, Sem 0.0106 LR: 6.828e-02 Score 99.617 Data time: 0.1985, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7485 [2021-07-08 02:57:41,174][root][INFO] - ===> Epoch[4147](20735/5): Loss 0.0104, Sem 0.0104 LR: 6.828e-02 Score 99.624 Data time: 0.2901, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8805 [2021-07-08 02:57:55,402][root][INFO] - ===> Epoch[4148](20740/5): Loss 0.0101, Sem 0.0101 LR: 6.827e-02 Score 99.635 Data time: 0.2062, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7990 [2021-07-08 02:58:09,081][root][INFO] - ===> Epoch[4149](20745/5): Loss 0.0110, Sem 0.0110 LR: 6.826e-02 Score 99.599 Data time: 0.2015, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6945 [2021-07-08 02:58:23,089][root][INFO] - ===> Epoch[4150](20750/5): Loss 0.0107, Sem 0.0107 LR: 6.825e-02 Score 99.611 Data time: 0.2546, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7554 [2021-07-08 02:58:37,218][root][INFO] - ===> Epoch[4151](20755/5): Loss 0.0109, Sem 0.0109 LR: 6.825e-02 Score 99.608 Data time: 0.2583, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7715 [2021-07-08 02:58:50,917][root][INFO] - ===> Epoch[4152](20760/5): Loss 0.0105, Sem 0.0105 LR: 6.824e-02 Score 99.619 Data time: 0.2418, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6920 [2021-07-08 02:59:05,222][root][INFO] - ===> Epoch[4153](20765/5): Loss 0.0106, Sem 0.0106 LR: 6.823e-02 Score 99.614 Data time: 0.2258, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8096 [2021-07-08 02:59:19,560][root][INFO] - ===> Epoch[4154](20770/5): Loss 0.0105, Sem 0.0105 LR: 6.822e-02 Score 99.625 Data time: 0.2760, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8151 [2021-07-08 02:59:33,702][root][INFO] - ===> Epoch[4155](20775/5): Loss 0.0104, Sem 0.0104 LR: 6.821e-02 Score 99.621 Data time: 0.2089, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7873 [2021-07-08 02:59:47,762][root][INFO] - ===> Epoch[4156](20780/5): Loss 0.0110, Sem 0.0110 LR: 6.821e-02 Score 99.605 Data time: 0.1914, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7703 [2021-07-08 03:00:01,510][root][INFO] - ===> Epoch[4157](20785/5): Loss 0.0112, Sem 0.0112 LR: 6.820e-02 Score 99.604 Data time: 0.2559, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7042 [2021-07-08 03:00:15,614][root][INFO] - ===> Epoch[4158](20790/5): Loss 0.0106, Sem 0.0106 LR: 6.819e-02 Score 99.616 Data time: 0.2108, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7634 [2021-07-08 03:00:29,870][root][INFO] - ===> Epoch[4159](20795/5): Loss 0.0110, Sem 0.0110 LR: 6.818e-02 Score 99.606 Data time: 0.2389, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8046 [2021-07-08 03:00:43,585][root][INFO] - ===> Epoch[4160](20800/5): Loss 0.0110, Sem 0.0110 LR: 6.817e-02 Score 99.606 Data time: 0.2312, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6922 [2021-07-08 03:00:43,587][root][INFO] - ===> Start testing [2021-07-08 03:01:19,842][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1431 Loss 0.261 (AVG: 0.568) Score 93.652 (AVG: 91.136) mIOU 71.277 mAP 83.548 mAcc 77.100 IOU: 65.603 0.000 85.383 75.921 95.545 92.700 27.095 84.098 98.539 77.156 80.306 86.055 58.208 mAP: 84.114 5.190 93.500 87.104 98.680 98.645 54.553 97.097 99.787 92.261 92.079 93.560 89.554 mAcc: 80.980 0.000 90.018 83.595 96.847 97.216 30.793 92.723 99.438 81.304 90.094 96.244 63.055 [2021-07-08 03:01:19,842][root][INFO] - Finished test. Elapsed time: 36.2546 [2021-07-08 03:01:20,985][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 03:01:34,651][root][INFO] - ===> Epoch[4161](20805/5): Loss 0.0101, Sem 0.0101 LR: 6.817e-02 Score 99.635 Data time: 0.1661, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6973 [2021-07-08 03:01:48,651][root][INFO] - ===> Epoch[4162](20810/5): Loss 0.0106, Sem 0.0106 LR: 6.816e-02 Score 99.616 Data time: 0.1696, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7636 [2021-07-08 03:02:02,085][root][INFO] - ===> Epoch[4163](20815/5): Loss 0.0104, Sem 0.0104 LR: 6.815e-02 Score 99.621 Data time: 0.2642, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6425 [2021-07-08 03:02:15,872][root][INFO] - ===> Epoch[4164](20820/5): Loss 0.0106, Sem 0.0106 LR: 6.814e-02 Score 99.621 Data time: 0.2320, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7095 [2021-07-08 03:02:29,553][root][INFO] - ===> Epoch[4165](20825/5): Loss 0.0109, Sem 0.0109 LR: 6.814e-02 Score 99.603 Data time: 0.2242, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6873 [2021-07-08 03:02:44,319][root][INFO] - ===> Epoch[4166](20830/5): Loss 0.0109, Sem 0.0109 LR: 6.813e-02 Score 99.606 Data time: 0.2381, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.9097 [2021-07-08 03:02:57,781][root][INFO] - ===> Epoch[4167](20835/5): Loss 0.0113, Sem 0.0113 LR: 6.812e-02 Score 99.593 Data time: 0.2506, Forward time: 1.2649, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.6455 [2021-07-08 03:03:12,308][root][INFO] - ===> Epoch[4168](20840/5): Loss 0.0106, Sem 0.0106 LR: 6.811e-02 Score 99.616 Data time: 0.2469, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8670 [2021-07-08 03:03:26,154][root][INFO] - ===> Epoch[4169](20845/5): Loss 0.0105, Sem 0.0105 LR: 6.810e-02 Score 99.622 Data time: 0.3027, Forward time: 1.2649, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7229 [2021-07-08 03:03:40,433][root][INFO] - ===> Epoch[4170](20850/5): Loss 0.0101, Sem 0.0101 LR: 6.810e-02 Score 99.631 Data time: 0.2240, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8252 [2021-07-08 03:03:54,552][root][INFO] - ===> Epoch[4171](20855/5): Loss 0.0107, Sem 0.0107 LR: 6.809e-02 Score 99.615 Data time: 0.2364, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7717 [2021-07-08 03:04:08,778][root][INFO] - ===> Epoch[4172](20860/5): Loss 0.0109, Sem 0.0109 LR: 6.808e-02 Score 99.612 Data time: 0.2262, Forward time: 1.2648, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7883 [2021-07-08 03:04:22,195][root][INFO] - ===> Epoch[4173](20865/5): Loss 0.0112, Sem 0.0112 LR: 6.807e-02 Score 99.595 Data time: 0.1656, Forward time: 1.2648, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.6321 [2021-07-08 03:04:36,585][root][INFO] - ===> Epoch[4174](20870/5): Loss 0.0113, Sem 0.0113 LR: 6.807e-02 Score 99.599 Data time: 0.1645, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8376 [2021-07-08 03:04:51,275][root][INFO] - ===> Epoch[4175](20875/5): Loss 0.0122, Sem 0.0122 LR: 6.806e-02 Score 99.566 Data time: 0.2087, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8993 [2021-07-08 03:05:04,931][root][INFO] - ===> Epoch[4176](20880/5): Loss 0.0115, Sem 0.0115 LR: 6.805e-02 Score 99.587 Data time: 0.1958, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6824 [2021-07-08 03:05:19,402][root][INFO] - ===> Epoch[4177](20885/5): Loss 0.0113, Sem 0.0113 LR: 6.804e-02 Score 99.594 Data time: 0.2695, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8418 [2021-07-08 03:05:33,516][root][INFO] - ===> Epoch[4178](20890/5): Loss 0.0112, Sem 0.0112 LR: 6.803e-02 Score 99.592 Data time: 0.1901, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7842 [2021-07-08 03:05:46,854][root][INFO] - ===> Epoch[4179](20895/5): Loss 0.0111, Sem 0.0111 LR: 6.803e-02 Score 99.602 Data time: 0.2666, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6177 [2021-07-08 03:06:01,097][root][INFO] - ===> Epoch[4180](20900/5): Loss 0.0118, Sem 0.0118 LR: 6.802e-02 Score 99.582 Data time: 0.2366, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7982 [2021-07-08 03:06:15,990][root][INFO] - ===> Epoch[4181](20905/5): Loss 0.0112, Sem 0.0112 LR: 6.801e-02 Score 99.602 Data time: 0.2705, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.9239 [2021-07-08 03:06:29,825][root][INFO] - ===> Epoch[4182](20910/5): Loss 0.0113, Sem 0.0113 LR: 6.800e-02 Score 99.595 Data time: 0.1838, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7225 [2021-07-08 03:06:43,126][root][INFO] - ===> Epoch[4183](20915/5): Loss 0.0113, Sem 0.0113 LR: 6.799e-02 Score 99.590 Data time: 0.2208, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6122 [2021-07-08 03:06:58,000][root][INFO] - ===> Epoch[4184](20920/5): Loss 0.0110, Sem 0.0110 LR: 6.799e-02 Score 99.605 Data time: 0.2800, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.9284 [2021-07-08 03:07:11,329][root][INFO] - ===> Epoch[4185](20925/5): Loss 0.0110, Sem 0.0110 LR: 6.798e-02 Score 99.600 Data time: 0.1903, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6287 [2021-07-08 03:07:25,122][root][INFO] - ===> Epoch[4186](20930/5): Loss 0.0115, Sem 0.0115 LR: 6.797e-02 Score 99.587 Data time: 0.2345, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7080 [2021-07-08 03:07:39,537][root][INFO] - ===> Epoch[4187](20935/5): Loss 0.0107, Sem 0.0107 LR: 6.796e-02 Score 99.613 Data time: 0.2077, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8228 [2021-07-08 03:07:53,408][root][INFO] - ===> Epoch[4188](20940/5): Loss 0.0112, Sem 0.0112 LR: 6.796e-02 Score 99.596 Data time: 0.2545, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7258 [2021-07-08 03:08:08,231][root][INFO] - ===> Epoch[4189](20945/5): Loss 0.0114, Sem 0.0114 LR: 6.795e-02 Score 99.594 Data time: 0.2936, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.9146 [2021-07-08 03:08:21,882][root][INFO] - ===> Epoch[4190](20950/5): Loss 0.0109, Sem 0.0109 LR: 6.794e-02 Score 99.605 Data time: 0.2532, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6916 [2021-07-08 03:08:36,517][root][INFO] - ===> Epoch[4191](20955/5): Loss 0.0109, Sem 0.0109 LR: 6.793e-02 Score 99.612 Data time: 0.2819, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8818 [2021-07-08 03:08:50,955][root][INFO] - ===> Epoch[4192](20960/5): Loss 0.0108, Sem 0.0108 LR: 6.792e-02 Score 99.607 Data time: 0.3106, Forward time: 1.2648, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8443 [2021-07-08 03:09:05,820][root][INFO] - ===> Epoch[4193](20965/5): Loss 0.0106, Sem 0.0106 LR: 6.792e-02 Score 99.619 Data time: 0.2776, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.9270 [2021-07-08 03:09:19,596][root][INFO] - ===> Epoch[4194](20970/5): Loss 0.0108, Sem 0.0108 LR: 6.791e-02 Score 99.613 Data time: 0.2444, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7068 [2021-07-08 03:09:33,515][root][INFO] - ===> Epoch[4195](20975/5): Loss 0.0110, Sem 0.0110 LR: 6.790e-02 Score 99.606 Data time: 0.2453, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7329 [2021-07-08 03:09:47,463][root][INFO] - ===> Epoch[4196](20980/5): Loss 0.0103, Sem 0.0103 LR: 6.789e-02 Score 99.624 Data time: 0.2295, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7515 [2021-07-08 03:10:00,940][root][INFO] - ===> Epoch[4197](20985/5): Loss 0.0109, Sem 0.0109 LR: 6.789e-02 Score 99.603 Data time: 0.2722, Forward time: 1.2649, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.6537 [2021-07-08 03:10:14,563][root][INFO] - ===> Epoch[4198](20990/5): Loss 0.0103, Sem 0.0103 LR: 6.788e-02 Score 99.638 Data time: 0.1976, Forward time: 1.2649, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.6859 [2021-07-08 03:10:29,542][root][INFO] - ===> Epoch[4199](20995/5): Loss 0.0111, Sem 0.0111 LR: 6.787e-02 Score 99.601 Data time: 0.2686, Forward time: 1.2650, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.9533 [2021-07-08 03:10:43,479][root][INFO] - ===> Epoch[4200](21000/5): Loss 0.0116, Sem 0.0116 LR: 6.786e-02 Score 99.587 Data time: 0.2024, Forward time: 1.2649, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7482 [2021-07-08 03:10:43,480][root][INFO] - ===> Start testing [2021-07-08 03:11:18,511][root][INFO] - 68/68: Data time: 0.0017, Iter time: 0.1396 Loss 0.368 (AVG: 0.541) Score 92.254 (AVG: 91.119) mIOU 71.274 mAP 83.633 mAcc 77.001 IOU: 65.541 0.000 86.288 75.176 95.241 92.125 23.851 85.459 98.516 77.923 80.281 86.051 60.112 mAP: 84.458 6.401 94.411 88.616 98.631 98.646 53.937 98.326 99.722 85.250 92.304 94.083 92.446 mAcc: 79.908 0.000 89.861 82.823 96.722 96.998 25.979 97.043 99.479 79.430 90.976 96.369 65.421 [2021-07-08 03:11:18,512][root][INFO] - Finished test. Elapsed time: 35.0310 [2021-07-08 03:11:19,684][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 03:11:31,849][root][INFO] - ===> Epoch[4201](21005/5): Loss 0.0105, Sem 0.0105 LR: 6.785e-02 Score 99.622 Data time: 0.2856, Forward time: 1.2649, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.3975 [2021-07-08 03:11:45,640][root][INFO] - ===> Epoch[4202](21010/5): Loss 0.0118, Sem 0.0118 LR: 6.785e-02 Score 99.587 Data time: 0.2031, Forward time: 1.2649, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7068 [2021-07-08 03:12:00,105][root][INFO] - ===> Epoch[4203](21015/5): Loss 0.0113, Sem 0.0113 LR: 6.784e-02 Score 99.593 Data time: 0.2240, Forward time: 1.2649, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8460 [2021-07-08 03:12:13,500][root][INFO] - ===> Epoch[4204](21020/5): Loss 0.0113, Sem 0.0113 LR: 6.783e-02 Score 99.591 Data time: 0.2126, Forward time: 1.2649, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.6211 [2021-07-08 03:12:27,597][root][INFO] - ===> Epoch[4205](21025/5): Loss 0.1178, Sem 0.1178 LR: 6.782e-02 Score 96.947 Data time: 0.2999, Forward time: 1.2649, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7653 [2021-07-08 03:12:41,584][root][INFO] - ===> Epoch[4206](21030/5): Loss 0.4576, Sem 0.4576 LR: 6.781e-02 Score 87.728 Data time: 0.1980, Forward time: 1.2649, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.7520 [2021-07-08 03:12:56,266][root][INFO] - ===> Epoch[4207](21035/5): Loss 0.4046, Sem 0.4046 LR: 6.781e-02 Score 88.435 Data time: 0.3236, Forward time: 1.2649, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8834 [2021-07-08 03:13:10,248][root][INFO] - ===> Epoch[4208](21040/5): Loss 0.3170, Sem 0.3170 LR: 6.780e-02 Score 90.659 Data time: 0.2628, Forward time: 1.2649, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.7470 [2021-07-08 03:13:23,538][root][INFO] - ===> Epoch[4209](21045/5): Loss 0.2672, Sem 0.2672 LR: 6.779e-02 Score 91.894 Data time: 0.2352, Forward time: 1.2649, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.5980 [2021-07-08 03:13:37,492][root][INFO] - ===> Epoch[4210](21050/5): Loss 0.2378, Sem 0.2378 LR: 6.778e-02 Score 92.700 Data time: 0.3019, Forward time: 1.2650, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7470 [2021-07-08 03:13:50,792][root][INFO] - ===> Epoch[4211](21055/5): Loss 0.2112, Sem 0.2112 LR: 6.778e-02 Score 93.524 Data time: 0.1891, Forward time: 1.2650, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6147 [2021-07-08 03:14:04,474][root][INFO] - ===> Epoch[4212](21060/5): Loss 0.1864, Sem 0.1864 LR: 6.777e-02 Score 94.194 Data time: 0.2619, Forward time: 1.2650, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6830 [2021-07-08 03:14:18,984][root][INFO] - ===> Epoch[4213](21065/5): Loss 0.1572, Sem 0.1572 LR: 6.776e-02 Score 95.108 Data time: 0.2939, Forward time: 1.2650, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8463 [2021-07-08 03:14:34,546][root][INFO] - ===> Epoch[4214](21070/5): Loss 0.1525, Sem 0.1525 LR: 6.775e-02 Score 95.212 Data time: 0.2689, Forward time: 1.2650, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 3.0612 [2021-07-08 03:14:48,577][root][INFO] - ===> Epoch[4215](21075/5): Loss 0.1270, Sem 0.1270 LR: 6.774e-02 Score 95.938 Data time: 0.2646, Forward time: 1.2650, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7510 [2021-07-08 03:15:03,400][root][INFO] - ===> Epoch[4216](21080/5): Loss 0.1192, Sem 0.1192 LR: 6.774e-02 Score 96.220 Data time: 0.2917, Forward time: 1.2650, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.9097 [2021-07-08 03:15:17,627][root][INFO] - ===> Epoch[4217](21085/5): Loss 0.1051, Sem 0.1051 LR: 6.773e-02 Score 96.617 Data time: 0.2489, Forward time: 1.2651, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8003 [2021-07-08 03:15:32,202][root][INFO] - ===> Epoch[4218](21090/5): Loss 0.0994, Sem 0.0994 LR: 6.772e-02 Score 96.781 Data time: 0.2145, Forward time: 1.2651, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8684 [2021-07-08 03:15:45,474][root][INFO] - ===> Epoch[4219](21095/5): Loss 0.0941, Sem 0.0941 LR: 6.771e-02 Score 96.940 Data time: 0.2328, Forward time: 1.2651, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6099 [2021-07-08 03:15:59,781][root][INFO] - ===> Epoch[4220](21100/5): Loss 0.0946, Sem 0.0946 LR: 6.771e-02 Score 96.879 Data time: 0.2505, Forward time: 1.2651, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8245 [2021-07-08 03:16:14,822][root][INFO] - ===> Epoch[4221](21105/5): Loss 0.0840, Sem 0.0840 LR: 6.770e-02 Score 97.264 Data time: 0.2908, Forward time: 1.2651, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.9691 [2021-07-08 03:16:29,272][root][INFO] - ===> Epoch[4222](21110/5): Loss 0.0840, Sem 0.0840 LR: 6.769e-02 Score 97.241 Data time: 0.2556, Forward time: 1.2651, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8511 [2021-07-08 03:16:43,233][root][INFO] - ===> Epoch[4223](21115/5): Loss 0.0768, Sem 0.0768 LR: 6.768e-02 Score 97.450 Data time: 0.2343, Forward time: 1.2651, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7436 [2021-07-08 03:16:57,407][root][INFO] - ===> Epoch[4224](21120/5): Loss 0.0782, Sem 0.0782 LR: 6.767e-02 Score 97.399 Data time: 0.1674, Forward time: 1.2651, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8067 [2021-07-08 03:17:11,076][root][INFO] - ===> Epoch[4225](21125/5): Loss 0.0722, Sem 0.0722 LR: 6.767e-02 Score 97.593 Data time: 0.1722, Forward time: 1.2652, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6820 [2021-07-08 03:17:24,326][root][INFO] - ===> Epoch[4226](21130/5): Loss 0.0664, Sem 0.0664 LR: 6.766e-02 Score 97.742 Data time: 0.2196, Forward time: 1.2651, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6074 [2021-07-08 03:17:38,649][root][INFO] - ===> Epoch[4227](21135/5): Loss 0.0673, Sem 0.0673 LR: 6.765e-02 Score 97.756 Data time: 0.2539, Forward time: 1.2651, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8059 [2021-07-08 03:17:52,654][root][INFO] - ===> Epoch[4228](21140/5): Loss 0.0613, Sem 0.0613 LR: 6.764e-02 Score 97.941 Data time: 0.1571, Forward time: 1.2651, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7584 [2021-07-08 03:18:05,988][root][INFO] - ===> Epoch[4229](21145/5): Loss 0.0627, Sem 0.0627 LR: 6.763e-02 Score 97.896 Data time: 0.2466, Forward time: 1.2651, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6176 [2021-07-08 03:18:19,405][root][INFO] - ===> Epoch[4230](21150/5): Loss 0.0618, Sem 0.0618 LR: 6.763e-02 Score 97.953 Data time: 0.2818, Forward time: 1.2651, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6386 [2021-07-08 03:18:33,656][root][INFO] - ===> Epoch[4231](21155/5): Loss 0.0584, Sem 0.0584 LR: 6.762e-02 Score 98.036 Data time: 0.2892, Forward time: 1.2651, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8066 [2021-07-08 03:18:47,045][root][INFO] - ===> Epoch[4232](21160/5): Loss 0.0542, Sem 0.0542 LR: 6.761e-02 Score 98.151 Data time: 0.2053, Forward time: 1.2651, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.6343 [2021-07-08 03:19:00,368][root][INFO] - ===> Epoch[4233](21165/5): Loss 0.0544, Sem 0.0544 LR: 6.760e-02 Score 98.128 Data time: 0.2316, Forward time: 1.2651, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6230 [2021-07-08 03:19:14,067][root][INFO] - ===> Epoch[4234](21170/5): Loss 0.0502, Sem 0.0502 LR: 6.760e-02 Score 98.277 Data time: 0.1901, Forward time: 1.2651, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6903 [2021-07-08 03:19:27,527][root][INFO] - ===> Epoch[4235](21175/5): Loss 0.0481, Sem 0.0481 LR: 6.759e-02 Score 98.357 Data time: 0.2418, Forward time: 1.2651, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.6435 [2021-07-08 03:19:41,821][root][INFO] - ===> Epoch[4236](21180/5): Loss 0.0487, Sem 0.0487 LR: 6.758e-02 Score 98.335 Data time: 0.2620, Forward time: 1.2652, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8121 [2021-07-08 03:19:55,803][root][INFO] - ===> Epoch[4237](21185/5): Loss 0.0475, Sem 0.0475 LR: 6.757e-02 Score 98.405 Data time: 0.2689, Forward time: 1.2652, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7365 [2021-07-08 03:20:10,044][root][INFO] - ===> Epoch[4238](21190/5): Loss 0.0451, Sem 0.0451 LR: 6.756e-02 Score 98.446 Data time: 0.2969, Forward time: 1.2652, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8025 [2021-07-08 03:20:24,602][root][INFO] - ===> Epoch[4239](21195/5): Loss 0.0455, Sem 0.0455 LR: 6.756e-02 Score 98.451 Data time: 0.2881, Forward time: 1.2652, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8544 [2021-07-08 03:20:38,851][root][INFO] - ===> Epoch[4240](21200/5): Loss 0.0435, Sem 0.0435 LR: 6.755e-02 Score 98.523 Data time: 0.3123, Forward time: 1.2652, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7981 [2021-07-08 03:20:38,852][root][INFO] - ===> Start testing [2021-07-08 03:21:15,543][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1308 Loss 0.300 (AVG: 0.431) Score 90.471 (AVG: 90.529) mIOU 69.465 mAP 84.164 mAcc 75.783 IOU: 63.092 0.029 78.951 73.863 95.774 91.025 27.147 79.727 98.321 75.257 77.597 85.752 56.505 mAP: 84.948 5.575 92.489 89.887 99.086 98.618 57.115 97.110 99.793 91.669 92.266 94.001 91.569 mAcc: 80.241 0.182 87.359 82.400 97.307 95.766 28.900 91.723 99.617 79.371 84.622 95.803 61.888 [2021-07-08 03:21:15,543][root][INFO] - Finished test. Elapsed time: 36.6905 [2021-07-08 03:21:16,698][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 03:21:29,712][root][INFO] - ===> Epoch[4241](21205/5): Loss 0.0448, Sem 0.0448 LR: 6.754e-02 Score 98.459 Data time: 0.1973, Forward time: 1.2652, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.5668 [2021-07-08 03:21:43,019][root][INFO] - ===> Epoch[4242](21210/5): Loss 0.0447, Sem 0.0447 LR: 6.753e-02 Score 98.461 Data time: 0.1756, Forward time: 1.2652, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6278 [2021-07-08 03:21:57,442][root][INFO] - ===> Epoch[4243](21215/5): Loss 0.0435, Sem 0.0435 LR: 6.753e-02 Score 98.511 Data time: 0.1616, Forward time: 1.2652, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8444 [2021-07-08 03:22:12,298][root][INFO] - ===> Epoch[4244](21220/5): Loss 0.0439, Sem 0.0439 LR: 6.752e-02 Score 98.478 Data time: 0.2331, Forward time: 1.2652, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.9259 [2021-07-08 03:22:26,158][root][INFO] - ===> Epoch[4245](21225/5): Loss 0.0436, Sem 0.0436 LR: 6.751e-02 Score 98.496 Data time: 0.2260, Forward time: 1.2652, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7180 [2021-07-08 03:22:39,888][root][INFO] - ===> Epoch[4246](21230/5): Loss 0.0425, Sem 0.0425 LR: 6.750e-02 Score 98.527 Data time: 0.2779, Forward time: 1.2652, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.6953 [2021-07-08 03:22:53,879][root][INFO] - ===> Epoch[4247](21235/5): Loss 0.0424, Sem 0.0424 LR: 6.749e-02 Score 98.550 Data time: 0.2261, Forward time: 1.2652, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7623 [2021-07-08 03:23:07,250][root][INFO] - ===> Epoch[4248](21240/5): Loss 0.0430, Sem 0.0430 LR: 6.749e-02 Score 98.534 Data time: 0.2719, Forward time: 1.2652, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.6306 [2021-07-08 03:23:21,167][root][INFO] - ===> Epoch[4249](21245/5): Loss 0.0424, Sem 0.0424 LR: 6.748e-02 Score 98.544 Data time: 0.2673, Forward time: 1.2652, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7374 [2021-07-08 03:23:34,908][root][INFO] - ===> Epoch[4250](21250/5): Loss 0.0396, Sem 0.0396 LR: 6.747e-02 Score 98.617 Data time: 0.2204, Forward time: 1.2652, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6926 [2021-07-08 03:23:48,565][root][INFO] - ===> Epoch[4251](21255/5): Loss 0.0424, Sem 0.0424 LR: 6.746e-02 Score 98.534 Data time: 0.2227, Forward time: 1.2652, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6825 [2021-07-08 03:24:02,866][root][INFO] - ===> Epoch[4252](21260/5): Loss 0.0406, Sem 0.0406 LR: 6.745e-02 Score 98.597 Data time: 0.2644, Forward time: 1.2652, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8136 [2021-07-08 03:24:16,639][root][INFO] - ===> Epoch[4253](21265/5): Loss 0.0412, Sem 0.0412 LR: 6.745e-02 Score 98.605 Data time: 0.2865, Forward time: 1.2652, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7043 [2021-07-08 03:24:31,276][root][INFO] - ===> Epoch[4254](21270/5): Loss 0.0408, Sem 0.0408 LR: 6.744e-02 Score 98.590 Data time: 0.2377, Forward time: 1.2652, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8827 [2021-07-08 03:24:45,545][root][INFO] - ===> Epoch[4255](21275/5): Loss 0.0410, Sem 0.0410 LR: 6.743e-02 Score 98.576 Data time: 0.2319, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8219 [2021-07-08 03:24:59,744][root][INFO] - ===> Epoch[4256](21280/5): Loss 0.0391, Sem 0.0391 LR: 6.742e-02 Score 98.638 Data time: 0.2710, Forward time: 1.2653, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.7876 [2021-07-08 03:25:13,788][root][INFO] - ===> Epoch[4257](21285/5): Loss 0.0411, Sem 0.0411 LR: 6.742e-02 Score 98.580 Data time: 0.1907, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7620 [2021-07-08 03:25:28,074][root][INFO] - ===> Epoch[4258](21290/5): Loss 0.0376, Sem 0.0376 LR: 6.741e-02 Score 98.701 Data time: 0.2599, Forward time: 1.2653, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.8157 [2021-07-08 03:25:43,528][root][INFO] - ===> Epoch[4259](21295/5): Loss 0.0409, Sem 0.0409 LR: 6.740e-02 Score 98.622 Data time: 0.2625, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 3.0406 [2021-07-08 03:25:56,924][root][INFO] - ===> Epoch[4260](21300/5): Loss 0.0384, Sem 0.0384 LR: 6.739e-02 Score 98.667 Data time: 0.2020, Forward time: 1.2653, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.6378 [2021-07-08 03:26:11,226][root][INFO] - ===> Epoch[4261](21305/5): Loss 0.0383, Sem 0.0383 LR: 6.738e-02 Score 98.659 Data time: 0.2254, Forward time: 1.2654, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.8158 [2021-07-08 03:26:24,802][root][INFO] - ===> Epoch[4262](21310/5): Loss 0.0382, Sem 0.0382 LR: 6.738e-02 Score 98.669 Data time: 0.1716, Forward time: 1.2654, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.6775 [2021-07-08 03:26:39,328][root][INFO] - ===> Epoch[4263](21315/5): Loss 0.0355, Sem 0.0355 LR: 6.737e-02 Score 98.762 Data time: 0.3006, Forward time: 1.2654, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.8547 [2021-07-08 03:26:53,500][root][INFO] - ===> Epoch[4264](21320/5): Loss 0.0353, Sem 0.0353 LR: 6.736e-02 Score 98.775 Data time: 0.2343, Forward time: 1.2654, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.7810 [2021-07-08 03:27:07,901][root][INFO] - ===> Epoch[4265](21325/5): Loss 0.0355, Sem 0.0355 LR: 6.735e-02 Score 98.771 Data time: 0.2825, Forward time: 1.2654, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.8339 [2021-07-08 03:27:21,748][root][INFO] - ===> Epoch[4266](21330/5): Loss 0.0335, Sem 0.0335 LR: 6.734e-02 Score 98.840 Data time: 0.2434, Forward time: 1.2654, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.7109 [2021-07-08 03:27:36,246][root][INFO] - ===> Epoch[4267](21335/5): Loss 0.0328, Sem 0.0328 LR: 6.734e-02 Score 98.857 Data time: 0.2699, Forward time: 1.2655, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.8505 [2021-07-08 03:27:51,810][root][INFO] - ===> Epoch[4268](21340/5): Loss 0.0329, Sem 0.0329 LR: 6.733e-02 Score 98.840 Data time: 0.2380, Forward time: 1.2655, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 3.0537 [2021-07-08 03:28:05,058][root][INFO] - ===> Epoch[4269](21345/5): Loss 0.0328, Sem 0.0328 LR: 6.732e-02 Score 98.850 Data time: 0.1832, Forward time: 1.2655, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.6056 [2021-07-08 03:28:19,686][root][INFO] - ===> Epoch[4270](21350/5): Loss 0.0319, Sem 0.0319 LR: 6.731e-02 Score 98.915 Data time: 0.2262, Forward time: 1.2655, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.8824 [2021-07-08 03:28:33,321][root][INFO] - ===> Epoch[4271](21355/5): Loss 0.0320, Sem 0.0320 LR: 6.731e-02 Score 98.886 Data time: 0.2379, Forward time: 1.2655, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.6818 [2021-07-08 03:28:47,876][root][INFO] - ===> Epoch[4272](21360/5): Loss 0.0295, Sem 0.0295 LR: 6.730e-02 Score 98.959 Data time: 0.2166, Forward time: 1.2655, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.8635 [2021-07-08 03:29:01,926][root][INFO] - ===> Epoch[4273](21365/5): Loss 0.0295, Sem 0.0295 LR: 6.729e-02 Score 98.967 Data time: 0.2427, Forward time: 1.2655, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.7650 [2021-07-08 03:29:15,971][root][INFO] - ===> Epoch[4274](21370/5): Loss 0.0303, Sem 0.0303 LR: 6.728e-02 Score 98.921 Data time: 0.2490, Forward time: 1.2655, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.7541 [2021-07-08 03:29:30,491][root][INFO] - ===> Epoch[4275](21375/5): Loss 0.0275, Sem 0.0275 LR: 6.727e-02 Score 99.034 Data time: 0.2225, Forward time: 1.2655, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.8654 [2021-07-08 03:29:44,102][root][INFO] - ===> Epoch[4276](21380/5): Loss 0.0289, Sem 0.0289 LR: 6.727e-02 Score 98.975 Data time: 0.1882, Forward time: 1.2655, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.6810 [2021-07-08 03:29:58,389][root][INFO] - ===> Epoch[4277](21385/5): Loss 0.0285, Sem 0.0285 LR: 6.726e-02 Score 99.001 Data time: 0.2213, Forward time: 1.2655, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.8165 [2021-07-08 03:30:12,063][root][INFO] - ===> Epoch[4278](21390/5): Loss 0.0267, Sem 0.0267 LR: 6.725e-02 Score 99.057 Data time: 0.2185, Forward time: 1.2655, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.6916 [2021-07-08 03:30:25,672][root][INFO] - ===> Epoch[4279](21395/5): Loss 0.0270, Sem 0.0270 LR: 6.724e-02 Score 99.051 Data time: 0.1927, Forward time: 1.2655, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.6824 [2021-07-08 03:30:38,877][root][INFO] - ===> Epoch[4280](21400/5): Loss 0.0284, Sem 0.0284 LR: 6.724e-02 Score 98.993 Data time: 0.1874, Forward time: 1.2655, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.5957 [2021-07-08 03:30:38,880][root][INFO] - ===> Start testing [2021-07-08 03:31:12,882][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1717 Loss 0.252 (AVG: 0.456) Score 94.076 (AVG: 90.705) mIOU 69.697 mAP 83.823 mAcc 75.811 IOU: 63.136 0.000 77.551 73.434 95.572 91.450 30.733 82.508 98.552 69.002 80.102 86.168 57.850 mAP: 84.545 5.002 89.762 88.896 98.980 98.550 62.897 97.561 99.813 84.618 92.734 95.110 91.222 mAcc: 81.197 0.000 84.441 80.916 96.688 97.010 34.731 91.914 99.596 71.133 89.182 96.119 62.618 [2021-07-08 03:31:12,882][root][INFO] - Finished test. Elapsed time: 34.0016 [2021-07-08 03:31:13,970][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 03:31:27,503][root][INFO] - ===> Epoch[4281](21405/5): Loss 0.0258, Sem 0.0258 LR: 6.723e-02 Score 99.084 Data time: 0.2054, Forward time: 1.2654, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.6623 [2021-07-08 03:31:41,098][root][INFO] - ===> Epoch[4282](21410/5): Loss 0.0270, Sem 0.0270 LR: 6.722e-02 Score 99.041 Data time: 0.2652, Forward time: 1.2654, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.6646 [2021-07-08 03:31:55,205][root][INFO] - ===> Epoch[4283](21415/5): Loss 0.0251, Sem 0.0251 LR: 6.721e-02 Score 99.106 Data time: 0.2979, Forward time: 1.2654, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.7703 [2021-07-08 03:32:09,538][root][INFO] - ===> Epoch[4284](21420/5): Loss 0.0255, Sem 0.0255 LR: 6.720e-02 Score 99.089 Data time: 0.2799, Forward time: 1.2655, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.8113 [2021-07-08 03:32:23,924][root][INFO] - ===> Epoch[4285](21425/5): Loss 0.0251, Sem 0.0251 LR: 6.720e-02 Score 99.106 Data time: 0.2358, Forward time: 1.2655, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.8294 [2021-07-08 03:32:37,888][root][INFO] - ===> Epoch[4286](21430/5): Loss 0.0256, Sem 0.0256 LR: 6.719e-02 Score 99.096 Data time: 0.2768, Forward time: 1.2655, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.7396 [2021-07-08 03:32:51,933][root][INFO] - ===> Epoch[4287](21435/5): Loss 0.0247, Sem 0.0247 LR: 6.718e-02 Score 99.118 Data time: 0.2568, Forward time: 1.2655, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.7616 [2021-07-08 03:33:05,503][root][INFO] - ===> Epoch[4288](21440/5): Loss 0.0259, Sem 0.0259 LR: 6.717e-02 Score 99.077 Data time: 0.2354, Forward time: 1.2655, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.6702 [2021-07-08 03:33:19,773][root][INFO] - ===> Epoch[4289](21445/5): Loss 0.0248, Sem 0.0248 LR: 6.716e-02 Score 99.117 Data time: 0.2021, Forward time: 1.2655, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.8184 [2021-07-08 03:33:34,064][root][INFO] - ===> Epoch[4290](21450/5): Loss 0.0251, Sem 0.0251 LR: 6.716e-02 Score 99.104 Data time: 0.1796, Forward time: 1.2655, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.8117 [2021-07-08 03:33:47,567][root][INFO] - ===> Epoch[4291](21455/5): Loss 0.0246, Sem 0.0246 LR: 6.715e-02 Score 99.132 Data time: 0.2534, Forward time: 1.2655, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.6474 [2021-07-08 03:34:01,002][root][INFO] - ===> Epoch[4292](21460/5): Loss 0.0241, Sem 0.0241 LR: 6.714e-02 Score 99.136 Data time: 0.1552, Forward time: 1.2655, Backward time: 1.2309, DDP time: 0.0042, Total iter time: 2.6458 [2021-07-08 03:34:15,438][root][INFO] - ===> Epoch[4293](21465/5): Loss 0.0243, Sem 0.0243 LR: 6.713e-02 Score 99.136 Data time: 0.2289, Forward time: 1.2655, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.8429 [2021-07-08 03:34:30,029][root][INFO] - ===> Epoch[4294](21470/5): Loss 0.0237, Sem 0.0237 LR: 6.713e-02 Score 99.162 Data time: 0.2362, Forward time: 1.2655, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.8722 [2021-07-08 03:34:43,461][root][INFO] - ===> Epoch[4295](21475/5): Loss 0.0227, Sem 0.0227 LR: 6.712e-02 Score 99.185 Data time: 0.2420, Forward time: 1.2655, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.6356 [2021-07-08 03:34:57,515][root][INFO] - ===> Epoch[4296](21480/5): Loss 0.0231, Sem 0.0231 LR: 6.711e-02 Score 99.171 Data time: 0.2586, Forward time: 1.2655, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.7630 [2021-07-08 03:35:11,138][root][INFO] - ===> Epoch[4297](21485/5): Loss 0.0252, Sem 0.0252 LR: 6.710e-02 Score 99.111 Data time: 0.2047, Forward time: 1.2654, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.6823 [2021-07-08 03:35:25,636][root][INFO] - ===> Epoch[4298](21490/5): Loss 0.0233, Sem 0.0233 LR: 6.709e-02 Score 99.178 Data time: 0.2703, Forward time: 1.2654, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.8507 [2021-07-08 03:35:40,045][root][INFO] - ===> Epoch[4299](21495/5): Loss 0.0241, Sem 0.0241 LR: 6.709e-02 Score 99.147 Data time: 0.2685, Forward time: 1.2654, Backward time: 1.2310, DDP time: 0.0042, Total iter time: 2.8251 [2021-07-08 03:35:54,894][root][INFO] - ===> Epoch[4300](21500/5): Loss 0.0245, Sem 0.0245 LR: 6.708e-02 Score 99.120 Data time: 0.1730, Forward time: 1.2654, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.9262 [2021-07-08 03:36:08,628][root][INFO] - ===> Epoch[4301](21505/5): Loss 0.0238, Sem 0.0238 LR: 6.707e-02 Score 99.162 Data time: 0.2185, Forward time: 1.2654, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.7003 [2021-07-08 03:36:23,415][root][INFO] - ===> Epoch[4302](21510/5): Loss 0.0227, Sem 0.0227 LR: 6.706e-02 Score 99.191 Data time: 0.2200, Forward time: 1.2654, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.9072 [2021-07-08 03:36:37,511][root][INFO] - ===> Epoch[4303](21515/5): Loss 0.0231, Sem 0.0231 LR: 6.705e-02 Score 99.172 Data time: 0.2145, Forward time: 1.2654, Backward time: 1.2311, DDP time: 0.0042, Total iter time: 2.7799 [2021-07-08 03:36:51,820][root][INFO] - ===> Epoch[4304](21520/5): Loss 0.0222, Sem 0.0222 LR: 6.705e-02 Score 99.204 Data time: 0.2354, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8121 [2021-07-08 03:37:06,053][root][INFO] - ===> Epoch[4305](21525/5): Loss 0.0225, Sem 0.0225 LR: 6.704e-02 Score 99.207 Data time: 0.1844, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7959 [2021-07-08 03:37:20,180][root][INFO] - ===> Epoch[4306](21530/5): Loss 0.0233, Sem 0.0233 LR: 6.703e-02 Score 99.163 Data time: 0.2599, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7708 [2021-07-08 03:37:34,336][root][INFO] - ===> Epoch[4307](21535/5): Loss 0.0215, Sem 0.0215 LR: 6.702e-02 Score 99.235 Data time: 0.2014, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7682 [2021-07-08 03:37:48,478][root][INFO] - ===> Epoch[4308](21540/5): Loss 0.0221, Sem 0.0221 LR: 6.702e-02 Score 99.207 Data time: 0.2692, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7870 [2021-07-08 03:38:03,410][root][INFO] - ===> Epoch[4309](21545/5): Loss 0.0221, Sem 0.0221 LR: 6.701e-02 Score 99.206 Data time: 0.2450, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.9429 [2021-07-08 03:38:17,853][root][INFO] - ===> Epoch[4310](21550/5): Loss 0.0216, Sem 0.0216 LR: 6.700e-02 Score 99.225 Data time: 0.2894, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8350 [2021-07-08 03:38:32,472][root][INFO] - ===> Epoch[4311](21555/5): Loss 0.0227, Sem 0.0227 LR: 6.699e-02 Score 99.191 Data time: 0.2840, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8740 [2021-07-08 03:38:47,127][root][INFO] - ===> Epoch[4312](21560/5): Loss 0.0215, Sem 0.0215 LR: 6.698e-02 Score 99.239 Data time: 0.2879, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8753 [2021-07-08 03:39:00,201][root][INFO] - ===> Epoch[4313](21565/5): Loss 0.0216, Sem 0.0216 LR: 6.698e-02 Score 99.233 Data time: 0.1913, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.5710 [2021-07-08 03:39:13,231][root][INFO] - ===> Epoch[4314](21570/5): Loss 0.0235, Sem 0.0235 LR: 6.697e-02 Score 99.165 Data time: 0.1816, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.5694 [2021-07-08 03:39:27,538][root][INFO] - ===> Epoch[4315](21575/5): Loss 0.0223, Sem 0.0223 LR: 6.696e-02 Score 99.204 Data time: 0.2162, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8143 [2021-07-08 03:39:41,338][root][INFO] - ===> Epoch[4316](21580/5): Loss 0.0247, Sem 0.0247 LR: 6.695e-02 Score 99.125 Data time: 0.2618, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7079 [2021-07-08 03:39:55,439][root][INFO] - ===> Epoch[4317](21585/5): Loss 0.0223, Sem 0.0223 LR: 6.695e-02 Score 99.207 Data time: 0.2256, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7662 [2021-07-08 03:40:09,554][root][INFO] - ===> Epoch[4318](21590/5): Loss 0.0232, Sem 0.0232 LR: 6.694e-02 Score 99.173 Data time: 0.2151, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7788 [2021-07-08 03:40:23,254][root][INFO] - ===> Epoch[4319](21595/5): Loss 0.0227, Sem 0.0227 LR: 6.693e-02 Score 99.193 Data time: 0.1979, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6890 [2021-07-08 03:40:36,594][root][INFO] - ===> Epoch[4320](21600/5): Loss 0.0227, Sem 0.0227 LR: 6.692e-02 Score 99.185 Data time: 0.2252, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6221 [2021-07-08 03:40:36,596][root][INFO] - ===> Start testing [2021-07-08 03:41:12,215][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1213 Loss 0.333 (AVG: 0.487) Score 91.088 (AVG: 90.692) mIOU 70.260 mAP 83.795 mAcc 76.733 IOU: 63.347 0.000 81.682 74.664 94.643 91.726 36.762 82.573 98.494 69.927 76.588 86.431 56.544 mAP: 85.058 6.344 92.215 88.733 98.941 98.343 64.850 97.898 99.800 79.408 91.675 95.571 90.500 mAcc: 80.835 0.000 86.341 81.070 96.062 96.001 43.308 93.754 99.494 71.300 92.293 95.593 61.482 [2021-07-08 03:41:12,216][root][INFO] - Finished test. Elapsed time: 35.6192 [2021-07-08 03:41:13,254][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 03:41:26,522][root][INFO] - ===> Epoch[4321](21605/5): Loss 0.0238, Sem 0.0238 LR: 6.691e-02 Score 99.151 Data time: 0.2376, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6101 [2021-07-08 03:41:40,560][root][INFO] - ===> Epoch[4322](21610/5): Loss 0.0233, Sem 0.0233 LR: 6.691e-02 Score 99.179 Data time: 0.1973, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7557 [2021-07-08 03:41:55,174][root][INFO] - ===> Epoch[4323](21615/5): Loss 0.0241, Sem 0.0241 LR: 6.690e-02 Score 99.177 Data time: 0.2333, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8743 [2021-07-08 03:42:09,897][root][INFO] - ===> Epoch[4324](21620/5): Loss 0.0268, Sem 0.0268 LR: 6.689e-02 Score 99.088 Data time: 0.2125, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.9063 [2021-07-08 03:42:24,343][root][INFO] - ===> Epoch[4325](21625/5): Loss 0.0246, Sem 0.0246 LR: 6.688e-02 Score 99.125 Data time: 0.1781, Forward time: 1.2653, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8406 [2021-07-08 03:42:37,505][root][INFO] - ===> Epoch[4326](21630/5): Loss 0.0257, Sem 0.0257 LR: 6.687e-02 Score 99.086 Data time: 0.2695, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.5936 [2021-07-08 03:42:51,299][root][INFO] - ===> Epoch[4327](21635/5): Loss 0.0249, Sem 0.0249 LR: 6.687e-02 Score 99.140 Data time: 0.2197, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7148 [2021-07-08 03:43:05,754][root][INFO] - ===> Epoch[4328](21640/5): Loss 0.0243, Sem 0.0243 LR: 6.686e-02 Score 99.139 Data time: 0.2747, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8364 [2021-07-08 03:43:20,239][root][INFO] - ===> Epoch[4329](21645/5): Loss 0.0226, Sem 0.0226 LR: 6.685e-02 Score 99.196 Data time: 0.2696, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8581 [2021-07-08 03:43:34,207][root][INFO] - ===> Epoch[4330](21650/5): Loss 0.0231, Sem 0.0231 LR: 6.684e-02 Score 99.173 Data time: 0.1642, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7511 [2021-07-08 03:43:48,915][root][INFO] - ===> Epoch[4331](21655/5): Loss 0.0227, Sem 0.0227 LR: 6.684e-02 Score 99.192 Data time: 0.2339, Forward time: 1.2653, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8917 [2021-07-08 03:44:02,353][root][INFO] - ===> Epoch[4332](21660/5): Loss 0.0214, Sem 0.0214 LR: 6.683e-02 Score 99.239 Data time: 0.2660, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6378 [2021-07-08 03:44:17,994][root][INFO] - ===> Epoch[4333](21665/5): Loss 0.0222, Sem 0.0222 LR: 6.682e-02 Score 99.206 Data time: 0.2292, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 3.0847 [2021-07-08 03:44:32,157][root][INFO] - ===> Epoch[4334](21670/5): Loss 0.0220, Sem 0.0220 LR: 6.681e-02 Score 99.222 Data time: 0.2278, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7890 [2021-07-08 03:44:46,144][root][INFO] - ===> Epoch[4335](21675/5): Loss 0.0203, Sem 0.0203 LR: 6.680e-02 Score 99.277 Data time: 0.2231, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7524 [2021-07-08 03:45:00,368][root][INFO] - ===> Epoch[4336](21680/5): Loss 0.0215, Sem 0.0215 LR: 6.680e-02 Score 99.230 Data time: 0.2557, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7931 [2021-07-08 03:45:14,298][root][INFO] - ===> Epoch[4337](21685/5): Loss 0.0220, Sem 0.0220 LR: 6.679e-02 Score 99.220 Data time: 0.2199, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7303 [2021-07-08 03:45:28,885][root][INFO] - ===> Epoch[4338](21690/5): Loss 0.0200, Sem 0.0200 LR: 6.678e-02 Score 99.279 Data time: 0.2433, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8583 [2021-07-08 03:45:42,630][root][INFO] - ===> Epoch[4339](21695/5): Loss 0.0201, Sem 0.0201 LR: 6.677e-02 Score 99.281 Data time: 0.2425, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7010 [2021-07-08 03:45:56,394][root][INFO] - ===> Epoch[4340](21700/5): Loss 0.0209, Sem 0.0209 LR: 6.676e-02 Score 99.251 Data time: 0.2269, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7176 [2021-07-08 03:46:10,340][root][INFO] - ===> Epoch[4341](21705/5): Loss 0.0204, Sem 0.0204 LR: 6.676e-02 Score 99.270 Data time: 0.2450, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7317 [2021-07-08 03:46:24,040][root][INFO] - ===> Epoch[4342](21710/5): Loss 0.0204, Sem 0.0204 LR: 6.675e-02 Score 99.272 Data time: 0.2455, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6812 [2021-07-08 03:46:37,709][root][INFO] - ===> Epoch[4343](21715/5): Loss 0.0193, Sem 0.0193 LR: 6.674e-02 Score 99.308 Data time: 0.1927, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6972 [2021-07-08 03:46:51,838][root][INFO] - ===> Epoch[4344](21720/5): Loss 0.0200, Sem 0.0200 LR: 6.673e-02 Score 99.280 Data time: 0.2468, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7897 [2021-07-08 03:47:06,129][root][INFO] - ===> Epoch[4345](21725/5): Loss 0.0200, Sem 0.0200 LR: 6.673e-02 Score 99.287 Data time: 0.2163, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7957 [2021-07-08 03:47:19,880][root][INFO] - ===> Epoch[4346](21730/5): Loss 0.0197, Sem 0.0197 LR: 6.672e-02 Score 99.309 Data time: 0.1481, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7114 [2021-07-08 03:47:34,273][root][INFO] - ===> Epoch[4347](21735/5): Loss 0.0195, Sem 0.0195 LR: 6.671e-02 Score 99.299 Data time: 0.2211, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8297 [2021-07-08 03:47:48,326][root][INFO] - ===> Epoch[4348](21740/5): Loss 0.0196, Sem 0.0196 LR: 6.670e-02 Score 99.308 Data time: 0.1791, Forward time: 1.2654, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7735 [2021-07-08 03:48:02,562][root][INFO] - ===> Epoch[4349](21745/5): Loss 0.0198, Sem 0.0198 LR: 6.669e-02 Score 99.294 Data time: 0.2522, Forward time: 1.2654, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8025 [2021-07-08 03:48:16,521][root][INFO] - ===> Epoch[4350](21750/5): Loss 0.0192, Sem 0.0192 LR: 6.669e-02 Score 99.316 Data time: 0.2273, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7372 [2021-07-08 03:48:30,261][root][INFO] - ===> Epoch[4351](21755/5): Loss 0.0195, Sem 0.0195 LR: 6.668e-02 Score 99.303 Data time: 0.2369, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.7002 [2021-07-08 03:48:44,604][root][INFO] - ===> Epoch[4352](21760/5): Loss 0.0212, Sem 0.0212 LR: 6.667e-02 Score 99.270 Data time: 0.2033, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8278 [2021-07-08 03:48:58,838][root][INFO] - ===> Epoch[4353](21765/5): Loss 0.0215, Sem 0.0215 LR: 6.666e-02 Score 99.230 Data time: 0.2107, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.8044 [2021-07-08 03:49:12,592][root][INFO] - ===> Epoch[4354](21770/5): Loss 0.0224, Sem 0.0224 LR: 6.665e-02 Score 99.229 Data time: 0.2864, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6931 [2021-07-08 03:49:26,633][root][INFO] - ===> Epoch[4355](21775/5): Loss 0.0208, Sem 0.0208 LR: 6.665e-02 Score 99.263 Data time: 0.1990, Forward time: 1.2654, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7597 [2021-07-08 03:49:40,066][root][INFO] - ===> Epoch[4356](21780/5): Loss 0.0200, Sem 0.0200 LR: 6.664e-02 Score 99.283 Data time: 0.2370, Forward time: 1.2654, Backward time: 1.2312, DDP time: 0.0042, Total iter time: 2.6412 [2021-07-08 03:49:55,558][root][INFO] - ===> Epoch[4357](21785/5): Loss 0.0194, Sem 0.0194 LR: 6.663e-02 Score 99.303 Data time: 0.2716, Forward time: 1.2654, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 3.0448 [2021-07-08 03:50:10,072][root][INFO] - ===> Epoch[4358](21790/5): Loss 0.0202, Sem 0.0202 LR: 6.662e-02 Score 99.273 Data time: 0.3153, Forward time: 1.2654, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.8503 [2021-07-08 03:50:24,183][root][INFO] - ===> Epoch[4359](21795/5): Loss 0.0192, Sem 0.0192 LR: 6.662e-02 Score 99.311 Data time: 0.2532, Forward time: 1.2654, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7624 [2021-07-08 03:50:38,992][root][INFO] - ===> Epoch[4360](21800/5): Loss 0.0189, Sem 0.0189 LR: 6.661e-02 Score 99.318 Data time: 0.2536, Forward time: 1.2654, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.9191 [2021-07-08 03:50:38,993][root][INFO] - ===> Start testing [2021-07-08 03:51:15,680][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1201 Loss 0.406 (AVG: 0.491) Score 90.465 (AVG: 91.062) mIOU 70.905 mAP 84.917 mAcc 77.315 IOU: 64.446 0.029 78.194 76.367 95.153 91.576 36.086 81.580 98.340 76.582 78.537 86.855 58.014 mAP: 85.863 6.824 93.287 89.017 99.035 98.145 64.527 98.488 99.810 89.812 92.544 94.977 91.590 mAcc: 79.899 0.273 85.037 84.111 96.454 95.582 40.716 95.858 99.393 79.272 90.313 95.892 62.299 [2021-07-08 03:51:15,680][root][INFO] - Finished test. Elapsed time: 36.6856 [2021-07-08 03:51:16,761][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 03:51:29,840][root][INFO] - ===> Epoch[4361](21805/5): Loss 0.0199, Sem 0.0199 LR: 6.660e-02 Score 99.285 Data time: 0.1978, Forward time: 1.2654, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.5682 [2021-07-08 03:51:43,979][root][INFO] - ===> Epoch[4362](21810/5): Loss 0.0192, Sem 0.0192 LR: 6.659e-02 Score 99.306 Data time: 0.2284, Forward time: 1.2653, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7779 [2021-07-08 03:51:57,908][root][INFO] - ===> Epoch[4363](21815/5): Loss 0.0186, Sem 0.0186 LR: 6.658e-02 Score 99.335 Data time: 0.2390, Forward time: 1.2653, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7304 [2021-07-08 03:52:11,883][root][INFO] - ===> Epoch[4364](21820/5): Loss 0.0202, Sem 0.0202 LR: 6.658e-02 Score 99.276 Data time: 0.2259, Forward time: 1.2653, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7491 [2021-07-08 03:52:25,712][root][INFO] - ===> Epoch[4365](21825/5): Loss 0.0187, Sem 0.0187 LR: 6.657e-02 Score 99.328 Data time: 0.1967, Forward time: 1.2653, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7137 [2021-07-08 03:52:38,900][root][INFO] - ===> Epoch[4366](21830/5): Loss 0.0204, Sem 0.0204 LR: 6.656e-02 Score 99.273 Data time: 0.1863, Forward time: 1.2653, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.6011 [2021-07-08 03:52:52,702][root][INFO] - ===> Epoch[4367](21835/5): Loss 0.0200, Sem 0.0200 LR: 6.655e-02 Score 99.286 Data time: 0.2035, Forward time: 1.2653, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7183 [2021-07-08 03:53:06,675][root][INFO] - ===> Epoch[4368](21840/5): Loss 0.0194, Sem 0.0194 LR: 6.654e-02 Score 99.299 Data time: 0.2111, Forward time: 1.2653, Backward time: 1.2313, DDP time: 0.0042, Total iter time: 2.7502 [2021-07-08 03:53:21,234][root][INFO] - ===> Epoch[4369](21845/5): Loss 0.0197, Sem 0.0197 LR: 6.654e-02 Score 99.289 Data time: 0.2154, Forward time: 1.2653, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8720 [2021-07-08 03:53:35,196][root][INFO] - ===> Epoch[4370](21850/5): Loss 0.0198, Sem 0.0198 LR: 6.653e-02 Score 99.294 Data time: 0.2440, Forward time: 1.2652, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7380 [2021-07-08 03:53:49,612][root][INFO] - ===> Epoch[4371](21855/5): Loss 0.0193, Sem 0.0193 LR: 6.652e-02 Score 99.299 Data time: 0.2016, Forward time: 1.2652, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8372 [2021-07-08 03:54:03,726][root][INFO] - ===> Epoch[4372](21860/5): Loss 0.0195, Sem 0.0195 LR: 6.651e-02 Score 99.301 Data time: 0.2125, Forward time: 1.2652, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7647 [2021-07-08 03:54:17,980][root][INFO] - ===> Epoch[4373](21865/5): Loss 0.0195, Sem 0.0195 LR: 6.651e-02 Score 99.297 Data time: 0.2565, Forward time: 1.2652, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.8069 [2021-07-08 03:54:32,008][root][INFO] - ===> Epoch[4374](21870/5): Loss 0.0189, Sem 0.0189 LR: 6.650e-02 Score 99.323 Data time: 0.2072, Forward time: 1.2652, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7594 [2021-07-08 03:54:45,751][root][INFO] - ===> Epoch[4375](21875/5): Loss 0.0202, Sem 0.0202 LR: 6.649e-02 Score 99.297 Data time: 0.2184, Forward time: 1.2652, Backward time: 1.2314, DDP time: 0.0042, Total iter time: 2.7002 [2021-07-08 03:55:00,328][root][INFO] - ===> Epoch[4376](21880/5): Loss 0.0209, Sem 0.0209 LR: 6.648e-02 Score 99.269 Data time: 0.2342, Forward time: 1.2652, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.8750 [2021-07-08 03:55:14,260][root][INFO] - ===> Epoch[4377](21885/5): Loss 0.0192, Sem 0.0192 LR: 6.647e-02 Score 99.314 Data time: 0.1804, Forward time: 1.2652, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7486 [2021-07-08 03:55:28,317][root][INFO] - ===> Epoch[4378](21890/5): Loss 0.0194, Sem 0.0194 LR: 6.647e-02 Score 99.300 Data time: 0.1948, Forward time: 1.2652, Backward time: 1.2315, DDP time: 0.0042, Total iter time: 2.7654 [2021-07-08 03:55:43,225][root][INFO] - ===> Epoch[4379](21895/5): Loss 0.0186, Sem 0.0186 LR: 6.646e-02 Score 99.328 Data time: 0.1570, Forward time: 1.2652, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.9233 [2021-07-08 03:55:57,092][root][INFO] - ===> Epoch[4380](21900/5): Loss 0.0189, Sem 0.0189 LR: 6.645e-02 Score 99.318 Data time: 0.2432, Forward time: 1.2652, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7361 [2021-07-08 03:56:11,361][root][INFO] - ===> Epoch[4381](21905/5): Loss 0.0191, Sem 0.0191 LR: 6.644e-02 Score 99.312 Data time: 0.2375, Forward time: 1.2652, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7987 [2021-07-08 03:56:25,844][root][INFO] - ===> Epoch[4382](21910/5): Loss 0.0183, Sem 0.0183 LR: 6.644e-02 Score 99.345 Data time: 0.2015, Forward time: 1.2651, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8555 [2021-07-08 03:56:40,220][root][INFO] - ===> Epoch[4383](21915/5): Loss 0.0174, Sem 0.0174 LR: 6.643e-02 Score 99.375 Data time: 0.2842, Forward time: 1.2652, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.8295 [2021-07-08 03:56:55,398][root][INFO] - ===> Epoch[4384](21920/5): Loss 0.0170, Sem 0.0170 LR: 6.642e-02 Score 99.387 Data time: 0.2914, Forward time: 1.2652, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.9858 [2021-07-08 03:57:09,119][root][INFO] - ===> Epoch[4385](21925/5): Loss 0.0184, Sem 0.0184 LR: 6.641e-02 Score 99.338 Data time: 0.1947, Forward time: 1.2652, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.6995 [2021-07-08 03:57:23,333][root][INFO] - ===> Epoch[4386](21930/5): Loss 0.0180, Sem 0.0180 LR: 6.640e-02 Score 99.349 Data time: 0.2597, Forward time: 1.2652, Backward time: 1.2316, DDP time: 0.0042, Total iter time: 2.7927 [2021-07-08 03:57:37,800][root][INFO] - ===> Epoch[4387](21935/5): Loss 0.0177, Sem 0.0177 LR: 6.640e-02 Score 99.367 Data time: 0.1656, Forward time: 1.2651, Backward time: 1.2317, DDP time: 0.0042, Total iter time: 2.8555 [2021-07-08 03:57:52,080][root][INFO] - ===> Epoch[4388](21940/5): Loss 0.0181, Sem 0.0181 LR: 6.639e-02 Score 99.348 Data time: 0.2474, Forward time: 1.2651, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8090 [2021-07-08 03:58:06,957][root][INFO] - ===> Epoch[4389](21945/5): Loss 0.0168, Sem 0.0168 LR: 6.638e-02 Score 99.393 Data time: 0.2395, Forward time: 1.2651, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.9300 [2021-07-08 03:58:21,134][root][INFO] - ===> Epoch[4390](21950/5): Loss 0.0182, Sem 0.0182 LR: 6.637e-02 Score 99.342 Data time: 0.1485, Forward time: 1.2651, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.8014 [2021-07-08 03:58:34,625][root][INFO] - ===> Epoch[4391](21955/5): Loss 0.0173, Sem 0.0173 LR: 6.636e-02 Score 99.377 Data time: 0.1604, Forward time: 1.2651, Backward time: 1.2318, DDP time: 0.0042, Total iter time: 2.6648 [2021-07-08 03:58:48,823][root][INFO] - ===> Epoch[4392](21960/5): Loss 0.0169, Sem 0.0169 LR: 6.636e-02 Score 99.391 Data time: 0.1924, Forward time: 1.2650, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.7896 [2021-07-08 03:59:03,234][root][INFO] - ===> Epoch[4393](21965/5): Loss 0.0173, Sem 0.0173 LR: 6.635e-02 Score 99.373 Data time: 0.1864, Forward time: 1.2650, Backward time: 1.2319, DDP time: 0.0042, Total iter time: 2.8493 [2021-07-08 03:59:17,045][root][INFO] - ===> Epoch[4394](21970/5): Loss 0.0178, Sem 0.0178 LR: 6.634e-02 Score 99.362 Data time: 0.1764, Forward time: 1.2650, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7225 [2021-07-08 03:59:31,578][root][INFO] - ===> Epoch[4395](21975/5): Loss 0.0168, Sem 0.0168 LR: 6.633e-02 Score 99.396 Data time: 0.2014, Forward time: 1.2650, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.8659 [2021-07-08 03:59:44,858][root][INFO] - ===> Epoch[4396](21980/5): Loss 0.0181, Sem 0.0181 LR: 6.633e-02 Score 99.355 Data time: 0.1887, Forward time: 1.2650, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6131 [2021-07-08 03:59:58,568][root][INFO] - ===> Epoch[4397](21985/5): Loss 0.0176, Sem 0.0176 LR: 6.632e-02 Score 99.369 Data time: 0.1986, Forward time: 1.2649, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6899 [2021-07-08 04:00:12,236][root][INFO] - ===> Epoch[4398](21990/5): Loss 0.0172, Sem 0.0172 LR: 6.631e-02 Score 99.383 Data time: 0.2092, Forward time: 1.2649, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.6894 [2021-07-08 04:00:26,375][root][INFO] - ===> Epoch[4399](21995/5): Loss 0.0173, Sem 0.0173 LR: 6.630e-02 Score 99.376 Data time: 0.2141, Forward time: 1.2649, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7704 [2021-07-08 04:00:41,385][root][INFO] - ===> Epoch[4400](22000/5): Loss 0.0174, Sem 0.0174 LR: 6.629e-02 Score 99.382 Data time: 0.2095, Forward time: 1.2649, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.9637 [2021-07-08 04:00:41,386][root][INFO] - ===> Start testing [2021-07-08 04:01:16,579][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1224 Loss 0.354 (AVG: 0.525) Score 91.618 (AVG: 90.884) mIOU 69.777 mAP 84.285 mAcc 77.275 IOU: 63.171 0.000 78.640 75.988 95.093 91.669 25.270 82.145 98.378 72.900 79.589 86.621 57.642 mAP: 83.548 6.683 94.023 89.299 98.956 98.043 53.352 98.248 99.834 94.277 93.083 94.867 91.497 mAcc: 78.462 0.000 87.658 84.315 96.390 97.016 26.778 94.078 99.293 91.002 91.131 96.401 62.047 [2021-07-08 04:01:16,579][root][INFO] - Finished test. Elapsed time: 35.1924 [2021-07-08 04:01:17,634][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 04:01:30,670][root][INFO] - ===> Epoch[4401](22005/5): Loss 0.0184, Sem 0.0184 LR: 6.629e-02 Score 99.362 Data time: 0.2030, Forward time: 1.2649, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.5690 [2021-07-08 04:01:44,761][root][INFO] - ===> Epoch[4402](22010/5): Loss 0.0173, Sem 0.0173 LR: 6.628e-02 Score 99.377 Data time: 0.2065, Forward time: 1.2649, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7798 [2021-07-08 04:01:59,002][root][INFO] - ===> Epoch[4403](22015/5): Loss 0.0173, Sem 0.0173 LR: 6.627e-02 Score 99.380 Data time: 0.1831, Forward time: 1.2649, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8090 [2021-07-08 04:02:13,012][root][INFO] - ===> Epoch[4404](22020/5): Loss 0.0180, Sem 0.0180 LR: 6.626e-02 Score 99.360 Data time: 0.1678, Forward time: 1.2649, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7660 [2021-07-08 04:02:26,927][root][INFO] - ===> Epoch[4405](22025/5): Loss 0.0173, Sem 0.0173 LR: 6.625e-02 Score 99.379 Data time: 0.2643, Forward time: 1.2649, Backward time: 1.2320, DDP time: 0.0042, Total iter time: 2.7362 [2021-07-08 04:02:41,147][root][INFO] - ===> Epoch[4406](22030/5): Loss 0.0167, Sem 0.0167 LR: 6.625e-02 Score 99.393 Data time: 0.2736, Forward time: 1.2649, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7997 [2021-07-08 04:02:54,975][root][INFO] - ===> Epoch[4407](22035/5): Loss 0.0168, Sem 0.0168 LR: 6.624e-02 Score 99.395 Data time: 0.1891, Forward time: 1.2649, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7258 [2021-07-08 04:03:09,250][root][INFO] - ===> Epoch[4408](22040/5): Loss 0.0170, Sem 0.0170 LR: 6.623e-02 Score 99.393 Data time: 0.2133, Forward time: 1.2649, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8146 [2021-07-08 04:03:22,983][root][INFO] - ===> Epoch[4409](22045/5): Loss 0.0166, Sem 0.0166 LR: 6.622e-02 Score 99.408 Data time: 0.2047, Forward time: 1.2649, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6896 [2021-07-08 04:03:37,349][root][INFO] - ===> Epoch[4410](22050/5): Loss 0.0176, Sem 0.0176 LR: 6.622e-02 Score 99.365 Data time: 0.1923, Forward time: 1.2649, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.8284 [2021-07-08 04:03:51,116][root][INFO] - ===> Epoch[4411](22055/5): Loss 0.0161, Sem 0.0161 LR: 6.621e-02 Score 99.416 Data time: 0.2244, Forward time: 1.2649, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.6949 [2021-07-08 04:04:05,372][root][INFO] - ===> Epoch[4412](22060/5): Loss 0.0192, Sem 0.0192 LR: 6.620e-02 Score 99.342 Data time: 0.1973, Forward time: 1.2648, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.8055 [2021-07-08 04:04:19,414][root][INFO] - ===> Epoch[4413](22065/5): Loss 0.0185, Sem 0.0185 LR: 6.619e-02 Score 99.334 Data time: 0.2370, Forward time: 1.2649, Backward time: 1.2321, DDP time: 0.0042, Total iter time: 2.7606 [2021-07-08 04:04:34,099][root][INFO] - ===> Epoch[4414](22070/5): Loss 0.0176, Sem 0.0176 LR: 6.618e-02 Score 99.371 Data time: 0.2065, Forward time: 1.2648, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.8864 [2021-07-08 04:04:47,959][root][INFO] - ===> Epoch[4415](22075/5): Loss 0.0180, Sem 0.0180 LR: 6.618e-02 Score 99.352 Data time: 0.1901, Forward time: 1.2648, Backward time: 1.2322, DDP time: 0.0042, Total iter time: 2.7313 [2021-07-08 04:05:03,251][root][INFO] - ===> Epoch[4416](22080/5): Loss 0.0177, Sem 0.0177 LR: 6.617e-02 Score 99.363 Data time: 0.2032, Forward time: 1.2648, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 3.0062 [2021-07-08 04:05:17,435][root][INFO] - ===> Epoch[4417](22085/5): Loss 0.0180, Sem 0.0180 LR: 6.616e-02 Score 99.354 Data time: 0.2059, Forward time: 1.2648, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7927 [2021-07-08 04:05:30,708][root][INFO] - ===> Epoch[4418](22090/5): Loss 0.0178, Sem 0.0178 LR: 6.615e-02 Score 99.389 Data time: 0.2283, Forward time: 1.2648, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.6022 [2021-07-08 04:05:45,003][root][INFO] - ===> Epoch[4419](22095/5): Loss 0.0176, Sem 0.0176 LR: 6.614e-02 Score 99.365 Data time: 0.1839, Forward time: 1.2648, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8073 [2021-07-08 04:05:59,076][root][INFO] - ===> Epoch[4420](22100/5): Loss 0.0178, Sem 0.0178 LR: 6.614e-02 Score 99.366 Data time: 0.2073, Forward time: 1.2648, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.7649 [2021-07-08 04:06:13,517][root][INFO] - ===> Epoch[4421](22105/5): Loss 0.0168, Sem 0.0168 LR: 6.613e-02 Score 99.399 Data time: 0.2234, Forward time: 1.2648, Backward time: 1.2323, DDP time: 0.0042, Total iter time: 2.8362 [2021-07-08 04:06:27,141][root][INFO] - ===> Epoch[4422](22110/5): Loss 0.0171, Sem 0.0171 LR: 6.612e-02 Score 99.389 Data time: 0.1767, Forward time: 1.2648, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6812 [2021-07-08 04:06:40,648][root][INFO] - ===> Epoch[4423](22115/5): Loss 0.0171, Sem 0.0171 LR: 6.611e-02 Score 99.383 Data time: 0.2032, Forward time: 1.2647, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.6528 [2021-07-08 04:06:54,607][root][INFO] - ===> Epoch[4424](22120/5): Loss 0.0159, Sem 0.0159 LR: 6.611e-02 Score 99.423 Data time: 0.2177, Forward time: 1.2647, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7243 [2021-07-08 04:07:09,126][root][INFO] - ===> Epoch[4425](22125/5): Loss 0.0163, Sem 0.0163 LR: 6.610e-02 Score 99.413 Data time: 0.2237, Forward time: 1.2647, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.8587 [2021-07-08 04:07:23,111][root][INFO] - ===> Epoch[4426](22130/5): Loss 0.0171, Sem 0.0171 LR: 6.609e-02 Score 99.394 Data time: 0.2087, Forward time: 1.2647, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7513 [2021-07-08 04:07:37,039][root][INFO] - ===> Epoch[4427](22135/5): Loss 0.0163, Sem 0.0163 LR: 6.608e-02 Score 99.411 Data time: 0.1573, Forward time: 1.2647, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7472 [2021-07-08 04:07:50,988][root][INFO] - ===> Epoch[4428](22140/5): Loss 0.0160, Sem 0.0160 LR: 6.607e-02 Score 99.420 Data time: 0.2323, Forward time: 1.2647, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7426 [2021-07-08 04:08:05,889][root][INFO] - ===> Epoch[4429](22145/5): Loss 0.0161, Sem 0.0161 LR: 6.607e-02 Score 99.424 Data time: 0.2495, Forward time: 1.2647, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9419 [2021-07-08 04:08:20,082][root][INFO] - ===> Epoch[4430](22150/5): Loss 0.0160, Sem 0.0160 LR: 6.606e-02 Score 99.423 Data time: 0.2877, Forward time: 1.2647, Backward time: 1.2324, DDP time: 0.0042, Total iter time: 2.7827 [2021-07-08 04:08:35,161][root][INFO] - ===> Epoch[4431](22155/5): Loss 0.0178, Sem 0.0178 LR: 6.605e-02 Score 99.382 Data time: 0.2960, Forward time: 1.2647, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.9675 [2021-07-08 04:08:49,878][root][INFO] - ===> Epoch[4432](22160/5): Loss 0.0174, Sem 0.0174 LR: 6.604e-02 Score 99.375 Data time: 0.3678, Forward time: 1.2647, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8950 [2021-07-08 04:09:03,463][root][INFO] - ===> Epoch[4433](22165/5): Loss 0.0169, Sem 0.0169 LR: 6.603e-02 Score 99.393 Data time: 0.1479, Forward time: 1.2647, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6764 [2021-07-08 04:09:17,311][root][INFO] - ===> Epoch[4434](22170/5): Loss 0.0167, Sem 0.0167 LR: 6.603e-02 Score 99.405 Data time: 0.2567, Forward time: 1.2647, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7306 [2021-07-08 04:09:31,722][root][INFO] - ===> Epoch[4435](22175/5): Loss 0.0167, Sem 0.0167 LR: 6.602e-02 Score 99.394 Data time: 0.2487, Forward time: 1.2647, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8328 [2021-07-08 04:09:46,273][root][INFO] - ===> Epoch[4436](22180/5): Loss 0.0161, Sem 0.0161 LR: 6.601e-02 Score 99.428 Data time: 0.2202, Forward time: 1.2647, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8717 [2021-07-08 04:10:00,912][root][INFO] - ===> Epoch[4437](22185/5): Loss 0.0158, Sem 0.0158 LR: 6.600e-02 Score 99.435 Data time: 0.2538, Forward time: 1.2647, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.8817 [2021-07-08 04:10:14,032][root][INFO] - ===> Epoch[4438](22190/5): Loss 0.0171, Sem 0.0171 LR: 6.600e-02 Score 99.385 Data time: 0.1952, Forward time: 1.2646, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.5851 [2021-07-08 04:10:27,149][root][INFO] - ===> Epoch[4439](22195/5): Loss 0.0162, Sem 0.0162 LR: 6.599e-02 Score 99.418 Data time: 0.1947, Forward time: 1.2646, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.5804 [2021-07-08 04:10:41,399][root][INFO] - ===> Epoch[4440](22200/5): Loss 0.0164, Sem 0.0164 LR: 6.598e-02 Score 99.409 Data time: 0.2304, Forward time: 1.2646, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7991 [2021-07-08 04:10:41,400][root][INFO] - ===> Start testing [2021-07-08 04:11:17,045][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1235 Loss 0.329 (AVG: 0.521) Score 93.276 (AVG: 91.024) mIOU 71.488 mAP 83.924 mAcc 77.348 IOU: 65.626 0.000 83.223 76.634 95.320 91.385 31.474 84.095 98.460 80.257 80.139 85.987 56.739 mAP: 85.111 5.510 89.088 89.095 98.901 98.040 56.800 98.110 99.804 93.428 92.814 94.219 90.092 mAcc: 82.887 0.000 84.978 82.985 96.593 95.425 39.739 94.808 99.500 83.139 89.823 95.498 60.151 [2021-07-08 04:11:17,046][root][INFO] - Finished test. Elapsed time: 35.6447 [2021-07-08 04:11:18,103][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 04:11:31,311][root][INFO] - ===> Epoch[4441](22205/5): Loss 0.0170, Sem 0.0170 LR: 6.597e-02 Score 99.397 Data time: 0.1990, Forward time: 1.2646, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.5985 [2021-07-08 04:11:44,989][root][INFO] - ===> Epoch[4442](22210/5): Loss 0.0175, Sem 0.0175 LR: 6.596e-02 Score 99.369 Data time: 0.1964, Forward time: 1.2646, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6959 [2021-07-08 04:11:59,866][root][INFO] - ===> Epoch[4443](22215/5): Loss 0.0181, Sem 0.0181 LR: 6.596e-02 Score 99.371 Data time: 0.2415, Forward time: 1.2646, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9294 [2021-07-08 04:12:13,348][root][INFO] - ===> Epoch[4444](22220/5): Loss 0.0172, Sem 0.0172 LR: 6.595e-02 Score 99.383 Data time: 0.2478, Forward time: 1.2646, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6415 [2021-07-08 04:12:26,982][root][INFO] - ===> Epoch[4445](22225/5): Loss 0.0176, Sem 0.0176 LR: 6.594e-02 Score 99.373 Data time: 0.2110, Forward time: 1.2646, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6861 [2021-07-08 04:12:40,615][root][INFO] - ===> Epoch[4446](22230/5): Loss 0.0177, Sem 0.0177 LR: 6.593e-02 Score 99.366 Data time: 0.2101, Forward time: 1.2646, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.6867 [2021-07-08 04:12:54,562][root][INFO] - ===> Epoch[4447](22235/5): Loss 0.0178, Sem 0.0178 LR: 6.592e-02 Score 99.362 Data time: 0.2095, Forward time: 1.2646, Backward time: 1.2325, DDP time: 0.0042, Total iter time: 2.7411 [2021-07-08 04:13:09,048][root][INFO] - ===> Epoch[4448](22240/5): Loss 0.0163, Sem 0.0163 LR: 6.592e-02 Score 99.415 Data time: 0.1964, Forward time: 1.2646, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.8485 [2021-07-08 04:13:22,764][root][INFO] - ===> Epoch[4449](22245/5): Loss 0.0170, Sem 0.0170 LR: 6.591e-02 Score 99.385 Data time: 0.1922, Forward time: 1.2646, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7051 [2021-07-08 04:13:36,517][root][INFO] - ===> Epoch[4450](22250/5): Loss 0.0169, Sem 0.0169 LR: 6.590e-02 Score 99.390 Data time: 0.1906, Forward time: 1.2645, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7151 [2021-07-08 04:13:50,445][root][INFO] - ===> Epoch[4451](22255/5): Loss 0.0168, Sem 0.0168 LR: 6.589e-02 Score 99.394 Data time: 0.2234, Forward time: 1.2645, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7413 [2021-07-08 04:14:04,508][root][INFO] - ===> Epoch[4452](22260/5): Loss 0.0169, Sem 0.0169 LR: 6.589e-02 Score 99.394 Data time: 0.3155, Forward time: 1.2645, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7694 [2021-07-08 04:14:18,534][root][INFO] - ===> Epoch[4453](22265/5): Loss 0.0172, Sem 0.0172 LR: 6.588e-02 Score 99.392 Data time: 0.2757, Forward time: 1.2645, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7559 [2021-07-08 04:14:32,768][root][INFO] - ===> Epoch[4454](22270/5): Loss 0.0169, Sem 0.0169 LR: 6.587e-02 Score 99.400 Data time: 0.2840, Forward time: 1.2645, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.7973 [2021-07-08 04:14:47,522][root][INFO] - ===> Epoch[4455](22275/5): Loss 0.0171, Sem 0.0171 LR: 6.586e-02 Score 99.387 Data time: 0.2883, Forward time: 1.2645, Backward time: 1.2326, DDP time: 0.0042, Total iter time: 2.9032 [2021-07-08 04:15:02,528][root][INFO] - ===> Epoch[4456](22280/5): Loss 0.0165, Sem 0.0165 LR: 6.585e-02 Score 99.404 Data time: 0.2233, Forward time: 1.2645, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.9558 [2021-07-08 04:15:16,079][root][INFO] - ===> Epoch[4457](22285/5): Loss 0.0169, Sem 0.0169 LR: 6.585e-02 Score 99.380 Data time: 0.1897, Forward time: 1.2645, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6678 [2021-07-08 04:15:29,700][root][INFO] - ===> Epoch[4458](22290/5): Loss 0.0170, Sem 0.0170 LR: 6.584e-02 Score 99.392 Data time: 0.2066, Forward time: 1.2645, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6840 [2021-07-08 04:15:44,233][root][INFO] - ===> Epoch[4459](22295/5): Loss 0.0164, Sem 0.0164 LR: 6.583e-02 Score 99.416 Data time: 0.2489, Forward time: 1.2644, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8490 [2021-07-08 04:15:58,161][root][INFO] - ===> Epoch[4460](22300/5): Loss 0.0156, Sem 0.0156 LR: 6.582e-02 Score 99.435 Data time: 0.2159, Forward time: 1.2644, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7378 [2021-07-08 04:16:12,106][root][INFO] - ===> Epoch[4461](22305/5): Loss 0.0157, Sem 0.0157 LR: 6.581e-02 Score 99.436 Data time: 0.2164, Forward time: 1.2644, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.7340 [2021-07-08 04:16:25,846][root][INFO] - ===> Epoch[4462](22310/5): Loss 0.0168, Sem 0.0168 LR: 6.581e-02 Score 99.397 Data time: 0.2386, Forward time: 1.2644, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6983 [2021-07-08 04:16:39,380][root][INFO] - ===> Epoch[4463](22315/5): Loss 0.0157, Sem 0.0157 LR: 6.580e-02 Score 99.430 Data time: 0.2343, Forward time: 1.2644, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6535 [2021-07-08 04:16:52,908][root][INFO] - ===> Epoch[4464](22320/5): Loss 0.0156, Sem 0.0156 LR: 6.579e-02 Score 99.437 Data time: 0.2009, Forward time: 1.2644, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.6640 [2021-07-08 04:17:07,134][root][INFO] - ===> Epoch[4465](22325/5): Loss 0.0163, Sem 0.0163 LR: 6.578e-02 Score 99.407 Data time: 0.2413, Forward time: 1.2644, Backward time: 1.2327, DDP time: 0.0042, Total iter time: 2.8019 [2021-07-08 04:17:21,256][root][INFO] - ===> Epoch[4466](22330/5): Loss 0.0163, Sem 0.0163 LR: 6.578e-02 Score 99.410 Data time: 0.1911, Forward time: 1.2643, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7736 [2021-07-08 04:17:35,190][root][INFO] - ===> Epoch[4467](22335/5): Loss 0.0151, Sem 0.0151 LR: 6.577e-02 Score 99.450 Data time: 0.1625, Forward time: 1.2643, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7488 [2021-07-08 04:17:49,132][root][INFO] - ===> Epoch[4468](22340/5): Loss 0.0164, Sem 0.0164 LR: 6.576e-02 Score 99.408 Data time: 0.1945, Forward time: 1.2643, Backward time: 1.2328, DDP time: 0.0042, Total iter time: 2.7414 [2021-07-08 04:18:03,263][root][INFO] - ===> Epoch[4469](22345/5): Loss 0.0154, Sem 0.0154 LR: 6.575e-02 Score 99.442 Data time: 0.2212, Forward time: 1.2642, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.7668 [2021-07-08 04:18:17,890][root][INFO] - ===> Epoch[4470](22350/5): Loss 0.0150, Sem 0.0150 LR: 6.574e-02 Score 99.458 Data time: 0.2720, Forward time: 1.2642, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.8681 [2021-07-08 04:18:31,281][root][INFO] - ===> Epoch[4471](22355/5): Loss 0.0154, Sem 0.0154 LR: 6.574e-02 Score 99.443 Data time: 0.1992, Forward time: 1.2642, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6356 [2021-07-08 04:18:46,113][root][INFO] - ===> Epoch[4472](22360/5): Loss 0.0144, Sem 0.0144 LR: 6.573e-02 Score 99.479 Data time: 0.2171, Forward time: 1.2642, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.9159 [2021-07-08 04:19:01,595][root][INFO] - ===> Epoch[4473](22365/5): Loss 0.0156, Sem 0.0156 LR: 6.572e-02 Score 99.446 Data time: 0.2055, Forward time: 1.2642, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 3.0475 [2021-07-08 04:19:15,788][root][INFO] - ===> Epoch[4474](22370/5): Loss 0.0157, Sem 0.0157 LR: 6.571e-02 Score 99.428 Data time: 0.2404, Forward time: 1.2642, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7932 [2021-07-08 04:19:29,275][root][INFO] - ===> Epoch[4475](22375/5): Loss 0.0160, Sem 0.0160 LR: 6.570e-02 Score 99.421 Data time: 0.2126, Forward time: 1.2642, Backward time: 1.2329, DDP time: 0.0042, Total iter time: 2.6534 [2021-07-08 04:19:43,555][root][INFO] - ===> Epoch[4476](22380/5): Loss 0.0159, Sem 0.0159 LR: 6.570e-02 Score 99.428 Data time: 0.2702, Forward time: 1.2642, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8072 [2021-07-08 04:19:58,328][root][INFO] - ===> Epoch[4477](22385/5): Loss 0.0156, Sem 0.0156 LR: 6.569e-02 Score 99.436 Data time: 0.2328, Forward time: 1.2643, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.9030 [2021-07-08 04:20:13,465][root][INFO] - ===> Epoch[4478](22390/5): Loss 0.0152, Sem 0.0152 LR: 6.568e-02 Score 99.455 Data time: 0.2047, Forward time: 1.2643, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.9857 [2021-07-08 04:20:28,056][root][INFO] - ===> Epoch[4479](22395/5): Loss 0.0156, Sem 0.0156 LR: 6.567e-02 Score 99.438 Data time: 0.2498, Forward time: 1.2643, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8641 [2021-07-08 04:20:42,462][root][INFO] - ===> Epoch[4480](22400/5): Loss 0.0153, Sem 0.0153 LR: 6.567e-02 Score 99.454 Data time: 0.2731, Forward time: 1.2643, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8363 [2021-07-08 04:20:42,463][root][INFO] - ===> Start testing [2021-07-08 04:21:16,658][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1263 Loss 0.380 (AVG: 0.482) Score 91.393 (AVG: 91.458) mIOU 72.049 mAP 85.043 mAcc 78.218 IOU: 64.847 0.000 84.495 77.298 95.000 91.916 35.835 83.904 98.389 78.351 81.257 87.326 58.023 mAP: 84.769 4.879 96.245 89.676 98.797 98.219 63.724 98.255 99.812 91.493 93.015 94.841 91.828 mAcc: 80.056 0.000 92.369 85.814 96.466 96.659 38.257 95.737 99.368 81.551 91.061 96.084 63.419 [2021-07-08 04:21:16,658][root][INFO] - Finished test. Elapsed time: 34.1940 [2021-07-08 04:21:17,743][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 04:21:29,939][root][INFO] - ===> Epoch[4481](22405/5): Loss 0.0155, Sem 0.0155 LR: 6.566e-02 Score 99.442 Data time: 0.1940, Forward time: 1.2642, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.4010 [2021-07-08 04:21:44,312][root][INFO] - ===> Epoch[4482](22410/5): Loss 0.0156, Sem 0.0156 LR: 6.565e-02 Score 99.435 Data time: 0.2000, Forward time: 1.2642, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8347 [2021-07-08 04:21:58,094][root][INFO] - ===> Epoch[4483](22415/5): Loss 0.0157, Sem 0.0157 LR: 6.564e-02 Score 99.436 Data time: 0.2707, Forward time: 1.2642, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7160 [2021-07-08 04:22:11,746][root][INFO] - ===> Epoch[4484](22420/5): Loss 0.0156, Sem 0.0156 LR: 6.563e-02 Score 99.435 Data time: 0.2564, Forward time: 1.2642, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6835 [2021-07-08 04:22:25,835][root][INFO] - ===> Epoch[4485](22425/5): Loss 0.0150, Sem 0.0150 LR: 6.563e-02 Score 99.455 Data time: 0.1839, Forward time: 1.2642, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7727 [2021-07-08 04:22:39,920][root][INFO] - ===> Epoch[4486](22430/5): Loss 0.0149, Sem 0.0149 LR: 6.562e-02 Score 99.464 Data time: 0.2625, Forward time: 1.2642, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7541 [2021-07-08 04:22:54,256][root][INFO] - ===> Epoch[4487](22435/5): Loss 0.0156, Sem 0.0156 LR: 6.561e-02 Score 99.436 Data time: 0.2880, Forward time: 1.2642, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8234 [2021-07-08 04:23:08,550][root][INFO] - ===> Epoch[4488](22440/5): Loss 0.0153, Sem 0.0153 LR: 6.560e-02 Score 99.448 Data time: 0.2816, Forward time: 1.2642, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8158 [2021-07-08 04:23:22,411][root][INFO] - ===> Epoch[4489](22445/5): Loss 0.0156, Sem 0.0156 LR: 6.559e-02 Score 99.444 Data time: 0.2875, Forward time: 1.2642, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7318 [2021-07-08 04:23:35,801][root][INFO] - ===> Epoch[4490](22450/5): Loss 0.0152, Sem 0.0152 LR: 6.559e-02 Score 99.448 Data time: 0.1859, Forward time: 1.2642, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6379 [2021-07-08 04:23:50,238][root][INFO] - ===> Epoch[4491](22455/5): Loss 0.0151, Sem 0.0151 LR: 6.558e-02 Score 99.457 Data time: 0.2276, Forward time: 1.2642, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8386 [2021-07-08 04:24:03,769][root][INFO] - ===> Epoch[4492](22460/5): Loss 0.0156, Sem 0.0156 LR: 6.557e-02 Score 99.441 Data time: 0.1863, Forward time: 1.2642, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6571 [2021-07-08 04:24:18,089][root][INFO] - ===> Epoch[4493](22465/5): Loss 0.0157, Sem 0.0157 LR: 6.556e-02 Score 99.440 Data time: 0.2330, Forward time: 1.2641, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8221 [2021-07-08 04:24:31,611][root][INFO] - ===> Epoch[4494](22470/5): Loss 0.0150, Sem 0.0150 LR: 6.556e-02 Score 99.460 Data time: 0.2335, Forward time: 1.2641, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6644 [2021-07-08 04:24:46,109][root][INFO] - ===> Epoch[4495](22475/5): Loss 0.0155, Sem 0.0155 LR: 6.555e-02 Score 99.440 Data time: 0.2514, Forward time: 1.2642, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8549 [2021-07-08 04:24:59,863][root][INFO] - ===> Epoch[4496](22480/5): Loss 0.0151, Sem 0.0151 LR: 6.554e-02 Score 99.456 Data time: 0.2356, Forward time: 1.2641, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7101 [2021-07-08 04:25:13,770][root][INFO] - ===> Epoch[4497](22485/5): Loss 0.0150, Sem 0.0150 LR: 6.553e-02 Score 99.466 Data time: 0.2019, Forward time: 1.2642, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.7419 [2021-07-08 04:25:28,001][root][INFO] - ===> Epoch[4498](22490/5): Loss 0.0155, Sem 0.0155 LR: 6.552e-02 Score 99.442 Data time: 0.2126, Forward time: 1.2641, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.8036 [2021-07-08 04:25:42,218][root][INFO] - ===> Epoch[4499](22495/5): Loss 0.0161, Sem 0.0161 LR: 6.552e-02 Score 99.432 Data time: 0.2360, Forward time: 1.2641, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7891 [2021-07-08 04:25:55,503][root][INFO] - ===> Epoch[4500](22500/5): Loss 0.0162, Sem 0.0162 LR: 6.551e-02 Score 99.427 Data time: 0.2641, Forward time: 1.2641, Backward time: 1.2330, DDP time: 0.0042, Total iter time: 2.6140 [2021-07-08 04:26:09,687][root][INFO] - ===> Epoch[4501](22505/5): Loss 0.0165, Sem 0.0165 LR: 6.550e-02 Score 99.409 Data time: 0.1937, Forward time: 1.2641, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7906 [2021-07-08 04:26:23,588][root][INFO] - ===> Epoch[4502](22510/5): Loss 0.0164, Sem 0.0164 LR: 6.549e-02 Score 99.409 Data time: 0.2370, Forward time: 1.2641, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7405 [2021-07-08 04:26:37,164][root][INFO] - ===> Epoch[4503](22515/5): Loss 0.0162, Sem 0.0162 LR: 6.548e-02 Score 99.414 Data time: 0.2540, Forward time: 1.2641, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.6671 [2021-07-08 04:26:52,493][root][INFO] - ===> Epoch[4504](22520/5): Loss 0.0162, Sem 0.0162 LR: 6.548e-02 Score 99.424 Data time: 0.2327, Forward time: 1.2641, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 3.0271 [2021-07-08 04:27:06,415][root][INFO] - ===> Epoch[4505](22525/5): Loss 0.0158, Sem 0.0158 LR: 6.547e-02 Score 99.433 Data time: 0.1887, Forward time: 1.2641, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.7443 [2021-07-08 04:27:21,088][root][INFO] - ===> Epoch[4506](22530/5): Loss 0.0156, Sem 0.0156 LR: 6.546e-02 Score 99.440 Data time: 0.2550, Forward time: 1.2641, Backward time: 1.2331, DDP time: 0.0042, Total iter time: 2.8923 [2021-07-08 04:27:34,848][root][INFO] - ===> Epoch[4507](22535/5): Loss 0.0159, Sem 0.0159 LR: 6.545e-02 Score 99.422 Data time: 0.1624, Forward time: 1.2641, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7113 [2021-07-08 04:27:50,255][root][INFO] - ===> Epoch[4508](22540/5): Loss 0.0150, Sem 0.0150 LR: 6.545e-02 Score 99.461 Data time: 0.1992, Forward time: 1.2641, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 3.0273 [2021-07-08 04:28:04,251][root][INFO] - ===> Epoch[4509](22545/5): Loss 0.0159, Sem 0.0159 LR: 6.544e-02 Score 99.428 Data time: 0.2595, Forward time: 1.2641, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7468 [2021-07-08 04:28:19,120][root][INFO] - ===> Epoch[4510](22550/5): Loss 0.0150, Sem 0.0150 LR: 6.543e-02 Score 99.454 Data time: 0.2745, Forward time: 1.2641, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.9301 [2021-07-08 04:28:33,253][root][INFO] - ===> Epoch[4511](22555/5): Loss 0.0161, Sem 0.0161 LR: 6.542e-02 Score 99.438 Data time: 0.2040, Forward time: 1.2641, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7824 [2021-07-08 04:28:47,122][root][INFO] - ===> Epoch[4512](22560/5): Loss 0.0155, Sem 0.0155 LR: 6.541e-02 Score 99.442 Data time: 0.2552, Forward time: 1.2641, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7280 [2021-07-08 04:29:01,039][root][INFO] - ===> Epoch[4513](22565/5): Loss 0.0153, Sem 0.0153 LR: 6.541e-02 Score 99.456 Data time: 0.2044, Forward time: 1.2641, Backward time: 1.2332, DDP time: 0.0042, Total iter time: 2.7299 [2021-07-08 04:29:15,487][root][INFO] - ===> Epoch[4514](22570/5): Loss 0.0153, Sem 0.0153 LR: 6.540e-02 Score 99.445 Data time: 0.1623, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8460 [2021-07-08 04:29:29,395][root][INFO] - ===> Epoch[4515](22575/5): Loss 0.0155, Sem 0.0155 LR: 6.539e-02 Score 99.439 Data time: 0.2296, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7311 [2021-07-08 04:29:43,531][root][INFO] - ===> Epoch[4516](22580/5): Loss 0.0152, Sem 0.0152 LR: 6.538e-02 Score 99.452 Data time: 0.2748, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7732 [2021-07-08 04:29:57,746][root][INFO] - ===> Epoch[4517](22585/5): Loss 0.0140, Sem 0.0140 LR: 6.537e-02 Score 99.489 Data time: 0.1757, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7996 [2021-07-08 04:30:10,976][root][INFO] - ===> Epoch[4518](22590/5): Loss 0.0150, Sem 0.0150 LR: 6.537e-02 Score 99.460 Data time: 0.1910, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.5932 [2021-07-08 04:30:25,615][root][INFO] - ===> Epoch[4519](22595/5): Loss 0.0148, Sem 0.0148 LR: 6.536e-02 Score 99.465 Data time: 0.2071, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8809 [2021-07-08 04:30:39,327][root][INFO] - ===> Epoch[4520](22600/5): Loss 0.0163, Sem 0.0163 LR: 6.535e-02 Score 99.421 Data time: 0.1982, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6838 [2021-07-08 04:30:39,331][root][INFO] - ===> Start testing [2021-07-08 04:31:15,275][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1232 Loss 0.310 (AVG: 0.503) Score 92.743 (AVG: 91.215) mIOU 71.988 mAP 84.869 mAcc 78.089 IOU: 65.215 0.127 83.030 76.324 95.133 92.344 41.635 82.314 98.473 77.971 80.063 86.871 56.338 mAP: 84.565 5.966 93.905 89.169 99.093 98.374 65.874 97.703 99.837 91.218 92.903 94.568 90.123 mAcc: 81.060 1.367 90.431 84.614 96.732 96.287 48.641 92.273 99.274 80.071 89.362 95.870 59.174 [2021-07-08 04:31:15,276][root][INFO] - Finished test. Elapsed time: 35.9433 [2021-07-08 04:31:16,336][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 04:31:29,469][root][INFO] - ===> Epoch[4521](22605/5): Loss 0.0157, Sem 0.0157 LR: 6.534e-02 Score 99.437 Data time: 0.1627, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.5995 [2021-07-08 04:31:42,789][root][INFO] - ===> Epoch[4522](22610/5): Loss 0.0160, Sem 0.0160 LR: 6.534e-02 Score 99.431 Data time: 0.2316, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6253 [2021-07-08 04:31:57,347][root][INFO] - ===> Epoch[4523](22615/5): Loss 0.0151, Sem 0.0151 LR: 6.533e-02 Score 99.463 Data time: 0.1459, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8756 [2021-07-08 04:32:11,403][root][INFO] - ===> Epoch[4524](22620/5): Loss 0.0155, Sem 0.0155 LR: 6.532e-02 Score 99.439 Data time: 0.2218, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7617 [2021-07-08 04:32:25,140][root][INFO] - ===> Epoch[4525](22625/5): Loss 0.0155, Sem 0.0155 LR: 6.531e-02 Score 99.440 Data time: 0.3106, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.6983 [2021-07-08 04:32:39,165][root][INFO] - ===> Epoch[4526](22630/5): Loss 0.0166, Sem 0.0166 LR: 6.530e-02 Score 99.435 Data time: 0.2370, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7576 [2021-07-08 04:32:53,234][root][INFO] - ===> Epoch[4527](22635/5): Loss 0.0182, Sem 0.0182 LR: 6.530e-02 Score 99.383 Data time: 0.2281, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.7557 [2021-07-08 04:33:08,171][root][INFO] - ===> Epoch[4528](22640/5): Loss 0.0165, Sem 0.0165 LR: 6.529e-02 Score 99.412 Data time: 0.2363, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.9375 [2021-07-08 04:33:22,581][root][INFO] - ===> Epoch[4529](22645/5): Loss 0.0165, Sem 0.0165 LR: 6.528e-02 Score 99.412 Data time: 0.2349, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8330 [2021-07-08 04:33:36,889][root][INFO] - ===> Epoch[4530](22650/5): Loss 0.0156, Sem 0.0156 LR: 6.527e-02 Score 99.437 Data time: 0.2196, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8051 [2021-07-08 04:33:51,280][root][INFO] - ===> Epoch[4531](22655/5): Loss 0.0153, Sem 0.0153 LR: 6.526e-02 Score 99.448 Data time: 0.2407, Forward time: 1.2640, Backward time: 1.2333, DDP time: 0.0042, Total iter time: 2.8315 [2021-07-08 04:34:05,745][root][INFO] - ===> Epoch[4532](22660/5): Loss 0.0156, Sem 0.0156 LR: 6.526e-02 Score 99.438 Data time: 0.2120, Forward time: 1.2640, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.8500 [2021-07-08 04:34:19,863][root][INFO] - ===> Epoch[4533](22665/5): Loss 0.0155, Sem 0.0155 LR: 6.525e-02 Score 99.440 Data time: 0.1891, Forward time: 1.2640, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7823 [2021-07-08 04:34:34,002][root][INFO] - ===> Epoch[4534](22670/5): Loss 0.0152, Sem 0.0152 LR: 6.524e-02 Score 99.446 Data time: 0.2054, Forward time: 1.2640, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7767 [2021-07-08 04:34:47,924][root][INFO] - ===> Epoch[4535](22675/5): Loss 0.0150, Sem 0.0150 LR: 6.523e-02 Score 99.454 Data time: 0.2382, Forward time: 1.2640, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7295 [2021-07-08 04:35:01,830][root][INFO] - ===> Epoch[4536](22680/5): Loss 0.0149, Sem 0.0149 LR: 6.523e-02 Score 99.466 Data time: 0.2579, Forward time: 1.2640, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7434 [2021-07-08 04:35:16,480][root][INFO] - ===> Epoch[4537](22685/5): Loss 0.0149, Sem 0.0149 LR: 6.522e-02 Score 99.461 Data time: 0.2431, Forward time: 1.2640, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.8817 [2021-07-08 04:35:30,309][root][INFO] - ===> Epoch[4538](22690/5): Loss 0.0145, Sem 0.0145 LR: 6.521e-02 Score 99.479 Data time: 0.2071, Forward time: 1.2640, Backward time: 1.2334, DDP time: 0.0042, Total iter time: 2.7225 [2021-07-08 04:35:43,902][root][INFO] - ===> Epoch[4539](22695/5): Loss 0.0148, Sem 0.0148 LR: 6.520e-02 Score 99.462 Data time: 0.1913, Forward time: 1.2639, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6791 [2021-07-08 04:35:57,900][root][INFO] - ===> Epoch[4540](22700/5): Loss 0.0150, Sem 0.0150 LR: 6.519e-02 Score 99.459 Data time: 0.2638, Forward time: 1.2639, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7594 [2021-07-08 04:36:11,376][root][INFO] - ===> Epoch[4541](22705/5): Loss 0.0147, Sem 0.0147 LR: 6.519e-02 Score 99.470 Data time: 0.2309, Forward time: 1.2639, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6465 [2021-07-08 04:36:25,764][root][INFO] - ===> Epoch[4542](22710/5): Loss 0.0144, Sem 0.0144 LR: 6.518e-02 Score 99.481 Data time: 0.2110, Forward time: 1.2639, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8311 [2021-07-08 04:36:39,215][root][INFO] - ===> Epoch[4543](22715/5): Loss 0.0144, Sem 0.0144 LR: 6.517e-02 Score 99.480 Data time: 0.1852, Forward time: 1.2639, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6445 [2021-07-08 04:36:52,851][root][INFO] - ===> Epoch[4544](22720/5): Loss 0.0139, Sem 0.0139 LR: 6.516e-02 Score 99.502 Data time: 0.2138, Forward time: 1.2638, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6811 [2021-07-08 04:37:07,934][root][INFO] - ===> Epoch[4545](22725/5): Loss 0.0140, Sem 0.0140 LR: 6.515e-02 Score 99.497 Data time: 0.2673, Forward time: 1.2639, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.9758 [2021-07-08 04:37:22,845][root][INFO] - ===> Epoch[4546](22730/5): Loss 0.0146, Sem 0.0146 LR: 6.515e-02 Score 99.475 Data time: 0.1966, Forward time: 1.2639, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.9429 [2021-07-08 04:37:36,387][root][INFO] - ===> Epoch[4547](22735/5): Loss 0.0144, Sem 0.0144 LR: 6.514e-02 Score 99.474 Data time: 0.1769, Forward time: 1.2639, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6671 [2021-07-08 04:37:50,560][root][INFO] - ===> Epoch[4548](22740/5): Loss 0.0144, Sem 0.0144 LR: 6.513e-02 Score 99.483 Data time: 0.1455, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7971 [2021-07-08 04:38:04,882][root][INFO] - ===> Epoch[4549](22745/5): Loss 0.0139, Sem 0.0139 LR: 6.512e-02 Score 99.499 Data time: 0.1999, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8188 [2021-07-08 04:38:18,434][root][INFO] - ===> Epoch[4550](22750/5): Loss 0.0148, Sem 0.0148 LR: 6.512e-02 Score 99.467 Data time: 0.2100, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6694 [2021-07-08 04:38:33,614][root][INFO] - ===> Epoch[4551](22755/5): Loss 0.0140, Sem 0.0140 LR: 6.511e-02 Score 99.494 Data time: 0.2244, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.9915 [2021-07-08 04:38:47,994][root][INFO] - ===> Epoch[4552](22760/5): Loss 0.0141, Sem 0.0141 LR: 6.510e-02 Score 99.490 Data time: 0.2836, Forward time: 1.2639, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8285 [2021-07-08 04:39:02,317][root][INFO] - ===> Epoch[4553](22765/5): Loss 0.0147, Sem 0.0147 LR: 6.509e-02 Score 99.467 Data time: 0.2993, Forward time: 1.2639, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8130 [2021-07-08 04:39:16,304][root][INFO] - ===> Epoch[4554](22770/5): Loss 0.0138, Sem 0.0138 LR: 6.508e-02 Score 99.502 Data time: 0.2410, Forward time: 1.2639, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7451 [2021-07-08 04:39:30,057][root][INFO] - ===> Epoch[4555](22775/5): Loss 0.0141, Sem 0.0141 LR: 6.508e-02 Score 99.487 Data time: 0.2611, Forward time: 1.2639, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7097 [2021-07-08 04:39:43,870][root][INFO] - ===> Epoch[4556](22780/5): Loss 0.0146, Sem 0.0146 LR: 6.507e-02 Score 99.469 Data time: 0.2117, Forward time: 1.2639, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7143 [2021-07-08 04:39:57,370][root][INFO] - ===> Epoch[4557](22785/5): Loss 0.0144, Sem 0.0144 LR: 6.506e-02 Score 99.475 Data time: 0.1813, Forward time: 1.2639, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6511 [2021-07-08 04:40:11,273][root][INFO] - ===> Epoch[4558](22790/5): Loss 0.0143, Sem 0.0143 LR: 6.505e-02 Score 99.486 Data time: 0.2517, Forward time: 1.2639, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7315 [2021-07-08 04:40:25,703][root][INFO] - ===> Epoch[4559](22795/5): Loss 0.0143, Sem 0.0143 LR: 6.504e-02 Score 99.484 Data time: 0.1661, Forward time: 1.2639, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8478 [2021-07-08 04:40:39,260][root][INFO] - ===> Epoch[4560](22800/5): Loss 0.0143, Sem 0.0143 LR: 6.504e-02 Score 99.481 Data time: 0.2459, Forward time: 1.2639, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6610 [2021-07-08 04:40:39,261][root][INFO] - ===> Start testing [2021-07-08 04:41:16,626][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1377 Loss 0.315 (AVG: 0.529) Score 94.044 (AVG: 91.318) mIOU 71.404 mAP 83.337 mAcc 77.260 IOU: 65.603 0.000 81.657 75.915 95.443 92.161 34.350 83.909 98.636 76.313 79.971 87.070 57.230 mAP: 84.398 3.420 89.331 88.185 98.927 98.138 57.349 98.558 99.769 88.577 92.168 94.482 90.077 mAcc: 83.006 0.000 84.646 82.393 96.783 96.748 37.116 96.259 99.466 79.588 90.847 96.307 61.222 [2021-07-08 04:41:16,626][root][INFO] - Finished test. Elapsed time: 37.3637 [2021-07-08 04:41:17,664][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 04:41:30,723][root][INFO] - ===> Epoch[4561](22805/5): Loss 0.0133, Sem 0.0133 LR: 6.503e-02 Score 99.517 Data time: 0.1988, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.5798 [2021-07-08 04:41:44,530][root][INFO] - ===> Epoch[4562](22810/5): Loss 0.0141, Sem 0.0141 LR: 6.502e-02 Score 99.491 Data time: 0.2436, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7209 [2021-07-08 04:41:58,520][root][INFO] - ===> Epoch[4563](22815/5): Loss 0.0139, Sem 0.0139 LR: 6.501e-02 Score 99.498 Data time: 0.2538, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7603 [2021-07-08 04:42:12,738][root][INFO] - ===> Epoch[4564](22820/5): Loss 0.0139, Sem 0.0139 LR: 6.500e-02 Score 99.499 Data time: 0.2381, Forward time: 1.2639, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7982 [2021-07-08 04:42:25,988][root][INFO] - ===> Epoch[4565](22825/5): Loss 0.0142, Sem 0.0142 LR: 6.500e-02 Score 99.487 Data time: 0.2115, Forward time: 1.2638, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.6003 [2021-07-08 04:42:40,092][root][INFO] - ===> Epoch[4566](22830/5): Loss 0.0143, Sem 0.0143 LR: 6.499e-02 Score 99.485 Data time: 0.1747, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7820 [2021-07-08 04:42:54,828][root][INFO] - ===> Epoch[4567](22835/5): Loss 0.0142, Sem 0.0142 LR: 6.498e-02 Score 99.490 Data time: 0.2105, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8974 [2021-07-08 04:43:08,304][root][INFO] - ===> Epoch[4568](22840/5): Loss 0.0141, Sem 0.0141 LR: 6.497e-02 Score 99.486 Data time: 0.2288, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6563 [2021-07-08 04:43:22,867][root][INFO] - ===> Epoch[4569](22845/5): Loss 0.0141, Sem 0.0141 LR: 6.497e-02 Score 99.493 Data time: 0.1614, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8668 [2021-07-08 04:43:37,259][root][INFO] - ===> Epoch[4570](22850/5): Loss 0.0143, Sem 0.0143 LR: 6.496e-02 Score 99.486 Data time: 0.2508, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8373 [2021-07-08 04:43:51,762][root][INFO] - ===> Epoch[4571](22855/5): Loss 0.0140, Sem 0.0140 LR: 6.495e-02 Score 99.493 Data time: 0.2069, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8533 [2021-07-08 04:44:05,986][root][INFO] - ===> Epoch[4572](22860/5): Loss 0.0139, Sem 0.0139 LR: 6.494e-02 Score 99.495 Data time: 0.2258, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8008 [2021-07-08 04:44:19,959][root][INFO] - ===> Epoch[4573](22865/5): Loss 0.0131, Sem 0.0131 LR: 6.493e-02 Score 99.527 Data time: 0.1442, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7584 [2021-07-08 04:44:34,282][root][INFO] - ===> Epoch[4574](22870/5): Loss 0.0138, Sem 0.0138 LR: 6.493e-02 Score 99.499 Data time: 0.1982, Forward time: 1.2637, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8170 [2021-07-08 04:44:48,172][root][INFO] - ===> Epoch[4575](22875/5): Loss 0.0139, Sem 0.0139 LR: 6.492e-02 Score 99.496 Data time: 0.2453, Forward time: 1.2637, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7365 [2021-07-08 04:45:02,145][root][INFO] - ===> Epoch[4576](22880/5): Loss 0.0138, Sem 0.0138 LR: 6.491e-02 Score 99.499 Data time: 0.2567, Forward time: 1.2637, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7450 [2021-07-08 04:45:15,529][root][INFO] - ===> Epoch[4577](22885/5): Loss 0.0141, Sem 0.0141 LR: 6.490e-02 Score 99.490 Data time: 0.2598, Forward time: 1.2637, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.6335 [2021-07-08 04:45:29,376][root][INFO] - ===> Epoch[4578](22890/5): Loss 0.0145, Sem 0.0145 LR: 6.489e-02 Score 99.478 Data time: 0.2417, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7267 [2021-07-08 04:45:43,875][root][INFO] - ===> Epoch[4579](22895/5): Loss 0.0139, Sem 0.0139 LR: 6.489e-02 Score 99.506 Data time: 0.2243, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8492 [2021-07-08 04:45:57,957][root][INFO] - ===> Epoch[4580](22900/5): Loss 0.0146, Sem 0.0146 LR: 6.488e-02 Score 99.475 Data time: 0.2905, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7698 [2021-07-08 04:46:12,412][root][INFO] - ===> Epoch[4581](22905/5): Loss 0.0138, Sem 0.0138 LR: 6.487e-02 Score 99.496 Data time: 0.1949, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8462 [2021-07-08 04:46:26,349][root][INFO] - ===> Epoch[4582](22910/5): Loss 0.0140, Sem 0.0140 LR: 6.486e-02 Score 99.492 Data time: 0.2460, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7363 [2021-07-08 04:46:40,141][root][INFO] - ===> Epoch[4583](22915/5): Loss 0.0130, Sem 0.0130 LR: 6.486e-02 Score 99.525 Data time: 0.2184, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7092 [2021-07-08 04:46:53,771][root][INFO] - ===> Epoch[4584](22920/5): Loss 0.0139, Sem 0.0139 LR: 6.485e-02 Score 99.497 Data time: 0.2019, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6824 [2021-07-08 04:47:08,022][root][INFO] - ===> Epoch[4585](22925/5): Loss 0.0139, Sem 0.0139 LR: 6.484e-02 Score 99.501 Data time: 0.2411, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8067 [2021-07-08 04:47:21,815][root][INFO] - ===> Epoch[4586](22930/5): Loss 0.0134, Sem 0.0134 LR: 6.483e-02 Score 99.513 Data time: 0.1892, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7146 [2021-07-08 04:47:35,019][root][INFO] - ===> Epoch[4587](22935/5): Loss 0.0141, Sem 0.0141 LR: 6.482e-02 Score 99.492 Data time: 0.2320, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.5896 [2021-07-08 04:47:49,022][root][INFO] - ===> Epoch[4588](22940/5): Loss 0.0136, Sem 0.0136 LR: 6.482e-02 Score 99.514 Data time: 0.1695, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7601 [2021-07-08 04:48:03,483][root][INFO] - ===> Epoch[4589](22945/5): Loss 0.0139, Sem 0.0139 LR: 6.481e-02 Score 99.501 Data time: 0.2079, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8461 [2021-07-08 04:48:17,448][root][INFO] - ===> Epoch[4590](22950/5): Loss 0.0137, Sem 0.0137 LR: 6.480e-02 Score 99.504 Data time: 0.2039, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7485 [2021-07-08 04:48:31,248][root][INFO] - ===> Epoch[4591](22955/5): Loss 0.0141, Sem 0.0141 LR: 6.479e-02 Score 99.491 Data time: 0.2345, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7059 [2021-07-08 04:48:44,852][root][INFO] - ===> Epoch[4592](22960/5): Loss 0.0139, Sem 0.0139 LR: 6.478e-02 Score 99.500 Data time: 0.2302, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6704 [2021-07-08 04:48:58,059][root][INFO] - ===> Epoch[4593](22965/5): Loss 0.0141, Sem 0.0141 LR: 6.478e-02 Score 99.491 Data time: 0.2692, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.5885 [2021-07-08 04:49:12,742][root][INFO] - ===> Epoch[4594](22970/5): Loss 0.0139, Sem 0.0139 LR: 6.477e-02 Score 99.500 Data time: 0.2210, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8936 [2021-07-08 04:49:27,214][root][INFO] - ===> Epoch[4595](22975/5): Loss 0.0135, Sem 0.0135 LR: 6.476e-02 Score 99.509 Data time: 0.1816, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8565 [2021-07-08 04:49:41,211][root][INFO] - ===> Epoch[4596](22980/5): Loss 0.0143, Sem 0.0143 LR: 6.475e-02 Score 99.482 Data time: 0.2740, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7564 [2021-07-08 04:49:55,209][root][INFO] - ===> Epoch[4597](22985/5): Loss 0.0139, Sem 0.0139 LR: 6.475e-02 Score 99.495 Data time: 0.2409, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7554 [2021-07-08 04:50:08,508][root][INFO] - ===> Epoch[4598](22990/5): Loss 0.0142, Sem 0.0142 LR: 6.474e-02 Score 99.488 Data time: 0.2282, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6017 [2021-07-08 04:50:22,614][root][INFO] - ===> Epoch[4599](22995/5): Loss 0.0139, Sem 0.0139 LR: 6.473e-02 Score 99.500 Data time: 0.1848, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7692 [2021-07-08 04:50:35,756][root][INFO] - ===> Epoch[4600](23000/5): Loss 0.0138, Sem 0.0138 LR: 6.472e-02 Score 99.500 Data time: 0.1642, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.5963 [2021-07-08 04:50:35,758][root][INFO] - ===> Start testing [2021-07-08 04:51:12,495][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1370 Loss 0.285 (AVG: 0.525) Score 93.780 (AVG: 91.333) mIOU 71.435 mAP 84.149 mAcc 76.951 IOU: 65.680 0.000 84.438 77.328 95.069 92.253 31.169 84.973 98.562 75.563 80.069 86.661 56.891 mAP: 84.698 8.386 91.284 89.756 99.061 98.209 57.824 98.317 99.786 88.946 92.383 94.817 90.470 mAcc: 81.713 0.000 88.025 85.018 96.225 96.186 33.141 93.624 99.565 77.457 91.598 96.426 61.379 [2021-07-08 04:51:12,495][root][INFO] - Finished test. Elapsed time: 36.7368 [2021-07-08 04:51:13,602][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 04:51:27,298][root][INFO] - ===> Epoch[4601](23005/5): Loss 0.0137, Sem 0.0137 LR: 6.471e-02 Score 99.506 Data time: 0.2201, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7006 [2021-07-08 04:51:41,940][root][INFO] - ===> Epoch[4602](23010/5): Loss 0.0134, Sem 0.0134 LR: 6.471e-02 Score 99.511 Data time: 0.2403, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8793 [2021-07-08 04:51:55,483][root][INFO] - ===> Epoch[4603](23015/5): Loss 0.0134, Sem 0.0134 LR: 6.470e-02 Score 99.514 Data time: 0.2609, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6576 [2021-07-08 04:52:09,808][root][INFO] - ===> Epoch[4604](23020/5): Loss 0.0132, Sem 0.0132 LR: 6.469e-02 Score 99.519 Data time: 0.3243, Forward time: 1.2637, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8124 [2021-07-08 04:52:24,256][root][INFO] - ===> Epoch[4605](23025/5): Loss 0.0133, Sem 0.0133 LR: 6.468e-02 Score 99.517 Data time: 0.2347, Forward time: 1.2637, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8446 [2021-07-08 04:52:38,251][root][INFO] - ===> Epoch[4606](23030/5): Loss 0.0131, Sem 0.0131 LR: 6.467e-02 Score 99.526 Data time: 0.2293, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7559 [2021-07-08 04:52:52,504][root][INFO] - ===> Epoch[4607](23035/5): Loss 0.0129, Sem 0.0129 LR: 6.467e-02 Score 99.542 Data time: 0.3149, Forward time: 1.2637, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8034 [2021-07-08 04:53:07,061][root][INFO] - ===> Epoch[4608](23040/5): Loss 0.0137, Sem 0.0137 LR: 6.466e-02 Score 99.504 Data time: 0.2175, Forward time: 1.2637, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8675 [2021-07-08 04:53:21,394][root][INFO] - ===> Epoch[4609](23045/5): Loss 0.0130, Sem 0.0130 LR: 6.465e-02 Score 99.535 Data time: 0.1736, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8235 [2021-07-08 04:53:35,354][root][INFO] - ===> Epoch[4610](23050/5): Loss 0.0136, Sem 0.0136 LR: 6.464e-02 Score 99.511 Data time: 0.1616, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7494 [2021-07-08 04:53:49,641][root][INFO] - ===> Epoch[4611](23055/5): Loss 0.0131, Sem 0.0131 LR: 6.464e-02 Score 99.524 Data time: 0.2175, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8116 [2021-07-08 04:54:03,291][root][INFO] - ===> Epoch[4612](23060/5): Loss 0.0139, Sem 0.0139 LR: 6.463e-02 Score 99.504 Data time: 0.2427, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6823 [2021-07-08 04:54:16,917][root][INFO] - ===> Epoch[4613](23065/5): Loss 0.0137, Sem 0.0137 LR: 6.462e-02 Score 99.508 Data time: 0.1785, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6866 [2021-07-08 04:54:30,839][root][INFO] - ===> Epoch[4614](23070/5): Loss 0.0130, Sem 0.0130 LR: 6.461e-02 Score 99.530 Data time: 0.2457, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7390 [2021-07-08 04:54:44,501][root][INFO] - ===> Epoch[4615](23075/5): Loss 0.0138, Sem 0.0138 LR: 6.460e-02 Score 99.500 Data time: 0.2848, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6844 [2021-07-08 04:54:58,090][root][INFO] - ===> Epoch[4616](23080/5): Loss 0.0140, Sem 0.0140 LR: 6.460e-02 Score 99.500 Data time: 0.2662, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6684 [2021-07-08 04:55:12,026][root][INFO] - ===> Epoch[4617](23085/5): Loss 0.0142, Sem 0.0142 LR: 6.459e-02 Score 99.488 Data time: 0.1870, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7445 [2021-07-08 04:55:25,517][root][INFO] - ===> Epoch[4618](23090/5): Loss 0.0134, Sem 0.0134 LR: 6.458e-02 Score 99.516 Data time: 0.1813, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6554 [2021-07-08 04:55:38,886][root][INFO] - ===> Epoch[4619](23095/5): Loss 0.0138, Sem 0.0138 LR: 6.457e-02 Score 99.503 Data time: 0.2020, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6283 [2021-07-08 04:55:53,005][root][INFO] - ===> Epoch[4620](23100/5): Loss 0.0132, Sem 0.0132 LR: 6.456e-02 Score 99.522 Data time: 0.2462, Forward time: 1.2638, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7625 [2021-07-08 04:56:07,387][root][INFO] - ===> Epoch[4621](23105/5): Loss 0.0135, Sem 0.0135 LR: 6.456e-02 Score 99.517 Data time: 0.2741, Forward time: 1.2638, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8271 [2021-07-08 04:56:21,253][root][INFO] - ===> Epoch[4622](23110/5): Loss 0.0136, Sem 0.0136 LR: 6.455e-02 Score 99.515 Data time: 0.1404, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7389 [2021-07-08 04:56:35,321][root][INFO] - ===> Epoch[4623](23115/5): Loss 0.0135, Sem 0.0135 LR: 6.454e-02 Score 99.515 Data time: 0.1719, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7723 [2021-07-08 04:56:48,735][root][INFO] - ===> Epoch[4624](23120/5): Loss 0.0135, Sem 0.0135 LR: 6.453e-02 Score 99.509 Data time: 0.2604, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6381 [2021-07-08 04:57:02,961][root][INFO] - ===> Epoch[4625](23125/5): Loss 0.0133, Sem 0.0133 LR: 6.452e-02 Score 99.528 Data time: 0.2583, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8008 [2021-07-08 04:57:16,955][root][INFO] - ===> Epoch[4626](23130/5): Loss 0.0136, Sem 0.0136 LR: 6.452e-02 Score 99.506 Data time: 0.1960, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7482 [2021-07-08 04:57:30,916][root][INFO] - ===> Epoch[4627](23135/5): Loss 0.0136, Sem 0.0136 LR: 6.451e-02 Score 99.511 Data time: 0.2466, Forward time: 1.2638, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7423 [2021-07-08 04:57:45,610][root][INFO] - ===> Epoch[4628](23140/5): Loss 0.0138, Sem 0.0138 LR: 6.450e-02 Score 99.497 Data time: 0.2970, Forward time: 1.2638, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8873 [2021-07-08 04:57:59,842][root][INFO] - ===> Epoch[4629](23145/5): Loss 0.0134, Sem 0.0134 LR: 6.449e-02 Score 99.518 Data time: 0.2405, Forward time: 1.2639, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.7986 [2021-07-08 04:58:14,519][root][INFO] - ===> Epoch[4630](23150/5): Loss 0.0134, Sem 0.0134 LR: 6.449e-02 Score 99.513 Data time: 0.2331, Forward time: 1.2639, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8976 [2021-07-08 04:58:29,243][root][INFO] - ===> Epoch[4631](23155/5): Loss 0.0137, Sem 0.0137 LR: 6.448e-02 Score 99.502 Data time: 0.1890, Forward time: 1.2639, Backward time: 1.2335, DDP time: 0.0042, Total iter time: 2.8914 [2021-07-08 04:58:43,314][root][INFO] - ===> Epoch[4632](23160/5): Loss 0.0137, Sem 0.0137 LR: 6.447e-02 Score 99.513 Data time: 0.1722, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7641 [2021-07-08 04:58:57,088][root][INFO] - ===> Epoch[4633](23165/5): Loss 0.0140, Sem 0.0140 LR: 6.446e-02 Score 99.490 Data time: 0.2222, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7100 [2021-07-08 04:59:10,726][root][INFO] - ===> Epoch[4634](23170/5): Loss 0.0135, Sem 0.0135 LR: 6.445e-02 Score 99.511 Data time: 0.1749, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6876 [2021-07-08 04:59:24,291][root][INFO] - ===> Epoch[4635](23175/5): Loss 0.0132, Sem 0.0132 LR: 6.445e-02 Score 99.520 Data time: 0.2262, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6717 [2021-07-08 04:59:38,224][root][INFO] - ===> Epoch[4636](23180/5): Loss 0.0134, Sem 0.0134 LR: 6.444e-02 Score 99.515 Data time: 0.1835, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7395 [2021-07-08 04:59:52,805][root][INFO] - ===> Epoch[4637](23185/5): Loss 0.0127, Sem 0.0127 LR: 6.443e-02 Score 99.541 Data time: 0.2333, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8682 [2021-07-08 05:00:06,616][root][INFO] - ===> Epoch[4638](23190/5): Loss 0.0130, Sem 0.0130 LR: 6.442e-02 Score 99.538 Data time: 0.2623, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7124 [2021-07-08 05:00:20,848][root][INFO] - ===> Epoch[4639](23195/5): Loss 0.0133, Sem 0.0133 LR: 6.441e-02 Score 99.520 Data time: 0.1971, Forward time: 1.2638, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.8048 [2021-07-08 05:00:35,531][root][INFO] - ===> Epoch[4640](23200/5): Loss 0.0129, Sem 0.0129 LR: 6.441e-02 Score 99.531 Data time: 0.1890, Forward time: 1.2638, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.9000 [2021-07-08 05:00:35,532][root][INFO] - ===> Start testing [2021-07-08 05:01:11,746][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1252 Loss 0.297 (AVG: 0.553) Score 93.825 (AVG: 91.073) mIOU 71.253 mAP 83.763 mAcc 77.109 IOU: 64.606 0.000 84.823 75.570 95.098 92.004 31.936 86.167 98.570 74.339 79.534 86.510 57.125 mAP: 84.059 4.938 91.430 87.208 98.915 98.341 57.957 98.603 99.777 91.125 92.599 94.355 89.607 mAcc: 82.467 0.000 88.077 82.393 96.475 96.608 34.134 94.928 99.598 80.485 89.428 96.084 61.737 [2021-07-08 05:01:11,746][root][INFO] - Finished test. Elapsed time: 36.2123 [2021-07-08 05:01:12,799][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 05:01:25,789][root][INFO] - ===> Epoch[4641](23205/5): Loss 0.0132, Sem 0.0132 LR: 6.440e-02 Score 99.518 Data time: 0.1867, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.5512 [2021-07-08 05:01:40,658][root][INFO] - ===> Epoch[4642](23210/5): Loss 0.0132, Sem 0.0132 LR: 6.439e-02 Score 99.526 Data time: 0.1824, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.9373 [2021-07-08 05:01:54,823][root][INFO] - ===> Epoch[4643](23215/5): Loss 0.0130, Sem 0.0130 LR: 6.438e-02 Score 99.534 Data time: 0.2218, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7874 [2021-07-08 05:02:08,895][root][INFO] - ===> Epoch[4644](23220/5): Loss 0.0140, Sem 0.0140 LR: 6.438e-02 Score 99.501 Data time: 0.2618, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7716 [2021-07-08 05:02:22,806][root][INFO] - ===> Epoch[4645](23225/5): Loss 0.0155, Sem 0.0155 LR: 6.437e-02 Score 99.469 Data time: 0.2279, Forward time: 1.2637, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7390 [2021-07-08 05:02:36,582][root][INFO] - ===> Epoch[4646](23230/5): Loss 0.0163, Sem 0.0163 LR: 6.436e-02 Score 99.444 Data time: 0.1924, Forward time: 1.2637, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7087 [2021-07-08 05:02:51,134][root][INFO] - ===> Epoch[4647](23235/5): Loss 0.0232, Sem 0.0232 LR: 6.435e-02 Score 99.253 Data time: 0.2685, Forward time: 1.2637, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8615 [2021-07-08 05:03:04,739][root][INFO] - ===> Epoch[4648](23240/5): Loss 0.0330, Sem 0.0330 LR: 6.434e-02 Score 98.945 Data time: 0.2056, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6772 [2021-07-08 05:03:17,822][root][INFO] - ===> Epoch[4649](23245/5): Loss 0.0452, Sem 0.0452 LR: 6.434e-02 Score 98.559 Data time: 0.2152, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.5714 [2021-07-08 05:03:31,578][root][INFO] - ===> Epoch[4650](23250/5): Loss 0.0484, Sem 0.0484 LR: 6.433e-02 Score 98.451 Data time: 0.2581, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7028 [2021-07-08 05:03:45,724][root][INFO] - ===> Epoch[4651](23255/5): Loss 0.0624, Sem 0.0624 LR: 6.432e-02 Score 98.019 Data time: 0.2356, Forward time: 1.2637, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.7793 [2021-07-08 05:03:59,401][root][INFO] - ===> Epoch[4652](23260/5): Loss 0.0547, Sem 0.0547 LR: 6.431e-02 Score 98.245 Data time: 0.1905, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6945 [2021-07-08 05:04:14,522][root][INFO] - ===> Epoch[4653](23265/5): Loss 0.0528, Sem 0.0528 LR: 6.430e-02 Score 98.307 Data time: 0.2646, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.9782 [2021-07-08 05:04:29,072][root][INFO] - ===> Epoch[4654](23270/5): Loss 0.0590, Sem 0.0590 LR: 6.430e-02 Score 98.118 Data time: 0.2789, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.8540 [2021-07-08 05:04:43,271][root][INFO] - ===> Epoch[4655](23275/5): Loss 0.0525, Sem 0.0525 LR: 6.429e-02 Score 98.257 Data time: 0.3051, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7911 [2021-07-08 05:04:57,239][root][INFO] - ===> Epoch[4656](23280/5): Loss 0.0563, Sem 0.0563 LR: 6.428e-02 Score 98.144 Data time: 0.2690, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7529 [2021-07-08 05:05:11,281][root][INFO] - ===> Epoch[4657](23285/5): Loss 0.0537, Sem 0.0537 LR: 6.427e-02 Score 98.249 Data time: 0.2781, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7559 [2021-07-08 05:05:24,654][root][INFO] - ===> Epoch[4658](23290/5): Loss 0.0562, Sem 0.0562 LR: 6.426e-02 Score 98.223 Data time: 0.2434, Forward time: 1.2637, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6321 [2021-07-08 05:05:38,360][root][INFO] - ===> Epoch[4659](23295/5): Loss 0.0537, Sem 0.0537 LR: 6.426e-02 Score 98.214 Data time: 0.1977, Forward time: 1.2637, Backward time: 1.2336, DDP time: 0.0042, Total iter time: 2.6904 [2021-07-08 05:05:53,552][root][INFO] - ===> Epoch[4660](23300/5): Loss 0.0553, Sem 0.0553 LR: 6.425e-02 Score 98.216 Data time: 0.2521, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.9849 [2021-07-08 05:06:07,238][root][INFO] - ===> Epoch[4661](23305/5): Loss 0.0484, Sem 0.0484 LR: 6.424e-02 Score 98.422 Data time: 0.2187, Forward time: 1.2637, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6808 [2021-07-08 05:06:20,648][root][INFO] - ===> Epoch[4662](23310/5): Loss 0.0433, Sem 0.0433 LR: 6.423e-02 Score 98.512 Data time: 0.2384, Forward time: 1.2636, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6403 [2021-07-08 05:06:34,804][root][INFO] - ===> Epoch[4663](23315/5): Loss 0.0412, Sem 0.0412 LR: 6.423e-02 Score 98.609 Data time: 0.2128, Forward time: 1.2636, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7845 [2021-07-08 05:06:49,003][root][INFO] - ===> Epoch[4664](23320/5): Loss 0.0374, Sem 0.0374 LR: 6.422e-02 Score 98.715 Data time: 0.2271, Forward time: 1.2636, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7850 [2021-07-08 05:07:03,797][root][INFO] - ===> Epoch[4665](23325/5): Loss 0.0365, Sem 0.0365 LR: 6.421e-02 Score 98.742 Data time: 0.2475, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.9195 [2021-07-08 05:07:16,918][root][INFO] - ===> Epoch[4666](23330/5): Loss 0.0322, Sem 0.0322 LR: 6.420e-02 Score 98.875 Data time: 0.1840, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.5803 [2021-07-08 05:07:30,568][root][INFO] - ===> Epoch[4667](23335/5): Loss 0.0308, Sem 0.0308 LR: 6.419e-02 Score 98.941 Data time: 0.2228, Forward time: 1.2636, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6721 [2021-07-08 05:07:43,953][root][INFO] - ===> Epoch[4668](23340/5): Loss 0.0274, Sem 0.0274 LR: 6.419e-02 Score 99.040 Data time: 0.1805, Forward time: 1.2636, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.6337 [2021-07-08 05:07:59,212][root][INFO] - ===> Epoch[4669](23345/5): Loss 0.0259, Sem 0.0259 LR: 6.418e-02 Score 99.097 Data time: 0.2006, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 3.0034 [2021-07-08 05:08:12,438][root][INFO] - ===> Epoch[4670](23350/5): Loss 0.0295, Sem 0.0295 LR: 6.417e-02 Score 98.963 Data time: 0.1811, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.6020 [2021-07-08 05:08:26,216][root][INFO] - ===> Epoch[4671](23355/5): Loss 0.0277, Sem 0.0277 LR: 6.416e-02 Score 99.016 Data time: 0.2453, Forward time: 1.2636, Backward time: 1.2337, DDP time: 0.0042, Total iter time: 2.7018 [2021-07-08 05:08:40,038][root][INFO] - ===> Epoch[4672](23360/5): Loss 0.0278, Sem 0.0278 LR: 6.415e-02 Score 99.039 Data time: 0.1749, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7228 [2021-07-08 05:08:54,591][root][INFO] - ===> Epoch[4673](23365/5): Loss 0.0267, Sem 0.0267 LR: 6.415e-02 Score 99.070 Data time: 0.2646, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8556 [2021-07-08 05:09:08,771][root][INFO] - ===> Epoch[4674](23370/5): Loss 0.0247, Sem 0.0247 LR: 6.414e-02 Score 99.124 Data time: 0.2316, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7887 [2021-07-08 05:09:22,383][root][INFO] - ===> Epoch[4675](23375/5): Loss 0.0264, Sem 0.0264 LR: 6.413e-02 Score 99.065 Data time: 0.2506, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.6836 [2021-07-08 05:09:36,810][root][INFO] - ===> Epoch[4676](23380/5): Loss 0.0257, Sem 0.0257 LR: 6.412e-02 Score 99.101 Data time: 0.2426, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8392 [2021-07-08 05:09:50,916][root][INFO] - ===> Epoch[4677](23385/5): Loss 0.0259, Sem 0.0259 LR: 6.412e-02 Score 99.097 Data time: 0.2013, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7738 [2021-07-08 05:10:04,675][root][INFO] - ===> Epoch[4678](23390/5): Loss 0.0299, Sem 0.0299 LR: 6.411e-02 Score 99.019 Data time: 0.1796, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7026 [2021-07-08 05:10:19,155][root][INFO] - ===> Epoch[4679](23395/5): Loss 0.0277, Sem 0.0277 LR: 6.410e-02 Score 99.072 Data time: 0.2679, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8374 [2021-07-08 05:10:33,491][root][INFO] - ===> Epoch[4680](23400/5): Loss 0.0313, Sem 0.0313 LR: 6.409e-02 Score 98.947 Data time: 0.2818, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8181 [2021-07-08 05:10:33,494][root][INFO] - ===> Start testing [2021-07-08 05:11:11,565][root][INFO] - 68/68: Data time: 0.0364, Iter time: 0.1371 Loss 0.412 (AVG: 0.495) Score 90.410 (AVG: 90.110) mIOU 69.039 mAP 84.537 mAcc 76.746 IOU: 61.543 0.128 78.778 73.752 94.795 89.812 35.726 76.286 97.886 70.877 78.790 86.446 52.684 mAP: 84.540 16.240 92.381 86.681 98.920 98.073 68.882 97.457 99.765 82.286 93.079 93.989 86.682 mAcc: 81.976 1.823 91.319 79.676 96.196 96.839 42.064 95.772 98.905 73.954 89.023 94.669 55.484 [2021-07-08 05:11:11,566][root][INFO] - Finished test. Elapsed time: 38.0712 [2021-07-08 05:11:12,674][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 05:11:25,854][root][INFO] - ===> Epoch[4681](23405/5): Loss 0.0273, Sem 0.0273 LR: 6.408e-02 Score 99.047 Data time: 0.1480, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.6050 [2021-07-08 05:11:40,052][root][INFO] - ===> Epoch[4682](23410/5): Loss 0.0286, Sem 0.0286 LR: 6.408e-02 Score 98.999 Data time: 0.1834, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8018 [2021-07-08 05:11:54,233][root][INFO] - ===> Epoch[4683](23415/5): Loss 0.0263, Sem 0.0263 LR: 6.407e-02 Score 99.081 Data time: 0.2607, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8003 [2021-07-08 05:12:08,447][root][INFO] - ===> Epoch[4684](23420/5): Loss 0.0254, Sem 0.0254 LR: 6.406e-02 Score 99.114 Data time: 0.2322, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7867 [2021-07-08 05:12:22,414][root][INFO] - ===> Epoch[4685](23425/5): Loss 0.0246, Sem 0.0246 LR: 6.405e-02 Score 99.141 Data time: 0.2141, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7485 [2021-07-08 05:12:36,349][root][INFO] - ===> Epoch[4686](23430/5): Loss 0.0227, Sem 0.0227 LR: 6.404e-02 Score 99.199 Data time: 0.2281, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7472 [2021-07-08 05:12:51,338][root][INFO] - ===> Epoch[4687](23435/5): Loss 0.0232, Sem 0.0232 LR: 6.404e-02 Score 99.181 Data time: 0.2212, Forward time: 1.2636, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.9458 [2021-07-08 05:13:04,564][root][INFO] - ===> Epoch[4688](23440/5): Loss 0.0273, Sem 0.0273 LR: 6.403e-02 Score 99.114 Data time: 0.1984, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.5981 [2021-07-08 05:13:17,863][root][INFO] - ===> Epoch[4689](23445/5): Loss 0.0235, Sem 0.0235 LR: 6.402e-02 Score 99.182 Data time: 0.2003, Forward time: 1.2635, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.6105 [2021-07-08 05:13:31,154][root][INFO] - ===> Epoch[4690](23450/5): Loss 0.0239, Sem 0.0239 LR: 6.401e-02 Score 99.186 Data time: 0.2502, Forward time: 1.2635, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.6006 [2021-07-08 05:13:44,765][root][INFO] - ===> Epoch[4691](23455/5): Loss 0.0231, Sem 0.0231 LR: 6.400e-02 Score 99.199 Data time: 0.2820, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.6715 [2021-07-08 05:13:59,201][root][INFO] - ===> Epoch[4692](23460/5): Loss 0.0224, Sem 0.0224 LR: 6.400e-02 Score 99.229 Data time: 0.2103, Forward time: 1.2635, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8432 [2021-07-08 05:14:13,064][root][INFO] - ===> Epoch[4693](23465/5): Loss 0.0225, Sem 0.0225 LR: 6.399e-02 Score 99.206 Data time: 0.2186, Forward time: 1.2635, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7232 [2021-07-08 05:14:27,463][root][INFO] - ===> Epoch[4694](23470/5): Loss 0.0205, Sem 0.0205 LR: 6.398e-02 Score 99.271 Data time: 0.1945, Forward time: 1.2636, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.8334 [2021-07-08 05:14:41,284][root][INFO] - ===> Epoch[4695](23475/5): Loss 0.0200, Sem 0.0200 LR: 6.397e-02 Score 99.281 Data time: 0.1804, Forward time: 1.2635, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7187 [2021-07-08 05:14:55,120][root][INFO] - ===> Epoch[4696](23480/5): Loss 0.0202, Sem 0.0202 LR: 6.397e-02 Score 99.278 Data time: 0.2797, Forward time: 1.2635, Backward time: 1.2338, DDP time: 0.0042, Total iter time: 2.7202 [2021-07-08 05:15:09,926][root][INFO] - ===> Epoch[4697](23485/5): Loss 0.0190, Sem 0.0190 LR: 6.396e-02 Score 99.349 Data time: 0.2241, Forward time: 1.2635, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.9188 [2021-07-08 05:15:24,434][root][INFO] - ===> Epoch[4698](23490/5): Loss 0.0199, Sem 0.0199 LR: 6.395e-02 Score 99.287 Data time: 0.1689, Forward time: 1.2635, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8559 [2021-07-08 05:15:38,376][root][INFO] - ===> Epoch[4699](23495/5): Loss 0.0200, Sem 0.0200 LR: 6.394e-02 Score 99.294 Data time: 0.2199, Forward time: 1.2635, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.7450 [2021-07-08 05:15:52,744][root][INFO] - ===> Epoch[4700](23500/5): Loss 0.0201, Sem 0.0201 LR: 6.393e-02 Score 99.294 Data time: 0.2305, Forward time: 1.2635, Backward time: 1.2339, DDP time: 0.0042, Total iter time: 2.8246 [2021-07-08 05:16:06,816][root][INFO] - ===> Epoch[4701](23505/5): Loss 0.0213, Sem 0.0213 LR: 6.393e-02 Score 99.256 Data time: 0.2117, Forward time: 1.2634, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.7714 [2021-07-08 05:16:22,018][root][INFO] - ===> Epoch[4702](23510/5): Loss 0.0207, Sem 0.0207 LR: 6.392e-02 Score 99.262 Data time: 0.2285, Forward time: 1.2634, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.9969 [2021-07-08 05:16:35,878][root][INFO] - ===> Epoch[4703](23515/5): Loss 0.0193, Sem 0.0193 LR: 6.391e-02 Score 99.312 Data time: 0.2273, Forward time: 1.2634, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.7317 [2021-07-08 05:16:49,704][root][INFO] - ===> Epoch[4704](23520/5): Loss 0.0188, Sem 0.0188 LR: 6.390e-02 Score 99.320 Data time: 0.3035, Forward time: 1.2634, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.7228 [2021-07-08 05:17:03,315][root][INFO] - ===> Epoch[4705](23525/5): Loss 0.0194, Sem 0.0194 LR: 6.389e-02 Score 99.310 Data time: 0.2769, Forward time: 1.2634, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.6715 [2021-07-08 05:17:16,941][root][INFO] - ===> Epoch[4706](23530/5): Loss 0.0186, Sem 0.0186 LR: 6.389e-02 Score 99.334 Data time: 0.2245, Forward time: 1.2634, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.6778 [2021-07-08 05:17:31,486][root][INFO] - ===> Epoch[4707](23535/5): Loss 0.0172, Sem 0.0172 LR: 6.388e-02 Score 99.384 Data time: 0.2282, Forward time: 1.2634, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.8650 [2021-07-08 05:17:45,700][root][INFO] - ===> Epoch[4708](23540/5): Loss 0.0172, Sem 0.0172 LR: 6.387e-02 Score 99.384 Data time: 0.1711, Forward time: 1.2634, Backward time: 1.2340, DDP time: 0.0042, Total iter time: 2.8036 [2021-07-08 05:18:00,503][root][INFO] - ===> Epoch[4709](23545/5): Loss 0.0172, Sem 0.0172 LR: 6.386e-02 Score 99.376 Data time: 0.2082, Forward time: 1.2634, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.9107 [2021-07-08 05:18:14,483][root][INFO] - ===> Epoch[4710](23550/5): Loss 0.0177, Sem 0.0177 LR: 6.386e-02 Score 99.372 Data time: 0.2152, Forward time: 1.2634, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7543 [2021-07-08 05:18:28,273][root][INFO] - ===> Epoch[4711](23555/5): Loss 0.0171, Sem 0.0171 LR: 6.385e-02 Score 99.384 Data time: 0.1438, Forward time: 1.2634, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7201 [2021-07-08 05:18:42,307][root][INFO] - ===> Epoch[4712](23560/5): Loss 0.0173, Sem 0.0173 LR: 6.384e-02 Score 99.384 Data time: 0.2427, Forward time: 1.2633, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7621 [2021-07-08 05:18:56,354][root][INFO] - ===> Epoch[4713](23565/5): Loss 0.0169, Sem 0.0169 LR: 6.383e-02 Score 99.395 Data time: 0.2041, Forward time: 1.2633, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7647 [2021-07-08 05:19:09,590][root][INFO] - ===> Epoch[4714](23570/5): Loss 0.0175, Sem 0.0175 LR: 6.382e-02 Score 99.372 Data time: 0.2309, Forward time: 1.2633, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.6032 [2021-07-08 05:19:23,957][root][INFO] - ===> Epoch[4715](23575/5): Loss 0.0170, Sem 0.0170 LR: 6.382e-02 Score 99.388 Data time: 0.2059, Forward time: 1.2634, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8358 [2021-07-08 05:19:38,187][root][INFO] - ===> Epoch[4716](23580/5): Loss 0.0160, Sem 0.0160 LR: 6.381e-02 Score 99.418 Data time: 0.2317, Forward time: 1.2633, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7932 [2021-07-08 05:19:51,804][root][INFO] - ===> Epoch[4717](23585/5): Loss 0.0175, Sem 0.0175 LR: 6.380e-02 Score 99.390 Data time: 0.2108, Forward time: 1.2633, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.6754 [2021-07-08 05:20:05,868][root][INFO] - ===> Epoch[4718](23590/5): Loss 0.0159, Sem 0.0159 LR: 6.379e-02 Score 99.424 Data time: 0.2432, Forward time: 1.2633, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7733 [2021-07-08 05:20:19,905][root][INFO] - ===> Epoch[4719](23595/5): Loss 0.0165, Sem 0.0165 LR: 6.378e-02 Score 99.409 Data time: 0.2192, Forward time: 1.2633, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7636 [2021-07-08 05:20:34,358][root][INFO] - ===> Epoch[4720](23600/5): Loss 0.0166, Sem 0.0166 LR: 6.378e-02 Score 99.407 Data time: 0.2496, Forward time: 1.2633, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8454 [2021-07-08 05:20:34,359][root][INFO] - ===> Start testing [2021-07-08 05:21:09,363][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1228 Loss 0.376 (AVG: 0.567) Score 91.509 (AVG: 90.549) mIOU 70.275 mAP 84.206 mAcc 76.807 IOU: 64.873 0.000 81.544 75.818 92.730 92.169 25.978 81.496 95.917 79.554 79.259 86.651 57.590 mAP: 85.909 8.770 91.602 88.561 98.115 98.466 62.142 98.175 99.152 84.556 92.982 95.056 91.190 mAcc: 80.860 0.000 85.564 82.429 94.028 96.937 28.496 96.795 99.392 83.416 92.299 96.258 62.021 [2021-07-08 05:21:09,363][root][INFO] - Finished test. Elapsed time: 35.0026 [2021-07-08 05:21:10,441][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 05:21:22,743][root][INFO] - ===> Epoch[4721](23605/5): Loss 0.0160, Sem 0.0160 LR: 6.377e-02 Score 99.426 Data time: 0.1570, Forward time: 1.2633, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.4348 [2021-07-08 05:21:36,656][root][INFO] - ===> Epoch[4722](23610/5): Loss 0.0165, Sem 0.0165 LR: 6.376e-02 Score 99.407 Data time: 0.2090, Forward time: 1.2633, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7342 [2021-07-08 05:21:49,963][root][INFO] - ===> Epoch[4723](23615/5): Loss 0.0163, Sem 0.0163 LR: 6.375e-02 Score 99.409 Data time: 0.2341, Forward time: 1.2633, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.6203 [2021-07-08 05:22:04,256][root][INFO] - ===> Epoch[4724](23620/5): Loss 0.0165, Sem 0.0165 LR: 6.374e-02 Score 99.402 Data time: 0.2180, Forward time: 1.2633, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.8136 [2021-07-08 05:22:18,146][root][INFO] - ===> Epoch[4725](23625/5): Loss 0.0157, Sem 0.0157 LR: 6.374e-02 Score 99.434 Data time: 0.2495, Forward time: 1.2633, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7271 [2021-07-08 05:22:32,216][root][INFO] - ===> Epoch[4726](23630/5): Loss 0.0155, Sem 0.0155 LR: 6.373e-02 Score 99.440 Data time: 0.1602, Forward time: 1.2633, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7755 [2021-07-08 05:22:46,138][root][INFO] - ===> Epoch[4727](23635/5): Loss 0.0156, Sem 0.0156 LR: 6.372e-02 Score 99.437 Data time: 0.1879, Forward time: 1.2632, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.7462 [2021-07-08 05:22:59,930][root][INFO] - ===> Epoch[4728](23640/5): Loss 0.0158, Sem 0.0158 LR: 6.371e-02 Score 99.432 Data time: 0.1623, Forward time: 1.2632, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.7154 [2021-07-08 05:23:14,006][root][INFO] - ===> Epoch[4729](23645/5): Loss 0.0157, Sem 0.0157 LR: 6.371e-02 Score 99.443 Data time: 0.2902, Forward time: 1.2632, Backward time: 1.2341, DDP time: 0.0042, Total iter time: 2.7620 [2021-07-08 05:23:28,859][root][INFO] - ===> Epoch[4730](23650/5): Loss 0.0148, Sem 0.0148 LR: 6.370e-02 Score 99.465 Data time: 0.2339, Forward time: 1.2633, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.9299 [2021-07-08 05:23:43,545][root][INFO] - ===> Epoch[4731](23655/5): Loss 0.0152, Sem 0.0152 LR: 6.369e-02 Score 99.451 Data time: 0.1993, Forward time: 1.2632, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.8913 [2021-07-08 05:23:58,565][root][INFO] - ===> Epoch[4732](23660/5): Loss 0.0146, Sem 0.0146 LR: 6.368e-02 Score 99.469 Data time: 0.2228, Forward time: 1.2632, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.9549 [2021-07-08 05:24:13,230][root][INFO] - ===> Epoch[4733](23665/5): Loss 0.0154, Sem 0.0154 LR: 6.367e-02 Score 99.451 Data time: 0.2470, Forward time: 1.2632, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.8785 [2021-07-08 05:24:26,973][root][INFO] - ===> Epoch[4734](23670/5): Loss 0.0148, Sem 0.0148 LR: 6.367e-02 Score 99.462 Data time: 0.1824, Forward time: 1.2633, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7099 [2021-07-08 05:24:41,286][root][INFO] - ===> Epoch[4735](23675/5): Loss 0.0152, Sem 0.0152 LR: 6.366e-02 Score 99.454 Data time: 0.2297, Forward time: 1.2633, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.8192 [2021-07-08 05:24:55,098][root][INFO] - ===> Epoch[4736](23680/5): Loss 0.0151, Sem 0.0151 LR: 6.365e-02 Score 99.456 Data time: 0.1940, Forward time: 1.2633, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7133 [2021-07-08 05:25:08,945][root][INFO] - ===> Epoch[4737](23685/5): Loss 0.0158, Sem 0.0158 LR: 6.364e-02 Score 99.431 Data time: 0.2420, Forward time: 1.2633, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.7223 [2021-07-08 05:25:22,857][root][INFO] - ===> Epoch[4738](23690/5): Loss 0.0141, Sem 0.0141 LR: 6.363e-02 Score 99.490 Data time: 0.2108, Forward time: 1.2633, Backward time: 1.2342, DDP time: 0.0042, Total iter time: 2.7471 [2021-07-08 05:25:37,134][root][INFO] - ===> Epoch[4739](23695/5): Loss 0.0150, Sem 0.0150 LR: 6.363e-02 Score 99.460 Data time: 0.2057, Forward time: 1.2633, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.8054 [2021-07-08 05:25:51,132][root][INFO] - ===> Epoch[4740](23700/5): Loss 0.0148, Sem 0.0148 LR: 6.362e-02 Score 99.469 Data time: 0.2158, Forward time: 1.2633, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7593 [2021-07-08 05:26:05,258][root][INFO] - ===> Epoch[4741](23705/5): Loss 0.0148, Sem 0.0148 LR: 6.361e-02 Score 99.464 Data time: 0.2225, Forward time: 1.2632, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7860 [2021-07-08 05:26:19,192][root][INFO] - ===> Epoch[4742](23710/5): Loss 0.0151, Sem 0.0151 LR: 6.360e-02 Score 99.452 Data time: 0.2902, Forward time: 1.2632, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7457 [2021-07-08 05:26:33,076][root][INFO] - ===> Epoch[4743](23715/5): Loss 0.0140, Sem 0.0140 LR: 6.359e-02 Score 99.493 Data time: 0.2627, Forward time: 1.2632, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7270 [2021-07-08 05:26:47,488][root][INFO] - ===> Epoch[4744](23720/5): Loss 0.0140, Sem 0.0140 LR: 6.359e-02 Score 99.499 Data time: 0.2535, Forward time: 1.2632, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.8295 [2021-07-08 05:27:01,953][root][INFO] - ===> Epoch[4745](23725/5): Loss 0.0144, Sem 0.0144 LR: 6.358e-02 Score 99.477 Data time: 0.2454, Forward time: 1.2632, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.8397 [2021-07-08 05:27:15,606][root][INFO] - ===> Epoch[4746](23730/5): Loss 0.0145, Sem 0.0145 LR: 6.357e-02 Score 99.477 Data time: 0.2438, Forward time: 1.2632, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.6873 [2021-07-08 05:27:29,897][root][INFO] - ===> Epoch[4747](23735/5): Loss 0.0149, Sem 0.0149 LR: 6.356e-02 Score 99.464 Data time: 0.2744, Forward time: 1.2632, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7977 [2021-07-08 05:27:43,634][root][INFO] - ===> Epoch[4748](23740/5): Loss 0.0137, Sem 0.0137 LR: 6.356e-02 Score 99.504 Data time: 0.1871, Forward time: 1.2632, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.6995 [2021-07-08 05:27:57,307][root][INFO] - ===> Epoch[4749](23745/5): Loss 0.0147, Sem 0.0147 LR: 6.355e-02 Score 99.477 Data time: 0.1976, Forward time: 1.2632, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.6894 [2021-07-08 05:28:11,299][root][INFO] - ===> Epoch[4750](23750/5): Loss 0.0147, Sem 0.0147 LR: 6.354e-02 Score 99.465 Data time: 0.2425, Forward time: 1.2632, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7533 [2021-07-08 05:28:26,134][root][INFO] - ===> Epoch[4751](23755/5): Loss 0.0141, Sem 0.0141 LR: 6.353e-02 Score 99.489 Data time: 0.2306, Forward time: 1.2632, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.9309 [2021-07-08 05:28:39,963][root][INFO] - ===> Epoch[4752](23760/5): Loss 0.0135, Sem 0.0135 LR: 6.352e-02 Score 99.507 Data time: 0.2628, Forward time: 1.2632, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7191 [2021-07-08 05:28:53,348][root][INFO] - ===> Epoch[4753](23765/5): Loss 0.0153, Sem 0.0153 LR: 6.352e-02 Score 99.466 Data time: 0.2483, Forward time: 1.2632, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.6321 [2021-07-08 05:29:06,609][root][INFO] - ===> Epoch[4754](23770/5): Loss 0.0156, Sem 0.0156 LR: 6.351e-02 Score 99.434 Data time: 0.1873, Forward time: 1.2632, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.6183 [2021-07-08 05:29:20,582][root][INFO] - ===> Epoch[4755](23775/5): Loss 0.0185, Sem 0.0185 LR: 6.350e-02 Score 99.379 Data time: 0.1591, Forward time: 1.2631, Backward time: 1.2343, DDP time: 0.0042, Total iter time: 2.7528 [2021-07-08 05:29:35,056][root][INFO] - ===> Epoch[4756](23780/5): Loss 0.0183, Sem 0.0183 LR: 6.349e-02 Score 99.375 Data time: 0.2312, Forward time: 1.2631, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.8499 [2021-07-08 05:29:49,210][root][INFO] - ===> Epoch[4757](23785/5): Loss 0.0190, Sem 0.0190 LR: 6.348e-02 Score 99.338 Data time: 0.2401, Forward time: 1.2631, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.7857 [2021-07-08 05:30:03,931][root][INFO] - ===> Epoch[4758](23790/5): Loss 0.0178, Sem 0.0178 LR: 6.348e-02 Score 99.367 Data time: 0.3225, Forward time: 1.2631, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.8841 [2021-07-08 05:30:18,863][root][INFO] - ===> Epoch[4759](23795/5): Loss 0.0173, Sem 0.0173 LR: 6.347e-02 Score 99.374 Data time: 0.2698, Forward time: 1.2632, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.9386 [2021-07-08 05:30:34,540][root][INFO] - ===> Epoch[4760](23800/5): Loss 0.0194, Sem 0.0194 LR: 6.346e-02 Score 99.328 Data time: 0.2547, Forward time: 1.2632, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 3.0949 [2021-07-08 05:30:34,541][root][INFO] - ===> Start testing [2021-07-08 05:31:11,229][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1358 Loss 0.899 (AVG: 0.563) Score 80.631 (AVG: 90.580) mIOU 69.472 mAP 83.155 mAcc 75.619 IOU: 62.955 0.000 80.979 75.407 94.212 90.092 20.562 85.233 97.332 73.490 80.143 85.833 56.899 mAP: 83.640 3.299 94.654 91.035 98.693 97.911 52.202 97.635 99.471 84.909 93.120 94.350 90.091 mAcc: 75.505 0.000 89.677 87.428 95.671 95.466 21.686 95.104 99.444 77.210 88.606 96.389 60.862 [2021-07-08 05:31:11,229][root][INFO] - Finished test. Elapsed time: 36.6882 [2021-07-08 05:31:12,291][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 05:31:25,019][root][INFO] - ===> Epoch[4761](23805/5): Loss 0.0214, Sem 0.0214 LR: 6.345e-02 Score 99.286 Data time: 0.1786, Forward time: 1.2632, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.5102 [2021-07-08 05:31:39,904][root][INFO] - ===> Epoch[4762](23810/5): Loss 0.0197, Sem 0.0197 LR: 6.345e-02 Score 99.299 Data time: 0.2446, Forward time: 1.2632, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.9338 [2021-07-08 05:31:53,077][root][INFO] - ===> Epoch[4763](23815/5): Loss 0.0231, Sem 0.0231 LR: 6.344e-02 Score 99.210 Data time: 0.1538, Forward time: 1.2632, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.5934 [2021-07-08 05:32:06,767][root][INFO] - ===> Epoch[4764](23820/5): Loss 0.0219, Sem 0.0219 LR: 6.343e-02 Score 99.239 Data time: 0.2006, Forward time: 1.2631, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.6950 [2021-07-08 05:32:20,550][root][INFO] - ===> Epoch[4765](23825/5): Loss 0.0215, Sem 0.0215 LR: 6.342e-02 Score 99.248 Data time: 0.2307, Forward time: 1.2631, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.6988 [2021-07-08 05:32:34,139][root][INFO] - ===> Epoch[4766](23830/5): Loss 0.0208, Sem 0.0208 LR: 6.341e-02 Score 99.260 Data time: 0.2478, Forward time: 1.2631, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.6682 [2021-07-08 05:32:49,090][root][INFO] - ===> Epoch[4767](23835/5): Loss 0.0222, Sem 0.0222 LR: 6.341e-02 Score 99.240 Data time: 0.2245, Forward time: 1.2631, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.9435 [2021-07-08 05:33:02,930][root][INFO] - ===> Epoch[4768](23840/5): Loss 0.0244, Sem 0.0244 LR: 6.340e-02 Score 99.184 Data time: 0.2214, Forward time: 1.2631, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.7297 [2021-07-08 05:33:16,591][root][INFO] - ===> Epoch[4769](23845/5): Loss 0.0266, Sem 0.0266 LR: 6.339e-02 Score 99.080 Data time: 0.2095, Forward time: 1.2631, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.6890 [2021-07-08 05:33:30,811][root][INFO] - ===> Epoch[4770](23850/5): Loss 0.0294, Sem 0.0294 LR: 6.338e-02 Score 99.000 Data time: 0.2191, Forward time: 1.2631, Backward time: 1.2345, DDP time: 0.0042, Total iter time: 2.8005 [2021-07-08 05:33:44,210][root][INFO] - ===> Epoch[4771](23855/5): Loss 0.0290, Sem 0.0290 LR: 6.337e-02 Score 98.999 Data time: 0.2112, Forward time: 1.2631, Backward time: 1.2344, DDP time: 0.0042, Total iter time: 2.6238 [2021-07-08 05:33:58,823][root][INFO] - ===> Epoch[4772](23860/5): Loss 0.0285, Sem 0.0285 LR: 6.337e-02 Score 99.027 Data time: 0.2035, Forward time: 1.2631, Backward time: 1.2345, DDP time: 0.0042, Total iter time: 2.8760 [2021-07-08 05:34:12,748][root][INFO] - ===> Epoch[4773](23865/5): Loss 0.0275, Sem 0.0275 LR: 6.336e-02 Score 99.045 Data time: 0.1899, Forward time: 1.2630, Backward time: 1.2345, DDP time: 0.0042, Total iter time: 2.7284 [2021-07-08 05:34:27,551][root][INFO] - ===> Epoch[4774](23870/5): Loss 0.0289, Sem 0.0289 LR: 6.335e-02 Score 99.000 Data time: 0.1815, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.9124 [2021-07-08 05:34:42,065][root][INFO] - ===> Epoch[4775](23875/5): Loss 0.0301, Sem 0.0301 LR: 6.334e-02 Score 98.975 Data time: 0.2664, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.8493 [2021-07-08 05:34:56,012][root][INFO] - ===> Epoch[4776](23880/5): Loss 0.0260, Sem 0.0260 LR: 6.333e-02 Score 99.088 Data time: 0.2188, Forward time: 1.2630, Backward time: 1.2345, DDP time: 0.0042, Total iter time: 2.7339 [2021-07-08 05:35:10,026][root][INFO] - ===> Epoch[4777](23885/5): Loss 0.0250, Sem 0.0250 LR: 6.333e-02 Score 99.129 Data time: 0.2650, Forward time: 1.2631, Backward time: 1.2345, DDP time: 0.0042, Total iter time: 2.7575 [2021-07-08 05:35:23,975][root][INFO] - ===> Epoch[4778](23890/5): Loss 0.0255, Sem 0.0255 LR: 6.332e-02 Score 99.120 Data time: 0.2440, Forward time: 1.2630, Backward time: 1.2345, DDP time: 0.0042, Total iter time: 2.7514 [2021-07-08 05:35:38,968][root][INFO] - ===> Epoch[4779](23895/5): Loss 0.0231, Sem 0.0231 LR: 6.331e-02 Score 99.187 Data time: 0.1927, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.9570 [2021-07-08 05:35:52,936][root][INFO] - ===> Epoch[4780](23900/5): Loss 0.0210, Sem 0.0210 LR: 6.330e-02 Score 99.252 Data time: 0.2372, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7385 [2021-07-08 05:36:07,472][root][INFO] - ===> Epoch[4781](23905/5): Loss 0.0209, Sem 0.0209 LR: 6.330e-02 Score 99.266 Data time: 0.2358, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.8523 [2021-07-08 05:36:22,410][root][INFO] - ===> Epoch[4782](23910/5): Loss 0.0208, Sem 0.0208 LR: 6.329e-02 Score 99.275 Data time: 0.2618, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.9454 [2021-07-08 05:36:36,790][root][INFO] - ===> Epoch[4783](23915/5): Loss 0.0189, Sem 0.0189 LR: 6.328e-02 Score 99.315 Data time: 0.3031, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.8297 [2021-07-08 05:36:50,430][root][INFO] - ===> Epoch[4784](23920/5): Loss 0.0195, Sem 0.0195 LR: 6.327e-02 Score 99.301 Data time: 0.2637, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.6834 [2021-07-08 05:37:04,690][root][INFO] - ===> Epoch[4785](23925/5): Loss 0.0182, Sem 0.0182 LR: 6.326e-02 Score 99.344 Data time: 0.3017, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7966 [2021-07-08 05:37:18,701][root][INFO] - ===> Epoch[4786](23930/5): Loss 0.0191, Sem 0.0191 LR: 6.326e-02 Score 99.318 Data time: 0.2649, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7450 [2021-07-08 05:37:32,126][root][INFO] - ===> Epoch[4787](23935/5): Loss 0.0175, Sem 0.0175 LR: 6.325e-02 Score 99.377 Data time: 0.2347, Forward time: 1.2631, Backward time: 1.2345, DDP time: 0.0042, Total iter time: 2.6348 [2021-07-08 05:37:46,984][root][INFO] - ===> Epoch[4788](23940/5): Loss 0.0175, Sem 0.0175 LR: 6.324e-02 Score 99.380 Data time: 0.1799, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.9330 [2021-07-08 05:38:01,119][root][INFO] - ===> Epoch[4789](23945/5): Loss 0.0168, Sem 0.0168 LR: 6.323e-02 Score 99.407 Data time: 0.2203, Forward time: 1.2631, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7732 [2021-07-08 05:38:14,504][root][INFO] - ===> Epoch[4790](23950/5): Loss 0.0166, Sem 0.0166 LR: 6.322e-02 Score 99.407 Data time: 0.2362, Forward time: 1.2631, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.6283 [2021-07-08 05:38:28,975][root][INFO] - ===> Epoch[4791](23955/5): Loss 0.0164, Sem 0.0164 LR: 6.322e-02 Score 99.417 Data time: 0.2639, Forward time: 1.2631, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.8452 [2021-07-08 05:38:42,365][root][INFO] - ===> Epoch[4792](23960/5): Loss 0.0169, Sem 0.0169 LR: 6.321e-02 Score 99.396 Data time: 0.1980, Forward time: 1.2631, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.6336 [2021-07-08 05:38:56,614][root][INFO] - ===> Epoch[4793](23965/5): Loss 0.0169, Sem 0.0169 LR: 6.320e-02 Score 99.399 Data time: 0.1759, Forward time: 1.2631, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7982 [2021-07-08 05:39:09,996][root][INFO] - ===> Epoch[4794](23970/5): Loss 0.0163, Sem 0.0163 LR: 6.319e-02 Score 99.415 Data time: 0.1944, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.6306 [2021-07-08 05:39:24,555][root][INFO] - ===> Epoch[4795](23975/5): Loss 0.0172, Sem 0.0172 LR: 6.318e-02 Score 99.399 Data time: 0.2148, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.8660 [2021-07-08 05:39:38,339][root][INFO] - ===> Epoch[4796](23980/5): Loss 0.0155, Sem 0.0155 LR: 6.318e-02 Score 99.444 Data time: 0.2573, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7030 [2021-07-08 05:39:51,554][root][INFO] - ===> Epoch[4797](23985/5): Loss 0.0167, Sem 0.0167 LR: 6.317e-02 Score 99.407 Data time: 0.1885, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.6086 [2021-07-08 05:40:05,379][root][INFO] - ===> Epoch[4798](23990/5): Loss 0.0160, Sem 0.0160 LR: 6.316e-02 Score 99.422 Data time: 0.1756, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7124 [2021-07-08 05:40:19,758][root][INFO] - ===> Epoch[4799](23995/5): Loss 0.0152, Sem 0.0152 LR: 6.315e-02 Score 99.446 Data time: 0.1883, Forward time: 1.2630, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.8437 [2021-07-08 05:40:34,501][root][INFO] - ===> Epoch[4800](24000/5): Loss 0.0157, Sem 0.0157 LR: 6.315e-02 Score 99.440 Data time: 0.2235, Forward time: 1.2630, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.8982 [2021-07-08 05:40:34,502][root][INFO] - ===> Start testing [2021-07-08 05:41:08,474][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1191 Loss 0.211 (AVG: 0.531) Score 94.738 (AVG: 90.983) mIOU 69.726 mAP 84.211 mAcc 76.498 IOU: 65.468 0.000 80.745 74.590 95.247 92.222 24.821 80.619 98.201 69.508 79.717 86.716 58.580 mAP: 85.788 5.945 90.837 87.520 98.947 98.763 56.727 97.835 99.783 93.713 92.538 95.269 91.077 mAcc: 81.800 0.000 85.850 81.358 96.638 96.367 26.750 94.153 99.510 82.468 90.567 96.779 62.239 [2021-07-08 05:41:08,474][root][INFO] - Finished test. Elapsed time: 33.9714 [2021-07-08 05:41:09,530][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 05:41:21,275][root][INFO] - ===> Epoch[4801](24005/5): Loss 0.0155, Sem 0.0155 LR: 6.314e-02 Score 99.442 Data time: 0.1856, Forward time: 1.2629, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.3124 [2021-07-08 05:41:35,394][root][INFO] - ===> Epoch[4802](24010/5): Loss 0.0156, Sem 0.0156 LR: 6.313e-02 Score 99.437 Data time: 0.2145, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7747 [2021-07-08 05:41:49,218][root][INFO] - ===> Epoch[4803](24015/5): Loss 0.0153, Sem 0.0153 LR: 6.312e-02 Score 99.452 Data time: 0.2277, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7184 [2021-07-08 05:42:03,185][root][INFO] - ===> Epoch[4804](24020/5): Loss 0.0150, Sem 0.0150 LR: 6.311e-02 Score 99.456 Data time: 0.2159, Forward time: 1.2629, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7333 [2021-07-08 05:42:17,183][root][INFO] - ===> Epoch[4805](24025/5): Loss 0.0149, Sem 0.0149 LR: 6.311e-02 Score 99.461 Data time: 0.1902, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7605 [2021-07-08 05:42:31,381][root][INFO] - ===> Epoch[4806](24030/5): Loss 0.0143, Sem 0.0143 LR: 6.310e-02 Score 99.482 Data time: 0.2717, Forward time: 1.2629, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7884 [2021-07-08 05:42:45,442][root][INFO] - ===> Epoch[4807](24035/5): Loss 0.0152, Sem 0.0152 LR: 6.309e-02 Score 99.454 Data time: 0.1884, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7736 [2021-07-08 05:43:00,283][root][INFO] - ===> Epoch[4808](24040/5): Loss 0.0144, Sem 0.0144 LR: 6.308e-02 Score 99.485 Data time: 0.2067, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.9323 [2021-07-08 05:43:15,406][root][INFO] - ===> Epoch[4809](24045/5): Loss 0.0153, Sem 0.0153 LR: 6.307e-02 Score 99.451 Data time: 0.2656, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.9752 [2021-07-08 05:43:30,159][root][INFO] - ===> Epoch[4810](24050/5): Loss 0.0159, Sem 0.0159 LR: 6.307e-02 Score 99.431 Data time: 0.2386, Forward time: 1.2630, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.9093 [2021-07-08 05:43:43,994][root][INFO] - ===> Epoch[4811](24055/5): Loss 0.0153, Sem 0.0153 LR: 6.306e-02 Score 99.445 Data time: 0.2009, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7219 [2021-07-08 05:43:58,760][root][INFO] - ===> Epoch[4812](24060/5): Loss 0.0147, Sem 0.0147 LR: 6.305e-02 Score 99.468 Data time: 0.2375, Forward time: 1.2630, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.8977 [2021-07-08 05:44:12,918][root][INFO] - ===> Epoch[4813](24065/5): Loss 0.0148, Sem 0.0148 LR: 6.304e-02 Score 99.465 Data time: 0.2725, Forward time: 1.2630, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7734 [2021-07-08 05:44:26,455][root][INFO] - ===> Epoch[4814](24070/5): Loss 0.0149, Sem 0.0149 LR: 6.303e-02 Score 99.458 Data time: 0.2312, Forward time: 1.2630, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.6693 [2021-07-08 05:44:40,301][root][INFO] - ===> Epoch[4815](24075/5): Loss 0.0147, Sem 0.0147 LR: 6.303e-02 Score 99.470 Data time: 0.2348, Forward time: 1.2630, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7259 [2021-07-08 05:44:53,709][root][INFO] - ===> Epoch[4816](24080/5): Loss 0.0148, Sem 0.0148 LR: 6.302e-02 Score 99.467 Data time: 0.2548, Forward time: 1.2630, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.6324 [2021-07-08 05:45:07,487][root][INFO] - ===> Epoch[4817](24085/5): Loss 0.0151, Sem 0.0151 LR: 6.301e-02 Score 99.449 Data time: 0.2179, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7065 [2021-07-08 05:45:21,444][root][INFO] - ===> Epoch[4818](24090/5): Loss 0.0149, Sem 0.0149 LR: 6.300e-02 Score 99.467 Data time: 0.2381, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7400 [2021-07-08 05:45:35,630][root][INFO] - ===> Epoch[4819](24095/5): Loss 0.0147, Sem 0.0147 LR: 6.300e-02 Score 99.468 Data time: 0.2052, Forward time: 1.2630, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.8010 [2021-07-08 05:45:49,407][root][INFO] - ===> Epoch[4820](24100/5): Loss 0.0146, Sem 0.0146 LR: 6.299e-02 Score 99.479 Data time: 0.2460, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7060 [2021-07-08 05:46:03,616][root][INFO] - ===> Epoch[4821](24105/5): Loss 0.0163, Sem 0.0163 LR: 6.298e-02 Score 99.448 Data time: 0.2475, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7854 [2021-07-08 05:46:17,144][root][INFO] - ===> Epoch[4822](24110/5): Loss 0.0164, Sem 0.0164 LR: 6.297e-02 Score 99.414 Data time: 0.1347, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.6732 [2021-07-08 05:46:30,961][root][INFO] - ===> Epoch[4823](24115/5): Loss 0.0155, Sem 0.0155 LR: 6.296e-02 Score 99.440 Data time: 0.2046, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7120 [2021-07-08 05:46:45,233][root][INFO] - ===> Epoch[4824](24120/5): Loss 0.0151, Sem 0.0151 LR: 6.296e-02 Score 99.457 Data time: 0.2207, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.8124 [2021-07-08 05:46:59,936][root][INFO] - ===> Epoch[4825](24125/5): Loss 0.0148, Sem 0.0148 LR: 6.295e-02 Score 99.466 Data time: 0.2046, Forward time: 1.2630, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.8955 [2021-07-08 05:47:13,717][root][INFO] - ===> Epoch[4826](24130/5): Loss 0.0148, Sem 0.0148 LR: 6.294e-02 Score 99.466 Data time: 0.2499, Forward time: 1.2630, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.6993 [2021-07-08 05:47:27,708][root][INFO] - ===> Epoch[4827](24135/5): Loss 0.0146, Sem 0.0146 LR: 6.293e-02 Score 99.476 Data time: 0.1857, Forward time: 1.2630, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7575 [2021-07-08 05:47:41,530][root][INFO] - ===> Epoch[4828](24140/5): Loss 0.0152, Sem 0.0152 LR: 6.292e-02 Score 99.453 Data time: 0.2100, Forward time: 1.2630, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7159 [2021-07-08 05:47:55,328][root][INFO] - ===> Epoch[4829](24145/5): Loss 0.0143, Sem 0.0143 LR: 6.292e-02 Score 99.484 Data time: 0.2538, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7149 [2021-07-08 05:48:09,336][root][INFO] - ===> Epoch[4830](24150/5): Loss 0.0163, Sem 0.0163 LR: 6.291e-02 Score 99.443 Data time: 0.2144, Forward time: 1.2630, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7606 [2021-07-08 05:48:23,156][root][INFO] - ===> Epoch[4831](24155/5): Loss 0.0153, Sem 0.0153 LR: 6.290e-02 Score 99.454 Data time: 0.2043, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7160 [2021-07-08 05:48:36,574][root][INFO] - ===> Epoch[4832](24160/5): Loss 0.0155, Sem 0.0155 LR: 6.289e-02 Score 99.445 Data time: 0.2229, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.6441 [2021-07-08 05:48:50,571][root][INFO] - ===> Epoch[4833](24165/5): Loss 0.0150, Sem 0.0150 LR: 6.288e-02 Score 99.462 Data time: 0.2259, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7554 [2021-07-08 05:49:05,086][root][INFO] - ===> Epoch[4834](24170/5): Loss 0.0147, Sem 0.0147 LR: 6.288e-02 Score 99.478 Data time: 0.2902, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.8488 [2021-07-08 05:49:19,114][root][INFO] - ===> Epoch[4835](24175/5): Loss 0.0148, Sem 0.0148 LR: 6.287e-02 Score 99.465 Data time: 0.2337, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7553 [2021-07-08 05:49:32,799][root][INFO] - ===> Epoch[4836](24180/5): Loss 0.0143, Sem 0.0143 LR: 6.286e-02 Score 99.491 Data time: 0.3082, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.6934 [2021-07-08 05:49:46,635][root][INFO] - ===> Epoch[4837](24185/5): Loss 0.0150, Sem 0.0150 LR: 6.285e-02 Score 99.463 Data time: 0.2535, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7232 [2021-07-08 05:50:00,661][root][INFO] - ===> Epoch[4838](24190/5): Loss 0.0140, Sem 0.0140 LR: 6.285e-02 Score 99.500 Data time: 0.2670, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7564 [2021-07-08 05:50:14,838][root][INFO] - ===> Epoch[4839](24195/5): Loss 0.0137, Sem 0.0137 LR: 6.284e-02 Score 99.510 Data time: 0.2068, Forward time: 1.2630, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7845 [2021-07-08 05:50:28,542][root][INFO] - ===> Epoch[4840](24200/5): Loss 0.0140, Sem 0.0140 LR: 6.283e-02 Score 99.494 Data time: 0.2318, Forward time: 1.2629, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7018 [2021-07-08 05:50:28,543][root][INFO] - ===> Start testing [2021-07-08 05:51:04,751][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1211 Loss 0.275 (AVG: 0.530) Score 93.427 (AVG: 91.183) mIOU 70.933 mAP 84.659 mAcc 77.522 IOU: 65.727 0.000 80.719 75.325 95.016 92.235 34.360 81.887 97.871 73.308 80.694 87.081 57.912 mAP: 85.904 6.972 92.003 87.780 98.844 98.775 61.995 98.212 99.775 92.211 92.375 95.288 90.435 mAcc: 82.191 0.000 84.419 82.266 96.296 97.060 38.340 96.374 99.480 82.695 90.444 96.318 61.905 [2021-07-08 05:51:04,751][root][INFO] - Finished test. Elapsed time: 36.2078 [2021-07-08 05:51:05,826][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 05:51:18,626][root][INFO] - ===> Epoch[4841](24205/5): Loss 0.0136, Sem 0.0136 LR: 6.282e-02 Score 99.512 Data time: 0.1877, Forward time: 1.2629, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.5205 [2021-07-08 05:51:32,240][root][INFO] - ===> Epoch[4842](24210/5): Loss 0.0142, Sem 0.0142 LR: 6.281e-02 Score 99.489 Data time: 0.2550, Forward time: 1.2629, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.6654 [2021-07-08 05:51:46,889][root][INFO] - ===> Epoch[4843](24215/5): Loss 0.0136, Sem 0.0136 LR: 6.281e-02 Score 99.505 Data time: 0.1871, Forward time: 1.2629, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.8888 [2021-07-08 05:52:00,939][root][INFO] - ===> Epoch[4844](24220/5): Loss 0.0145, Sem 0.0145 LR: 6.280e-02 Score 99.480 Data time: 0.2654, Forward time: 1.2629, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7576 [2021-07-08 05:52:14,916][root][INFO] - ===> Epoch[4845](24225/5): Loss 0.0140, Sem 0.0140 LR: 6.279e-02 Score 99.497 Data time: 0.2100, Forward time: 1.2629, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7614 [2021-07-08 05:52:28,772][root][INFO] - ===> Epoch[4846](24230/5): Loss 0.0130, Sem 0.0130 LR: 6.278e-02 Score 99.533 Data time: 0.1964, Forward time: 1.2629, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7226 [2021-07-08 05:52:43,203][root][INFO] - ===> Epoch[4847](24235/5): Loss 0.0137, Sem 0.0137 LR: 6.277e-02 Score 99.511 Data time: 0.2307, Forward time: 1.2629, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.8339 [2021-07-08 05:52:57,156][root][INFO] - ===> Epoch[4848](24240/5): Loss 0.0146, Sem 0.0146 LR: 6.277e-02 Score 99.474 Data time: 0.1989, Forward time: 1.2629, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7449 [2021-07-08 05:53:11,078][root][INFO] - ===> Epoch[4849](24245/5): Loss 0.0147, Sem 0.0147 LR: 6.276e-02 Score 99.472 Data time: 0.2638, Forward time: 1.2629, Backward time: 1.2346, DDP time: 0.0042, Total iter time: 2.7385 [2021-07-08 05:53:26,813][root][INFO] - ===> Epoch[4850](24250/5): Loss 0.0143, Sem 0.0143 LR: 6.275e-02 Score 99.480 Data time: 0.2402, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 3.0990 [2021-07-08 05:53:40,636][root][INFO] - ===> Epoch[4851](24255/5): Loss 0.0144, Sem 0.0144 LR: 6.274e-02 Score 99.481 Data time: 0.1587, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7196 [2021-07-08 05:53:54,698][root][INFO] - ===> Epoch[4852](24260/5): Loss 0.0139, Sem 0.0139 LR: 6.273e-02 Score 99.497 Data time: 0.2028, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7724 [2021-07-08 05:54:08,336][root][INFO] - ===> Epoch[4853](24265/5): Loss 0.0140, Sem 0.0140 LR: 6.273e-02 Score 99.497 Data time: 0.2212, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.6847 [2021-07-08 05:54:21,933][root][INFO] - ===> Epoch[4854](24270/5): Loss 0.0136, Sem 0.0136 LR: 6.272e-02 Score 99.506 Data time: 0.1837, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.6805 [2021-07-08 05:54:35,616][root][INFO] - ===> Epoch[4855](24275/5): Loss 0.0141, Sem 0.0141 LR: 6.271e-02 Score 99.488 Data time: 0.2069, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.6897 [2021-07-08 05:54:49,494][root][INFO] - ===> Epoch[4856](24280/5): Loss 0.0149, Sem 0.0149 LR: 6.270e-02 Score 99.464 Data time: 0.2034, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7330 [2021-07-08 05:55:03,209][root][INFO] - ===> Epoch[4857](24285/5): Loss 0.0134, Sem 0.0134 LR: 6.270e-02 Score 99.515 Data time: 0.2635, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.6961 [2021-07-08 05:55:16,933][root][INFO] - ===> Epoch[4858](24290/5): Loss 0.0139, Sem 0.0139 LR: 6.269e-02 Score 99.495 Data time: 0.1693, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.6998 [2021-07-08 05:55:31,070][root][INFO] - ===> Epoch[4859](24295/5): Loss 0.0139, Sem 0.0139 LR: 6.268e-02 Score 99.501 Data time: 0.2013, Forward time: 1.2628, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7846 [2021-07-08 05:55:46,479][root][INFO] - ===> Epoch[4860](24300/5): Loss 0.0139, Sem 0.0139 LR: 6.267e-02 Score 99.499 Data time: 0.3126, Forward time: 1.2628, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 3.0361 [2021-07-08 05:56:01,403][root][INFO] - ===> Epoch[4861](24305/5): Loss 0.0138, Sem 0.0138 LR: 6.266e-02 Score 99.502 Data time: 0.2923, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.9162 [2021-07-08 05:56:16,045][root][INFO] - ===> Epoch[4862](24310/5): Loss 0.0138, Sem 0.0138 LR: 6.266e-02 Score 99.506 Data time: 0.2158, Forward time: 1.2628, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 2.8798 [2021-07-08 05:56:29,355][root][INFO] - ===> Epoch[4863](24315/5): Loss 0.0132, Sem 0.0132 LR: 6.265e-02 Score 99.520 Data time: 0.2235, Forward time: 1.2628, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 2.6084 [2021-07-08 05:56:44,215][root][INFO] - ===> Epoch[4864](24320/5): Loss 0.0135, Sem 0.0135 LR: 6.264e-02 Score 99.511 Data time: 0.2578, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.9096 [2021-07-08 05:56:58,418][root][INFO] - ===> Epoch[4865](24325/5): Loss 0.0140, Sem 0.0140 LR: 6.263e-02 Score 99.493 Data time: 0.2186, Forward time: 1.2629, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 2.7953 [2021-07-08 05:57:12,172][root][INFO] - ===> Epoch[4866](24330/5): Loss 0.0132, Sem 0.0132 LR: 6.262e-02 Score 99.525 Data time: 0.2451, Forward time: 1.2629, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 2.7139 [2021-07-08 05:57:26,193][root][INFO] - ===> Epoch[4867](24335/5): Loss 0.0133, Sem 0.0133 LR: 6.262e-02 Score 99.517 Data time: 0.1895, Forward time: 1.2629, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 2.7665 [2021-07-08 05:57:40,016][root][INFO] - ===> Epoch[4868](24340/5): Loss 0.0134, Sem 0.0134 LR: 6.261e-02 Score 99.523 Data time: 0.2359, Forward time: 1.2629, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 2.7251 [2021-07-08 05:57:53,836][root][INFO] - ===> Epoch[4869](24345/5): Loss 0.0134, Sem 0.0134 LR: 6.260e-02 Score 99.517 Data time: 0.2324, Forward time: 1.2629, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 2.7125 [2021-07-08 05:58:07,787][root][INFO] - ===> Epoch[4870](24350/5): Loss 0.0129, Sem 0.0129 LR: 6.259e-02 Score 99.538 Data time: 0.2503, Forward time: 1.2629, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 2.7457 [2021-07-08 05:58:21,597][root][INFO] - ===> Epoch[4871](24355/5): Loss 0.0136, Sem 0.0136 LR: 6.258e-02 Score 99.510 Data time: 0.2266, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7181 [2021-07-08 05:58:34,936][root][INFO] - ===> Epoch[4872](24360/5): Loss 0.0133, Sem 0.0133 LR: 6.258e-02 Score 99.520 Data time: 0.2067, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.6267 [2021-07-08 05:58:49,303][root][INFO] - ===> Epoch[4873](24365/5): Loss 0.0136, Sem 0.0136 LR: 6.257e-02 Score 99.511 Data time: 0.2471, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.8290 [2021-07-08 05:59:03,129][root][INFO] - ===> Epoch[4874](24370/5): Loss 0.0133, Sem 0.0133 LR: 6.256e-02 Score 99.514 Data time: 0.1963, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7190 [2021-07-08 05:59:17,074][root][INFO] - ===> Epoch[4875](24375/5): Loss 0.0131, Sem 0.0131 LR: 6.255e-02 Score 99.526 Data time: 0.2384, Forward time: 1.2628, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7419 [2021-07-08 05:59:31,200][root][INFO] - ===> Epoch[4876](24380/5): Loss 0.0130, Sem 0.0130 LR: 6.255e-02 Score 99.528 Data time: 0.2066, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7758 [2021-07-08 05:59:45,722][root][INFO] - ===> Epoch[4877](24385/5): Loss 0.0133, Sem 0.0133 LR: 6.254e-02 Score 99.519 Data time: 0.2580, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.8558 [2021-07-08 05:59:59,846][root][INFO] - ===> Epoch[4878](24390/5): Loss 0.0131, Sem 0.0131 LR: 6.253e-02 Score 99.526 Data time: 0.2372, Forward time: 1.2629, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 2.7689 [2021-07-08 06:00:14,545][root][INFO] - ===> Epoch[4879](24395/5): Loss 0.0127, Sem 0.0127 LR: 6.252e-02 Score 99.543 Data time: 0.2422, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.8978 [2021-07-08 06:00:28,738][root][INFO] - ===> Epoch[4880](24400/5): Loss 0.0133, Sem 0.0133 LR: 6.251e-02 Score 99.521 Data time: 0.2115, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7962 [2021-07-08 06:00:28,739][root][INFO] - ===> Start testing [2021-07-08 06:01:04,705][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1222 Loss 0.322 (AVG: 0.545) Score 91.705 (AVG: 91.086) mIOU 70.057 mAP 83.886 mAcc 76.214 IOU: 65.578 0.000 81.301 74.983 94.988 92.182 25.619 85.609 97.701 67.071 79.499 86.535 59.673 mAP: 84.665 7.969 92.754 87.887 98.811 98.732 56.974 98.390 99.734 85.436 91.869 95.344 91.950 mAcc: 81.229 0.000 85.034 82.512 96.221 97.047 27.028 96.832 99.453 76.884 88.186 97.111 63.249 [2021-07-08 06:01:04,706][root][INFO] - Finished test. Elapsed time: 35.9660 [2021-07-08 06:01:05,731][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 06:01:18,886][root][INFO] - ===> Epoch[4881](24405/5): Loss 0.0126, Sem 0.0126 LR: 6.251e-02 Score 99.540 Data time: 0.2136, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.5856 [2021-07-08 06:01:32,258][root][INFO] - ===> Epoch[4882](24410/5): Loss 0.0134, Sem 0.0134 LR: 6.250e-02 Score 99.525 Data time: 0.2237, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.6303 [2021-07-08 06:01:46,527][root][INFO] - ===> Epoch[4883](24415/5): Loss 0.0132, Sem 0.0132 LR: 6.249e-02 Score 99.520 Data time: 0.1932, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.8051 [2021-07-08 06:02:00,977][root][INFO] - ===> Epoch[4884](24420/5): Loss 0.0132, Sem 0.0132 LR: 6.248e-02 Score 99.528 Data time: 0.2365, Forward time: 1.2629, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 2.8406 [2021-07-08 06:02:15,146][root][INFO] - ===> Epoch[4885](24425/5): Loss 0.0128, Sem 0.0128 LR: 6.247e-02 Score 99.541 Data time: 0.2849, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.7897 [2021-07-08 06:02:30,062][root][INFO] - ===> Epoch[4886](24430/5): Loss 0.0129, Sem 0.0129 LR: 6.247e-02 Score 99.531 Data time: 0.2813, Forward time: 1.2629, Backward time: 1.2347, DDP time: 0.0042, Total iter time: 2.9474 [2021-07-08 06:02:44,586][root][INFO] - ===> Epoch[4887](24435/5): Loss 0.0125, Sem 0.0125 LR: 6.246e-02 Score 99.548 Data time: 0.2053, Forward time: 1.2629, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 2.8622 [2021-07-08 06:02:59,979][root][INFO] - ===> Epoch[4888](24440/5): Loss 0.0127, Sem 0.0127 LR: 6.245e-02 Score 99.539 Data time: 0.2421, Forward time: 1.2629, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 3.0358 [2021-07-08 06:03:14,028][root][INFO] - ===> Epoch[4889](24445/5): Loss 0.0127, Sem 0.0127 LR: 6.244e-02 Score 99.542 Data time: 0.1974, Forward time: 1.2629, Backward time: 1.2348, DDP time: 0.0042, Total iter time: 2.7682 [2021-07-08 06:03:28,172][root][INFO] - ===> Epoch[4890](24450/5): Loss 0.0130, Sem 0.0130 LR: 6.243e-02 Score 99.532 Data time: 0.2044, Forward time: 1.2629, Backward time: 1.2349, DDP time: 0.0042, Total iter time: 2.7802 [2021-07-08 06:03:42,214][root][INFO] - ===> Epoch[4891](24455/5): Loss 0.0133, Sem 0.0133 LR: 6.243e-02 Score 99.521 Data time: 0.2010, Forward time: 1.2628, Backward time: 1.2349, DDP time: 0.0042, Total iter time: 2.7676 [2021-07-08 06:03:56,055][root][INFO] - ===> Epoch[4892](24460/5): Loss 0.0124, Sem 0.0124 LR: 6.242e-02 Score 99.551 Data time: 0.2084, Forward time: 1.2628, Backward time: 1.2349, DDP time: 0.0042, Total iter time: 2.7195 [2021-07-08 06:04:09,331][root][INFO] - ===> Epoch[4893](24465/5): Loss 0.0141, Sem 0.0141 LR: 6.241e-02 Score 99.513 Data time: 0.1804, Forward time: 1.2628, Backward time: 1.2349, DDP time: 0.0042, Total iter time: 2.6041 [2021-07-08 06:04:23,488][root][INFO] - ===> Epoch[4894](24470/5): Loss 0.0132, Sem 0.0132 LR: 6.240e-02 Score 99.521 Data time: 0.2331, Forward time: 1.2628, Backward time: 1.2349, DDP time: 0.0042, Total iter time: 2.7766 [2021-07-08 06:04:37,153][root][INFO] - ===> Epoch[4895](24475/5): Loss 0.0136, Sem 0.0136 LR: 6.239e-02 Score 99.506 Data time: 0.2200, Forward time: 1.2628, Backward time: 1.2349, DDP time: 0.0042, Total iter time: 2.6849 [2021-07-08 06:04:51,593][root][INFO] - ===> Epoch[4896](24480/5): Loss 0.0128, Sem 0.0128 LR: 6.239e-02 Score 99.538 Data time: 0.2397, Forward time: 1.2628, Backward time: 1.2349, DDP time: 0.0042, Total iter time: 2.8401 [2021-07-08 06:05:04,972][root][INFO] - ===> Epoch[4897](24485/5): Loss 0.0127, Sem 0.0127 LR: 6.238e-02 Score 99.536 Data time: 0.2468, Forward time: 1.2628, Backward time: 1.2349, DDP time: 0.0042, Total iter time: 2.6318 [2021-07-08 06:05:19,128][root][INFO] - ===> Epoch[4898](24490/5): Loss 0.0129, Sem 0.0129 LR: 6.237e-02 Score 99.533 Data time: 0.1660, Forward time: 1.2628, Backward time: 1.2349, DDP time: 0.0042, Total iter time: 2.7782 [2021-07-08 06:05:33,603][root][INFO] - ===> Epoch[4899](24495/5): Loss 0.0134, Sem 0.0134 LR: 6.236e-02 Score 99.525 Data time: 0.1651, Forward time: 1.2627, Backward time: 1.2350, DDP time: 0.0042, Total iter time: 2.8455 [2021-07-08 06:05:47,611][root][INFO] - ===> Epoch[4900](24500/5): Loss 0.0122, Sem 0.0122 LR: 6.236e-02 Score 99.564 Data time: 0.2115, Forward time: 1.2627, Backward time: 1.2350, DDP time: 0.0042, Total iter time: 2.7610 [2021-07-08 06:06:01,873][root][INFO] - ===> Epoch[4901](24505/5): Loss 0.0134, Sem 0.0134 LR: 6.235e-02 Score 99.513 Data time: 0.2069, Forward time: 1.2627, Backward time: 1.2350, DDP time: 0.0042, Total iter time: 2.8054 [2021-07-08 06:06:16,254][root][INFO] - ===> Epoch[4902](24510/5): Loss 0.0128, Sem 0.0128 LR: 6.234e-02 Score 99.539 Data time: 0.2739, Forward time: 1.2627, Backward time: 1.2350, DDP time: 0.0042, Total iter time: 2.8247 [2021-07-08 06:06:30,958][root][INFO] - ===> Epoch[4903](24515/5): Loss 0.0123, Sem 0.0123 LR: 6.233e-02 Score 99.556 Data time: 0.2186, Forward time: 1.2627, Backward time: 1.2351, DDP time: 0.0042, Total iter time: 2.9085 [2021-07-08 06:06:44,347][root][INFO] - ===> Epoch[4904](24520/5): Loss 0.0129, Sem 0.0129 LR: 6.232e-02 Score 99.533 Data time: 0.2361, Forward time: 1.2627, Backward time: 1.2350, DDP time: 0.0042, Total iter time: 2.6406 [2021-07-08 06:06:58,433][root][INFO] - ===> Epoch[4905](24525/5): Loss 0.0129, Sem 0.0129 LR: 6.232e-02 Score 99.535 Data time: 0.1546, Forward time: 1.2627, Backward time: 1.2351, DDP time: 0.0042, Total iter time: 2.7710 [2021-07-08 06:07:12,044][root][INFO] - ===> Epoch[4906](24530/5): Loss 0.0126, Sem 0.0126 LR: 6.231e-02 Score 99.541 Data time: 0.1864, Forward time: 1.2627, Backward time: 1.2351, DDP time: 0.0042, Total iter time: 2.6787 [2021-07-08 06:07:25,698][root][INFO] - ===> Epoch[4907](24535/5): Loss 0.0128, Sem 0.0128 LR: 6.230e-02 Score 99.541 Data time: 0.2382, Forward time: 1.2627, Backward time: 1.2350, DDP time: 0.0042, Total iter time: 2.6833 [2021-07-08 06:07:40,565][root][INFO] - ===> Epoch[4908](24540/5): Loss 0.0122, Sem 0.0122 LR: 6.229e-02 Score 99.556 Data time: 0.2190, Forward time: 1.2627, Backward time: 1.2351, DDP time: 0.0042, Total iter time: 2.9292 [2021-07-08 06:07:54,196][root][INFO] - ===> Epoch[4909](24545/5): Loss 0.0125, Sem 0.0125 LR: 6.228e-02 Score 99.548 Data time: 0.1760, Forward time: 1.2627, Backward time: 1.2351, DDP time: 0.0042, Total iter time: 2.6749 [2021-07-08 06:08:07,880][root][INFO] - ===> Epoch[4910](24550/5): Loss 0.0125, Sem 0.0125 LR: 6.228e-02 Score 99.548 Data time: 0.1828, Forward time: 1.2627, Backward time: 1.2350, DDP time: 0.0042, Total iter time: 2.6900 [2021-07-08 06:08:21,618][root][INFO] - ===> Epoch[4911](24555/5): Loss 0.0126, Sem 0.0126 LR: 6.227e-02 Score 99.547 Data time: 0.2145, Forward time: 1.2627, Backward time: 1.2350, DDP time: 0.0042, Total iter time: 2.6887 [2021-07-08 06:08:35,417][root][INFO] - ===> Epoch[4912](24560/5): Loss 0.0127, Sem 0.0127 LR: 6.226e-02 Score 99.537 Data time: 0.2519, Forward time: 1.2627, Backward time: 1.2350, DDP time: 0.0042, Total iter time: 2.7177 [2021-07-08 06:08:51,013][root][INFO] - ===> Epoch[4913](24565/5): Loss 0.0123, Sem 0.0123 LR: 6.225e-02 Score 99.553 Data time: 0.1510, Forward time: 1.2627, Backward time: 1.2351, DDP time: 0.0042, Total iter time: 3.0795 [2021-07-08 06:09:05,372][root][INFO] - ===> Epoch[4914](24570/5): Loss 0.0130, Sem 0.0130 LR: 6.224e-02 Score 99.530 Data time: 0.2921, Forward time: 1.2627, Backward time: 1.2351, DDP time: 0.0042, Total iter time: 2.8275 [2021-07-08 06:09:20,756][root][INFO] - ===> Epoch[4915](24575/5): Loss 0.0131, Sem 0.0131 LR: 6.224e-02 Score 99.526 Data time: 0.2711, Forward time: 1.2628, Backward time: 1.2351, DDP time: 0.0042, Total iter time: 3.0256 [2021-07-08 06:09:35,304][root][INFO] - ===> Epoch[4916](24580/5): Loss 0.0131, Sem 0.0131 LR: 6.223e-02 Score 99.529 Data time: 0.3186, Forward time: 1.2628, Backward time: 1.2351, DDP time: 0.0042, Total iter time: 2.8553 [2021-07-08 06:09:50,341][root][INFO] - ===> Epoch[4917](24585/5): Loss 0.0124, Sem 0.0124 LR: 6.222e-02 Score 99.548 Data time: 0.2449, Forward time: 1.2628, Backward time: 1.2351, DDP time: 0.0042, Total iter time: 2.9699 [2021-07-08 06:10:03,275][root][INFO] - ===> Epoch[4918](24590/5): Loss 0.0122, Sem 0.0122 LR: 6.221e-02 Score 99.556 Data time: 0.2516, Forward time: 1.2627, Backward time: 1.2351, DDP time: 0.0042, Total iter time: 2.5383 [2021-07-08 06:10:17,602][root][INFO] - ===> Epoch[4919](24595/5): Loss 0.0127, Sem 0.0127 LR: 6.221e-02 Score 99.544 Data time: 0.2396, Forward time: 1.2627, Backward time: 1.2351, DDP time: 0.0042, Total iter time: 2.8220 [2021-07-08 06:10:32,102][root][INFO] - ===> Epoch[4920](24600/5): Loss 0.0132, Sem 0.0132 LR: 6.220e-02 Score 99.523 Data time: 0.1820, Forward time: 1.2627, Backward time: 1.2352, DDP time: 0.0042, Total iter time: 2.8592 [2021-07-08 06:10:32,104][root][INFO] - ===> Start testing [2021-07-08 06:11:07,976][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1259 Loss 0.256 (AVG: 0.516) Score 93.694 (AVG: 91.322) mIOU 71.156 mAP 84.546 mAcc 77.852 IOU: 65.457 0.000 84.084 75.236 94.880 92.056 37.991 81.078 98.269 69.157 79.611 87.758 59.451 mAP: 85.038 10.179 94.078 87.276 98.942 98.638 63.071 98.341 99.778 85.220 92.077 95.256 91.203 mAcc: 80.983 0.000 89.718 82.895 96.045 96.703 41.584 97.241 99.507 76.717 89.870 96.502 64.317 [2021-07-08 06:11:07,976][root][INFO] - Finished test. Elapsed time: 35.8711 [2021-07-08 06:11:09,023][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 06:11:23,186][root][INFO] - ===> Epoch[4921](24605/5): Loss 0.0129, Sem 0.0129 LR: 6.219e-02 Score 99.531 Data time: 0.2350, Forward time: 1.2627, Backward time: 1.2352, DDP time: 0.0042, Total iter time: 2.7866 [2021-07-08 06:11:37,448][root][INFO] - ===> Epoch[4922](24610/5): Loss 0.0122, Sem 0.0122 LR: 6.218e-02 Score 99.554 Data time: 0.2145, Forward time: 1.2627, Backward time: 1.2352, DDP time: 0.0042, Total iter time: 2.8112 [2021-07-08 06:11:51,735][root][INFO] - ===> Epoch[4923](24615/5): Loss 0.0126, Sem 0.0126 LR: 6.217e-02 Score 99.542 Data time: 0.1859, Forward time: 1.2627, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.8160 [2021-07-08 06:12:05,261][root][INFO] - ===> Epoch[4924](24620/5): Loss 0.0127, Sem 0.0127 LR: 6.217e-02 Score 99.544 Data time: 0.2257, Forward time: 1.2627, Backward time: 1.2352, DDP time: 0.0042, Total iter time: 2.6561 [2021-07-08 06:12:18,771][root][INFO] - ===> Epoch[4925](24625/5): Loss 0.0126, Sem 0.0126 LR: 6.216e-02 Score 99.542 Data time: 0.2300, Forward time: 1.2627, Backward time: 1.2352, DDP time: 0.0042, Total iter time: 2.6567 [2021-07-08 06:12:33,040][root][INFO] - ===> Epoch[4926](24630/5): Loss 0.0123, Sem 0.0123 LR: 6.215e-02 Score 99.553 Data time: 0.2034, Forward time: 1.2627, Backward time: 1.2352, DDP time: 0.0042, Total iter time: 2.8070 [2021-07-08 06:12:47,385][root][INFO] - ===> Epoch[4927](24635/5): Loss 0.0122, Sem 0.0122 LR: 6.214e-02 Score 99.558 Data time: 0.2490, Forward time: 1.2627, Backward time: 1.2352, DDP time: 0.0042, Total iter time: 2.8248 [2021-07-08 06:13:00,629][root][INFO] - ===> Epoch[4928](24640/5): Loss 0.0125, Sem 0.0125 LR: 6.213e-02 Score 99.548 Data time: 0.2106, Forward time: 1.2627, Backward time: 1.2352, DDP time: 0.0042, Total iter time: 2.6145 [2021-07-08 06:13:14,564][root][INFO] - ===> Epoch[4929](24645/5): Loss 0.0125, Sem 0.0125 LR: 6.213e-02 Score 99.549 Data time: 0.1967, Forward time: 1.2626, Backward time: 1.2352, DDP time: 0.0042, Total iter time: 2.7341 [2021-07-08 06:13:28,664][root][INFO] - ===> Epoch[4930](24650/5): Loss 0.0126, Sem 0.0126 LR: 6.212e-02 Score 99.546 Data time: 0.1983, Forward time: 1.2626, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.7768 [2021-07-08 06:13:42,922][root][INFO] - ===> Epoch[4931](24655/5): Loss 0.0124, Sem 0.0124 LR: 6.211e-02 Score 99.557 Data time: 0.1822, Forward time: 1.2626, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.8004 [2021-07-08 06:13:57,272][root][INFO] - ===> Epoch[4932](24660/5): Loss 0.0128, Sem 0.0128 LR: 6.210e-02 Score 99.537 Data time: 0.2276, Forward time: 1.2626, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.8085 [2021-07-08 06:14:10,986][root][INFO] - ===> Epoch[4933](24665/5): Loss 0.0124, Sem 0.0124 LR: 6.209e-02 Score 99.551 Data time: 0.3003, Forward time: 1.2626, Backward time: 1.2352, DDP time: 0.0042, Total iter time: 2.6894 [2021-07-08 06:14:24,783][root][INFO] - ===> Epoch[4934](24670/5): Loss 0.0127, Sem 0.0127 LR: 6.209e-02 Score 99.539 Data time: 0.2103, Forward time: 1.2626, Backward time: 1.2352, DDP time: 0.0042, Total iter time: 2.7126 [2021-07-08 06:14:40,051][root][INFO] - ===> Epoch[4935](24675/5): Loss 0.0124, Sem 0.0124 LR: 6.208e-02 Score 99.548 Data time: 0.2234, Forward time: 1.2626, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 3.0092 [2021-07-08 06:14:53,788][root][INFO] - ===> Epoch[4936](24680/5): Loss 0.0127, Sem 0.0127 LR: 6.207e-02 Score 99.538 Data time: 0.1991, Forward time: 1.2626, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.7005 [2021-07-08 06:15:08,922][root][INFO] - ===> Epoch[4937](24685/5): Loss 0.0124, Sem 0.0124 LR: 6.206e-02 Score 99.549 Data time: 0.3169, Forward time: 1.2626, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.9684 [2021-07-08 06:15:23,023][root][INFO] - ===> Epoch[4938](24690/5): Loss 0.0130, Sem 0.0130 LR: 6.205e-02 Score 99.548 Data time: 0.3089, Forward time: 1.2627, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.7618 [2021-07-08 06:15:37,538][root][INFO] - ===> Epoch[4939](24695/5): Loss 0.0128, Sem 0.0128 LR: 6.205e-02 Score 99.542 Data time: 0.2294, Forward time: 1.2627, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.8628 [2021-07-08 06:15:52,086][root][INFO] - ===> Epoch[4940](24700/5): Loss 0.0129, Sem 0.0129 LR: 6.204e-02 Score 99.534 Data time: 0.2350, Forward time: 1.2627, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.8621 [2021-07-08 06:16:06,769][root][INFO] - ===> Epoch[4941](24705/5): Loss 0.0130, Sem 0.0130 LR: 6.203e-02 Score 99.533 Data time: 0.2669, Forward time: 1.2627, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.8864 [2021-07-08 06:16:20,553][root][INFO] - ===> Epoch[4942](24710/5): Loss 0.0124, Sem 0.0124 LR: 6.202e-02 Score 99.556 Data time: 0.2720, Forward time: 1.2627, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.7086 [2021-07-08 06:16:35,146][root][INFO] - ===> Epoch[4943](24715/5): Loss 0.0134, Sem 0.0134 LR: 6.202e-02 Score 99.523 Data time: 0.2463, Forward time: 1.2627, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.8655 [2021-07-08 06:16:49,123][root][INFO] - ===> Epoch[4944](24720/5): Loss 0.0131, Sem 0.0131 LR: 6.201e-02 Score 99.523 Data time: 0.1892, Forward time: 1.2627, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.7399 [2021-07-08 06:17:03,159][root][INFO] - ===> Epoch[4945](24725/5): Loss 0.0132, Sem 0.0132 LR: 6.200e-02 Score 99.521 Data time: 0.2066, Forward time: 1.2627, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.7665 [2021-07-08 06:17:17,401][root][INFO] - ===> Epoch[4946](24730/5): Loss 0.0130, Sem 0.0130 LR: 6.199e-02 Score 99.531 Data time: 0.2071, Forward time: 1.2627, Backward time: 1.2353, DDP time: 0.0042, Total iter time: 2.8031 [2021-07-08 06:17:32,714][root][INFO] - ===> Epoch[4947](24735/5): Loss 0.0127, Sem 0.0127 LR: 6.198e-02 Score 99.539 Data time: 0.2236, Forward time: 1.2627, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 3.0129 [2021-07-08 06:17:46,217][root][INFO] - ===> Epoch[4948](24740/5): Loss 0.0127, Sem 0.0127 LR: 6.198e-02 Score 99.539 Data time: 0.1564, Forward time: 1.2627, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.6601 [2021-07-08 06:17:59,220][root][INFO] - ===> Epoch[4949](24745/5): Loss 0.0129, Sem 0.0129 LR: 6.197e-02 Score 99.531 Data time: 0.2323, Forward time: 1.2627, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.5581 [2021-07-08 06:18:13,344][root][INFO] - ===> Epoch[4950](24750/5): Loss 0.0126, Sem 0.0126 LR: 6.196e-02 Score 99.543 Data time: 0.2181, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.7779 [2021-07-08 06:18:26,700][root][INFO] - ===> Epoch[4951](24755/5): Loss 0.0131, Sem 0.0131 LR: 6.195e-02 Score 99.531 Data time: 0.2517, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.6281 [2021-07-08 06:18:40,874][root][INFO] - ===> Epoch[4952](24760/5): Loss 0.0119, Sem 0.0119 LR: 6.194e-02 Score 99.573 Data time: 0.2162, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.8002 [2021-07-08 06:18:54,763][root][INFO] - ===> Epoch[4953](24765/5): Loss 0.0127, Sem 0.0127 LR: 6.194e-02 Score 99.544 Data time: 0.2565, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.7270 [2021-07-08 06:19:08,478][root][INFO] - ===> Epoch[4954](24770/5): Loss 0.0125, Sem 0.0125 LR: 6.193e-02 Score 99.556 Data time: 0.2076, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.6979 [2021-07-08 06:19:22,960][root][INFO] - ===> Epoch[4955](24775/5): Loss 0.0132, Sem 0.0132 LR: 6.192e-02 Score 99.531 Data time: 0.2228, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.8588 [2021-07-08 06:19:37,100][root][INFO] - ===> Epoch[4956](24780/5): Loss 0.0121, Sem 0.0121 LR: 6.191e-02 Score 99.560 Data time: 0.1970, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.7923 [2021-07-08 06:19:50,836][root][INFO] - ===> Epoch[4957](24785/5): Loss 0.0122, Sem 0.0122 LR: 6.190e-02 Score 99.557 Data time: 0.2128, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.6889 [2021-07-08 06:20:05,668][root][INFO] - ===> Epoch[4958](24790/5): Loss 0.0122, Sem 0.0122 LR: 6.190e-02 Score 99.559 Data time: 0.1739, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.9316 [2021-07-08 06:20:19,892][root][INFO] - ===> Epoch[4959](24795/5): Loss 0.0125, Sem 0.0125 LR: 6.189e-02 Score 99.560 Data time: 0.2703, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.7899 [2021-07-08 06:20:33,891][root][INFO] - ===> Epoch[4960](24800/5): Loss 0.0127, Sem 0.0127 LR: 6.188e-02 Score 99.546 Data time: 0.2484, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.7564 [2021-07-08 06:20:33,893][root][INFO] - ===> Start testing [2021-07-08 06:21:07,967][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1222 Loss 0.317 (AVG: 0.528) Score 92.004 (AVG: 91.275) mIOU 70.676 mAP 84.063 mAcc 76.711 IOU: 65.368 0.000 83.411 75.430 94.824 92.048 36.301 82.626 98.016 65.213 78.762 87.645 59.149 mAP: 84.545 11.776 94.075 87.904 98.877 98.626 63.070 98.384 99.778 78.454 91.667 95.329 90.339 mAcc: 80.674 0.000 88.684 83.220 95.884 96.973 39.078 97.218 99.383 66.693 89.853 96.996 62.587 [2021-07-08 06:21:07,967][root][INFO] - Finished test. Elapsed time: 34.0736 [2021-07-08 06:21:09,076][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 06:21:22,408][root][INFO] - ===> Epoch[4961](24805/5): Loss 0.0124, Sem 0.0124 LR: 6.187e-02 Score 99.553 Data time: 0.2066, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.6251 [2021-07-08 06:21:37,282][root][INFO] - ===> Epoch[4962](24810/5): Loss 0.0125, Sem 0.0125 LR: 6.187e-02 Score 99.549 Data time: 0.1586, Forward time: 1.2626, Backward time: 1.2355, DDP time: 0.0042, Total iter time: 2.9430 [2021-07-08 06:21:51,299][root][INFO] - ===> Epoch[4963](24815/5): Loss 0.0124, Sem 0.0124 LR: 6.186e-02 Score 99.549 Data time: 0.2322, Forward time: 1.2626, Backward time: 1.2355, DDP time: 0.0042, Total iter time: 2.7637 [2021-07-08 06:22:05,248][root][INFO] - ===> Epoch[4964](24820/5): Loss 0.0123, Sem 0.0123 LR: 6.185e-02 Score 99.557 Data time: 0.2126, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.7536 [2021-07-08 06:22:19,255][root][INFO] - ===> Epoch[4965](24825/5): Loss 0.0126, Sem 0.0126 LR: 6.184e-02 Score 99.549 Data time: 0.1801, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.7497 [2021-07-08 06:22:33,635][root][INFO] - ===> Epoch[4966](24830/5): Loss 0.0123, Sem 0.0123 LR: 6.183e-02 Score 99.555 Data time: 0.2761, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.8272 [2021-07-08 06:22:47,548][root][INFO] - ===> Epoch[4967](24835/5): Loss 0.0124, Sem 0.0124 LR: 6.183e-02 Score 99.552 Data time: 0.2159, Forward time: 1.2627, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.7296 [2021-07-08 06:23:02,007][root][INFO] - ===> Epoch[4968](24840/5): Loss 0.0122, Sem 0.0122 LR: 6.182e-02 Score 99.558 Data time: 0.2584, Forward time: 1.2627, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.8361 [2021-07-08 06:23:15,543][root][INFO] - ===> Epoch[4969](24845/5): Loss 0.0129, Sem 0.0129 LR: 6.181e-02 Score 99.534 Data time: 0.1751, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.6617 [2021-07-08 06:23:29,123][root][INFO] - ===> Epoch[4970](24850/5): Loss 0.0125, Sem 0.0125 LR: 6.180e-02 Score 99.545 Data time: 0.2474, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.6726 [2021-07-08 06:23:43,416][root][INFO] - ===> Epoch[4971](24855/5): Loss 0.0124, Sem 0.0124 LR: 6.179e-02 Score 99.552 Data time: 0.2347, Forward time: 1.2626, Backward time: 1.2354, DDP time: 0.0042, Total iter time: 2.8191 [2021-07-08 06:23:58,420][root][INFO] - ===> Epoch[4972](24860/5): Loss 0.0119, Sem 0.0119 LR: 6.179e-02 Score 99.570 Data time: 0.2113, Forward time: 1.2626, Backward time: 1.2355, DDP time: 0.0042, Total iter time: 2.9536 [2021-07-08 06:24:12,577][root][INFO] - ===> Epoch[4973](24865/5): Loss 0.0120, Sem 0.0120 LR: 6.178e-02 Score 99.565 Data time: 0.2170, Forward time: 1.2626, Backward time: 1.2355, DDP time: 0.0042, Total iter time: 2.7847 [2021-07-08 06:24:27,005][root][INFO] - ===> Epoch[4974](24870/5): Loss 0.0127, Sem 0.0127 LR: 6.177e-02 Score 99.543 Data time: 0.1741, Forward time: 1.2626, Backward time: 1.2355, DDP time: 0.0042, Total iter time: 2.8439 [2021-07-08 06:24:40,495][root][INFO] - ===> Epoch[4975](24875/5): Loss 0.0123, Sem 0.0123 LR: 6.176e-02 Score 99.555 Data time: 0.1969, Forward time: 1.2626, Backward time: 1.2355, DDP time: 0.0042, Total iter time: 2.6519 [2021-07-08 06:24:54,997][root][INFO] - ===> Epoch[4976](24880/5): Loss 0.0121, Sem 0.0121 LR: 6.175e-02 Score 99.564 Data time: 0.2221, Forward time: 1.2626, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.8633 [2021-07-08 06:25:08,560][root][INFO] - ===> Epoch[4977](24885/5): Loss 0.0119, Sem 0.0119 LR: 6.175e-02 Score 99.569 Data time: 0.2071, Forward time: 1.2626, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.6759 [2021-07-08 06:25:23,078][root][INFO] - ===> Epoch[4978](24890/5): Loss 0.0121, Sem 0.0121 LR: 6.174e-02 Score 99.563 Data time: 0.1920, Forward time: 1.2626, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.8577 [2021-07-08 06:25:37,719][root][INFO] - ===> Epoch[4979](24895/5): Loss 0.0120, Sem 0.0120 LR: 6.173e-02 Score 99.568 Data time: 0.2876, Forward time: 1.2626, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.8795 [2021-07-08 06:25:52,101][root][INFO] - ===> Epoch[4980](24900/5): Loss 0.0127, Sem 0.0127 LR: 6.172e-02 Score 99.536 Data time: 0.1510, Forward time: 1.2625, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.8373 [2021-07-08 06:26:05,156][root][INFO] - ===> Epoch[4981](24905/5): Loss 0.0118, Sem 0.0118 LR: 6.171e-02 Score 99.576 Data time: 0.2224, Forward time: 1.2625, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.5579 [2021-07-08 06:26:19,127][root][INFO] - ===> Epoch[4982](24910/5): Loss 0.0121, Sem 0.0121 LR: 6.171e-02 Score 99.564 Data time: 0.2416, Forward time: 1.2626, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.7364 [2021-07-08 06:26:32,907][root][INFO] - ===> Epoch[4983](24915/5): Loss 0.0122, Sem 0.0122 LR: 6.170e-02 Score 99.562 Data time: 0.2590, Forward time: 1.2626, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.7124 [2021-07-08 06:26:47,509][root][INFO] - ===> Epoch[4984](24920/5): Loss 0.0121, Sem 0.0121 LR: 6.169e-02 Score 99.559 Data time: 0.2190, Forward time: 1.2626, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.8737 [2021-07-08 06:27:01,889][root][INFO] - ===> Epoch[4985](24925/5): Loss 0.0117, Sem 0.0117 LR: 6.168e-02 Score 99.578 Data time: 0.2285, Forward time: 1.2626, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.8253 [2021-07-08 06:27:15,889][root][INFO] - ===> Epoch[4986](24930/5): Loss 0.0120, Sem 0.0120 LR: 6.168e-02 Score 99.564 Data time: 0.2121, Forward time: 1.2626, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.7539 [2021-07-08 06:27:30,126][root][INFO] - ===> Epoch[4987](24935/5): Loss 0.0119, Sem 0.0119 LR: 6.167e-02 Score 99.566 Data time: 0.2187, Forward time: 1.2626, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.8021 [2021-07-08 06:27:44,353][root][INFO] - ===> Epoch[4988](24940/5): Loss 0.0125, Sem 0.0125 LR: 6.166e-02 Score 99.552 Data time: 0.1925, Forward time: 1.2626, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.8025 [2021-07-08 06:27:58,423][root][INFO] - ===> Epoch[4989](24945/5): Loss 0.0115, Sem 0.0115 LR: 6.165e-02 Score 99.588 Data time: 0.2164, Forward time: 1.2626, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.7583 [2021-07-08 06:28:11,836][root][INFO] - ===> Epoch[4990](24950/5): Loss 0.0119, Sem 0.0119 LR: 6.164e-02 Score 99.567 Data time: 0.2066, Forward time: 1.2626, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.6426 [2021-07-08 06:28:25,712][root][INFO] - ===> Epoch[4991](24955/5): Loss 0.0123, Sem 0.0123 LR: 6.164e-02 Score 99.555 Data time: 0.2082, Forward time: 1.2626, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.7301 [2021-07-08 06:28:40,137][root][INFO] - ===> Epoch[4992](24960/5): Loss 0.0123, Sem 0.0123 LR: 6.163e-02 Score 99.552 Data time: 0.1779, Forward time: 1.2625, Backward time: 1.2357, DDP time: 0.0042, Total iter time: 2.8490 [2021-07-08 06:28:53,996][root][INFO] - ===> Epoch[4993](24965/5): Loss 0.0123, Sem 0.0123 LR: 6.162e-02 Score 99.558 Data time: 0.2446, Forward time: 1.2625, Backward time: 1.2357, DDP time: 0.0042, Total iter time: 2.7291 [2021-07-08 06:29:08,087][root][INFO] - ===> Epoch[4994](24970/5): Loss 0.0119, Sem 0.0119 LR: 6.161e-02 Score 99.572 Data time: 0.3152, Forward time: 1.2625, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.7584 [2021-07-08 06:29:22,233][root][INFO] - ===> Epoch[4995](24975/5): Loss 0.0124, Sem 0.0124 LR: 6.160e-02 Score 99.554 Data time: 0.1680, Forward time: 1.2625, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.7808 [2021-07-08 06:29:36,323][root][INFO] - ===> Epoch[4996](24980/5): Loss 0.0127, Sem 0.0127 LR: 6.160e-02 Score 99.547 Data time: 0.2627, Forward time: 1.2626, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.7755 [2021-07-08 06:29:50,248][root][INFO] - ===> Epoch[4997](24985/5): Loss 0.0122, Sem 0.0122 LR: 6.159e-02 Score 99.560 Data time: 0.2207, Forward time: 1.2626, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.7377 [2021-07-08 06:30:04,498][root][INFO] - ===> Epoch[4998](24990/5): Loss 0.0123, Sem 0.0123 LR: 6.158e-02 Score 99.556 Data time: 0.1932, Forward time: 1.2626, Backward time: 1.2356, DDP time: 0.0042, Total iter time: 2.8079 [2021-07-08 06:30:18,953][root][INFO] - ===> Epoch[4999](24995/5): Loss 0.0123, Sem 0.0123 LR: 6.157e-02 Score 99.554 Data time: 0.1529, Forward time: 1.2625, Backward time: 1.2357, DDP time: 0.0042, Total iter time: 2.8573 [2021-07-08 06:30:32,716][root][INFO] - ===> Epoch[5000](25000/5): Loss 0.0119, Sem 0.0119 LR: 6.156e-02 Score 99.570 Data time: 0.1521, Forward time: 1.2625, Backward time: 1.2357, DDP time: 0.0042, Total iter time: 2.7193 [2021-07-08 06:30:32,717][root][INFO] - ===> Start testing [2021-07-08 06:31:06,297][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1248 Loss 0.271 (AVG: 0.523) Score 93.356 (AVG: 91.413) mIOU 71.378 mAP 84.582 mAcc 77.993 IOU: 65.834 0.000 83.658 75.362 95.334 92.334 39.153 85.315 98.066 68.039 79.108 87.337 58.374 mAP: 84.867 7.886 92.787 87.239 98.916 98.649 65.393 98.456 99.746 88.384 91.362 95.204 90.677 mAcc: 80.613 0.000 87.996 82.522 96.698 96.750 42.538 97.247 99.544 80.525 89.588 96.807 63.079 [2021-07-08 06:31:06,297][root][INFO] - Finished test. Elapsed time: 33.5791 [2021-07-08 06:31:07,410][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 06:31:22,110][root][INFO] - ===> Epoch[5001](25005/5): Loss 0.0121, Sem 0.0121 LR: 6.156e-02 Score 99.562 Data time: 0.1934, Forward time: 1.2625, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.9063 [2021-07-08 06:31:35,650][root][INFO] - ===> Epoch[5002](25010/5): Loss 0.0122, Sem 0.0122 LR: 6.155e-02 Score 99.553 Data time: 0.2225, Forward time: 1.2625, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.6549 [2021-07-08 06:31:48,631][root][INFO] - ===> Epoch[5003](25015/5): Loss 0.0122, Sem 0.0122 LR: 6.154e-02 Score 99.567 Data time: 0.1809, Forward time: 1.2625, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.5477 [2021-07-08 06:32:03,010][root][INFO] - ===> Epoch[5004](25020/5): Loss 0.0123, Sem 0.0123 LR: 6.153e-02 Score 99.562 Data time: 0.1866, Forward time: 1.2625, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.8259 [2021-07-08 06:32:16,659][root][INFO] - ===> Epoch[5005](25025/5): Loss 0.0120, Sem 0.0120 LR: 6.152e-02 Score 99.566 Data time: 0.2190, Forward time: 1.2624, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.6868 [2021-07-08 06:32:30,130][root][INFO] - ===> Epoch[5006](25030/5): Loss 0.0120, Sem 0.0120 LR: 6.152e-02 Score 99.566 Data time: 0.2217, Forward time: 1.2624, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.6522 [2021-07-08 06:32:44,220][root][INFO] - ===> Epoch[5007](25035/5): Loss 0.0124, Sem 0.0124 LR: 6.151e-02 Score 99.556 Data time: 0.2400, Forward time: 1.2624, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.7649 [2021-07-08 06:32:57,859][root][INFO] - ===> Epoch[5008](25040/5): Loss 0.0129, Sem 0.0129 LR: 6.150e-02 Score 99.543 Data time: 0.2034, Forward time: 1.2624, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.6793 [2021-07-08 06:33:11,675][root][INFO] - ===> Epoch[5009](25045/5): Loss 0.0123, Sem 0.0123 LR: 6.149e-02 Score 99.563 Data time: 0.2364, Forward time: 1.2624, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.7132 [2021-07-08 06:33:26,346][root][INFO] - ===> Epoch[5010](25050/5): Loss 0.0122, Sem 0.0122 LR: 6.149e-02 Score 99.558 Data time: 0.2725, Forward time: 1.2624, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.8942 [2021-07-08 06:33:40,321][root][INFO] - ===> Epoch[5011](25055/5): Loss 0.0127, Sem 0.0127 LR: 6.148e-02 Score 99.538 Data time: 0.1506, Forward time: 1.2624, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.7585 [2021-07-08 06:33:53,988][root][INFO] - ===> Epoch[5012](25060/5): Loss 0.0121, Sem 0.0121 LR: 6.147e-02 Score 99.565 Data time: 0.1702, Forward time: 1.2624, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.6923 [2021-07-08 06:34:07,876][root][INFO] - ===> Epoch[5013](25065/5): Loss 0.0121, Sem 0.0121 LR: 6.146e-02 Score 99.568 Data time: 0.2310, Forward time: 1.2624, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.7249 [2021-07-08 06:34:21,261][root][INFO] - ===> Epoch[5014](25070/5): Loss 0.0127, Sem 0.0127 LR: 6.145e-02 Score 99.550 Data time: 0.2042, Forward time: 1.2624, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.6329 [2021-07-08 06:34:35,665][root][INFO] - ===> Epoch[5015](25075/5): Loss 0.0126, Sem 0.0126 LR: 6.145e-02 Score 99.543 Data time: 0.1665, Forward time: 1.2624, Backward time: 1.2358, DDP time: 0.0042, Total iter time: 2.8317 [2021-07-08 06:34:50,039][root][INFO] - ===> Epoch[5016](25080/5): Loss 0.0122, Sem 0.0122 LR: 6.144e-02 Score 99.558 Data time: 0.1890, Forward time: 1.2624, Backward time: 1.2359, DDP time: 0.0042, Total iter time: 2.8356 [2021-07-08 06:35:05,433][root][INFO] - ===> Epoch[5017](25085/5): Loss 0.0120, Sem 0.0120 LR: 6.143e-02 Score 99.568 Data time: 0.2196, Forward time: 1.2624, Backward time: 1.2359, DDP time: 0.0042, Total iter time: 3.0251 [2021-07-08 06:35:19,455][root][INFO] - ===> Epoch[5018](25090/5): Loss 0.0117, Sem 0.0117 LR: 6.142e-02 Score 99.572 Data time: 0.2457, Forward time: 1.2624, Backward time: 1.2359, DDP time: 0.0042, Total iter time: 2.7573 [2021-07-08 06:35:33,617][root][INFO] - ===> Epoch[5019](25095/5): Loss 0.0116, Sem 0.0116 LR: 6.141e-02 Score 99.584 Data time: 0.2419, Forward time: 1.2624, Backward time: 1.2359, DDP time: 0.0042, Total iter time: 2.7845 [2021-07-08 06:35:47,688][root][INFO] - ===> Epoch[5020](25100/5): Loss 0.0124, Sem 0.0124 LR: 6.141e-02 Score 99.558 Data time: 0.2125, Forward time: 1.2624, Backward time: 1.2359, DDP time: 0.0042, Total iter time: 2.7706 [2021-07-08 06:36:02,013][root][INFO] - ===> Epoch[5021](25105/5): Loss 0.0124, Sem 0.0124 LR: 6.140e-02 Score 99.555 Data time: 0.3167, Forward time: 1.2624, Backward time: 1.2359, DDP time: 0.0042, Total iter time: 2.8200 [2021-07-08 06:36:16,057][root][INFO] - ===> Epoch[5022](25110/5): Loss 0.0113, Sem 0.0113 LR: 6.139e-02 Score 99.591 Data time: 0.1521, Forward time: 1.2623, Backward time: 1.2360, DDP time: 0.0042, Total iter time: 2.7710 [2021-07-08 06:36:30,129][root][INFO] - ===> Epoch[5023](25115/5): Loss 0.0116, Sem 0.0116 LR: 6.138e-02 Score 99.578 Data time: 0.2079, Forward time: 1.2624, Backward time: 1.2359, DDP time: 0.0042, Total iter time: 2.7630 [2021-07-08 06:36:43,971][root][INFO] - ===> Epoch[5024](25120/5): Loss 0.0117, Sem 0.0117 LR: 6.137e-02 Score 99.578 Data time: 0.1727, Forward time: 1.2624, Backward time: 1.2359, DDP time: 0.0042, Total iter time: 2.7226 [2021-07-08 06:36:57,741][root][INFO] - ===> Epoch[5025](25125/5): Loss 0.0119, Sem 0.0119 LR: 6.137e-02 Score 99.567 Data time: 0.2367, Forward time: 1.2624, Backward time: 1.2359, DDP time: 0.0042, Total iter time: 2.7019 [2021-07-08 06:37:12,225][root][INFO] - ===> Epoch[5026](25130/5): Loss 0.0119, Sem 0.0119 LR: 6.136e-02 Score 99.567 Data time: 0.1927, Forward time: 1.2623, Backward time: 1.2360, DDP time: 0.0042, Total iter time: 2.8499 [2021-07-08 06:37:25,255][root][INFO] - ===> Epoch[5027](25135/5): Loss 0.0121, Sem 0.0121 LR: 6.135e-02 Score 99.564 Data time: 0.1773, Forward time: 1.2623, Backward time: 1.2359, DDP time: 0.0042, Total iter time: 2.5603 [2021-07-08 06:37:39,671][root][INFO] - ===> Epoch[5028](25140/5): Loss 0.0125, Sem 0.0125 LR: 6.134e-02 Score 99.549 Data time: 0.2380, Forward time: 1.2623, Backward time: 1.2360, DDP time: 0.0042, Total iter time: 2.8333 [2021-07-08 06:37:53,500][root][INFO] - ===> Epoch[5029](25145/5): Loss 0.0117, Sem 0.0117 LR: 6.133e-02 Score 99.580 Data time: 0.1854, Forward time: 1.2623, Backward time: 1.2360, DDP time: 0.0042, Total iter time: 2.7318 [2021-07-08 06:38:08,406][root][INFO] - ===> Epoch[5030](25150/5): Loss 0.0116, Sem 0.0116 LR: 6.133e-02 Score 99.581 Data time: 0.2659, Forward time: 1.2623, Backward time: 1.2360, DDP time: 0.0042, Total iter time: 2.9300 [2021-07-08 06:38:22,286][root][INFO] - ===> Epoch[5031](25155/5): Loss 0.0119, Sem 0.0119 LR: 6.132e-02 Score 99.567 Data time: 0.2271, Forward time: 1.2624, Backward time: 1.2360, DDP time: 0.0042, Total iter time: 2.7331 [2021-07-08 06:38:36,479][root][INFO] - ===> Epoch[5032](25160/5): Loss 0.0115, Sem 0.0115 LR: 6.131e-02 Score 99.587 Data time: 0.2376, Forward time: 1.2623, Backward time: 1.2360, DDP time: 0.0042, Total iter time: 2.7847 [2021-07-08 06:38:49,866][root][INFO] - ===> Epoch[5033](25165/5): Loss 0.0120, Sem 0.0120 LR: 6.130e-02 Score 99.566 Data time: 0.1959, Forward time: 1.2623, Backward time: 1.2360, DDP time: 0.0042, Total iter time: 2.6304 [2021-07-08 06:39:04,398][root][INFO] - ===> Epoch[5034](25170/5): Loss 0.0113, Sem 0.0113 LR: 6.130e-02 Score 99.593 Data time: 0.2040, Forward time: 1.2623, Backward time: 1.2360, DDP time: 0.0042, Total iter time: 2.8604 [2021-07-08 06:39:18,254][root][INFO] - ===> Epoch[5035](25175/5): Loss 0.0118, Sem 0.0118 LR: 6.129e-02 Score 99.572 Data time: 0.1828, Forward time: 1.2623, Backward time: 1.2360, DDP time: 0.0042, Total iter time: 2.7317 [2021-07-08 06:39:32,011][root][INFO] - ===> Epoch[5036](25180/5): Loss 0.0121, Sem 0.0121 LR: 6.128e-02 Score 99.560 Data time: 0.2312, Forward time: 1.2623, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.7020 [2021-07-08 06:39:46,557][root][INFO] - ===> Epoch[5037](25185/5): Loss 0.0118, Sem 0.0118 LR: 6.127e-02 Score 99.574 Data time: 0.1703, Forward time: 1.2622, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.8650 [2021-07-08 06:40:00,395][root][INFO] - ===> Epoch[5038](25190/5): Loss 0.0120, Sem 0.0120 LR: 6.126e-02 Score 99.567 Data time: 0.2069, Forward time: 1.2622, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.7124 [2021-07-08 06:40:15,577][root][INFO] - ===> Epoch[5039](25195/5): Loss 0.0115, Sem 0.0115 LR: 6.126e-02 Score 99.585 Data time: 0.2510, Forward time: 1.2622, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.9890 [2021-07-08 06:40:30,077][root][INFO] - ===> Epoch[5040](25200/5): Loss 0.0120, Sem 0.0120 LR: 6.125e-02 Score 99.570 Data time: 0.2664, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.8525 [2021-07-08 06:40:30,079][root][INFO] - ===> Start testing [2021-07-08 06:41:05,102][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1248 Loss 0.264 (AVG: 0.546) Score 94.118 (AVG: 91.248) mIOU 71.080 mAP 84.351 mAcc 77.367 IOU: 65.754 0.000 83.074 75.324 95.133 92.696 31.919 84.479 97.910 72.797 78.546 86.995 59.415 mAP: 84.336 8.837 92.981 87.457 98.860 98.289 60.993 98.193 99.780 88.747 92.059 94.804 91.221 mAcc: 80.606 0.000 87.321 82.323 96.449 96.409 33.534 96.478 99.282 82.242 90.562 97.111 63.457 [2021-07-08 06:41:05,102][root][INFO] - Finished test. Elapsed time: 35.0224 [2021-07-08 06:41:06,158][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 06:41:19,734][root][INFO] - ===> Epoch[5041](25205/5): Loss 0.0121, Sem 0.0121 LR: 6.124e-02 Score 99.566 Data time: 0.1701, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6744 [2021-07-08 06:41:34,580][root][INFO] - ===> Epoch[5042](25210/5): Loss 0.0119, Sem 0.0119 LR: 6.123e-02 Score 99.573 Data time: 0.2417, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.9258 [2021-07-08 06:41:47,908][root][INFO] - ===> Epoch[5043](25215/5): Loss 0.0116, Sem 0.0116 LR: 6.122e-02 Score 99.582 Data time: 0.2027, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6228 [2021-07-08 06:42:02,940][root][INFO] - ===> Epoch[5044](25220/5): Loss 0.0117, Sem 0.0117 LR: 6.122e-02 Score 99.578 Data time: 0.3030, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.9512 [2021-07-08 06:42:17,784][root][INFO] - ===> Epoch[5045](25225/5): Loss 0.0118, Sem 0.0118 LR: 6.121e-02 Score 99.570 Data time: 0.2311, Forward time: 1.2623, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.9225 [2021-07-08 06:42:31,694][root][INFO] - ===> Epoch[5046](25230/5): Loss 0.0117, Sem 0.0117 LR: 6.120e-02 Score 99.578 Data time: 0.2456, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7427 [2021-07-08 06:42:46,488][root][INFO] - ===> Epoch[5047](25235/5): Loss 0.0122, Sem 0.0122 LR: 6.119e-02 Score 99.566 Data time: 0.2899, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.9151 [2021-07-08 06:43:00,968][root][INFO] - ===> Epoch[5048](25240/5): Loss 0.0120, Sem 0.0120 LR: 6.118e-02 Score 99.566 Data time: 0.2859, Forward time: 1.2623, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.8418 [2021-07-08 06:43:14,510][root][INFO] - ===> Epoch[5049](25245/5): Loss 0.0116, Sem 0.0116 LR: 6.118e-02 Score 99.580 Data time: 0.1923, Forward time: 1.2623, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6719 [2021-07-08 06:43:28,844][root][INFO] - ===> Epoch[5050](25250/5): Loss 0.0118, Sem 0.0118 LR: 6.117e-02 Score 99.576 Data time: 0.2557, Forward time: 1.2623, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.8145 [2021-07-08 06:43:42,417][root][INFO] - ===> Epoch[5051](25255/5): Loss 0.0121, Sem 0.0121 LR: 6.116e-02 Score 99.567 Data time: 0.2045, Forward time: 1.2623, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6733 [2021-07-08 06:43:56,027][root][INFO] - ===> Epoch[5052](25260/5): Loss 0.0122, Sem 0.0122 LR: 6.115e-02 Score 99.556 Data time: 0.2101, Forward time: 1.2623, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.6810 [2021-07-08 06:44:09,388][root][INFO] - ===> Epoch[5053](25265/5): Loss 0.0123, Sem 0.0123 LR: 6.114e-02 Score 99.553 Data time: 0.1701, Forward time: 1.2623, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.6417 [2021-07-08 06:44:23,146][root][INFO] - ===> Epoch[5054](25270/5): Loss 0.0112, Sem 0.0112 LR: 6.114e-02 Score 99.594 Data time: 0.2194, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7082 [2021-07-08 06:44:37,735][root][INFO] - ===> Epoch[5055](25275/5): Loss 0.0117, Sem 0.0117 LR: 6.113e-02 Score 99.576 Data time: 0.2408, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.8703 [2021-07-08 06:44:51,428][root][INFO] - ===> Epoch[5056](25280/5): Loss 0.0124, Sem 0.0124 LR: 6.112e-02 Score 99.556 Data time: 0.2417, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6874 [2021-07-08 06:45:05,304][root][INFO] - ===> Epoch[5057](25285/5): Loss 0.0115, Sem 0.0115 LR: 6.111e-02 Score 99.584 Data time: 0.1998, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7261 [2021-07-08 06:45:19,004][root][INFO] - ===> Epoch[5058](25290/5): Loss 0.0113, Sem 0.0113 LR: 6.111e-02 Score 99.591 Data time: 0.2246, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6830 [2021-07-08 06:45:33,462][root][INFO] - ===> Epoch[5059](25295/5): Loss 0.0118, Sem 0.0118 LR: 6.110e-02 Score 99.570 Data time: 0.2653, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.8413 [2021-07-08 06:45:47,659][root][INFO] - ===> Epoch[5060](25300/5): Loss 0.0112, Sem 0.0112 LR: 6.109e-02 Score 99.588 Data time: 0.2451, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7998 [2021-07-08 06:46:02,493][root][INFO] - ===> Epoch[5061](25305/5): Loss 0.0117, Sem 0.0117 LR: 6.108e-02 Score 99.582 Data time: 0.2225, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.9243 [2021-07-08 06:46:17,045][root][INFO] - ===> Epoch[5062](25310/5): Loss 0.0111, Sem 0.0111 LR: 6.107e-02 Score 99.598 Data time: 0.1731, Forward time: 1.2622, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8690 [2021-07-08 06:46:30,851][root][INFO] - ===> Epoch[5063](25315/5): Loss 0.0114, Sem 0.0114 LR: 6.107e-02 Score 99.585 Data time: 0.2220, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7182 [2021-07-08 06:46:44,367][root][INFO] - ===> Epoch[5064](25320/5): Loss 0.0115, Sem 0.0115 LR: 6.106e-02 Score 99.580 Data time: 0.1739, Forward time: 1.2622, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6601 [2021-07-08 06:46:57,867][root][INFO] - ===> Epoch[5065](25325/5): Loss 0.0110, Sem 0.0110 LR: 6.105e-02 Score 99.602 Data time: 0.1979, Forward time: 1.2622, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6552 [2021-07-08 06:47:12,193][root][INFO] - ===> Epoch[5066](25330/5): Loss 0.0112, Sem 0.0112 LR: 6.104e-02 Score 99.591 Data time: 0.1860, Forward time: 1.2622, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8235 [2021-07-08 06:47:25,213][root][INFO] - ===> Epoch[5067](25335/5): Loss 0.0114, Sem 0.0114 LR: 6.103e-02 Score 99.587 Data time: 0.2224, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.5568 [2021-07-08 06:47:38,396][root][INFO] - ===> Epoch[5068](25340/5): Loss 0.0118, Sem 0.0118 LR: 6.103e-02 Score 99.574 Data time: 0.2199, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.5964 [2021-07-08 06:47:52,962][root][INFO] - ===> Epoch[5069](25345/5): Loss 0.0114, Sem 0.0114 LR: 6.102e-02 Score 99.588 Data time: 0.2254, Forward time: 1.2622, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.8527 [2021-07-08 06:48:06,222][root][INFO] - ===> Epoch[5070](25350/5): Loss 0.0120, Sem 0.0120 LR: 6.101e-02 Score 99.565 Data time: 0.1931, Forward time: 1.2621, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6058 [2021-07-08 06:48:20,587][root][INFO] - ===> Epoch[5071](25355/5): Loss 0.0109, Sem 0.0109 LR: 6.100e-02 Score 99.602 Data time: 0.1899, Forward time: 1.2621, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8201 [2021-07-08 06:48:34,670][root][INFO] - ===> Epoch[5072](25360/5): Loss 0.0119, Sem 0.0119 LR: 6.099e-02 Score 99.567 Data time: 0.2428, Forward time: 1.2621, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7622 [2021-07-08 06:48:49,647][root][INFO] - ===> Epoch[5073](25365/5): Loss 0.0116, Sem 0.0116 LR: 6.099e-02 Score 99.579 Data time: 0.1831, Forward time: 1.2621, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.9453 [2021-07-08 06:49:03,223][root][INFO] - ===> Epoch[5074](25370/5): Loss 0.0114, Sem 0.0114 LR: 6.098e-02 Score 99.586 Data time: 0.2672, Forward time: 1.2621, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6630 [2021-07-08 06:49:17,980][root][INFO] - ===> Epoch[5075](25375/5): Loss 0.0111, Sem 0.0111 LR: 6.097e-02 Score 99.593 Data time: 0.2755, Forward time: 1.2621, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8970 [2021-07-08 06:49:32,637][root][INFO] - ===> Epoch[5076](25380/5): Loss 0.0113, Sem 0.0113 LR: 6.096e-02 Score 99.593 Data time: 0.2526, Forward time: 1.2621, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8802 [2021-07-08 06:49:46,362][root][INFO] - ===> Epoch[5077](25385/5): Loss 0.0122, Sem 0.0122 LR: 6.095e-02 Score 99.556 Data time: 0.2775, Forward time: 1.2621, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6952 [2021-07-08 06:50:00,034][root][INFO] - ===> Epoch[5078](25390/5): Loss 0.0112, Sem 0.0112 LR: 6.095e-02 Score 99.594 Data time: 0.2580, Forward time: 1.2621, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6857 [2021-07-08 06:50:14,131][root][INFO] - ===> Epoch[5079](25395/5): Loss 0.0116, Sem 0.0116 LR: 6.094e-02 Score 99.585 Data time: 0.2214, Forward time: 1.2621, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7665 [2021-07-08 06:50:28,113][root][INFO] - ===> Epoch[5080](25400/5): Loss 0.0113, Sem 0.0113 LR: 6.093e-02 Score 99.595 Data time: 0.2779, Forward time: 1.2621, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7571 [2021-07-08 06:50:28,114][root][INFO] - ===> Start testing [2021-07-08 06:51:04,487][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1301 Loss 0.200 (AVG: 0.542) Score 95.287 (AVG: 91.396) mIOU 71.410 mAP 84.899 mAcc 78.296 IOU: 65.564 0.000 83.834 75.770 95.040 92.004 36.877 84.357 97.868 70.219 80.297 87.473 59.029 mAP: 84.485 8.503 93.879 87.836 98.916 98.528 63.536 98.289 99.699 91.823 92.019 94.973 91.200 mAcc: 81.333 0.000 89.046 82.502 96.342 97.475 40.111 96.516 99.500 84.649 90.992 96.702 62.687 [2021-07-08 06:51:04,487][root][INFO] - Finished test. Elapsed time: 36.3721 [2021-07-08 06:51:05,572][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 06:51:18,637][root][INFO] - ===> Epoch[5081](25405/5): Loss 0.0114, Sem 0.0114 LR: 6.092e-02 Score 99.587 Data time: 0.2223, Forward time: 1.2621, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.5687 [2021-07-08 06:51:32,179][root][INFO] - ===> Epoch[5082](25410/5): Loss 0.0118, Sem 0.0118 LR: 6.091e-02 Score 99.576 Data time: 0.2211, Forward time: 1.2621, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6664 [2021-07-08 06:51:46,199][root][INFO] - ===> Epoch[5083](25415/5): Loss 0.0117, Sem 0.0117 LR: 6.091e-02 Score 99.579 Data time: 0.2435, Forward time: 1.2621, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7647 [2021-07-08 06:52:00,228][root][INFO] - ===> Epoch[5084](25420/5): Loss 0.0116, Sem 0.0116 LR: 6.090e-02 Score 99.577 Data time: 0.2249, Forward time: 1.2621, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7646 [2021-07-08 06:52:13,182][root][INFO] - ===> Epoch[5085](25425/5): Loss 0.0118, Sem 0.0118 LR: 6.089e-02 Score 99.576 Data time: 0.2271, Forward time: 1.2621, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.5506 [2021-07-08 06:52:27,816][root][INFO] - ===> Epoch[5086](25430/5): Loss 0.0121, Sem 0.0121 LR: 6.088e-02 Score 99.569 Data time: 0.2352, Forward time: 1.2621, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.8841 [2021-07-08 06:52:42,484][root][INFO] - ===> Epoch[5087](25435/5): Loss 0.0120, Sem 0.0120 LR: 6.088e-02 Score 99.570 Data time: 0.2369, Forward time: 1.2621, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8867 [2021-07-08 06:52:55,965][root][INFO] - ===> Epoch[5088](25440/5): Loss 0.0121, Sem 0.0121 LR: 6.087e-02 Score 99.566 Data time: 0.1866, Forward time: 1.2621, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6611 [2021-07-08 06:53:09,854][root][INFO] - ===> Epoch[5089](25445/5): Loss 0.0114, Sem 0.0114 LR: 6.086e-02 Score 99.588 Data time: 0.1628, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7419 [2021-07-08 06:53:23,103][root][INFO] - ===> Epoch[5090](25450/5): Loss 0.0118, Sem 0.0118 LR: 6.085e-02 Score 99.575 Data time: 0.2344, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6030 [2021-07-08 06:53:36,804][root][INFO] - ===> Epoch[5091](25455/5): Loss 0.0115, Sem 0.0115 LR: 6.084e-02 Score 99.586 Data time: 0.2462, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6908 [2021-07-08 06:53:50,360][root][INFO] - ===> Epoch[5092](25460/5): Loss 0.0113, Sem 0.0113 LR: 6.084e-02 Score 99.588 Data time: 0.2293, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6628 [2021-07-08 06:54:04,673][root][INFO] - ===> Epoch[5093](25465/5): Loss 0.0115, Sem 0.0115 LR: 6.083e-02 Score 99.581 Data time: 0.3055, Forward time: 1.2621, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.8024 [2021-07-08 06:54:17,696][root][INFO] - ===> Epoch[5094](25470/5): Loss 0.0116, Sem 0.0116 LR: 6.082e-02 Score 99.579 Data time: 0.1815, Forward time: 1.2621, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.5624 [2021-07-08 06:54:32,032][root][INFO] - ===> Epoch[5095](25475/5): Loss 0.0110, Sem 0.0110 LR: 6.081e-02 Score 99.601 Data time: 0.2223, Forward time: 1.2621, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.8192 [2021-07-08 06:54:45,376][root][INFO] - ===> Epoch[5096](25480/5): Loss 0.0113, Sem 0.0113 LR: 6.080e-02 Score 99.591 Data time: 0.2212, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6273 [2021-07-08 06:54:59,594][root][INFO] - ===> Epoch[5097](25485/5): Loss 0.0113, Sem 0.0113 LR: 6.080e-02 Score 99.589 Data time: 0.2478, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7969 [2021-07-08 06:55:12,956][root][INFO] - ===> Epoch[5098](25490/5): Loss 0.0115, Sem 0.0115 LR: 6.079e-02 Score 99.586 Data time: 0.1922, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6212 [2021-07-08 06:55:26,385][root][INFO] - ===> Epoch[5099](25495/5): Loss 0.0114, Sem 0.0114 LR: 6.078e-02 Score 99.588 Data time: 0.2116, Forward time: 1.2620, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.6364 [2021-07-08 06:55:40,457][root][INFO] - ===> Epoch[5100](25500/5): Loss 0.0110, Sem 0.0110 LR: 6.077e-02 Score 99.608 Data time: 0.2100, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7649 [2021-07-08 06:55:54,596][root][INFO] - ===> Epoch[5101](25505/5): Loss 0.0113, Sem 0.0113 LR: 6.076e-02 Score 99.590 Data time: 0.1957, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7874 [2021-07-08 06:56:08,660][root][INFO] - ===> Epoch[5102](25510/5): Loss 0.0112, Sem 0.0112 LR: 6.076e-02 Score 99.596 Data time: 0.2885, Forward time: 1.2620, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.7472 [2021-07-08 06:56:22,502][root][INFO] - ===> Epoch[5103](25515/5): Loss 0.0118, Sem 0.0118 LR: 6.075e-02 Score 99.574 Data time: 0.2538, Forward time: 1.2620, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.7192 [2021-07-08 06:56:36,387][root][INFO] - ===> Epoch[5104](25520/5): Loss 0.0122, Sem 0.0122 LR: 6.074e-02 Score 99.558 Data time: 0.2563, Forward time: 1.2620, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.7296 [2021-07-08 06:56:50,699][root][INFO] - ===> Epoch[5105](25525/5): Loss 0.0111, Sem 0.0111 LR: 6.073e-02 Score 99.597 Data time: 0.3009, Forward time: 1.2620, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.8200 [2021-07-08 06:57:04,464][root][INFO] - ===> Epoch[5106](25530/5): Loss 0.0113, Sem 0.0113 LR: 6.072e-02 Score 99.598 Data time: 0.2187, Forward time: 1.2621, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.7059 [2021-07-08 06:57:18,972][root][INFO] - ===> Epoch[5107](25535/5): Loss 0.0114, Sem 0.0114 LR: 6.072e-02 Score 99.588 Data time: 0.1890, Forward time: 1.2621, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.8523 [2021-07-08 06:57:32,508][root][INFO] - ===> Epoch[5108](25540/5): Loss 0.0115, Sem 0.0115 LR: 6.071e-02 Score 99.583 Data time: 0.2584, Forward time: 1.2621, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.6604 [2021-07-08 06:57:47,179][root][INFO] - ===> Epoch[5109](25545/5): Loss 0.0114, Sem 0.0114 LR: 6.070e-02 Score 99.585 Data time: 0.2066, Forward time: 1.2621, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.8782 [2021-07-08 06:58:00,963][root][INFO] - ===> Epoch[5110](25550/5): Loss 0.0112, Sem 0.0112 LR: 6.069e-02 Score 99.595 Data time: 0.1897, Forward time: 1.2621, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.7149 [2021-07-08 06:58:15,301][root][INFO] - ===> Epoch[5111](25555/5): Loss 0.0112, Sem 0.0112 LR: 6.069e-02 Score 99.593 Data time: 0.2207, Forward time: 1.2621, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.8194 [2021-07-08 06:58:28,940][root][INFO] - ===> Epoch[5112](25560/5): Loss 0.0120, Sem 0.0120 LR: 6.068e-02 Score 99.570 Data time: 0.2213, Forward time: 1.2621, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.6859 [2021-07-08 06:58:43,061][root][INFO] - ===> Epoch[5113](25565/5): Loss 0.0114, Sem 0.0114 LR: 6.067e-02 Score 99.590 Data time: 0.1905, Forward time: 1.2621, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.7795 [2021-07-08 06:58:57,616][root][INFO] - ===> Epoch[5114](25570/5): Loss 0.0111, Sem 0.0111 LR: 6.066e-02 Score 99.596 Data time: 0.2607, Forward time: 1.2621, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.8609 [2021-07-08 06:59:12,357][root][INFO] - ===> Epoch[5115](25575/5): Loss 0.0118, Sem 0.0118 LR: 6.065e-02 Score 99.570 Data time: 0.2000, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.8975 [2021-07-08 06:59:27,045][root][INFO] - ===> Epoch[5116](25580/5): Loss 0.0110, Sem 0.0110 LR: 6.065e-02 Score 99.599 Data time: 0.2212, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.8869 [2021-07-08 06:59:40,404][root][INFO] - ===> Epoch[5117](25585/5): Loss 0.0113, Sem 0.0113 LR: 6.064e-02 Score 99.589 Data time: 0.2362, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6223 [2021-07-08 06:59:54,393][root][INFO] - ===> Epoch[5118](25590/5): Loss 0.0117, Sem 0.0117 LR: 6.063e-02 Score 99.577 Data time: 0.2256, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7496 [2021-07-08 07:00:08,346][root][INFO] - ===> Epoch[5119](25595/5): Loss 0.0115, Sem 0.0115 LR: 6.062e-02 Score 99.581 Data time: 0.2102, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7485 [2021-07-08 07:00:22,512][root][INFO] - ===> Epoch[5120](25600/5): Loss 0.0113, Sem 0.0113 LR: 6.061e-02 Score 99.591 Data time: 0.2475, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7802 [2021-07-08 07:00:22,513][root][INFO] - ===> Start testing [2021-07-08 07:01:01,284][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1271 Loss 0.220 (AVG: 0.556) Score 94.927 (AVG: 91.222) mIOU 71.156 mAP 84.417 mAcc 77.613 IOU: 65.474 0.000 84.335 74.859 94.843 92.114 34.720 82.714 97.807 71.401 79.712 87.494 59.557 mAP: 84.376 7.383 92.805 86.720 98.930 98.396 62.649 98.089 99.774 90.093 92.345 94.726 91.135 mAcc: 81.003 0.000 87.909 82.136 96.156 96.812 37.523 96.189 99.483 81.581 88.947 96.892 64.344 [2021-07-08 07:01:01,284][root][INFO] - Finished test. Elapsed time: 38.7694 [2021-07-08 07:01:02,346][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 07:01:15,777][root][INFO] - ===> Epoch[5121](25605/5): Loss 0.0112, Sem 0.0112 LR: 6.061e-02 Score 99.596 Data time: 0.1339, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6521 [2021-07-08 07:01:30,533][root][INFO] - ===> Epoch[5122](25610/5): Loss 0.0107, Sem 0.0107 LR: 6.060e-02 Score 99.615 Data time: 0.2406, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.9135 [2021-07-08 07:01:44,787][root][INFO] - ===> Epoch[5123](25615/5): Loss 0.0115, Sem 0.0115 LR: 6.059e-02 Score 99.582 Data time: 0.2049, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7944 [2021-07-08 07:01:59,150][root][INFO] - ===> Epoch[5124](25620/5): Loss 0.0113, Sem 0.0113 LR: 6.058e-02 Score 99.590 Data time: 0.2005, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8283 [2021-07-08 07:02:12,619][root][INFO] - ===> Epoch[5125](25625/5): Loss 0.0113, Sem 0.0113 LR: 6.057e-02 Score 99.592 Data time: 0.2019, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6497 [2021-07-08 07:02:26,592][root][INFO] - ===> Epoch[5126](25630/5): Loss 0.0107, Sem 0.0107 LR: 6.057e-02 Score 99.613 Data time: 0.1830, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7548 [2021-07-08 07:02:40,584][root][INFO] - ===> Epoch[5127](25635/5): Loss 0.0115, Sem 0.0115 LR: 6.056e-02 Score 99.584 Data time: 0.2402, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7524 [2021-07-08 07:02:53,509][root][INFO] - ===> Epoch[5128](25640/5): Loss 0.0120, Sem 0.0120 LR: 6.055e-02 Score 99.568 Data time: 0.1699, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.5447 [2021-07-08 07:03:07,428][root][INFO] - ===> Epoch[5129](25645/5): Loss 0.0114, Sem 0.0114 LR: 6.054e-02 Score 99.587 Data time: 0.3113, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7312 [2021-07-08 07:03:20,652][root][INFO] - ===> Epoch[5130](25650/5): Loss 0.0116, Sem 0.0116 LR: 6.053e-02 Score 99.578 Data time: 0.1766, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6022 [2021-07-08 07:03:34,875][root][INFO] - ===> Epoch[5131](25655/5): Loss 0.0120, Sem 0.0120 LR: 6.053e-02 Score 99.569 Data time: 0.2960, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7933 [2021-07-08 07:03:50,436][root][INFO] - ===> Epoch[5132](25660/5): Loss 0.0123, Sem 0.0123 LR: 6.052e-02 Score 99.553 Data time: 0.2921, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 3.0606 [2021-07-08 07:04:04,016][root][INFO] - ===> Epoch[5133](25665/5): Loss 0.0126, Sem 0.0126 LR: 6.051e-02 Score 99.552 Data time: 0.2524, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6676 [2021-07-08 07:04:18,104][root][INFO] - ===> Epoch[5134](25670/5): Loss 0.0154, Sem 0.0154 LR: 6.050e-02 Score 99.504 Data time: 0.2534, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7762 [2021-07-08 07:04:32,165][root][INFO] - ===> Epoch[5135](25675/5): Loss 0.0145, Sem 0.0145 LR: 6.049e-02 Score 99.484 Data time: 0.2438, Forward time: 1.2620, Backward time: 1.2361, DDP time: 0.0042, Total iter time: 2.7657 [2021-07-08 07:04:46,495][root][INFO] - ===> Epoch[5136](25680/5): Loss 0.0152, Sem 0.0152 LR: 6.049e-02 Score 99.473 Data time: 0.1842, Forward time: 1.2621, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.8163 [2021-07-08 07:04:59,995][root][INFO] - ===> Epoch[5137](25685/5): Loss 0.0147, Sem 0.0147 LR: 6.048e-02 Score 99.475 Data time: 0.1981, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6651 [2021-07-08 07:05:13,320][root][INFO] - ===> Epoch[5138](25690/5): Loss 0.0141, Sem 0.0141 LR: 6.047e-02 Score 99.494 Data time: 0.2324, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.6176 [2021-07-08 07:05:27,617][root][INFO] - ===> Epoch[5139](25695/5): Loss 0.0143, Sem 0.0143 LR: 6.046e-02 Score 99.497 Data time: 0.2063, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.8194 [2021-07-08 07:05:41,585][root][INFO] - ===> Epoch[5140](25700/5): Loss 0.0132, Sem 0.0132 LR: 6.046e-02 Score 99.526 Data time: 0.2243, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7367 [2021-07-08 07:05:55,631][root][INFO] - ===> Epoch[5141](25705/5): Loss 0.0127, Sem 0.0127 LR: 6.045e-02 Score 99.545 Data time: 0.2353, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7631 [2021-07-08 07:06:09,533][root][INFO] - ===> Epoch[5142](25710/5): Loss 0.0124, Sem 0.0124 LR: 6.044e-02 Score 99.552 Data time: 0.1917, Forward time: 1.2620, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.7445 [2021-07-08 07:06:23,876][root][INFO] - ===> Epoch[5143](25715/5): Loss 0.0124, Sem 0.0124 LR: 6.043e-02 Score 99.551 Data time: 0.2380, Forward time: 1.2619, Backward time: 1.2362, DDP time: 0.0042, Total iter time: 2.8265 [2021-07-08 07:06:38,948][root][INFO] - ===> Epoch[5144](25720/5): Loss 0.0121, Sem 0.0121 LR: 6.042e-02 Score 99.568 Data time: 0.2775, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.9540 [2021-07-08 07:06:53,422][root][INFO] - ===> Epoch[5145](25725/5): Loss 0.0124, Sem 0.0124 LR: 6.042e-02 Score 99.553 Data time: 0.2509, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8358 [2021-07-08 07:07:07,806][root][INFO] - ===> Epoch[5146](25730/5): Loss 0.0120, Sem 0.0120 LR: 6.041e-02 Score 99.563 Data time: 0.2184, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8342 [2021-07-08 07:07:21,985][root][INFO] - ===> Epoch[5147](25735/5): Loss 0.0115, Sem 0.0115 LR: 6.040e-02 Score 99.586 Data time: 0.1906, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7946 [2021-07-08 07:07:35,614][root][INFO] - ===> Epoch[5148](25740/5): Loss 0.0121, Sem 0.0121 LR: 6.039e-02 Score 99.562 Data time: 0.2316, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6849 [2021-07-08 07:07:49,563][root][INFO] - ===> Epoch[5149](25745/5): Loss 0.0122, Sem 0.0122 LR: 6.038e-02 Score 99.562 Data time: 0.1643, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7406 [2021-07-08 07:08:03,258][root][INFO] - ===> Epoch[5150](25750/5): Loss 0.0119, Sem 0.0119 LR: 6.038e-02 Score 99.570 Data time: 0.2265, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6909 [2021-07-08 07:08:18,178][root][INFO] - ===> Epoch[5151](25755/5): Loss 0.0123, Sem 0.0123 LR: 6.037e-02 Score 99.554 Data time: 0.2404, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.9395 [2021-07-08 07:08:32,722][root][INFO] - ===> Epoch[5152](25760/5): Loss 0.0119, Sem 0.0119 LR: 6.036e-02 Score 99.575 Data time: 0.2165, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8670 [2021-07-08 07:08:46,319][root][INFO] - ===> Epoch[5153](25765/5): Loss 0.0122, Sem 0.0122 LR: 6.035e-02 Score 99.559 Data time: 0.2264, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6801 [2021-07-08 07:09:00,248][root][INFO] - ===> Epoch[5154](25770/5): Loss 0.0117, Sem 0.0117 LR: 6.034e-02 Score 99.577 Data time: 0.3000, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7318 [2021-07-08 07:09:14,342][root][INFO] - ===> Epoch[5155](25775/5): Loss 0.0122, Sem 0.0122 LR: 6.034e-02 Score 99.561 Data time: 0.1764, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7714 [2021-07-08 07:09:28,061][root][INFO] - ===> Epoch[5156](25780/5): Loss 0.0121, Sem 0.0121 LR: 6.033e-02 Score 99.566 Data time: 0.2268, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6957 [2021-07-08 07:09:42,065][root][INFO] - ===> Epoch[5157](25785/5): Loss 0.0128, Sem 0.0128 LR: 6.032e-02 Score 99.542 Data time: 0.2627, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7511 [2021-07-08 07:09:55,756][root][INFO] - ===> Epoch[5158](25790/5): Loss 0.0134, Sem 0.0134 LR: 6.031e-02 Score 99.528 Data time: 0.2030, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6816 [2021-07-08 07:10:10,638][root][INFO] - ===> Epoch[5159](25795/5): Loss 0.0157, Sem 0.0157 LR: 6.030e-02 Score 99.449 Data time: 0.2217, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.9388 [2021-07-08 07:10:25,278][root][INFO] - ===> Epoch[5160](25800/5): Loss 0.0176, Sem 0.0176 LR: 6.030e-02 Score 99.392 Data time: 0.2245, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8742 [2021-07-08 07:10:25,279][root][INFO] - ===> Start testing [2021-07-08 07:11:07,552][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1347 Loss 0.273 (AVG: 0.637) Score 94.198 (AVG: 90.255) mIOU 68.131 mAP 82.099 mAcc 74.308 IOU: 63.767 0.000 78.377 71.842 94.758 92.180 24.400 82.988 98.234 58.948 78.900 85.420 55.889 mAP: 83.169 8.988 90.587 84.800 99.134 98.582 52.983 97.527 99.781 80.987 91.403 93.090 86.263 mAcc: 78.102 0.000 82.670 78.679 95.955 95.847 27.905 94.301 99.494 60.478 90.431 97.035 65.109 [2021-07-08 07:11:07,552][root][INFO] - Finished test. Elapsed time: 42.2727 [2021-07-08 07:11:08,631][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 07:11:22,121][root][INFO] - ===> Epoch[5161](25805/5): Loss 0.0228, Sem 0.0228 LR: 6.029e-02 Score 99.234 Data time: 0.1538, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6519 [2021-07-08 07:11:36,143][root][INFO] - ===> Epoch[5162](25810/5): Loss 0.0210, Sem 0.0210 LR: 6.028e-02 Score 99.280 Data time: 0.1955, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7541 [2021-07-08 07:11:50,248][root][INFO] - ===> Epoch[5163](25815/5): Loss 0.0199, Sem 0.0199 LR: 6.027e-02 Score 99.309 Data time: 0.2128, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7707 [2021-07-08 07:12:04,421][root][INFO] - ===> Epoch[5164](25820/5): Loss 0.0183, Sem 0.0183 LR: 6.026e-02 Score 99.363 Data time: 0.2179, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7907 [2021-07-08 07:12:18,852][root][INFO] - ===> Epoch[5165](25825/5): Loss 0.0194, Sem 0.0194 LR: 6.026e-02 Score 99.327 Data time: 0.2298, Forward time: 1.2620, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.8395 [2021-07-08 07:12:32,847][root][INFO] - ===> Epoch[5166](25830/5): Loss 0.0182, Sem 0.0182 LR: 6.025e-02 Score 99.355 Data time: 0.2282, Forward time: 1.2620, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7551 [2021-07-08 07:12:47,259][root][INFO] - ===> Epoch[5167](25835/5): Loss 0.0175, Sem 0.0175 LR: 6.024e-02 Score 99.377 Data time: 0.1789, Forward time: 1.2620, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.8332 [2021-07-08 07:13:00,973][root][INFO] - ===> Epoch[5168](25840/5): Loss 0.0179, Sem 0.0179 LR: 6.023e-02 Score 99.363 Data time: 0.2065, Forward time: 1.2619, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7032 [2021-07-08 07:13:15,194][root][INFO] - ===> Epoch[5169](25845/5): Loss 0.0158, Sem 0.0158 LR: 6.023e-02 Score 99.436 Data time: 0.1755, Forward time: 1.2619, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.8047 [2021-07-08 07:13:29,105][root][INFO] - ===> Epoch[5170](25850/5): Loss 0.0162, Sem 0.0162 LR: 6.022e-02 Score 99.428 Data time: 0.2715, Forward time: 1.2619, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7325 [2021-07-08 07:13:43,079][root][INFO] - ===> Epoch[5171](25855/5): Loss 0.0173, Sem 0.0173 LR: 6.021e-02 Score 99.384 Data time: 0.2363, Forward time: 1.2619, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7408 [2021-07-08 07:13:56,771][root][INFO] - ===> Epoch[5172](25860/5): Loss 0.0155, Sem 0.0155 LR: 6.020e-02 Score 99.442 Data time: 0.1772, Forward time: 1.2619, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.6970 [2021-07-08 07:14:10,769][root][INFO] - ===> Epoch[5173](25865/5): Loss 0.0142, Sem 0.0142 LR: 6.019e-02 Score 99.488 Data time: 0.1921, Forward time: 1.2619, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7598 [2021-07-08 07:14:23,933][root][INFO] - ===> Epoch[5174](25870/5): Loss 0.0145, Sem 0.0145 LR: 6.019e-02 Score 99.476 Data time: 0.2157, Forward time: 1.2619, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.5888 [2021-07-08 07:14:37,425][root][INFO] - ===> Epoch[5175](25875/5): Loss 0.0141, Sem 0.0141 LR: 6.018e-02 Score 99.493 Data time: 0.2557, Forward time: 1.2619, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.6606 [2021-07-08 07:14:51,316][root][INFO] - ===> Epoch[5176](25880/5): Loss 0.0144, Sem 0.0144 LR: 6.017e-02 Score 99.493 Data time: 0.2318, Forward time: 1.2619, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7305 [2021-07-08 07:15:05,277][root][INFO] - ===> Epoch[5177](25885/5): Loss 0.0131, Sem 0.0131 LR: 6.016e-02 Score 99.531 Data time: 0.2295, Forward time: 1.2619, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7517 [2021-07-08 07:15:18,858][root][INFO] - ===> Epoch[5178](25890/5): Loss 0.0138, Sem 0.0138 LR: 6.015e-02 Score 99.499 Data time: 0.1996, Forward time: 1.2619, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.6699 [2021-07-08 07:15:32,402][root][INFO] - ===> Epoch[5179](25895/5): Loss 0.0132, Sem 0.0132 LR: 6.015e-02 Score 99.527 Data time: 0.2107, Forward time: 1.2619, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.6598 [2021-07-08 07:15:46,994][root][INFO] - ===> Epoch[5180](25900/5): Loss 0.0137, Sem 0.0137 LR: 6.014e-02 Score 99.510 Data time: 0.1855, Forward time: 1.2618, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.8825 [2021-07-08 07:16:00,995][root][INFO] - ===> Epoch[5181](25905/5): Loss 0.0131, Sem 0.0131 LR: 6.013e-02 Score 99.532 Data time: 0.1907, Forward time: 1.2619, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7557 [2021-07-08 07:16:15,289][root][INFO] - ===> Epoch[5182](25910/5): Loss 0.0132, Sem 0.0132 LR: 6.012e-02 Score 99.522 Data time: 0.2341, Forward time: 1.2618, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.8125 [2021-07-08 07:16:29,083][root][INFO] - ===> Epoch[5183](25915/5): Loss 0.0140, Sem 0.0140 LR: 6.011e-02 Score 99.508 Data time: 0.2624, Forward time: 1.2618, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7044 [2021-07-08 07:16:42,355][root][INFO] - ===> Epoch[5184](25920/5): Loss 0.0138, Sem 0.0138 LR: 6.011e-02 Score 99.506 Data time: 0.2042, Forward time: 1.2618, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.6148 [2021-07-08 07:16:57,659][root][INFO] - ===> Epoch[5185](25925/5): Loss 0.0145, Sem 0.0145 LR: 6.010e-02 Score 99.502 Data time: 0.2432, Forward time: 1.2619, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 3.0080 [2021-07-08 07:17:11,265][root][INFO] - ===> Epoch[5186](25930/5): Loss 0.0152, Sem 0.0152 LR: 6.009e-02 Score 99.486 Data time: 0.2330, Forward time: 1.2619, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.6734 [2021-07-08 07:17:25,249][root][INFO] - ===> Epoch[5187](25935/5): Loss 0.0138, Sem 0.0138 LR: 6.008e-02 Score 99.505 Data time: 0.2600, Forward time: 1.2619, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7492 [2021-07-08 07:17:39,842][root][INFO] - ===> Epoch[5188](25940/5): Loss 0.0162, Sem 0.0162 LR: 6.007e-02 Score 99.453 Data time: 0.2938, Forward time: 1.2619, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.8741 [2021-07-08 07:17:54,143][root][INFO] - ===> Epoch[5189](25945/5): Loss 0.0158, Sem 0.0158 LR: 6.007e-02 Score 99.461 Data time: 0.3195, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8070 [2021-07-08 07:18:08,429][root][INFO] - ===> Epoch[5190](25950/5): Loss 0.0151, Sem 0.0151 LR: 6.006e-02 Score 99.464 Data time: 0.2590, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8067 [2021-07-08 07:18:22,405][root][INFO] - ===> Epoch[5191](25955/5): Loss 0.0162, Sem 0.0162 LR: 6.005e-02 Score 99.439 Data time: 0.2243, Forward time: 1.2620, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7608 [2021-07-08 07:18:35,587][root][INFO] - ===> Epoch[5192](25960/5): Loss 0.0184, Sem 0.0184 LR: 6.004e-02 Score 99.391 Data time: 0.1822, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.5868 [2021-07-08 07:18:49,110][root][INFO] - ===> Epoch[5193](25965/5): Loss 0.0164, Sem 0.0164 LR: 6.003e-02 Score 99.419 Data time: 0.2044, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6511 [2021-07-08 07:19:03,547][root][INFO] - ===> Epoch[5194](25970/5): Loss 0.0163, Sem 0.0163 LR: 6.003e-02 Score 99.432 Data time: 0.2407, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8365 [2021-07-08 07:19:17,811][root][INFO] - ===> Epoch[5195](25975/5): Loss 0.0147, Sem 0.0147 LR: 6.002e-02 Score 99.474 Data time: 0.2424, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8077 [2021-07-08 07:19:31,860][root][INFO] - ===> Epoch[5196](25980/5): Loss 0.0159, Sem 0.0159 LR: 6.001e-02 Score 99.444 Data time: 0.1698, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7651 [2021-07-08 07:19:45,216][root][INFO] - ===> Epoch[5197](25985/5): Loss 0.0148, Sem 0.0148 LR: 6.000e-02 Score 99.471 Data time: 0.1821, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6355 [2021-07-08 07:19:58,702][root][INFO] - ===> Epoch[5198](25990/5): Loss 0.0148, Sem 0.0148 LR: 5.999e-02 Score 99.472 Data time: 0.2020, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6548 [2021-07-08 07:20:12,819][root][INFO] - ===> Epoch[5199](25995/5): Loss 0.0135, Sem 0.0135 LR: 5.999e-02 Score 99.515 Data time: 0.2497, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7868 [2021-07-08 07:20:26,928][root][INFO] - ===> Epoch[5200](26000/5): Loss 0.0138, Sem 0.0138 LR: 5.998e-02 Score 99.503 Data time: 0.2275, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7766 [2021-07-08 07:20:26,929][root][INFO] - ===> Start testing [2021-07-08 07:21:11,577][root][INFO] - 68/68: Data time: 0.0440, Iter time: 0.1560 Loss 0.260 (AVG: 0.540) Score 93.832 (AVG: 91.025) mIOU 70.319 mAP 83.613 mAcc 76.289 IOU: 64.351 0.000 85.070 75.703 94.595 91.703 30.928 86.088 98.214 63.086 79.572 86.246 58.586 mAP: 84.721 5.166 95.017 88.072 98.901 98.840 61.929 98.171 99.772 77.992 92.151 95.126 91.112 mAcc: 79.272 0.000 89.607 83.488 96.492 97.394 35.226 95.925 99.376 66.062 88.519 96.386 64.005 [2021-07-08 07:21:11,578][root][INFO] - Finished test. Elapsed time: 44.6477 [2021-07-08 07:21:12,625][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 07:21:25,623][root][INFO] - ===> Epoch[5201](26005/5): Loss 0.0137, Sem 0.0137 LR: 5.997e-02 Score 99.510 Data time: 0.2542, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.5553 [2021-07-08 07:21:39,294][root][INFO] - ===> Epoch[5202](26010/5): Loss 0.0150, Sem 0.0150 LR: 5.996e-02 Score 99.509 Data time: 0.1829, Forward time: 1.2618, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6920 [2021-07-08 07:21:53,373][root][INFO] - ===> Epoch[5203](26015/5): Loss 0.0165, Sem 0.0165 LR: 5.996e-02 Score 99.456 Data time: 0.1922, Forward time: 1.2618, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7691 [2021-07-08 07:22:07,400][root][INFO] - ===> Epoch[5204](26020/5): Loss 0.0151, Sem 0.0151 LR: 5.995e-02 Score 99.467 Data time: 0.2110, Forward time: 1.2618, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7562 [2021-07-08 07:22:20,996][root][INFO] - ===> Epoch[5205](26025/5): Loss 0.0170, Sem 0.0170 LR: 5.994e-02 Score 99.395 Data time: 0.1720, Forward time: 1.2618, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.6737 [2021-07-08 07:22:35,497][root][INFO] - ===> Epoch[5206](26030/5): Loss 0.0195, Sem 0.0195 LR: 5.993e-02 Score 99.338 Data time: 0.2620, Forward time: 1.2618, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.8559 [2021-07-08 07:22:49,348][root][INFO] - ===> Epoch[5207](26035/5): Loss 0.0203, Sem 0.0203 LR: 5.992e-02 Score 99.311 Data time: 0.2393, Forward time: 1.2618, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7164 [2021-07-08 07:23:03,616][root][INFO] - ===> Epoch[5208](26040/5): Loss 0.0216, Sem 0.0216 LR: 5.992e-02 Score 99.251 Data time: 0.2006, Forward time: 1.2618, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7960 [2021-07-08 07:23:16,953][root][INFO] - ===> Epoch[5209](26045/5): Loss 0.0258, Sem 0.0258 LR: 5.991e-02 Score 99.166 Data time: 0.2458, Forward time: 1.2618, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.6109 [2021-07-08 07:23:30,860][root][INFO] - ===> Epoch[5210](26050/5): Loss 0.0278, Sem 0.0278 LR: 5.990e-02 Score 99.068 Data time: 0.2172, Forward time: 1.2618, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7321 [2021-07-08 07:23:45,595][root][INFO] - ===> Epoch[5211](26055/5): Loss 0.0250, Sem 0.0250 LR: 5.989e-02 Score 99.139 Data time: 0.1985, Forward time: 1.2618, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.9004 [2021-07-08 07:23:58,971][root][INFO] - ===> Epoch[5212](26060/5): Loss 0.0295, Sem 0.0295 LR: 5.988e-02 Score 99.024 Data time: 0.2776, Forward time: 1.2618, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6186 [2021-07-08 07:24:13,192][root][INFO] - ===> Epoch[5213](26065/5): Loss 0.0347, Sem 0.0347 LR: 5.988e-02 Score 98.881 Data time: 0.2638, Forward time: 1.2618, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8020 [2021-07-08 07:24:28,258][root][INFO] - ===> Epoch[5214](26070/5): Loss 0.0350, Sem 0.0350 LR: 5.987e-02 Score 98.852 Data time: 0.2434, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.9657 [2021-07-08 07:24:42,004][root][INFO] - ===> Epoch[5215](26075/5): Loss 0.0326, Sem 0.0326 LR: 5.986e-02 Score 98.915 Data time: 0.2373, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7046 [2021-07-08 07:24:56,265][root][INFO] - ===> Epoch[5216](26080/5): Loss 0.0318, Sem 0.0318 LR: 5.985e-02 Score 98.920 Data time: 0.2700, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7988 [2021-07-08 07:25:09,899][root][INFO] - ===> Epoch[5217](26085/5): Loss 0.0350, Sem 0.0350 LR: 5.984e-02 Score 98.844 Data time: 0.2197, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6853 [2021-07-08 07:25:23,808][root][INFO] - ===> Epoch[5218](26090/5): Loss 0.0353, Sem 0.0353 LR: 5.984e-02 Score 98.832 Data time: 0.1995, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.7372 [2021-07-08 07:25:38,131][root][INFO] - ===> Epoch[5219](26095/5): Loss 0.0371, Sem 0.0371 LR: 5.983e-02 Score 98.775 Data time: 0.2256, Forward time: 1.2619, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.8146 [2021-07-08 07:25:51,477][root][INFO] - ===> Epoch[5220](26100/5): Loss 0.0363, Sem 0.0363 LR: 5.982e-02 Score 98.776 Data time: 0.2188, Forward time: 1.2618, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6204 [2021-07-08 07:26:05,124][root][INFO] - ===> Epoch[5221](26105/5): Loss 0.0364, Sem 0.0364 LR: 5.981e-02 Score 98.788 Data time: 0.2449, Forward time: 1.2618, Backward time: 1.2363, DDP time: 0.0042, Total iter time: 2.6792 [2021-07-08 07:26:19,565][root][INFO] - ===> Epoch[5222](26110/5): Loss 0.0389, Sem 0.0389 LR: 5.980e-02 Score 98.724 Data time: 0.2094, Forward time: 1.2618, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.8420 [2021-07-08 07:26:34,303][root][INFO] - ===> Epoch[5223](26115/5): Loss 0.0425, Sem 0.0425 LR: 5.980e-02 Score 98.603 Data time: 0.1597, Forward time: 1.2618, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.8881 [2021-07-08 07:26:47,716][root][INFO] - ===> Epoch[5224](26120/5): Loss 0.0405, Sem 0.0405 LR: 5.979e-02 Score 98.654 Data time: 0.2357, Forward time: 1.2617, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.6342 [2021-07-08 07:27:01,162][root][INFO] - ===> Epoch[5225](26125/5): Loss 0.0372, Sem 0.0372 LR: 5.978e-02 Score 98.729 Data time: 0.2069, Forward time: 1.2617, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.6420 [2021-07-08 07:27:14,847][root][INFO] - ===> Epoch[5226](26130/5): Loss 0.0343, Sem 0.0343 LR: 5.977e-02 Score 98.849 Data time: 0.2338, Forward time: 1.2617, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.6836 [2021-07-08 07:27:28,673][root][INFO] - ===> Epoch[5227](26135/5): Loss 0.0354, Sem 0.0354 LR: 5.976e-02 Score 98.827 Data time: 0.1820, Forward time: 1.2617, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7205 [2021-07-08 07:27:42,740][root][INFO] - ===> Epoch[5228](26140/5): Loss 0.0299, Sem 0.0299 LR: 5.976e-02 Score 99.010 Data time: 0.2514, Forward time: 1.2617, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7637 [2021-07-08 07:27:56,662][root][INFO] - ===> Epoch[5229](26145/5): Loss 0.0312, Sem 0.0312 LR: 5.975e-02 Score 98.974 Data time: 0.2204, Forward time: 1.2617, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7394 [2021-07-08 07:28:11,354][root][INFO] - ===> Epoch[5230](26150/5): Loss 0.0329, Sem 0.0329 LR: 5.974e-02 Score 98.926 Data time: 0.2058, Forward time: 1.2617, Backward time: 1.2365, DDP time: 0.0042, Total iter time: 2.8859 [2021-07-08 07:28:25,556][root][INFO] - ===> Epoch[5231](26155/5): Loss 0.0348, Sem 0.0348 LR: 5.973e-02 Score 98.832 Data time: 0.1932, Forward time: 1.2617, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.8003 [2021-07-08 07:28:39,612][root][INFO] - ===> Epoch[5232](26160/5): Loss 0.0309, Sem 0.0309 LR: 5.973e-02 Score 98.933 Data time: 0.1706, Forward time: 1.2617, Backward time: 1.2364, DDP time: 0.0042, Total iter time: 2.7595 [2021-07-08 07:28:53,645][root][INFO] - ===> Epoch[5233](26165/5): Loss 0.0366, Sem 0.0366 LR: 5.972e-02 Score 98.828 Data time: 0.2149, Forward time: 1.2617, Backward time: 1.2365, DDP time: 0.0042, Total iter time: 2.7577 [2021-07-08 07:29:07,455][root][INFO] - ===> Epoch[5234](26170/5): Loss 0.0359, Sem 0.0359 LR: 5.971e-02 Score 98.816 Data time: 0.1737, Forward time: 1.2617, Backward time: 1.2365, DDP time: 0.0042, Total iter time: 2.7167 [2021-07-08 07:29:21,635][root][INFO] - ===> Epoch[5235](26175/5): Loss 0.0445, Sem 0.0445 LR: 5.970e-02 Score 98.562 Data time: 0.2277, Forward time: 1.2617, Backward time: 1.2365, DDP time: 0.0042, Total iter time: 2.7852 [2021-07-08 07:29:35,910][root][INFO] - ===> Epoch[5236](26180/5): Loss 0.0387, Sem 0.0387 LR: 5.969e-02 Score 98.711 Data time: 0.2244, Forward time: 1.2617, Backward time: 1.2365, DDP time: 0.0042, Total iter time: 2.8083 [2021-07-08 07:29:49,942][root][INFO] - ===> Epoch[5237](26185/5): Loss 0.0382, Sem 0.0382 LR: 5.969e-02 Score 98.738 Data time: 0.2022, Forward time: 1.2617, Backward time: 1.2365, DDP time: 0.0042, Total iter time: 2.7650 [2021-07-08 07:30:03,366][root][INFO] - ===> Epoch[5238](26190/5): Loss 0.0393, Sem 0.0393 LR: 5.968e-02 Score 98.704 Data time: 0.2005, Forward time: 1.2617, Backward time: 1.2365, DDP time: 0.0042, Total iter time: 2.6427 [2021-07-08 07:30:18,350][root][INFO] - ===> Epoch[5239](26195/5): Loss 0.0359, Sem 0.0359 LR: 5.967e-02 Score 98.783 Data time: 0.2206, Forward time: 1.2617, Backward time: 1.2365, DDP time: 0.0042, Total iter time: 2.9562 [2021-07-08 07:30:32,963][root][INFO] - ===> Epoch[5240](26200/5): Loss 0.0370, Sem 0.0370 LR: 5.966e-02 Score 98.754 Data time: 0.1990, Forward time: 1.2617, Backward time: 1.2365, DDP time: 0.0042, Total iter time: 2.8793 [2021-07-08 07:30:32,964][root][INFO] - ===> Start testing [2021-07-08 07:31:14,414][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1285 Loss 0.227 (AVG: 0.443) Score 94.053 (AVG: 89.822) mIOU 67.021 mAP 83.859 mAcc 75.802 IOU: 59.728 0.000 78.957 72.965 94.004 88.039 35.421 76.594 98.104 50.544 72.597 86.405 57.922 mAP: 83.050 2.236 87.468 89.337 98.458 98.620 66.638 94.898 99.833 93.272 91.031 95.119 90.204 mAcc: 81.150 0.000 82.618 82.130 95.341 96.273 40.050 88.943 99.274 83.396 78.967 95.058 62.225 [2021-07-08 07:31:14,414][root][INFO] - Finished test. Elapsed time: 41.4490 [2021-07-08 07:31:15,481][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 07:31:28,522][root][INFO] - ===> Epoch[5241](26205/5): Loss 0.0345, Sem 0.0345 LR: 5.965e-02 Score 98.857 Data time: 0.2034, Forward time: 1.2617, Backward time: 1.2366, DDP time: 0.0042, Total iter time: 2.5719 [2021-07-08 07:31:42,258][root][INFO] - ===> Epoch[5242](26210/5): Loss 0.0330, Sem 0.0330 LR: 5.965e-02 Score 98.878 Data time: 0.2514, Forward time: 1.2617, Backward time: 1.2365, DDP time: 0.0042, Total iter time: 2.6921 [2021-07-08 07:31:57,269][root][INFO] - ===> Epoch[5243](26215/5): Loss 0.0319, Sem 0.0319 LR: 5.964e-02 Score 98.921 Data time: 0.2764, Forward time: 1.2617, Backward time: 1.2366, DDP time: 0.0042, Total iter time: 2.9624 [2021-07-08 07:32:11,068][root][INFO] - ===> Epoch[5244](26220/5): Loss 0.0300, Sem 0.0300 LR: 5.963e-02 Score 98.963 Data time: 0.2134, Forward time: 1.2616, Backward time: 1.2366, DDP time: 0.0042, Total iter time: 2.7267 [2021-07-08 07:32:25,375][root][INFO] - ===> Epoch[5245](26225/5): Loss 0.0280, Sem 0.0280 LR: 5.962e-02 Score 99.041 Data time: 0.2047, Forward time: 1.2616, Backward time: 1.2366, DDP time: 0.0042, Total iter time: 2.8166 [2021-07-08 07:32:39,111][root][INFO] - ===> Epoch[5246](26230/5): Loss 0.0266, Sem 0.0266 LR: 5.961e-02 Score 99.066 Data time: 0.2136, Forward time: 1.2616, Backward time: 1.2366, DDP time: 0.0042, Total iter time: 2.7086 [2021-07-08 07:32:52,587][root][INFO] - ===> Epoch[5247](26235/5): Loss 0.0279, Sem 0.0279 LR: 5.961e-02 Score 99.068 Data time: 0.2020, Forward time: 1.2616, Backward time: 1.2366, DDP time: 0.0042, Total iter time: 2.6591 [2021-07-08 07:33:06,761][root][INFO] - ===> Epoch[5248](26240/5): Loss 0.0252, Sem 0.0252 LR: 5.960e-02 Score 99.119 Data time: 0.2293, Forward time: 1.2615, Backward time: 1.2367, DDP time: 0.0042, Total iter time: 2.7921 [2021-07-08 07:33:19,899][root][INFO] - ===> Epoch[5249](26245/5): Loss 0.0237, Sem 0.0237 LR: 5.959e-02 Score 99.165 Data time: 0.1884, Forward time: 1.2615, Backward time: 1.2367, DDP time: 0.0042, Total iter time: 2.5852 [2021-07-08 07:33:34,662][root][INFO] - ===> Epoch[5250](26250/5): Loss 0.0241, Sem 0.0241 LR: 5.958e-02 Score 99.163 Data time: 0.2585, Forward time: 1.2615, Backward time: 1.2367, DDP time: 0.0042, Total iter time: 2.9070 [2021-07-08 07:33:48,488][root][INFO] - ===> Epoch[5251](26255/5): Loss 0.0249, Sem 0.0249 LR: 5.957e-02 Score 99.131 Data time: 0.2948, Forward time: 1.2615, Backward time: 1.2367, DDP time: 0.0042, Total iter time: 2.7172 [2021-07-08 07:34:02,376][root][INFO] - ===> Epoch[5252](26260/5): Loss 0.0237, Sem 0.0237 LR: 5.957e-02 Score 99.183 Data time: 0.2289, Forward time: 1.2615, Backward time: 1.2367, DDP time: 0.0042, Total iter time: 2.7362 [2021-07-08 07:34:16,948][root][INFO] - ===> Epoch[5253](26265/5): Loss 0.0253, Sem 0.0253 LR: 5.956e-02 Score 99.112 Data time: 0.1935, Forward time: 1.2615, Backward time: 1.2367, DDP time: 0.0042, Total iter time: 2.8706 [2021-07-08 07:34:30,349][root][INFO] - ===> Epoch[5254](26270/5): Loss 0.0221, Sem 0.0221 LR: 5.955e-02 Score 99.216 Data time: 0.2030, Forward time: 1.2615, Backward time: 1.2367, DDP time: 0.0042, Total iter time: 2.6409 [2021-07-08 07:34:44,415][root][INFO] - ===> Epoch[5255](26275/5): Loss 0.0219, Sem 0.0219 LR: 5.954e-02 Score 99.231 Data time: 0.2263, Forward time: 1.2615, Backward time: 1.2367, DDP time: 0.0042, Total iter time: 2.7631 [2021-07-08 07:34:58,507][root][INFO] - ===> Epoch[5256](26280/5): Loss 0.0247, Sem 0.0247 LR: 5.953e-02 Score 99.168 Data time: 0.2489, Forward time: 1.2615, Backward time: 1.2367, DDP time: 0.0042, Total iter time: 2.7687 [2021-07-08 07:35:12,916][root][INFO] - ===> Epoch[5257](26285/5): Loss 0.0224, Sem 0.0224 LR: 5.953e-02 Score 99.223 Data time: 0.2528, Forward time: 1.2615, Backward time: 1.2367, DDP time: 0.0042, Total iter time: 2.8318 [2021-07-08 07:35:27,858][root][INFO] - ===> Epoch[5258](26290/5): Loss 0.0230, Sem 0.0230 LR: 5.952e-02 Score 99.219 Data time: 0.2164, Forward time: 1.2614, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.9416 [2021-07-08 07:35:41,625][root][INFO] - ===> Epoch[5259](26295/5): Loss 0.0211, Sem 0.0211 LR: 5.951e-02 Score 99.260 Data time: 0.2309, Forward time: 1.2614, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.6995 [2021-07-08 07:35:55,815][root][INFO] - ===> Epoch[5260](26300/5): Loss 0.0202, Sem 0.0202 LR: 5.950e-02 Score 99.287 Data time: 0.2015, Forward time: 1.2614, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.7990 [2021-07-08 07:36:09,598][root][INFO] - ===> Epoch[5261](26305/5): Loss 0.0195, Sem 0.0195 LR: 5.949e-02 Score 99.303 Data time: 0.2138, Forward time: 1.2614, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.7182 [2021-07-08 07:36:24,041][root][INFO] - ===> Epoch[5262](26310/5): Loss 0.0190, Sem 0.0190 LR: 5.949e-02 Score 99.323 Data time: 0.2056, Forward time: 1.2614, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.8492 [2021-07-08 07:36:38,525][root][INFO] - ===> Epoch[5263](26315/5): Loss 0.0185, Sem 0.0185 LR: 5.948e-02 Score 99.342 Data time: 0.1846, Forward time: 1.2614, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.8556 [2021-07-08 07:36:52,336][root][INFO] - ===> Epoch[5264](26320/5): Loss 0.0183, Sem 0.0183 LR: 5.947e-02 Score 99.338 Data time: 0.2680, Forward time: 1.2615, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.7164 [2021-07-08 07:37:06,054][root][INFO] - ===> Epoch[5265](26325/5): Loss 0.0181, Sem 0.0181 LR: 5.946e-02 Score 99.368 Data time: 0.2192, Forward time: 1.2615, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.6892 [2021-07-08 07:37:19,543][root][INFO] - ===> Epoch[5266](26330/5): Loss 0.0182, Sem 0.0182 LR: 5.945e-02 Score 99.360 Data time: 0.1747, Forward time: 1.2615, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.6530 [2021-07-08 07:37:33,560][root][INFO] - ===> Epoch[5267](26335/5): Loss 0.0178, Sem 0.0178 LR: 5.945e-02 Score 99.371 Data time: 0.2144, Forward time: 1.2615, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.7532 [2021-07-08 07:37:46,994][root][INFO] - ===> Epoch[5268](26340/5): Loss 0.0171, Sem 0.0171 LR: 5.944e-02 Score 99.388 Data time: 0.1999, Forward time: 1.2615, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.6497 [2021-07-08 07:38:00,361][root][INFO] - ===> Epoch[5269](26345/5): Loss 0.0182, Sem 0.0182 LR: 5.943e-02 Score 99.371 Data time: 0.2181, Forward time: 1.2614, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.6295 [2021-07-08 07:38:14,935][root][INFO] - ===> Epoch[5270](26350/5): Loss 0.0182, Sem 0.0182 LR: 5.942e-02 Score 99.370 Data time: 0.1749, Forward time: 1.2614, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.8709 [2021-07-08 07:38:28,811][root][INFO] - ===> Epoch[5271](26355/5): Loss 0.0182, Sem 0.0182 LR: 5.942e-02 Score 99.364 Data time: 0.1938, Forward time: 1.2614, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.7294 [2021-07-08 07:38:42,436][root][INFO] - ===> Epoch[5272](26360/5): Loss 0.0181, Sem 0.0181 LR: 5.941e-02 Score 99.356 Data time: 0.2410, Forward time: 1.2614, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.6780 [2021-07-08 07:38:57,429][root][INFO] - ===> Epoch[5273](26365/5): Loss 0.0180, Sem 0.0180 LR: 5.940e-02 Score 99.363 Data time: 0.2380, Forward time: 1.2614, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.9467 [2021-07-08 07:39:11,839][root][INFO] - ===> Epoch[5274](26370/5): Loss 0.0171, Sem 0.0171 LR: 5.939e-02 Score 99.394 Data time: 0.2521, Forward time: 1.2614, Backward time: 1.2368, DDP time: 0.0042, Total iter time: 2.8141 [2021-07-08 07:39:26,910][root][INFO] - ===> Epoch[5275](26375/5): Loss 0.0163, Sem 0.0163 LR: 5.938e-02 Score 99.430 Data time: 0.2710, Forward time: 1.2614, Backward time: 1.2369, DDP time: 0.0042, Total iter time: 2.9553 [2021-07-08 07:39:41,428][root][INFO] - ===> Epoch[5276](26380/5): Loss 0.0179, Sem 0.0179 LR: 5.938e-02 Score 99.370 Data time: 0.2107, Forward time: 1.2614, Backward time: 1.2369, DDP time: 0.0042, Total iter time: 2.8661 [2021-07-08 07:39:55,603][root][INFO] - ===> Epoch[5277](26385/5): Loss 0.0174, Sem 0.0174 LR: 5.937e-02 Score 99.382 Data time: 0.1869, Forward time: 1.2613, Backward time: 1.2370, DDP time: 0.0042, Total iter time: 2.7814 [2021-07-08 07:40:09,257][root][INFO] - ===> Epoch[5278](26390/5): Loss 0.0182, Sem 0.0182 LR: 5.936e-02 Score 99.361 Data time: 0.2381, Forward time: 1.2614, Backward time: 1.2369, DDP time: 0.0042, Total iter time: 2.6802 [2021-07-08 07:40:23,813][root][INFO] - ===> Epoch[5279](26395/5): Loss 0.0188, Sem 0.0188 LR: 5.935e-02 Score 99.339 Data time: 0.2721, Forward time: 1.2613, Backward time: 1.2370, DDP time: 0.0042, Total iter time: 2.8660 [2021-07-08 07:40:37,911][root][INFO] - ===> Epoch[5280](26400/5): Loss 0.0163, Sem 0.0163 LR: 5.934e-02 Score 99.418 Data time: 0.1839, Forward time: 1.2613, Backward time: 1.2370, DDP time: 0.0042, Total iter time: 2.7811 [2021-07-08 07:40:37,912][root][INFO] - ===> Start testing [2021-07-08 07:41:16,968][root][INFO] - 68/68: Data time: 0.0019, Iter time: 0.1329 Loss 0.359 (AVG: 0.536) Score 91.522 (AVG: 90.290) mIOU 69.802 mAP 83.396 mAcc 76.610 IOU: 63.158 0.000 80.222 73.390 94.833 91.401 26.815 80.135 98.542 77.923 77.609 85.494 57.908 mAP: 84.344 2.434 88.108 88.507 98.786 98.630 53.894 97.578 99.791 94.933 91.497 94.323 91.320 mAcc: 82.033 0.000 80.872 79.409 96.441 96.676 32.807 95.104 99.433 85.211 87.893 94.832 65.215 [2021-07-08 07:41:16,969][root][INFO] - Finished test. Elapsed time: 39.0553 [2021-07-08 07:41:18,098][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 07:41:30,755][root][INFO] - ===> Epoch[5281](26405/5): Loss 0.0163, Sem 0.0163 LR: 5.934e-02 Score 99.417 Data time: 0.1593, Forward time: 1.2613, Backward time: 1.2370, DDP time: 0.0042, Total iter time: 2.4989 [2021-07-08 07:41:44,801][root][INFO] - ===> Epoch[5282](26410/5): Loss 0.0170, Sem 0.0170 LR: 5.933e-02 Score 99.393 Data time: 0.2242, Forward time: 1.2613, Backward time: 1.2370, DDP time: 0.0042, Total iter time: 2.7595 [2021-07-08 07:41:59,642][root][INFO] - ===> Epoch[5283](26415/5): Loss 0.0256, Sem 0.0256 LR: 5.932e-02 Score 99.243 Data time: 0.2179, Forward time: 1.2613, Backward time: 1.2370, DDP time: 0.0042, Total iter time: 2.9306 [2021-07-08 07:42:13,446][root][INFO] - ===> Epoch[5284](26420/5): Loss 0.0195, Sem 0.0195 LR: 5.931e-02 Score 99.308 Data time: 0.2273, Forward time: 1.2613, Backward time: 1.2370, DDP time: 0.0042, Total iter time: 2.7103 [2021-07-08 07:42:27,087][root][INFO] - ===> Epoch[5285](26425/5): Loss 0.0194, Sem 0.0194 LR: 5.930e-02 Score 99.311 Data time: 0.1839, Forward time: 1.2613, Backward time: 1.2370, DDP time: 0.0042, Total iter time: 2.6885 [2021-07-08 07:42:41,147][root][INFO] - ===> Epoch[5286](26430/5): Loss 0.0177, Sem 0.0177 LR: 5.930e-02 Score 99.366 Data time: 0.2424, Forward time: 1.2613, Backward time: 1.2370, DDP time: 0.0042, Total iter time: 2.7550 [2021-07-08 07:42:54,503][root][INFO] - ===> Epoch[5287](26435/5): Loss 0.0178, Sem 0.0178 LR: 5.929e-02 Score 99.374 Data time: 0.2130, Forward time: 1.2613, Backward time: 1.2370, DDP time: 0.0042, Total iter time: 2.6281 [2021-07-08 07:43:09,231][root][INFO] - ===> Epoch[5288](26440/5): Loss 0.0175, Sem 0.0175 LR: 5.928e-02 Score 99.394 Data time: 0.1837, Forward time: 1.2612, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.8846 [2021-07-08 07:43:23,332][root][INFO] - ===> Epoch[5289](26445/5): Loss 0.0160, Sem 0.0160 LR: 5.927e-02 Score 99.429 Data time: 0.1983, Forward time: 1.2612, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.7779 [2021-07-08 07:43:36,922][root][INFO] - ===> Epoch[5290](26450/5): Loss 0.0169, Sem 0.0169 LR: 5.926e-02 Score 99.401 Data time: 0.2396, Forward time: 1.2613, Backward time: 1.2370, DDP time: 0.0042, Total iter time: 2.6661 [2021-07-08 07:43:50,492][root][INFO] - ===> Epoch[5291](26455/5): Loss 0.0163, Sem 0.0163 LR: 5.926e-02 Score 99.429 Data time: 0.1526, Forward time: 1.2612, Backward time: 1.2370, DDP time: 0.0042, Total iter time: 2.6789 [2021-07-08 07:44:04,676][root][INFO] - ===> Epoch[5292](26460/5): Loss 0.0150, Sem 0.0150 LR: 5.925e-02 Score 99.458 Data time: 0.1433, Forward time: 1.2612, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.8073 [2021-07-08 07:44:17,706][root][INFO] - ===> Epoch[5293](26465/5): Loss 0.0155, Sem 0.0155 LR: 5.924e-02 Score 99.441 Data time: 0.1707, Forward time: 1.2612, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.5684 [2021-07-08 07:44:31,789][root][INFO] - ===> Epoch[5294](26470/5): Loss 0.0149, Sem 0.0149 LR: 5.923e-02 Score 99.463 Data time: 0.2523, Forward time: 1.2612, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.7611 [2021-07-08 07:44:46,350][root][INFO] - ===> Epoch[5295](26475/5): Loss 0.0153, Sem 0.0153 LR: 5.922e-02 Score 99.457 Data time: 0.2479, Forward time: 1.2612, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.8505 [2021-07-08 07:44:59,843][root][INFO] - ===> Epoch[5296](26480/5): Loss 0.0155, Sem 0.0155 LR: 5.922e-02 Score 99.477 Data time: 0.2438, Forward time: 1.2612, Backward time: 1.2370, DDP time: 0.0042, Total iter time: 2.6563 [2021-07-08 07:45:13,727][root][INFO] - ===> Epoch[5297](26485/5): Loss 0.0157, Sem 0.0157 LR: 5.921e-02 Score 99.441 Data time: 0.2452, Forward time: 1.2612, Backward time: 1.2370, DDP time: 0.0042, Total iter time: 2.7290 [2021-07-08 07:45:28,332][root][INFO] - ===> Epoch[5298](26490/5): Loss 0.0147, Sem 0.0147 LR: 5.920e-02 Score 99.471 Data time: 0.2285, Forward time: 1.2612, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.8839 [2021-07-08 07:45:42,213][root][INFO] - ===> Epoch[5299](26495/5): Loss 0.0145, Sem 0.0145 LR: 5.919e-02 Score 99.481 Data time: 0.2283, Forward time: 1.2612, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.7381 [2021-07-08 07:45:56,631][root][INFO] - ===> Epoch[5300](26500/5): Loss 0.0146, Sem 0.0146 LR: 5.918e-02 Score 99.475 Data time: 0.2442, Forward time: 1.2612, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.8358 [2021-07-08 07:46:10,661][root][INFO] - ===> Epoch[5301](26505/5): Loss 0.0150, Sem 0.0150 LR: 5.918e-02 Score 99.460 Data time: 0.2242, Forward time: 1.2612, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.7592 [2021-07-08 07:46:24,975][root][INFO] - ===> Epoch[5302](26510/5): Loss 0.0147, Sem 0.0147 LR: 5.917e-02 Score 99.465 Data time: 0.2778, Forward time: 1.2612, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.8146 [2021-07-08 07:46:39,781][root][INFO] - ===> Epoch[5303](26515/5): Loss 0.0145, Sem 0.0145 LR: 5.916e-02 Score 99.476 Data time: 0.2703, Forward time: 1.2613, Backward time: 1.2370, DDP time: 0.0042, Total iter time: 2.8988 [2021-07-08 07:46:53,624][root][INFO] - ===> Epoch[5304](26520/5): Loss 0.0145, Sem 0.0145 LR: 5.915e-02 Score 99.480 Data time: 0.2021, Forward time: 1.2613, Backward time: 1.2370, DDP time: 0.0042, Total iter time: 2.7159 [2021-07-08 07:47:08,268][root][INFO] - ===> Epoch[5305](26525/5): Loss 0.0148, Sem 0.0148 LR: 5.914e-02 Score 99.464 Data time: 0.1822, Forward time: 1.2613, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.8819 [2021-07-08 07:47:22,856][root][INFO] - ===> Epoch[5306](26530/5): Loss 0.0142, Sem 0.0142 LR: 5.914e-02 Score 99.486 Data time: 0.2290, Forward time: 1.2613, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.8699 [2021-07-08 07:47:36,827][root][INFO] - ===> Epoch[5307](26535/5): Loss 0.0140, Sem 0.0140 LR: 5.913e-02 Score 99.497 Data time: 0.2467, Forward time: 1.2613, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.7534 [2021-07-08 07:47:51,454][root][INFO] - ===> Epoch[5308](26540/5): Loss 0.0149, Sem 0.0149 LR: 5.912e-02 Score 99.478 Data time: 0.1994, Forward time: 1.2612, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.8835 [2021-07-08 07:48:05,265][root][INFO] - ===> Epoch[5309](26545/5): Loss 0.0141, Sem 0.0141 LR: 5.911e-02 Score 99.494 Data time: 0.2569, Forward time: 1.2612, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.7145 [2021-07-08 07:48:20,947][root][INFO] - ===> Epoch[5310](26550/5): Loss 0.0139, Sem 0.0139 LR: 5.911e-02 Score 99.500 Data time: 0.2438, Forward time: 1.2613, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 3.0679 [2021-07-08 07:48:35,161][root][INFO] - ===> Epoch[5311](26555/5): Loss 0.0148, Sem 0.0148 LR: 5.910e-02 Score 99.482 Data time: 0.2020, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.8076 [2021-07-08 07:48:48,726][root][INFO] - ===> Epoch[5312](26560/5): Loss 0.0151, Sem 0.0151 LR: 5.909e-02 Score 99.465 Data time: 0.2502, Forward time: 1.2613, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.6616 [2021-07-08 07:49:02,296][root][INFO] - ===> Epoch[5313](26565/5): Loss 0.0148, Sem 0.0148 LR: 5.908e-02 Score 99.473 Data time: 0.2233, Forward time: 1.2613, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.6613 [2021-07-08 07:49:16,388][root][INFO] - ===> Epoch[5314](26570/5): Loss 0.0187, Sem 0.0187 LR: 5.907e-02 Score 99.367 Data time: 0.1851, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.7836 [2021-07-08 07:49:29,974][root][INFO] - ===> Epoch[5315](26575/5): Loss 0.0165, Sem 0.0165 LR: 5.907e-02 Score 99.408 Data time: 0.2384, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.6713 [2021-07-08 07:49:43,655][root][INFO] - ===> Epoch[5316](26580/5): Loss 0.0163, Sem 0.0163 LR: 5.906e-02 Score 99.423 Data time: 0.2260, Forward time: 1.2612, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.6922 [2021-07-08 07:49:58,281][root][INFO] - ===> Epoch[5317](26585/5): Loss 0.0158, Sem 0.0158 LR: 5.905e-02 Score 99.434 Data time: 0.2131, Forward time: 1.2613, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.8802 [2021-07-08 07:50:12,457][root][INFO] - ===> Epoch[5318](26590/5): Loss 0.0150, Sem 0.0150 LR: 5.904e-02 Score 99.464 Data time: 0.2021, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.7901 [2021-07-08 07:50:27,409][root][INFO] - ===> Epoch[5319](26595/5): Loss 0.0144, Sem 0.0144 LR: 5.903e-02 Score 99.476 Data time: 0.2147, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.9357 [2021-07-08 07:50:41,316][root][INFO] - ===> Epoch[5320](26600/5): Loss 0.0143, Sem 0.0143 LR: 5.903e-02 Score 99.483 Data time: 0.2711, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.7271 [2021-07-08 07:50:41,318][root][INFO] - ===> Start testing [2021-07-08 07:51:21,864][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1238 Loss 0.479 (AVG: 0.532) Score 90.349 (AVG: 90.741) mIOU 70.028 mAP 83.676 mAcc 76.856 IOU: 63.613 0.000 78.561 75.274 94.316 91.010 29.716 82.213 98.321 73.142 76.858 86.512 60.834 mAP: 85.057 3.508 93.641 87.107 98.821 98.088 61.383 97.487 99.803 82.242 92.413 94.916 93.328 mAcc: 80.049 0.000 85.267 82.788 95.406 96.442 33.007 95.798 99.350 78.749 91.425 95.876 64.977 [2021-07-08 07:51:21,864][root][INFO] - Finished test. Elapsed time: 40.5455 [2021-07-08 07:51:22,904][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 07:51:36,301][root][INFO] - ===> Epoch[5321](26605/5): Loss 0.0139, Sem 0.0139 LR: 5.902e-02 Score 99.501 Data time: 0.2175, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.6371 [2021-07-08 07:51:50,440][root][INFO] - ===> Epoch[5322](26610/5): Loss 0.0135, Sem 0.0135 LR: 5.901e-02 Score 99.509 Data time: 0.2983, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.7753 [2021-07-08 07:52:03,863][root][INFO] - ===> Epoch[5323](26615/5): Loss 0.0136, Sem 0.0136 LR: 5.900e-02 Score 99.514 Data time: 0.2164, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.6377 [2021-07-08 07:52:17,455][root][INFO] - ===> Epoch[5324](26620/5): Loss 0.0141, Sem 0.0141 LR: 5.899e-02 Score 99.497 Data time: 0.2093, Forward time: 1.2612, Backward time: 1.2371, DDP time: 0.0042, Total iter time: 2.6687 [2021-07-08 07:52:32,531][root][INFO] - ===> Epoch[5325](26625/5): Loss 0.0139, Sem 0.0139 LR: 5.899e-02 Score 99.510 Data time: 0.2263, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.9575 [2021-07-08 07:52:45,406][root][INFO] - ===> Epoch[5326](26630/5): Loss 0.0139, Sem 0.0139 LR: 5.898e-02 Score 99.496 Data time: 0.2355, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.5307 [2021-07-08 07:53:00,534][root][INFO] - ===> Epoch[5327](26635/5): Loss 0.0138, Sem 0.0138 LR: 5.897e-02 Score 99.502 Data time: 0.2288, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.9842 [2021-07-08 07:53:14,643][root][INFO] - ===> Epoch[5328](26640/5): Loss 0.0140, Sem 0.0140 LR: 5.896e-02 Score 99.495 Data time: 0.2369, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.7760 [2021-07-08 07:53:28,645][root][INFO] - ===> Epoch[5329](26645/5): Loss 0.0147, Sem 0.0147 LR: 5.895e-02 Score 99.463 Data time: 0.2134, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.7545 [2021-07-08 07:53:42,594][root][INFO] - ===> Epoch[5330](26650/5): Loss 0.0154, Sem 0.0154 LR: 5.895e-02 Score 99.468 Data time: 0.2722, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.7501 [2021-07-08 07:53:56,879][root][INFO] - ===> Epoch[5331](26655/5): Loss 0.0152, Sem 0.0152 LR: 5.894e-02 Score 99.459 Data time: 0.2157, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.8135 [2021-07-08 07:54:10,175][root][INFO] - ===> Epoch[5332](26660/5): Loss 0.0138, Sem 0.0138 LR: 5.893e-02 Score 99.499 Data time: 0.2137, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.6160 [2021-07-08 07:54:24,029][root][INFO] - ===> Epoch[5333](26665/5): Loss 0.0140, Sem 0.0140 LR: 5.892e-02 Score 99.495 Data time: 0.2275, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.7247 [2021-07-08 07:54:37,831][root][INFO] - ===> Epoch[5334](26670/5): Loss 0.0142, Sem 0.0142 LR: 5.891e-02 Score 99.498 Data time: 0.2110, Forward time: 1.2612, Backward time: 1.2372, DDP time: 0.0042, Total iter time: 2.7178 [2021-07-08 07:54:52,879][root][INFO] - ===> Epoch[5335](26675/5): Loss 0.0147, Sem 0.0147 LR: 5.891e-02 Score 99.480 Data time: 0.1815, Forward time: 1.2611, Backward time: 1.2373, DDP time: 0.0042, Total iter time: 2.9677 [2021-07-08 07:55:07,043][root][INFO] - ===> Epoch[5336](26680/5): Loss 0.0148, Sem 0.0148 LR: 5.890e-02 Score 99.472 Data time: 0.2455, Forward time: 1.2611, Backward time: 1.2373, DDP time: 0.0042, Total iter time: 2.7855 [2021-07-08 07:55:20,565][root][INFO] - ===> Epoch[5337](26685/5): Loss 0.0134, Sem 0.0134 LR: 5.889e-02 Score 99.518 Data time: 0.2150, Forward time: 1.2611, Backward time: 1.2373, DDP time: 0.0042, Total iter time: 2.6566 [2021-07-08 07:55:34,592][root][INFO] - ===> Epoch[5338](26690/5): Loss 0.0135, Sem 0.0135 LR: 5.888e-02 Score 99.511 Data time: 0.1742, Forward time: 1.2611, Backward time: 1.2373, DDP time: 0.0042, Total iter time: 2.7609 [2021-07-08 07:55:47,725][root][INFO] - ===> Epoch[5339](26695/5): Loss 0.0134, Sem 0.0134 LR: 5.887e-02 Score 99.522 Data time: 0.2273, Forward time: 1.2611, Backward time: 1.2373, DDP time: 0.0042, Total iter time: 2.5723 [2021-07-08 07:56:02,636][root][INFO] - ===> Epoch[5340](26700/5): Loss 0.0136, Sem 0.0136 LR: 5.887e-02 Score 99.507 Data time: 0.1772, Forward time: 1.2611, Backward time: 1.2373, DDP time: 0.0042, Total iter time: 2.9372 [2021-07-08 07:56:17,342][root][INFO] - ===> Epoch[5341](26705/5): Loss 0.0141, Sem 0.0141 LR: 5.886e-02 Score 99.489 Data time: 0.2386, Forward time: 1.2611, Backward time: 1.2373, DDP time: 0.0042, Total iter time: 2.8923 [2021-07-08 07:56:31,625][root][INFO] - ===> Epoch[5342](26710/5): Loss 0.0139, Sem 0.0139 LR: 5.885e-02 Score 99.504 Data time: 0.2250, Forward time: 1.2611, Backward time: 1.2374, DDP time: 0.0042, Total iter time: 2.8086 [2021-07-08 07:56:45,453][root][INFO] - ===> Epoch[5343](26715/5): Loss 0.0145, Sem 0.0145 LR: 5.884e-02 Score 99.476 Data time: 0.2070, Forward time: 1.2611, Backward time: 1.2374, DDP time: 0.0042, Total iter time: 2.7226 [2021-07-08 07:56:58,730][root][INFO] - ===> Epoch[5344](26720/5): Loss 0.0133, Sem 0.0133 LR: 5.883e-02 Score 99.520 Data time: 0.1802, Forward time: 1.2611, Backward time: 1.2374, DDP time: 0.0042, Total iter time: 2.6266 [2021-07-08 07:57:11,943][root][INFO] - ===> Epoch[5345](26725/5): Loss 0.0138, Sem 0.0138 LR: 5.883e-02 Score 99.508 Data time: 0.2477, Forward time: 1.2610, Backward time: 1.2373, DDP time: 0.0042, Total iter time: 2.6054 [2021-07-08 07:57:25,177][root][INFO] - ===> Epoch[5346](26730/5): Loss 0.0155, Sem 0.0155 LR: 5.882e-02 Score 99.460 Data time: 0.1846, Forward time: 1.2610, Backward time: 1.2373, DDP time: 0.0042, Total iter time: 2.6072 [2021-07-08 07:57:38,549][root][INFO] - ===> Epoch[5347](26735/5): Loss 0.0138, Sem 0.0138 LR: 5.881e-02 Score 99.500 Data time: 0.2546, Forward time: 1.2610, Backward time: 1.2373, DDP time: 0.0042, Total iter time: 2.6266 [2021-07-08 07:57:52,878][root][INFO] - ===> Epoch[5348](26740/5): Loss 0.0142, Sem 0.0142 LR: 5.880e-02 Score 99.482 Data time: 0.2192, Forward time: 1.2610, Backward time: 1.2374, DDP time: 0.0042, Total iter time: 2.8205 [2021-07-08 07:58:07,801][root][INFO] - ===> Epoch[5349](26745/5): Loss 0.0135, Sem 0.0135 LR: 5.879e-02 Score 99.515 Data time: 0.2229, Forward time: 1.2609, Backward time: 1.2374, DDP time: 0.0042, Total iter time: 2.9390 [2021-07-08 07:58:21,896][root][INFO] - ===> Epoch[5350](26750/5): Loss 0.0137, Sem 0.0137 LR: 5.879e-02 Score 99.505 Data time: 0.2140, Forward time: 1.2609, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.7699 [2021-07-08 07:58:35,565][root][INFO] - ===> Epoch[5351](26755/5): Loss 0.0139, Sem 0.0139 LR: 5.878e-02 Score 99.501 Data time: 0.2080, Forward time: 1.2609, Backward time: 1.2374, DDP time: 0.0042, Total iter time: 2.6887 [2021-07-08 07:58:48,832][root][INFO] - ===> Epoch[5352](26760/5): Loss 0.0136, Sem 0.0136 LR: 5.877e-02 Score 99.512 Data time: 0.2267, Forward time: 1.2609, Backward time: 1.2374, DDP time: 0.0042, Total iter time: 2.5902 [2021-07-08 07:59:02,586][root][INFO] - ===> Epoch[5353](26765/5): Loss 0.0136, Sem 0.0136 LR: 5.876e-02 Score 99.511 Data time: 0.2114, Forward time: 1.2609, Backward time: 1.2374, DDP time: 0.0042, Total iter time: 2.7077 [2021-07-08 07:59:16,749][root][INFO] - ===> Epoch[5354](26770/5): Loss 0.0139, Sem 0.0139 LR: 5.876e-02 Score 99.500 Data time: 0.2748, Forward time: 1.2609, Backward time: 1.2374, DDP time: 0.0042, Total iter time: 2.7873 [2021-07-08 07:59:30,915][root][INFO] - ===> Epoch[5355](26775/5): Loss 0.0135, Sem 0.0135 LR: 5.875e-02 Score 99.511 Data time: 0.2019, Forward time: 1.2609, Backward time: 1.2374, DDP time: 0.0042, Total iter time: 2.7920 [2021-07-08 07:59:44,849][root][INFO] - ===> Epoch[5356](26780/5): Loss 0.0133, Sem 0.0133 LR: 5.874e-02 Score 99.522 Data time: 0.2434, Forward time: 1.2609, Backward time: 1.2374, DDP time: 0.0042, Total iter time: 2.7337 [2021-07-08 07:59:59,400][root][INFO] - ===> Epoch[5357](26785/5): Loss 0.0127, Sem 0.0127 LR: 5.873e-02 Score 99.537 Data time: 0.1814, Forward time: 1.2609, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.8692 [2021-07-08 08:00:13,468][root][INFO] - ===> Epoch[5358](26790/5): Loss 0.0133, Sem 0.0133 LR: 5.872e-02 Score 99.520 Data time: 0.2312, Forward time: 1.2609, Backward time: 1.2374, DDP time: 0.0042, Total iter time: 2.7665 [2021-07-08 08:00:27,679][root][INFO] - ===> Epoch[5359](26795/5): Loss 0.0130, Sem 0.0130 LR: 5.872e-02 Score 99.536 Data time: 0.2754, Forward time: 1.2609, Backward time: 1.2374, DDP time: 0.0042, Total iter time: 2.7928 [2021-07-08 08:00:42,346][root][INFO] - ===> Epoch[5360](26800/5): Loss 0.0121, Sem 0.0121 LR: 5.871e-02 Score 99.564 Data time: 0.1651, Forward time: 1.2609, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.8831 [2021-07-08 08:00:42,348][root][INFO] - ===> Start testing [2021-07-08 08:01:29,762][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1299 Loss 0.463 (AVG: 0.555) Score 90.738 (AVG: 91.007) mIOU 70.022 mAP 83.561 mAcc 76.328 IOU: 65.021 0.000 84.630 76.112 94.853 91.875 21.454 81.915 98.508 71.779 78.203 86.575 59.357 mAP: 85.132 5.095 93.649 87.627 98.915 98.626 56.260 98.340 99.765 83.858 92.300 94.431 92.300 mAcc: 81.376 0.000 89.556 82.793 96.268 97.272 22.693 96.600 99.409 75.681 90.339 96.373 63.901 [2021-07-08 08:01:29,762][root][INFO] - Finished test. Elapsed time: 47.4140 [2021-07-08 08:01:30,827][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 08:01:43,596][root][INFO] - ===> Epoch[5361](26805/5): Loss 0.0133, Sem 0.0133 LR: 5.870e-02 Score 99.526 Data time: 0.2121, Forward time: 1.2609, Backward time: 1.2374, DDP time: 0.0042, Total iter time: 2.5045 [2021-07-08 08:01:58,297][root][INFO] - ===> Epoch[5362](26810/5): Loss 0.0132, Sem 0.0132 LR: 5.869e-02 Score 99.525 Data time: 0.1900, Forward time: 1.2609, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.9018 [2021-07-08 08:02:12,488][root][INFO] - ===> Epoch[5363](26815/5): Loss 0.0137, Sem 0.0137 LR: 5.868e-02 Score 99.510 Data time: 0.2364, Forward time: 1.2609, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.7720 [2021-07-08 08:02:25,585][root][INFO] - ===> Epoch[5364](26820/5): Loss 0.0136, Sem 0.0136 LR: 5.868e-02 Score 99.513 Data time: 0.2000, Forward time: 1.2609, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.5762 [2021-07-08 08:02:40,107][root][INFO] - ===> Epoch[5365](26825/5): Loss 0.0132, Sem 0.0132 LR: 5.867e-02 Score 99.530 Data time: 0.2377, Forward time: 1.2609, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.8569 [2021-07-08 08:02:54,446][root][INFO] - ===> Epoch[5366](26830/5): Loss 0.0138, Sem 0.0138 LR: 5.866e-02 Score 99.511 Data time: 0.2245, Forward time: 1.2609, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.8115 [2021-07-08 08:03:09,043][root][INFO] - ===> Epoch[5367](26835/5): Loss 0.0133, Sem 0.0133 LR: 5.865e-02 Score 99.523 Data time: 0.2300, Forward time: 1.2609, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.8688 [2021-07-08 08:03:22,327][root][INFO] - ===> Epoch[5368](26840/5): Loss 0.0136, Sem 0.0136 LR: 5.864e-02 Score 99.512 Data time: 0.2280, Forward time: 1.2609, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.6144 [2021-07-08 08:03:36,589][root][INFO] - ===> Epoch[5369](26845/5): Loss 0.0130, Sem 0.0130 LR: 5.864e-02 Score 99.538 Data time: 0.2383, Forward time: 1.2609, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.8057 [2021-07-08 08:03:49,751][root][INFO] - ===> Epoch[5370](26850/5): Loss 0.0125, Sem 0.0125 LR: 5.863e-02 Score 99.551 Data time: 0.2227, Forward time: 1.2609, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.5912 [2021-07-08 08:04:03,889][root][INFO] - ===> Epoch[5371](26855/5): Loss 0.0126, Sem 0.0126 LR: 5.862e-02 Score 99.546 Data time: 0.1863, Forward time: 1.2609, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.7829 [2021-07-08 08:04:17,592][root][INFO] - ===> Epoch[5372](26860/5): Loss 0.0128, Sem 0.0128 LR: 5.861e-02 Score 99.541 Data time: 0.1918, Forward time: 1.2609, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.6996 [2021-07-08 08:04:30,715][root][INFO] - ===> Epoch[5373](26865/5): Loss 0.0130, Sem 0.0130 LR: 5.860e-02 Score 99.530 Data time: 0.1837, Forward time: 1.2608, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.5825 [2021-07-08 08:04:44,425][root][INFO] - ===> Epoch[5374](26870/5): Loss 0.0129, Sem 0.0129 LR: 5.860e-02 Score 99.533 Data time: 0.1909, Forward time: 1.2608, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.7043 [2021-07-08 08:04:58,141][root][INFO] - ===> Epoch[5375](26875/5): Loss 0.0122, Sem 0.0122 LR: 5.859e-02 Score 99.561 Data time: 0.2597, Forward time: 1.2608, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.6979 [2021-07-08 08:05:12,523][root][INFO] - ===> Epoch[5376](26880/5): Loss 0.0121, Sem 0.0121 LR: 5.858e-02 Score 99.560 Data time: 0.2122, Forward time: 1.2608, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.8268 [2021-07-08 08:05:26,277][root][INFO] - ===> Epoch[5377](26885/5): Loss 0.0128, Sem 0.0128 LR: 5.857e-02 Score 99.536 Data time: 0.1992, Forward time: 1.2608, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.7078 [2021-07-08 08:05:40,385][root][INFO] - ===> Epoch[5378](26890/5): Loss 0.0131, Sem 0.0131 LR: 5.856e-02 Score 99.521 Data time: 0.2211, Forward time: 1.2608, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.7722 [2021-07-08 08:05:55,512][root][INFO] - ===> Epoch[5379](26895/5): Loss 0.0117, Sem 0.0117 LR: 5.856e-02 Score 99.579 Data time: 0.2294, Forward time: 1.2608, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.9678 [2021-07-08 08:06:09,322][root][INFO] - ===> Epoch[5380](26900/5): Loss 0.0122, Sem 0.0122 LR: 5.855e-02 Score 99.560 Data time: 0.2172, Forward time: 1.2608, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7085 [2021-07-08 08:06:23,447][root][INFO] - ===> Epoch[5381](26905/5): Loss 0.0126, Sem 0.0126 LR: 5.854e-02 Score 99.545 Data time: 0.1990, Forward time: 1.2608, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7780 [2021-07-08 08:06:37,102][root][INFO] - ===> Epoch[5382](26910/5): Loss 0.0133, Sem 0.0133 LR: 5.853e-02 Score 99.527 Data time: 0.1959, Forward time: 1.2607, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.6869 [2021-07-08 08:06:51,197][root][INFO] - ===> Epoch[5383](26915/5): Loss 0.0124, Sem 0.0124 LR: 5.852e-02 Score 99.553 Data time: 0.1815, Forward time: 1.2607, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7717 [2021-07-08 08:07:05,039][root][INFO] - ===> Epoch[5384](26920/5): Loss 0.0144, Sem 0.0144 LR: 5.852e-02 Score 99.500 Data time: 0.2445, Forward time: 1.2608, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7185 [2021-07-08 08:07:18,798][root][INFO] - ===> Epoch[5385](26925/5): Loss 0.0147, Sem 0.0147 LR: 5.851e-02 Score 99.473 Data time: 0.2465, Forward time: 1.2607, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.6958 [2021-07-08 08:07:32,178][root][INFO] - ===> Epoch[5386](26930/5): Loss 0.0153, Sem 0.0153 LR: 5.850e-02 Score 99.456 Data time: 0.2821, Forward time: 1.2607, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.6370 [2021-07-08 08:07:46,566][root][INFO] - ===> Epoch[5387](26935/5): Loss 0.0138, Sem 0.0138 LR: 5.849e-02 Score 99.502 Data time: 0.2469, Forward time: 1.2607, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.8327 [2021-07-08 08:08:00,735][root][INFO] - ===> Epoch[5388](26940/5): Loss 0.0129, Sem 0.0129 LR: 5.848e-02 Score 99.530 Data time: 0.2901, Forward time: 1.2607, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7824 [2021-07-08 08:08:14,738][root][INFO] - ===> Epoch[5389](26945/5): Loss 0.0135, Sem 0.0135 LR: 5.848e-02 Score 99.513 Data time: 0.2629, Forward time: 1.2607, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7496 [2021-07-08 08:08:29,049][root][INFO] - ===> Epoch[5390](26950/5): Loss 0.0130, Sem 0.0130 LR: 5.847e-02 Score 99.529 Data time: 0.1888, Forward time: 1.2607, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.8132 [2021-07-08 08:08:44,257][root][INFO] - ===> Epoch[5391](26955/5): Loss 0.0131, Sem 0.0131 LR: 5.846e-02 Score 99.521 Data time: 0.2482, Forward time: 1.2607, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.9980 [2021-07-08 08:08:57,643][root][INFO] - ===> Epoch[5392](26960/5): Loss 0.0129, Sem 0.0129 LR: 5.845e-02 Score 99.535 Data time: 0.2273, Forward time: 1.2607, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.6273 [2021-07-08 08:09:11,680][root][INFO] - ===> Epoch[5393](26965/5): Loss 0.0137, Sem 0.0137 LR: 5.844e-02 Score 99.504 Data time: 0.2510, Forward time: 1.2607, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7624 [2021-07-08 08:09:26,265][root][INFO] - ===> Epoch[5394](26970/5): Loss 0.0186, Sem 0.0186 LR: 5.844e-02 Score 99.436 Data time: 0.1972, Forward time: 1.2607, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8674 [2021-07-08 08:09:40,648][root][INFO] - ===> Epoch[5395](26975/5): Loss 0.0152, Sem 0.0152 LR: 5.843e-02 Score 99.467 Data time: 0.2909, Forward time: 1.2607, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.8384 [2021-07-08 08:09:54,448][root][INFO] - ===> Epoch[5396](26980/5): Loss 0.0149, Sem 0.0149 LR: 5.842e-02 Score 99.480 Data time: 0.2459, Forward time: 1.2607, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7164 [2021-07-08 08:10:09,160][root][INFO] - ===> Epoch[5397](26985/5): Loss 0.0147, Sem 0.0147 LR: 5.841e-02 Score 99.469 Data time: 0.2517, Forward time: 1.2607, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8943 [2021-07-08 08:10:22,919][root][INFO] - ===> Epoch[5398](26990/5): Loss 0.0140, Sem 0.0140 LR: 5.841e-02 Score 99.494 Data time: 0.2092, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7115 [2021-07-08 08:10:36,458][root][INFO] - ===> Epoch[5399](26995/5): Loss 0.0144, Sem 0.0144 LR: 5.840e-02 Score 99.482 Data time: 0.2098, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6631 [2021-07-08 08:10:50,808][root][INFO] - ===> Epoch[5400](27000/5): Loss 0.0134, Sem 0.0134 LR: 5.839e-02 Score 99.521 Data time: 0.2455, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8216 [2021-07-08 08:10:50,810][root][INFO] - ===> Start testing [2021-07-08 08:11:36,914][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1217 Loss 0.308 (AVG: 0.558) Score 92.955 (AVG: 90.735) mIOU 69.975 mAP 82.999 mAcc 76.183 IOU: 64.639 0.000 84.588 76.271 94.357 91.393 22.945 81.720 98.242 74.458 77.522 85.955 57.583 mAP: 84.777 5.347 90.781 87.006 98.875 98.768 53.931 97.995 99.772 84.809 92.464 94.048 90.418 mAcc: 81.142 0.000 88.330 83.481 95.681 97.378 26.000 95.536 99.321 77.220 90.401 96.096 59.794 [2021-07-08 08:11:36,914][root][INFO] - Finished test. Elapsed time: 46.1039 [2021-07-08 08:11:37,966][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 08:11:50,587][root][INFO] - ===> Epoch[5401](27005/5): Loss 0.0138, Sem 0.0138 LR: 5.838e-02 Score 99.502 Data time: 0.2038, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.4895 [2021-07-08 08:12:04,378][root][INFO] - ===> Epoch[5402](27010/5): Loss 0.0126, Sem 0.0126 LR: 5.837e-02 Score 99.547 Data time: 0.1709, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7205 [2021-07-08 08:12:18,434][root][INFO] - ===> Epoch[5403](27015/5): Loss 0.0128, Sem 0.0128 LR: 5.837e-02 Score 99.540 Data time: 0.2045, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7657 [2021-07-08 08:12:31,614][root][INFO] - ===> Epoch[5404](27020/5): Loss 0.0127, Sem 0.0127 LR: 5.836e-02 Score 99.543 Data time: 0.2326, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.5826 [2021-07-08 08:12:45,648][root][INFO] - ===> Epoch[5405](27025/5): Loss 0.0131, Sem 0.0131 LR: 5.835e-02 Score 99.533 Data time: 0.2775, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7598 [2021-07-08 08:12:59,723][root][INFO] - ===> Epoch[5406](27030/5): Loss 0.0126, Sem 0.0126 LR: 5.834e-02 Score 99.543 Data time: 0.2365, Forward time: 1.2606, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7714 [2021-07-08 08:13:13,894][root][INFO] - ===> Epoch[5407](27035/5): Loss 0.0124, Sem 0.0124 LR: 5.833e-02 Score 99.550 Data time: 0.2320, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7938 [2021-07-08 08:13:28,468][root][INFO] - ===> Epoch[5408](27040/5): Loss 0.0123, Sem 0.0123 LR: 5.833e-02 Score 99.554 Data time: 0.2134, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8742 [2021-07-08 08:13:42,714][root][INFO] - ===> Epoch[5409](27045/5): Loss 0.0117, Sem 0.0117 LR: 5.832e-02 Score 99.581 Data time: 0.1885, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8081 [2021-07-08 08:13:57,868][root][INFO] - ===> Epoch[5410](27050/5): Loss 0.0120, Sem 0.0120 LR: 5.831e-02 Score 99.566 Data time: 0.2583, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.9890 [2021-07-08 08:14:10,467][root][INFO] - ===> Epoch[5411](27055/5): Loss 0.0125, Sem 0.0125 LR: 5.830e-02 Score 99.545 Data time: 0.1933, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.4648 [2021-07-08 08:14:25,015][root][INFO] - ===> Epoch[5412](27060/5): Loss 0.0124, Sem 0.0124 LR: 5.829e-02 Score 99.550 Data time: 0.2795, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8535 [2021-07-08 08:14:40,039][root][INFO] - ===> Epoch[5413](27065/5): Loss 0.0129, Sem 0.0129 LR: 5.829e-02 Score 99.559 Data time: 0.2639, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.9519 [2021-07-08 08:14:53,811][root][INFO] - ===> Epoch[5414](27070/5): Loss 0.0131, Sem 0.0131 LR: 5.828e-02 Score 99.531 Data time: 0.2361, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7134 [2021-07-08 08:15:07,707][root][INFO] - ===> Epoch[5415](27075/5): Loss 0.0128, Sem 0.0128 LR: 5.827e-02 Score 99.539 Data time: 0.2258, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7336 [2021-07-08 08:15:21,863][root][INFO] - ===> Epoch[5416](27080/5): Loss 0.0127, Sem 0.0127 LR: 5.826e-02 Score 99.542 Data time: 0.2354, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7757 [2021-07-08 08:15:35,903][root][INFO] - ===> Epoch[5417](27085/5): Loss 0.0123, Sem 0.0123 LR: 5.825e-02 Score 99.557 Data time: 0.2435, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7648 [2021-07-08 08:15:50,018][root][INFO] - ===> Epoch[5418](27090/5): Loss 0.0126, Sem 0.0126 LR: 5.825e-02 Score 99.547 Data time: 0.2135, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7798 [2021-07-08 08:16:03,809][root][INFO] - ===> Epoch[5419](27095/5): Loss 0.0128, Sem 0.0128 LR: 5.824e-02 Score 99.541 Data time: 0.2417, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7150 [2021-07-08 08:16:18,468][root][INFO] - ===> Epoch[5420](27100/5): Loss 0.0121, Sem 0.0121 LR: 5.823e-02 Score 99.564 Data time: 0.1885, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8857 [2021-07-08 08:16:31,750][root][INFO] - ===> Epoch[5421](27105/5): Loss 0.0129, Sem 0.0129 LR: 5.822e-02 Score 99.531 Data time: 0.2445, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6161 [2021-07-08 08:16:45,884][root][INFO] - ===> Epoch[5422](27110/5): Loss 0.0119, Sem 0.0119 LR: 5.821e-02 Score 99.571 Data time: 0.2189, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7849 [2021-07-08 08:16:59,671][root][INFO] - ===> Epoch[5423](27115/5): Loss 0.0124, Sem 0.0124 LR: 5.821e-02 Score 99.553 Data time: 0.2252, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7213 [2021-07-08 08:17:13,945][root][INFO] - ===> Epoch[5424](27120/5): Loss 0.0122, Sem 0.0122 LR: 5.820e-02 Score 99.556 Data time: 0.2157, Forward time: 1.2606, Backward time: 1.2378, DDP time: 0.0042, Total iter time: 2.8133 [2021-07-08 08:17:26,966][root][INFO] - ===> Epoch[5425](27125/5): Loss 0.0118, Sem 0.0118 LR: 5.819e-02 Score 99.575 Data time: 0.2112, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.5520 [2021-07-08 08:17:40,823][root][INFO] - ===> Epoch[5426](27130/5): Loss 0.0116, Sem 0.0116 LR: 5.818e-02 Score 99.582 Data time: 0.2212, Forward time: 1.2606, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7260 [2021-07-08 08:17:54,870][root][INFO] - ===> Epoch[5427](27135/5): Loss 0.0119, Sem 0.0119 LR: 5.817e-02 Score 99.569 Data time: 0.2522, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7547 [2021-07-08 08:18:07,810][root][INFO] - ===> Epoch[5428](27140/5): Loss 0.0121, Sem 0.0121 LR: 5.817e-02 Score 99.564 Data time: 0.2588, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.5378 [2021-07-08 08:18:21,163][root][INFO] - ===> Epoch[5429](27145/5): Loss 0.0120, Sem 0.0120 LR: 5.816e-02 Score 99.565 Data time: 0.2342, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6129 [2021-07-08 08:18:34,686][root][INFO] - ===> Epoch[5430](27150/5): Loss 0.0118, Sem 0.0118 LR: 5.815e-02 Score 99.574 Data time: 0.1831, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6664 [2021-07-08 08:18:49,243][root][INFO] - ===> Epoch[5431](27155/5): Loss 0.0115, Sem 0.0115 LR: 5.814e-02 Score 99.584 Data time: 0.1897, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8689 [2021-07-08 08:19:03,054][root][INFO] - ===> Epoch[5432](27160/5): Loss 0.0124, Sem 0.0124 LR: 5.813e-02 Score 99.552 Data time: 0.1764, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7272 [2021-07-08 08:19:17,562][root][INFO] - ===> Epoch[5433](27165/5): Loss 0.0123, Sem 0.0123 LR: 5.813e-02 Score 99.556 Data time: 0.2326, Forward time: 1.2605, Backward time: 1.2378, DDP time: 0.0042, Total iter time: 2.8466 [2021-07-08 08:19:31,575][root][INFO] - ===> Epoch[5434](27170/5): Loss 0.0115, Sem 0.0115 LR: 5.812e-02 Score 99.582 Data time: 0.2722, Forward time: 1.2605, Backward time: 1.2378, DDP time: 0.0042, Total iter time: 2.7504 [2021-07-08 08:19:44,830][root][INFO] - ===> Epoch[5435](27175/5): Loss 0.0118, Sem 0.0118 LR: 5.811e-02 Score 99.575 Data time: 0.2051, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6028 [2021-07-08 08:19:58,557][root][INFO] - ===> Epoch[5436](27180/5): Loss 0.0123, Sem 0.0123 LR: 5.810e-02 Score 99.560 Data time: 0.2438, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6913 [2021-07-08 08:20:12,451][root][INFO] - ===> Epoch[5437](27185/5): Loss 0.0118, Sem 0.0118 LR: 5.809e-02 Score 99.575 Data time: 0.2691, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7118 [2021-07-08 08:20:26,278][root][INFO] - ===> Epoch[5438](27190/5): Loss 0.0116, Sem 0.0116 LR: 5.809e-02 Score 99.579 Data time: 0.2620, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7238 [2021-07-08 08:20:39,534][root][INFO] - ===> Epoch[5439](27195/5): Loss 0.0114, Sem 0.0114 LR: 5.808e-02 Score 99.588 Data time: 0.2146, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6072 [2021-07-08 08:20:54,342][root][INFO] - ===> Epoch[5440](27200/5): Loss 0.0115, Sem 0.0115 LR: 5.807e-02 Score 99.583 Data time: 0.2212, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.9158 [2021-07-08 08:20:54,343][root][INFO] - ===> Start testing [2021-07-08 08:21:37,797][root][INFO] - 68/68: Data time: 0.0242, Iter time: 0.1455 Loss 0.471 (AVG: 0.557) Score 90.976 (AVG: 91.087) mIOU 70.779 mAP 84.273 mAcc 76.834 IOU: 64.832 0.000 84.386 76.700 94.715 92.255 25.463 83.814 98.437 74.413 78.457 86.668 59.991 mAP: 84.673 10.539 94.280 87.306 98.877 98.753 56.584 98.674 99.783 88.060 91.304 94.285 92.429 mAcc: 81.939 0.000 89.329 83.784 95.908 97.578 27.075 97.187 99.358 79.134 88.027 96.351 63.172 [2021-07-08 08:21:37,798][root][INFO] - Finished test. Elapsed time: 43.4536 [2021-07-08 08:21:39,010][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 08:21:52,233][root][INFO] - ===> Epoch[5441](27205/5): Loss 0.0116, Sem 0.0116 LR: 5.806e-02 Score 99.580 Data time: 0.1842, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6006 [2021-07-08 08:22:05,763][root][INFO] - ===> Epoch[5442](27210/5): Loss 0.0124, Sem 0.0124 LR: 5.805e-02 Score 99.558 Data time: 0.2489, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6677 [2021-07-08 08:22:20,302][root][INFO] - ===> Epoch[5443](27215/5): Loss 0.0130, Sem 0.0130 LR: 5.805e-02 Score 99.534 Data time: 0.2649, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8641 [2021-07-08 08:22:34,413][root][INFO] - ===> Epoch[5444](27220/5): Loss 0.0136, Sem 0.0136 LR: 5.804e-02 Score 99.522 Data time: 0.2766, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7628 [2021-07-08 08:22:48,373][root][INFO] - ===> Epoch[5445](27225/5): Loss 0.0128, Sem 0.0128 LR: 5.803e-02 Score 99.540 Data time: 0.2952, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7440 [2021-07-08 08:23:03,156][root][INFO] - ===> Epoch[5446](27230/5): Loss 0.0120, Sem 0.0120 LR: 5.802e-02 Score 99.569 Data time: 0.2863, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.9061 [2021-07-08 08:23:17,201][root][INFO] - ===> Epoch[5447](27235/5): Loss 0.0126, Sem 0.0126 LR: 5.801e-02 Score 99.547 Data time: 0.2718, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7582 [2021-07-08 08:23:30,925][root][INFO] - ===> Epoch[5448](27240/5): Loss 0.0124, Sem 0.0124 LR: 5.801e-02 Score 99.558 Data time: 0.2580, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7008 [2021-07-08 08:23:45,407][root][INFO] - ===> Epoch[5449](27245/5): Loss 0.0122, Sem 0.0122 LR: 5.800e-02 Score 99.561 Data time: 0.1705, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.8616 [2021-07-08 08:23:59,179][root][INFO] - ===> Epoch[5450](27250/5): Loss 0.0122, Sem 0.0122 LR: 5.799e-02 Score 99.559 Data time: 0.2695, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7083 [2021-07-08 08:24:13,792][root][INFO] - ===> Epoch[5451](27255/5): Loss 0.0120, Sem 0.0120 LR: 5.798e-02 Score 99.566 Data time: 0.2042, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8814 [2021-07-08 08:24:27,533][root][INFO] - ===> Epoch[5452](27260/5): Loss 0.0126, Sem 0.0126 LR: 5.797e-02 Score 99.546 Data time: 0.2016, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6937 [2021-07-08 08:24:40,904][root][INFO] - ===> Epoch[5453](27265/5): Loss 0.0118, Sem 0.0118 LR: 5.797e-02 Score 99.576 Data time: 0.2037, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6276 [2021-07-08 08:24:54,955][root][INFO] - ===> Epoch[5454](27270/5): Loss 0.0128, Sem 0.0128 LR: 5.796e-02 Score 99.539 Data time: 0.2432, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7641 [2021-07-08 08:25:08,833][root][INFO] - ===> Epoch[5455](27275/5): Loss 0.0121, Sem 0.0121 LR: 5.795e-02 Score 99.560 Data time: 0.1696, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7347 [2021-07-08 08:25:24,081][root][INFO] - ===> Epoch[5456](27280/5): Loss 0.0122, Sem 0.0122 LR: 5.794e-02 Score 99.559 Data time: 0.2131, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 3.0033 [2021-07-08 08:25:37,663][root][INFO] - ===> Epoch[5457](27285/5): Loss 0.0118, Sem 0.0118 LR: 5.794e-02 Score 99.573 Data time: 0.2387, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6728 [2021-07-08 08:25:51,638][root][INFO] - ===> Epoch[5458](27290/5): Loss 0.0115, Sem 0.0115 LR: 5.793e-02 Score 99.581 Data time: 0.2083, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7545 [2021-07-08 08:26:06,100][root][INFO] - ===> Epoch[5459](27295/5): Loss 0.0117, Sem 0.0117 LR: 5.792e-02 Score 99.574 Data time: 0.2674, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8320 [2021-07-08 08:26:19,917][root][INFO] - ===> Epoch[5460](27300/5): Loss 0.0111, Sem 0.0111 LR: 5.791e-02 Score 99.599 Data time: 0.2363, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7146 [2021-07-08 08:26:33,644][root][INFO] - ===> Epoch[5461](27305/5): Loss 0.0118, Sem 0.0118 LR: 5.790e-02 Score 99.577 Data time: 0.2076, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7015 [2021-07-08 08:26:47,444][root][INFO] - ===> Epoch[5462](27310/5): Loss 0.0116, Sem 0.0116 LR: 5.790e-02 Score 99.581 Data time: 0.2205, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7084 [2021-07-08 08:27:01,785][root][INFO] - ===> Epoch[5463](27315/5): Loss 0.0125, Sem 0.0125 LR: 5.789e-02 Score 99.546 Data time: 0.1809, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8191 [2021-07-08 08:27:15,657][root][INFO] - ===> Epoch[5464](27320/5): Loss 0.0126, Sem 0.0126 LR: 5.788e-02 Score 99.544 Data time: 0.2275, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7219 [2021-07-08 08:27:29,333][root][INFO] - ===> Epoch[5465](27325/5): Loss 0.0126, Sem 0.0126 LR: 5.787e-02 Score 99.555 Data time: 0.1868, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6892 [2021-07-08 08:27:43,337][root][INFO] - ===> Epoch[5466](27330/5): Loss 0.0121, Sem 0.0121 LR: 5.786e-02 Score 99.562 Data time: 0.2250, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7512 [2021-07-08 08:27:57,103][root][INFO] - ===> Epoch[5467](27335/5): Loss 0.0119, Sem 0.0119 LR: 5.786e-02 Score 99.574 Data time: 0.2060, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7005 [2021-07-08 08:28:11,742][root][INFO] - ===> Epoch[5468](27340/5): Loss 0.0113, Sem 0.0113 LR: 5.785e-02 Score 99.592 Data time: 0.2476, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8793 [2021-07-08 08:28:26,052][root][INFO] - ===> Epoch[5469](27345/5): Loss 0.0120, Sem 0.0120 LR: 5.784e-02 Score 99.563 Data time: 0.2028, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8198 [2021-07-08 08:28:40,552][root][INFO] - ===> Epoch[5470](27350/5): Loss 0.0123, Sem 0.0123 LR: 5.783e-02 Score 99.554 Data time: 0.2146, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8478 [2021-07-08 08:28:54,487][root][INFO] - ===> Epoch[5471](27355/5): Loss 0.0112, Sem 0.0112 LR: 5.782e-02 Score 99.595 Data time: 0.1901, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7368 [2021-07-08 08:29:08,252][root][INFO] - ===> Epoch[5472](27360/5): Loss 0.0125, Sem 0.0125 LR: 5.782e-02 Score 99.543 Data time: 0.2752, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7111 [2021-07-08 08:29:22,207][root][INFO] - ===> Epoch[5473](27365/5): Loss 0.0125, Sem 0.0125 LR: 5.781e-02 Score 99.543 Data time: 0.2395, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7454 [2021-07-08 08:29:36,315][root][INFO] - ===> Epoch[5474](27370/5): Loss 0.0114, Sem 0.0114 LR: 5.780e-02 Score 99.591 Data time: 0.2414, Forward time: 1.2604, Backward time: 1.2378, DDP time: 0.0042, Total iter time: 2.7726 [2021-07-08 08:29:50,409][root][INFO] - ===> Epoch[5475](27375/5): Loss 0.0119, Sem 0.0119 LR: 5.779e-02 Score 99.571 Data time: 0.3003, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7801 [2021-07-08 08:30:05,222][root][INFO] - ===> Epoch[5476](27380/5): Loss 0.0113, Sem 0.0113 LR: 5.778e-02 Score 99.591 Data time: 0.2445, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.9255 [2021-07-08 08:30:19,913][root][INFO] - ===> Epoch[5477](27385/5): Loss 0.0113, Sem 0.0113 LR: 5.778e-02 Score 99.588 Data time: 0.2446, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8773 [2021-07-08 08:30:33,601][root][INFO] - ===> Epoch[5478](27390/5): Loss 0.0119, Sem 0.0119 LR: 5.777e-02 Score 99.567 Data time: 0.1945, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6844 [2021-07-08 08:30:47,236][root][INFO] - ===> Epoch[5479](27395/5): Loss 0.0110, Sem 0.0110 LR: 5.776e-02 Score 99.605 Data time: 0.2805, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6709 [2021-07-08 08:31:02,795][root][INFO] - ===> Epoch[5480](27400/5): Loss 0.0121, Sem 0.0121 LR: 5.775e-02 Score 99.569 Data time: 0.3479, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 3.0604 [2021-07-08 08:31:02,796][root][INFO] - ===> Start testing [2021-07-08 08:31:48,677][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1379 Loss 0.469 (AVG: 0.556) Score 90.889 (AVG: 90.978) mIOU 70.542 mAP 84.271 mAcc 77.404 IOU: 64.854 0.000 83.630 76.262 94.365 91.711 27.220 80.426 97.920 76.932 77.358 86.906 59.462 mAP: 85.428 6.068 94.964 88.557 98.765 98.455 57.686 98.312 99.777 89.918 90.929 94.655 92.016 mAcc: 80.527 0.000 91.545 83.715 95.502 97.507 29.853 96.135 99.370 83.968 89.062 96.501 62.570 [2021-07-08 08:31:48,677][root][INFO] - Finished test. Elapsed time: 45.8804 [2021-07-08 08:31:49,772][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 08:32:03,890][root][INFO] - ===> Epoch[5481](27405/5): Loss 0.0121, Sem 0.0121 LR: 5.774e-02 Score 99.562 Data time: 0.1657, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7923 [2021-07-08 08:32:16,863][root][INFO] - ===> Epoch[5482](27410/5): Loss 0.0116, Sem 0.0116 LR: 5.774e-02 Score 99.578 Data time: 0.2151, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.5533 [2021-07-08 08:32:30,851][root][INFO] - ===> Epoch[5483](27415/5): Loss 0.0120, Sem 0.0120 LR: 5.773e-02 Score 99.568 Data time: 0.2574, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7555 [2021-07-08 08:32:44,630][root][INFO] - ===> Epoch[5484](27420/5): Loss 0.0112, Sem 0.0112 LR: 5.772e-02 Score 99.598 Data time: 0.2651, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7070 [2021-07-08 08:32:58,196][root][INFO] - ===> Epoch[5485](27425/5): Loss 0.0122, Sem 0.0122 LR: 5.771e-02 Score 99.562 Data time: 0.1701, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6690 [2021-07-08 08:33:12,056][root][INFO] - ===> Epoch[5486](27430/5): Loss 0.0118, Sem 0.0118 LR: 5.770e-02 Score 99.570 Data time: 0.2414, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7285 [2021-07-08 08:33:26,639][root][INFO] - ===> Epoch[5487](27435/5): Loss 0.0111, Sem 0.0111 LR: 5.770e-02 Score 99.599 Data time: 0.2466, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8698 [2021-07-08 08:33:40,864][root][INFO] - ===> Epoch[5488](27440/5): Loss 0.0117, Sem 0.0117 LR: 5.769e-02 Score 99.579 Data time: 0.2249, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7998 [2021-07-08 08:33:54,154][root][INFO] - ===> Epoch[5489](27445/5): Loss 0.0113, Sem 0.0113 LR: 5.768e-02 Score 99.593 Data time: 0.2035, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6155 [2021-07-08 08:34:07,518][root][INFO] - ===> Epoch[5490](27450/5): Loss 0.0112, Sem 0.0112 LR: 5.767e-02 Score 99.597 Data time: 0.2524, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.6235 [2021-07-08 08:34:21,853][root][INFO] - ===> Epoch[5491](27455/5): Loss 0.0115, Sem 0.0115 LR: 5.766e-02 Score 99.582 Data time: 0.1985, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.8178 [2021-07-08 08:34:35,581][root][INFO] - ===> Epoch[5492](27460/5): Loss 0.0110, Sem 0.0110 LR: 5.766e-02 Score 99.599 Data time: 0.1578, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7109 [2021-07-08 08:34:49,128][root][INFO] - ===> Epoch[5493](27465/5): Loss 0.0116, Sem 0.0116 LR: 5.765e-02 Score 99.580 Data time: 0.2544, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.6572 [2021-07-08 08:35:02,932][root][INFO] - ===> Epoch[5494](27470/5): Loss 0.0114, Sem 0.0114 LR: 5.764e-02 Score 99.591 Data time: 0.1957, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7241 [2021-07-08 08:35:16,640][root][INFO] - ===> Epoch[5495](27475/5): Loss 0.0111, Sem 0.0111 LR: 5.763e-02 Score 99.595 Data time: 0.1666, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7063 [2021-07-08 08:35:30,493][root][INFO] - ===> Epoch[5496](27480/5): Loss 0.0115, Sem 0.0115 LR: 5.762e-02 Score 99.584 Data time: 0.2066, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7252 [2021-07-08 08:35:44,472][root][INFO] - ===> Epoch[5497](27485/5): Loss 0.0115, Sem 0.0115 LR: 5.762e-02 Score 99.583 Data time: 0.2224, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7491 [2021-07-08 08:35:59,116][root][INFO] - ===> Epoch[5498](27490/5): Loss 0.0108, Sem 0.0108 LR: 5.761e-02 Score 99.610 Data time: 0.2032, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8830 [2021-07-08 08:36:13,210][root][INFO] - ===> Epoch[5499](27495/5): Loss 0.0112, Sem 0.0112 LR: 5.760e-02 Score 99.597 Data time: 0.1738, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7735 [2021-07-08 08:36:26,854][root][INFO] - ===> Epoch[5500](27500/5): Loss 0.0115, Sem 0.0115 LR: 5.759e-02 Score 99.586 Data time: 0.2195, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6861 [2021-07-08 08:36:40,135][root][INFO] - ===> Epoch[5501](27505/5): Loss 0.0113, Sem 0.0113 LR: 5.758e-02 Score 99.588 Data time: 0.2133, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.6127 [2021-07-08 08:36:54,219][root][INFO] - ===> Epoch[5502](27510/5): Loss 0.0111, Sem 0.0111 LR: 5.758e-02 Score 99.593 Data time: 0.2432, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7670 [2021-07-08 08:37:07,661][root][INFO] - ===> Epoch[5503](27515/5): Loss 0.0110, Sem 0.0110 LR: 5.757e-02 Score 99.601 Data time: 0.1681, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.6528 [2021-07-08 08:37:21,070][root][INFO] - ===> Epoch[5504](27520/5): Loss 0.0114, Sem 0.0114 LR: 5.756e-02 Score 99.584 Data time: 0.2625, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.6365 [2021-07-08 08:37:35,012][root][INFO] - ===> Epoch[5505](27525/5): Loss 0.0113, Sem 0.0113 LR: 5.755e-02 Score 99.597 Data time: 0.3022, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7296 [2021-07-08 08:37:49,548][root][INFO] - ===> Epoch[5506](27530/5): Loss 0.0116, Sem 0.0116 LR: 5.754e-02 Score 99.580 Data time: 0.3145, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.8528 [2021-07-08 08:38:03,854][root][INFO] - ===> Epoch[5507](27535/5): Loss 0.0108, Sem 0.0108 LR: 5.754e-02 Score 99.610 Data time: 0.2617, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.8095 [2021-07-08 08:38:17,913][root][INFO] - ===> Epoch[5508](27540/5): Loss 0.0114, Sem 0.0114 LR: 5.753e-02 Score 99.581 Data time: 0.2597, Forward time: 1.2605, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.7724 [2021-07-08 08:38:32,119][root][INFO] - ===> Epoch[5509](27545/5): Loss 0.0112, Sem 0.0112 LR: 5.752e-02 Score 99.591 Data time: 0.2319, Forward time: 1.2605, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.7883 [2021-07-08 08:38:46,358][root][INFO] - ===> Epoch[5510](27550/5): Loss 0.0109, Sem 0.0109 LR: 5.751e-02 Score 99.602 Data time: 0.2176, Forward time: 1.2605, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.8021 [2021-07-08 08:39:00,316][root][INFO] - ===> Epoch[5511](27555/5): Loss 0.0112, Sem 0.0112 LR: 5.750e-02 Score 99.601 Data time: 0.2116, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7513 [2021-07-08 08:39:13,517][root][INFO] - ===> Epoch[5512](27560/5): Loss 0.0116, Sem 0.0116 LR: 5.750e-02 Score 99.582 Data time: 0.2155, Forward time: 1.2605, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.5887 [2021-07-08 08:39:27,507][root][INFO] - ===> Epoch[5513](27565/5): Loss 0.0109, Sem 0.0109 LR: 5.749e-02 Score 99.604 Data time: 0.2088, Forward time: 1.2605, Backward time: 1.2375, DDP time: 0.0042, Total iter time: 2.7457 [2021-07-08 08:39:42,058][root][INFO] - ===> Epoch[5514](27570/5): Loss 0.0112, Sem 0.0112 LR: 5.748e-02 Score 99.590 Data time: 0.2440, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.8581 [2021-07-08 08:39:57,546][root][INFO] - ===> Epoch[5515](27575/5): Loss 0.0107, Sem 0.0107 LR: 5.747e-02 Score 99.614 Data time: 0.1914, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 3.0383 [2021-07-08 08:40:11,065][root][INFO] - ===> Epoch[5516](27580/5): Loss 0.0114, Sem 0.0114 LR: 5.746e-02 Score 99.586 Data time: 0.2351, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.6477 [2021-07-08 08:40:25,729][root][INFO] - ===> Epoch[5517](27585/5): Loss 0.0114, Sem 0.0114 LR: 5.746e-02 Score 99.587 Data time: 0.1853, Forward time: 1.2605, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8856 [2021-07-08 08:40:39,576][root][INFO] - ===> Epoch[5518](27590/5): Loss 0.0111, Sem 0.0111 LR: 5.745e-02 Score 99.598 Data time: 0.2348, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7230 [2021-07-08 08:40:53,554][root][INFO] - ===> Epoch[5519](27595/5): Loss 0.0110, Sem 0.0110 LR: 5.744e-02 Score 99.600 Data time: 0.2251, Forward time: 1.2605, Backward time: 1.2376, DDP time: 0.0042, Total iter time: 2.7584 [2021-07-08 08:41:08,010][root][INFO] - ===> Epoch[5520](27600/5): Loss 0.0112, Sem 0.0112 LR: 5.743e-02 Score 99.591 Data time: 0.2160, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8574 [2021-07-08 08:41:08,011][root][INFO] - ===> Start testing [2021-07-08 08:41:49,465][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.2449 Loss 0.321 (AVG: 0.553) Score 92.755 (AVG: 91.174) mIOU 71.028 mAP 83.630 mAcc 77.382 IOU: 65.311 0.000 84.732 76.272 95.183 92.140 27.148 82.696 98.251 77.828 78.202 86.894 58.705 mAP: 84.485 4.904 93.522 87.252 98.808 98.708 55.757 98.569 99.794 87.985 91.576 94.258 91.570 mAcc: 81.547 0.000 89.872 82.775 96.560 97.329 29.718 97.396 99.379 82.212 90.425 96.298 62.461 [2021-07-08 08:41:49,466][root][INFO] - Finished test. Elapsed time: 41.4538 [2021-07-08 08:41:50,522][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 08:42:03,951][root][INFO] - ===> Epoch[5521](27605/5): Loss 0.0110, Sem 0.0110 LR: 5.742e-02 Score 99.602 Data time: 0.2342, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6410 [2021-07-08 08:42:18,702][root][INFO] - ===> Epoch[5522](27610/5): Loss 0.0111, Sem 0.0111 LR: 5.742e-02 Score 99.599 Data time: 0.2500, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.9067 [2021-07-08 08:42:32,940][root][INFO] - ===> Epoch[5523](27615/5): Loss 0.0110, Sem 0.0110 LR: 5.741e-02 Score 99.603 Data time: 0.2296, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7922 [2021-07-08 08:42:46,963][root][INFO] - ===> Epoch[5524](27620/5): Loss 0.0108, Sem 0.0108 LR: 5.740e-02 Score 99.611 Data time: 0.2136, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7598 [2021-07-08 08:43:00,857][root][INFO] - ===> Epoch[5525](27625/5): Loss 0.0117, Sem 0.0117 LR: 5.739e-02 Score 99.587 Data time: 0.2346, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7368 [2021-07-08 08:43:14,700][root][INFO] - ===> Epoch[5526](27630/5): Loss 0.0109, Sem 0.0109 LR: 5.738e-02 Score 99.603 Data time: 0.2109, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7280 [2021-07-08 08:43:28,431][root][INFO] - ===> Epoch[5527](27635/5): Loss 0.0105, Sem 0.0105 LR: 5.738e-02 Score 99.618 Data time: 0.2242, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6894 [2021-07-08 08:43:42,834][root][INFO] - ===> Epoch[5528](27640/5): Loss 0.0111, Sem 0.0111 LR: 5.737e-02 Score 99.591 Data time: 0.2707, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.8348 [2021-07-08 08:43:57,395][root][INFO] - ===> Epoch[5529](27645/5): Loss 0.0111, Sem 0.0111 LR: 5.736e-02 Score 99.601 Data time: 0.1727, Forward time: 1.2604, Backward time: 1.2378, DDP time: 0.0042, Total iter time: 2.8773 [2021-07-08 08:44:11,451][root][INFO] - ===> Epoch[5530](27650/5): Loss 0.0110, Sem 0.0110 LR: 5.735e-02 Score 99.608 Data time: 0.2294, Forward time: 1.2604, Backward time: 1.2378, DDP time: 0.0042, Total iter time: 2.7590 [2021-07-08 08:44:24,987][root][INFO] - ===> Epoch[5531](27655/5): Loss 0.0109, Sem 0.0109 LR: 5.735e-02 Score 99.606 Data time: 0.2380, Forward time: 1.2604, Backward time: 1.2378, DDP time: 0.0042, Total iter time: 2.6620 [2021-07-08 08:44:38,874][root][INFO] - ===> Epoch[5532](27660/5): Loss 0.0116, Sem 0.0116 LR: 5.734e-02 Score 99.599 Data time: 0.2139, Forward time: 1.2604, Backward time: 1.2378, DDP time: 0.0042, Total iter time: 2.7366 [2021-07-08 08:44:53,093][root][INFO] - ===> Epoch[5533](27665/5): Loss 0.0105, Sem 0.0105 LR: 5.733e-02 Score 99.619 Data time: 0.2387, Forward time: 1.2603, Backward time: 1.2378, DDP time: 0.0042, Total iter time: 2.7910 [2021-07-08 08:45:07,731][root][INFO] - ===> Epoch[5534](27670/5): Loss 0.0106, Sem 0.0106 LR: 5.732e-02 Score 99.615 Data time: 0.3510, Forward time: 1.2603, Backward time: 1.2378, DDP time: 0.0042, Total iter time: 2.8853 [2021-07-08 08:45:21,302][root][INFO] - ===> Epoch[5535](27675/5): Loss 0.0114, Sem 0.0114 LR: 5.731e-02 Score 99.592 Data time: 0.2907, Forward time: 1.2603, Backward time: 1.2378, DDP time: 0.0042, Total iter time: 2.6649 [2021-07-08 08:45:34,897][root][INFO] - ===> Epoch[5536](27680/5): Loss 0.0106, Sem 0.0106 LR: 5.731e-02 Score 99.616 Data time: 0.2593, Forward time: 1.2603, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.6713 [2021-07-08 08:45:48,883][root][INFO] - ===> Epoch[5537](27685/5): Loss 0.0111, Sem 0.0111 LR: 5.730e-02 Score 99.600 Data time: 0.2382, Forward time: 1.2604, Backward time: 1.2377, DDP time: 0.0042, Total iter time: 2.7467 [2021-07-08 08:46:03,057][root][INFO] - ===> Epoch[5538](27690/5): Loss 0.0112, Sem 0.0112 LR: 5.729e-02 Score 99.601 Data time: 0.2389, Forward time: 1.2603, Backward time: 1.2378, DDP time: 0.0042, Total iter time: 2.7982 [2021-07-08 08:46:18,064][root][INFO] - ===> Epoch[5539](27695/5): Loss 0.0107, Sem 0.0107 LR: 5.728e-02 Score 99.609 Data time: 0.2064, Forward time: 1.2603, Backward time: 1.2378, DDP time: 0.0042, Total iter time: 2.9460 [2021-07-08 08:46:32,268][root][INFO] - ===> Epoch[5540](27700/5): Loss 0.0108, Sem 0.0108 LR: 5.727e-02 Score 99.612 Data time: 0.2503, Forward time: 1.2603, Backward time: 1.2378, DDP time: 0.0042, Total iter time: 2.7977 [2021-07-08 08:46:45,989][root][INFO] - ===> Epoch[5541](27705/5): Loss 0.0110, Sem 0.0110 LR: 5.727e-02 Score 99.602 Data time: 0.1790, Forward time: 1.2603, Backward time: 1.2378, DDP time: 0.0042, Total iter time: 2.6894 [2021-07-08 08:47:00,100][root][INFO] - ===> Epoch[5542](27710/5): Loss 0.0106, Sem 0.0106 LR: 5.726e-02 Score 99.619 Data time: 0.2120, Forward time: 1.2603, Backward time: 1.2378, DDP time: 0.0042, Total iter time: 2.7841 [2021-07-08 08:47:14,378][root][INFO] - ===> Epoch[5543](27715/5): Loss 0.0113, Sem 0.0113 LR: 5.725e-02 Score 99.592 Data time: 0.1912, Forward time: 1.2602, Backward time: 1.2379, DDP time: 0.0042, Total iter time: 2.8124 [2021-07-08 08:47:28,625][root][INFO] - ===> Epoch[5544](27720/5): Loss 0.0105, Sem 0.0105 LR: 5.724e-02 Score 99.620 Data time: 0.1886, Forward time: 1.2602, Backward time: 1.2379, DDP time: 0.0042, Total iter time: 2.8038 [2021-07-08 08:47:42,798][root][INFO] - ===> Epoch[5545](27725/5): Loss 0.0103, Sem 0.0103 LR: 5.723e-02 Score 99.627 Data time: 0.2000, Forward time: 1.2602, Backward time: 1.2379, DDP time: 0.0042, Total iter time: 2.7887 [2021-07-08 08:47:56,742][root][INFO] - ===> Epoch[5546](27730/5): Loss 0.0111, Sem 0.0111 LR: 5.723e-02 Score 99.605 Data time: 0.2265, Forward time: 1.2602, Backward time: 1.2379, DDP time: 0.0042, Total iter time: 2.7463 [2021-07-08 08:48:11,421][root][INFO] - ===> Epoch[5547](27735/5): Loss 0.0109, Sem 0.0109 LR: 5.722e-02 Score 99.607 Data time: 0.2046, Forward time: 1.2602, Backward time: 1.2379, DDP time: 0.0042, Total iter time: 2.8895 [2021-07-08 08:48:25,105][root][INFO] - ===> Epoch[5548](27740/5): Loss 0.0112, Sem 0.0112 LR: 5.721e-02 Score 99.597 Data time: 0.1968, Forward time: 1.2602, Backward time: 1.2379, DDP time: 0.0042, Total iter time: 2.6982 [2021-07-08 08:48:39,101][root][INFO] - ===> Epoch[5549](27745/5): Loss 0.0107, Sem 0.0107 LR: 5.720e-02 Score 99.612 Data time: 0.2524, Forward time: 1.2602, Backward time: 1.2379, DDP time: 0.0042, Total iter time: 2.7563 [2021-07-08 08:48:52,863][root][INFO] - ===> Epoch[5550](27750/5): Loss 0.0106, Sem 0.0106 LR: 5.719e-02 Score 99.613 Data time: 0.2208, Forward time: 1.2602, Backward time: 1.2379, DDP time: 0.0042, Total iter time: 2.7076 [2021-07-08 08:49:06,459][root][INFO] - ===> Epoch[5551](27755/5): Loss 0.0108, Sem 0.0108 LR: 5.719e-02 Score 99.614 Data time: 0.2598, Forward time: 1.2602, Backward time: 1.2379, DDP time: 0.0042, Total iter time: 2.6749 [2021-07-08 08:49:20,555][root][INFO] - ===> Epoch[5552](27760/5): Loss 0.0108, Sem 0.0108 LR: 5.718e-02 Score 99.610 Data time: 0.2802, Forward time: 1.2602, Backward time: 1.2379, DDP time: 0.0042, Total iter time: 2.7695 [2021-07-08 08:49:34,513][root][INFO] - ===> Epoch[5553](27765/5): Loss 0.0109, Sem 0.0109 LR: 5.717e-02 Score 99.602 Data time: 0.1786, Forward time: 1.2602, Backward time: 1.2379, DDP time: 0.0042, Total iter time: 2.7583 [2021-07-08 08:49:48,340][root][INFO] - ===> Epoch[5554](27770/5): Loss 0.0106, Sem 0.0106 LR: 5.716e-02 Score 99.616 Data time: 0.1692, Forward time: 1.2602, Backward time: 1.2380, DDP time: 0.0042, Total iter time: 2.7230 [2021-07-08 08:50:02,735][root][INFO] - ===> Epoch[5555](27775/5): Loss 0.0109, Sem 0.0109 LR: 5.715e-02 Score 99.608 Data time: 0.1954, Forward time: 1.2601, Backward time: 1.2380, DDP time: 0.0042, Total iter time: 2.8379 [2021-07-08 08:50:16,931][root][INFO] - ===> Epoch[5556](27780/5): Loss 0.0108, Sem 0.0108 LR: 5.715e-02 Score 99.615 Data time: 0.2167, Forward time: 1.2601, Backward time: 1.2380, DDP time: 0.0042, Total iter time: 2.7964 [2021-07-08 08:50:30,414][root][INFO] - ===> Epoch[5557](27785/5): Loss 0.0109, Sem 0.0109 LR: 5.714e-02 Score 99.607 Data time: 0.2137, Forward time: 1.2601, Backward time: 1.2380, DDP time: 0.0042, Total iter time: 2.6557 [2021-07-08 08:50:44,394][root][INFO] - ===> Epoch[5558](27790/5): Loss 0.0108, Sem 0.0108 LR: 5.713e-02 Score 99.608 Data time: 0.1899, Forward time: 1.2601, Backward time: 1.2380, DDP time: 0.0042, Total iter time: 2.7513 [2021-07-08 08:50:58,942][root][INFO] - ===> Epoch[5559](27795/5): Loss 0.0111, Sem 0.0111 LR: 5.712e-02 Score 99.598 Data time: 0.2127, Forward time: 1.2601, Backward time: 1.2380, DDP time: 0.0042, Total iter time: 2.8683 [2021-07-08 08:51:12,446][root][INFO] - ===> Epoch[5560](27800/5): Loss 0.0114, Sem 0.0114 LR: 5.711e-02 Score 99.584 Data time: 0.2172, Forward time: 1.2601, Backward time: 1.2380, DDP time: 0.0042, Total iter time: 2.6576 [2021-07-08 08:51:12,447][root][INFO] - ===> Start testing [2021-07-08 08:51:50,539][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1220 Loss 0.408 (AVG: 0.548) Score 92.171 (AVG: 90.916) mIOU 70.636 mAP 84.188 mAcc 77.233 IOU: 63.735 0.000 83.532 76.058 94.351 92.048 27.512 83.498 98.335 75.368 78.021 86.484 59.331 mAP: 85.042 5.666 96.105 89.228 98.773 98.555 55.515 98.441 99.760 89.948 91.957 93.891 91.567 mAcc: 76.378 0.000 91.564 86.024 95.759 96.475 31.161 95.481 99.518 82.291 90.265 96.464 62.648 [2021-07-08 08:51:50,540][root][INFO] - Finished test. Elapsed time: 38.0918 [2021-07-08 08:51:51,628][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 08:52:03,788][root][INFO] - ===> Epoch[5561](27805/5): Loss 0.0114, Sem 0.0114 LR: 5.711e-02 Score 99.592 Data time: 0.1575, Forward time: 1.2601, Backward time: 1.2380, DDP time: 0.0042, Total iter time: 2.3971 [2021-07-08 08:52:17,240][root][INFO] - ===> Epoch[5562](27810/5): Loss 0.0117, Sem 0.0117 LR: 5.710e-02 Score 99.581 Data time: 0.2062, Forward time: 1.2601, Backward time: 1.2380, DDP time: 0.0042, Total iter time: 2.6544 [2021-07-08 08:52:31,710][root][INFO] - ===> Epoch[5563](27815/5): Loss 0.0113, Sem 0.0113 LR: 5.709e-02 Score 99.592 Data time: 0.2074, Forward time: 1.2601, Backward time: 1.2380, DDP time: 0.0042, Total iter time: 2.8510 [2021-07-08 08:52:46,225][root][INFO] - ===> Epoch[5564](27820/5): Loss 0.0120, Sem 0.0120 LR: 5.708e-02 Score 99.569 Data time: 0.2007, Forward time: 1.2601, Backward time: 1.2381, DDP time: 0.0042, Total iter time: 2.8475 [2021-07-08 08:53:00,789][root][INFO] - ===> Epoch[5565](27825/5): Loss 0.0118, Sem 0.0118 LR: 5.707e-02 Score 99.571 Data time: 0.2290, Forward time: 1.2601, Backward time: 1.2381, DDP time: 0.0042, Total iter time: 2.8649 [2021-07-08 08:53:14,798][root][INFO] - ===> Epoch[5566](27830/5): Loss 0.0114, Sem 0.0114 LR: 5.707e-02 Score 99.586 Data time: 0.2502, Forward time: 1.2601, Backward time: 1.2381, DDP time: 0.0042, Total iter time: 2.7482 [2021-07-08 08:53:29,073][root][INFO] - ===> Epoch[5567](27835/5): Loss 0.0119, Sem 0.0119 LR: 5.706e-02 Score 99.579 Data time: 0.2867, Forward time: 1.2601, Backward time: 1.2380, DDP time: 0.0042, Total iter time: 2.7985 [2021-07-08 08:53:42,603][root][INFO] - ===> Epoch[5568](27840/5): Loss 0.0124, Sem 0.0124 LR: 5.705e-02 Score 99.557 Data time: 0.1652, Forward time: 1.2601, Backward time: 1.2381, DDP time: 0.0042, Total iter time: 2.6637 [2021-07-08 08:53:56,818][root][INFO] - ===> Epoch[5569](27845/5): Loss 0.0121, Sem 0.0121 LR: 5.704e-02 Score 99.559 Data time: 0.2118, Forward time: 1.2601, Backward time: 1.2381, DDP time: 0.0042, Total iter time: 2.8049 [2021-07-08 08:54:10,673][root][INFO] - ===> Epoch[5570](27850/5): Loss 0.0117, Sem 0.0117 LR: 5.703e-02 Score 99.581 Data time: 0.2360, Forward time: 1.2601, Backward time: 1.2381, DDP time: 0.0042, Total iter time: 2.7128 [2021-07-08 08:54:24,739][root][INFO] - ===> Epoch[5571](27855/5): Loss 0.0120, Sem 0.0120 LR: 5.703e-02 Score 99.578 Data time: 0.1804, Forward time: 1.2601, Backward time: 1.2381, DDP time: 0.0042, Total iter time: 2.7768 [2021-07-08 08:54:38,334][root][INFO] - ===> Epoch[5572](27860/5): Loss 0.0120, Sem 0.0120 LR: 5.702e-02 Score 99.568 Data time: 0.1862, Forward time: 1.2601, Backward time: 1.2381, DDP time: 0.0042, Total iter time: 2.6788 [2021-07-08 08:54:52,469][root][INFO] - ===> Epoch[5573](27865/5): Loss 0.0119, Sem 0.0119 LR: 5.701e-02 Score 99.570 Data time: 0.1951, Forward time: 1.2600, Backward time: 1.2381, DDP time: 0.0042, Total iter time: 2.7866 [2021-07-08 08:55:06,921][root][INFO] - ===> Epoch[5574](27870/5): Loss 0.0126, Sem 0.0126 LR: 5.700e-02 Score 99.553 Data time: 0.2370, Forward time: 1.2600, Backward time: 1.2381, DDP time: 0.0042, Total iter time: 2.8507 [2021-07-08 08:55:21,055][root][INFO] - ===> Epoch[5575](27875/5): Loss 0.0115, Sem 0.0115 LR: 5.699e-02 Score 99.585 Data time: 0.2152, Forward time: 1.2600, Backward time: 1.2382, DDP time: 0.0042, Total iter time: 2.7756 [2021-07-08 08:55:34,800][root][INFO] - ===> Epoch[5576](27880/5): Loss 0.0114, Sem 0.0114 LR: 5.699e-02 Score 99.587 Data time: 0.1863, Forward time: 1.2600, Backward time: 1.2382, DDP time: 0.0042, Total iter time: 2.6987 [2021-07-08 08:55:49,991][root][INFO] - ===> Epoch[5577](27885/5): Loss 0.0113, Sem 0.0113 LR: 5.698e-02 Score 99.593 Data time: 0.1818, Forward time: 1.2600, Backward time: 1.2382, DDP time: 0.0042, Total iter time: 2.9986 [2021-07-08 08:56:04,009][root][INFO] - ===> Epoch[5578](27890/5): Loss 0.0114, Sem 0.0114 LR: 5.697e-02 Score 99.590 Data time: 0.2071, Forward time: 1.2600, Backward time: 1.2382, DDP time: 0.0042, Total iter time: 2.7563 [2021-07-08 08:56:18,242][root][INFO] - ===> Epoch[5579](27895/5): Loss 0.0113, Sem 0.0113 LR: 5.696e-02 Score 99.594 Data time: 0.2394, Forward time: 1.2600, Backward time: 1.2382, DDP time: 0.0042, Total iter time: 2.8067 [2021-07-08 08:56:32,804][root][INFO] - ===> Epoch[5580](27900/5): Loss 0.0110, Sem 0.0110 LR: 5.695e-02 Score 99.603 Data time: 0.2527, Forward time: 1.2600, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.8685 [2021-07-08 08:56:46,646][root][INFO] - ===> Epoch[5581](27905/5): Loss 0.0115, Sem 0.0115 LR: 5.695e-02 Score 99.584 Data time: 0.2268, Forward time: 1.2600, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.7274 [2021-07-08 08:57:00,673][root][INFO] - ===> Epoch[5582](27910/5): Loss 0.0111, Sem 0.0111 LR: 5.694e-02 Score 99.599 Data time: 0.2034, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.7709 [2021-07-08 08:57:14,432][root][INFO] - ===> Epoch[5583](27915/5): Loss 0.0116, Sem 0.0116 LR: 5.693e-02 Score 99.582 Data time: 0.2357, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.7116 [2021-07-08 08:57:27,577][root][INFO] - ===> Epoch[5584](27920/5): Loss 0.0116, Sem 0.0116 LR: 5.692e-02 Score 99.583 Data time: 0.2068, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.5881 [2021-07-08 08:57:41,887][root][INFO] - ===> Epoch[5585](27925/5): Loss 0.0111, Sem 0.0111 LR: 5.691e-02 Score 99.594 Data time: 0.2472, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.7996 [2021-07-08 08:57:57,289][root][INFO] - ===> Epoch[5586](27930/5): Loss 0.0108, Sem 0.0108 LR: 5.691e-02 Score 99.609 Data time: 0.1840, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 3.0297 [2021-07-08 08:58:10,856][root][INFO] - ===> Epoch[5587](27935/5): Loss 0.0115, Sem 0.0115 LR: 5.690e-02 Score 99.590 Data time: 0.2335, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.6602 [2021-07-08 08:58:24,874][root][INFO] - ===> Epoch[5588](27940/5): Loss 0.0104, Sem 0.0104 LR: 5.689e-02 Score 99.623 Data time: 0.1929, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.7570 [2021-07-08 08:58:38,965][root][INFO] - ===> Epoch[5589](27945/5): Loss 0.0106, Sem 0.0106 LR: 5.688e-02 Score 99.616 Data time: 0.2038, Forward time: 1.2599, Backward time: 1.2384, DDP time: 0.0042, Total iter time: 2.7753 [2021-07-08 08:58:52,658][root][INFO] - ===> Epoch[5590](27950/5): Loss 0.0108, Sem 0.0108 LR: 5.687e-02 Score 99.607 Data time: 0.2077, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.6958 [2021-07-08 08:59:06,190][root][INFO] - ===> Epoch[5591](27955/5): Loss 0.0105, Sem 0.0105 LR: 5.687e-02 Score 99.620 Data time: 0.1914, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.6612 [2021-07-08 08:59:19,918][root][INFO] - ===> Epoch[5592](27960/5): Loss 0.0109, Sem 0.0109 LR: 5.686e-02 Score 99.603 Data time: 0.2607, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.6953 [2021-07-08 08:59:33,640][root][INFO] - ===> Epoch[5593](27965/5): Loss 0.0106, Sem 0.0106 LR: 5.685e-02 Score 99.620 Data time: 0.2116, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.6971 [2021-07-08 08:59:48,055][root][INFO] - ===> Epoch[5594](27970/5): Loss 0.0109, Sem 0.0109 LR: 5.684e-02 Score 99.606 Data time: 0.2156, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.8434 [2021-07-08 09:00:01,937][root][INFO] - ===> Epoch[5595](27975/5): Loss 0.0110, Sem 0.0110 LR: 5.683e-02 Score 99.604 Data time: 0.2447, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.7270 [2021-07-08 09:00:16,263][root][INFO] - ===> Epoch[5596](27980/5): Loss 0.0109, Sem 0.0109 LR: 5.683e-02 Score 99.608 Data time: 0.2120, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.8253 [2021-07-08 09:00:29,747][root][INFO] - ===> Epoch[5597](27985/5): Loss 0.0110, Sem 0.0110 LR: 5.682e-02 Score 99.605 Data time: 0.2638, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.6479 [2021-07-08 09:00:43,932][root][INFO] - ===> Epoch[5598](27990/5): Loss 0.0109, Sem 0.0109 LR: 5.681e-02 Score 99.606 Data time: 0.2467, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.7948 [2021-07-08 09:00:58,766][root][INFO] - ===> Epoch[5599](27995/5): Loss 0.0108, Sem 0.0108 LR: 5.680e-02 Score 99.608 Data time: 0.2461, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.9273 [2021-07-08 09:01:12,793][root][INFO] - ===> Epoch[5600](28000/5): Loss 0.0103, Sem 0.0103 LR: 5.679e-02 Score 99.629 Data time: 0.1815, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.7572 [2021-07-08 09:01:12,796][root][INFO] - ===> Start testing [2021-07-08 09:01:47,160][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1429 Loss 0.344 (AVG: 0.561) Score 92.897 (AVG: 91.048) mIOU 70.471 mAP 83.503 mAcc 76.839 IOU: 64.765 0.000 84.194 76.893 94.657 92.319 21.156 82.842 98.292 75.088 79.630 86.463 59.828 mAP: 84.538 6.262 95.501 88.233 98.784 98.655 49.227 98.642 99.751 88.061 92.010 93.917 91.962 mAcc: 81.000 0.000 91.626 84.463 95.817 97.006 22.824 96.061 99.467 79.815 90.767 96.122 63.938 [2021-07-08 09:01:47,160][root][INFO] - Finished test. Elapsed time: 34.3627 [2021-07-08 09:01:48,248][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 09:02:00,733][root][INFO] - ===> Epoch[5601](28005/5): Loss 0.0106, Sem 0.0106 LR: 5.679e-02 Score 99.618 Data time: 0.2670, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.4539 [2021-07-08 09:02:15,716][root][INFO] - ===> Epoch[5602](28010/5): Loss 0.0106, Sem 0.0106 LR: 5.678e-02 Score 99.617 Data time: 0.2269, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.9433 [2021-07-08 09:02:30,138][root][INFO] - ===> Epoch[5603](28015/5): Loss 0.0103, Sem 0.0103 LR: 5.677e-02 Score 99.627 Data time: 0.2278, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.8437 [2021-07-08 09:02:44,141][root][INFO] - ===> Epoch[5604](28020/5): Loss 0.0104, Sem 0.0104 LR: 5.676e-02 Score 99.624 Data time: 0.2076, Forward time: 1.2599, Backward time: 1.2383, DDP time: 0.0042, Total iter time: 2.7557 [2021-07-08 09:02:58,349][root][INFO] - ===> Epoch[5605](28025/5): Loss 0.0106, Sem 0.0106 LR: 5.675e-02 Score 99.614 Data time: 0.2405, Forward time: 1.2599, Backward time: 1.2384, DDP time: 0.0042, Total iter time: 2.8033 [2021-07-08 09:03:13,024][root][INFO] - ===> Epoch[5606](28030/5): Loss 0.0108, Sem 0.0108 LR: 5.675e-02 Score 99.606 Data time: 0.2401, Forward time: 1.2599, Backward time: 1.2384, DDP time: 0.0042, Total iter time: 2.8827 [2021-07-08 09:03:26,734][root][INFO] - ===> Epoch[5607](28035/5): Loss 0.0108, Sem 0.0108 LR: 5.674e-02 Score 99.611 Data time: 0.2339, Forward time: 1.2599, Backward time: 1.2384, DDP time: 0.0042, Total iter time: 2.6968 [2021-07-08 09:03:40,582][root][INFO] - ===> Epoch[5608](28040/5): Loss 0.0103, Sem 0.0103 LR: 5.673e-02 Score 99.626 Data time: 0.2470, Forward time: 1.2599, Backward time: 1.2384, DDP time: 0.0042, Total iter time: 2.7311 [2021-07-08 09:03:55,249][root][INFO] - ===> Epoch[5609](28045/5): Loss 0.0107, Sem 0.0107 LR: 5.672e-02 Score 99.614 Data time: 0.2210, Forward time: 1.2599, Backward time: 1.2384, DDP time: 0.0042, Total iter time: 2.8885 [2021-07-08 09:04:09,083][root][INFO] - ===> Epoch[5610](28050/5): Loss 0.0109, Sem 0.0109 LR: 5.671e-02 Score 99.605 Data time: 0.2115, Forward time: 1.2598, Backward time: 1.2384, DDP time: 0.0042, Total iter time: 2.7195 [2021-07-08 09:04:22,522][root][INFO] - ===> Epoch[5611](28055/5): Loss 0.0103, Sem 0.0103 LR: 5.671e-02 Score 99.626 Data time: 0.1741, Forward time: 1.2598, Backward time: 1.2384, DDP time: 0.0042, Total iter time: 2.6366 [2021-07-08 09:04:36,649][root][INFO] - ===> Epoch[5612](28060/5): Loss 0.0107, Sem 0.0107 LR: 5.670e-02 Score 99.610 Data time: 0.1961, Forward time: 1.2598, Backward time: 1.2384, DDP time: 0.0042, Total iter time: 2.7720 [2021-07-08 09:04:50,545][root][INFO] - ===> Epoch[5613](28065/5): Loss 0.0107, Sem 0.0107 LR: 5.669e-02 Score 99.615 Data time: 0.2147, Forward time: 1.2598, Backward time: 1.2384, DDP time: 0.0042, Total iter time: 2.7379 [2021-07-08 09:05:04,475][root][INFO] - ===> Epoch[5614](28070/5): Loss 0.0108, Sem 0.0108 LR: 5.668e-02 Score 99.610 Data time: 0.2457, Forward time: 1.2598, Backward time: 1.2384, DDP time: 0.0042, Total iter time: 2.7394 [2021-07-08 09:05:18,896][root][INFO] - ===> Epoch[5615](28075/5): Loss 0.0103, Sem 0.0103 LR: 5.667e-02 Score 99.630 Data time: 0.1778, Forward time: 1.2598, Backward time: 1.2384, DDP time: 0.0042, Total iter time: 2.8429 [2021-07-08 09:05:32,783][root][INFO] - ===> Epoch[5616](28080/5): Loss 0.0102, Sem 0.0102 LR: 5.667e-02 Score 99.635 Data time: 0.2779, Forward time: 1.2598, Backward time: 1.2384, DDP time: 0.0042, Total iter time: 2.7224 [2021-07-08 09:05:48,898][root][INFO] - ===> Epoch[5617](28085/5): Loss 0.0107, Sem 0.0107 LR: 5.666e-02 Score 99.613 Data time: 0.2524, Forward time: 1.2598, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 3.1738 [2021-07-08 09:06:03,239][root][INFO] - ===> Epoch[5618](28090/5): Loss 0.0105, Sem 0.0105 LR: 5.665e-02 Score 99.619 Data time: 0.2136, Forward time: 1.2598, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.8166 [2021-07-08 09:06:16,639][root][INFO] - ===> Epoch[5619](28095/5): Loss 0.0111, Sem 0.0111 LR: 5.664e-02 Score 99.599 Data time: 0.2607, Forward time: 1.2598, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.6275 [2021-07-08 09:06:29,899][root][INFO] - ===> Epoch[5620](28100/5): Loss 0.0107, Sem 0.0107 LR: 5.663e-02 Score 99.614 Data time: 0.2258, Forward time: 1.2598, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.6080 [2021-07-08 09:06:44,028][root][INFO] - ===> Epoch[5621](28105/5): Loss 0.0109, Sem 0.0109 LR: 5.663e-02 Score 99.608 Data time: 0.2541, Forward time: 1.2597, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.7715 [2021-07-08 09:06:57,686][root][INFO] - ===> Epoch[5622](28110/5): Loss 0.0107, Sem 0.0107 LR: 5.662e-02 Score 99.611 Data time: 0.1974, Forward time: 1.2597, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.6788 [2021-07-08 09:07:11,473][root][INFO] - ===> Epoch[5623](28115/5): Loss 0.0105, Sem 0.0105 LR: 5.661e-02 Score 99.619 Data time: 0.2160, Forward time: 1.2597, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.7122 [2021-07-08 09:07:25,242][root][INFO] - ===> Epoch[5624](28120/5): Loss 0.0108, Sem 0.0108 LR: 5.660e-02 Score 99.613 Data time: 0.2347, Forward time: 1.2597, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.7110 [2021-07-08 09:07:39,223][root][INFO] - ===> Epoch[5625](28125/5): Loss 0.0109, Sem 0.0109 LR: 5.659e-02 Score 99.602 Data time: 0.2026, Forward time: 1.2597, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.7529 [2021-07-08 09:07:53,368][root][INFO] - ===> Epoch[5626](28130/5): Loss 0.0106, Sem 0.0106 LR: 5.659e-02 Score 99.614 Data time: 0.2473, Forward time: 1.2597, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.7786 [2021-07-08 09:08:07,587][root][INFO] - ===> Epoch[5627](28135/5): Loss 0.0105, Sem 0.0105 LR: 5.658e-02 Score 99.620 Data time: 0.2869, Forward time: 1.2597, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.7841 [2021-07-08 09:08:21,162][root][INFO] - ===> Epoch[5628](28140/5): Loss 0.0109, Sem 0.0109 LR: 5.657e-02 Score 99.607 Data time: 0.2452, Forward time: 1.2597, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.6663 [2021-07-08 09:08:35,830][root][INFO] - ===> Epoch[5629](28145/5): Loss 0.0103, Sem 0.0103 LR: 5.656e-02 Score 99.629 Data time: 0.3023, Forward time: 1.2597, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.8870 [2021-07-08 09:08:49,752][root][INFO] - ===> Epoch[5630](28150/5): Loss 0.0105, Sem 0.0105 LR: 5.655e-02 Score 99.618 Data time: 0.2003, Forward time: 1.2597, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.7389 [2021-07-08 09:09:04,058][root][INFO] - ===> Epoch[5631](28155/5): Loss 0.0105, Sem 0.0105 LR: 5.655e-02 Score 99.616 Data time: 0.1980, Forward time: 1.2597, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.8264 [2021-07-08 09:09:18,341][root][INFO] - ===> Epoch[5632](28160/5): Loss 0.0099, Sem 0.0099 LR: 5.654e-02 Score 99.642 Data time: 0.2494, Forward time: 1.2597, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.8051 [2021-07-08 09:09:32,380][root][INFO] - ===> Epoch[5633](28165/5): Loss 0.0109, Sem 0.0109 LR: 5.653e-02 Score 99.603 Data time: 0.2962, Forward time: 1.2597, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.7599 [2021-07-08 09:09:46,116][root][INFO] - ===> Epoch[5634](28170/5): Loss 0.0110, Sem 0.0110 LR: 5.652e-02 Score 99.605 Data time: 0.2109, Forward time: 1.2597, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.7110 [2021-07-08 09:10:01,115][root][INFO] - ===> Epoch[5635](28175/5): Loss 0.0104, Sem 0.0104 LR: 5.651e-02 Score 99.623 Data time: 0.2460, Forward time: 1.2597, Backward time: 1.2386, DDP time: 0.0042, Total iter time: 2.9552 [2021-07-08 09:10:15,921][root][INFO] - ===> Epoch[5636](28180/5): Loss 0.0106, Sem 0.0106 LR: 5.651e-02 Score 99.619 Data time: 0.3222, Forward time: 1.2597, Backward time: 1.2386, DDP time: 0.0042, Total iter time: 2.9053 [2021-07-08 09:10:30,845][root][INFO] - ===> Epoch[5637](28185/5): Loss 0.0104, Sem 0.0104 LR: 5.650e-02 Score 99.625 Data time: 0.2175, Forward time: 1.2597, Backward time: 1.2386, DDP time: 0.0042, Total iter time: 2.9434 [2021-07-08 09:10:43,840][root][INFO] - ===> Epoch[5638](28190/5): Loss 0.0106, Sem 0.0106 LR: 5.649e-02 Score 99.619 Data time: 0.2128, Forward time: 1.2597, Backward time: 1.2386, DDP time: 0.0042, Total iter time: 2.5608 [2021-07-08 09:10:58,119][root][INFO] - ===> Epoch[5639](28195/5): Loss 0.0106, Sem 0.0106 LR: 5.648e-02 Score 99.618 Data time: 0.2460, Forward time: 1.2597, Backward time: 1.2386, DDP time: 0.0042, Total iter time: 2.8020 [2021-07-08 09:11:12,764][root][INFO] - ===> Epoch[5640](28200/5): Loss 0.0100, Sem 0.0100 LR: 5.647e-02 Score 99.636 Data time: 0.2224, Forward time: 1.2597, Backward time: 1.2386, DDP time: 0.0042, Total iter time: 2.8816 [2021-07-08 09:11:12,765][root][INFO] - ===> Start testing [2021-07-08 09:11:54,093][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1255 Loss 0.375 (AVG: 0.572) Score 92.508 (AVG: 91.098) mIOU 70.896 mAP 83.527 mAcc 77.305 IOU: 64.690 0.000 84.857 76.243 94.851 91.936 27.855 81.564 98.300 76.329 78.234 86.956 59.828 mAP: 84.057 5.402 94.809 87.259 98.745 98.638 53.183 97.779 99.723 89.188 91.357 94.069 91.646 mAcc: 80.414 0.000 90.036 83.764 96.202 96.801 30.387 97.174 99.370 81.285 89.547 96.255 63.735 [2021-07-08 09:11:54,093][root][INFO] - Finished test. Elapsed time: 41.3272 [2021-07-08 09:11:55,234][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 09:12:07,338][root][INFO] - ===> Epoch[5641](28205/5): Loss 0.0107, Sem 0.0107 LR: 5.647e-02 Score 99.613 Data time: 0.2280, Forward time: 1.2596, Backward time: 1.2386, DDP time: 0.0042, Total iter time: 2.3772 [2021-07-08 09:12:20,829][root][INFO] - ===> Epoch[5642](28210/5): Loss 0.0104, Sem 0.0104 LR: 5.646e-02 Score 99.623 Data time: 0.2710, Forward time: 1.2596, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.6453 [2021-07-08 09:12:34,988][root][INFO] - ===> Epoch[5643](28215/5): Loss 0.0102, Sem 0.0102 LR: 5.645e-02 Score 99.628 Data time: 0.1813, Forward time: 1.2596, Backward time: 1.2385, DDP time: 0.0042, Total iter time: 2.7857 [2021-07-08 09:12:49,039][root][INFO] - ===> Epoch[5644](28220/5): Loss 0.0104, Sem 0.0104 LR: 5.644e-02 Score 99.619 Data time: 0.2351, Forward time: 1.2596, Backward time: 1.2386, DDP time: 0.0042, Total iter time: 2.7620 [2021-07-08 09:13:02,699][root][INFO] - ===> Epoch[5645](28225/5): Loss 0.0106, Sem 0.0106 LR: 5.643e-02 Score 99.619 Data time: 0.2609, Forward time: 1.2596, Backward time: 1.2386, DDP time: 0.0042, Total iter time: 2.6930 [2021-07-08 09:13:16,276][root][INFO] - ===> Epoch[5646](28230/5): Loss 0.0107, Sem 0.0107 LR: 5.643e-02 Score 99.612 Data time: 0.2224, Forward time: 1.2596, Backward time: 1.2386, DDP time: 0.0042, Total iter time: 2.6700 [2021-07-08 09:13:30,770][root][INFO] - ===> Epoch[5647](28235/5): Loss 0.0113, Sem 0.0113 LR: 5.642e-02 Score 99.602 Data time: 0.1656, Forward time: 1.2596, Backward time: 1.2386, DDP time: 0.0042, Total iter time: 2.8605 [2021-07-08 09:13:44,647][root][INFO] - ===> Epoch[5648](28240/5): Loss 0.0108, Sem 0.0108 LR: 5.641e-02 Score 99.609 Data time: 0.2221, Forward time: 1.2596, Backward time: 1.2386, DDP time: 0.0042, Total iter time: 2.7152 [2021-07-08 09:13:58,894][root][INFO] - ===> Epoch[5649](28245/5): Loss 0.0106, Sem 0.0106 LR: 5.640e-02 Score 99.617 Data time: 0.1887, Forward time: 1.2596, Backward time: 1.2386, DDP time: 0.0042, Total iter time: 2.8126 [2021-07-08 09:14:13,379][root][INFO] - ===> Epoch[5650](28250/5): Loss 0.0113, Sem 0.0113 LR: 5.639e-02 Score 99.597 Data time: 0.2452, Forward time: 1.2596, Backward time: 1.2386, DDP time: 0.0042, Total iter time: 2.8394 [2021-07-08 09:14:27,466][root][INFO] - ===> Epoch[5651](28255/5): Loss 0.0108, Sem 0.0108 LR: 5.639e-02 Score 99.610 Data time: 0.2305, Forward time: 1.2596, Backward time: 1.2386, DDP time: 0.0042, Total iter time: 2.7658 [2021-07-08 09:14:41,925][root][INFO] - ===> Epoch[5652](28260/5): Loss 0.0105, Sem 0.0105 LR: 5.638e-02 Score 99.622 Data time: 0.2263, Forward time: 1.2596, Backward time: 1.2386, DDP time: 0.0042, Total iter time: 2.8444 [2021-07-08 09:14:55,715][root][INFO] - ===> Epoch[5653](28265/5): Loss 0.0104, Sem 0.0104 LR: 5.637e-02 Score 99.622 Data time: 0.2468, Forward time: 1.2596, Backward time: 1.2386, DDP time: 0.0042, Total iter time: 2.7074 [2021-07-08 09:15:09,995][root][INFO] - ===> Epoch[5654](28270/5): Loss 0.0108, Sem 0.0108 LR: 5.636e-02 Score 99.606 Data time: 0.1946, Forward time: 1.2595, Backward time: 1.2387, DDP time: 0.0042, Total iter time: 2.8209 [2021-07-08 09:15:24,091][root][INFO] - ===> Epoch[5655](28275/5): Loss 0.0106, Sem 0.0106 LR: 5.635e-02 Score 99.619 Data time: 0.2332, Forward time: 1.2596, Backward time: 1.2387, DDP time: 0.0042, Total iter time: 2.7783 [2021-07-08 09:15:38,362][root][INFO] - ===> Epoch[5656](28280/5): Loss 0.0108, Sem 0.0108 LR: 5.635e-02 Score 99.609 Data time: 0.2405, Forward time: 1.2595, Backward time: 1.2387, DDP time: 0.0042, Total iter time: 2.8128 [2021-07-08 09:15:52,298][root][INFO] - ===> Epoch[5657](28285/5): Loss 0.0106, Sem 0.0106 LR: 5.634e-02 Score 99.620 Data time: 0.2305, Forward time: 1.2595, Backward time: 1.2387, DDP time: 0.0042, Total iter time: 2.7369 [2021-07-08 09:16:06,763][root][INFO] - ===> Epoch[5658](28290/5): Loss 0.0107, Sem 0.0107 LR: 5.633e-02 Score 99.614 Data time: 0.2035, Forward time: 1.2595, Backward time: 1.2387, DDP time: 0.0042, Total iter time: 2.8362 [2021-07-08 09:16:21,922][root][INFO] - ===> Epoch[5659](28295/5): Loss 0.0103, Sem 0.0103 LR: 5.632e-02 Score 99.624 Data time: 0.2268, Forward time: 1.2595, Backward time: 1.2387, DDP time: 0.0042, Total iter time: 2.9902 [2021-07-08 09:16:36,169][root][INFO] - ===> Epoch[5660](28300/5): Loss 0.0103, Sem 0.0103 LR: 5.631e-02 Score 99.630 Data time: 0.1862, Forward time: 1.2595, Backward time: 1.2387, DDP time: 0.0042, Total iter time: 2.8076 [2021-07-08 09:16:49,592][root][INFO] - ===> Epoch[5661](28305/5): Loss 0.0109, Sem 0.0109 LR: 5.631e-02 Score 99.606 Data time: 0.2300, Forward time: 1.2595, Backward time: 1.2387, DDP time: 0.0042, Total iter time: 2.6337 [2021-07-08 09:17:03,938][root][INFO] - ===> Epoch[5662](28310/5): Loss 0.0104, Sem 0.0104 LR: 5.630e-02 Score 99.622 Data time: 0.2518, Forward time: 1.2595, Backward time: 1.2387, DDP time: 0.0042, Total iter time: 2.8228 [2021-07-08 09:17:18,352][root][INFO] - ===> Epoch[5663](28315/5): Loss 0.0110, Sem 0.0110 LR: 5.629e-02 Score 99.606 Data time: 0.2548, Forward time: 1.2595, Backward time: 1.2387, DDP time: 0.0042, Total iter time: 2.8374 [2021-07-08 09:17:32,088][root][INFO] - ===> Epoch[5664](28320/5): Loss 0.0107, Sem 0.0107 LR: 5.628e-02 Score 99.613 Data time: 0.2273, Forward time: 1.2595, Backward time: 1.2387, DDP time: 0.0042, Total iter time: 2.7015 [2021-07-08 09:17:46,072][root][INFO] - ===> Epoch[5665](28325/5): Loss 0.0102, Sem 0.0102 LR: 5.627e-02 Score 99.631 Data time: 0.1931, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.7518 [2021-07-08 09:17:59,243][root][INFO] - ===> Epoch[5666](28330/5): Loss 0.0110, Sem 0.0110 LR: 5.627e-02 Score 99.606 Data time: 0.1996, Forward time: 1.2595, Backward time: 1.2387, DDP time: 0.0042, Total iter time: 2.5924 [2021-07-08 09:18:13,732][root][INFO] - ===> Epoch[5667](28335/5): Loss 0.0103, Sem 0.0103 LR: 5.626e-02 Score 99.627 Data time: 0.1961, Forward time: 1.2595, Backward time: 1.2387, DDP time: 0.0042, Total iter time: 2.8459 [2021-07-08 09:18:27,260][root][INFO] - ===> Epoch[5668](28340/5): Loss 0.0103, Sem 0.0103 LR: 5.625e-02 Score 99.632 Data time: 0.2474, Forward time: 1.2595, Backward time: 1.2387, DDP time: 0.0042, Total iter time: 2.6586 [2021-07-08 09:18:42,483][root][INFO] - ===> Epoch[5669](28345/5): Loss 0.0102, Sem 0.0102 LR: 5.624e-02 Score 99.630 Data time: 0.1901, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 3.0001 [2021-07-08 09:18:56,228][root][INFO] - ===> Epoch[5670](28350/5): Loss 0.0107, Sem 0.0107 LR: 5.623e-02 Score 99.613 Data time: 0.2277, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.6914 [2021-07-08 09:19:10,659][root][INFO] - ===> Epoch[5671](28355/5): Loss 0.0101, Sem 0.0101 LR: 5.623e-02 Score 99.636 Data time: 0.1856, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.8487 [2021-07-08 09:19:24,738][root][INFO] - ===> Epoch[5672](28360/5): Loss 0.0104, Sem 0.0104 LR: 5.622e-02 Score 99.625 Data time: 0.1925, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.7753 [2021-07-08 09:19:38,893][root][INFO] - ===> Epoch[5673](28365/5): Loss 0.0114, Sem 0.0114 LR: 5.621e-02 Score 99.591 Data time: 0.2549, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.7942 [2021-07-08 09:19:53,731][root][INFO] - ===> Epoch[5674](28370/5): Loss 0.0109, Sem 0.0109 LR: 5.620e-02 Score 99.609 Data time: 0.2321, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.9077 [2021-07-08 09:20:06,958][root][INFO] - ===> Epoch[5675](28375/5): Loss 0.0104, Sem 0.0104 LR: 5.619e-02 Score 99.622 Data time: 0.2271, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.5971 [2021-07-08 09:20:20,641][root][INFO] - ===> Epoch[5676](28380/5): Loss 0.0105, Sem 0.0105 LR: 5.619e-02 Score 99.619 Data time: 0.2322, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.6909 [2021-07-08 09:20:36,024][root][INFO] - ===> Epoch[5677](28385/5): Loss 0.0104, Sem 0.0104 LR: 5.618e-02 Score 99.624 Data time: 0.1953, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 3.0368 [2021-07-08 09:20:49,656][root][INFO] - ===> Epoch[5678](28390/5): Loss 0.0105, Sem 0.0105 LR: 5.617e-02 Score 99.621 Data time: 0.1927, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.6855 [2021-07-08 09:21:03,675][root][INFO] - ===> Epoch[5679](28395/5): Loss 0.0108, Sem 0.0108 LR: 5.616e-02 Score 99.606 Data time: 0.2304, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.7593 [2021-07-08 09:21:17,474][root][INFO] - ===> Epoch[5680](28400/5): Loss 0.0103, Sem 0.0103 LR: 5.615e-02 Score 99.627 Data time: 0.1891, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.7145 [2021-07-08 09:21:17,475][root][INFO] - ===> Start testing [2021-07-08 09:22:02,846][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1234 Loss 0.351 (AVG: 0.545) Score 92.714 (AVG: 91.166) mIOU 71.573 mAP 84.052 mAcc 77.770 IOU: 65.150 0.000 85.221 76.304 94.737 92.032 33.159 84.786 98.388 75.189 78.445 86.858 60.182 mAP: 83.975 5.686 92.363 87.244 98.831 98.527 58.806 98.619 99.728 90.924 91.550 94.256 92.168 mAcc: 82.379 0.000 88.681 82.846 95.924 97.110 37.770 96.308 99.467 81.294 89.332 95.992 63.906 [2021-07-08 09:22:02,846][root][INFO] - Finished test. Elapsed time: 45.3697 [2021-07-08 09:22:03,877][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 09:22:17,301][root][INFO] - ===> Epoch[5681](28405/5): Loss 0.0103, Sem 0.0103 LR: 5.615e-02 Score 99.631 Data time: 0.2027, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.6373 [2021-07-08 09:22:31,433][root][INFO] - ===> Epoch[5682](28410/5): Loss 0.0103, Sem 0.0103 LR: 5.614e-02 Score 99.627 Data time: 0.2023, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.7769 [2021-07-08 09:22:44,692][root][INFO] - ===> Epoch[5683](28415/5): Loss 0.0101, Sem 0.0101 LR: 5.613e-02 Score 99.635 Data time: 0.1860, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.6119 [2021-07-08 09:22:58,565][root][INFO] - ===> Epoch[5684](28420/5): Loss 0.0102, Sem 0.0102 LR: 5.612e-02 Score 99.631 Data time: 0.2150, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.7272 [2021-07-08 09:23:12,247][root][INFO] - ===> Epoch[5685](28425/5): Loss 0.0102, Sem 0.0102 LR: 5.611e-02 Score 99.635 Data time: 0.2221, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.6898 [2021-07-08 09:23:26,265][root][INFO] - ===> Epoch[5686](28430/5): Loss 0.0102, Sem 0.0102 LR: 5.611e-02 Score 99.632 Data time: 0.2182, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.7563 [2021-07-08 09:23:40,680][root][INFO] - ===> Epoch[5687](28435/5): Loss 0.0100, Sem 0.0100 LR: 5.610e-02 Score 99.641 Data time: 0.2393, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.8368 [2021-07-08 09:23:54,674][root][INFO] - ===> Epoch[5688](28440/5): Loss 0.0108, Sem 0.0108 LR: 5.609e-02 Score 99.609 Data time: 0.2605, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.7607 [2021-07-08 09:24:07,791][root][INFO] - ===> Epoch[5689](28445/5): Loss 0.0104, Sem 0.0104 LR: 5.608e-02 Score 99.622 Data time: 0.2194, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.5789 [2021-07-08 09:24:22,542][root][INFO] - ===> Epoch[5690](28450/5): Loss 0.0098, Sem 0.0098 LR: 5.607e-02 Score 99.648 Data time: 0.2397, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.9013 [2021-07-08 09:24:35,971][root][INFO] - ===> Epoch[5691](28455/5): Loss 0.0104, Sem 0.0104 LR: 5.607e-02 Score 99.623 Data time: 0.2297, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.6425 [2021-07-08 09:24:49,613][root][INFO] - ===> Epoch[5692](28460/5): Loss 0.0108, Sem 0.0108 LR: 5.606e-02 Score 99.610 Data time: 0.2449, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.6769 [2021-07-08 09:25:03,824][root][INFO] - ===> Epoch[5693](28465/5): Loss 0.0102, Sem 0.0102 LR: 5.605e-02 Score 99.633 Data time: 0.2364, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.7912 [2021-07-08 09:25:18,298][root][INFO] - ===> Epoch[5694](28470/5): Loss 0.0104, Sem 0.0104 LR: 5.604e-02 Score 99.627 Data time: 0.2092, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.8538 [2021-07-08 09:25:32,087][root][INFO] - ===> Epoch[5695](28475/5): Loss 0.0104, Sem 0.0104 LR: 5.603e-02 Score 99.624 Data time: 0.2136, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.7016 [2021-07-08 09:25:46,240][root][INFO] - ===> Epoch[5696](28480/5): Loss 0.0105, Sem 0.0105 LR: 5.603e-02 Score 99.623 Data time: 0.2212, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.7921 [2021-07-08 09:26:00,443][root][INFO] - ===> Epoch[5697](28485/5): Loss 0.0103, Sem 0.0103 LR: 5.602e-02 Score 99.627 Data time: 0.1841, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.8001 [2021-07-08 09:26:14,638][root][INFO] - ===> Epoch[5698](28490/5): Loss 0.0102, Sem 0.0102 LR: 5.601e-02 Score 99.631 Data time: 0.2236, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.7985 [2021-07-08 09:26:28,142][root][INFO] - ===> Epoch[5699](28495/5): Loss 0.0103, Sem 0.0103 LR: 5.600e-02 Score 99.625 Data time: 0.2076, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.6555 [2021-07-08 09:26:41,779][root][INFO] - ===> Epoch[5700](28500/5): Loss 0.0101, Sem 0.0101 LR: 5.600e-02 Score 99.635 Data time: 0.2645, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.6847 [2021-07-08 09:26:55,442][root][INFO] - ===> Epoch[5701](28505/5): Loss 0.0101, Sem 0.0101 LR: 5.599e-02 Score 99.637 Data time: 0.2155, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.6857 [2021-07-08 09:27:08,434][root][INFO] - ===> Epoch[5702](28510/5): Loss 0.0103, Sem 0.0103 LR: 5.598e-02 Score 99.627 Data time: 0.2490, Forward time: 1.2595, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.5575 [2021-07-08 09:27:22,714][root][INFO] - ===> Epoch[5703](28515/5): Loss 0.0102, Sem 0.0102 LR: 5.597e-02 Score 99.630 Data time: 0.2848, Forward time: 1.2594, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.8177 [2021-07-08 09:27:36,284][root][INFO] - ===> Epoch[5704](28520/5): Loss 0.0105, Sem 0.0105 LR: 5.596e-02 Score 99.624 Data time: 0.2048, Forward time: 1.2594, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.6681 [2021-07-08 09:27:50,456][root][INFO] - ===> Epoch[5705](28525/5): Loss 0.0106, Sem 0.0106 LR: 5.596e-02 Score 99.619 Data time: 0.2120, Forward time: 1.2594, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.8004 [2021-07-08 09:28:04,912][root][INFO] - ===> Epoch[5706](28530/5): Loss 0.0099, Sem 0.0099 LR: 5.595e-02 Score 99.643 Data time: 0.2045, Forward time: 1.2594, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.8379 [2021-07-08 09:28:18,766][root][INFO] - ===> Epoch[5707](28535/5): Loss 0.0103, Sem 0.0103 LR: 5.594e-02 Score 99.626 Data time: 0.1656, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7321 [2021-07-08 09:28:32,676][root][INFO] - ===> Epoch[5708](28540/5): Loss 0.0103, Sem 0.0103 LR: 5.593e-02 Score 99.627 Data time: 0.1852, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7402 [2021-07-08 09:28:46,606][root][INFO] - ===> Epoch[5709](28545/5): Loss 0.0104, Sem 0.0104 LR: 5.592e-02 Score 99.623 Data time: 0.2015, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7351 [2021-07-08 09:29:00,156][root][INFO] - ===> Epoch[5710](28550/5): Loss 0.0100, Sem 0.0100 LR: 5.592e-02 Score 99.639 Data time: 0.2653, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6586 [2021-07-08 09:29:14,488][root][INFO] - ===> Epoch[5711](28555/5): Loss 0.0100, Sem 0.0100 LR: 5.591e-02 Score 99.639 Data time: 0.2473, Forward time: 1.2594, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.8115 [2021-07-08 09:29:27,918][root][INFO] - ===> Epoch[5712](28560/5): Loss 0.0103, Sem 0.0103 LR: 5.590e-02 Score 99.628 Data time: 0.1773, Forward time: 1.2594, Backward time: 1.2388, DDP time: 0.0042, Total iter time: 2.6437 [2021-07-08 09:29:42,727][root][INFO] - ===> Epoch[5713](28565/5): Loss 0.0105, Sem 0.0105 LR: 5.589e-02 Score 99.619 Data time: 0.1765, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.9273 [2021-07-08 09:29:56,088][root][INFO] - ===> Epoch[5714](28570/5): Loss 0.0100, Sem 0.0100 LR: 5.588e-02 Score 99.638 Data time: 0.1924, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6196 [2021-07-08 09:30:10,488][root][INFO] - ===> Epoch[5715](28575/5): Loss 0.0096, Sem 0.0096 LR: 5.588e-02 Score 99.653 Data time: 0.1706, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.8392 [2021-07-08 09:30:24,496][root][INFO] - ===> Epoch[5716](28580/5): Loss 0.0105, Sem 0.0105 LR: 5.587e-02 Score 99.620 Data time: 0.2139, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7594 [2021-07-08 09:30:37,976][root][INFO] - ===> Epoch[5717](28585/5): Loss 0.0105, Sem 0.0105 LR: 5.586e-02 Score 99.618 Data time: 0.2524, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6505 [2021-07-08 09:30:52,358][root][INFO] - ===> Epoch[5718](28590/5): Loss 0.0103, Sem 0.0103 LR: 5.585e-02 Score 99.627 Data time: 0.2511, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.8251 [2021-07-08 09:31:07,099][root][INFO] - ===> Epoch[5719](28595/5): Loss 0.0100, Sem 0.0100 LR: 5.584e-02 Score 99.639 Data time: 0.2228, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.9074 [2021-07-08 09:31:20,844][root][INFO] - ===> Epoch[5720](28600/5): Loss 0.0105, Sem 0.0105 LR: 5.583e-02 Score 99.620 Data time: 0.2919, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6959 [2021-07-08 09:31:20,846][root][INFO] - ===> Start testing [2021-07-08 09:32:00,444][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1325 Loss 0.440 (AVG: 0.570) Score 91.422 (AVG: 90.994) mIOU 70.836 mAP 83.375 mAcc 76.917 IOU: 64.460 0.000 84.641 76.565 94.786 92.161 26.123 83.198 98.294 76.456 78.771 86.380 59.027 mAP: 83.925 5.355 92.952 87.961 98.735 98.523 53.517 98.424 99.766 88.083 91.466 93.814 91.350 mAcc: 80.181 0.000 89.197 84.002 96.201 97.060 28.255 95.812 99.267 81.442 89.194 96.374 62.935 [2021-07-08 09:32:00,444][root][INFO] - Finished test. Elapsed time: 39.5979 [2021-07-08 09:32:01,532][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 09:32:14,228][root][INFO] - ===> Epoch[5721](28605/5): Loss 0.0099, Sem 0.0099 LR: 5.583e-02 Score 99.644 Data time: 0.2068, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.4955 [2021-07-08 09:32:28,208][root][INFO] - ===> Epoch[5722](28610/5): Loss 0.0101, Sem 0.0101 LR: 5.582e-02 Score 99.634 Data time: 0.2325, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7460 [2021-07-08 09:32:43,642][root][INFO] - ===> Epoch[5723](28615/5): Loss 0.0099, Sem 0.0099 LR: 5.581e-02 Score 99.640 Data time: 0.2465, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 3.0439 [2021-07-08 09:32:57,022][root][INFO] - ===> Epoch[5724](28620/5): Loss 0.0098, Sem 0.0098 LR: 5.580e-02 Score 99.646 Data time: 0.2459, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6327 [2021-07-08 09:33:10,976][root][INFO] - ===> Epoch[5725](28625/5): Loss 0.0104, Sem 0.0104 LR: 5.579e-02 Score 99.626 Data time: 0.1930, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7470 [2021-07-08 09:33:25,054][root][INFO] - ===> Epoch[5726](28630/5): Loss 0.0107, Sem 0.0107 LR: 5.579e-02 Score 99.622 Data time: 0.2444, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7640 [2021-07-08 09:33:38,810][root][INFO] - ===> Epoch[5727](28635/5): Loss 0.0102, Sem 0.0102 LR: 5.578e-02 Score 99.632 Data time: 0.2155, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7034 [2021-07-08 09:33:52,789][root][INFO] - ===> Epoch[5728](28640/5): Loss 0.0103, Sem 0.0103 LR: 5.577e-02 Score 99.628 Data time: 0.2176, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7503 [2021-07-08 09:34:07,526][root][INFO] - ===> Epoch[5729](28645/5): Loss 0.0103, Sem 0.0103 LR: 5.576e-02 Score 99.625 Data time: 0.2302, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.9054 [2021-07-08 09:34:22,021][root][INFO] - ===> Epoch[5730](28650/5): Loss 0.0104, Sem 0.0104 LR: 5.575e-02 Score 99.624 Data time: 0.2324, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.8491 [2021-07-08 09:34:35,547][root][INFO] - ===> Epoch[5731](28655/5): Loss 0.0104, Sem 0.0104 LR: 5.575e-02 Score 99.628 Data time: 0.2270, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6654 [2021-07-08 09:34:49,849][root][INFO] - ===> Epoch[5732](28660/5): Loss 0.0103, Sem 0.0103 LR: 5.574e-02 Score 99.628 Data time: 0.2636, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.8111 [2021-07-08 09:35:03,438][root][INFO] - ===> Epoch[5733](28665/5): Loss 0.0102, Sem 0.0102 LR: 5.573e-02 Score 99.631 Data time: 0.2290, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6722 [2021-07-08 09:35:18,040][root][INFO] - ===> Epoch[5734](28670/5): Loss 0.0100, Sem 0.0100 LR: 5.572e-02 Score 99.641 Data time: 0.2305, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.8741 [2021-07-08 09:35:31,719][root][INFO] - ===> Epoch[5735](28675/5): Loss 0.0105, Sem 0.0105 LR: 5.571e-02 Score 99.617 Data time: 0.1877, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6911 [2021-07-08 09:35:46,353][root][INFO] - ===> Epoch[5736](28680/5): Loss 0.0104, Sem 0.0104 LR: 5.571e-02 Score 99.627 Data time: 0.2423, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.8747 [2021-07-08 09:36:00,836][root][INFO] - ===> Epoch[5737](28685/5): Loss 0.0099, Sem 0.0099 LR: 5.570e-02 Score 99.642 Data time: 0.2262, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.8542 [2021-07-08 09:36:14,882][root][INFO] - ===> Epoch[5738](28690/5): Loss 0.0102, Sem 0.0102 LR: 5.569e-02 Score 99.632 Data time: 0.2213, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7699 [2021-07-08 09:36:28,291][root][INFO] - ===> Epoch[5739](28695/5): Loss 0.0102, Sem 0.0102 LR: 5.568e-02 Score 99.632 Data time: 0.2553, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6301 [2021-07-08 09:36:41,994][root][INFO] - ===> Epoch[5740](28700/5): Loss 0.0104, Sem 0.0104 LR: 5.567e-02 Score 99.623 Data time: 0.1728, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7078 [2021-07-08 09:36:55,295][root][INFO] - ===> Epoch[5741](28705/5): Loss 0.0103, Sem 0.0103 LR: 5.567e-02 Score 99.625 Data time: 0.2385, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6146 [2021-07-08 09:37:10,087][root][INFO] - ===> Epoch[5742](28710/5): Loss 0.0097, Sem 0.0097 LR: 5.566e-02 Score 99.651 Data time: 0.1667, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.9070 [2021-07-08 09:37:23,849][root][INFO] - ===> Epoch[5743](28715/5): Loss 0.0108, Sem 0.0108 LR: 5.565e-02 Score 99.612 Data time: 0.2498, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6943 [2021-07-08 09:37:37,733][root][INFO] - ===> Epoch[5744](28720/5): Loss 0.0097, Sem 0.0097 LR: 5.564e-02 Score 99.649 Data time: 0.2005, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7424 [2021-07-08 09:37:51,548][root][INFO] - ===> Epoch[5745](28725/5): Loss 0.0104, Sem 0.0104 LR: 5.563e-02 Score 99.625 Data time: 0.2097, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7249 [2021-07-08 09:38:05,366][root][INFO] - ===> Epoch[5746](28730/5): Loss 0.0105, Sem 0.0105 LR: 5.563e-02 Score 99.618 Data time: 0.1492, Forward time: 1.2593, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7300 [2021-07-08 09:38:18,791][root][INFO] - ===> Epoch[5747](28735/5): Loss 0.0099, Sem 0.0099 LR: 5.562e-02 Score 99.640 Data time: 0.2521, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6398 [2021-07-08 09:38:33,957][root][INFO] - ===> Epoch[5748](28740/5): Loss 0.0101, Sem 0.0101 LR: 5.561e-02 Score 99.641 Data time: 0.2351, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.9919 [2021-07-08 09:38:48,465][root][INFO] - ===> Epoch[5749](28745/5): Loss 0.0104, Sem 0.0104 LR: 5.560e-02 Score 99.624 Data time: 0.2206, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.8526 [2021-07-08 09:39:02,602][root][INFO] - ===> Epoch[5750](28750/5): Loss 0.0099, Sem 0.0099 LR: 5.559e-02 Score 99.641 Data time: 0.2801, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7779 [2021-07-08 09:39:16,851][root][INFO] - ===> Epoch[5751](28755/5): Loss 0.0106, Sem 0.0106 LR: 5.559e-02 Score 99.619 Data time: 0.2708, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.8129 [2021-07-08 09:39:30,569][root][INFO] - ===> Epoch[5752](28760/5): Loss 0.0101, Sem 0.0101 LR: 5.558e-02 Score 99.636 Data time: 0.1781, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7066 [2021-07-08 09:39:44,236][root][INFO] - ===> Epoch[5753](28765/5): Loss 0.0103, Sem 0.0103 LR: 5.557e-02 Score 99.627 Data time: 0.2937, Forward time: 1.2594, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6883 [2021-07-08 09:39:57,697][root][INFO] - ===> Epoch[5754](28770/5): Loss 0.0100, Sem 0.0100 LR: 5.556e-02 Score 99.640 Data time: 0.2140, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6439 [2021-07-08 09:40:11,431][root][INFO] - ===> Epoch[5755](28775/5): Loss 0.0103, Sem 0.0103 LR: 5.555e-02 Score 99.630 Data time: 0.2174, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6916 [2021-07-08 09:40:24,985][root][INFO] - ===> Epoch[5756](28780/5): Loss 0.0100, Sem 0.0100 LR: 5.555e-02 Score 99.640 Data time: 0.2426, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6687 [2021-07-08 09:40:39,305][root][INFO] - ===> Epoch[5757](28785/5): Loss 0.0101, Sem 0.0101 LR: 5.554e-02 Score 99.640 Data time: 0.2407, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.8158 [2021-07-08 09:40:52,687][root][INFO] - ===> Epoch[5758](28790/5): Loss 0.0109, Sem 0.0109 LR: 5.553e-02 Score 99.613 Data time: 0.2598, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6330 [2021-07-08 09:41:06,784][root][INFO] - ===> Epoch[5759](28795/5): Loss 0.0105, Sem 0.0105 LR: 5.552e-02 Score 99.624 Data time: 0.2346, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7755 [2021-07-08 09:41:20,930][root][INFO] - ===> Epoch[5760](28800/5): Loss 0.0112, Sem 0.0112 LR: 5.551e-02 Score 99.594 Data time: 0.2446, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7784 [2021-07-08 09:41:20,934][root][INFO] - ===> Start testing [2021-07-08 09:41:54,982][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1201 Loss 0.371 (AVG: 0.570) Score 92.045 (AVG: 90.852) mIOU 71.444 mAP 84.045 mAcc 78.489 IOU: 63.321 0.000 83.737 74.717 94.613 92.381 42.703 82.517 98.356 72.998 78.098 86.899 58.433 mAP: 83.428 5.867 90.455 86.608 98.776 98.510 66.332 98.158 99.720 88.146 92.108 93.363 91.113 mAcc: 78.848 0.000 87.607 84.477 96.101 97.698 50.607 95.249 99.393 82.547 90.757 94.649 62.419 [2021-07-08 09:41:54,982][root][INFO] - Finished test. Elapsed time: 34.0474 [2021-07-08 09:41:56,032][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 09:42:09,249][root][INFO] - ===> Epoch[5761](28805/5): Loss 0.0114, Sem 0.0114 LR: 5.551e-02 Score 99.588 Data time: 0.1779, Forward time: 1.2592, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.6053 [2021-07-08 09:42:23,523][root][INFO] - ===> Epoch[5762](28810/5): Loss 0.0110, Sem 0.0110 LR: 5.550e-02 Score 99.601 Data time: 0.1925, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.8055 [2021-07-08 09:42:37,915][root][INFO] - ===> Epoch[5763](28815/5): Loss 0.0107, Sem 0.0107 LR: 5.549e-02 Score 99.617 Data time: 0.2354, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.8364 [2021-07-08 09:42:50,911][root][INFO] - ===> Epoch[5764](28820/5): Loss 0.0112, Sem 0.0112 LR: 5.548e-02 Score 99.595 Data time: 0.1780, Forward time: 1.2592, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.5492 [2021-07-08 09:43:05,371][root][INFO] - ===> Epoch[5765](28825/5): Loss 0.0111, Sem 0.0111 LR: 5.547e-02 Score 99.604 Data time: 0.2475, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.8435 [2021-07-08 09:43:19,388][root][INFO] - ===> Epoch[5766](28830/5): Loss 0.0107, Sem 0.0107 LR: 5.547e-02 Score 99.615 Data time: 0.2055, Forward time: 1.2593, Backward time: 1.2389, DDP time: 0.0042, Total iter time: 2.7600 [2021-07-08 09:43:33,520][root][INFO] - ===> Epoch[5767](28835/5): Loss 0.0104, Sem 0.0104 LR: 5.546e-02 Score 99.629 Data time: 0.1511, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7858 [2021-07-08 09:43:46,827][root][INFO] - ===> Epoch[5768](28840/5): Loss 0.0104, Sem 0.0104 LR: 5.545e-02 Score 99.624 Data time: 0.1900, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.6194 [2021-07-08 09:44:01,594][root][INFO] - ===> Epoch[5769](28845/5): Loss 0.0106, Sem 0.0106 LR: 5.544e-02 Score 99.618 Data time: 0.2164, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.9112 [2021-07-08 09:44:15,795][root][INFO] - ===> Epoch[5770](28850/5): Loss 0.0101, Sem 0.0101 LR: 5.543e-02 Score 99.632 Data time: 0.1958, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.8013 [2021-07-08 09:44:29,863][root][INFO] - ===> Epoch[5771](28855/5): Loss 0.0106, Sem 0.0106 LR: 5.543e-02 Score 99.616 Data time: 0.2054, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7678 [2021-07-08 09:44:43,817][root][INFO] - ===> Epoch[5772](28860/5): Loss 0.0102, Sem 0.0102 LR: 5.542e-02 Score 99.635 Data time: 0.2098, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7443 [2021-07-08 09:44:58,659][root][INFO] - ===> Epoch[5773](28865/5): Loss 0.0102, Sem 0.0102 LR: 5.541e-02 Score 99.631 Data time: 0.2605, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.9209 [2021-07-08 09:45:12,344][root][INFO] - ===> Epoch[5774](28870/5): Loss 0.0100, Sem 0.0100 LR: 5.540e-02 Score 99.639 Data time: 0.1837, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.6867 [2021-07-08 09:45:26,092][root][INFO] - ===> Epoch[5775](28875/5): Loss 0.0099, Sem 0.0099 LR: 5.539e-02 Score 99.641 Data time: 0.2271, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7107 [2021-07-08 09:45:39,378][root][INFO] - ===> Epoch[5776](28880/5): Loss 0.0102, Sem 0.0102 LR: 5.539e-02 Score 99.628 Data time: 0.2229, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.6142 [2021-07-08 09:45:53,193][root][INFO] - ===> Epoch[5777](28885/5): Loss 0.0103, Sem 0.0103 LR: 5.538e-02 Score 99.633 Data time: 0.2166, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7152 [2021-07-08 09:46:07,045][root][INFO] - ===> Epoch[5778](28890/5): Loss 0.0104, Sem 0.0104 LR: 5.537e-02 Score 99.626 Data time: 0.2185, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7311 [2021-07-08 09:46:21,333][root][INFO] - ===> Epoch[5779](28895/5): Loss 0.0131, Sem 0.0131 LR: 5.536e-02 Score 99.557 Data time: 0.3347, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.8109 [2021-07-08 09:46:35,293][root][INFO] - ===> Epoch[5780](28900/5): Loss 0.0178, Sem 0.0178 LR: 5.535e-02 Score 99.411 Data time: 0.2473, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7478 [2021-07-08 09:46:49,145][root][INFO] - ===> Epoch[5781](28905/5): Loss 0.0193, Sem 0.0193 LR: 5.535e-02 Score 99.335 Data time: 0.2911, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7197 [2021-07-08 09:47:04,055][root][INFO] - ===> Epoch[5782](28910/5): Loss 0.0182, Sem 0.0182 LR: 5.534e-02 Score 99.384 Data time: 0.2468, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.9312 [2021-07-08 09:47:17,777][root][INFO] - ===> Epoch[5783](28915/5): Loss 0.0195, Sem 0.0195 LR: 5.533e-02 Score 99.326 Data time: 0.2288, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7035 [2021-07-08 09:47:32,277][root][INFO] - ===> Epoch[5784](28920/5): Loss 0.0205, Sem 0.0205 LR: 5.532e-02 Score 99.284 Data time: 0.2403, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.8614 [2021-07-08 09:47:45,358][root][INFO] - ===> Epoch[5785](28925/5): Loss 0.0205, Sem 0.0205 LR: 5.531e-02 Score 99.296 Data time: 0.2520, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.5661 [2021-07-08 09:47:59,602][root][INFO] - ===> Epoch[5786](28930/5): Loss 0.0174, Sem 0.0174 LR: 5.531e-02 Score 99.386 Data time: 0.2366, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7955 [2021-07-08 09:48:13,107][root][INFO] - ===> Epoch[5787](28935/5): Loss 0.0190, Sem 0.0190 LR: 5.530e-02 Score 99.338 Data time: 0.2383, Forward time: 1.2592, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.6415 [2021-07-08 09:48:27,061][root][INFO] - ===> Epoch[5788](28940/5): Loss 0.0166, Sem 0.0166 LR: 5.529e-02 Score 99.406 Data time: 0.2038, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7458 [2021-07-08 09:48:41,915][root][INFO] - ===> Epoch[5789](28945/5): Loss 0.0165, Sem 0.0165 LR: 5.528e-02 Score 99.423 Data time: 0.2136, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.9251 [2021-07-08 09:48:55,391][root][INFO] - ===> Epoch[5790](28950/5): Loss 0.0153, Sem 0.0153 LR: 5.527e-02 Score 99.458 Data time: 0.2561, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.6516 [2021-07-08 09:49:09,495][root][INFO] - ===> Epoch[5791](28955/5): Loss 0.0168, Sem 0.0168 LR: 5.527e-02 Score 99.427 Data time: 0.1875, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7794 [2021-07-08 09:49:22,906][root][INFO] - ===> Epoch[5792](28960/5): Loss 0.0172, Sem 0.0172 LR: 5.526e-02 Score 99.417 Data time: 0.2258, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.6438 [2021-07-08 09:49:37,892][root][INFO] - ===> Epoch[5793](28965/5): Loss 0.0168, Sem 0.0168 LR: 5.525e-02 Score 99.413 Data time: 0.1961, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.9529 [2021-07-08 09:49:51,848][root][INFO] - ===> Epoch[5794](28970/5): Loss 0.0162, Sem 0.0162 LR: 5.524e-02 Score 99.425 Data time: 0.2325, Forward time: 1.2591, Backward time: 1.2391, DDP time: 0.0042, Total iter time: 2.7454 [2021-07-08 09:50:05,650][root][INFO] - ===> Epoch[5795](28975/5): Loss 0.0178, Sem 0.0178 LR: 5.523e-02 Score 99.383 Data time: 0.2427, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7095 [2021-07-08 09:50:19,761][root][INFO] - ===> Epoch[5796](28980/5): Loss 0.0179, Sem 0.0179 LR: 5.523e-02 Score 99.383 Data time: 0.1588, Forward time: 1.2591, Backward time: 1.2391, DDP time: 0.0042, Total iter time: 2.7802 [2021-07-08 09:50:33,463][root][INFO] - ===> Epoch[5797](28985/5): Loss 0.0179, Sem 0.0179 LR: 5.522e-02 Score 99.387 Data time: 0.2514, Forward time: 1.2591, Backward time: 1.2391, DDP time: 0.0042, Total iter time: 2.6863 [2021-07-08 09:50:46,688][root][INFO] - ===> Epoch[5798](28990/5): Loss 0.0173, Sem 0.0173 LR: 5.521e-02 Score 99.388 Data time: 0.2103, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.6016 [2021-07-08 09:51:00,563][root][INFO] - ===> Epoch[5799](28995/5): Loss 0.0182, Sem 0.0182 LR: 5.520e-02 Score 99.371 Data time: 0.2595, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7246 [2021-07-08 09:51:14,793][root][INFO] - ===> Epoch[5800](29000/5): Loss 0.0181, Sem 0.0181 LR: 5.519e-02 Score 99.370 Data time: 0.2356, Forward time: 1.2591, Backward time: 1.2391, DDP time: 0.0042, Total iter time: 2.8004 [2021-07-08 09:51:14,794][root][INFO] - ===> Start testing [2021-07-08 09:51:55,234][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1269 Loss 0.587 (AVG: 0.517) Score 87.972 (AVG: 90.828) mIOU 70.590 mAP 84.127 mAcc 78.257 IOU: 64.185 0.000 81.312 76.524 94.843 91.366 30.726 80.522 98.165 76.948 78.304 86.294 58.476 mAP: 84.015 4.591 96.463 91.739 98.723 98.562 55.341 98.058 99.716 89.160 92.082 93.877 91.324 mAcc: 77.169 0.000 92.822 85.113 96.220 97.431 35.759 96.933 98.911 87.569 91.527 95.476 62.405 [2021-07-08 09:51:55,234][root][INFO] - Finished test. Elapsed time: 40.4389 [2021-07-08 09:51:56,301][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 09:52:09,529][root][INFO] - ===> Epoch[5801](29005/5): Loss 0.0162, Sem 0.0162 LR: 5.519e-02 Score 99.422 Data time: 0.1778, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.6064 [2021-07-08 09:52:23,187][root][INFO] - ===> Epoch[5802](29010/5): Loss 0.0165, Sem 0.0165 LR: 5.518e-02 Score 99.409 Data time: 0.2083, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.6766 [2021-07-08 09:52:36,696][root][INFO] - ===> Epoch[5803](29015/5): Loss 0.0143, Sem 0.0143 LR: 5.517e-02 Score 99.486 Data time: 0.2083, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.6600 [2021-07-08 09:52:51,013][root][INFO] - ===> Epoch[5804](29020/5): Loss 0.0155, Sem 0.0155 LR: 5.516e-02 Score 99.443 Data time: 0.2636, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.8148 [2021-07-08 09:53:05,037][root][INFO] - ===> Epoch[5805](29025/5): Loss 0.0139, Sem 0.0139 LR: 5.515e-02 Score 99.507 Data time: 0.2805, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7520 [2021-07-08 09:53:18,472][root][INFO] - ===> Epoch[5806](29030/5): Loss 0.0153, Sem 0.0153 LR: 5.515e-02 Score 99.468 Data time: 0.1956, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.6518 [2021-07-08 09:53:32,209][root][INFO] - ===> Epoch[5807](29035/5): Loss 0.0134, Sem 0.0134 LR: 5.514e-02 Score 99.521 Data time: 0.2162, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7024 [2021-07-08 09:53:47,617][root][INFO] - ===> Epoch[5808](29040/5): Loss 0.0130, Sem 0.0130 LR: 5.513e-02 Score 99.529 Data time: 0.1816, Forward time: 1.2591, Backward time: 1.2391, DDP time: 0.0042, Total iter time: 3.0450 [2021-07-08 09:54:01,791][root][INFO] - ===> Epoch[5809](29045/5): Loss 0.0130, Sem 0.0130 LR: 5.512e-02 Score 99.530 Data time: 0.2164, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7851 [2021-07-08 09:54:16,777][root][INFO] - ===> Epoch[5810](29050/5): Loss 0.0126, Sem 0.0126 LR: 5.511e-02 Score 99.548 Data time: 0.3077, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.9516 [2021-07-08 09:54:31,009][root][INFO] - ===> Epoch[5811](29055/5): Loss 0.0132, Sem 0.0132 LR: 5.511e-02 Score 99.532 Data time: 0.2346, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.7983 [2021-07-08 09:54:44,917][root][INFO] - ===> Epoch[5812](29060/5): Loss 0.0128, Sem 0.0128 LR: 5.510e-02 Score 99.539 Data time: 0.1754, Forward time: 1.2591, Backward time: 1.2391, DDP time: 0.0042, Total iter time: 2.7523 [2021-07-08 09:54:58,507][root][INFO] - ===> Epoch[5813](29065/5): Loss 0.0128, Sem 0.0128 LR: 5.509e-02 Score 99.547 Data time: 0.2298, Forward time: 1.2591, Backward time: 1.2390, DDP time: 0.0042, Total iter time: 2.6696 [2021-07-08 09:55:13,521][root][INFO] - ===> Epoch[5814](29070/5): Loss 0.0127, Sem 0.0127 LR: 5.508e-02 Score 99.560 Data time: 0.2153, Forward time: 1.2591, Backward time: 1.2391, DDP time: 0.0042, Total iter time: 2.9566 [2021-07-08 09:55:28,144][root][INFO] - ===> Epoch[5815](29075/5): Loss 0.0131, Sem 0.0131 LR: 5.507e-02 Score 99.538 Data time: 0.2115, Forward time: 1.2591, Backward time: 1.2391, DDP time: 0.0042, Total iter time: 2.8818 [2021-07-08 09:55:41,930][root][INFO] - ===> Epoch[5816](29080/5): Loss 0.0133, Sem 0.0133 LR: 5.507e-02 Score 99.524 Data time: 0.1908, Forward time: 1.2591, Backward time: 1.2391, DDP time: 0.0042, Total iter time: 2.7074 [2021-07-08 09:55:56,079][root][INFO] - ===> Epoch[5817](29085/5): Loss 0.0133, Sem 0.0133 LR: 5.506e-02 Score 99.530 Data time: 0.2627, Forward time: 1.2591, Backward time: 1.2391, DDP time: 0.0042, Total iter time: 2.7797 [2021-07-08 09:56:09,517][root][INFO] - ===> Epoch[5818](29090/5): Loss 0.0143, Sem 0.0143 LR: 5.505e-02 Score 99.484 Data time: 0.1521, Forward time: 1.2590, Backward time: 1.2391, DDP time: 0.0042, Total iter time: 2.6461 [2021-07-08 09:56:23,201][root][INFO] - ===> Epoch[5819](29095/5): Loss 0.0136, Sem 0.0136 LR: 5.504e-02 Score 99.519 Data time: 0.2048, Forward time: 1.2590, Backward time: 1.2391, DDP time: 0.0042, Total iter time: 2.6867 [2021-07-08 09:56:37,332][root][INFO] - ===> Epoch[5820](29100/5): Loss 0.0123, Sem 0.0123 LR: 5.503e-02 Score 99.559 Data time: 0.2253, Forward time: 1.2590, Backward time: 1.2391, DDP time: 0.0042, Total iter time: 2.7811 [2021-07-08 09:56:51,484][root][INFO] - ===> Epoch[5821](29105/5): Loss 0.0126, Sem 0.0126 LR: 5.503e-02 Score 99.544 Data time: 0.1598, Forward time: 1.2590, Backward time: 1.2392, DDP time: 0.0042, Total iter time: 2.7929 [2021-07-08 09:57:04,802][root][INFO] - ===> Epoch[5822](29110/5): Loss 0.0125, Sem 0.0125 LR: 5.502e-02 Score 99.551 Data time: 0.2704, Forward time: 1.2590, Backward time: 1.2391, DDP time: 0.0042, Total iter time: 2.6221 [2021-07-08 09:57:19,277][root][INFO] - ===> Epoch[5823](29115/5): Loss 0.0120, Sem 0.0120 LR: 5.501e-02 Score 99.570 Data time: 0.2156, Forward time: 1.2590, Backward time: 1.2392, DDP time: 0.0042, Total iter time: 2.8380 [2021-07-08 09:57:33,189][root][INFO] - ===> Epoch[5824](29120/5): Loss 0.0130, Sem 0.0130 LR: 5.500e-02 Score 99.551 Data time: 0.2557, Forward time: 1.2590, Backward time: 1.2392, DDP time: 0.0042, Total iter time: 2.7284 [2021-07-08 09:57:46,872][root][INFO] - ===> Epoch[5825](29125/5): Loss 0.0141, Sem 0.0141 LR: 5.499e-02 Score 99.522 Data time: 0.2133, Forward time: 1.2590, Backward time: 1.2392, DDP time: 0.0042, Total iter time: 2.6888 [2021-07-08 09:58:01,538][root][INFO] - ===> Epoch[5826](29130/5): Loss 0.0140, Sem 0.0140 LR: 5.499e-02 Score 99.507 Data time: 0.3168, Forward time: 1.2590, Backward time: 1.2391, DDP time: 0.0042, Total iter time: 2.8715 [2021-07-08 09:58:16,084][root][INFO] - ===> Epoch[5827](29135/5): Loss 0.0125, Sem 0.0125 LR: 5.498e-02 Score 99.551 Data time: 0.1996, Forward time: 1.2590, Backward time: 1.2392, DDP time: 0.0042, Total iter time: 2.8601 [2021-07-08 09:58:29,528][root][INFO] - ===> Epoch[5828](29140/5): Loss 0.0122, Sem 0.0122 LR: 5.497e-02 Score 99.568 Data time: 0.2135, Forward time: 1.2590, Backward time: 1.2391, DDP time: 0.0042, Total iter time: 2.6500 [2021-07-08 09:58:43,493][root][INFO] - ===> Epoch[5829](29145/5): Loss 0.0133, Sem 0.0133 LR: 5.496e-02 Score 99.521 Data time: 0.1933, Forward time: 1.2590, Backward time: 1.2392, DDP time: 0.0042, Total iter time: 2.7520 [2021-07-08 09:58:57,553][root][INFO] - ===> Epoch[5830](29150/5): Loss 0.0125, Sem 0.0125 LR: 5.495e-02 Score 99.550 Data time: 0.2170, Forward time: 1.2590, Backward time: 1.2392, DDP time: 0.0042, Total iter time: 2.7604 [2021-07-08 09:59:10,879][root][INFO] - ===> Epoch[5831](29155/5): Loss 0.0120, Sem 0.0120 LR: 5.495e-02 Score 99.566 Data time: 0.1981, Forward time: 1.2590, Backward time: 1.2392, DDP time: 0.0042, Total iter time: 2.6215 [2021-07-08 09:59:25,320][root][INFO] - ===> Epoch[5832](29160/5): Loss 0.0116, Sem 0.0116 LR: 5.494e-02 Score 99.584 Data time: 0.1942, Forward time: 1.2590, Backward time: 1.2392, DDP time: 0.0042, Total iter time: 2.8426 [2021-07-08 09:59:39,019][root][INFO] - ===> Epoch[5833](29165/5): Loss 0.0124, Sem 0.0124 LR: 5.493e-02 Score 99.560 Data time: 0.1885, Forward time: 1.2590, Backward time: 1.2392, DDP time: 0.0042, Total iter time: 2.7066 [2021-07-08 09:59:52,945][root][INFO] - ===> Epoch[5834](29170/5): Loss 0.0118, Sem 0.0118 LR: 5.492e-02 Score 99.576 Data time: 0.2548, Forward time: 1.2590, Backward time: 1.2392, DDP time: 0.0042, Total iter time: 2.7351 [2021-07-08 10:00:08,148][root][INFO] - ===> Epoch[5835](29175/5): Loss 0.0118, Sem 0.0118 LR: 5.491e-02 Score 99.574 Data time: 0.1858, Forward time: 1.2589, Backward time: 1.2392, DDP time: 0.0042, Total iter time: 2.9940 [2021-07-08 10:00:22,114][root][INFO] - ===> Epoch[5836](29180/5): Loss 0.0115, Sem 0.0115 LR: 5.491e-02 Score 99.587 Data time: 0.2342, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.7448 [2021-07-08 10:00:36,259][root][INFO] - ===> Epoch[5837](29185/5): Loss 0.0123, Sem 0.0123 LR: 5.490e-02 Score 99.559 Data time: 0.2105, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.7818 [2021-07-08 10:00:50,623][root][INFO] - ===> Epoch[5838](29190/5): Loss 0.0118, Sem 0.0118 LR: 5.489e-02 Score 99.577 Data time: 0.1765, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.8251 [2021-07-08 10:01:04,286][root][INFO] - ===> Epoch[5839](29195/5): Loss 0.0115, Sem 0.0115 LR: 5.488e-02 Score 99.585 Data time: 0.1948, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.6882 [2021-07-08 10:01:18,458][root][INFO] - ===> Epoch[5840](29200/5): Loss 0.0114, Sem 0.0114 LR: 5.487e-02 Score 99.586 Data time: 0.1878, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.7824 [2021-07-08 10:01:18,460][root][INFO] - ===> Start testing [2021-07-08 10:01:53,803][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1278 Loss 0.387 (AVG: 0.545) Score 91.907 (AVG: 91.181) mIOU 71.448 mAP 84.775 mAcc 77.967 IOU: 65.227 0.000 84.567 76.725 94.589 92.614 36.782 83.703 98.299 71.434 80.429 86.473 57.986 mAP: 85.915 9.089 95.170 89.145 98.653 98.892 62.406 97.797 99.777 87.316 92.491 94.378 91.051 mAcc: 83.031 0.000 90.911 83.288 95.773 97.177 43.051 94.557 99.353 78.256 90.903 95.684 61.590 [2021-07-08 10:01:53,803][root][INFO] - Finished test. Elapsed time: 35.3432 [2021-07-08 10:01:54,876][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 10:02:07,992][root][INFO] - ===> Epoch[5841](29205/5): Loss 0.0109, Sem 0.0109 LR: 5.487e-02 Score 99.606 Data time: 0.1996, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.5834 [2021-07-08 10:02:22,484][root][INFO] - ===> Epoch[5842](29210/5): Loss 0.0113, Sem 0.0113 LR: 5.486e-02 Score 99.594 Data time: 0.2636, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.8506 [2021-07-08 10:02:36,845][root][INFO] - ===> Epoch[5843](29215/5): Loss 0.0110, Sem 0.0110 LR: 5.485e-02 Score 99.607 Data time: 0.2402, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.8271 [2021-07-08 10:02:50,766][root][INFO] - ===> Epoch[5844](29220/5): Loss 0.0112, Sem 0.0112 LR: 5.484e-02 Score 99.599 Data time: 0.2485, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.7410 [2021-07-08 10:03:04,980][root][INFO] - ===> Epoch[5845](29225/5): Loss 0.0106, Sem 0.0106 LR: 5.483e-02 Score 99.616 Data time: 0.1761, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.8023 [2021-07-08 10:03:18,096][root][INFO] - ===> Epoch[5846](29230/5): Loss 0.0112, Sem 0.0112 LR: 5.483e-02 Score 99.596 Data time: 0.2079, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.5629 [2021-07-08 10:03:32,404][root][INFO] - ===> Epoch[5847](29235/5): Loss 0.0109, Sem 0.0109 LR: 5.482e-02 Score 99.608 Data time: 0.2957, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.8144 [2021-07-08 10:03:46,669][root][INFO] - ===> Epoch[5848](29240/5): Loss 0.0107, Sem 0.0107 LR: 5.481e-02 Score 99.613 Data time: 0.1905, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.8076 [2021-07-08 10:04:00,871][root][INFO] - ===> Epoch[5849](29245/5): Loss 0.0109, Sem 0.0109 LR: 5.480e-02 Score 99.607 Data time: 0.2048, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.7975 [2021-07-08 10:04:13,885][root][INFO] - ===> Epoch[5850](29250/5): Loss 0.0110, Sem 0.0110 LR: 5.479e-02 Score 99.604 Data time: 0.1967, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.5577 [2021-07-08 10:04:28,136][root][INFO] - ===> Epoch[5851](29255/5): Loss 0.0110, Sem 0.0110 LR: 5.479e-02 Score 99.602 Data time: 0.2318, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.7957 [2021-07-08 10:04:41,694][root][INFO] - ===> Epoch[5852](29260/5): Loss 0.0105, Sem 0.0105 LR: 5.478e-02 Score 99.622 Data time: 0.2244, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.6628 [2021-07-08 10:04:56,323][root][INFO] - ===> Epoch[5853](29265/5): Loss 0.0114, Sem 0.0114 LR: 5.477e-02 Score 99.589 Data time: 0.2436, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.8795 [2021-07-08 10:05:10,740][root][INFO] - ===> Epoch[5854](29270/5): Loss 0.0109, Sem 0.0109 LR: 5.476e-02 Score 99.608 Data time: 0.2913, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.8319 [2021-07-08 10:05:24,616][root][INFO] - ===> Epoch[5855](29275/5): Loss 0.0111, Sem 0.0111 LR: 5.475e-02 Score 99.595 Data time: 0.2109, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.7318 [2021-07-08 10:05:39,256][root][INFO] - ===> Epoch[5856](29280/5): Loss 0.0108, Sem 0.0108 LR: 5.475e-02 Score 99.611 Data time: 0.1654, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.8917 [2021-07-08 10:05:54,560][root][INFO] - ===> Epoch[5857](29285/5): Loss 0.0104, Sem 0.0104 LR: 5.474e-02 Score 99.621 Data time: 0.2105, Forward time: 1.2589, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 3.0163 [2021-07-08 10:06:08,209][root][INFO] - ===> Epoch[5858](29290/5): Loss 0.0113, Sem 0.0113 LR: 5.473e-02 Score 99.597 Data time: 0.2353, Forward time: 1.2589, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6824 [2021-07-08 10:06:22,209][root][INFO] - ===> Epoch[5859](29295/5): Loss 0.0117, Sem 0.0117 LR: 5.472e-02 Score 99.586 Data time: 0.2418, Forward time: 1.2589, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7561 [2021-07-08 10:06:34,981][root][INFO] - ===> Epoch[5860](29300/5): Loss 0.0119, Sem 0.0119 LR: 5.471e-02 Score 99.582 Data time: 0.2059, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.5127 [2021-07-08 10:06:48,065][root][INFO] - ===> Epoch[5861](29305/5): Loss 0.0116, Sem 0.0116 LR: 5.471e-02 Score 99.584 Data time: 0.1794, Forward time: 1.2589, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.5705 [2021-07-08 10:07:02,165][root][INFO] - ===> Epoch[5862](29310/5): Loss 0.0119, Sem 0.0119 LR: 5.470e-02 Score 99.577 Data time: 0.2054, Forward time: 1.2588, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7793 [2021-07-08 10:07:16,191][root][INFO] - ===> Epoch[5863](29315/5): Loss 0.0109, Sem 0.0109 LR: 5.469e-02 Score 99.608 Data time: 0.2398, Forward time: 1.2588, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7636 [2021-07-08 10:07:29,647][root][INFO] - ===> Epoch[5864](29320/5): Loss 0.0110, Sem 0.0110 LR: 5.468e-02 Score 99.603 Data time: 0.1918, Forward time: 1.2588, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6536 [2021-07-08 10:07:44,192][root][INFO] - ===> Epoch[5865](29325/5): Loss 0.0118, Sem 0.0118 LR: 5.467e-02 Score 99.581 Data time: 0.2303, Forward time: 1.2588, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.8598 [2021-07-08 10:07:58,279][root][INFO] - ===> Epoch[5866](29330/5): Loss 0.0112, Sem 0.0112 LR: 5.467e-02 Score 99.598 Data time: 0.2480, Forward time: 1.2588, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7606 [2021-07-08 10:08:12,206][root][INFO] - ===> Epoch[5867](29335/5): Loss 0.0105, Sem 0.0105 LR: 5.466e-02 Score 99.620 Data time: 0.2425, Forward time: 1.2588, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7433 [2021-07-08 10:08:25,990][root][INFO] - ===> Epoch[5868](29340/5): Loss 0.0111, Sem 0.0111 LR: 5.465e-02 Score 99.600 Data time: 0.2420, Forward time: 1.2588, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7073 [2021-07-08 10:08:39,993][root][INFO] - ===> Epoch[5869](29345/5): Loss 0.0148, Sem 0.0148 LR: 5.464e-02 Score 99.511 Data time: 0.2216, Forward time: 1.2588, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7507 [2021-07-08 10:08:54,107][root][INFO] - ===> Epoch[5870](29350/5): Loss 0.0173, Sem 0.0173 LR: 5.463e-02 Score 99.408 Data time: 0.1977, Forward time: 1.2588, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7804 [2021-07-08 10:09:08,203][root][INFO] - ===> Epoch[5871](29355/5): Loss 0.0150, Sem 0.0150 LR: 5.463e-02 Score 99.467 Data time: 0.2325, Forward time: 1.2588, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7788 [2021-07-08 10:09:21,500][root][INFO] - ===> Epoch[5872](29360/5): Loss 0.0160, Sem 0.0160 LR: 5.462e-02 Score 99.435 Data time: 0.1903, Forward time: 1.2588, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6234 [2021-07-08 10:09:35,751][root][INFO] - ===> Epoch[5873](29365/5): Loss 0.0136, Sem 0.0136 LR: 5.461e-02 Score 99.516 Data time: 0.2888, Forward time: 1.2588, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7934 [2021-07-08 10:09:49,440][root][INFO] - ===> Epoch[5874](29370/5): Loss 0.0143, Sem 0.0143 LR: 5.460e-02 Score 99.494 Data time: 0.2868, Forward time: 1.2588, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6875 [2021-07-08 10:10:03,648][root][INFO] - ===> Epoch[5875](29375/5): Loss 0.0143, Sem 0.0143 LR: 5.459e-02 Score 99.495 Data time: 0.2653, Forward time: 1.2588, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7863 [2021-07-08 10:10:17,560][root][INFO] - ===> Epoch[5876](29380/5): Loss 0.0143, Sem 0.0143 LR: 5.459e-02 Score 99.491 Data time: 0.2137, Forward time: 1.2588, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7394 [2021-07-08 10:10:31,551][root][INFO] - ===> Epoch[5877](29385/5): Loss 0.0139, Sem 0.0139 LR: 5.458e-02 Score 99.517 Data time: 0.2891, Forward time: 1.2588, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7505 [2021-07-08 10:10:45,312][root][INFO] - ===> Epoch[5878](29390/5): Loss 0.0148, Sem 0.0148 LR: 5.457e-02 Score 99.482 Data time: 0.2394, Forward time: 1.2588, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7046 [2021-07-08 10:10:59,047][root][INFO] - ===> Epoch[5879](29395/5): Loss 0.0163, Sem 0.0163 LR: 5.456e-02 Score 99.464 Data time: 0.1943, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7044 [2021-07-08 10:11:12,068][root][INFO] - ===> Epoch[5880](29400/5): Loss 0.0150, Sem 0.0150 LR: 5.455e-02 Score 99.495 Data time: 0.1900, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.5532 [2021-07-08 10:11:12,070][root][INFO] - ===> Start testing [2021-07-08 10:11:57,886][root][INFO] - 68/68: Data time: 0.1022, Iter time: 0.1284 Loss 0.451 (AVG: 0.555) Score 90.054 (AVG: 90.752) mIOU 70.099 mAP 83.862 mAcc 76.536 IOU: 64.460 0.000 85.038 75.356 94.926 91.117 29.126 79.426 98.208 71.983 79.798 86.355 55.491 mAP: 83.678 5.211 92.908 88.005 98.601 98.497 62.755 96.821 99.809 88.606 92.596 93.886 88.826 mAcc: 80.822 0.000 90.981 81.799 96.280 97.490 32.640 95.858 99.053 75.562 89.674 96.449 58.355 [2021-07-08 10:11:57,886][root][INFO] - Finished test. Elapsed time: 45.8150 [2021-07-08 10:11:59,012][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 10:12:12,316][root][INFO] - ===> Epoch[5881](29405/5): Loss 0.0145, Sem 0.0145 LR: 5.455e-02 Score 99.488 Data time: 0.2023, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6239 [2021-07-08 10:12:26,164][root][INFO] - ===> Epoch[5882](29410/5): Loss 0.0139, Sem 0.0139 LR: 5.454e-02 Score 99.511 Data time: 0.2436, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7291 [2021-07-08 10:12:40,245][root][INFO] - ===> Epoch[5883](29415/5): Loss 0.0142, Sem 0.0142 LR: 5.453e-02 Score 99.498 Data time: 0.2533, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7692 [2021-07-08 10:12:54,839][root][INFO] - ===> Epoch[5884](29420/5): Loss 0.0138, Sem 0.0138 LR: 5.452e-02 Score 99.513 Data time: 0.2378, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.8637 [2021-07-08 10:13:09,490][root][INFO] - ===> Epoch[5885](29425/5): Loss 0.0132, Sem 0.0132 LR: 5.451e-02 Score 99.527 Data time: 0.2680, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.8803 [2021-07-08 10:13:24,178][root][INFO] - ===> Epoch[5886](29430/5): Loss 0.0140, Sem 0.0140 LR: 5.450e-02 Score 99.510 Data time: 0.2721, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.8662 [2021-07-08 10:13:36,737][root][INFO] - ===> Epoch[5887](29435/5): Loss 0.0166, Sem 0.0166 LR: 5.450e-02 Score 99.426 Data time: 0.1812, Forward time: 1.2587, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.4825 [2021-07-08 10:13:52,552][root][INFO] - ===> Epoch[5888](29440/5): Loss 0.0151, Sem 0.0151 LR: 5.449e-02 Score 99.472 Data time: 0.1813, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 3.1226 [2021-07-08 10:14:05,731][root][INFO] - ===> Epoch[5889](29445/5): Loss 0.0164, Sem 0.0164 LR: 5.448e-02 Score 99.437 Data time: 0.1721, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.5943 [2021-07-08 10:14:19,684][root][INFO] - ===> Epoch[5890](29450/5): Loss 0.0151, Sem 0.0151 LR: 5.447e-02 Score 99.469 Data time: 0.2783, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7424 [2021-07-08 10:14:33,766][root][INFO] - ===> Epoch[5891](29455/5): Loss 0.0145, Sem 0.0145 LR: 5.446e-02 Score 99.489 Data time: 0.1512, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7733 [2021-07-08 10:14:46,973][root][INFO] - ===> Epoch[5892](29460/5): Loss 0.0161, Sem 0.0161 LR: 5.446e-02 Score 99.449 Data time: 0.2343, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6003 [2021-07-08 10:15:00,828][root][INFO] - ===> Epoch[5893](29465/5): Loss 0.0180, Sem 0.0180 LR: 5.445e-02 Score 99.409 Data time: 0.2405, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7132 [2021-07-08 10:15:14,609][root][INFO] - ===> Epoch[5894](29470/5): Loss 0.0190, Sem 0.0190 LR: 5.444e-02 Score 99.339 Data time: 0.1709, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7222 [2021-07-08 10:15:28,366][root][INFO] - ===> Epoch[5895](29475/5): Loss 0.0176, Sem 0.0176 LR: 5.443e-02 Score 99.387 Data time: 0.1946, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6994 [2021-07-08 10:15:42,393][root][INFO] - ===> Epoch[5896](29480/5): Loss 0.0199, Sem 0.0199 LR: 5.442e-02 Score 99.300 Data time: 0.2226, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7449 [2021-07-08 10:15:56,276][root][INFO] - ===> Epoch[5897](29485/5): Loss 0.0179, Sem 0.0179 LR: 5.442e-02 Score 99.373 Data time: 0.2312, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7333 [2021-07-08 10:16:10,677][root][INFO] - ===> Epoch[5898](29490/5): Loss 0.0183, Sem 0.0183 LR: 5.441e-02 Score 99.357 Data time: 0.2476, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.8307 [2021-07-08 10:16:24,090][root][INFO] - ===> Epoch[5899](29495/5): Loss 0.0169, Sem 0.0169 LR: 5.440e-02 Score 99.412 Data time: 0.2446, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6284 [2021-07-08 10:16:38,417][root][INFO] - ===> Epoch[5900](29500/5): Loss 0.0162, Sem 0.0162 LR: 5.439e-02 Score 99.418 Data time: 0.2260, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.8169 [2021-07-08 10:16:52,929][root][INFO] - ===> Epoch[5901](29505/5): Loss 0.0153, Sem 0.0153 LR: 5.438e-02 Score 99.451 Data time: 0.2048, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.8425 [2021-07-08 10:17:06,147][root][INFO] - ===> Epoch[5902](29510/5): Loss 0.0156, Sem 0.0156 LR: 5.438e-02 Score 99.447 Data time: 0.2729, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.5941 [2021-07-08 10:17:20,339][root][INFO] - ===> Epoch[5903](29515/5): Loss 0.0151, Sem 0.0151 LR: 5.437e-02 Score 99.462 Data time: 0.1879, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7937 [2021-07-08 10:17:34,583][root][INFO] - ===> Epoch[5904](29520/5): Loss 0.0139, Sem 0.0139 LR: 5.436e-02 Score 99.505 Data time: 0.2409, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7976 [2021-07-08 10:17:48,762][root][INFO] - ===> Epoch[5905](29525/5): Loss 0.0146, Sem 0.0146 LR: 5.435e-02 Score 99.478 Data time: 0.3248, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7909 [2021-07-08 10:18:02,115][root][INFO] - ===> Epoch[5906](29530/5): Loss 0.0138, Sem 0.0138 LR: 5.434e-02 Score 99.502 Data time: 0.2554, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6237 [2021-07-08 10:18:16,159][root][INFO] - ===> Epoch[5907](29535/5): Loss 0.0134, Sem 0.0134 LR: 5.434e-02 Score 99.518 Data time: 0.2231, Forward time: 1.2587, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.7630 [2021-07-08 10:18:29,903][root][INFO] - ===> Epoch[5908](29540/5): Loss 0.0138, Sem 0.0138 LR: 5.433e-02 Score 99.515 Data time: 0.2650, Forward time: 1.2587, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.7062 [2021-07-08 10:18:43,888][root][INFO] - ===> Epoch[5909](29545/5): Loss 0.0129, Sem 0.0129 LR: 5.432e-02 Score 99.533 Data time: 0.2296, Forward time: 1.2587, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.7589 [2021-07-08 10:18:57,986][root][INFO] - ===> Epoch[5910](29550/5): Loss 0.0144, Sem 0.0144 LR: 5.431e-02 Score 99.511 Data time: 0.2276, Forward time: 1.2587, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.7785 [2021-07-08 10:19:11,837][root][INFO] - ===> Epoch[5911](29555/5): Loss 0.0141, Sem 0.0141 LR: 5.430e-02 Score 99.512 Data time: 0.2011, Forward time: 1.2587, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.7300 [2021-07-08 10:19:25,379][root][INFO] - ===> Epoch[5912](29560/5): Loss 0.0142, Sem 0.0142 LR: 5.430e-02 Score 99.510 Data time: 0.2171, Forward time: 1.2587, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.6501 [2021-07-08 10:19:39,773][root][INFO] - ===> Epoch[5913](29565/5): Loss 0.0132, Sem 0.0132 LR: 5.429e-02 Score 99.529 Data time: 0.2008, Forward time: 1.2587, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.8437 [2021-07-08 10:19:53,586][root][INFO] - ===> Epoch[5914](29570/5): Loss 0.0140, Sem 0.0140 LR: 5.428e-02 Score 99.497 Data time: 0.2486, Forward time: 1.2587, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.7212 [2021-07-08 10:20:07,810][root][INFO] - ===> Epoch[5915](29575/5): Loss 0.0123, Sem 0.0123 LR: 5.427e-02 Score 99.559 Data time: 0.1961, Forward time: 1.2587, Backward time: 1.2393, DDP time: 0.0042, Total iter time: 2.7959 [2021-07-08 10:20:22,459][root][INFO] - ===> Epoch[5916](29580/5): Loss 0.0126, Sem 0.0126 LR: 5.426e-02 Score 99.543 Data time: 0.2689, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.8742 [2021-07-08 10:20:36,336][root][INFO] - ===> Epoch[5917](29585/5): Loss 0.0133, Sem 0.0133 LR: 5.426e-02 Score 99.526 Data time: 0.2380, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7231 [2021-07-08 10:20:51,098][root][INFO] - ===> Epoch[5918](29590/5): Loss 0.0120, Sem 0.0120 LR: 5.425e-02 Score 99.567 Data time: 0.2416, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.8994 [2021-07-08 10:21:05,298][root][INFO] - ===> Epoch[5919](29595/5): Loss 0.0120, Sem 0.0120 LR: 5.424e-02 Score 99.570 Data time: 0.1707, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7915 [2021-07-08 10:21:18,686][root][INFO] - ===> Epoch[5920](29600/5): Loss 0.0118, Sem 0.0118 LR: 5.423e-02 Score 99.575 Data time: 0.2709, Forward time: 1.2587, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6342 [2021-07-08 10:21:18,688][root][INFO] - ===> Start testing [2021-07-08 10:21:56,299][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1227 Loss 0.332 (AVG: 0.508) Score 92.479 (AVG: 91.135) mIOU 70.794 mAP 84.256 mAcc 77.901 IOU: 64.240 0.000 83.560 75.784 94.647 91.946 42.077 83.833 98.266 60.217 79.054 87.117 59.578 mAP: 84.924 7.087 90.507 89.321 98.758 98.828 65.721 97.234 99.757 85.216 91.967 94.519 91.481 mAcc: 79.056 0.000 87.221 83.585 96.117 97.091 54.792 95.487 99.499 68.933 91.628 95.547 63.756 [2021-07-08 10:21:56,299][root][INFO] - Finished test. Elapsed time: 37.6108 [2021-07-08 10:21:57,369][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 10:22:10,769][root][INFO] - ===> Epoch[5921](29605/5): Loss 0.0123, Sem 0.0123 LR: 5.422e-02 Score 99.560 Data time: 0.2157, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6444 [2021-07-08 10:22:25,358][root][INFO] - ===> Epoch[5922](29610/5): Loss 0.0114, Sem 0.0114 LR: 5.422e-02 Score 99.588 Data time: 0.2321, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.8656 [2021-07-08 10:22:38,965][root][INFO] - ===> Epoch[5923](29615/5): Loss 0.0123, Sem 0.0123 LR: 5.421e-02 Score 99.566 Data time: 0.2496, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6753 [2021-07-08 10:22:52,905][root][INFO] - ===> Epoch[5924](29620/5): Loss 0.0120, Sem 0.0120 LR: 5.420e-02 Score 99.570 Data time: 0.1953, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7551 [2021-07-08 10:23:06,682][root][INFO] - ===> Epoch[5925](29625/5): Loss 0.0120, Sem 0.0120 LR: 5.419e-02 Score 99.569 Data time: 0.2618, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7156 [2021-07-08 10:23:20,233][root][INFO] - ===> Epoch[5926](29630/5): Loss 0.0117, Sem 0.0117 LR: 5.418e-02 Score 99.580 Data time: 0.2504, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6590 [2021-07-08 10:23:34,940][root][INFO] - ===> Epoch[5927](29635/5): Loss 0.0113, Sem 0.0113 LR: 5.418e-02 Score 99.597 Data time: 0.2410, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.8947 [2021-07-08 10:23:48,259][root][INFO] - ===> Epoch[5928](29640/5): Loss 0.0118, Sem 0.0118 LR: 5.417e-02 Score 99.574 Data time: 0.2246, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6227 [2021-07-08 10:24:01,985][root][INFO] - ===> Epoch[5929](29645/5): Loss 0.0117, Sem 0.0117 LR: 5.416e-02 Score 99.578 Data time: 0.2526, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6856 [2021-07-08 10:24:16,093][root][INFO] - ===> Epoch[5930](29650/5): Loss 0.0117, Sem 0.0117 LR: 5.415e-02 Score 99.582 Data time: 0.2057, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7666 [2021-07-08 10:24:30,395][root][INFO] - ===> Epoch[5931](29655/5): Loss 0.0130, Sem 0.0130 LR: 5.414e-02 Score 99.545 Data time: 0.2188, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.8091 [2021-07-08 10:24:44,176][root][INFO] - ===> Epoch[5932](29660/5): Loss 0.0126, Sem 0.0126 LR: 5.414e-02 Score 99.574 Data time: 0.2103, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7128 [2021-07-08 10:24:58,097][root][INFO] - ===> Epoch[5933](29665/5): Loss 0.0121, Sem 0.0121 LR: 5.413e-02 Score 99.575 Data time: 0.2337, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7319 [2021-07-08 10:25:12,197][root][INFO] - ===> Epoch[5934](29670/5): Loss 0.0137, Sem 0.0137 LR: 5.412e-02 Score 99.540 Data time: 0.2524, Forward time: 1.2585, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7701 [2021-07-08 10:25:26,439][root][INFO] - ===> Epoch[5935](29675/5): Loss 0.0128, Sem 0.0128 LR: 5.411e-02 Score 99.546 Data time: 0.2373, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7963 [2021-07-08 10:25:40,275][root][INFO] - ===> Epoch[5936](29680/5): Loss 0.0134, Sem 0.0134 LR: 5.410e-02 Score 99.527 Data time: 0.2788, Forward time: 1.2585, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7214 [2021-07-08 10:25:55,300][root][INFO] - ===> Epoch[5937](29685/5): Loss 0.0129, Sem 0.0129 LR: 5.410e-02 Score 99.538 Data time: 0.2919, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.9410 [2021-07-08 10:26:10,753][root][INFO] - ===> Epoch[5938](29690/5): Loss 0.0125, Sem 0.0125 LR: 5.409e-02 Score 99.556 Data time: 0.2901, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 3.0423 [2021-07-08 10:26:26,231][root][INFO] - ===> Epoch[5939](29695/5): Loss 0.0125, Sem 0.0125 LR: 5.408e-02 Score 99.552 Data time: 0.2691, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 3.0340 [2021-07-08 10:26:39,465][root][INFO] - ===> Epoch[5940](29700/5): Loss 0.0121, Sem 0.0121 LR: 5.407e-02 Score 99.561 Data time: 0.1730, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6102 [2021-07-08 10:26:53,554][root][INFO] - ===> Epoch[5941](29705/5): Loss 0.0124, Sem 0.0124 LR: 5.406e-02 Score 99.557 Data time: 0.2497, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7579 [2021-07-08 10:27:06,852][root][INFO] - ===> Epoch[5942](29710/5): Loss 0.0119, Sem 0.0119 LR: 5.406e-02 Score 99.578 Data time: 0.1902, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6085 [2021-07-08 10:27:20,934][root][INFO] - ===> Epoch[5943](29715/5): Loss 0.0118, Sem 0.0118 LR: 5.405e-02 Score 99.579 Data time: 0.2257, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7768 [2021-07-08 10:27:34,612][root][INFO] - ===> Epoch[5944](29720/5): Loss 0.0131, Sem 0.0131 LR: 5.404e-02 Score 99.539 Data time: 0.2838, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.6870 [2021-07-08 10:27:48,463][root][INFO] - ===> Epoch[5945](29725/5): Loss 0.0126, Sem 0.0126 LR: 5.403e-02 Score 99.548 Data time: 0.1697, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7339 [2021-07-08 10:28:02,313][root][INFO] - ===> Epoch[5946](29730/5): Loss 0.0120, Sem 0.0120 LR: 5.402e-02 Score 99.570 Data time: 0.1829, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7308 [2021-07-08 10:28:15,536][root][INFO] - ===> Epoch[5947](29735/5): Loss 0.0123, Sem 0.0123 LR: 5.402e-02 Score 99.561 Data time: 0.2383, Forward time: 1.2586, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.5967 [2021-07-08 10:28:29,949][root][INFO] - ===> Epoch[5948](29740/5): Loss 0.0119, Sem 0.0119 LR: 5.401e-02 Score 99.575 Data time: 0.1663, Forward time: 1.2585, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.8387 [2021-07-08 10:28:43,790][root][INFO] - ===> Epoch[5949](29745/5): Loss 0.0120, Sem 0.0120 LR: 5.400e-02 Score 99.577 Data time: 0.2545, Forward time: 1.2585, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.7250 [2021-07-08 10:28:58,453][root][INFO] - ===> Epoch[5950](29750/5): Loss 0.0136, Sem 0.0136 LR: 5.399e-02 Score 99.527 Data time: 0.2555, Forward time: 1.2585, Backward time: 1.2394, DDP time: 0.0042, Total iter time: 2.8922 [2021-07-08 10:29:12,986][root][INFO] - ===> Epoch[5951](29755/5): Loss 0.0142, Sem 0.0142 LR: 5.398e-02 Score 99.489 Data time: 0.1787, Forward time: 1.2585, Backward time: 1.2395, DDP time: 0.0042, Total iter time: 2.8624 [2021-07-08 10:29:27,046][root][INFO] - ===> Epoch[5952](29760/5): Loss 0.0133, Sem 0.0133 LR: 5.398e-02 Score 99.522 Data time: 0.2251, Forward time: 1.2585, Backward time: 1.2395, DDP time: 0.0042, Total iter time: 2.7634 [2021-07-08 10:29:40,666][root][INFO] - ===> Epoch[5953](29765/5): Loss 0.0139, Sem 0.0139 LR: 5.397e-02 Score 99.504 Data time: 0.2285, Forward time: 1.2585, Backward time: 1.2395, DDP time: 0.0042, Total iter time: 2.6827 [2021-07-08 10:29:54,325][root][INFO] - ===> Epoch[5954](29770/5): Loss 0.0140, Sem 0.0140 LR: 5.396e-02 Score 99.507 Data time: 0.1624, Forward time: 1.2585, Backward time: 1.2395, DDP time: 0.0042, Total iter time: 2.6915 [2021-07-08 10:30:08,108][root][INFO] - ===> Epoch[5955](29775/5): Loss 0.0138, Sem 0.0138 LR: 5.395e-02 Score 99.512 Data time: 0.2214, Forward time: 1.2585, Backward time: 1.2395, DDP time: 0.0042, Total iter time: 2.7049 [2021-07-08 10:30:21,684][root][INFO] - ===> Epoch[5956](29780/5): Loss 0.0132, Sem 0.0132 LR: 5.394e-02 Score 99.527 Data time: 0.2714, Forward time: 1.2585, Backward time: 1.2395, DDP time: 0.0042, Total iter time: 2.6633 [2021-07-08 10:30:34,856][root][INFO] - ===> Epoch[5957](29785/5): Loss 0.0134, Sem 0.0134 LR: 5.393e-02 Score 99.521 Data time: 0.2567, Forward time: 1.2584, Backward time: 1.2395, DDP time: 0.0042, Total iter time: 2.5800 [2021-07-08 10:30:49,980][root][INFO] - ===> Epoch[5958](29790/5): Loss 0.0126, Sem 0.0126 LR: 5.393e-02 Score 99.554 Data time: 0.1986, Forward time: 1.2584, Backward time: 1.2395, DDP time: 0.0042, Total iter time: 2.9697 [2021-07-08 10:31:03,791][root][INFO] - ===> Epoch[5959](29795/5): Loss 0.0134, Sem 0.0134 LR: 5.392e-02 Score 99.532 Data time: 0.2143, Forward time: 1.2585, Backward time: 1.2395, DDP time: 0.0042, Total iter time: 2.7202 [2021-07-08 10:31:17,394][root][INFO] - ===> Epoch[5960](29800/5): Loss 0.0132, Sem 0.0132 LR: 5.391e-02 Score 99.538 Data time: 0.2034, Forward time: 1.2584, Backward time: 1.2395, DDP time: 0.0042, Total iter time: 2.6812 [2021-07-08 10:31:17,396][root][INFO] - ===> Start testing [2021-07-08 10:32:00,194][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1334 Loss 0.387 (AVG: 0.549) Score 92.084 (AVG: 90.726) mIOU 70.750 mAP 84.204 mAcc 77.322 IOU: 63.077 0.000 84.401 73.795 94.294 92.066 37.132 81.953 98.328 72.147 78.559 86.380 57.611 mAP: 84.284 8.558 90.883 87.756 98.580 98.785 65.464 96.879 99.757 86.101 91.736 94.173 91.697 mAcc: 77.227 0.000 88.951 82.026 96.108 97.061 40.732 94.105 99.359 77.407 90.919 96.252 65.038 [2021-07-08 10:32:00,194][root][INFO] - Finished test. Elapsed time: 42.7972 [2021-07-08 10:32:01,194][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 10:32:13,507][root][INFO] - ===> Epoch[5961](29805/5): Loss 0.0153, Sem 0.0153 LR: 5.390e-02 Score 99.471 Data time: 0.1818, Forward time: 1.2584, Backward time: 1.2395, DDP time: 0.0042, Total iter time: 2.4152 [2021-07-08 10:32:27,835][root][INFO] - ===> Epoch[5962](29810/5): Loss 0.0268, Sem 0.0268 LR: 5.389e-02 Score 99.167 Data time: 0.1793, Forward time: 1.2584, Backward time: 1.2395, DDP time: 0.0042, Total iter time: 2.8186 [2021-07-08 10:32:41,328][root][INFO] - ===> Epoch[5963](29815/5): Loss 0.0287, Sem 0.0287 LR: 5.389e-02 Score 99.054 Data time: 0.1523, Forward time: 1.2584, Backward time: 1.2395, DDP time: 0.0042, Total iter time: 2.6650 [2021-07-08 10:32:55,194][root][INFO] - ===> Epoch[5964](29820/5): Loss 0.0263, Sem 0.0263 LR: 5.388e-02 Score 99.131 Data time: 0.2860, Forward time: 1.2584, Backward time: 1.2395, DDP time: 0.0042, Total iter time: 2.7225 [2021-07-08 10:33:09,943][root][INFO] - ===> Epoch[5965](29825/5): Loss 0.0245, Sem 0.0245 LR: 5.387e-02 Score 99.159 Data time: 0.2191, Forward time: 1.2584, Backward time: 1.2395, DDP time: 0.0042, Total iter time: 2.9133 [2021-07-08 10:33:25,041][root][INFO] - ===> Epoch[5966](29830/5): Loss 0.0226, Sem 0.0226 LR: 5.386e-02 Score 99.219 Data time: 0.3014, Forward time: 1.2584, Backward time: 1.2395, DDP time: 0.0042, Total iter time: 2.9672 [2021-07-08 10:33:39,987][root][INFO] - ===> Epoch[5967](29835/5): Loss 0.0193, Sem 0.0193 LR: 5.385e-02 Score 99.323 Data time: 0.2020, Forward time: 1.2584, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.9542 [2021-07-08 10:33:54,090][root][INFO] - ===> Epoch[5968](29840/5): Loss 0.0199, Sem 0.0199 LR: 5.385e-02 Score 99.304 Data time: 0.2258, Forward time: 1.2584, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7762 [2021-07-08 10:34:07,677][root][INFO] - ===> Epoch[5969](29845/5): Loss 0.0190, Sem 0.0190 LR: 5.384e-02 Score 99.334 Data time: 0.1646, Forward time: 1.2584, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.6795 [2021-07-08 10:34:22,176][root][INFO] - ===> Epoch[5970](29850/5): Loss 0.0172, Sem 0.0172 LR: 5.383e-02 Score 99.393 Data time: 0.2520, Forward time: 1.2584, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.8603 [2021-07-08 10:34:36,256][root][INFO] - ===> Epoch[5971](29855/5): Loss 0.0168, Sem 0.0168 LR: 5.382e-02 Score 99.410 Data time: 0.1919, Forward time: 1.2584, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7682 [2021-07-08 10:34:49,831][root][INFO] - ===> Epoch[5972](29860/5): Loss 0.0165, Sem 0.0165 LR: 5.381e-02 Score 99.417 Data time: 0.2131, Forward time: 1.2584, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.6679 [2021-07-08 10:35:04,133][root][INFO] - ===> Epoch[5973](29865/5): Loss 0.0160, Sem 0.0160 LR: 5.381e-02 Score 99.447 Data time: 0.1746, Forward time: 1.2584, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.8184 [2021-07-08 10:35:17,882][root][INFO] - ===> Epoch[5974](29870/5): Loss 0.0173, Sem 0.0173 LR: 5.380e-02 Score 99.391 Data time: 0.2630, Forward time: 1.2584, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7088 [2021-07-08 10:35:32,708][root][INFO] - ===> Epoch[5975](29875/5): Loss 0.0175, Sem 0.0175 LR: 5.379e-02 Score 99.387 Data time: 0.2142, Forward time: 1.2584, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.9186 [2021-07-08 10:35:46,325][root][INFO] - ===> Epoch[5976](29880/5): Loss 0.0178, Sem 0.0178 LR: 5.378e-02 Score 99.378 Data time: 0.2387, Forward time: 1.2584, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.6712 [2021-07-08 10:35:59,409][root][INFO] - ===> Epoch[5977](29885/5): Loss 0.0172, Sem 0.0172 LR: 5.377e-02 Score 99.394 Data time: 0.2374, Forward time: 1.2584, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.5772 [2021-07-08 10:36:13,673][root][INFO] - ===> Epoch[5978](29890/5): Loss 0.0163, Sem 0.0163 LR: 5.377e-02 Score 99.416 Data time: 0.2397, Forward time: 1.2584, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.8080 [2021-07-08 10:36:27,936][root][INFO] - ===> Epoch[5979](29895/5): Loss 0.0158, Sem 0.0158 LR: 5.376e-02 Score 99.442 Data time: 0.2067, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.8101 [2021-07-08 10:36:42,079][root][INFO] - ===> Epoch[5980](29900/5): Loss 0.0146, Sem 0.0146 LR: 5.375e-02 Score 99.486 Data time: 0.2701, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7784 [2021-07-08 10:36:56,493][root][INFO] - ===> Epoch[5981](29905/5): Loss 0.0136, Sem 0.0136 LR: 5.374e-02 Score 99.511 Data time: 0.2306, Forward time: 1.2583, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.8411 [2021-07-08 10:37:11,072][root][INFO] - ===> Epoch[5982](29910/5): Loss 0.0133, Sem 0.0133 LR: 5.373e-02 Score 99.523 Data time: 0.2198, Forward time: 1.2583, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.8642 [2021-07-08 10:37:25,169][root][INFO] - ===> Epoch[5983](29915/5): Loss 0.0137, Sem 0.0137 LR: 5.373e-02 Score 99.514 Data time: 0.2602, Forward time: 1.2583, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.7656 [2021-07-08 10:37:38,691][root][INFO] - ===> Epoch[5984](29920/5): Loss 0.0129, Sem 0.0129 LR: 5.372e-02 Score 99.541 Data time: 0.2527, Forward time: 1.2583, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.6462 [2021-07-08 10:37:52,422][root][INFO] - ===> Epoch[5985](29925/5): Loss 0.0133, Sem 0.0133 LR: 5.371e-02 Score 99.530 Data time: 0.1978, Forward time: 1.2583, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.6973 [2021-07-08 10:38:06,513][root][INFO] - ===> Epoch[5986](29930/5): Loss 0.0124, Sem 0.0124 LR: 5.370e-02 Score 99.552 Data time: 0.2308, Forward time: 1.2583, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.7727 [2021-07-08 10:38:20,135][root][INFO] - ===> Epoch[5987](29935/5): Loss 0.0122, Sem 0.0122 LR: 5.369e-02 Score 99.562 Data time: 0.2527, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.6803 [2021-07-08 10:38:33,893][root][INFO] - ===> Epoch[5988](29940/5): Loss 0.0126, Sem 0.0126 LR: 5.369e-02 Score 99.547 Data time: 0.2465, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.6979 [2021-07-08 10:38:47,718][root][INFO] - ===> Epoch[5989](29945/5): Loss 0.0123, Sem 0.0123 LR: 5.368e-02 Score 99.557 Data time: 0.1730, Forward time: 1.2583, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.7228 [2021-07-08 10:39:01,862][root][INFO] - ===> Epoch[5990](29950/5): Loss 0.0122, Sem 0.0122 LR: 5.367e-02 Score 99.562 Data time: 0.2056, Forward time: 1.2583, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.7790 [2021-07-08 10:39:14,813][root][INFO] - ===> Epoch[5991](29955/5): Loss 0.0120, Sem 0.0120 LR: 5.366e-02 Score 99.564 Data time: 0.2592, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.5371 [2021-07-08 10:39:29,378][root][INFO] - ===> Epoch[5992](29960/5): Loss 0.0124, Sem 0.0124 LR: 5.365e-02 Score 99.556 Data time: 0.2472, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.8683 [2021-07-08 10:39:43,215][root][INFO] - ===> Epoch[5993](29965/5): Loss 0.0121, Sem 0.0121 LR: 5.365e-02 Score 99.565 Data time: 0.1756, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7182 [2021-07-08 10:39:56,617][root][INFO] - ===> Epoch[5994](29970/5): Loss 0.0125, Sem 0.0125 LR: 5.364e-02 Score 99.558 Data time: 0.1908, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.6336 [2021-07-08 10:40:10,755][root][INFO] - ===> Epoch[5995](29975/5): Loss 0.0116, Sem 0.0116 LR: 5.363e-02 Score 99.583 Data time: 0.2139, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7843 [2021-07-08 10:40:24,351][root][INFO] - ===> Epoch[5996](29980/5): Loss 0.0119, Sem 0.0119 LR: 5.362e-02 Score 99.572 Data time: 0.1926, Forward time: 1.2583, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.6777 [2021-07-08 10:40:38,279][root][INFO] - ===> Epoch[5997](29985/5): Loss 0.0114, Sem 0.0114 LR: 5.361e-02 Score 99.590 Data time: 0.2115, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.7365 [2021-07-08 10:40:52,738][root][INFO] - ===> Epoch[5998](29990/5): Loss 0.0120, Sem 0.0120 LR: 5.361e-02 Score 99.567 Data time: 0.2183, Forward time: 1.2583, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.8339 [2021-07-08 10:41:06,724][root][INFO] - ===> Epoch[5999](29995/5): Loss 0.0115, Sem 0.0115 LR: 5.360e-02 Score 99.587 Data time: 0.1853, Forward time: 1.2583, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.7529 [2021-07-08 10:41:20,475][root][INFO] - ===> Epoch[6000](30000/5): Loss 0.0117, Sem 0.0117 LR: 5.359e-02 Score 99.577 Data time: 0.3052, Forward time: 1.2583, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.7120 [2021-07-08 10:41:20,476][root][INFO] - ===> Start testing [2021-07-08 10:41:56,428][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1268 Loss 0.291 (AVG: 0.540) Score 94.012 (AVG: 91.158) mIOU 71.698 mAP 84.640 mAcc 77.913 IOU: 65.320 0.000 85.417 75.632 94.873 92.149 38.412 82.937 98.512 75.175 78.790 86.751 58.102 mAP: 85.048 9.179 93.126 89.332 98.829 98.540 61.870 98.162 99.787 88.960 91.737 94.325 91.423 mAcc: 83.174 0.000 88.795 81.865 96.069 96.848 43.277 95.172 99.390 80.397 90.214 96.017 61.654 [2021-07-08 10:41:56,428][root][INFO] - Finished test. Elapsed time: 35.9512 [2021-07-08 10:41:57,456][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 10:42:10,537][root][INFO] - ===> Epoch[6001](30005/5): Loss 0.0112, Sem 0.0112 LR: 5.358e-02 Score 99.599 Data time: 0.1842, Forward time: 1.2582, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.5780 [2021-07-08 10:42:25,297][root][INFO] - ===> Epoch[6002](30010/5): Loss 0.0112, Sem 0.0112 LR: 5.357e-02 Score 99.594 Data time: 0.2375, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.9018 [2021-07-08 10:42:39,421][root][INFO] - ===> Epoch[6003](30015/5): Loss 0.0114, Sem 0.0114 LR: 5.357e-02 Score 99.587 Data time: 0.2453, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7812 [2021-07-08 10:42:52,908][root][INFO] - ===> Epoch[6004](30020/5): Loss 0.0114, Sem 0.0114 LR: 5.356e-02 Score 99.590 Data time: 0.2369, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.6491 [2021-07-08 10:43:06,732][root][INFO] - ===> Epoch[6005](30025/5): Loss 0.0116, Sem 0.0116 LR: 5.355e-02 Score 99.585 Data time: 0.1687, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7281 [2021-07-08 10:43:20,674][root][INFO] - ===> Epoch[6006](30030/5): Loss 0.0117, Sem 0.0117 LR: 5.354e-02 Score 99.576 Data time: 0.1480, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7518 [2021-07-08 10:43:34,479][root][INFO] - ===> Epoch[6007](30035/5): Loss 0.0114, Sem 0.0114 LR: 5.353e-02 Score 99.590 Data time: 0.2227, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7129 [2021-07-08 10:43:47,966][root][INFO] - ===> Epoch[6008](30040/5): Loss 0.0120, Sem 0.0120 LR: 5.353e-02 Score 99.571 Data time: 0.1923, Forward time: 1.2582, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.6585 [2021-07-08 10:44:01,909][root][INFO] - ===> Epoch[6009](30045/5): Loss 0.0118, Sem 0.0118 LR: 5.352e-02 Score 99.576 Data time: 0.2536, Forward time: 1.2582, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7225 [2021-07-08 10:44:15,576][root][INFO] - ===> Epoch[6010](30050/5): Loss 0.0115, Sem 0.0115 LR: 5.351e-02 Score 99.587 Data time: 0.3052, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.6889 [2021-07-08 10:44:29,797][root][INFO] - ===> Epoch[6011](30055/5): Loss 0.0117, Sem 0.0117 LR: 5.350e-02 Score 99.578 Data time: 0.2575, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7903 [2021-07-08 10:44:44,067][root][INFO] - ===> Epoch[6012](30060/5): Loss 0.0130, Sem 0.0130 LR: 5.349e-02 Score 99.539 Data time: 0.2645, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.8094 [2021-07-08 10:44:57,563][root][INFO] - ===> Epoch[6013](30065/5): Loss 0.0119, Sem 0.0119 LR: 5.348e-02 Score 99.573 Data time: 0.2202, Forward time: 1.2582, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.6506 [2021-07-08 10:45:12,177][root][INFO] - ===> Epoch[6014](30070/5): Loss 0.0125, Sem 0.0125 LR: 5.348e-02 Score 99.550 Data time: 0.1925, Forward time: 1.2582, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.8765 [2021-07-08 10:45:26,442][root][INFO] - ===> Epoch[6015](30075/5): Loss 0.0126, Sem 0.0126 LR: 5.347e-02 Score 99.548 Data time: 0.1885, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.8097 [2021-07-08 10:45:40,958][root][INFO] - ===> Epoch[6016](30080/5): Loss 0.0122, Sem 0.0122 LR: 5.346e-02 Score 99.569 Data time: 0.2408, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.8477 [2021-07-08 10:45:54,708][root][INFO] - ===> Epoch[6017](30085/5): Loss 0.0121, Sem 0.0121 LR: 5.345e-02 Score 99.565 Data time: 0.2029, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7005 [2021-07-08 10:46:08,784][root][INFO] - ===> Epoch[6018](30090/5): Loss 0.0116, Sem 0.0116 LR: 5.344e-02 Score 99.585 Data time: 0.2237, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7689 [2021-07-08 10:46:22,665][root][INFO] - ===> Epoch[6019](30095/5): Loss 0.0117, Sem 0.0117 LR: 5.344e-02 Score 99.580 Data time: 0.1635, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7268 [2021-07-08 10:46:36,611][root][INFO] - ===> Epoch[6020](30100/5): Loss 0.0116, Sem 0.0116 LR: 5.343e-02 Score 99.580 Data time: 0.1916, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7463 [2021-07-08 10:46:51,290][root][INFO] - ===> Epoch[6021](30105/5): Loss 0.0112, Sem 0.0112 LR: 5.342e-02 Score 99.596 Data time: 0.1931, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.8969 [2021-07-08 10:47:05,031][root][INFO] - ===> Epoch[6022](30110/5): Loss 0.0118, Sem 0.0118 LR: 5.341e-02 Score 99.574 Data time: 0.1778, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7077 [2021-07-08 10:47:18,381][root][INFO] - ===> Epoch[6023](30115/5): Loss 0.0116, Sem 0.0116 LR: 5.340e-02 Score 99.580 Data time: 0.1685, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.6284 [2021-07-08 10:47:31,885][root][INFO] - ===> Epoch[6024](30120/5): Loss 0.0120, Sem 0.0120 LR: 5.340e-02 Score 99.569 Data time: 0.2171, Forward time: 1.2583, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.6436 [2021-07-08 10:47:45,350][root][INFO] - ===> Epoch[6025](30125/5): Loss 0.0143, Sem 0.0143 LR: 5.339e-02 Score 99.510 Data time: 0.2378, Forward time: 1.2582, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.6395 [2021-07-08 10:47:59,127][root][INFO] - ===> Epoch[6026](30130/5): Loss 0.0136, Sem 0.0136 LR: 5.338e-02 Score 99.522 Data time: 0.2248, Forward time: 1.2582, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.6985 [2021-07-08 10:48:12,778][root][INFO] - ===> Epoch[6027](30135/5): Loss 0.0140, Sem 0.0140 LR: 5.337e-02 Score 99.508 Data time: 0.2405, Forward time: 1.2582, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.6854 [2021-07-08 10:48:27,461][root][INFO] - ===> Epoch[6028](30140/5): Loss 0.0137, Sem 0.0137 LR: 5.336e-02 Score 99.521 Data time: 0.2652, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.8857 [2021-07-08 10:48:40,584][root][INFO] - ===> Epoch[6029](30145/5): Loss 0.0144, Sem 0.0144 LR: 5.336e-02 Score 99.487 Data time: 0.2097, Forward time: 1.2582, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.5765 [2021-07-08 10:48:55,027][root][INFO] - ===> Epoch[6030](30150/5): Loss 0.0138, Sem 0.0138 LR: 5.335e-02 Score 99.512 Data time: 0.1901, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.8496 [2021-07-08 10:49:09,660][root][INFO] - ===> Epoch[6031](30155/5): Loss 0.0145, Sem 0.0145 LR: 5.334e-02 Score 99.502 Data time: 0.2155, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.8859 [2021-07-08 10:49:23,820][root][INFO] - ===> Epoch[6032](30160/5): Loss 0.0133, Sem 0.0133 LR: 5.333e-02 Score 99.534 Data time: 0.1760, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.7856 [2021-07-08 10:49:38,702][root][INFO] - ===> Epoch[6033](30165/5): Loss 0.0129, Sem 0.0129 LR: 5.332e-02 Score 99.541 Data time: 0.2733, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.9337 [2021-07-08 10:49:52,669][root][INFO] - ===> Epoch[6034](30170/5): Loss 0.0123, Sem 0.0123 LR: 5.332e-02 Score 99.559 Data time: 0.2861, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.7440 [2021-07-08 10:50:06,608][root][INFO] - ===> Epoch[6035](30175/5): Loss 0.0121, Sem 0.0121 LR: 5.331e-02 Score 99.564 Data time: 0.1965, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.7383 [2021-07-08 10:50:20,454][root][INFO] - ===> Epoch[6036](30180/5): Loss 0.0121, Sem 0.0121 LR: 5.330e-02 Score 99.568 Data time: 0.2262, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.7212 [2021-07-08 10:50:35,378][root][INFO] - ===> Epoch[6037](30185/5): Loss 0.0124, Sem 0.0124 LR: 5.329e-02 Score 99.554 Data time: 0.3215, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.9327 [2021-07-08 10:50:49,392][root][INFO] - ===> Epoch[6038](30190/5): Loss 0.0119, Sem 0.0119 LR: 5.328e-02 Score 99.577 Data time: 0.1942, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.7581 [2021-07-08 10:51:03,417][root][INFO] - ===> Epoch[6039](30195/5): Loss 0.0122, Sem 0.0122 LR: 5.328e-02 Score 99.561 Data time: 0.1881, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.7637 [2021-07-08 10:51:16,571][root][INFO] - ===> Epoch[6040](30200/5): Loss 0.0120, Sem 0.0120 LR: 5.327e-02 Score 99.577 Data time: 0.2085, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.5827 [2021-07-08 10:51:16,574][root][INFO] - ===> Start testing [2021-07-08 10:51:51,045][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1274 Loss 0.309 (AVG: 0.534) Score 93.896 (AVG: 91.063) mIOU 71.284 mAP 84.476 mAcc 78.053 IOU: 65.398 0.000 84.642 75.858 94.944 92.206 36.351 81.216 98.474 74.100 79.144 86.317 58.044 mAP: 84.740 6.377 92.493 89.406 98.871 98.773 63.287 97.876 99.757 89.260 91.881 94.632 90.839 mAcc: 80.910 0.000 88.533 82.155 96.482 96.889 43.070 95.132 99.419 81.827 90.193 95.712 64.371 [2021-07-08 10:51:51,045][root][INFO] - Finished test. Elapsed time: 34.4705 [2021-07-08 10:51:52,139][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 10:52:04,903][root][INFO] - ===> Epoch[6041](30205/5): Loss 0.0111, Sem 0.0111 LR: 5.326e-02 Score 99.599 Data time: 0.1794, Forward time: 1.2582, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.5151 [2021-07-08 10:52:18,433][root][INFO] - ===> Epoch[6042](30210/5): Loss 0.0117, Sem 0.0117 LR: 5.325e-02 Score 99.579 Data time: 0.2426, Forward time: 1.2582, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.6618 [2021-07-08 10:52:33,323][root][INFO] - ===> Epoch[6043](30215/5): Loss 0.0112, Sem 0.0112 LR: 5.324e-02 Score 99.603 Data time: 0.2360, Forward time: 1.2582, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.9242 [2021-07-08 10:52:47,229][root][INFO] - ===> Epoch[6044](30220/5): Loss 0.0111, Sem 0.0111 LR: 5.324e-02 Score 99.602 Data time: 0.2251, Forward time: 1.2582, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7440 [2021-07-08 10:53:00,999][root][INFO] - ===> Epoch[6045](30225/5): Loss 0.0110, Sem 0.0110 LR: 5.323e-02 Score 99.603 Data time: 0.2250, Forward time: 1.2582, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7109 [2021-07-08 10:53:15,068][root][INFO] - ===> Epoch[6046](30230/5): Loss 0.0111, Sem 0.0111 LR: 5.322e-02 Score 99.600 Data time: 0.1966, Forward time: 1.2582, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.7680 [2021-07-08 10:53:28,552][root][INFO] - ===> Epoch[6047](30235/5): Loss 0.0112, Sem 0.0112 LR: 5.321e-02 Score 99.597 Data time: 0.1754, Forward time: 1.2582, Backward time: 1.2396, DDP time: 0.0042, Total iter time: 2.6602 [2021-07-08 10:53:43,637][root][INFO] - ===> Epoch[6048](30240/5): Loss 0.0123, Sem 0.0123 LR: 5.320e-02 Score 99.573 Data time: 0.2615, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.9773 [2021-07-08 10:53:58,823][root][INFO] - ===> Epoch[6049](30245/5): Loss 0.0123, Sem 0.0123 LR: 5.320e-02 Score 99.560 Data time: 0.2455, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.9937 [2021-07-08 10:54:12,797][root][INFO] - ===> Epoch[6050](30250/5): Loss 0.0123, Sem 0.0123 LR: 5.319e-02 Score 99.554 Data time: 0.2067, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.7588 [2021-07-08 10:54:27,487][root][INFO] - ===> Epoch[6051](30255/5): Loss 0.0123, Sem 0.0123 LR: 5.318e-02 Score 99.562 Data time: 0.1795, Forward time: 1.2582, Backward time: 1.2398, DDP time: 0.0042, Total iter time: 2.8966 [2021-07-08 10:54:41,631][root][INFO] - ===> Epoch[6052](30260/5): Loss 0.0117, Sem 0.0117 LR: 5.317e-02 Score 99.578 Data time: 0.2710, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.7757 [2021-07-08 10:54:56,833][root][INFO] - ===> Epoch[6053](30265/5): Loss 0.0119, Sem 0.0119 LR: 5.316e-02 Score 99.571 Data time: 0.3324, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.9771 [2021-07-08 10:55:10,934][root][INFO] - ===> Epoch[6054](30270/5): Loss 0.0115, Sem 0.0115 LR: 5.316e-02 Score 99.583 Data time: 0.2902, Forward time: 1.2582, Backward time: 1.2397, DDP time: 0.0042, Total iter time: 2.7685 [2021-07-08 10:55:25,793][root][INFO] - ===> Epoch[6055](30275/5): Loss 0.0115, Sem 0.0115 LR: 5.315e-02 Score 99.588 Data time: 0.1410, Forward time: 1.2582, Backward time: 1.2398, DDP time: 0.0042, Total iter time: 2.9306 [2021-07-08 10:55:39,217][root][INFO] - ===> Epoch[6056](30280/5): Loss 0.0119, Sem 0.0119 LR: 5.314e-02 Score 99.571 Data time: 0.1998, Forward time: 1.2582, Backward time: 1.2398, DDP time: 0.0042, Total iter time: 2.6396 [2021-07-08 10:55:52,636][root][INFO] - ===> Epoch[6057](30285/5): Loss 0.0111, Sem 0.0111 LR: 5.313e-02 Score 99.596 Data time: 0.2006, Forward time: 1.2582, Backward time: 1.2398, DDP time: 0.0042, Total iter time: 2.6276 [2021-07-08 10:56:06,648][root][INFO] - ===> Epoch[6058](30290/5): Loss 0.0119, Sem 0.0119 LR: 5.312e-02 Score 99.574 Data time: 0.1791, Forward time: 1.2581, Backward time: 1.2398, DDP time: 0.0042, Total iter time: 2.7494 [2021-07-08 10:56:21,208][root][INFO] - ===> Epoch[6059](30295/5): Loss 0.0115, Sem 0.0115 LR: 5.311e-02 Score 99.584 Data time: 0.3112, Forward time: 1.2582, Backward time: 1.2398, DDP time: 0.0042, Total iter time: 2.8658 [2021-07-08 10:56:35,776][root][INFO] - ===> Epoch[6060](30300/5): Loss 0.0110, Sem 0.0110 LR: 5.311e-02 Score 99.606 Data time: 0.1615, Forward time: 1.2582, Backward time: 1.2398, DDP time: 0.0042, Total iter time: 2.8671 [2021-07-08 10:56:49,358][root][INFO] - ===> Epoch[6061](30305/5): Loss 0.0114, Sem 0.0114 LR: 5.310e-02 Score 99.587 Data time: 0.1674, Forward time: 1.2581, Backward time: 1.2399, DDP time: 0.0042, Total iter time: 2.6777 [2021-07-08 10:57:03,728][root][INFO] - ===> Epoch[6062](30310/5): Loss 0.0106, Sem 0.0106 LR: 5.309e-02 Score 99.612 Data time: 0.1999, Forward time: 1.2582, Backward time: 1.2399, DDP time: 0.0042, Total iter time: 2.8192 [2021-07-08 10:57:18,052][root][INFO] - ===> Epoch[6063](30315/5): Loss 0.0111, Sem 0.0111 LR: 5.308e-02 Score 99.599 Data time: 0.1852, Forward time: 1.2581, Backward time: 1.2399, DDP time: 0.0042, Total iter time: 2.8231 [2021-07-08 10:57:32,512][root][INFO] - ===> Epoch[6064](30320/5): Loss 0.0112, Sem 0.0112 LR: 5.307e-02 Score 99.599 Data time: 0.2647, Forward time: 1.2582, Backward time: 1.2399, DDP time: 0.0042, Total iter time: 2.8575 [2021-07-08 10:57:47,419][root][INFO] - ===> Epoch[6065](30325/5): Loss 0.0106, Sem 0.0106 LR: 5.307e-02 Score 99.615 Data time: 0.2596, Forward time: 1.2582, Backward time: 1.2399, DDP time: 0.0042, Total iter time: 2.9407 [2021-07-08 10:58:01,431][root][INFO] - ===> Epoch[6066](30330/5): Loss 0.0113, Sem 0.0113 LR: 5.306e-02 Score 99.592 Data time: 0.2375, Forward time: 1.2581, Backward time: 1.2399, DDP time: 0.0042, Total iter time: 2.7611 [2021-07-08 10:58:15,063][root][INFO] - ===> Epoch[6067](30335/5): Loss 0.0113, Sem 0.0113 LR: 5.305e-02 Score 99.598 Data time: 0.2777, Forward time: 1.2582, Backward time: 1.2399, DDP time: 0.0042, Total iter time: 2.6832 [2021-07-08 10:58:28,766][root][INFO] - ===> Epoch[6068](30340/5): Loss 0.0120, Sem 0.0120 LR: 5.304e-02 Score 99.568 Data time: 0.3089, Forward time: 1.2581, Backward time: 1.2399, DDP time: 0.0042, Total iter time: 2.6944 [2021-07-08 10:58:42,760][root][INFO] - ===> Epoch[6069](30345/5): Loss 0.0112, Sem 0.0112 LR: 5.303e-02 Score 99.595 Data time: 0.2694, Forward time: 1.2581, Backward time: 1.2399, DDP time: 0.0042, Total iter time: 2.7505 [2021-07-08 10:58:57,202][root][INFO] - ===> Epoch[6070](30350/5): Loss 0.0108, Sem 0.0108 LR: 5.303e-02 Score 99.609 Data time: 0.3029, Forward time: 1.2581, Backward time: 1.2399, DDP time: 0.0042, Total iter time: 2.8406 [2021-07-08 10:59:11,198][root][INFO] - ===> Epoch[6071](30355/5): Loss 0.0106, Sem 0.0106 LR: 5.302e-02 Score 99.618 Data time: 0.1891, Forward time: 1.2581, Backward time: 1.2399, DDP time: 0.0042, Total iter time: 2.7500 [2021-07-08 10:59:25,053][root][INFO] - ===> Epoch[6072](30360/5): Loss 0.0108, Sem 0.0108 LR: 5.301e-02 Score 99.613 Data time: 0.2153, Forward time: 1.2581, Backward time: 1.2399, DDP time: 0.0042, Total iter time: 2.7290 [2021-07-08 10:59:40,689][root][INFO] - ===> Epoch[6073](30365/5): Loss 0.0107, Sem 0.0107 LR: 5.300e-02 Score 99.617 Data time: 0.2847, Forward time: 1.2581, Backward time: 1.2399, DDP time: 0.0042, Total iter time: 3.0744 [2021-07-08 10:59:53,926][root][INFO] - ===> Epoch[6074](30370/5): Loss 0.0112, Sem 0.0112 LR: 5.299e-02 Score 99.590 Data time: 0.1830, Forward time: 1.2581, Backward time: 1.2399, DDP time: 0.0042, Total iter time: 2.6018 [2021-07-08 11:00:08,471][root][INFO] - ===> Epoch[6075](30375/5): Loss 0.0115, Sem 0.0115 LR: 5.299e-02 Score 99.602 Data time: 0.2210, Forward time: 1.2581, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.8587 [2021-07-08 11:00:23,005][root][INFO] - ===> Epoch[6076](30380/5): Loss 0.0110, Sem 0.0110 LR: 5.298e-02 Score 99.601 Data time: 0.1709, Forward time: 1.2581, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.8648 [2021-07-08 11:00:36,746][root][INFO] - ===> Epoch[6077](30385/5): Loss 0.0109, Sem 0.0109 LR: 5.297e-02 Score 99.604 Data time: 0.2416, Forward time: 1.2581, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.7058 [2021-07-08 11:00:51,128][root][INFO] - ===> Epoch[6078](30390/5): Loss 0.0108, Sem 0.0108 LR: 5.296e-02 Score 99.608 Data time: 0.1707, Forward time: 1.2581, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.8447 [2021-07-08 11:01:05,667][root][INFO] - ===> Epoch[6079](30395/5): Loss 0.0111, Sem 0.0111 LR: 5.295e-02 Score 99.600 Data time: 0.1912, Forward time: 1.2581, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.8711 [2021-07-08 11:01:20,071][root][INFO] - ===> Epoch[6080](30400/5): Loss 0.0107, Sem 0.0107 LR: 5.295e-02 Score 99.613 Data time: 0.1204, Forward time: 1.2581, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.8494 [2021-07-08 11:01:20,072][root][INFO] - ===> Start testing [2021-07-08 11:02:00,016][root][INFO] - 68/68: Data time: 0.0598, Iter time: 0.1228 Loss 0.306 (AVG: 0.546) Score 93.234 (AVG: 91.288) mIOU 71.622 mAP 84.308 mAcc 78.285 IOU: 65.780 0.000 83.619 77.439 95.073 92.124 38.395 83.916 98.585 74.326 78.589 86.369 56.867 mAP: 85.355 5.907 92.196 90.123 98.691 98.654 61.362 98.088 99.778 89.892 91.315 94.405 90.238 mAcc: 81.780 0.000 88.271 84.302 96.349 97.560 45.196 96.561 99.469 81.916 90.033 95.539 60.728 [2021-07-08 11:02:00,017][root][INFO] - Finished test. Elapsed time: 39.9436 [2021-07-08 11:02:01,096][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 11:02:14,119][root][INFO] - ===> Epoch[6081](30405/5): Loss 0.0114, Sem 0.0114 LR: 5.294e-02 Score 99.603 Data time: 0.1833, Forward time: 1.2581, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.5754 [2021-07-08 11:02:27,648][root][INFO] - ===> Epoch[6082](30410/5): Loss 0.0113, Sem 0.0113 LR: 5.293e-02 Score 99.593 Data time: 0.2162, Forward time: 1.2581, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.6593 [2021-07-08 11:02:42,396][root][INFO] - ===> Epoch[6083](30415/5): Loss 0.0111, Sem 0.0111 LR: 5.292e-02 Score 99.599 Data time: 0.2355, Forward time: 1.2581, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.8917 [2021-07-08 11:02:55,980][root][INFO] - ===> Epoch[6084](30420/5): Loss 0.0112, Sem 0.0112 LR: 5.291e-02 Score 99.598 Data time: 0.2299, Forward time: 1.2581, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.6692 [2021-07-08 11:03:10,344][root][INFO] - ===> Epoch[6085](30425/5): Loss 0.0119, Sem 0.0119 LR: 5.291e-02 Score 99.573 Data time: 0.2413, Forward time: 1.2581, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.8281 [2021-07-08 11:03:24,590][root][INFO] - ===> Epoch[6086](30430/5): Loss 0.0125, Sem 0.0125 LR: 5.290e-02 Score 99.573 Data time: 0.1962, Forward time: 1.2581, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.8111 [2021-07-08 11:03:37,975][root][INFO] - ===> Epoch[6087](30435/5): Loss 0.0115, Sem 0.0115 LR: 5.289e-02 Score 99.588 Data time: 0.1938, Forward time: 1.2580, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.6377 [2021-07-08 11:03:51,899][root][INFO] - ===> Epoch[6088](30440/5): Loss 0.0129, Sem 0.0129 LR: 5.288e-02 Score 99.543 Data time: 0.2165, Forward time: 1.2581, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.7317 [2021-07-08 11:04:05,362][root][INFO] - ===> Epoch[6089](30445/5): Loss 0.0124, Sem 0.0124 LR: 5.287e-02 Score 99.559 Data time: 0.2463, Forward time: 1.2581, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.6442 [2021-07-08 11:04:18,912][root][INFO] - ===> Epoch[6090](30450/5): Loss 0.0121, Sem 0.0121 LR: 5.287e-02 Score 99.574 Data time: 0.2484, Forward time: 1.2581, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.6609 [2021-07-08 11:04:32,969][root][INFO] - ===> Epoch[6091](30455/5): Loss 0.0119, Sem 0.0119 LR: 5.286e-02 Score 99.570 Data time: 0.1529, Forward time: 1.2580, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.7767 [2021-07-08 11:04:46,912][root][INFO] - ===> Epoch[6092](30460/5): Loss 0.0117, Sem 0.0117 LR: 5.285e-02 Score 99.583 Data time: 0.1842, Forward time: 1.2580, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.7460 [2021-07-08 11:05:01,206][root][INFO] - ===> Epoch[6093](30465/5): Loss 0.0116, Sem 0.0116 LR: 5.284e-02 Score 99.583 Data time: 0.2009, Forward time: 1.2580, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.8095 [2021-07-08 11:05:14,544][root][INFO] - ===> Epoch[6094](30470/5): Loss 0.0121, Sem 0.0121 LR: 5.283e-02 Score 99.564 Data time: 0.1939, Forward time: 1.2580, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.6293 [2021-07-08 11:05:28,248][root][INFO] - ===> Epoch[6095](30475/5): Loss 0.0113, Sem 0.0113 LR: 5.283e-02 Score 99.590 Data time: 0.2239, Forward time: 1.2580, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.6910 [2021-07-08 11:05:42,664][root][INFO] - ===> Epoch[6096](30480/5): Loss 0.0118, Sem 0.0118 LR: 5.282e-02 Score 99.585 Data time: 0.2842, Forward time: 1.2580, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.8339 [2021-07-08 11:05:57,033][root][INFO] - ===> Epoch[6097](30485/5): Loss 0.0119, Sem 0.0119 LR: 5.281e-02 Score 99.580 Data time: 0.2695, Forward time: 1.2580, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.8291 [2021-07-08 11:06:10,827][root][INFO] - ===> Epoch[6098](30490/5): Loss 0.0109, Sem 0.0109 LR: 5.280e-02 Score 99.609 Data time: 0.2454, Forward time: 1.2580, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.7087 [2021-07-08 11:06:26,655][root][INFO] - ===> Epoch[6099](30495/5): Loss 0.0110, Sem 0.0110 LR: 5.279e-02 Score 99.601 Data time: 0.2399, Forward time: 1.2581, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 3.1181 [2021-07-08 11:06:40,599][root][INFO] - ===> Epoch[6100](30500/5): Loss 0.0108, Sem 0.0108 LR: 5.278e-02 Score 99.608 Data time: 0.2544, Forward time: 1.2581, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.7403 [2021-07-08 11:06:54,841][root][INFO] - ===> Epoch[6101](30505/5): Loss 0.0104, Sem 0.0104 LR: 5.278e-02 Score 99.621 Data time: 0.2256, Forward time: 1.2581, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.7964 [2021-07-08 11:07:09,873][root][INFO] - ===> Epoch[6102](30510/5): Loss 0.0107, Sem 0.0107 LR: 5.277e-02 Score 99.613 Data time: 0.2419, Forward time: 1.2581, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.9522 [2021-07-08 11:07:22,650][root][INFO] - ===> Epoch[6103](30515/5): Loss 0.0108, Sem 0.0108 LR: 5.276e-02 Score 99.610 Data time: 0.1969, Forward time: 1.2581, Backward time: 1.2400, DDP time: 0.0042, Total iter time: 2.5116 [2021-07-08 11:07:37,861][root][INFO] - ===> Epoch[6104](30520/5): Loss 0.0105, Sem 0.0105 LR: 5.275e-02 Score 99.624 Data time: 0.2675, Forward time: 1.2581, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.9900 [2021-07-08 11:07:51,206][root][INFO] - ===> Epoch[6105](30525/5): Loss 0.0114, Sem 0.0114 LR: 5.274e-02 Score 99.588 Data time: 0.1671, Forward time: 1.2580, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.6160 [2021-07-08 11:08:04,792][root][INFO] - ===> Epoch[6106](30530/5): Loss 0.0102, Sem 0.0102 LR: 5.274e-02 Score 99.630 Data time: 0.2445, Forward time: 1.2580, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.6728 [2021-07-08 11:08:18,361][root][INFO] - ===> Epoch[6107](30535/5): Loss 0.0103, Sem 0.0103 LR: 5.273e-02 Score 99.626 Data time: 0.1503, Forward time: 1.2580, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.6588 [2021-07-08 11:08:32,061][root][INFO] - ===> Epoch[6108](30540/5): Loss 0.0109, Sem 0.0109 LR: 5.272e-02 Score 99.605 Data time: 0.1773, Forward time: 1.2580, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.6875 [2021-07-08 11:08:46,146][root][INFO] - ===> Epoch[6109](30545/5): Loss 0.0100, Sem 0.0100 LR: 5.271e-02 Score 99.640 Data time: 0.2281, Forward time: 1.2580, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.7738 [2021-07-08 11:09:00,344][root][INFO] - ===> Epoch[6110](30550/5): Loss 0.0103, Sem 0.0103 LR: 5.270e-02 Score 99.629 Data time: 0.2329, Forward time: 1.2580, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.7958 [2021-07-08 11:09:14,654][root][INFO] - ===> Epoch[6111](30555/5): Loss 0.0111, Sem 0.0111 LR: 5.270e-02 Score 99.597 Data time: 0.2570, Forward time: 1.2580, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.8227 [2021-07-08 11:09:27,840][root][INFO] - ===> Epoch[6112](30560/5): Loss 0.0103, Sem 0.0103 LR: 5.269e-02 Score 99.631 Data time: 0.1892, Forward time: 1.2580, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.5901 [2021-07-08 11:09:41,400][root][INFO] - ===> Epoch[6113](30565/5): Loss 0.0107, Sem 0.0107 LR: 5.268e-02 Score 99.612 Data time: 0.1892, Forward time: 1.2579, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.6690 [2021-07-08 11:09:55,409][root][INFO] - ===> Epoch[6114](30570/5): Loss 0.0110, Sem 0.0110 LR: 5.267e-02 Score 99.609 Data time: 0.2506, Forward time: 1.2579, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.7577 [2021-07-08 11:10:09,290][root][INFO] - ===> Epoch[6115](30575/5): Loss 0.0110, Sem 0.0110 LR: 5.266e-02 Score 99.602 Data time: 0.2458, Forward time: 1.2579, Backward time: 1.2401, DDP time: 0.0042, Total iter time: 2.7222 [2021-07-08 11:10:23,200][root][INFO] - ===> Epoch[6116](30580/5): Loss 0.0105, Sem 0.0105 LR: 5.266e-02 Score 99.621 Data time: 0.1772, Forward time: 1.2579, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.7261 [2021-07-08 11:10:36,672][root][INFO] - ===> Epoch[6117](30585/5): Loss 0.0109, Sem 0.0109 LR: 5.265e-02 Score 99.604 Data time: 0.2219, Forward time: 1.2579, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.6544 [2021-07-08 11:10:50,496][root][INFO] - ===> Epoch[6118](30590/5): Loss 0.0103, Sem 0.0103 LR: 5.264e-02 Score 99.623 Data time: 0.2258, Forward time: 1.2579, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.7224 [2021-07-08 11:11:04,427][root][INFO] - ===> Epoch[6119](30595/5): Loss 0.0105, Sem 0.0105 LR: 5.263e-02 Score 99.621 Data time: 0.2076, Forward time: 1.2579, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.7383 [2021-07-08 11:11:18,697][root][INFO] - ===> Epoch[6120](30600/5): Loss 0.0104, Sem 0.0104 LR: 5.262e-02 Score 99.629 Data time: 0.1842, Forward time: 1.2579, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.8069 [2021-07-08 11:11:18,700][root][INFO] - ===> Start testing [2021-07-08 11:11:59,524][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1352 Loss 0.296 (AVG: 0.553) Score 93.491 (AVG: 91.287) mIOU 71.440 mAP 84.192 mAcc 77.691 IOU: 66.769 0.063 85.002 76.833 95.499 91.841 30.418 85.055 98.265 75.793 79.241 86.133 57.806 mAP: 85.768 7.490 91.772 89.225 98.787 98.629 57.127 98.383 99.783 90.648 91.673 93.579 91.629 mAcc: 81.778 0.456 87.539 83.892 96.877 97.499 34.198 96.547 99.109 84.116 90.188 96.099 61.679 [2021-07-08 11:11:59,525][root][INFO] - Finished test. Elapsed time: 40.8242 [2021-07-08 11:12:00,607][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 11:12:13,992][root][INFO] - ===> Epoch[6121](30605/5): Loss 0.0109, Sem 0.0109 LR: 5.262e-02 Score 99.609 Data time: 0.1909, Forward time: 1.2579, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.6286 [2021-07-08 11:12:27,421][root][INFO] - ===> Epoch[6122](30610/5): Loss 0.0103, Sem 0.0103 LR: 5.261e-02 Score 99.627 Data time: 0.1684, Forward time: 1.2578, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.6465 [2021-07-08 11:12:42,141][root][INFO] - ===> Epoch[6123](30615/5): Loss 0.0104, Sem 0.0104 LR: 5.260e-02 Score 99.625 Data time: 0.2285, Forward time: 1.2578, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.8800 [2021-07-08 11:12:55,682][root][INFO] - ===> Epoch[6124](30620/5): Loss 0.0105, Sem 0.0105 LR: 5.259e-02 Score 99.620 Data time: 0.2172, Forward time: 1.2578, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.6620 [2021-07-08 11:13:09,361][root][INFO] - ===> Epoch[6125](30625/5): Loss 0.0102, Sem 0.0102 LR: 5.258e-02 Score 99.632 Data time: 0.2136, Forward time: 1.2578, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.6959 [2021-07-08 11:13:23,345][root][INFO] - ===> Epoch[6126](30630/5): Loss 0.0101, Sem 0.0101 LR: 5.258e-02 Score 99.638 Data time: 0.2118, Forward time: 1.2578, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.7542 [2021-07-08 11:13:36,935][root][INFO] - ===> Epoch[6127](30635/5): Loss 0.0103, Sem 0.0103 LR: 5.257e-02 Score 99.628 Data time: 0.2443, Forward time: 1.2578, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.6735 [2021-07-08 11:13:50,972][root][INFO] - ===> Epoch[6128](30640/5): Loss 0.0106, Sem 0.0106 LR: 5.256e-02 Score 99.614 Data time: 0.1895, Forward time: 1.2578, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.7700 [2021-07-08 11:14:05,546][root][INFO] - ===> Epoch[6129](30645/5): Loss 0.0112, Sem 0.0112 LR: 5.255e-02 Score 99.601 Data time: 0.2358, Forward time: 1.2578, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.8678 [2021-07-08 11:14:19,676][root][INFO] - ===> Epoch[6130](30650/5): Loss 0.0104, Sem 0.0104 LR: 5.254e-02 Score 99.625 Data time: 0.2342, Forward time: 1.2579, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.7798 [2021-07-08 11:14:34,297][root][INFO] - ===> Epoch[6131](30655/5): Loss 0.0101, Sem 0.0101 LR: 5.254e-02 Score 99.635 Data time: 0.2914, Forward time: 1.2579, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.8738 [2021-07-08 11:14:48,877][root][INFO] - ===> Epoch[6132](30660/5): Loss 0.0100, Sem 0.0100 LR: 5.253e-02 Score 99.638 Data time: 0.2567, Forward time: 1.2579, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.8632 [2021-07-08 11:15:02,774][root][INFO] - ===> Epoch[6133](30665/5): Loss 0.0105, Sem 0.0105 LR: 5.252e-02 Score 99.625 Data time: 0.2483, Forward time: 1.2579, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.7357 [2021-07-08 11:15:17,107][root][INFO] - ===> Epoch[6134](30670/5): Loss 0.0101, Sem 0.0101 LR: 5.251e-02 Score 99.636 Data time: 0.1756, Forward time: 1.2579, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.8295 [2021-07-08 11:15:31,159][root][INFO] - ===> Epoch[6135](30675/5): Loss 0.0098, Sem 0.0098 LR: 5.250e-02 Score 99.643 Data time: 0.2105, Forward time: 1.2579, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.7731 [2021-07-08 11:15:44,656][root][INFO] - ===> Epoch[6136](30680/5): Loss 0.0107, Sem 0.0107 LR: 5.250e-02 Score 99.611 Data time: 0.1837, Forward time: 1.2579, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.6505 [2021-07-08 11:15:58,954][root][INFO] - ===> Epoch[6137](30685/5): Loss 0.0102, Sem 0.0102 LR: 5.249e-02 Score 99.634 Data time: 0.2628, Forward time: 1.2579, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.8055 [2021-07-08 11:16:13,403][root][INFO] - ===> Epoch[6138](30690/5): Loss 0.0103, Sem 0.0103 LR: 5.248e-02 Score 99.629 Data time: 0.2081, Forward time: 1.2578, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.8417 [2021-07-08 11:16:27,837][root][INFO] - ===> Epoch[6139](30695/5): Loss 0.0105, Sem 0.0105 LR: 5.247e-02 Score 99.623 Data time: 0.1968, Forward time: 1.2578, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8478 [2021-07-08 11:16:42,482][root][INFO] - ===> Epoch[6140](30700/5): Loss 0.0099, Sem 0.0099 LR: 5.246e-02 Score 99.643 Data time: 0.2320, Forward time: 1.2578, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8759 [2021-07-08 11:16:56,066][root][INFO] - ===> Epoch[6141](30705/5): Loss 0.0104, Sem 0.0104 LR: 5.245e-02 Score 99.626 Data time: 0.1930, Forward time: 1.2578, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6762 [2021-07-08 11:17:09,253][root][INFO] - ===> Epoch[6142](30710/5): Loss 0.0104, Sem 0.0104 LR: 5.245e-02 Score 99.626 Data time: 0.2545, Forward time: 1.2578, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.5918 [2021-07-08 11:17:23,196][root][INFO] - ===> Epoch[6143](30715/5): Loss 0.0100, Sem 0.0100 LR: 5.244e-02 Score 99.638 Data time: 0.2482, Forward time: 1.2578, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7442 [2021-07-08 11:17:37,679][root][INFO] - ===> Epoch[6144](30720/5): Loss 0.0097, Sem 0.0097 LR: 5.243e-02 Score 99.649 Data time: 0.2311, Forward time: 1.2578, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8448 [2021-07-08 11:17:51,247][root][INFO] - ===> Epoch[6145](30725/5): Loss 0.0103, Sem 0.0103 LR: 5.242e-02 Score 99.625 Data time: 0.2277, Forward time: 1.2578, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6672 [2021-07-08 11:18:05,852][root][INFO] - ===> Epoch[6146](30730/5): Loss 0.0098, Sem 0.0098 LR: 5.241e-02 Score 99.647 Data time: 0.2132, Forward time: 1.2578, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8728 [2021-07-08 11:18:19,309][root][INFO] - ===> Epoch[6147](30735/5): Loss 0.0105, Sem 0.0105 LR: 5.241e-02 Score 99.622 Data time: 0.1778, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6424 [2021-07-08 11:18:33,464][root][INFO] - ===> Epoch[6148](30740/5): Loss 0.0105, Sem 0.0105 LR: 5.240e-02 Score 99.622 Data time: 0.1776, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7921 [2021-07-08 11:18:47,113][root][INFO] - ===> Epoch[6149](30745/5): Loss 0.0099, Sem 0.0099 LR: 5.239e-02 Score 99.646 Data time: 0.2218, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6835 [2021-07-08 11:19:00,969][root][INFO] - ===> Epoch[6150](30750/5): Loss 0.0104, Sem 0.0104 LR: 5.238e-02 Score 99.622 Data time: 0.2175, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7152 [2021-07-08 11:19:15,149][root][INFO] - ===> Epoch[6151](30755/5): Loss 0.0098, Sem 0.0098 LR: 5.237e-02 Score 99.646 Data time: 0.2618, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7849 [2021-07-08 11:19:29,738][root][INFO] - ===> Epoch[6152](30760/5): Loss 0.0097, Sem 0.0097 LR: 5.237e-02 Score 99.652 Data time: 0.1982, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8751 [2021-07-08 11:19:43,391][root][INFO] - ===> Epoch[6153](30765/5): Loss 0.0099, Sem 0.0099 LR: 5.236e-02 Score 99.641 Data time: 0.2210, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6717 [2021-07-08 11:19:57,985][root][INFO] - ===> Epoch[6154](30770/5): Loss 0.0099, Sem 0.0099 LR: 5.235e-02 Score 99.643 Data time: 0.2297, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8769 [2021-07-08 11:20:12,205][root][INFO] - ===> Epoch[6155](30775/5): Loss 0.0099, Sem 0.0099 LR: 5.234e-02 Score 99.644 Data time: 0.2620, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7896 [2021-07-08 11:20:25,842][root][INFO] - ===> Epoch[6156](30780/5): Loss 0.0101, Sem 0.0101 LR: 5.233e-02 Score 99.634 Data time: 0.2164, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6765 [2021-07-08 11:20:39,802][root][INFO] - ===> Epoch[6157](30785/5): Loss 0.0101, Sem 0.0101 LR: 5.233e-02 Score 99.638 Data time: 0.1967, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7476 [2021-07-08 11:20:52,851][root][INFO] - ===> Epoch[6158](30790/5): Loss 0.0102, Sem 0.0102 LR: 5.232e-02 Score 99.633 Data time: 0.2025, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.5674 [2021-07-08 11:21:06,506][root][INFO] - ===> Epoch[6159](30795/5): Loss 0.0099, Sem 0.0099 LR: 5.231e-02 Score 99.640 Data time: 0.2122, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6786 [2021-07-08 11:21:20,552][root][INFO] - ===> Epoch[6160](30800/5): Loss 0.0101, Sem 0.0101 LR: 5.230e-02 Score 99.638 Data time: 0.2003, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7728 [2021-07-08 11:21:20,553][root][INFO] - ===> Start testing [2021-07-08 11:22:04,301][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1717 Loss 0.330 (AVG: 0.554) Score 92.383 (AVG: 91.293) mIOU 71.588 mAP 84.215 mAcc 77.545 IOU: 65.660 0.000 83.240 77.221 95.257 92.631 29.784 88.593 98.539 76.110 79.242 86.144 58.217 mAP: 85.389 8.327 92.603 89.386 98.647 98.545 55.579 98.466 99.766 91.730 91.838 93.501 91.015 mAcc: 81.254 0.000 87.858 84.247 96.746 97.018 32.654 94.642 99.419 84.550 90.757 96.203 62.737 [2021-07-08 11:22:04,301][root][INFO] - Finished test. Elapsed time: 43.7471 [2021-07-08 11:22:05,397][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 11:22:17,743][root][INFO] - ===> Epoch[6161](30805/5): Loss 0.0107, Sem 0.0107 LR: 5.229e-02 Score 99.614 Data time: 0.2286, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.4372 [2021-07-08 11:22:32,214][root][INFO] - ===> Epoch[6162](30810/5): Loss 0.0097, Sem 0.0097 LR: 5.229e-02 Score 99.648 Data time: 0.1828, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8559 [2021-07-08 11:22:46,285][root][INFO] - ===> Epoch[6163](30815/5): Loss 0.0103, Sem 0.0103 LR: 5.228e-02 Score 99.632 Data time: 0.2180, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7731 [2021-07-08 11:23:00,687][root][INFO] - ===> Epoch[6164](30820/5): Loss 0.0098, Sem 0.0098 LR: 5.227e-02 Score 99.644 Data time: 0.2658, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8420 [2021-07-08 11:23:14,543][root][INFO] - ===> Epoch[6165](30825/5): Loss 0.0102, Sem 0.0102 LR: 5.226e-02 Score 99.635 Data time: 0.2673, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7313 [2021-07-08 11:23:29,768][root][INFO] - ===> Epoch[6166](30830/5): Loss 0.0099, Sem 0.0099 LR: 5.225e-02 Score 99.641 Data time: 0.3111, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.9972 [2021-07-08 11:23:43,600][root][INFO] - ===> Epoch[6167](30835/5): Loss 0.0101, Sem 0.0101 LR: 5.225e-02 Score 99.633 Data time: 0.2221, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7147 [2021-07-08 11:23:57,719][root][INFO] - ===> Epoch[6168](30840/5): Loss 0.0102, Sem 0.0102 LR: 5.224e-02 Score 99.631 Data time: 0.2281, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7808 [2021-07-08 11:24:11,462][root][INFO] - ===> Epoch[6169](30845/5): Loss 0.0101, Sem 0.0101 LR: 5.223e-02 Score 99.635 Data time: 0.2298, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6988 [2021-07-08 11:24:25,721][root][INFO] - ===> Epoch[6170](30850/5): Loss 0.0096, Sem 0.0096 LR: 5.222e-02 Score 99.655 Data time: 0.2094, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8179 [2021-07-08 11:24:39,038][root][INFO] - ===> Epoch[6171](30855/5): Loss 0.0102, Sem 0.0102 LR: 5.221e-02 Score 99.632 Data time: 0.2286, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6247 [2021-07-08 11:24:52,831][root][INFO] - ===> Epoch[6172](30860/5): Loss 0.0103, Sem 0.0103 LR: 5.220e-02 Score 99.623 Data time: 0.1891, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7271 [2021-07-08 11:25:06,452][root][INFO] - ===> Epoch[6173](30865/5): Loss 0.0097, Sem 0.0097 LR: 5.220e-02 Score 99.649 Data time: 0.1756, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6882 [2021-07-08 11:25:19,919][root][INFO] - ===> Epoch[6174](30870/5): Loss 0.0100, Sem 0.0100 LR: 5.219e-02 Score 99.645 Data time: 0.2669, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6463 [2021-07-08 11:25:33,340][root][INFO] - ===> Epoch[6175](30875/5): Loss 0.0104, Sem 0.0104 LR: 5.218e-02 Score 99.626 Data time: 0.1960, Forward time: 1.2577, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.6333 [2021-07-08 11:25:47,232][root][INFO] - ===> Epoch[6176](30880/5): Loss 0.0102, Sem 0.0102 LR: 5.217e-02 Score 99.627 Data time: 0.2302, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7341 [2021-07-08 11:26:01,359][root][INFO] - ===> Epoch[6177](30885/5): Loss 0.0098, Sem 0.0098 LR: 5.216e-02 Score 99.647 Data time: 0.2091, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7797 [2021-07-08 11:26:15,766][root][INFO] - ===> Epoch[6178](30890/5): Loss 0.0104, Sem 0.0104 LR: 5.216e-02 Score 99.627 Data time: 0.2817, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8223 [2021-07-08 11:26:29,308][root][INFO] - ===> Epoch[6179](30895/5): Loss 0.0101, Sem 0.0101 LR: 5.215e-02 Score 99.634 Data time: 0.2524, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6684 [2021-07-08 11:26:43,689][root][INFO] - ===> Epoch[6180](30900/5): Loss 0.0101, Sem 0.0101 LR: 5.214e-02 Score 99.633 Data time: 0.2553, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8297 [2021-07-08 11:26:58,779][root][INFO] - ===> Epoch[6181](30905/5): Loss 0.0095, Sem 0.0095 LR: 5.213e-02 Score 99.661 Data time: 0.2892, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.9623 [2021-07-08 11:27:12,664][root][INFO] - ===> Epoch[6182](30910/5): Loss 0.0098, Sem 0.0098 LR: 5.212e-02 Score 99.648 Data time: 0.2003, Forward time: 1.2578, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.7344 [2021-07-08 11:27:26,906][root][INFO] - ===> Epoch[6183](30915/5): Loss 0.0099, Sem 0.0099 LR: 5.212e-02 Score 99.646 Data time: 0.2129, Forward time: 1.2578, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.8090 [2021-07-08 11:27:41,329][root][INFO] - ===> Epoch[6184](30920/5): Loss 0.0102, Sem 0.0102 LR: 5.211e-02 Score 99.631 Data time: 0.2530, Forward time: 1.2578, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.8242 [2021-07-08 11:27:55,419][root][INFO] - ===> Epoch[6185](30925/5): Loss 0.0102, Sem 0.0102 LR: 5.210e-02 Score 99.631 Data time: 0.2166, Forward time: 1.2578, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7788 [2021-07-08 11:28:09,654][root][INFO] - ===> Epoch[6186](30930/5): Loss 0.0093, Sem 0.0093 LR: 5.209e-02 Score 99.666 Data time: 0.2486, Forward time: 1.2578, Backward time: 1.2402, DDP time: 0.0042, Total iter time: 2.8040 [2021-07-08 11:28:24,797][root][INFO] - ===> Epoch[6187](30935/5): Loss 0.0098, Sem 0.0098 LR: 5.208e-02 Score 99.648 Data time: 0.2165, Forward time: 1.2578, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.9865 [2021-07-08 11:28:38,877][root][INFO] - ===> Epoch[6188](30940/5): Loss 0.0095, Sem 0.0095 LR: 5.208e-02 Score 99.657 Data time: 0.1975, Forward time: 1.2578, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7752 [2021-07-08 11:28:52,745][root][INFO] - ===> Epoch[6189](30945/5): Loss 0.0104, Sem 0.0104 LR: 5.207e-02 Score 99.622 Data time: 0.1864, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7344 [2021-07-08 11:29:06,954][root][INFO] - ===> Epoch[6190](30950/5): Loss 0.0099, Sem 0.0099 LR: 5.206e-02 Score 99.646 Data time: 0.2606, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7891 [2021-07-08 11:29:21,369][root][INFO] - ===> Epoch[6191](30955/5): Loss 0.0098, Sem 0.0098 LR: 5.205e-02 Score 99.645 Data time: 0.2322, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8356 [2021-07-08 11:29:35,156][root][INFO] - ===> Epoch[6192](30960/5): Loss 0.0097, Sem 0.0097 LR: 5.204e-02 Score 99.650 Data time: 0.1844, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7202 [2021-07-08 11:29:48,382][root][INFO] - ===> Epoch[6193](30965/5): Loss 0.0101, Sem 0.0101 LR: 5.204e-02 Score 99.635 Data time: 0.2710, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.5793 [2021-07-08 11:30:02,453][root][INFO] - ===> Epoch[6194](30970/5): Loss 0.0100, Sem 0.0100 LR: 5.203e-02 Score 99.633 Data time: 0.2305, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7640 [2021-07-08 11:30:16,207][root][INFO] - ===> Epoch[6195](30975/5): Loss 0.0100, Sem 0.0100 LR: 5.202e-02 Score 99.639 Data time: 0.2533, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7079 [2021-07-08 11:30:30,719][root][INFO] - ===> Epoch[6196](30980/5): Loss 0.0100, Sem 0.0100 LR: 5.201e-02 Score 99.642 Data time: 0.2157, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8617 [2021-07-08 11:30:45,618][root][INFO] - ===> Epoch[6197](30985/5): Loss 0.0099, Sem 0.0099 LR: 5.200e-02 Score 99.645 Data time: 0.2228, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.9321 [2021-07-08 11:31:00,349][root][INFO] - ===> Epoch[6198](30990/5): Loss 0.0103, Sem 0.0103 LR: 5.200e-02 Score 99.630 Data time: 0.2906, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.9077 [2021-07-08 11:31:14,127][root][INFO] - ===> Epoch[6199](30995/5): Loss 0.0100, Sem 0.0100 LR: 5.199e-02 Score 99.638 Data time: 0.3207, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6990 [2021-07-08 11:31:28,973][root][INFO] - ===> Epoch[6200](31000/5): Loss 0.0100, Sem 0.0100 LR: 5.198e-02 Score 99.639 Data time: 0.2558, Forward time: 1.2578, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.9237 [2021-07-08 11:31:28,975][root][INFO] - ===> Start testing [2021-07-08 11:32:06,222][root][INFO] - 68/68: Data time: 0.0017, Iter time: 0.1344 Loss 0.297 (AVG: 0.550) Score 93.867 (AVG: 91.310) mIOU 71.729 mAP 84.331 mAcc 77.560 IOU: 65.896 0.000 86.427 76.830 95.383 92.268 32.237 84.034 98.415 76.868 79.237 86.597 58.281 mAP: 85.432 9.481 92.680 88.755 98.806 98.554 57.255 98.268 99.747 89.710 92.101 93.869 91.645 mAcc: 81.371 0.000 87.799 83.845 96.875 96.864 35.832 95.671 99.285 81.798 90.332 96.216 62.389 [2021-07-08 11:32:06,222][root][INFO] - Finished test. Elapsed time: 37.2469 [2021-07-08 11:32:07,327][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 11:32:20,525][root][INFO] - ===> Epoch[6201](31005/5): Loss 0.0095, Sem 0.0095 LR: 5.197e-02 Score 99.656 Data time: 0.1680, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6008 [2021-07-08 11:32:34,747][root][INFO] - ===> Epoch[6202](31010/5): Loss 0.0101, Sem 0.0101 LR: 5.196e-02 Score 99.636 Data time: 0.1767, Forward time: 1.2578, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8020 [2021-07-08 11:32:48,346][root][INFO] - ===> Epoch[6203](31015/5): Loss 0.0096, Sem 0.0096 LR: 5.195e-02 Score 99.654 Data time: 0.2253, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6662 [2021-07-08 11:33:02,298][root][INFO] - ===> Epoch[6204](31020/5): Loss 0.0094, Sem 0.0094 LR: 5.195e-02 Score 99.661 Data time: 0.2206, Forward time: 1.2578, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7423 [2021-07-08 11:33:15,690][root][INFO] - ===> Epoch[6205](31025/5): Loss 0.0099, Sem 0.0099 LR: 5.194e-02 Score 99.647 Data time: 0.1787, Forward time: 1.2578, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6365 [2021-07-08 11:33:29,489][root][INFO] - ===> Epoch[6206](31030/5): Loss 0.0101, Sem 0.0101 LR: 5.193e-02 Score 99.635 Data time: 0.2375, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7210 [2021-07-08 11:33:43,069][root][INFO] - ===> Epoch[6207](31035/5): Loss 0.0099, Sem 0.0099 LR: 5.192e-02 Score 99.640 Data time: 0.1927, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6803 [2021-07-08 11:33:57,122][root][INFO] - ===> Epoch[6208](31040/5): Loss 0.0097, Sem 0.0097 LR: 5.191e-02 Score 99.650 Data time: 0.1919, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7688 [2021-07-08 11:34:11,863][root][INFO] - ===> Epoch[6209](31045/5): Loss 0.0100, Sem 0.0100 LR: 5.191e-02 Score 99.638 Data time: 0.2273, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.9085 [2021-07-08 11:34:25,615][root][INFO] - ===> Epoch[6210](31050/5): Loss 0.0098, Sem 0.0098 LR: 5.190e-02 Score 99.645 Data time: 0.2082, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7125 [2021-07-08 11:34:39,855][root][INFO] - ===> Epoch[6211](31055/5): Loss 0.0101, Sem 0.0101 LR: 5.189e-02 Score 99.642 Data time: 0.2288, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8103 [2021-07-08 11:34:53,839][root][INFO] - ===> Epoch[6212](31060/5): Loss 0.0100, Sem 0.0100 LR: 5.188e-02 Score 99.640 Data time: 0.2171, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7533 [2021-07-08 11:35:07,338][root][INFO] - ===> Epoch[6213](31065/5): Loss 0.0098, Sem 0.0098 LR: 5.187e-02 Score 99.645 Data time: 0.1895, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6611 [2021-07-08 11:35:22,033][root][INFO] - ===> Epoch[6214](31070/5): Loss 0.0094, Sem 0.0094 LR: 5.187e-02 Score 99.660 Data time: 0.2027, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8927 [2021-07-08 11:35:35,069][root][INFO] - ===> Epoch[6215](31075/5): Loss 0.0107, Sem 0.0107 LR: 5.186e-02 Score 99.624 Data time: 0.1786, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.5647 [2021-07-08 11:35:49,886][root][INFO] - ===> Epoch[6216](31080/5): Loss 0.0098, Sem 0.0098 LR: 5.185e-02 Score 99.652 Data time: 0.2054, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.9101 [2021-07-08 11:36:03,965][root][INFO] - ===> Epoch[6217](31085/5): Loss 0.0099, Sem 0.0099 LR: 5.184e-02 Score 99.644 Data time: 0.2055, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7746 [2021-07-08 11:36:18,149][root][INFO] - ===> Epoch[6218](31090/5): Loss 0.0097, Sem 0.0097 LR: 5.183e-02 Score 99.649 Data time: 0.2602, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7936 [2021-07-08 11:36:33,168][root][INFO] - ===> Epoch[6219](31095/5): Loss 0.0094, Sem 0.0094 LR: 5.183e-02 Score 99.664 Data time: 0.1816, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.9577 [2021-07-08 11:36:47,433][root][INFO] - ===> Epoch[6220](31100/5): Loss 0.0103, Sem 0.0103 LR: 5.182e-02 Score 99.630 Data time: 0.2096, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7937 [2021-07-08 11:37:02,077][root][INFO] - ===> Epoch[6221](31105/5): Loss 0.0101, Sem 0.0101 LR: 5.181e-02 Score 99.647 Data time: 0.2138, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8868 [2021-07-08 11:37:15,522][root][INFO] - ===> Epoch[6222](31110/5): Loss 0.0102, Sem 0.0102 LR: 5.180e-02 Score 99.633 Data time: 0.2295, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.6449 [2021-07-08 11:37:30,020][root][INFO] - ===> Epoch[6223](31115/5): Loss 0.0099, Sem 0.0099 LR: 5.179e-02 Score 99.646 Data time: 0.2404, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8529 [2021-07-08 11:37:43,519][root][INFO] - ===> Epoch[6224](31120/5): Loss 0.0106, Sem 0.0106 LR: 5.179e-02 Score 99.618 Data time: 0.2057, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.6651 [2021-07-08 11:37:56,875][root][INFO] - ===> Epoch[6225](31125/5): Loss 0.0100, Sem 0.0100 LR: 5.178e-02 Score 99.641 Data time: 0.2171, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.6113 [2021-07-08 11:38:11,052][root][INFO] - ===> Epoch[6226](31130/5): Loss 0.0101, Sem 0.0101 LR: 5.177e-02 Score 99.635 Data time: 0.2466, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7809 [2021-07-08 11:38:25,174][root][INFO] - ===> Epoch[6227](31135/5): Loss 0.0097, Sem 0.0097 LR: 5.176e-02 Score 99.652 Data time: 0.2217, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7700 [2021-07-08 11:38:38,583][root][INFO] - ===> Epoch[6228](31140/5): Loss 0.0104, Sem 0.0104 LR: 5.175e-02 Score 99.625 Data time: 0.2356, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.6209 [2021-07-08 11:38:53,350][root][INFO] - ===> Epoch[6229](31145/5): Loss 0.0103, Sem 0.0103 LR: 5.175e-02 Score 99.629 Data time: 0.2554, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.9002 [2021-07-08 11:39:07,282][root][INFO] - ===> Epoch[6230](31150/5): Loss 0.0098, Sem 0.0098 LR: 5.174e-02 Score 99.648 Data time: 0.1758, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7428 [2021-07-08 11:39:22,004][root][INFO] - ===> Epoch[6231](31155/5): Loss 0.0098, Sem 0.0098 LR: 5.173e-02 Score 99.647 Data time: 0.2385, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8965 [2021-07-08 11:39:36,584][root][INFO] - ===> Epoch[6232](31160/5): Loss 0.0099, Sem 0.0099 LR: 5.172e-02 Score 99.645 Data time: 0.2838, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8691 [2021-07-08 11:39:51,420][root][INFO] - ===> Epoch[6233](31165/5): Loss 0.0096, Sem 0.0096 LR: 5.171e-02 Score 99.653 Data time: 0.2557, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.9181 [2021-07-08 11:40:05,944][root][INFO] - ===> Epoch[6234](31170/5): Loss 0.0097, Sem 0.0097 LR: 5.170e-02 Score 99.653 Data time: 0.2335, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8608 [2021-07-08 11:40:19,726][root][INFO] - ===> Epoch[6235](31175/5): Loss 0.0103, Sem 0.0103 LR: 5.170e-02 Score 99.628 Data time: 0.2629, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7173 [2021-07-08 11:40:34,873][root][INFO] - ===> Epoch[6236](31180/5): Loss 0.0100, Sem 0.0100 LR: 5.169e-02 Score 99.641 Data time: 0.1857, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.9916 [2021-07-08 11:40:48,093][root][INFO] - ===> Epoch[6237](31185/5): Loss 0.0102, Sem 0.0102 LR: 5.168e-02 Score 99.625 Data time: 0.2043, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.5976 [2021-07-08 11:41:02,239][root][INFO] - ===> Epoch[6238](31190/5): Loss 0.0097, Sem 0.0097 LR: 5.167e-02 Score 99.650 Data time: 0.2589, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7887 [2021-07-08 11:41:16,232][root][INFO] - ===> Epoch[6239](31195/5): Loss 0.0097, Sem 0.0097 LR: 5.166e-02 Score 99.650 Data time: 0.2150, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7470 [2021-07-08 11:41:30,410][root][INFO] - ===> Epoch[6240](31200/5): Loss 0.0095, Sem 0.0095 LR: 5.166e-02 Score 99.657 Data time: 0.2068, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7914 [2021-07-08 11:41:30,414][root][INFO] - ===> Start testing [2021-07-08 11:42:08,610][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1294 Loss 0.297 (AVG: 0.569) Score 93.989 (AVG: 91.197) mIOU 71.501 mAP 84.179 mAcc 77.718 IOU: 65.412 0.000 85.255 76.492 95.283 92.205 31.671 87.783 98.377 74.156 78.946 86.030 57.905 mAP: 84.418 6.474 93.900 88.738 98.807 98.528 57.213 98.305 99.736 91.911 91.913 92.928 91.451 mAcc: 79.956 0.000 89.421 82.806 96.782 97.340 34.818 95.924 99.349 84.984 90.760 96.719 61.468 [2021-07-08 11:42:08,610][root][INFO] - Finished test. Elapsed time: 38.1945 [2021-07-08 11:42:09,694][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 11:42:22,346][root][INFO] - ===> Epoch[6241](31205/5): Loss 0.0102, Sem 0.0102 LR: 5.165e-02 Score 99.634 Data time: 0.1831, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.4968 [2021-07-08 11:42:35,553][root][INFO] - ===> Epoch[6242](31210/5): Loss 0.0103, Sem 0.0103 LR: 5.164e-02 Score 99.630 Data time: 0.2486, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.5913 [2021-07-08 11:42:49,648][root][INFO] - ===> Epoch[6243](31215/5): Loss 0.0091, Sem 0.0091 LR: 5.163e-02 Score 99.671 Data time: 0.1949, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7745 [2021-07-08 11:43:03,771][root][INFO] - ===> Epoch[6244](31220/5): Loss 0.0099, Sem 0.0099 LR: 5.162e-02 Score 99.640 Data time: 0.1836, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7872 [2021-07-08 11:43:17,735][root][INFO] - ===> Epoch[6245](31225/5): Loss 0.0099, Sem 0.0099 LR: 5.162e-02 Score 99.640 Data time: 0.2412, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7492 [2021-07-08 11:43:32,065][root][INFO] - ===> Epoch[6246](31230/5): Loss 0.0103, Sem 0.0103 LR: 5.161e-02 Score 99.633 Data time: 0.2540, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8118 [2021-07-08 11:43:45,552][root][INFO] - ===> Epoch[6247](31235/5): Loss 0.0099, Sem 0.0099 LR: 5.160e-02 Score 99.643 Data time: 0.2101, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.6600 [2021-07-08 11:43:59,079][root][INFO] - ===> Epoch[6248](31240/5): Loss 0.0103, Sem 0.0103 LR: 5.159e-02 Score 99.626 Data time: 0.2006, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.6650 [2021-07-08 11:44:12,079][root][INFO] - ===> Epoch[6249](31245/5): Loss 0.0099, Sem 0.0099 LR: 5.158e-02 Score 99.643 Data time: 0.1980, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.5409 [2021-07-08 11:44:26,836][root][INFO] - ===> Epoch[6250](31250/5): Loss 0.0115, Sem 0.0115 LR: 5.158e-02 Score 99.617 Data time: 0.2273, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.9027 [2021-07-08 11:44:40,875][root][INFO] - ===> Epoch[6251](31255/5): Loss 0.0102, Sem 0.0102 LR: 5.157e-02 Score 99.636 Data time: 0.1816, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7641 [2021-07-08 11:44:55,021][root][INFO] - ===> Epoch[6252](31260/5): Loss 0.0100, Sem 0.0100 LR: 5.156e-02 Score 99.636 Data time: 0.2467, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7683 [2021-07-08 11:45:09,264][root][INFO] - ===> Epoch[6253](31265/5): Loss 0.0094, Sem 0.0094 LR: 5.155e-02 Score 99.664 Data time: 0.2536, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7990 [2021-07-08 11:45:24,481][root][INFO] - ===> Epoch[6254](31270/5): Loss 0.0097, Sem 0.0097 LR: 5.154e-02 Score 99.647 Data time: 0.2380, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.9959 [2021-07-08 11:45:38,843][root][INFO] - ===> Epoch[6255](31275/5): Loss 0.0095, Sem 0.0095 LR: 5.154e-02 Score 99.657 Data time: 0.2494, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8241 [2021-07-08 11:45:52,893][root][INFO] - ===> Epoch[6256](31280/5): Loss 0.0101, Sem 0.0101 LR: 5.153e-02 Score 99.638 Data time: 0.1852, Forward time: 1.2576, Backward time: 1.2406, DDP time: 0.0042, Total iter time: 2.7745 [2021-07-08 11:46:06,762][root][INFO] - ===> Epoch[6257](31285/5): Loss 0.0104, Sem 0.0104 LR: 5.152e-02 Score 99.625 Data time: 0.2024, Forward time: 1.2576, Backward time: 1.2406, DDP time: 0.0042, Total iter time: 2.7363 [2021-07-08 11:46:20,476][root][INFO] - ===> Epoch[6258](31290/5): Loss 0.0103, Sem 0.0103 LR: 5.151e-02 Score 99.634 Data time: 0.2560, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.6996 [2021-07-08 11:46:34,673][root][INFO] - ===> Epoch[6259](31295/5): Loss 0.0099, Sem 0.0099 LR: 5.150e-02 Score 99.645 Data time: 0.2587, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7922 [2021-07-08 11:46:48,209][root][INFO] - ===> Epoch[6260](31300/5): Loss 0.0101, Sem 0.0101 LR: 5.149e-02 Score 99.636 Data time: 0.2358, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.6607 [2021-07-08 11:47:01,942][root][INFO] - ===> Epoch[6261](31305/5): Loss 0.0099, Sem 0.0099 LR: 5.149e-02 Score 99.642 Data time: 0.2187, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7050 [2021-07-08 11:47:15,052][root][INFO] - ===> Epoch[6262](31310/5): Loss 0.0097, Sem 0.0097 LR: 5.148e-02 Score 99.650 Data time: 0.1649, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.5804 [2021-07-08 11:47:29,050][root][INFO] - ===> Epoch[6263](31315/5): Loss 0.0100, Sem 0.0100 LR: 5.147e-02 Score 99.638 Data time: 0.2207, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7583 [2021-07-08 11:47:42,990][root][INFO] - ===> Epoch[6264](31320/5): Loss 0.0097, Sem 0.0097 LR: 5.146e-02 Score 99.649 Data time: 0.2595, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7468 [2021-07-08 11:47:56,375][root][INFO] - ===> Epoch[6265](31325/5): Loss 0.0103, Sem 0.0103 LR: 5.145e-02 Score 99.628 Data time: 0.2443, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.6295 [2021-07-08 11:48:10,414][root][INFO] - ===> Epoch[6266](31330/5): Loss 0.0099, Sem 0.0099 LR: 5.145e-02 Score 99.645 Data time: 0.2346, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7695 [2021-07-08 11:48:24,864][root][INFO] - ===> Epoch[6267](31335/5): Loss 0.0097, Sem 0.0097 LR: 5.144e-02 Score 99.653 Data time: 0.2905, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8359 [2021-07-08 11:48:37,955][root][INFO] - ===> Epoch[6268](31340/5): Loss 0.0099, Sem 0.0099 LR: 5.143e-02 Score 99.641 Data time: 0.2037, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.5746 [2021-07-08 11:48:51,952][root][INFO] - ===> Epoch[6269](31345/5): Loss 0.0098, Sem 0.0098 LR: 5.142e-02 Score 99.648 Data time: 0.2371, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7454 [2021-07-08 11:49:05,629][root][INFO] - ===> Epoch[6270](31350/5): Loss 0.0097, Sem 0.0097 LR: 5.141e-02 Score 99.651 Data time: 0.2313, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6910 [2021-07-08 11:49:19,522][root][INFO] - ===> Epoch[6271](31355/5): Loss 0.0095, Sem 0.0095 LR: 5.141e-02 Score 99.657 Data time: 0.2447, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7377 [2021-07-08 11:49:32,456][root][INFO] - ===> Epoch[6272](31360/5): Loss 0.0097, Sem 0.0097 LR: 5.140e-02 Score 99.648 Data time: 0.2214, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.5437 [2021-07-08 11:49:45,635][root][INFO] - ===> Epoch[6273](31365/5): Loss 0.0098, Sem 0.0098 LR: 5.139e-02 Score 99.645 Data time: 0.2091, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.5897 [2021-07-08 11:49:58,699][root][INFO] - ===> Epoch[6274](31370/5): Loss 0.0098, Sem 0.0098 LR: 5.138e-02 Score 99.647 Data time: 0.2727, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.5590 [2021-07-08 11:50:12,278][root][INFO] - ===> Epoch[6275](31375/5): Loss 0.0093, Sem 0.0093 LR: 5.137e-02 Score 99.667 Data time: 0.2503, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6762 [2021-07-08 11:50:26,657][root][INFO] - ===> Epoch[6276](31380/5): Loss 0.0102, Sem 0.0102 LR: 5.137e-02 Score 99.632 Data time: 0.1552, Forward time: 1.2576, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8345 [2021-07-08 11:50:40,863][root][INFO] - ===> Epoch[6277](31385/5): Loss 0.0094, Sem 0.0094 LR: 5.136e-02 Score 99.661 Data time: 0.2207, Forward time: 1.2576, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7987 [2021-07-08 11:50:55,012][root][INFO] - ===> Epoch[6278](31390/5): Loss 0.0094, Sem 0.0094 LR: 5.135e-02 Score 99.661 Data time: 0.2746, Forward time: 1.2576, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7743 [2021-07-08 11:51:09,509][root][INFO] - ===> Epoch[6279](31395/5): Loss 0.0095, Sem 0.0095 LR: 5.134e-02 Score 99.659 Data time: 0.2327, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8487 [2021-07-08 11:51:23,152][root][INFO] - ===> Epoch[6280](31400/5): Loss 0.0098, Sem 0.0098 LR: 5.133e-02 Score 99.642 Data time: 0.2243, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6820 [2021-07-08 11:51:23,153][root][INFO] - ===> Start testing [2021-07-08 11:52:00,337][root][INFO] - 68/68: Data time: 0.1190, Iter time: 0.1344 Loss 0.407 (AVG: 0.576) Score 90.883 (AVG: 91.199) mIOU 70.900 mAP 83.825 mAcc 77.202 IOU: 65.868 0.000 85.744 76.857 95.171 92.305 24.877 84.947 98.394 74.238 78.458 86.246 58.600 mAP: 84.508 8.965 93.177 88.468 98.616 98.343 53.277 98.236 99.743 89.642 91.321 93.592 91.832 mAcc: 80.241 0.000 89.070 84.716 96.538 97.075 26.607 97.204 99.359 82.903 90.735 96.237 62.943 [2021-07-08 11:52:00,338][root][INFO] - Finished test. Elapsed time: 37.1840 [2021-07-08 11:52:01,453][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 11:52:14,833][root][INFO] - ===> Epoch[6281](31405/5): Loss 0.0093, Sem 0.0093 LR: 5.133e-02 Score 99.663 Data time: 0.2171, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6394 [2021-07-08 11:52:29,230][root][INFO] - ===> Epoch[6282](31410/5): Loss 0.0097, Sem 0.0097 LR: 5.132e-02 Score 99.648 Data time: 0.2007, Forward time: 1.2576, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8389 [2021-07-08 11:52:43,392][root][INFO] - ===> Epoch[6283](31415/5): Loss 0.0096, Sem 0.0096 LR: 5.131e-02 Score 99.653 Data time: 0.2226, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7829 [2021-07-08 11:52:57,241][root][INFO] - ===> Epoch[6284](31420/5): Loss 0.0099, Sem 0.0099 LR: 5.130e-02 Score 99.644 Data time: 0.2469, Forward time: 1.2576, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7190 [2021-07-08 11:53:10,977][root][INFO] - ===> Epoch[6285](31425/5): Loss 0.0095, Sem 0.0095 LR: 5.129e-02 Score 99.656 Data time: 0.1954, Forward time: 1.2576, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6969 [2021-07-08 11:53:24,983][root][INFO] - ===> Epoch[6286](31430/5): Loss 0.0100, Sem 0.0100 LR: 5.128e-02 Score 99.641 Data time: 0.2266, Forward time: 1.2576, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7585 [2021-07-08 11:53:39,689][root][INFO] - ===> Epoch[6287](31435/5): Loss 0.0094, Sem 0.0094 LR: 5.128e-02 Score 99.664 Data time: 0.1429, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8986 [2021-07-08 11:53:53,486][root][INFO] - ===> Epoch[6288](31440/5): Loss 0.0136, Sem 0.0136 LR: 5.127e-02 Score 99.551 Data time: 0.2644, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7130 [2021-07-08 11:54:07,509][root][INFO] - ===> Epoch[6289](31445/5): Loss 0.0261, Sem 0.0261 LR: 5.126e-02 Score 99.185 Data time: 0.2213, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7542 [2021-07-08 11:54:22,344][root][INFO] - ===> Epoch[6290](31450/5): Loss 0.0852, Sem 0.0852 LR: 5.125e-02 Score 97.521 Data time: 0.2299, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.9170 [2021-07-08 11:54:36,183][root][INFO] - ===> Epoch[6291](31455/5): Loss 0.0933, Sem 0.0933 LR: 5.124e-02 Score 97.370 Data time: 0.1269, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7346 [2021-07-08 11:54:50,204][root][INFO] - ===> Epoch[6292](31460/5): Loss 0.0933, Sem 0.0933 LR: 5.124e-02 Score 97.176 Data time: 0.2035, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7605 [2021-07-08 11:55:04,460][root][INFO] - ===> Epoch[6293](31465/5): Loss 0.0821, Sem 0.0821 LR: 5.123e-02 Score 97.439 Data time: 0.2373, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7927 [2021-07-08 11:55:18,777][root][INFO] - ===> Epoch[6294](31470/5): Loss 0.0772, Sem 0.0772 LR: 5.122e-02 Score 97.545 Data time: 0.1859, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8153 [2021-07-08 11:55:33,676][root][INFO] - ===> Epoch[6295](31475/5): Loss 0.0690, Sem 0.0690 LR: 5.121e-02 Score 97.730 Data time: 0.1876, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.9344 [2021-07-08 11:55:46,572][root][INFO] - ===> Epoch[6296](31480/5): Loss 0.0655, Sem 0.0655 LR: 5.120e-02 Score 97.895 Data time: 0.1973, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.5335 [2021-07-08 11:56:00,590][root][INFO] - ===> Epoch[6297](31485/5): Loss 0.0690, Sem 0.0690 LR: 5.120e-02 Score 97.828 Data time: 0.2780, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7558 [2021-07-08 11:56:15,036][root][INFO] - ===> Epoch[6298](31490/5): Loss 0.0592, Sem 0.0592 LR: 5.119e-02 Score 98.033 Data time: 0.3083, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8427 [2021-07-08 11:56:29,004][root][INFO] - ===> Epoch[6299](31495/5): Loss 0.0694, Sem 0.0694 LR: 5.118e-02 Score 97.939 Data time: 0.2886, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7379 [2021-07-08 11:56:43,391][root][INFO] - ===> Epoch[6300](31500/5): Loss 0.2111, Sem 0.2111 LR: 5.117e-02 Score 94.458 Data time: 0.1877, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8295 [2021-07-08 11:56:57,345][root][INFO] - ===> Epoch[6301](31505/5): Loss 0.2842, Sem 0.2842 LR: 5.116e-02 Score 91.529 Data time: 0.2773, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7438 [2021-07-08 11:57:11,794][root][INFO] - ===> Epoch[6302](31510/5): Loss 0.2609, Sem 0.2609 LR: 5.116e-02 Score 92.087 Data time: 0.2950, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8482 [2021-07-08 11:57:25,237][root][INFO] - ===> Epoch[6303](31515/5): Loss 0.2330, Sem 0.2330 LR: 5.115e-02 Score 92.994 Data time: 0.2325, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6474 [2021-07-08 11:57:39,008][root][INFO] - ===> Epoch[6304](31520/5): Loss 0.2043, Sem 0.2043 LR: 5.114e-02 Score 93.561 Data time: 0.1794, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7186 [2021-07-08 11:57:53,373][root][INFO] - ===> Epoch[6305](31525/5): Loss 0.1842, Sem 0.1842 LR: 5.113e-02 Score 94.392 Data time: 0.2358, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8313 [2021-07-08 11:58:06,851][root][INFO] - ===> Epoch[6306](31530/5): Loss 0.1707, Sem 0.1707 LR: 5.112e-02 Score 94.680 Data time: 0.2146, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6473 [2021-07-08 11:58:20,016][root][INFO] - ===> Epoch[6307](31535/5): Loss 0.1479, Sem 0.1479 LR: 5.112e-02 Score 95.386 Data time: 0.2242, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.5838 [2021-07-08 11:58:34,364][root][INFO] - ===> Epoch[6308](31540/5): Loss 0.1279, Sem 0.1279 LR: 5.111e-02 Score 96.014 Data time: 0.2201, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8208 [2021-07-08 11:58:48,215][root][INFO] - ===> Epoch[6309](31545/5): Loss 0.1235, Sem 0.1235 LR: 5.110e-02 Score 96.043 Data time: 0.2356, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7138 [2021-07-08 11:59:02,766][root][INFO] - ===> Epoch[6310](31550/5): Loss 0.0995, Sem 0.0995 LR: 5.109e-02 Score 96.798 Data time: 0.2572, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8598 [2021-07-08 11:59:17,004][root][INFO] - ===> Epoch[6311](31555/5): Loss 0.0943, Sem 0.0943 LR: 5.108e-02 Score 97.033 Data time: 0.1848, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7970 [2021-07-08 11:59:31,098][root][INFO] - ===> Epoch[6312](31560/5): Loss 0.0826, Sem 0.0826 LR: 5.107e-02 Score 97.315 Data time: 0.2000, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7711 [2021-07-08 11:59:45,257][root][INFO] - ===> Epoch[6313](31565/5): Loss 0.0757, Sem 0.0757 LR: 5.107e-02 Score 97.500 Data time: 0.2478, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7829 [2021-07-08 11:59:58,828][root][INFO] - ===> Epoch[6314](31570/5): Loss 0.0691, Sem 0.0691 LR: 5.106e-02 Score 97.759 Data time: 0.2649, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6635 [2021-07-08 12:00:12,723][root][INFO] - ===> Epoch[6315](31575/5): Loss 0.0648, Sem 0.0648 LR: 5.105e-02 Score 97.847 Data time: 0.2209, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7380 [2021-07-08 12:00:27,254][root][INFO] - ===> Epoch[6316](31580/5): Loss 0.0606, Sem 0.0606 LR: 5.104e-02 Score 98.009 Data time: 0.1803, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8698 [2021-07-08 12:00:41,256][root][INFO] - ===> Epoch[6317](31585/5): Loss 0.0670, Sem 0.0670 LR: 5.103e-02 Score 97.794 Data time: 0.1889, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7549 [2021-07-08 12:00:55,400][root][INFO] - ===> Epoch[6318](31590/5): Loss 0.0592, Sem 0.0592 LR: 5.103e-02 Score 98.029 Data time: 0.1824, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7813 [2021-07-08 12:01:09,465][root][INFO] - ===> Epoch[6319](31595/5): Loss 0.0536, Sem 0.0536 LR: 5.102e-02 Score 98.194 Data time: 0.2042, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7670 [2021-07-08 12:01:24,254][root][INFO] - ===> Epoch[6320](31600/5): Loss 0.0486, Sem 0.0486 LR: 5.101e-02 Score 98.353 Data time: 0.2414, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.9080 [2021-07-08 12:01:24,256][root][INFO] - ===> Start testing [2021-07-08 12:02:05,474][root][INFO] - 68/68: Data time: 0.0016, Iter time: 0.1660 Loss 0.257 (AVG: 0.498) Score 92.550 (AVG: 89.602) mIOU 67.207 mAP 82.600 mAcc 74.071 IOU: 62.054 0.040 84.471 72.383 93.300 89.527 22.514 73.424 98.403 62.945 73.804 84.563 56.262 mAP: 85.168 9.454 94.093 87.405 98.773 98.659 44.809 91.912 99.857 86.167 92.827 93.841 90.831 mAcc: 76.828 0.456 89.113 79.033 95.732 95.921 24.676 80.380 99.636 72.869 91.417 96.974 59.883 [2021-07-08 12:02:05,474][root][INFO] - Finished test. Elapsed time: 41.2180 [2021-07-08 12:02:06,565][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 12:02:20,216][root][INFO] - ===> Epoch[6321](31605/5): Loss 0.0587, Sem 0.0587 LR: 5.100e-02 Score 98.073 Data time: 0.2455, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6832 [2021-07-08 12:02:33,418][root][INFO] - ===> Epoch[6322](31610/5): Loss 0.0534, Sem 0.0534 LR: 5.099e-02 Score 98.168 Data time: 0.2121, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.5868 [2021-07-08 12:02:47,820][root][INFO] - ===> Epoch[6323](31615/5): Loss 0.0487, Sem 0.0487 LR: 5.099e-02 Score 98.373 Data time: 0.2470, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8421 [2021-07-08 12:03:01,219][root][INFO] - ===> Epoch[6324](31620/5): Loss 0.0504, Sem 0.0504 LR: 5.098e-02 Score 98.325 Data time: 0.2362, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6241 [2021-07-08 12:03:14,965][root][INFO] - ===> Epoch[6325](31625/5): Loss 0.0512, Sem 0.0512 LR: 5.097e-02 Score 98.303 Data time: 0.2533, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6928 [2021-07-08 12:03:29,351][root][INFO] - ===> Epoch[6326](31630/5): Loss 0.0451, Sem 0.0451 LR: 5.096e-02 Score 98.471 Data time: 0.2176, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8255 [2021-07-08 12:03:42,804][root][INFO] - ===> Epoch[6327](31635/5): Loss 0.0435, Sem 0.0435 LR: 5.095e-02 Score 98.511 Data time: 0.2623, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6409 [2021-07-08 12:03:56,889][root][INFO] - ===> Epoch[6328](31640/5): Loss 0.0451, Sem 0.0451 LR: 5.095e-02 Score 98.483 Data time: 0.1443, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7773 [2021-07-08 12:04:11,751][root][INFO] - ===> Epoch[6329](31645/5): Loss 0.0393, Sem 0.0393 LR: 5.094e-02 Score 98.669 Data time: 0.2056, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.9239 [2021-07-08 12:04:25,570][root][INFO] - ===> Epoch[6330](31650/5): Loss 0.0401, Sem 0.0401 LR: 5.093e-02 Score 98.673 Data time: 0.2239, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7133 [2021-07-08 12:04:39,637][root][INFO] - ===> Epoch[6331](31655/5): Loss 0.0378, Sem 0.0378 LR: 5.092e-02 Score 98.728 Data time: 0.2728, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7677 [2021-07-08 12:04:53,973][root][INFO] - ===> Epoch[6332](31660/5): Loss 0.0350, Sem 0.0350 LR: 5.091e-02 Score 98.793 Data time: 0.2695, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8262 [2021-07-08 12:05:08,418][root][INFO] - ===> Epoch[6333](31665/5): Loss 0.0345, Sem 0.0345 LR: 5.091e-02 Score 98.799 Data time: 0.2358, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8481 [2021-07-08 12:05:22,486][root][INFO] - ===> Epoch[6334](31670/5): Loss 0.0318, Sem 0.0318 LR: 5.090e-02 Score 98.902 Data time: 0.3010, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7679 [2021-07-08 12:05:36,305][root][INFO] - ===> Epoch[6335](31675/5): Loss 0.0316, Sem 0.0316 LR: 5.089e-02 Score 98.898 Data time: 0.2243, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7179 [2021-07-08 12:05:50,736][root][INFO] - ===> Epoch[6336](31680/5): Loss 0.0312, Sem 0.0312 LR: 5.088e-02 Score 98.930 Data time: 0.2172, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8443 [2021-07-08 12:06:04,459][root][INFO] - ===> Epoch[6337](31685/5): Loss 0.0301, Sem 0.0301 LR: 5.087e-02 Score 98.951 Data time: 0.2146, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6840 [2021-07-08 12:06:18,948][root][INFO] - ===> Epoch[6338](31690/5): Loss 0.0299, Sem 0.0299 LR: 5.086e-02 Score 98.960 Data time: 0.2511, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8423 [2021-07-08 12:06:33,152][root][INFO] - ===> Epoch[6339](31695/5): Loss 0.0295, Sem 0.0295 LR: 5.086e-02 Score 98.964 Data time: 0.2439, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7880 [2021-07-08 12:06:46,695][root][INFO] - ===> Epoch[6340](31700/5): Loss 0.0285, Sem 0.0285 LR: 5.085e-02 Score 99.013 Data time: 0.2020, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6622 [2021-07-08 12:06:59,834][root][INFO] - ===> Epoch[6341](31705/5): Loss 0.0276, Sem 0.0276 LR: 5.084e-02 Score 99.032 Data time: 0.2142, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.5881 [2021-07-08 12:07:14,163][root][INFO] - ===> Epoch[6342](31710/5): Loss 0.0292, Sem 0.0292 LR: 5.083e-02 Score 98.980 Data time: 0.2278, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8149 [2021-07-08 12:07:29,068][root][INFO] - ===> Epoch[6343](31715/5): Loss 0.0278, Sem 0.0278 LR: 5.082e-02 Score 99.025 Data time: 0.2084, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.9393 [2021-07-08 12:07:42,699][root][INFO] - ===> Epoch[6344](31720/5): Loss 0.0270, Sem 0.0270 LR: 5.082e-02 Score 99.038 Data time: 0.2162, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6799 [2021-07-08 12:07:56,229][root][INFO] - ===> Epoch[6345](31725/5): Loss 0.0252, Sem 0.0252 LR: 5.081e-02 Score 99.107 Data time: 0.1843, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6584 [2021-07-08 12:08:10,276][root][INFO] - ===> Epoch[6346](31730/5): Loss 0.0248, Sem 0.0248 LR: 5.080e-02 Score 99.121 Data time: 0.1940, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7663 [2021-07-08 12:08:25,069][root][INFO] - ===> Epoch[6347](31735/5): Loss 0.0238, Sem 0.0238 LR: 5.079e-02 Score 99.156 Data time: 0.2497, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.9169 [2021-07-08 12:08:38,275][root][INFO] - ===> Epoch[6348](31740/5): Loss 0.0238, Sem 0.0238 LR: 5.078e-02 Score 99.158 Data time: 0.2814, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.5895 [2021-07-08 12:08:52,889][root][INFO] - ===> Epoch[6349](31745/5): Loss 0.0238, Sem 0.0238 LR: 5.078e-02 Score 99.159 Data time: 0.2524, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8789 [2021-07-08 12:09:06,945][root][INFO] - ===> Epoch[6350](31750/5): Loss 0.0247, Sem 0.0247 LR: 5.077e-02 Score 99.137 Data time: 0.2076, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7616 [2021-07-08 12:09:21,275][root][INFO] - ===> Epoch[6351](31755/5): Loss 0.0239, Sem 0.0239 LR: 5.076e-02 Score 99.164 Data time: 0.2232, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8138 [2021-07-08 12:09:34,982][root][INFO] - ===> Epoch[6352](31760/5): Loss 0.0235, Sem 0.0235 LR: 5.075e-02 Score 99.187 Data time: 0.2087, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6935 [2021-07-08 12:09:48,218][root][INFO] - ===> Epoch[6353](31765/5): Loss 0.0235, Sem 0.0235 LR: 5.074e-02 Score 99.165 Data time: 0.1616, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.5991 [2021-07-08 12:10:01,918][root][INFO] - ===> Epoch[6354](31770/5): Loss 0.0224, Sem 0.0224 LR: 5.074e-02 Score 99.212 Data time: 0.2269, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6945 [2021-07-08 12:10:15,849][root][INFO] - ===> Epoch[6355](31775/5): Loss 0.0214, Sem 0.0214 LR: 5.073e-02 Score 99.246 Data time: 0.2612, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7424 [2021-07-08 12:10:30,043][root][INFO] - ===> Epoch[6356](31780/5): Loss 0.0211, Sem 0.0211 LR: 5.072e-02 Score 99.254 Data time: 0.3432, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7781 [2021-07-08 12:10:43,596][root][INFO] - ===> Epoch[6357](31785/5): Loss 0.0209, Sem 0.0209 LR: 5.071e-02 Score 99.256 Data time: 0.2655, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6699 [2021-07-08 12:10:58,110][root][INFO] - ===> Epoch[6358](31790/5): Loss 0.0209, Sem 0.0209 LR: 5.070e-02 Score 99.259 Data time: 0.2586, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8459 [2021-07-08 12:11:12,579][root][INFO] - ===> Epoch[6359](31795/5): Loss 0.0208, Sem 0.0208 LR: 5.069e-02 Score 99.261 Data time: 0.2327, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8417 [2021-07-08 12:11:27,616][root][INFO] - ===> Epoch[6360](31800/5): Loss 0.0220, Sem 0.0220 LR: 5.069e-02 Score 99.224 Data time: 0.2374, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.9562 [2021-07-08 12:11:27,617][root][INFO] - ===> Start testing [2021-07-08 12:12:09,983][root][INFO] - 68/68: Data time: 0.0021, Iter time: 0.1477 Loss 0.313 (AVG: 0.512) Score 92.248 (AVG: 90.385) mIOU 68.849 mAP 83.601 mAcc 75.537 IOU: 62.906 0.000 80.741 75.199 93.421 89.546 23.091 78.599 98.580 72.623 78.527 85.690 56.115 mAP: 83.607 11.558 89.792 88.794 98.589 98.634 53.694 96.628 99.838 86.351 93.296 94.546 91.486 mAcc: 81.371 0.000 87.002 82.341 94.609 96.057 23.645 91.711 99.633 78.986 91.185 96.504 58.937 [2021-07-08 12:12:09,983][root][INFO] - Finished test. Elapsed time: 42.3646 [2021-07-08 12:12:11,032][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 12:12:24,479][root][INFO] - ===> Epoch[6361](31805/5): Loss 0.0230, Sem 0.0230 LR: 5.068e-02 Score 99.208 Data time: 0.1704, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6559 [2021-07-08 12:12:38,220][root][INFO] - ===> Epoch[6362](31810/5): Loss 0.0243, Sem 0.0243 LR: 5.067e-02 Score 99.166 Data time: 0.1803, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7057 [2021-07-08 12:12:51,983][root][INFO] - ===> Epoch[6363](31815/5): Loss 0.0264, Sem 0.0264 LR: 5.066e-02 Score 99.123 Data time: 0.2147, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7020 [2021-07-08 12:13:06,694][root][INFO] - ===> Epoch[6364](31820/5): Loss 0.0248, Sem 0.0248 LR: 5.065e-02 Score 99.135 Data time: 0.2570, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.9019 [2021-07-08 12:13:21,099][root][INFO] - ===> Epoch[6365](31825/5): Loss 0.0243, Sem 0.0243 LR: 5.065e-02 Score 99.151 Data time: 0.2925, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8228 [2021-07-08 12:13:35,753][root][INFO] - ===> Epoch[6366](31830/5): Loss 0.0223, Sem 0.0223 LR: 5.064e-02 Score 99.203 Data time: 0.2400, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8794 [2021-07-08 12:13:49,766][root][INFO] - ===> Epoch[6367](31835/5): Loss 0.0231, Sem 0.0231 LR: 5.063e-02 Score 99.186 Data time: 0.2029, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7587 [2021-07-08 12:14:03,832][root][INFO] - ===> Epoch[6368](31840/5): Loss 0.0208, Sem 0.0208 LR: 5.062e-02 Score 99.255 Data time: 0.2547, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7753 [2021-07-08 12:14:17,288][root][INFO] - ===> Epoch[6369](31845/5): Loss 0.0216, Sem 0.0216 LR: 5.061e-02 Score 99.233 Data time: 0.2700, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6204 [2021-07-08 12:14:31,587][root][INFO] - ===> Epoch[6370](31850/5): Loss 0.0209, Sem 0.0209 LR: 5.061e-02 Score 99.258 Data time: 0.2310, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8104 [2021-07-08 12:14:45,549][root][INFO] - ===> Epoch[6371](31855/5): Loss 0.0213, Sem 0.0213 LR: 5.060e-02 Score 99.242 Data time: 0.2284, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7458 [2021-07-08 12:14:59,060][root][INFO] - ===> Epoch[6372](31860/5): Loss 0.0203, Sem 0.0203 LR: 5.059e-02 Score 99.280 Data time: 0.2571, Forward time: 1.2576, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6561 [2021-07-08 12:15:12,558][root][INFO] - ===> Epoch[6373](31865/5): Loss 0.0238, Sem 0.0238 LR: 5.058e-02 Score 99.215 Data time: 0.1974, Forward time: 1.2576, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6566 [2021-07-08 12:15:26,857][root][INFO] - ===> Epoch[6374](31870/5): Loss 0.0213, Sem 0.0213 LR: 5.057e-02 Score 99.249 Data time: 0.2109, Forward time: 1.2576, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8119 [2021-07-08 12:15:41,612][root][INFO] - ===> Epoch[6375](31875/5): Loss 0.0202, Sem 0.0202 LR: 5.057e-02 Score 99.292 Data time: 0.2840, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8985 [2021-07-08 12:15:55,502][root][INFO] - ===> Epoch[6376](31880/5): Loss 0.0205, Sem 0.0205 LR: 5.056e-02 Score 99.286 Data time: 0.2336, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7250 [2021-07-08 12:16:09,512][root][INFO] - ===> Epoch[6377](31885/5): Loss 0.0206, Sem 0.0206 LR: 5.055e-02 Score 99.268 Data time: 0.2564, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7587 [2021-07-08 12:16:23,228][root][INFO] - ===> Epoch[6378](31890/5): Loss 0.0200, Sem 0.0200 LR: 5.054e-02 Score 99.302 Data time: 0.1770, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.6994 [2021-07-08 12:16:37,066][root][INFO] - ===> Epoch[6379](31895/5): Loss 0.0199, Sem 0.0199 LR: 5.053e-02 Score 99.291 Data time: 0.1916, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.7264 [2021-07-08 12:16:51,649][root][INFO] - ===> Epoch[6380](31900/5): Loss 0.0191, Sem 0.0191 LR: 5.053e-02 Score 99.320 Data time: 0.2118, Forward time: 1.2577, Backward time: 1.2403, DDP time: 0.0042, Total iter time: 2.8722 [2021-07-08 12:17:05,756][root][INFO] - ===> Epoch[6381](31905/5): Loss 0.0189, Sem 0.0189 LR: 5.052e-02 Score 99.330 Data time: 0.1741, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7928 [2021-07-08 12:17:20,054][root][INFO] - ===> Epoch[6382](31910/5): Loss 0.0193, Sem 0.0193 LR: 5.051e-02 Score 99.320 Data time: 0.1878, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8132 [2021-07-08 12:17:33,791][root][INFO] - ===> Epoch[6383](31915/5): Loss 0.0183, Sem 0.0183 LR: 5.050e-02 Score 99.349 Data time: 0.1841, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7018 [2021-07-08 12:17:47,972][root][INFO] - ===> Epoch[6384](31920/5): Loss 0.0181, Sem 0.0181 LR: 5.049e-02 Score 99.364 Data time: 0.2112, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7968 [2021-07-08 12:18:01,577][root][INFO] - ===> Epoch[6385](31925/5): Loss 0.0193, Sem 0.0193 LR: 5.048e-02 Score 99.330 Data time: 0.2124, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6713 [2021-07-08 12:18:15,228][root][INFO] - ===> Epoch[6386](31930/5): Loss 0.0186, Sem 0.0186 LR: 5.048e-02 Score 99.342 Data time: 0.2422, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6857 [2021-07-08 12:18:29,368][root][INFO] - ===> Epoch[6387](31935/5): Loss 0.0198, Sem 0.0198 LR: 5.047e-02 Score 99.301 Data time: 0.2106, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7716 [2021-07-08 12:18:44,311][root][INFO] - ===> Epoch[6388](31940/5): Loss 0.0189, Sem 0.0189 LR: 5.046e-02 Score 99.335 Data time: 0.2560, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.9329 [2021-07-08 12:18:58,189][root][INFO] - ===> Epoch[6389](31945/5): Loss 0.0179, Sem 0.0179 LR: 5.045e-02 Score 99.362 Data time: 0.2274, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7291 [2021-07-08 12:19:12,906][root][INFO] - ===> Epoch[6390](31950/5): Loss 0.0184, Sem 0.0184 LR: 5.044e-02 Score 99.354 Data time: 0.2212, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8965 [2021-07-08 12:19:27,027][root][INFO] - ===> Epoch[6391](31955/5): Loss 0.0185, Sem 0.0185 LR: 5.044e-02 Score 99.352 Data time: 0.1645, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7862 [2021-07-08 12:19:40,826][root][INFO] - ===> Epoch[6392](31960/5): Loss 0.0201, Sem 0.0201 LR: 5.043e-02 Score 99.323 Data time: 0.1981, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7188 [2021-07-08 12:19:56,340][root][INFO] - ===> Epoch[6393](31965/5): Loss 0.0196, Sem 0.0196 LR: 5.042e-02 Score 99.322 Data time: 0.2064, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 3.0593 [2021-07-08 12:20:09,860][root][INFO] - ===> Epoch[6394](31970/5): Loss 0.0201, Sem 0.0201 LR: 5.041e-02 Score 99.298 Data time: 0.2714, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.6516 [2021-07-08 12:20:24,141][root][INFO] - ===> Epoch[6395](31975/5): Loss 0.0197, Sem 0.0197 LR: 5.040e-02 Score 99.299 Data time: 0.2059, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8191 [2021-07-08 12:20:38,048][root][INFO] - ===> Epoch[6396](31980/5): Loss 0.0193, Sem 0.0193 LR: 5.040e-02 Score 99.311 Data time: 0.2286, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7328 [2021-07-08 12:20:52,172][root][INFO] - ===> Epoch[6397](31985/5): Loss 0.0199, Sem 0.0199 LR: 5.039e-02 Score 99.302 Data time: 0.2438, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7818 [2021-07-08 12:21:06,143][root][INFO] - ===> Epoch[6398](31990/5): Loss 0.0197, Sem 0.0197 LR: 5.038e-02 Score 99.305 Data time: 0.2497, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7428 [2021-07-08 12:21:20,100][root][INFO] - ===> Epoch[6399](31995/5): Loss 0.0198, Sem 0.0198 LR: 5.037e-02 Score 99.325 Data time: 0.2135, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7477 [2021-07-08 12:21:34,366][root][INFO] - ===> Epoch[6400](32000/5): Loss 0.0186, Sem 0.0186 LR: 5.036e-02 Score 99.350 Data time: 0.2207, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8105 [2021-07-08 12:21:34,368][root][INFO] - ===> Start testing [2021-07-08 12:22:17,760][root][INFO] - 68/68: Data time: 0.0757, Iter time: 0.1394 Loss 0.383 (AVG: 0.535) Score 91.316 (AVG: 90.506) mIOU 70.884 mAP 84.770 mAcc 77.615 IOU: 62.933 0.000 83.844 74.362 93.908 91.117 30.450 81.951 98.358 81.099 79.786 86.070 57.617 mAP: 84.086 6.420 93.622 88.385 98.831 98.887 63.165 97.571 99.825 93.330 92.970 94.756 90.162 mAcc: 81.337 0.000 89.464 82.231 95.802 97.856 35.406 94.897 99.258 85.892 89.087 94.750 63.015 [2021-07-08 12:22:17,760][root][INFO] - Finished test. Elapsed time: 43.3920 [2021-07-08 12:22:18,773][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 12:22:32,234][root][INFO] - ===> Epoch[6401](32005/5): Loss 0.0183, Sem 0.0183 LR: 5.036e-02 Score 99.353 Data time: 0.2422, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6541 [2021-07-08 12:22:46,737][root][INFO] - ===> Epoch[6402](32010/5): Loss 0.0183, Sem 0.0183 LR: 5.035e-02 Score 99.347 Data time: 0.2642, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8416 [2021-07-08 12:23:00,615][root][INFO] - ===> Epoch[6403](32015/5): Loss 0.0201, Sem 0.0201 LR: 5.034e-02 Score 99.321 Data time: 0.2156, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7290 [2021-07-08 12:23:14,586][root][INFO] - ===> Epoch[6404](32020/5): Loss 0.0180, Sem 0.0180 LR: 5.033e-02 Score 99.363 Data time: 0.2054, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7508 [2021-07-08 12:23:28,457][root][INFO] - ===> Epoch[6405](32025/5): Loss 0.0180, Sem 0.0180 LR: 5.032e-02 Score 99.360 Data time: 0.2486, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7316 [2021-07-08 12:23:43,541][root][INFO] - ===> Epoch[6406](32030/5): Loss 0.0178, Sem 0.0178 LR: 5.031e-02 Score 99.369 Data time: 0.2724, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.9717 [2021-07-08 12:23:57,282][root][INFO] - ===> Epoch[6407](32035/5): Loss 0.0203, Sem 0.0203 LR: 5.031e-02 Score 99.327 Data time: 0.2236, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7063 [2021-07-08 12:24:11,985][root][INFO] - ===> Epoch[6408](32040/5): Loss 0.0183, Sem 0.0183 LR: 5.030e-02 Score 99.346 Data time: 0.2067, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8927 [2021-07-08 12:24:25,580][root][INFO] - ===> Epoch[6409](32045/5): Loss 0.0174, Sem 0.0174 LR: 5.029e-02 Score 99.371 Data time: 0.2530, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.6699 [2021-07-08 12:24:39,718][root][INFO] - ===> Epoch[6410](32050/5): Loss 0.0175, Sem 0.0175 LR: 5.028e-02 Score 99.379 Data time: 0.1828, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7836 [2021-07-08 12:24:53,103][root][INFO] - ===> Epoch[6411](32055/5): Loss 0.0170, Sem 0.0170 LR: 5.027e-02 Score 99.389 Data time: 0.1700, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.6448 [2021-07-08 12:25:06,197][root][INFO] - ===> Epoch[6412](32060/5): Loss 0.0169, Sem 0.0169 LR: 5.027e-02 Score 99.390 Data time: 0.1889, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.5769 [2021-07-08 12:25:21,113][root][INFO] - ===> Epoch[6413](32065/5): Loss 0.0166, Sem 0.0166 LR: 5.026e-02 Score 99.406 Data time: 0.2349, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.9418 [2021-07-08 12:25:35,078][root][INFO] - ===> Epoch[6414](32070/5): Loss 0.0166, Sem 0.0166 LR: 5.025e-02 Score 99.407 Data time: 0.3097, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7455 [2021-07-08 12:25:49,007][root][INFO] - ===> Epoch[6415](32075/5): Loss 0.0163, Sem 0.0163 LR: 5.024e-02 Score 99.424 Data time: 0.2260, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7406 [2021-07-08 12:26:02,957][root][INFO] - ===> Epoch[6416](32080/5): Loss 0.0197, Sem 0.0197 LR: 5.023e-02 Score 99.324 Data time: 0.2128, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7459 [2021-07-08 12:26:16,804][root][INFO] - ===> Epoch[6417](32085/5): Loss 0.0289, Sem 0.0289 LR: 5.023e-02 Score 98.993 Data time: 0.1946, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7260 [2021-07-08 12:26:30,467][root][INFO] - ===> Epoch[6418](32090/5): Loss 0.0294, Sem 0.0294 LR: 5.022e-02 Score 99.000 Data time: 0.1911, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.6847 [2021-07-08 12:26:44,996][root][INFO] - ===> Epoch[6419](32095/5): Loss 0.0291, Sem 0.0291 LR: 5.021e-02 Score 99.025 Data time: 0.2460, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8601 [2021-07-08 12:26:58,552][root][INFO] - ===> Epoch[6420](32100/5): Loss 0.0332, Sem 0.0332 LR: 5.020e-02 Score 98.946 Data time: 0.2207, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.6610 [2021-07-08 12:27:12,456][root][INFO] - ===> Epoch[6421](32105/5): Loss 0.0323, Sem 0.0323 LR: 5.019e-02 Score 98.901 Data time: 0.2290, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7381 [2021-07-08 12:27:26,360][root][INFO] - ===> Epoch[6422](32110/5): Loss 0.0304, Sem 0.0304 LR: 5.019e-02 Score 98.953 Data time: 0.2313, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7275 [2021-07-08 12:27:40,718][root][INFO] - ===> Epoch[6423](32115/5): Loss 0.0308, Sem 0.0308 LR: 5.018e-02 Score 98.964 Data time: 0.1859, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8170 [2021-07-08 12:27:54,968][root][INFO] - ===> Epoch[6424](32120/5): Loss 0.0284, Sem 0.0284 LR: 5.017e-02 Score 98.998 Data time: 0.2017, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8060 [2021-07-08 12:28:08,886][root][INFO] - ===> Epoch[6425](32125/5): Loss 0.0249, Sem 0.0249 LR: 5.016e-02 Score 99.127 Data time: 0.1884, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7354 [2021-07-08 12:28:22,731][root][INFO] - ===> Epoch[6426](32130/5): Loss 0.0251, Sem 0.0251 LR: 5.015e-02 Score 99.125 Data time: 0.2084, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7290 [2021-07-08 12:28:35,753][root][INFO] - ===> Epoch[6427](32135/5): Loss 0.0256, Sem 0.0256 LR: 5.014e-02 Score 99.119 Data time: 0.2425, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.5629 [2021-07-08 12:28:50,122][root][INFO] - ===> Epoch[6428](32140/5): Loss 0.0251, Sem 0.0251 LR: 5.014e-02 Score 99.139 Data time: 0.2333, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8329 [2021-07-08 12:29:04,312][root][INFO] - ===> Epoch[6429](32145/5): Loss 0.0246, Sem 0.0246 LR: 5.013e-02 Score 99.156 Data time: 0.1982, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7886 [2021-07-08 12:29:18,571][root][INFO] - ===> Epoch[6430](32150/5): Loss 0.0235, Sem 0.0235 LR: 5.012e-02 Score 99.176 Data time: 0.2429, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8022 [2021-07-08 12:29:31,997][root][INFO] - ===> Epoch[6431](32155/5): Loss 0.0246, Sem 0.0246 LR: 5.011e-02 Score 99.159 Data time: 0.2132, Forward time: 1.2576, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6369 [2021-07-08 12:29:46,204][root][INFO] - ===> Epoch[6432](32160/5): Loss 0.0214, Sem 0.0214 LR: 5.010e-02 Score 99.243 Data time: 0.2121, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7956 [2021-07-08 12:30:02,194][root][INFO] - ===> Epoch[6433](32165/5): Loss 0.0209, Sem 0.0209 LR: 5.010e-02 Score 99.264 Data time: 0.2466, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 3.1536 [2021-07-08 12:30:16,108][root][INFO] - ===> Epoch[6434](32170/5): Loss 0.0211, Sem 0.0211 LR: 5.009e-02 Score 99.262 Data time: 0.2707, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7215 [2021-07-08 12:30:30,328][root][INFO] - ===> Epoch[6435](32175/5): Loss 0.0197, Sem 0.0197 LR: 5.008e-02 Score 99.301 Data time: 0.2791, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8002 [2021-07-08 12:30:44,804][root][INFO] - ===> Epoch[6436](32180/5): Loss 0.0225, Sem 0.0225 LR: 5.007e-02 Score 99.258 Data time: 0.3233, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8429 [2021-07-08 12:30:58,141][root][INFO] - ===> Epoch[6437](32185/5): Loss 0.0219, Sem 0.0219 LR: 5.006e-02 Score 99.255 Data time: 0.2193, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6253 [2021-07-08 12:31:12,142][root][INFO] - ===> Epoch[6438](32190/5): Loss 0.0209, Sem 0.0209 LR: 5.006e-02 Score 99.282 Data time: 0.1988, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7535 [2021-07-08 12:31:27,298][root][INFO] - ===> Epoch[6439](32195/5): Loss 0.0197, Sem 0.0197 LR: 5.005e-02 Score 99.310 Data time: 0.2289, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.9876 [2021-07-08 12:31:41,444][root][INFO] - ===> Epoch[6440](32200/5): Loss 0.0209, Sem 0.0209 LR: 5.004e-02 Score 99.273 Data time: 0.2590, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7773 [2021-07-08 12:31:41,446][root][INFO] - ===> Start testing [2021-07-08 12:32:15,502][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1279 Loss 0.301 (AVG: 0.489) Score 92.768 (AVG: 90.541) mIOU 70.943 mAP 84.684 mAcc 76.970 IOU: 63.022 0.000 85.804 73.652 94.357 90.462 28.911 86.115 98.474 80.834 78.964 85.258 56.404 mAP: 85.684 3.194 91.663 89.385 98.708 98.547 62.799 98.097 99.821 93.866 93.123 94.347 91.656 mAcc: 78.662 0.000 89.475 81.848 96.278 96.480 30.824 94.425 99.335 84.718 89.657 96.037 62.876 [2021-07-08 12:32:15,502][root][INFO] - Finished test. Elapsed time: 34.0554 [2021-07-08 12:32:16,743][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 12:32:29,936][root][INFO] - ===> Epoch[6441](32205/5): Loss 0.0187, Sem 0.0187 LR: 5.003e-02 Score 99.336 Data time: 0.2195, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.5984 [2021-07-08 12:32:44,733][root][INFO] - ===> Epoch[6442](32210/5): Loss 0.0199, Sem 0.0199 LR: 5.002e-02 Score 99.298 Data time: 0.2166, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.9134 [2021-07-08 12:32:58,467][root][INFO] - ===> Epoch[6443](32215/5): Loss 0.0200, Sem 0.0200 LR: 5.002e-02 Score 99.304 Data time: 0.2571, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6966 [2021-07-08 12:33:13,037][root][INFO] - ===> Epoch[6444](32220/5): Loss 0.0184, Sem 0.0184 LR: 5.001e-02 Score 99.344 Data time: 0.2650, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8618 [2021-07-08 12:33:27,655][root][INFO] - ===> Epoch[6445](32225/5): Loss 0.0180, Sem 0.0180 LR: 5.000e-02 Score 99.360 Data time: 0.2306, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8773 [2021-07-08 12:33:40,887][root][INFO] - ===> Epoch[6446](32230/5): Loss 0.0178, Sem 0.0178 LR: 4.999e-02 Score 99.362 Data time: 0.2494, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6011 [2021-07-08 12:33:55,366][root][INFO] - ===> Epoch[6447](32235/5): Loss 0.0173, Sem 0.0173 LR: 4.998e-02 Score 99.379 Data time: 0.2048, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8581 [2021-07-08 12:34:09,524][root][INFO] - ===> Epoch[6448](32240/5): Loss 0.0174, Sem 0.0174 LR: 4.997e-02 Score 99.383 Data time: 0.2003, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7885 [2021-07-08 12:34:23,867][root][INFO] - ===> Epoch[6449](32245/5): Loss 0.0175, Sem 0.0175 LR: 4.997e-02 Score 99.372 Data time: 0.2171, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8280 [2021-07-08 12:34:37,446][root][INFO] - ===> Epoch[6450](32250/5): Loss 0.0171, Sem 0.0171 LR: 4.996e-02 Score 99.388 Data time: 0.3000, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6737 [2021-07-08 12:34:51,489][root][INFO] - ===> Epoch[6451](32255/5): Loss 0.0172, Sem 0.0172 LR: 4.995e-02 Score 99.382 Data time: 0.2516, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7646 [2021-07-08 12:35:06,519][root][INFO] - ===> Epoch[6452](32260/5): Loss 0.0165, Sem 0.0165 LR: 4.994e-02 Score 99.417 Data time: 0.1861, Forward time: 1.2578, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.9554 [2021-07-08 12:35:20,134][root][INFO] - ===> Epoch[6453](32265/5): Loss 0.0182, Sem 0.0182 LR: 4.993e-02 Score 99.357 Data time: 0.2431, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6786 [2021-07-08 12:35:34,372][root][INFO] - ===> Epoch[6454](32270/5): Loss 0.0164, Sem 0.0164 LR: 4.993e-02 Score 99.405 Data time: 0.1989, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7976 [2021-07-08 12:35:47,706][root][INFO] - ===> Epoch[6455](32275/5): Loss 0.0169, Sem 0.0169 LR: 4.992e-02 Score 99.395 Data time: 0.2044, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6211 [2021-07-08 12:36:01,257][root][INFO] - ===> Epoch[6456](32280/5): Loss 0.0160, Sem 0.0160 LR: 4.991e-02 Score 99.434 Data time: 0.1882, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6536 [2021-07-08 12:36:14,584][root][INFO] - ===> Epoch[6457](32285/5): Loss 0.0163, Sem 0.0163 LR: 4.990e-02 Score 99.414 Data time: 0.2320, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6092 [2021-07-08 12:36:29,007][root][INFO] - ===> Epoch[6458](32290/5): Loss 0.0166, Sem 0.0166 LR: 4.989e-02 Score 99.408 Data time: 0.2425, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8309 [2021-07-08 12:36:42,761][root][INFO] - ===> Epoch[6459](32295/5): Loss 0.0164, Sem 0.0164 LR: 4.989e-02 Score 99.414 Data time: 0.2016, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7114 [2021-07-08 12:36:56,624][root][INFO] - ===> Epoch[6460](32300/5): Loss 0.0157, Sem 0.0157 LR: 4.988e-02 Score 99.438 Data time: 0.2197, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7289 [2021-07-08 12:37:10,776][root][INFO] - ===> Epoch[6461](32305/5): Loss 0.0163, Sem 0.0163 LR: 4.987e-02 Score 99.420 Data time: 0.2218, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7870 [2021-07-08 12:37:24,518][root][INFO] - ===> Epoch[6462](32310/5): Loss 0.0154, Sem 0.0154 LR: 4.986e-02 Score 99.445 Data time: 0.1738, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.7090 [2021-07-08 12:37:39,257][root][INFO] - ===> Epoch[6463](32315/5): Loss 0.0149, Sem 0.0149 LR: 4.985e-02 Score 99.465 Data time: 0.2629, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.8964 [2021-07-08 12:37:52,702][root][INFO] - ===> Epoch[6464](32320/5): Loss 0.0158, Sem 0.0158 LR: 4.984e-02 Score 99.432 Data time: 0.1581, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6467 [2021-07-08 12:38:07,246][root][INFO] - ===> Epoch[6465](32325/5): Loss 0.0153, Sem 0.0153 LR: 4.984e-02 Score 99.450 Data time: 0.1460, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8660 [2021-07-08 12:38:21,899][root][INFO] - ===> Epoch[6466](32330/5): Loss 0.0143, Sem 0.0143 LR: 4.983e-02 Score 99.484 Data time: 0.2280, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8911 [2021-07-08 12:38:36,499][root][INFO] - ===> Epoch[6467](32335/5): Loss 0.0143, Sem 0.0143 LR: 4.982e-02 Score 99.484 Data time: 0.2095, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8741 [2021-07-08 12:38:50,436][root][INFO] - ===> Epoch[6468](32340/5): Loss 0.0149, Sem 0.0149 LR: 4.981e-02 Score 99.466 Data time: 0.3401, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7337 [2021-07-08 12:39:04,998][root][INFO] - ===> Epoch[6469](32345/5): Loss 0.0145, Sem 0.0145 LR: 4.980e-02 Score 99.483 Data time: 0.2862, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8529 [2021-07-08 12:39:19,045][root][INFO] - ===> Epoch[6470](32350/5): Loss 0.0149, Sem 0.0149 LR: 4.980e-02 Score 99.463 Data time: 0.1950, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7649 [2021-07-08 12:39:33,372][root][INFO] - ===> Epoch[6471](32355/5): Loss 0.0155, Sem 0.0155 LR: 4.979e-02 Score 99.445 Data time: 0.2924, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8281 [2021-07-08 12:39:48,515][root][INFO] - ===> Epoch[6472](32360/5): Loss 0.0138, Sem 0.0138 LR: 4.978e-02 Score 99.500 Data time: 0.2524, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.9921 [2021-07-08 12:40:02,625][root][INFO] - ===> Epoch[6473](32365/5): Loss 0.0146, Sem 0.0146 LR: 4.977e-02 Score 99.477 Data time: 0.1841, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7878 [2021-07-08 12:40:16,220][root][INFO] - ===> Epoch[6474](32370/5): Loss 0.0146, Sem 0.0146 LR: 4.976e-02 Score 99.470 Data time: 0.2670, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.6814 [2021-07-08 12:40:29,497][root][INFO] - ===> Epoch[6475](32375/5): Loss 0.0145, Sem 0.0145 LR: 4.976e-02 Score 99.486 Data time: 0.2407, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.6056 [2021-07-08 12:40:43,428][root][INFO] - ===> Epoch[6476](32380/5): Loss 0.0144, Sem 0.0144 LR: 4.975e-02 Score 99.481 Data time: 0.1815, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7434 [2021-07-08 12:40:56,377][root][INFO] - ===> Epoch[6477](32385/5): Loss 0.0146, Sem 0.0146 LR: 4.974e-02 Score 99.474 Data time: 0.2019, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.5423 [2021-07-08 12:41:10,160][root][INFO] - ===> Epoch[6478](32390/5): Loss 0.0151, Sem 0.0151 LR: 4.973e-02 Score 99.453 Data time: 0.1869, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7172 [2021-07-08 12:41:24,221][root][INFO] - ===> Epoch[6479](32395/5): Loss 0.0145, Sem 0.0145 LR: 4.972e-02 Score 99.482 Data time: 0.2212, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.7652 [2021-07-08 12:41:37,776][root][INFO] - ===> Epoch[6480](32400/5): Loss 0.0147, Sem 0.0147 LR: 4.972e-02 Score 99.469 Data time: 0.2391, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.6678 [2021-07-08 12:41:37,777][root][INFO] - ===> Start testing [2021-07-08 12:42:22,301][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1312 Loss 0.266 (AVG: 0.543) Score 93.873 (AVG: 90.906) mIOU 71.485 mAP 84.361 mAcc 77.178 IOU: 64.584 0.000 85.365 74.527 94.936 92.265 25.000 85.927 98.606 84.306 80.214 85.918 57.654 mAP: 85.225 3.586 91.370 87.059 98.699 98.980 58.986 98.384 99.821 94.932 93.487 94.406 91.764 mAcc: 81.921 0.000 89.372 81.538 96.107 97.112 26.122 95.186 99.485 87.391 90.595 96.440 62.048 [2021-07-08 12:42:22,301][root][INFO] - Finished test. Elapsed time: 44.5229 [2021-07-08 12:42:23,391][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 12:42:36,800][root][INFO] - ===> Epoch[6481](32405/5): Loss 0.0145, Sem 0.0145 LR: 4.971e-02 Score 99.482 Data time: 0.1780, Forward time: 1.2577, Backward time: 1.2404, DDP time: 0.0042, Total iter time: 2.6427 [2021-07-08 12:42:51,424][root][INFO] - ===> Epoch[6482](32410/5): Loss 0.0146, Sem 0.0146 LR: 4.970e-02 Score 99.472 Data time: 0.2106, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8759 [2021-07-08 12:43:05,984][root][INFO] - ===> Epoch[6483](32415/5): Loss 0.0152, Sem 0.0152 LR: 4.969e-02 Score 99.457 Data time: 0.2284, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8607 [2021-07-08 12:43:20,235][root][INFO] - ===> Epoch[6484](32420/5): Loss 0.0137, Sem 0.0137 LR: 4.968e-02 Score 99.507 Data time: 0.2046, Forward time: 1.2577, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8066 [2021-07-08 12:43:34,450][root][INFO] - ===> Epoch[6485](32425/5): Loss 0.0146, Sem 0.0146 LR: 4.967e-02 Score 99.478 Data time: 0.1561, Forward time: 1.2576, Backward time: 1.2405, DDP time: 0.0042, Total iter time: 2.8009 [2021-07-08 12:43:48,969][root][INFO] - ===> Epoch[6486](32430/5): Loss 0.0147, Sem 0.0147 LR: 4.967e-02 Score 99.472 Data time: 0.1884, Forward time: 1.2576, Backward time: 1.2406, DDP time: 0.0042, Total iter time: 2.8695 [2021-07-08 12:44:02,818][root][INFO] - ===> Epoch[6487](32435/5): Loss 0.0144, Sem 0.0144 LR: 4.966e-02 Score 99.484 Data time: 0.1946, Forward time: 1.2576, Backward time: 1.2406, DDP time: 0.0042, Total iter time: 2.7146 [2021-07-08 12:44:17,092][root][INFO] - ===> Epoch[6488](32440/5): Loss 0.0146, Sem 0.0146 LR: 4.965e-02 Score 99.473 Data time: 0.2123, Forward time: 1.2576, Backward time: 1.2406, DDP time: 0.0042, Total iter time: 2.7955 [2021-07-08 12:44:31,722][root][INFO] - ===> Epoch[6489](32445/5): Loss 0.0135, Sem 0.0135 LR: 4.964e-02 Score 99.511 Data time: 0.2129, Forward time: 1.2576, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.8766 [2021-07-08 12:44:45,692][root][INFO] - ===> Epoch[6490](32450/5): Loss 0.0150, Sem 0.0150 LR: 4.963e-02 Score 99.461 Data time: 0.2265, Forward time: 1.2575, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.7453 [2021-07-08 12:44:59,827][root][INFO] - ===> Epoch[6491](32455/5): Loss 0.0143, Sem 0.0143 LR: 4.963e-02 Score 99.488 Data time: 0.2361, Forward time: 1.2575, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.7817 [2021-07-08 12:45:13,450][root][INFO] - ===> Epoch[6492](32460/5): Loss 0.0146, Sem 0.0146 LR: 4.962e-02 Score 99.483 Data time: 0.1707, Forward time: 1.2575, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.6835 [2021-07-08 12:45:27,259][root][INFO] - ===> Epoch[6493](32465/5): Loss 0.0141, Sem 0.0141 LR: 4.961e-02 Score 99.490 Data time: 0.1964, Forward time: 1.2575, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.7219 [2021-07-08 12:45:41,085][root][INFO] - ===> Epoch[6494](32470/5): Loss 0.0139, Sem 0.0139 LR: 4.960e-02 Score 99.498 Data time: 0.2091, Forward time: 1.2575, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.7220 [2021-07-08 12:45:54,638][root][INFO] - ===> Epoch[6495](32475/5): Loss 0.0139, Sem 0.0139 LR: 4.959e-02 Score 99.503 Data time: 0.1980, Forward time: 1.2575, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.6602 [2021-07-08 12:46:07,904][root][INFO] - ===> Epoch[6496](32480/5): Loss 0.0139, Sem 0.0139 LR: 4.959e-02 Score 99.492 Data time: 0.1976, Forward time: 1.2575, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.6018 [2021-07-08 12:46:23,015][root][INFO] - ===> Epoch[6497](32485/5): Loss 0.0135, Sem 0.0135 LR: 4.958e-02 Score 99.509 Data time: 0.1983, Forward time: 1.2575, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.9790 [2021-07-08 12:46:37,709][root][INFO] - ===> Epoch[6498](32490/5): Loss 0.0140, Sem 0.0140 LR: 4.957e-02 Score 99.498 Data time: 0.2412, Forward time: 1.2576, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.8967 [2021-07-08 12:46:52,188][root][INFO] - ===> Epoch[6499](32495/5): Loss 0.0135, Sem 0.0135 LR: 4.956e-02 Score 99.511 Data time: 0.2279, Forward time: 1.2575, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.8498 [2021-07-08 12:47:06,881][root][INFO] - ===> Epoch[6500](32500/5): Loss 0.0136, Sem 0.0136 LR: 4.955e-02 Score 99.509 Data time: 0.2609, Forward time: 1.2575, Backward time: 1.2408, DDP time: 0.0042, Total iter time: 2.8842 [2021-07-08 12:47:21,551][root][INFO] - ===> Epoch[6501](32505/5): Loss 0.0135, Sem 0.0135 LR: 4.955e-02 Score 99.514 Data time: 0.2217, Forward time: 1.2575, Backward time: 1.2408, DDP time: 0.0042, Total iter time: 2.8817 [2021-07-08 12:47:35,783][root][INFO] - ===> Epoch[6502](32510/5): Loss 0.0137, Sem 0.0137 LR: 4.954e-02 Score 99.502 Data time: 0.2834, Forward time: 1.2575, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.7884 [2021-07-08 12:47:49,483][root][INFO] - ===> Epoch[6503](32515/5): Loss 0.0139, Sem 0.0139 LR: 4.953e-02 Score 99.504 Data time: 0.2242, Forward time: 1.2575, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.6928 [2021-07-08 12:48:04,181][root][INFO] - ===> Epoch[6504](32520/5): Loss 0.0137, Sem 0.0137 LR: 4.952e-02 Score 99.510 Data time: 0.2881, Forward time: 1.2576, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.8898 [2021-07-08 12:48:18,264][root][INFO] - ===> Epoch[6505](32525/5): Loss 0.0136, Sem 0.0136 LR: 4.951e-02 Score 99.512 Data time: 0.2636, Forward time: 1.2576, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.7606 [2021-07-08 12:48:31,877][root][INFO] - ===> Epoch[6506](32530/5): Loss 0.0137, Sem 0.0137 LR: 4.950e-02 Score 99.512 Data time: 0.2298, Forward time: 1.2576, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.6751 [2021-07-08 12:48:46,197][root][INFO] - ===> Epoch[6507](32535/5): Loss 0.0137, Sem 0.0137 LR: 4.950e-02 Score 99.506 Data time: 0.2283, Forward time: 1.2576, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.8176 [2021-07-08 12:49:00,410][root][INFO] - ===> Epoch[6508](32540/5): Loss 0.0140, Sem 0.0140 LR: 4.949e-02 Score 99.500 Data time: 0.2523, Forward time: 1.2576, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.7987 [2021-07-08 12:49:14,577][root][INFO] - ===> Epoch[6509](32545/5): Loss 0.0132, Sem 0.0132 LR: 4.948e-02 Score 99.521 Data time: 0.2023, Forward time: 1.2576, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.7923 [2021-07-08 12:49:29,498][root][INFO] - ===> Epoch[6510](32550/5): Loss 0.0139, Sem 0.0139 LR: 4.947e-02 Score 99.493 Data time: 0.2363, Forward time: 1.2576, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.9329 [2021-07-08 12:49:42,722][root][INFO] - ===> Epoch[6511](32555/5): Loss 0.0132, Sem 0.0132 LR: 4.946e-02 Score 99.525 Data time: 0.1975, Forward time: 1.2576, Backward time: 1.2407, DDP time: 0.0042, Total iter time: 2.6002 [2021-07-08 12:49:57,382][root][INFO] - ===> Epoch[6512](32560/5): Loss 0.0139, Sem 0.0139 LR: 4.946e-02 Score 99.502 Data time: 0.2273, Forward time: 1.2575, Backward time: 1.2408, DDP time: 0.0042, Total iter time: 2.8900 [2021-07-08 12:50:11,902][root][INFO] - ===> Epoch[6513](32565/5): Loss 0.0145, Sem 0.0145 LR: 4.945e-02 Score 99.503 Data time: 0.2497, Forward time: 1.2576, Backward time: 1.2408, DDP time: 0.0042, Total iter time: 2.8586 [2021-07-08 12:50:26,098][root][INFO] - ===> Epoch[6514](32570/5): Loss 0.0137, Sem 0.0137 LR: 4.944e-02 Score 99.507 Data time: 0.2081, Forward time: 1.2575, Backward time: 1.2408, DDP time: 0.0042, Total iter time: 2.7967 [2021-07-08 12:50:39,912][root][INFO] - ===> Epoch[6515](32575/5): Loss 0.0127, Sem 0.0127 LR: 4.943e-02 Score 99.547 Data time: 0.1902, Forward time: 1.2575, Backward time: 1.2408, DDP time: 0.0042, Total iter time: 2.7226 [2021-07-08 12:50:54,589][root][INFO] - ===> Epoch[6516](32580/5): Loss 0.0138, Sem 0.0138 LR: 4.942e-02 Score 99.496 Data time: 0.1860, Forward time: 1.2575, Backward time: 1.2408, DDP time: 0.0042, Total iter time: 2.8975 [2021-07-08 12:51:07,769][root][INFO] - ===> Epoch[6517](32585/5): Loss 0.0137, Sem 0.0137 LR: 4.942e-02 Score 99.503 Data time: 0.1475, Forward time: 1.2575, Backward time: 1.2408, DDP time: 0.0042, Total iter time: 2.5889 [2021-07-08 12:51:22,940][root][INFO] - ===> Epoch[6518](32590/5): Loss 0.0131, Sem 0.0131 LR: 4.941e-02 Score 99.529 Data time: 0.2340, Forward time: 1.2575, Backward time: 1.2409, DDP time: 0.0042, Total iter time: 2.9827 [2021-07-08 12:51:36,952][root][INFO] - ===> Epoch[6519](32595/5): Loss 0.0132, Sem 0.0132 LR: 4.940e-02 Score 99.521 Data time: 0.1887, Forward time: 1.2575, Backward time: 1.2409, DDP time: 0.0042, Total iter time: 2.7580 [2021-07-08 12:51:50,459][root][INFO] - ===> Epoch[6520](32600/5): Loss 0.0144, Sem 0.0144 LR: 4.939e-02 Score 99.481 Data time: 0.1985, Forward time: 1.2575, Backward time: 1.2409, DDP time: 0.0042, Total iter time: 2.6605 [2021-07-08 12:51:50,460][root][INFO] - ===> Start testing [2021-07-08 12:52:36,732][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1321 Loss 0.312 (AVG: 0.524) Score 93.305 (AVG: 90.984) mIOU 72.032 mAP 84.587 mAcc 78.122 IOU: 63.721 0.000 85.303 74.717 94.721 91.760 37.127 85.216 98.550 80.465 80.108 86.497 58.236 mAP: 85.077 3.562 93.474 87.343 98.543 98.816 67.050 98.467 99.821 88.669 92.860 94.562 91.386 mAcc: 79.938 0.000 91.216 83.128 95.917 96.858 40.944 95.078 99.341 83.593 89.918 95.951 63.709 [2021-07-08 12:52:36,732][root][INFO] - Finished test. Elapsed time: 46.2719 [2021-07-08 12:52:37,827][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 12:52:50,723][root][INFO] - ===> Epoch[6521](32605/5): Loss 0.0138, Sem 0.0138 LR: 4.938e-02 Score 99.507 Data time: 0.2734, Forward time: 1.2575, Backward time: 1.2408, DDP time: 0.0042, Total iter time: 2.5408 [2021-07-08 12:53:04,453][root][INFO] - ===> Epoch[6522](32610/5): Loss 0.0137, Sem 0.0137 LR: 4.937e-02 Score 99.507 Data time: 0.2541, Forward time: 1.2575, Backward time: 1.2408, DDP time: 0.0042, Total iter time: 2.6948 [2021-07-08 12:53:18,080][root][INFO] - ===> Epoch[6523](32615/5): Loss 0.0141, Sem 0.0141 LR: 4.937e-02 Score 99.492 Data time: 0.2022, Forward time: 1.2574, Backward time: 1.2409, DDP time: 0.0042, Total iter time: 2.6778 [2021-07-08 12:53:31,343][root][INFO] - ===> Epoch[6524](32620/5): Loss 0.0135, Sem 0.0135 LR: 4.936e-02 Score 99.513 Data time: 0.2437, Forward time: 1.2574, Backward time: 1.2408, DDP time: 0.0042, Total iter time: 2.6112 [2021-07-08 12:53:45,600][root][INFO] - ===> Epoch[6525](32625/5): Loss 0.0136, Sem 0.0136 LR: 4.935e-02 Score 99.515 Data time: 0.2884, Forward time: 1.2574, Backward time: 1.2408, DDP time: 0.0042, Total iter time: 2.8011 [2021-07-08 12:53:59,672][root][INFO] - ===> Epoch[6526](32630/5): Loss 0.0130, Sem 0.0130 LR: 4.934e-02 Score 99.534 Data time: 0.2417, Forward time: 1.2574, Backward time: 1.2409, DDP time: 0.0042, Total iter time: 2.7684 [2021-07-08 12:54:14,234][root][INFO] - ===> Epoch[6527](32635/5): Loss 0.0139, Sem 0.0139 LR: 4.933e-02 Score 99.502 Data time: 0.2205, Forward time: 1.2574, Backward time: 1.2409, DDP time: 0.0042, Total iter time: 2.8563 [2021-07-08 12:54:27,529][root][INFO] - ===> Epoch[6528](32640/5): Loss 0.0139, Sem 0.0139 LR: 4.933e-02 Score 99.498 Data time: 0.1726, Forward time: 1.2574, Backward time: 1.2409, DDP time: 0.0042, Total iter time: 2.6197 [2021-07-08 12:54:41,104][root][INFO] - ===> Epoch[6529](32645/5): Loss 0.0128, Sem 0.0128 LR: 4.932e-02 Score 99.537 Data time: 0.2074, Forward time: 1.2574, Backward time: 1.2409, DDP time: 0.0042, Total iter time: 2.6702 [2021-07-08 12:54:55,573][root][INFO] - ===> Epoch[6530](32650/5): Loss 0.0130, Sem 0.0130 LR: 4.931e-02 Score 99.531 Data time: 0.1916, Forward time: 1.2574, Backward time: 1.2409, DDP time: 0.0042, Total iter time: 2.8558 [2021-07-08 12:55:09,265][root][INFO] - ===> Epoch[6531](32655/5): Loss 0.0136, Sem 0.0136 LR: 4.930e-02 Score 99.526 Data time: 0.2052, Forward time: 1.2574, Backward time: 1.2409, DDP time: 0.0042, Total iter time: 2.6913 [2021-07-08 12:55:23,915][root][INFO] - ===> Epoch[6532](32660/5): Loss 0.0153, Sem 0.0153 LR: 4.929e-02 Score 99.474 Data time: 0.2697, Forward time: 1.2574, Backward time: 1.2409, DDP time: 0.0042, Total iter time: 2.8726 [2021-07-08 12:55:38,591][root][INFO] - ===> Epoch[6533](32665/5): Loss 0.0129, Sem 0.0129 LR: 4.929e-02 Score 99.539 Data time: 0.2184, Forward time: 1.2574, Backward time: 1.2409, DDP time: 0.0042, Total iter time: 2.8908 [2021-07-08 12:55:52,373][root][INFO] - ===> Epoch[6534](32670/5): Loss 0.0132, Sem 0.0132 LR: 4.928e-02 Score 99.521 Data time: 0.2191, Forward time: 1.2574, Backward time: 1.2409, DDP time: 0.0042, Total iter time: 2.7135 [2021-07-08 12:56:07,414][root][INFO] - ===> Epoch[6535](32675/5): Loss 0.0134, Sem 0.0134 LR: 4.927e-02 Score 99.517 Data time: 0.2242, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.9627 [2021-07-08 12:56:20,328][root][INFO] - ===> Epoch[6536](32680/5): Loss 0.0133, Sem 0.0133 LR: 4.926e-02 Score 99.524 Data time: 0.1956, Forward time: 1.2574, Backward time: 1.2409, DDP time: 0.0042, Total iter time: 2.5406 [2021-07-08 12:56:34,933][root][INFO] - ===> Epoch[6537](32685/5): Loss 0.0146, Sem 0.0146 LR: 4.925e-02 Score 99.497 Data time: 0.2596, Forward time: 1.2574, Backward time: 1.2409, DDP time: 0.0042, Total iter time: 2.8752 [2021-07-08 12:56:49,979][root][INFO] - ===> Epoch[6538](32690/5): Loss 0.0131, Sem 0.0131 LR: 4.924e-02 Score 99.532 Data time: 0.2826, Forward time: 1.2574, Backward time: 1.2409, DDP time: 0.0042, Total iter time: 2.9610 [2021-07-08 12:57:04,690][root][INFO] - ===> Epoch[6539](32695/5): Loss 0.0134, Sem 0.0134 LR: 4.924e-02 Score 99.521 Data time: 0.2171, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.8911 [2021-07-08 12:57:19,212][root][INFO] - ===> Epoch[6540](32700/5): Loss 0.0135, Sem 0.0135 LR: 4.923e-02 Score 99.518 Data time: 0.2049, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.8679 [2021-07-08 12:57:34,470][root][INFO] - ===> Epoch[6541](32705/5): Loss 0.0132, Sem 0.0132 LR: 4.922e-02 Score 99.530 Data time: 0.1389, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 3.0178 [2021-07-08 12:57:48,128][root][INFO] - ===> Epoch[6542](32710/5): Loss 0.0138, Sem 0.0138 LR: 4.921e-02 Score 99.507 Data time: 0.2270, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.6833 [2021-07-08 12:58:02,043][root][INFO] - ===> Epoch[6543](32715/5): Loss 0.0139, Sem 0.0139 LR: 4.920e-02 Score 99.507 Data time: 0.2281, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.7237 [2021-07-08 12:58:16,835][root][INFO] - ===> Epoch[6544](32720/5): Loss 0.0147, Sem 0.0147 LR: 4.920e-02 Score 99.488 Data time: 0.2104, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.9169 [2021-07-08 12:58:30,876][root][INFO] - ===> Epoch[6545](32725/5): Loss 0.0136, Sem 0.0136 LR: 4.919e-02 Score 99.514 Data time: 0.1717, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7592 [2021-07-08 12:58:45,825][root][INFO] - ===> Epoch[6546](32730/5): Loss 0.0141, Sem 0.0141 LR: 4.918e-02 Score 99.501 Data time: 0.2817, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.9316 [2021-07-08 12:59:00,407][root][INFO] - ===> Epoch[6547](32735/5): Loss 0.0129, Sem 0.0129 LR: 4.917e-02 Score 99.543 Data time: 0.2278, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.8756 [2021-07-08 12:59:14,299][root][INFO] - ===> Epoch[6548](32740/5): Loss 0.0139, Sem 0.0139 LR: 4.916e-02 Score 99.492 Data time: 0.2106, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7356 [2021-07-08 12:59:28,011][root][INFO] - ===> Epoch[6549](32745/5): Loss 0.0129, Sem 0.0129 LR: 4.916e-02 Score 99.533 Data time: 0.2388, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6958 [2021-07-08 12:59:41,593][root][INFO] - ===> Epoch[6550](32750/5): Loss 0.0134, Sem 0.0134 LR: 4.915e-02 Score 99.516 Data time: 0.2449, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.6564 [2021-07-08 12:59:56,116][root][INFO] - ===> Epoch[6551](32755/5): Loss 0.0129, Sem 0.0129 LR: 4.914e-02 Score 99.530 Data time: 0.2080, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.8585 [2021-07-08 13:00:11,021][root][INFO] - ===> Epoch[6552](32760/5): Loss 0.0133, Sem 0.0133 LR: 4.913e-02 Score 99.524 Data time: 0.2031, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.9411 [2021-07-08 13:00:25,571][root][INFO] - ===> Epoch[6553](32765/5): Loss 0.0131, Sem 0.0131 LR: 4.912e-02 Score 99.531 Data time: 0.2379, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.8646 [2021-07-08 13:00:40,187][root][INFO] - ===> Epoch[6554](32770/5): Loss 0.0138, Sem 0.0138 LR: 4.912e-02 Score 99.497 Data time: 0.2906, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.8775 [2021-07-08 13:00:53,677][root][INFO] - ===> Epoch[6555](32775/5): Loss 0.0132, Sem 0.0132 LR: 4.911e-02 Score 99.530 Data time: 0.2583, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6455 [2021-07-08 13:01:07,344][root][INFO] - ===> Epoch[6556](32780/5): Loss 0.0128, Sem 0.0128 LR: 4.910e-02 Score 99.542 Data time: 0.2120, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6897 [2021-07-08 13:01:22,044][root][INFO] - ===> Epoch[6557](32785/5): Loss 0.0133, Sem 0.0133 LR: 4.909e-02 Score 99.523 Data time: 0.2300, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.8899 [2021-07-08 13:01:35,902][root][INFO] - ===> Epoch[6558](32790/5): Loss 0.0124, Sem 0.0124 LR: 4.908e-02 Score 99.554 Data time: 0.2142, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7223 [2021-07-08 13:01:49,578][root][INFO] - ===> Epoch[6559](32795/5): Loss 0.0130, Sem 0.0130 LR: 4.907e-02 Score 99.531 Data time: 0.2369, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6964 [2021-07-08 13:02:03,344][root][INFO] - ===> Epoch[6560](32800/5): Loss 0.0126, Sem 0.0126 LR: 4.907e-02 Score 99.543 Data time: 0.1908, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7128 [2021-07-08 13:02:03,345][root][INFO] - ===> Start testing [2021-07-08 13:02:36,556][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1238 Loss 0.278 (AVG: 0.554) Score 94.330 (AVG: 91.031) mIOU 71.461 mAP 84.374 mAcc 77.176 IOU: 65.023 0.000 85.761 74.256 95.444 91.773 32.800 84.334 98.631 77.627 80.262 86.226 56.859 mAP: 84.925 7.231 91.751 86.956 98.833 98.993 61.679 98.240 99.825 90.703 92.693 94.588 90.439 mAcc: 80.645 0.000 88.935 81.811 96.622 96.895 35.487 94.160 99.484 80.170 90.598 96.562 61.913 [2021-07-08 13:02:36,556][root][INFO] - Finished test. Elapsed time: 33.2105 [2021-07-08 13:02:37,621][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 13:02:49,933][root][INFO] - ===> Epoch[6561](32805/5): Loss 0.0124, Sem 0.0124 LR: 4.906e-02 Score 99.552 Data time: 0.1838, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.4250 [2021-07-08 13:03:03,409][root][INFO] - ===> Epoch[6562](32810/5): Loss 0.0129, Sem 0.0129 LR: 4.905e-02 Score 99.537 Data time: 0.2478, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.6301 [2021-07-08 13:03:17,022][root][INFO] - ===> Epoch[6563](32815/5): Loss 0.0119, Sem 0.0119 LR: 4.904e-02 Score 99.572 Data time: 0.2094, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.6756 [2021-07-08 13:03:30,570][root][INFO] - ===> Epoch[6564](32820/5): Loss 0.0128, Sem 0.0128 LR: 4.903e-02 Score 99.536 Data time: 0.2598, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.6584 [2021-07-08 13:03:43,968][root][INFO] - ===> Epoch[6565](32825/5): Loss 0.0129, Sem 0.0129 LR: 4.903e-02 Score 99.535 Data time: 0.2046, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.6335 [2021-07-08 13:03:58,241][root][INFO] - ===> Epoch[6566](32830/5): Loss 0.0124, Sem 0.0124 LR: 4.902e-02 Score 99.554 Data time: 0.1434, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.8173 [2021-07-08 13:04:12,057][root][INFO] - ===> Epoch[6567](32835/5): Loss 0.0127, Sem 0.0127 LR: 4.901e-02 Score 99.544 Data time: 0.1748, Forward time: 1.2573, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.7089 [2021-07-08 13:04:25,693][root][INFO] - ===> Epoch[6568](32840/5): Loss 0.0132, Sem 0.0132 LR: 4.900e-02 Score 99.527 Data time: 0.2332, Forward time: 1.2573, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.6823 [2021-07-08 13:04:40,437][root][INFO] - ===> Epoch[6569](32845/5): Loss 0.0129, Sem 0.0129 LR: 4.899e-02 Score 99.539 Data time: 0.2526, Forward time: 1.2573, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.9016 [2021-07-08 13:04:55,034][root][INFO] - ===> Epoch[6570](32850/5): Loss 0.0130, Sem 0.0130 LR: 4.899e-02 Score 99.532 Data time: 0.2510, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.8707 [2021-07-08 13:05:09,491][root][INFO] - ===> Epoch[6571](32855/5): Loss 0.0125, Sem 0.0125 LR: 4.898e-02 Score 99.548 Data time: 0.2025, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.8433 [2021-07-08 13:05:23,820][root][INFO] - ===> Epoch[6572](32860/5): Loss 0.0129, Sem 0.0129 LR: 4.897e-02 Score 99.537 Data time: 0.2322, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.8170 [2021-07-08 13:05:37,837][root][INFO] - ===> Epoch[6573](32865/5): Loss 0.0124, Sem 0.0124 LR: 4.896e-02 Score 99.551 Data time: 0.2202, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.7588 [2021-07-08 13:05:51,496][root][INFO] - ===> Epoch[6574](32870/5): Loss 0.0122, Sem 0.0122 LR: 4.895e-02 Score 99.558 Data time: 0.2137, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.6847 [2021-07-08 13:06:05,838][root][INFO] - ===> Epoch[6575](32875/5): Loss 0.0131, Sem 0.0131 LR: 4.894e-02 Score 99.525 Data time: 0.1904, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.8246 [2021-07-08 13:06:19,889][root][INFO] - ===> Epoch[6576](32880/5): Loss 0.0123, Sem 0.0123 LR: 4.894e-02 Score 99.555 Data time: 0.2258, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.7654 [2021-07-08 13:06:33,401][root][INFO] - ===> Epoch[6577](32885/5): Loss 0.0124, Sem 0.0124 LR: 4.893e-02 Score 99.553 Data time: 0.2476, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.6579 [2021-07-08 13:06:47,747][root][INFO] - ===> Epoch[6578](32890/5): Loss 0.0132, Sem 0.0132 LR: 4.892e-02 Score 99.530 Data time: 0.2111, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.8261 [2021-07-08 13:07:02,159][root][INFO] - ===> Epoch[6579](32895/5): Loss 0.0132, Sem 0.0132 LR: 4.891e-02 Score 99.527 Data time: 0.2330, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.8307 [2021-07-08 13:07:15,811][root][INFO] - ===> Epoch[6580](32900/5): Loss 0.0129, Sem 0.0129 LR: 4.890e-02 Score 99.540 Data time: 0.1957, Forward time: 1.2574, Backward time: 1.2410, DDP time: 0.0042, Total iter time: 2.6726 [2021-07-08 13:07:31,133][root][INFO] - ===> Epoch[6581](32905/5): Loss 0.0127, Sem 0.0127 LR: 4.890e-02 Score 99.538 Data time: 0.2236, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 3.0183 [2021-07-08 13:07:45,039][root][INFO] - ===> Epoch[6582](32910/5): Loss 0.0131, Sem 0.0131 LR: 4.889e-02 Score 99.544 Data time: 0.2350, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7293 [2021-07-08 13:07:59,167][root][INFO] - ===> Epoch[6583](32915/5): Loss 0.0136, Sem 0.0136 LR: 4.888e-02 Score 99.523 Data time: 0.2245, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7818 [2021-07-08 13:08:13,171][root][INFO] - ===> Epoch[6584](32920/5): Loss 0.0125, Sem 0.0125 LR: 4.887e-02 Score 99.554 Data time: 0.1925, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7506 [2021-07-08 13:08:26,610][root][INFO] - ===> Epoch[6585](32925/5): Loss 0.0131, Sem 0.0131 LR: 4.886e-02 Score 99.530 Data time: 0.2158, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6462 [2021-07-08 13:08:41,234][root][INFO] - ===> Epoch[6586](32930/5): Loss 0.0131, Sem 0.0131 LR: 4.886e-02 Score 99.530 Data time: 0.2321, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.8747 [2021-07-08 13:08:55,943][root][INFO] - ===> Epoch[6587](32935/5): Loss 0.0135, Sem 0.0135 LR: 4.885e-02 Score 99.518 Data time: 0.2518, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.8914 [2021-07-08 13:09:09,535][root][INFO] - ===> Epoch[6588](32940/5): Loss 0.0126, Sem 0.0126 LR: 4.884e-02 Score 99.548 Data time: 0.2389, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6790 [2021-07-08 13:09:23,409][root][INFO] - ===> Epoch[6589](32945/5): Loss 0.0129, Sem 0.0129 LR: 4.883e-02 Score 99.531 Data time: 0.1669, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7310 [2021-07-08 13:09:36,817][root][INFO] - ===> Epoch[6590](32950/5): Loss 0.0133, Sem 0.0133 LR: 4.882e-02 Score 99.526 Data time: 0.1986, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6426 [2021-07-08 13:09:51,070][root][INFO] - ===> Epoch[6591](32955/5): Loss 0.0124, Sem 0.0124 LR: 4.881e-02 Score 99.555 Data time: 0.2248, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.8128 [2021-07-08 13:10:04,717][root][INFO] - ===> Epoch[6592](32960/5): Loss 0.0131, Sem 0.0131 LR: 4.881e-02 Score 99.527 Data time: 0.2054, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.6844 [2021-07-08 13:10:18,310][root][INFO] - ===> Epoch[6593](32965/5): Loss 0.0126, Sem 0.0126 LR: 4.880e-02 Score 99.549 Data time: 0.2758, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6647 [2021-07-08 13:10:32,479][root][INFO] - ===> Epoch[6594](32970/5): Loss 0.0131, Sem 0.0131 LR: 4.879e-02 Score 99.527 Data time: 0.1859, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.7929 [2021-07-08 13:10:45,996][root][INFO] - ===> Epoch[6595](32975/5): Loss 0.0125, Sem 0.0125 LR: 4.878e-02 Score 99.552 Data time: 0.2486, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6524 [2021-07-08 13:10:59,971][root][INFO] - ===> Epoch[6596](32980/5): Loss 0.0131, Sem 0.0131 LR: 4.877e-02 Score 99.525 Data time: 0.2401, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7406 [2021-07-08 13:11:14,258][root][INFO] - ===> Epoch[6597](32985/5): Loss 0.0119, Sem 0.0119 LR: 4.877e-02 Score 99.572 Data time: 0.2391, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.8077 [2021-07-08 13:11:28,621][root][INFO] - ===> Epoch[6598](32990/5): Loss 0.0121, Sem 0.0121 LR: 4.876e-02 Score 99.563 Data time: 0.2377, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.8285 [2021-07-08 13:11:42,747][root][INFO] - ===> Epoch[6599](32995/5): Loss 0.0125, Sem 0.0125 LR: 4.875e-02 Score 99.549 Data time: 0.2264, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7797 [2021-07-08 13:11:56,928][root][INFO] - ===> Epoch[6600](33000/5): Loss 0.0123, Sem 0.0123 LR: 4.874e-02 Score 99.549 Data time: 0.2516, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7903 [2021-07-08 13:11:56,930][root][INFO] - ===> Start testing [2021-07-08 13:12:36,576][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1252 Loss 0.262 (AVG: 0.567) Score 94.105 (AVG: 90.726) mIOU 71.016 mAP 83.743 mAcc 77.148 IOU: 63.512 0.000 85.613 74.045 94.626 92.075 27.771 81.858 98.586 80.840 79.085 86.217 58.979 mAP: 84.138 4.098 90.486 85.989 98.563 98.702 59.239 98.106 99.844 92.418 92.458 94.081 90.539 mAcc: 81.041 0.000 88.735 81.380 96.006 96.829 30.234 94.211 99.377 84.915 90.821 96.033 63.339 [2021-07-08 13:12:36,576][root][INFO] - Finished test. Elapsed time: 39.6461 [2021-07-08 13:12:37,668][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 13:12:49,842][root][INFO] - ===> Epoch[6601](33005/5): Loss 0.0128, Sem 0.0128 LR: 4.873e-02 Score 99.547 Data time: 0.1406, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.4053 [2021-07-08 13:13:03,529][root][INFO] - ===> Epoch[6602](33010/5): Loss 0.0131, Sem 0.0131 LR: 4.873e-02 Score 99.529 Data time: 0.1936, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6917 [2021-07-08 13:13:17,685][root][INFO] - ===> Epoch[6603](33015/5): Loss 0.0122, Sem 0.0122 LR: 4.872e-02 Score 99.559 Data time: 0.2303, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7813 [2021-07-08 13:13:31,836][root][INFO] - ===> Epoch[6604](33020/5): Loss 0.0124, Sem 0.0124 LR: 4.871e-02 Score 99.551 Data time: 0.2129, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7797 [2021-07-08 13:13:45,370][root][INFO] - ===> Epoch[6605](33025/5): Loss 0.0119, Sem 0.0119 LR: 4.870e-02 Score 99.569 Data time: 0.2559, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6642 [2021-07-08 13:14:00,322][root][INFO] - ===> Epoch[6606](33030/5): Loss 0.0124, Sem 0.0124 LR: 4.869e-02 Score 99.554 Data time: 0.2695, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.9426 [2021-07-08 13:14:14,339][root][INFO] - ===> Epoch[6607](33035/5): Loss 0.0120, Sem 0.0120 LR: 4.868e-02 Score 99.564 Data time: 0.2207, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7552 [2021-07-08 13:14:28,528][root][INFO] - ===> Epoch[6608](33040/5): Loss 0.0124, Sem 0.0124 LR: 4.868e-02 Score 99.552 Data time: 0.2756, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7866 [2021-07-08 13:14:43,333][root][INFO] - ===> Epoch[6609](33045/5): Loss 0.0120, Sem 0.0120 LR: 4.867e-02 Score 99.566 Data time: 0.2218, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.9111 [2021-07-08 13:14:57,785][root][INFO] - ===> Epoch[6610](33050/5): Loss 0.0115, Sem 0.0115 LR: 4.866e-02 Score 99.584 Data time: 0.1950, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.8430 [2021-07-08 13:15:11,346][root][INFO] - ===> Epoch[6611](33055/5): Loss 0.0127, Sem 0.0127 LR: 4.865e-02 Score 99.547 Data time: 0.2204, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6632 [2021-07-08 13:15:25,053][root][INFO] - ===> Epoch[6612](33060/5): Loss 0.0121, Sem 0.0121 LR: 4.864e-02 Score 99.566 Data time: 0.2168, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6947 [2021-07-08 13:15:39,409][root][INFO] - ===> Epoch[6613](33065/5): Loss 0.0121, Sem 0.0121 LR: 4.864e-02 Score 99.562 Data time: 0.2232, Forward time: 1.2574, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.8278 [2021-07-08 13:15:53,021][root][INFO] - ===> Epoch[6614](33070/5): Loss 0.0117, Sem 0.0117 LR: 4.863e-02 Score 99.576 Data time: 0.2134, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6792 [2021-07-08 13:16:07,174][root][INFO] - ===> Epoch[6615](33075/5): Loss 0.0118, Sem 0.0118 LR: 4.862e-02 Score 99.572 Data time: 0.2233, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7892 [2021-07-08 13:16:20,884][root][INFO] - ===> Epoch[6616](33080/5): Loss 0.0123, Sem 0.0123 LR: 4.861e-02 Score 99.554 Data time: 0.2227, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6896 [2021-07-08 13:16:34,491][root][INFO] - ===> Epoch[6617](33085/5): Loss 0.0121, Sem 0.0121 LR: 4.860e-02 Score 99.565 Data time: 0.2224, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6758 [2021-07-08 13:16:49,239][root][INFO] - ===> Epoch[6618](33090/5): Loss 0.0113, Sem 0.0113 LR: 4.860e-02 Score 99.593 Data time: 0.2109, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.9024 [2021-07-08 13:17:03,064][root][INFO] - ===> Epoch[6619](33095/5): Loss 0.0121, Sem 0.0121 LR: 4.859e-02 Score 99.561 Data time: 0.1815, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7192 [2021-07-08 13:17:18,176][root][INFO] - ===> Epoch[6620](33100/5): Loss 0.0115, Sem 0.0115 LR: 4.858e-02 Score 99.582 Data time: 0.2434, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.9729 [2021-07-08 13:17:32,123][root][INFO] - ===> Epoch[6621](33105/5): Loss 0.0116, Sem 0.0116 LR: 4.857e-02 Score 99.581 Data time: 0.1853, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7486 [2021-07-08 13:17:46,749][root][INFO] - ===> Epoch[6622](33110/5): Loss 0.0117, Sem 0.0117 LR: 4.856e-02 Score 99.576 Data time: 0.1819, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.8837 [2021-07-08 13:18:00,194][root][INFO] - ===> Epoch[6623](33115/5): Loss 0.0124, Sem 0.0124 LR: 4.855e-02 Score 99.553 Data time: 0.2095, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.6525 [2021-07-08 13:18:14,304][root][INFO] - ===> Epoch[6624](33120/5): Loss 0.0116, Sem 0.0116 LR: 4.855e-02 Score 99.587 Data time: 0.1953, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.7755 [2021-07-08 13:18:28,153][root][INFO] - ===> Epoch[6625](33125/5): Loss 0.0118, Sem 0.0118 LR: 4.854e-02 Score 99.576 Data time: 0.1887, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.7281 [2021-07-08 13:18:42,272][root][INFO] - ===> Epoch[6626](33130/5): Loss 0.0112, Sem 0.0112 LR: 4.853e-02 Score 99.594 Data time: 0.2244, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.7792 [2021-07-08 13:18:55,960][root][INFO] - ===> Epoch[6627](33135/5): Loss 0.0127, Sem 0.0127 LR: 4.852e-02 Score 99.540 Data time: 0.2363, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.6915 [2021-07-08 13:19:09,862][root][INFO] - ===> Epoch[6628](33140/5): Loss 0.0115, Sem 0.0115 LR: 4.851e-02 Score 99.580 Data time: 0.2591, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.7275 [2021-07-08 13:19:23,741][root][INFO] - ===> Epoch[6629](33145/5): Loss 0.0119, Sem 0.0119 LR: 4.851e-02 Score 99.569 Data time: 0.1731, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.7378 [2021-07-08 13:19:38,168][root][INFO] - ===> Epoch[6630](33150/5): Loss 0.0116, Sem 0.0116 LR: 4.850e-02 Score 99.581 Data time: 0.2117, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.8451 [2021-07-08 13:19:51,797][root][INFO] - ===> Epoch[6631](33155/5): Loss 0.0118, Sem 0.0118 LR: 4.849e-02 Score 99.573 Data time: 0.2326, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.6836 [2021-07-08 13:20:05,866][root][INFO] - ===> Epoch[6632](33160/5): Loss 0.0114, Sem 0.0114 LR: 4.848e-02 Score 99.586 Data time: 0.1721, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.7749 [2021-07-08 13:20:18,797][root][INFO] - ===> Epoch[6633](33165/5): Loss 0.0120, Sem 0.0120 LR: 4.847e-02 Score 99.569 Data time: 0.2156, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.5352 [2021-07-08 13:20:33,277][root][INFO] - ===> Epoch[6634](33170/5): Loss 0.0116, Sem 0.0116 LR: 4.847e-02 Score 99.579 Data time: 0.1881, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.8535 [2021-07-08 13:20:46,221][root][INFO] - ===> Epoch[6635](33175/5): Loss 0.0118, Sem 0.0118 LR: 4.846e-02 Score 99.571 Data time: 0.2182, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.5390 [2021-07-08 13:21:00,250][root][INFO] - ===> Epoch[6636](33180/5): Loss 0.0127, Sem 0.0127 LR: 4.845e-02 Score 99.540 Data time: 0.2066, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.7552 [2021-07-08 13:21:14,442][root][INFO] - ===> Epoch[6637](33185/5): Loss 0.0112, Sem 0.0112 LR: 4.844e-02 Score 99.590 Data time: 0.2631, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.7893 [2021-07-08 13:21:28,177][root][INFO] - ===> Epoch[6638](33190/5): Loss 0.0118, Sem 0.0118 LR: 4.843e-02 Score 99.574 Data time: 0.2218, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.7035 [2021-07-08 13:21:42,549][root][INFO] - ===> Epoch[6639](33195/5): Loss 0.0121, Sem 0.0121 LR: 4.842e-02 Score 99.563 Data time: 0.2377, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.8238 [2021-07-08 13:21:57,038][root][INFO] - ===> Epoch[6640](33200/5): Loss 0.0115, Sem 0.0115 LR: 4.842e-02 Score 99.582 Data time: 0.2382, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.8395 [2021-07-08 13:21:57,040][root][INFO] - ===> Start testing [2021-07-08 13:22:36,896][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1295 Loss 0.386 (AVG: 0.576) Score 92.274 (AVG: 90.953) mIOU 71.280 mAP 83.860 mAcc 76.844 IOU: 64.674 0.000 86.955 74.680 94.905 91.556 24.725 85.173 98.601 81.488 80.592 85.706 57.584 mAP: 85.048 4.536 92.567 87.244 98.690 98.578 55.066 98.096 99.796 93.826 92.460 94.187 90.081 mAcc: 79.653 0.000 90.109 83.291 96.241 96.797 25.652 94.202 99.380 85.339 89.772 97.013 61.517 [2021-07-08 13:22:36,896][root][INFO] - Finished test. Elapsed time: 39.8557 [2021-07-08 13:22:38,046][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 13:22:51,296][root][INFO] - ===> Epoch[6641](33205/5): Loss 0.0116, Sem 0.0116 LR: 4.841e-02 Score 99.581 Data time: 0.2241, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6168 [2021-07-08 13:23:05,888][root][INFO] - ===> Epoch[6642](33210/5): Loss 0.0119, Sem 0.0119 LR: 4.840e-02 Score 99.573 Data time: 0.2800, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.8509 [2021-07-08 13:23:19,319][root][INFO] - ===> Epoch[6643](33215/5): Loss 0.0114, Sem 0.0114 LR: 4.839e-02 Score 99.581 Data time: 0.2414, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6346 [2021-07-08 13:23:33,380][root][INFO] - ===> Epoch[6644](33220/5): Loss 0.0112, Sem 0.0112 LR: 4.838e-02 Score 99.594 Data time: 0.2290, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7734 [2021-07-08 13:23:47,944][root][INFO] - ===> Epoch[6645](33225/5): Loss 0.0112, Sem 0.0112 LR: 4.838e-02 Score 99.601 Data time: 0.2389, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.8630 [2021-07-08 13:24:02,365][root][INFO] - ===> Epoch[6646](33230/5): Loss 0.0118, Sem 0.0118 LR: 4.837e-02 Score 99.574 Data time: 0.2924, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.8345 [2021-07-08 13:24:17,066][root][INFO] - ===> Epoch[6647](33235/5): Loss 0.0108, Sem 0.0108 LR: 4.836e-02 Score 99.610 Data time: 0.2315, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.8957 [2021-07-08 13:24:30,484][root][INFO] - ===> Epoch[6648](33240/5): Loss 0.0114, Sem 0.0114 LR: 4.835e-02 Score 99.584 Data time: 0.2167, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.6352 [2021-07-08 13:24:44,319][root][INFO] - ===> Epoch[6649](33245/5): Loss 0.0114, Sem 0.0114 LR: 4.834e-02 Score 99.586 Data time: 0.2238, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7200 [2021-07-08 13:24:58,303][root][INFO] - ===> Epoch[6650](33250/5): Loss 0.0115, Sem 0.0115 LR: 4.834e-02 Score 99.582 Data time: 0.2612, Forward time: 1.2573, Backward time: 1.2411, DDP time: 0.0042, Total iter time: 2.7403 [2021-07-08 13:25:12,965][root][INFO] - ===> Epoch[6651](33255/5): Loss 0.0116, Sem 0.0116 LR: 4.833e-02 Score 99.580 Data time: 0.2173, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.8844 [2021-07-08 13:25:26,473][root][INFO] - ===> Epoch[6652](33260/5): Loss 0.0116, Sem 0.0116 LR: 4.832e-02 Score 99.580 Data time: 0.1980, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.6553 [2021-07-08 13:25:40,875][root][INFO] - ===> Epoch[6653](33265/5): Loss 0.0109, Sem 0.0109 LR: 4.831e-02 Score 99.608 Data time: 0.1640, Forward time: 1.2573, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.8337 [2021-07-08 13:25:54,052][root][INFO] - ===> Epoch[6654](33270/5): Loss 0.0112, Sem 0.0112 LR: 4.830e-02 Score 99.593 Data time: 0.1791, Forward time: 1.2572, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.5860 [2021-07-08 13:26:09,353][root][INFO] - ===> Epoch[6655](33275/5): Loss 0.0115, Sem 0.0115 LR: 4.829e-02 Score 99.586 Data time: 0.2349, Forward time: 1.2572, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 3.0165 [2021-07-08 13:26:23,383][root][INFO] - ===> Epoch[6656](33280/5): Loss 0.0111, Sem 0.0111 LR: 4.829e-02 Score 99.595 Data time: 0.2978, Forward time: 1.2572, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.7452 [2021-07-08 13:26:37,207][root][INFO] - ===> Epoch[6657](33285/5): Loss 0.0114, Sem 0.0114 LR: 4.828e-02 Score 99.590 Data time: 0.2066, Forward time: 1.2572, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.7197 [2021-07-08 13:26:51,034][root][INFO] - ===> Epoch[6658](33290/5): Loss 0.0121, Sem 0.0121 LR: 4.827e-02 Score 99.566 Data time: 0.2301, Forward time: 1.2572, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.7242 [2021-07-08 13:27:04,902][root][INFO] - ===> Epoch[6659](33295/5): Loss 0.0110, Sem 0.0110 LR: 4.826e-02 Score 99.602 Data time: 0.1851, Forward time: 1.2572, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.7350 [2021-07-08 13:27:17,542][root][INFO] - ===> Epoch[6660](33300/5): Loss 0.0117, Sem 0.0117 LR: 4.825e-02 Score 99.577 Data time: 0.2218, Forward time: 1.2572, Backward time: 1.2412, DDP time: 0.0042, Total iter time: 2.4898 [2021-07-08 13:27:31,961][root][INFO] - ===> Epoch[6661](33305/5): Loss 0.0111, Sem 0.0111 LR: 4.825e-02 Score 99.596 Data time: 0.1915, Forward time: 1.2572, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.8449 [2021-07-08 13:27:45,861][root][INFO] - ===> Epoch[6662](33310/5): Loss 0.0110, Sem 0.0110 LR: 4.824e-02 Score 99.604 Data time: 0.2407, Forward time: 1.2572, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.7340 [2021-07-08 13:28:00,086][root][INFO] - ===> Epoch[6663](33315/5): Loss 0.0116, Sem 0.0116 LR: 4.823e-02 Score 99.579 Data time: 0.2463, Forward time: 1.2572, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.8014 [2021-07-08 13:28:14,285][root][INFO] - ===> Epoch[6664](33320/5): Loss 0.0116, Sem 0.0116 LR: 4.822e-02 Score 99.581 Data time: 0.2421, Forward time: 1.2572, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.7889 [2021-07-08 13:28:28,520][root][INFO] - ===> Epoch[6665](33325/5): Loss 0.0108, Sem 0.0108 LR: 4.821e-02 Score 99.611 Data time: 0.2319, Forward time: 1.2572, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.7906 [2021-07-08 13:28:42,219][root][INFO] - ===> Epoch[6666](33330/5): Loss 0.0116, Sem 0.0116 LR: 4.821e-02 Score 99.581 Data time: 0.1995, Forward time: 1.2572, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.6945 [2021-07-08 13:28:56,238][root][INFO] - ===> Epoch[6667](33335/5): Loss 0.0113, Sem 0.0113 LR: 4.820e-02 Score 99.592 Data time: 0.1688, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.7535 [2021-07-08 13:29:10,821][root][INFO] - ===> Epoch[6668](33340/5): Loss 0.0117, Sem 0.0117 LR: 4.819e-02 Score 99.580 Data time: 0.2611, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.8554 [2021-07-08 13:29:24,293][root][INFO] - ===> Epoch[6669](33345/5): Loss 0.0118, Sem 0.0118 LR: 4.818e-02 Score 99.587 Data time: 0.1994, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.6409 [2021-07-08 13:29:38,209][root][INFO] - ===> Epoch[6670](33350/5): Loss 0.0112, Sem 0.0112 LR: 4.817e-02 Score 99.598 Data time: 0.2432, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.7361 [2021-07-08 13:29:52,204][root][INFO] - ===> Epoch[6671](33355/5): Loss 0.0109, Sem 0.0109 LR: 4.816e-02 Score 99.611 Data time: 0.1988, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.7594 [2021-07-08 13:30:06,271][root][INFO] - ===> Epoch[6672](33360/5): Loss 0.0112, Sem 0.0112 LR: 4.816e-02 Score 99.594 Data time: 0.2315, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.7525 [2021-07-08 13:30:19,952][root][INFO] - ===> Epoch[6673](33365/5): Loss 0.0112, Sem 0.0112 LR: 4.815e-02 Score 99.596 Data time: 0.2445, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.6824 [2021-07-08 13:30:34,816][root][INFO] - ===> Epoch[6674](33370/5): Loss 0.0118, Sem 0.0118 LR: 4.814e-02 Score 99.575 Data time: 0.1830, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.9208 [2021-07-08 13:30:48,715][root][INFO] - ===> Epoch[6675](33375/5): Loss 0.0113, Sem 0.0113 LR: 4.813e-02 Score 99.588 Data time: 0.2540, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.7320 [2021-07-08 13:31:03,194][root][INFO] - ===> Epoch[6676](33380/5): Loss 0.0114, Sem 0.0114 LR: 4.812e-02 Score 99.590 Data time: 0.2402, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.8562 [2021-07-08 13:31:17,329][root][INFO] - ===> Epoch[6677](33385/5): Loss 0.0110, Sem 0.0110 LR: 4.812e-02 Score 99.604 Data time: 0.2565, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.7779 [2021-07-08 13:31:31,140][root][INFO] - ===> Epoch[6678](33390/5): Loss 0.0109, Sem 0.0109 LR: 4.811e-02 Score 99.605 Data time: 0.3190, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.7108 [2021-07-08 13:31:45,706][root][INFO] - ===> Epoch[6679](33395/5): Loss 0.0118, Sem 0.0118 LR: 4.810e-02 Score 99.570 Data time: 0.2910, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.8756 [2021-07-08 13:32:00,272][root][INFO] - ===> Epoch[6680](33400/5): Loss 0.0109, Sem 0.0109 LR: 4.809e-02 Score 99.604 Data time: 0.2841, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.8528 [2021-07-08 13:32:00,274][root][INFO] - ===> Start testing [2021-07-08 13:32:39,849][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1239 Loss 0.369 (AVG: 0.583) Score 92.155 (AVG: 90.920) mIOU 70.580 mAP 83.705 mAcc 76.452 IOU: 64.539 0.000 85.138 75.031 95.147 91.760 23.461 82.341 98.567 78.540 79.459 86.208 57.347 mAP: 85.161 5.392 91.397 87.165 98.539 98.779 55.934 98.318 99.790 90.925 92.010 94.337 90.419 mAcc: 81.725 0.000 87.728 82.795 96.219 96.797 24.316 95.206 99.424 81.492 90.831 96.374 60.972 [2021-07-08 13:32:39,849][root][INFO] - Finished test. Elapsed time: 39.5746 [2021-07-08 13:32:40,963][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 13:32:53,656][root][INFO] - ===> Epoch[6681](33405/5): Loss 0.0122, Sem 0.0122 LR: 4.808e-02 Score 99.567 Data time: 0.1779, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.5087 [2021-07-08 13:33:07,798][root][INFO] - ===> Epoch[6682](33410/5): Loss 0.0113, Sem 0.0113 LR: 4.807e-02 Score 99.594 Data time: 0.2240, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.7790 [2021-07-08 13:33:20,878][root][INFO] - ===> Epoch[6683](33415/5): Loss 0.0117, Sem 0.0117 LR: 4.807e-02 Score 99.574 Data time: 0.2147, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.5789 [2021-07-08 13:33:34,057][root][INFO] - ===> Epoch[6684](33420/5): Loss 0.0112, Sem 0.0112 LR: 4.806e-02 Score 99.597 Data time: 0.2278, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.5990 [2021-07-08 13:33:48,452][root][INFO] - ===> Epoch[6685](33425/5): Loss 0.0113, Sem 0.0113 LR: 4.805e-02 Score 99.590 Data time: 0.2287, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.8436 [2021-07-08 13:34:02,260][root][INFO] - ===> Epoch[6686](33430/5): Loss 0.0109, Sem 0.0109 LR: 4.804e-02 Score 99.606 Data time: 0.2588, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.7108 [2021-07-08 13:34:16,829][root][INFO] - ===> Epoch[6687](33435/5): Loss 0.0109, Sem 0.0109 LR: 4.803e-02 Score 99.602 Data time: 0.2173, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.8727 [2021-07-08 13:34:31,881][root][INFO] - ===> Epoch[6688](33440/5): Loss 0.0113, Sem 0.0113 LR: 4.803e-02 Score 99.587 Data time: 0.2271, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.9644 [2021-07-08 13:34:45,571][root][INFO] - ===> Epoch[6689](33445/5): Loss 0.0113, Sem 0.0113 LR: 4.802e-02 Score 99.594 Data time: 0.2043, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.6980 [2021-07-08 13:34:59,673][root][INFO] - ===> Epoch[6690](33450/5): Loss 0.0113, Sem 0.0113 LR: 4.801e-02 Score 99.589 Data time: 0.2498, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.7784 [2021-07-08 13:35:14,327][root][INFO] - ===> Epoch[6691](33455/5): Loss 0.0114, Sem 0.0114 LR: 4.800e-02 Score 99.593 Data time: 0.2630, Forward time: 1.2571, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.8887 [2021-07-08 13:35:28,592][root][INFO] - ===> Epoch[6692](33460/5): Loss 0.0114, Sem 0.0114 LR: 4.799e-02 Score 99.588 Data time: 0.2474, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8006 [2021-07-08 13:35:42,350][root][INFO] - ===> Epoch[6693](33465/5): Loss 0.0109, Sem 0.0109 LR: 4.799e-02 Score 99.606 Data time: 0.2021, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.7033 [2021-07-08 13:35:56,026][root][INFO] - ===> Epoch[6694](33470/5): Loss 0.0112, Sem 0.0112 LR: 4.798e-02 Score 99.594 Data time: 0.2266, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.6856 [2021-07-08 13:36:10,762][root][INFO] - ===> Epoch[6695](33475/5): Loss 0.0109, Sem 0.0109 LR: 4.797e-02 Score 99.604 Data time: 0.2006, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8979 [2021-07-08 13:36:24,003][root][INFO] - ===> Epoch[6696](33480/5): Loss 0.0115, Sem 0.0115 LR: 4.796e-02 Score 99.585 Data time: 0.2012, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.5957 [2021-07-08 13:36:38,520][root][INFO] - ===> Epoch[6697](33485/5): Loss 0.0113, Sem 0.0113 LR: 4.795e-02 Score 99.586 Data time: 0.1864, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8594 [2021-07-08 13:36:51,715][root][INFO] - ===> Epoch[6698](33490/5): Loss 0.0116, Sem 0.0116 LR: 4.794e-02 Score 99.579 Data time: 0.2031, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.5983 [2021-07-08 13:37:05,629][root][INFO] - ===> Epoch[6699](33495/5): Loss 0.0110, Sem 0.0110 LR: 4.794e-02 Score 99.600 Data time: 0.2275, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.7296 [2021-07-08 13:37:19,193][root][INFO] - ===> Epoch[6700](33500/5): Loss 0.0115, Sem 0.0115 LR: 4.793e-02 Score 99.587 Data time: 0.2680, Forward time: 1.2570, Backward time: 1.2413, DDP time: 0.0042, Total iter time: 2.6612 [2021-07-08 13:37:33,714][root][INFO] - ===> Epoch[6701](33505/5): Loss 0.0112, Sem 0.0112 LR: 4.792e-02 Score 99.602 Data time: 0.1972, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8661 [2021-07-08 13:37:47,795][root][INFO] - ===> Epoch[6702](33510/5): Loss 0.0112, Sem 0.0112 LR: 4.791e-02 Score 99.596 Data time: 0.1939, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.7767 [2021-07-08 13:38:02,241][root][INFO] - ===> Epoch[6703](33515/5): Loss 0.0108, Sem 0.0108 LR: 4.790e-02 Score 99.607 Data time: 0.2237, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8326 [2021-07-08 13:38:16,203][root][INFO] - ===> Epoch[6704](33520/5): Loss 0.0108, Sem 0.0108 LR: 4.790e-02 Score 99.607 Data time: 0.2327, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.7314 [2021-07-08 13:38:30,045][root][INFO] - ===> Epoch[6705](33525/5): Loss 0.0112, Sem 0.0112 LR: 4.789e-02 Score 99.595 Data time: 0.2081, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.7180 [2021-07-08 13:38:44,241][root][INFO] - ===> Epoch[6706](33530/5): Loss 0.0107, Sem 0.0107 LR: 4.788e-02 Score 99.613 Data time: 0.2456, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8035 [2021-07-08 13:38:57,869][root][INFO] - ===> Epoch[6707](33535/5): Loss 0.0110, Sem 0.0110 LR: 4.787e-02 Score 99.602 Data time: 0.2297, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.6695 [2021-07-08 13:39:11,350][root][INFO] - ===> Epoch[6708](33540/5): Loss 0.0109, Sem 0.0109 LR: 4.786e-02 Score 99.607 Data time: 0.2488, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.6457 [2021-07-08 13:39:25,553][root][INFO] - ===> Epoch[6709](33545/5): Loss 0.0114, Sem 0.0114 LR: 4.786e-02 Score 99.590 Data time: 0.2150, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8006 [2021-07-08 13:39:39,475][root][INFO] - ===> Epoch[6710](33550/5): Loss 0.0109, Sem 0.0109 LR: 4.785e-02 Score 99.605 Data time: 0.2608, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.7432 [2021-07-08 13:39:53,457][root][INFO] - ===> Epoch[6711](33555/5): Loss 0.0116, Sem 0.0116 LR: 4.784e-02 Score 99.580 Data time: 0.2491, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.7519 [2021-07-08 13:40:07,417][root][INFO] - ===> Epoch[6712](33560/5): Loss 0.0106, Sem 0.0106 LR: 4.783e-02 Score 99.617 Data time: 0.2241, Forward time: 1.2569, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.7504 [2021-07-08 13:40:22,107][root][INFO] - ===> Epoch[6713](33565/5): Loss 0.0111, Sem 0.0111 LR: 4.782e-02 Score 99.603 Data time: 0.3179, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8811 [2021-07-08 13:40:36,559][root][INFO] - ===> Epoch[6714](33570/5): Loss 0.0112, Sem 0.0112 LR: 4.781e-02 Score 99.595 Data time: 0.2267, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8265 [2021-07-08 13:40:50,665][root][INFO] - ===> Epoch[6715](33575/5): Loss 0.0113, Sem 0.0113 LR: 4.781e-02 Score 99.595 Data time: 0.2702, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.7735 [2021-07-08 13:41:04,130][root][INFO] - ===> Epoch[6716](33580/5): Loss 0.0118, Sem 0.0118 LR: 4.780e-02 Score 99.578 Data time: 0.2160, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.6416 [2021-07-08 13:41:18,551][root][INFO] - ===> Epoch[6717](33585/5): Loss 0.0117, Sem 0.0117 LR: 4.779e-02 Score 99.579 Data time: 0.2154, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8300 [2021-07-08 13:41:33,147][root][INFO] - ===> Epoch[6718](33590/5): Loss 0.0115, Sem 0.0115 LR: 4.778e-02 Score 99.583 Data time: 0.2294, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8840 [2021-07-08 13:41:47,795][root][INFO] - ===> Epoch[6719](33595/5): Loss 0.0118, Sem 0.0118 LR: 4.777e-02 Score 99.573 Data time: 0.2208, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8875 [2021-07-08 13:42:02,267][root][INFO] - ===> Epoch[6720](33600/5): Loss 0.0119, Sem 0.0119 LR: 4.777e-02 Score 99.577 Data time: 0.2192, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8520 [2021-07-08 13:42:02,268][root][INFO] - ===> Start testing [2021-07-08 13:42:41,793][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1253 Loss 0.393 (AVG: 0.571) Score 91.699 (AVG: 91.010) mIOU 70.913 mAP 83.985 mAcc 77.293 IOU: 65.008 0.000 85.821 75.415 94.876 91.991 22.320 82.434 98.637 80.302 80.418 86.327 58.324 mAP: 85.161 8.088 92.868 87.363 98.794 98.805 51.321 98.036 99.822 93.956 92.353 94.146 91.097 mAcc: 79.758 0.000 89.964 84.063 96.225 97.332 23.341 95.333 99.395 88.723 91.411 96.388 62.881 [2021-07-08 13:42:41,793][root][INFO] - Finished test. Elapsed time: 39.5236 [2021-07-08 13:42:42,824][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 13:42:55,851][root][INFO] - ===> Epoch[6721](33605/5): Loss 0.0125, Sem 0.0125 LR: 4.776e-02 Score 99.553 Data time: 0.1987, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.5614 [2021-07-08 13:43:09,150][root][INFO] - ===> Epoch[6722](33610/5): Loss 0.0119, Sem 0.0119 LR: 4.775e-02 Score 99.574 Data time: 0.2277, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.6192 [2021-07-08 13:43:23,481][root][INFO] - ===> Epoch[6723](33615/5): Loss 0.0122, Sem 0.0122 LR: 4.774e-02 Score 99.570 Data time: 0.2127, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8153 [2021-07-08 13:43:37,971][root][INFO] - ===> Epoch[6724](33620/5): Loss 0.0122, Sem 0.0122 LR: 4.773e-02 Score 99.562 Data time: 0.2163, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8455 [2021-07-08 13:43:52,422][root][INFO] - ===> Epoch[6725](33625/5): Loss 0.0124, Sem 0.0124 LR: 4.772e-02 Score 99.551 Data time: 0.2157, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8389 [2021-07-08 13:44:06,006][root][INFO] - ===> Epoch[6726](33630/5): Loss 0.0123, Sem 0.0123 LR: 4.772e-02 Score 99.557 Data time: 0.2308, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.6696 [2021-07-08 13:44:19,224][root][INFO] - ===> Epoch[6727](33635/5): Loss 0.0122, Sem 0.0122 LR: 4.771e-02 Score 99.561 Data time: 0.2196, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.5925 [2021-07-08 13:44:32,810][root][INFO] - ===> Epoch[6728](33640/5): Loss 0.0122, Sem 0.0122 LR: 4.770e-02 Score 99.561 Data time: 0.1734, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.6740 [2021-07-08 13:44:46,854][root][INFO] - ===> Epoch[6729](33645/5): Loss 0.0119, Sem 0.0119 LR: 4.769e-02 Score 99.572 Data time: 0.2418, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.7584 [2021-07-08 13:45:00,449][root][INFO] - ===> Epoch[6730](33650/5): Loss 0.0119, Sem 0.0119 LR: 4.768e-02 Score 99.568 Data time: 0.2049, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.6767 [2021-07-08 13:45:14,553][root][INFO] - ===> Epoch[6731](33655/5): Loss 0.0130, Sem 0.0130 LR: 4.768e-02 Score 99.547 Data time: 0.2465, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.7855 [2021-07-08 13:45:29,023][root][INFO] - ===> Epoch[6732](33660/5): Loss 0.0128, Sem 0.0128 LR: 4.767e-02 Score 99.543 Data time: 0.1785, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8581 [2021-07-08 13:45:42,419][root][INFO] - ===> Epoch[6733](33665/5): Loss 0.0127, Sem 0.0127 LR: 4.766e-02 Score 99.543 Data time: 0.2368, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.6363 [2021-07-08 13:45:56,690][root][INFO] - ===> Epoch[6734](33670/5): Loss 0.0127, Sem 0.0127 LR: 4.765e-02 Score 99.543 Data time: 0.2253, Forward time: 1.2570, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.8107 [2021-07-08 13:46:10,299][root][INFO] - ===> Epoch[6735](33675/5): Loss 0.0126, Sem 0.0126 LR: 4.764e-02 Score 99.544 Data time: 0.2449, Forward time: 1.2569, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.6719 [2021-07-08 13:46:23,653][root][INFO] - ===> Epoch[6736](33680/5): Loss 0.0126, Sem 0.0126 LR: 4.764e-02 Score 99.546 Data time: 0.1740, Forward time: 1.2569, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.6270 [2021-07-08 13:46:37,641][root][INFO] - ===> Epoch[6737](33685/5): Loss 0.0119, Sem 0.0119 LR: 4.763e-02 Score 99.570 Data time: 0.2011, Forward time: 1.2569, Backward time: 1.2414, DDP time: 0.0042, Total iter time: 2.7502 [2021-07-08 13:46:52,718][root][INFO] - ===> Epoch[6738](33690/5): Loss 0.0126, Sem 0.0126 LR: 4.762e-02 Score 99.544 Data time: 0.2237, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.9623 [2021-07-08 13:47:06,505][root][INFO] - ===> Epoch[6739](33695/5): Loss 0.0118, Sem 0.0118 LR: 4.761e-02 Score 99.571 Data time: 0.2343, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.7008 [2021-07-08 13:47:20,605][root][INFO] - ===> Epoch[6740](33700/5): Loss 0.0117, Sem 0.0117 LR: 4.760e-02 Score 99.575 Data time: 0.2430, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.7678 [2021-07-08 13:47:35,077][root][INFO] - ===> Epoch[6741](33705/5): Loss 0.0113, Sem 0.0113 LR: 4.759e-02 Score 99.592 Data time: 0.2363, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.8375 [2021-07-08 13:47:49,009][root][INFO] - ===> Epoch[6742](33710/5): Loss 0.0120, Sem 0.0120 LR: 4.759e-02 Score 99.566 Data time: 0.2105, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.7279 [2021-07-08 13:48:03,241][root][INFO] - ===> Epoch[6743](33715/5): Loss 0.0113, Sem 0.0113 LR: 4.758e-02 Score 99.595 Data time: 0.1968, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.7890 [2021-07-08 13:48:17,511][root][INFO] - ===> Epoch[6744](33720/5): Loss 0.0116, Sem 0.0116 LR: 4.757e-02 Score 99.580 Data time: 0.1823, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.8121 [2021-07-08 13:48:32,000][root][INFO] - ===> Epoch[6745](33725/5): Loss 0.0115, Sem 0.0115 LR: 4.756e-02 Score 99.590 Data time: 0.2835, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.8349 [2021-07-08 13:48:46,619][root][INFO] - ===> Epoch[6746](33730/5): Loss 0.0121, Sem 0.0121 LR: 4.755e-02 Score 99.563 Data time: 0.2362, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.8837 [2021-07-08 13:49:00,915][root][INFO] - ===> Epoch[6747](33735/5): Loss 0.0114, Sem 0.0114 LR: 4.755e-02 Score 99.589 Data time: 0.2523, Forward time: 1.2570, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.8170 [2021-07-08 13:49:15,127][root][INFO] - ===> Epoch[6748](33740/5): Loss 0.0113, Sem 0.0113 LR: 4.754e-02 Score 99.589 Data time: 0.1989, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.7949 [2021-07-08 13:49:29,254][root][INFO] - ===> Epoch[6749](33745/5): Loss 0.0112, Sem 0.0112 LR: 4.753e-02 Score 99.593 Data time: 0.2465, Forward time: 1.2570, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.7849 [2021-07-08 13:49:43,597][root][INFO] - ===> Epoch[6750](33750/5): Loss 0.0114, Sem 0.0114 LR: 4.752e-02 Score 99.587 Data time: 0.2006, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.8225 [2021-07-08 13:49:56,697][root][INFO] - ===> Epoch[6751](33755/5): Loss 0.0114, Sem 0.0114 LR: 4.751e-02 Score 99.592 Data time: 0.1957, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.5768 [2021-07-08 13:50:11,119][root][INFO] - ===> Epoch[6752](33760/5): Loss 0.0116, Sem 0.0116 LR: 4.751e-02 Score 99.580 Data time: 0.2481, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.8439 [2021-07-08 13:50:25,419][root][INFO] - ===> Epoch[6753](33765/5): Loss 0.0111, Sem 0.0111 LR: 4.750e-02 Score 99.600 Data time: 0.2701, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.8206 [2021-07-08 13:50:39,005][root][INFO] - ===> Epoch[6754](33770/5): Loss 0.0109, Sem 0.0109 LR: 4.749e-02 Score 99.606 Data time: 0.2304, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.6670 [2021-07-08 13:50:52,637][root][INFO] - ===> Epoch[6755](33775/5): Loss 0.0112, Sem 0.0112 LR: 4.748e-02 Score 99.594 Data time: 0.2415, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.6762 [2021-07-08 13:51:07,712][root][INFO] - ===> Epoch[6756](33780/5): Loss 0.0109, Sem 0.0109 LR: 4.747e-02 Score 99.608 Data time: 0.1890, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.9588 [2021-07-08 13:51:21,306][root][INFO] - ===> Epoch[6757](33785/5): Loss 0.0120, Sem 0.0120 LR: 4.746e-02 Score 99.566 Data time: 0.2724, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.6649 [2021-07-08 13:51:35,381][root][INFO] - ===> Epoch[6758](33790/5): Loss 0.0124, Sem 0.0124 LR: 4.746e-02 Score 99.584 Data time: 0.2236, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.7646 [2021-07-08 13:51:49,776][root][INFO] - ===> Epoch[6759](33795/5): Loss 0.0122, Sem 0.0122 LR: 4.745e-02 Score 99.563 Data time: 0.2294, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.8381 [2021-07-08 13:52:04,362][root][INFO] - ===> Epoch[6760](33800/5): Loss 0.0116, Sem 0.0116 LR: 4.744e-02 Score 99.583 Data time: 0.2141, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.8703 [2021-07-08 13:52:04,363][root][INFO] - ===> Start testing [2021-07-08 13:52:47,811][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1267 Loss 0.335 (AVG: 0.589) Score 93.286 (AVG: 90.902) mIOU 70.843 mAP 83.054 mAcc 76.646 IOU: 64.070 0.000 84.158 75.076 94.964 91.974 24.227 82.335 98.592 82.869 79.096 86.070 57.527 mAP: 83.391 2.854 91.759 86.949 98.623 98.809 53.073 97.048 99.825 91.112 92.484 93.795 89.983 mAcc: 79.015 0.000 87.650 83.328 96.520 96.824 25.555 93.166 99.306 85.191 91.272 96.874 61.694 [2021-07-08 13:52:47,811][root][INFO] - Finished test. Elapsed time: 43.4465 [2021-07-08 13:52:48,959][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 13:53:01,781][root][INFO] - ===> Epoch[6761](33805/5): Loss 0.0118, Sem 0.0118 LR: 4.743e-02 Score 99.575 Data time: 0.1887, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.5268 [2021-07-08 13:53:15,929][root][INFO] - ===> Epoch[6762](33810/5): Loss 0.0117, Sem 0.0117 LR: 4.742e-02 Score 99.578 Data time: 0.2154, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.7727 [2021-07-08 13:53:29,626][root][INFO] - ===> Epoch[6763](33815/5): Loss 0.0121, Sem 0.0121 LR: 4.742e-02 Score 99.564 Data time: 0.2098, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.6935 [2021-07-08 13:53:43,540][root][INFO] - ===> Epoch[6764](33820/5): Loss 0.0115, Sem 0.0115 LR: 4.741e-02 Score 99.582 Data time: 0.2705, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.7394 [2021-07-08 13:53:56,961][root][INFO] - ===> Epoch[6765](33825/5): Loss 0.0109, Sem 0.0109 LR: 4.740e-02 Score 99.602 Data time: 0.2052, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.6350 [2021-07-08 13:54:10,721][root][INFO] - ===> Epoch[6766](33830/5): Loss 0.0109, Sem 0.0109 LR: 4.739e-02 Score 99.604 Data time: 0.2145, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.7018 [2021-07-08 13:54:24,941][root][INFO] - ===> Epoch[6767](33835/5): Loss 0.0112, Sem 0.0112 LR: 4.738e-02 Score 99.596 Data time: 0.2240, Forward time: 1.2568, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.7891 [2021-07-08 13:54:38,733][root][INFO] - ===> Epoch[6768](33840/5): Loss 0.0112, Sem 0.0112 LR: 4.737e-02 Score 99.596 Data time: 0.1760, Forward time: 1.2568, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.7059 [2021-07-08 13:54:52,033][root][INFO] - ===> Epoch[6769](33845/5): Loss 0.0119, Sem 0.0119 LR: 4.737e-02 Score 99.570 Data time: 0.2012, Forward time: 1.2568, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.6068 [2021-07-08 13:55:05,753][root][INFO] - ===> Epoch[6770](33850/5): Loss 0.0108, Sem 0.0108 LR: 4.736e-02 Score 99.608 Data time: 0.1773, Forward time: 1.2568, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.6927 [2021-07-08 13:55:20,090][root][INFO] - ===> Epoch[6771](33855/5): Loss 0.0111, Sem 0.0111 LR: 4.735e-02 Score 99.599 Data time: 0.2061, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8312 [2021-07-08 13:55:34,395][root][INFO] - ===> Epoch[6772](33860/5): Loss 0.0111, Sem 0.0111 LR: 4.734e-02 Score 99.597 Data time: 0.2175, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8181 [2021-07-08 13:55:48,188][root][INFO] - ===> Epoch[6773](33865/5): Loss 0.0111, Sem 0.0111 LR: 4.733e-02 Score 99.598 Data time: 0.2194, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7196 [2021-07-08 13:56:02,550][root][INFO] - ===> Epoch[6774](33870/5): Loss 0.0107, Sem 0.0107 LR: 4.733e-02 Score 99.611 Data time: 0.2684, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8280 [2021-07-08 13:56:16,390][root][INFO] - ===> Epoch[6775](33875/5): Loss 0.0106, Sem 0.0106 LR: 4.732e-02 Score 99.615 Data time: 0.2832, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7253 [2021-07-08 13:56:30,363][root][INFO] - ===> Epoch[6776](33880/5): Loss 0.0112, Sem 0.0112 LR: 4.731e-02 Score 99.595 Data time: 0.2308, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7491 [2021-07-08 13:56:44,295][root][INFO] - ===> Epoch[6777](33885/5): Loss 0.0106, Sem 0.0106 LR: 4.730e-02 Score 99.617 Data time: 0.2344, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7305 [2021-07-08 13:56:59,466][root][INFO] - ===> Epoch[6778](33890/5): Loss 0.0110, Sem 0.0110 LR: 4.729e-02 Score 99.602 Data time: 0.2249, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.9873 [2021-07-08 13:57:13,438][root][INFO] - ===> Epoch[6779](33895/5): Loss 0.0105, Sem 0.0105 LR: 4.729e-02 Score 99.620 Data time: 0.3468, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7382 [2021-07-08 13:57:28,491][root][INFO] - ===> Epoch[6780](33900/5): Loss 0.0105, Sem 0.0105 LR: 4.728e-02 Score 99.622 Data time: 0.2331, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.9604 [2021-07-08 13:57:42,613][root][INFO] - ===> Epoch[6781](33905/5): Loss 0.0108, Sem 0.0108 LR: 4.727e-02 Score 99.609 Data time: 0.2432, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7851 [2021-07-08 13:57:57,061][root][INFO] - ===> Epoch[6782](33910/5): Loss 0.0109, Sem 0.0109 LR: 4.726e-02 Score 99.609 Data time: 0.2400, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8399 [2021-07-08 13:58:10,850][root][INFO] - ===> Epoch[6783](33915/5): Loss 0.0115, Sem 0.0115 LR: 4.725e-02 Score 99.584 Data time: 0.2601, Forward time: 1.2568, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.7062 [2021-07-08 13:58:25,833][root][INFO] - ===> Epoch[6784](33920/5): Loss 0.0110, Sem 0.0110 LR: 4.724e-02 Score 99.607 Data time: 0.2572, Forward time: 1.2569, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.9452 [2021-07-08 13:58:39,699][root][INFO] - ===> Epoch[6785](33925/5): Loss 0.0109, Sem 0.0109 LR: 4.724e-02 Score 99.610 Data time: 0.2393, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7282 [2021-07-08 13:58:53,922][root][INFO] - ===> Epoch[6786](33930/5): Loss 0.0105, Sem 0.0105 LR: 4.723e-02 Score 99.621 Data time: 0.1966, Forward time: 1.2569, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8013 [2021-07-08 13:59:08,015][root][INFO] - ===> Epoch[6787](33935/5): Loss 0.0109, Sem 0.0109 LR: 4.722e-02 Score 99.607 Data time: 0.1787, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7824 [2021-07-08 13:59:22,616][root][INFO] - ===> Epoch[6788](33940/5): Loss 0.0109, Sem 0.0109 LR: 4.721e-02 Score 99.606 Data time: 0.1994, Forward time: 1.2569, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8872 [2021-07-08 13:59:36,847][root][INFO] - ===> Epoch[6789](33945/5): Loss 0.0114, Sem 0.0114 LR: 4.720e-02 Score 99.586 Data time: 0.1897, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8074 [2021-07-08 13:59:50,232][root][INFO] - ===> Epoch[6790](33950/5): Loss 0.0109, Sem 0.0109 LR: 4.720e-02 Score 99.610 Data time: 0.2701, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6333 [2021-07-08 14:00:04,158][root][INFO] - ===> Epoch[6791](33955/5): Loss 0.0111, Sem 0.0111 LR: 4.719e-02 Score 99.599 Data time: 0.1671, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7423 [2021-07-08 14:00:17,835][root][INFO] - ===> Epoch[6792](33960/5): Loss 0.0105, Sem 0.0105 LR: 4.718e-02 Score 99.622 Data time: 0.2381, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6780 [2021-07-08 14:00:31,245][root][INFO] - ===> Epoch[6793](33965/5): Loss 0.0113, Sem 0.0113 LR: 4.717e-02 Score 99.600 Data time: 0.2013, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6435 [2021-07-08 14:00:45,229][root][INFO] - ===> Epoch[6794](33970/5): Loss 0.0108, Sem 0.0108 LR: 4.716e-02 Score 99.613 Data time: 0.1922, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7481 [2021-07-08 14:00:58,847][root][INFO] - ===> Epoch[6795](33975/5): Loss 0.0107, Sem 0.0107 LR: 4.715e-02 Score 99.614 Data time: 0.1655, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6823 [2021-07-08 14:01:12,327][root][INFO] - ===> Epoch[6796](33980/5): Loss 0.0109, Sem 0.0109 LR: 4.715e-02 Score 99.608 Data time: 0.2204, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6493 [2021-07-08 14:01:25,753][root][INFO] - ===> Epoch[6797](33985/5): Loss 0.0105, Sem 0.0105 LR: 4.714e-02 Score 99.622 Data time: 0.2046, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6354 [2021-07-08 14:01:39,503][root][INFO] - ===> Epoch[6798](33990/5): Loss 0.0114, Sem 0.0114 LR: 4.713e-02 Score 99.587 Data time: 0.2137, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7076 [2021-07-08 14:01:53,739][root][INFO] - ===> Epoch[6799](33995/5): Loss 0.0107, Sem 0.0107 LR: 4.712e-02 Score 99.619 Data time: 0.2449, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8021 [2021-07-08 14:02:07,646][root][INFO] - ===> Epoch[6800](34000/5): Loss 0.0108, Sem 0.0108 LR: 4.711e-02 Score 99.605 Data time: 0.2056, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7384 [2021-07-08 14:02:07,648][root][INFO] - ===> Start testing [2021-07-08 14:02:49,207][root][INFO] - 68/68: Data time: 0.0016, Iter time: 0.1289 Loss 0.309 (AVG: 0.572) Score 93.295 (AVG: 91.052) mIOU 71.613 mAP 84.119 mAcc 77.835 IOU: 64.067 0.000 84.040 75.605 94.590 92.329 31.949 84.027 98.593 80.933 79.850 86.413 58.567 mAP: 83.672 4.486 92.444 87.390 98.491 98.890 59.191 98.180 99.764 93.821 91.912 94.485 90.825 mAcc: 79.296 0.000 88.495 84.108 95.966 96.575 34.560 94.335 99.604 88.487 90.534 96.326 63.566 [2021-07-08 14:02:49,207][root][INFO] - Finished test. Elapsed time: 41.5588 [2021-07-08 14:02:50,273][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 14:03:03,787][root][INFO] - ===> Epoch[6801](34005/5): Loss 0.0113, Sem 0.0113 LR: 4.711e-02 Score 99.596 Data time: 0.2052, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6653 [2021-07-08 14:03:18,624][root][INFO] - ===> Epoch[6802](34010/5): Loss 0.0105, Sem 0.0105 LR: 4.710e-02 Score 99.617 Data time: 0.3037, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.9081 [2021-07-08 14:03:33,092][root][INFO] - ===> Epoch[6803](34015/5): Loss 0.0100, Sem 0.0100 LR: 4.709e-02 Score 99.636 Data time: 0.1989, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8497 [2021-07-08 14:03:46,772][root][INFO] - ===> Epoch[6804](34020/5): Loss 0.0108, Sem 0.0108 LR: 4.708e-02 Score 99.610 Data time: 0.2195, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6929 [2021-07-08 14:04:00,658][root][INFO] - ===> Epoch[6805](34025/5): Loss 0.0105, Sem 0.0105 LR: 4.707e-02 Score 99.621 Data time: 0.1929, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7260 [2021-07-08 14:04:14,891][root][INFO] - ===> Epoch[6806](34030/5): Loss 0.0103, Sem 0.0103 LR: 4.706e-02 Score 99.625 Data time: 0.2269, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7927 [2021-07-08 14:04:29,739][root][INFO] - ===> Epoch[6807](34035/5): Loss 0.0104, Sem 0.0104 LR: 4.706e-02 Score 99.625 Data time: 0.1934, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.9166 [2021-07-08 14:04:43,721][root][INFO] - ===> Epoch[6808](34040/5): Loss 0.0110, Sem 0.0110 LR: 4.705e-02 Score 99.600 Data time: 0.2168, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7489 [2021-07-08 14:04:58,587][root][INFO] - ===> Epoch[6809](34045/5): Loss 0.0101, Sem 0.0101 LR: 4.704e-02 Score 99.637 Data time: 0.2396, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.9325 [2021-07-08 14:05:12,153][root][INFO] - ===> Epoch[6810](34050/5): Loss 0.0106, Sem 0.0106 LR: 4.703e-02 Score 99.621 Data time: 0.2430, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6687 [2021-07-08 14:05:26,437][root][INFO] - ===> Epoch[6811](34055/5): Loss 0.0110, Sem 0.0110 LR: 4.702e-02 Score 99.602 Data time: 0.2374, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8084 [2021-07-08 14:05:39,974][root][INFO] - ===> Epoch[6812](34060/5): Loss 0.0104, Sem 0.0104 LR: 4.702e-02 Score 99.624 Data time: 0.1577, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6620 [2021-07-08 14:05:53,761][root][INFO] - ===> Epoch[6813](34065/5): Loss 0.0110, Sem 0.0110 LR: 4.701e-02 Score 99.604 Data time: 0.1493, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7159 [2021-07-08 14:06:09,154][root][INFO] - ===> Epoch[6814](34070/5): Loss 0.0107, Sem 0.0107 LR: 4.700e-02 Score 99.611 Data time: 0.3541, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 3.0369 [2021-07-08 14:06:23,513][root][INFO] - ===> Epoch[6815](34075/5): Loss 0.0104, Sem 0.0104 LR: 4.699e-02 Score 99.621 Data time: 0.3352, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8163 [2021-07-08 14:06:37,339][root][INFO] - ===> Epoch[6816](34080/5): Loss 0.0102, Sem 0.0102 LR: 4.698e-02 Score 99.631 Data time: 0.2139, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7302 [2021-07-08 14:06:51,420][root][INFO] - ===> Epoch[6817](34085/5): Loss 0.0107, Sem 0.0107 LR: 4.698e-02 Score 99.616 Data time: 0.2287, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7557 [2021-07-08 14:07:05,799][root][INFO] - ===> Epoch[6818](34090/5): Loss 0.0104, Sem 0.0104 LR: 4.697e-02 Score 99.622 Data time: 0.3040, Forward time: 1.2569, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8302 [2021-07-08 14:07:19,608][root][INFO] - ===> Epoch[6819](34095/5): Loss 0.0108, Sem 0.0108 LR: 4.696e-02 Score 99.611 Data time: 0.2003, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7307 [2021-07-08 14:07:33,398][root][INFO] - ===> Epoch[6820](34100/5): Loss 0.0102, Sem 0.0102 LR: 4.695e-02 Score 99.630 Data time: 0.2107, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7154 [2021-07-08 14:07:46,706][root][INFO] - ===> Epoch[6821](34105/5): Loss 0.0103, Sem 0.0103 LR: 4.694e-02 Score 99.627 Data time: 0.1975, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6190 [2021-07-08 14:08:00,047][root][INFO] - ===> Epoch[6822](34110/5): Loss 0.0107, Sem 0.0107 LR: 4.693e-02 Score 99.616 Data time: 0.2269, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6243 [2021-07-08 14:08:14,426][root][INFO] - ===> Epoch[6823](34115/5): Loss 0.0105, Sem 0.0105 LR: 4.693e-02 Score 99.621 Data time: 0.1674, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8379 [2021-07-08 14:08:28,520][root][INFO] - ===> Epoch[6824](34120/5): Loss 0.0108, Sem 0.0108 LR: 4.692e-02 Score 99.612 Data time: 0.2497, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7698 [2021-07-08 14:08:42,500][root][INFO] - ===> Epoch[6825](34125/5): Loss 0.0110, Sem 0.0110 LR: 4.691e-02 Score 99.604 Data time: 0.1735, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7528 [2021-07-08 14:08:57,048][root][INFO] - ===> Epoch[6826](34130/5): Loss 0.0112, Sem 0.0112 LR: 4.690e-02 Score 99.596 Data time: 0.2270, Forward time: 1.2569, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8586 [2021-07-08 14:09:11,526][root][INFO] - ===> Epoch[6827](34135/5): Loss 0.0107, Sem 0.0107 LR: 4.689e-02 Score 99.612 Data time: 0.2191, Forward time: 1.2569, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8496 [2021-07-08 14:09:26,533][root][INFO] - ===> Epoch[6828](34140/5): Loss 0.0105, Sem 0.0105 LR: 4.689e-02 Score 99.623 Data time: 0.1641, Forward time: 1.2569, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.9689 [2021-07-08 14:09:40,475][root][INFO] - ===> Epoch[6829](34145/5): Loss 0.0111, Sem 0.0111 LR: 4.688e-02 Score 99.603 Data time: 0.2056, Forward time: 1.2569, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7419 [2021-07-08 14:09:54,225][root][INFO] - ===> Epoch[6830](34150/5): Loss 0.0105, Sem 0.0105 LR: 4.687e-02 Score 99.619 Data time: 0.1819, Forward time: 1.2569, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7020 [2021-07-08 14:10:07,794][root][INFO] - ===> Epoch[6831](34155/5): Loss 0.0106, Sem 0.0106 LR: 4.686e-02 Score 99.618 Data time: 0.1820, Forward time: 1.2569, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6745 [2021-07-08 14:10:21,347][root][INFO] - ===> Epoch[6832](34160/5): Loss 0.0109, Sem 0.0109 LR: 4.685e-02 Score 99.608 Data time: 0.2108, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6649 [2021-07-08 14:10:36,045][root][INFO] - ===> Epoch[6833](34165/5): Loss 0.0106, Sem 0.0106 LR: 4.684e-02 Score 99.613 Data time: 0.2257, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.8862 [2021-07-08 14:10:49,675][root][INFO] - ===> Epoch[6834](34170/5): Loss 0.0105, Sem 0.0105 LR: 4.684e-02 Score 99.617 Data time: 0.1893, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.6826 [2021-07-08 14:11:04,457][root][INFO] - ===> Epoch[6835](34175/5): Loss 0.0099, Sem 0.0099 LR: 4.683e-02 Score 99.640 Data time: 0.2318, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.9114 [2021-07-08 14:11:19,032][root][INFO] - ===> Epoch[6836](34180/5): Loss 0.0107, Sem 0.0107 LR: 4.682e-02 Score 99.611 Data time: 0.2237, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.8605 [2021-07-08 14:11:33,013][root][INFO] - ===> Epoch[6837](34185/5): Loss 0.0108, Sem 0.0108 LR: 4.681e-02 Score 99.611 Data time: 0.1827, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7609 [2021-07-08 14:11:47,135][root][INFO] - ===> Epoch[6838](34190/5): Loss 0.0104, Sem 0.0104 LR: 4.680e-02 Score 99.627 Data time: 0.1930, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7893 [2021-07-08 14:12:00,818][root][INFO] - ===> Epoch[6839](34195/5): Loss 0.0104, Sem 0.0104 LR: 4.680e-02 Score 99.625 Data time: 0.2300, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.6891 [2021-07-08 14:12:14,847][root][INFO] - ===> Epoch[6840](34200/5): Loss 0.0103, Sem 0.0103 LR: 4.679e-02 Score 99.624 Data time: 0.2235, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7436 [2021-07-08 14:12:14,849][root][INFO] - ===> Start testing [2021-07-08 14:12:52,629][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1462 Loss 0.354 (AVG: 0.583) Score 92.723 (AVG: 90.992) mIOU 71.407 mAP 83.956 mAcc 77.415 IOU: 64.021 0.000 82.782 74.939 94.617 92.436 30.938 84.424 98.649 81.696 79.902 86.382 57.499 mAP: 84.386 4.533 91.142 86.964 98.711 98.794 59.513 98.321 99.773 93.544 91.854 93.865 90.029 mAcc: 79.581 0.000 87.996 83.777 95.960 97.035 32.482 94.653 99.543 86.701 90.302 96.527 61.837 [2021-07-08 14:12:52,629][root][INFO] - Finished test. Elapsed time: 37.7795 [2021-07-08 14:12:53,697][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 14:13:06,429][root][INFO] - ===> Epoch[6841](34205/5): Loss 0.0109, Sem 0.0109 LR: 4.678e-02 Score 99.607 Data time: 0.1972, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.5112 [2021-07-08 14:13:20,439][root][INFO] - ===> Epoch[6842](34210/5): Loss 0.0111, Sem 0.0111 LR: 4.677e-02 Score 99.613 Data time: 0.1697, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7668 [2021-07-08 14:13:34,585][root][INFO] - ===> Epoch[6843](34215/5): Loss 0.0104, Sem 0.0104 LR: 4.676e-02 Score 99.622 Data time: 0.2628, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7796 [2021-07-08 14:13:48,582][root][INFO] - ===> Epoch[6844](34220/5): Loss 0.0104, Sem 0.0104 LR: 4.675e-02 Score 99.627 Data time: 0.2522, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7572 [2021-07-08 14:14:02,045][root][INFO] - ===> Epoch[6845](34225/5): Loss 0.0100, Sem 0.0100 LR: 4.675e-02 Score 99.641 Data time: 0.2792, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.6517 [2021-07-08 14:14:15,683][root][INFO] - ===> Epoch[6846](34230/5): Loss 0.0102, Sem 0.0102 LR: 4.674e-02 Score 99.632 Data time: 0.1993, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.6786 [2021-07-08 14:14:29,951][root][INFO] - ===> Epoch[6847](34235/5): Loss 0.0106, Sem 0.0106 LR: 4.673e-02 Score 99.615 Data time: 0.2454, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.8046 [2021-07-08 14:14:43,443][root][INFO] - ===> Epoch[6848](34240/5): Loss 0.0108, Sem 0.0108 LR: 4.672e-02 Score 99.615 Data time: 0.1677, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.6637 [2021-07-08 14:14:57,309][root][INFO] - ===> Epoch[6849](34245/5): Loss 0.0101, Sem 0.0101 LR: 4.671e-02 Score 99.637 Data time: 0.2437, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7241 [2021-07-08 14:15:10,669][root][INFO] - ===> Epoch[6850](34250/5): Loss 0.0104, Sem 0.0104 LR: 4.671e-02 Score 99.622 Data time: 0.2514, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.6187 [2021-07-08 14:15:25,121][root][INFO] - ===> Epoch[6851](34255/5): Loss 0.0105, Sem 0.0105 LR: 4.670e-02 Score 99.620 Data time: 0.2329, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.8434 [2021-07-08 14:15:39,677][root][INFO] - ===> Epoch[6852](34260/5): Loss 0.0103, Sem 0.0103 LR: 4.669e-02 Score 99.631 Data time: 0.2457, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.8694 [2021-07-08 14:15:53,639][root][INFO] - ===> Epoch[6853](34265/5): Loss 0.0101, Sem 0.0101 LR: 4.668e-02 Score 99.637 Data time: 0.2716, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7413 [2021-07-08 14:16:07,737][root][INFO] - ===> Epoch[6854](34270/5): Loss 0.0106, Sem 0.0106 LR: 4.667e-02 Score 99.614 Data time: 0.2185, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7753 [2021-07-08 14:16:21,966][root][INFO] - ===> Epoch[6855](34275/5): Loss 0.0106, Sem 0.0106 LR: 4.667e-02 Score 99.617 Data time: 0.2380, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8051 [2021-07-08 14:16:36,487][root][INFO] - ===> Epoch[6856](34280/5): Loss 0.0102, Sem 0.0102 LR: 4.666e-02 Score 99.632 Data time: 0.1997, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.8615 [2021-07-08 14:16:50,794][root][INFO] - ===> Epoch[6857](34285/5): Loss 0.0102, Sem 0.0102 LR: 4.665e-02 Score 99.634 Data time: 0.2526, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.8013 [2021-07-08 14:17:06,007][root][INFO] - ===> Epoch[6858](34290/5): Loss 0.0101, Sem 0.0101 LR: 4.664e-02 Score 99.635 Data time: 0.2007, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.9982 [2021-07-08 14:17:19,488][root][INFO] - ===> Epoch[6859](34295/5): Loss 0.0105, Sem 0.0105 LR: 4.663e-02 Score 99.621 Data time: 0.1891, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.6569 [2021-07-08 14:17:33,280][root][INFO] - ===> Epoch[6860](34300/5): Loss 0.0107, Sem 0.0107 LR: 4.662e-02 Score 99.612 Data time: 0.1746, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7206 [2021-07-08 14:17:46,363][root][INFO] - ===> Epoch[6861](34305/5): Loss 0.0103, Sem 0.0103 LR: 4.662e-02 Score 99.628 Data time: 0.2900, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.5724 [2021-07-08 14:18:00,129][root][INFO] - ===> Epoch[6862](34310/5): Loss 0.0106, Sem 0.0106 LR: 4.661e-02 Score 99.615 Data time: 0.2186, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.6992 [2021-07-08 14:18:14,422][root][INFO] - ===> Epoch[6863](34315/5): Loss 0.0100, Sem 0.0100 LR: 4.660e-02 Score 99.638 Data time: 0.2439, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.8142 [2021-07-08 14:18:27,799][root][INFO] - ===> Epoch[6864](34320/5): Loss 0.0104, Sem 0.0104 LR: 4.659e-02 Score 99.623 Data time: 0.2106, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.6383 [2021-07-08 14:18:41,609][root][INFO] - ===> Epoch[6865](34325/5): Loss 0.0099, Sem 0.0099 LR: 4.658e-02 Score 99.639 Data time: 0.1925, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7234 [2021-07-08 14:18:55,237][root][INFO] - ===> Epoch[6866](34330/5): Loss 0.0101, Sem 0.0101 LR: 4.658e-02 Score 99.631 Data time: 0.1901, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.6840 [2021-07-08 14:19:08,978][root][INFO] - ===> Epoch[6867](34335/5): Loss 0.0105, Sem 0.0105 LR: 4.657e-02 Score 99.620 Data time: 0.2303, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7095 [2021-07-08 14:19:23,414][root][INFO] - ===> Epoch[6868](34340/5): Loss 0.0105, Sem 0.0105 LR: 4.656e-02 Score 99.625 Data time: 0.2573, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.8368 [2021-07-08 14:19:36,645][root][INFO] - ===> Epoch[6869](34345/5): Loss 0.0106, Sem 0.0106 LR: 4.655e-02 Score 99.614 Data time: 0.2490, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.5947 [2021-07-08 14:19:50,599][root][INFO] - ===> Epoch[6870](34350/5): Loss 0.0097, Sem 0.0097 LR: 4.654e-02 Score 99.647 Data time: 0.2585, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7342 [2021-07-08 14:20:05,085][root][INFO] - ===> Epoch[6871](34355/5): Loss 0.0101, Sem 0.0101 LR: 4.653e-02 Score 99.638 Data time: 0.1901, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.8499 [2021-07-08 14:20:19,216][root][INFO] - ===> Epoch[6872](34360/5): Loss 0.0103, Sem 0.0103 LR: 4.653e-02 Score 99.626 Data time: 0.2256, Forward time: 1.2568, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7749 [2021-07-08 14:20:32,753][root][INFO] - ===> Epoch[6873](34365/5): Loss 0.0098, Sem 0.0098 LR: 4.652e-02 Score 99.646 Data time: 0.2098, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6659 [2021-07-08 14:20:46,564][root][INFO] - ===> Epoch[6874](34370/5): Loss 0.0102, Sem 0.0102 LR: 4.651e-02 Score 99.630 Data time: 0.2181, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7058 [2021-07-08 14:21:00,821][root][INFO] - ===> Epoch[6875](34375/5): Loss 0.0103, Sem 0.0103 LR: 4.650e-02 Score 99.629 Data time: 0.2463, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8022 [2021-07-08 14:21:14,624][root][INFO] - ===> Epoch[6876](34380/5): Loss 0.0100, Sem 0.0100 LR: 4.649e-02 Score 99.636 Data time: 0.2161, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7234 [2021-07-08 14:21:28,673][root][INFO] - ===> Epoch[6877](34385/5): Loss 0.0104, Sem 0.0104 LR: 4.649e-02 Score 99.635 Data time: 0.2197, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7464 [2021-07-08 14:21:43,266][root][INFO] - ===> Epoch[6878](34390/5): Loss 0.0106, Sem 0.0106 LR: 4.648e-02 Score 99.619 Data time: 0.2406, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8726 [2021-07-08 14:21:57,452][root][INFO] - ===> Epoch[6879](34395/5): Loss 0.0110, Sem 0.0110 LR: 4.647e-02 Score 99.609 Data time: 0.2424, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7937 [2021-07-08 14:22:11,132][root][INFO] - ===> Epoch[6880](34400/5): Loss 0.0105, Sem 0.0105 LR: 4.646e-02 Score 99.621 Data time: 0.2205, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6910 [2021-07-08 14:22:11,135][root][INFO] - ===> Start testing [2021-07-08 14:22:48,544][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1312 Loss 0.375 (AVG: 0.582) Score 92.682 (AVG: 90.992) mIOU 71.430 mAP 84.101 mAcc 77.516 IOU: 64.058 0.000 84.905 75.614 94.464 92.304 33.597 81.343 98.477 79.972 79.965 86.543 57.350 mAP: 83.910 5.614 93.782 87.564 98.648 98.740 60.470 98.148 99.811 91.888 91.901 93.878 88.963 mAcc: 80.104 0.000 89.326 84.541 95.692 96.846 36.067 94.661 99.363 83.199 89.693 96.144 62.078 [2021-07-08 14:22:48,545][root][INFO] - Finished test. Elapsed time: 37.4087 [2021-07-08 14:22:49,609][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 14:23:03,201][root][INFO] - ===> Epoch[6881](34405/5): Loss 0.0105, Sem 0.0105 LR: 4.645e-02 Score 99.621 Data time: 0.1907, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6732 [2021-07-08 14:23:17,135][root][INFO] - ===> Epoch[6882](34410/5): Loss 0.0101, Sem 0.0101 LR: 4.644e-02 Score 99.632 Data time: 0.2083, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7466 [2021-07-08 14:23:30,286][root][INFO] - ===> Epoch[6883](34415/5): Loss 0.0106, Sem 0.0106 LR: 4.644e-02 Score 99.621 Data time: 0.1945, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.5876 [2021-07-08 14:23:43,301][root][INFO] - ===> Epoch[6884](34420/5): Loss 0.0106, Sem 0.0106 LR: 4.643e-02 Score 99.619 Data time: 0.2230, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.5482 [2021-07-08 14:23:56,781][root][INFO] - ===> Epoch[6885](34425/5): Loss 0.0102, Sem 0.0102 LR: 4.642e-02 Score 99.630 Data time: 0.2408, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6482 [2021-07-08 14:24:11,604][root][INFO] - ===> Epoch[6886](34430/5): Loss 0.0102, Sem 0.0102 LR: 4.641e-02 Score 99.633 Data time: 0.2310, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.9204 [2021-07-08 14:24:25,205][root][INFO] - ===> Epoch[6887](34435/5): Loss 0.0105, Sem 0.0105 LR: 4.640e-02 Score 99.619 Data time: 0.2713, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6636 [2021-07-08 14:24:38,645][root][INFO] - ===> Epoch[6888](34440/5): Loss 0.0108, Sem 0.0108 LR: 4.640e-02 Score 99.609 Data time: 0.2579, Forward time: 1.2568, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.6385 [2021-07-08 14:24:52,028][root][INFO] - ===> Epoch[6889](34445/5): Loss 0.0102, Sem 0.0102 LR: 4.639e-02 Score 99.630 Data time: 0.2634, Forward time: 1.2568, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.6263 [2021-07-08 14:25:07,019][root][INFO] - ===> Epoch[6890](34450/5): Loss 0.0106, Sem 0.0106 LR: 4.638e-02 Score 99.621 Data time: 0.2743, Forward time: 1.2568, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.9520 [2021-07-08 14:25:21,335][root][INFO] - ===> Epoch[6891](34455/5): Loss 0.0106, Sem 0.0106 LR: 4.637e-02 Score 99.621 Data time: 0.2991, Forward time: 1.2568, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.8016 [2021-07-08 14:25:35,812][root][INFO] - ===> Epoch[6892](34460/5): Loss 0.0101, Sem 0.0101 LR: 4.636e-02 Score 99.634 Data time: 0.2753, Forward time: 1.2568, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.8464 [2021-07-08 14:25:49,828][root][INFO] - ===> Epoch[6893](34465/5): Loss 0.0108, Sem 0.0108 LR: 4.635e-02 Score 99.611 Data time: 0.2186, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.7409 [2021-07-08 14:26:04,140][root][INFO] - ===> Epoch[6894](34470/5): Loss 0.0107, Sem 0.0107 LR: 4.635e-02 Score 99.614 Data time: 0.2356, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.8142 [2021-07-08 14:26:18,464][root][INFO] - ===> Epoch[6895](34475/5): Loss 0.0114, Sem 0.0114 LR: 4.634e-02 Score 99.605 Data time: 0.1350, Forward time: 1.2568, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.8331 [2021-07-08 14:26:32,568][root][INFO] - ===> Epoch[6896](34480/5): Loss 0.0105, Sem 0.0105 LR: 4.633e-02 Score 99.621 Data time: 0.2263, Forward time: 1.2569, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.7822 [2021-07-08 14:26:46,245][root][INFO] - ===> Epoch[6897](34485/5): Loss 0.0103, Sem 0.0103 LR: 4.632e-02 Score 99.629 Data time: 0.2203, Forward time: 1.2568, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.6952 [2021-07-08 14:27:00,256][root][INFO] - ===> Epoch[6898](34490/5): Loss 0.0105, Sem 0.0105 LR: 4.631e-02 Score 99.619 Data time: 0.1894, Forward time: 1.2568, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 2.7662 [2021-07-08 14:27:15,553][root][INFO] - ===> Epoch[6899](34495/5): Loss 0.0102, Sem 0.0102 LR: 4.631e-02 Score 99.629 Data time: 0.2519, Forward time: 1.2568, Backward time: 1.2415, DDP time: 0.0042, Total iter time: 3.0139 [2021-07-08 14:27:28,981][root][INFO] - ===> Epoch[6900](34500/5): Loss 0.0107, Sem 0.0107 LR: 4.630e-02 Score 99.615 Data time: 0.1942, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6411 [2021-07-08 14:27:43,395][root][INFO] - ===> Epoch[6901](34505/5): Loss 0.0101, Sem 0.0101 LR: 4.629e-02 Score 99.633 Data time: 0.2253, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8421 [2021-07-08 14:27:57,096][root][INFO] - ===> Epoch[6902](34510/5): Loss 0.0104, Sem 0.0104 LR: 4.628e-02 Score 99.622 Data time: 0.1854, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6916 [2021-07-08 14:28:11,027][root][INFO] - ===> Epoch[6903](34515/5): Loss 0.0111, Sem 0.0111 LR: 4.627e-02 Score 99.615 Data time: 0.2489, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7405 [2021-07-08 14:28:24,553][root][INFO] - ===> Epoch[6904](34520/5): Loss 0.0104, Sem 0.0104 LR: 4.626e-02 Score 99.620 Data time: 0.2143, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6570 [2021-07-08 14:28:38,181][root][INFO] - ===> Epoch[6905](34525/5): Loss 0.0111, Sem 0.0111 LR: 4.626e-02 Score 99.594 Data time: 0.2294, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6842 [2021-07-08 14:28:52,557][root][INFO] - ===> Epoch[6906](34530/5): Loss 0.0100, Sem 0.0100 LR: 4.625e-02 Score 99.637 Data time: 0.2812, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8269 [2021-07-08 14:29:06,463][root][INFO] - ===> Epoch[6907](34535/5): Loss 0.0101, Sem 0.0101 LR: 4.624e-02 Score 99.635 Data time: 0.2323, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7363 [2021-07-08 14:29:20,381][root][INFO] - ===> Epoch[6908](34540/5): Loss 0.0105, Sem 0.0105 LR: 4.623e-02 Score 99.621 Data time: 0.2170, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7377 [2021-07-08 14:29:33,565][root][INFO] - ===> Epoch[6909](34545/5): Loss 0.0103, Sem 0.0103 LR: 4.622e-02 Score 99.628 Data time: 0.2356, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.5852 [2021-07-08 14:29:47,917][root][INFO] - ===> Epoch[6910](34550/5): Loss 0.0104, Sem 0.0104 LR: 4.622e-02 Score 99.627 Data time: 0.2527, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8285 [2021-07-08 14:30:01,728][root][INFO] - ===> Epoch[6911](34555/5): Loss 0.0099, Sem 0.0099 LR: 4.621e-02 Score 99.642 Data time: 0.2084, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7117 [2021-07-08 14:30:16,382][root][INFO] - ===> Epoch[6912](34560/5): Loss 0.0104, Sem 0.0104 LR: 4.620e-02 Score 99.623 Data time: 0.1824, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8855 [2021-07-08 14:30:29,862][root][INFO] - ===> Epoch[6913](34565/5): Loss 0.0099, Sem 0.0099 LR: 4.619e-02 Score 99.642 Data time: 0.1957, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6599 [2021-07-08 14:30:44,719][root][INFO] - ===> Epoch[6914](34570/5): Loss 0.0100, Sem 0.0100 LR: 4.618e-02 Score 99.640 Data time: 0.2594, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.9215 [2021-07-08 14:30:58,424][root][INFO] - ===> Epoch[6915](34575/5): Loss 0.0104, Sem 0.0104 LR: 4.618e-02 Score 99.625 Data time: 0.2379, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6874 [2021-07-08 14:31:12,102][root][INFO] - ===> Epoch[6916](34580/5): Loss 0.0105, Sem 0.0105 LR: 4.617e-02 Score 99.618 Data time: 0.1945, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6927 [2021-07-08 14:31:25,794][root][INFO] - ===> Epoch[6917](34585/5): Loss 0.0102, Sem 0.0102 LR: 4.616e-02 Score 99.630 Data time: 0.1604, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7035 [2021-07-08 14:31:40,286][root][INFO] - ===> Epoch[6918](34590/5): Loss 0.0098, Sem 0.0098 LR: 4.615e-02 Score 99.648 Data time: 0.2430, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8558 [2021-07-08 14:31:53,983][root][INFO] - ===> Epoch[6919](34595/5): Loss 0.0105, Sem 0.0105 LR: 4.614e-02 Score 99.620 Data time: 0.1755, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6962 [2021-07-08 14:32:08,381][root][INFO] - ===> Epoch[6920](34600/5): Loss 0.0103, Sem 0.0103 LR: 4.613e-02 Score 99.625 Data time: 0.2069, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8310 [2021-07-08 14:32:08,383][root][INFO] - ===> Start testing [2021-07-08 14:32:43,566][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1231 Loss 0.378 (AVG: 0.585) Score 92.184 (AVG: 90.864) mIOU 71.333 mAP 83.637 mAcc 77.392 IOU: 63.698 0.000 84.851 74.338 94.775 91.608 35.583 83.334 98.528 78.731 78.387 86.459 57.038 mAP: 83.201 3.952 94.006 87.203 98.791 98.810 60.322 97.783 99.794 87.856 91.611 94.119 89.829 mAcc: 80.466 0.000 90.522 82.204 95.978 97.233 38.055 94.041 99.352 80.890 88.434 96.335 62.589 [2021-07-08 14:32:43,566][root][INFO] - Finished test. Elapsed time: 35.1824 [2021-07-08 14:32:44,641][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 14:32:57,863][root][INFO] - ===> Epoch[6921](34605/5): Loss 0.0100, Sem 0.0100 LR: 4.613e-02 Score 99.633 Data time: 0.1986, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6072 [2021-07-08 14:33:10,748][root][INFO] - ===> Epoch[6922](34610/5): Loss 0.0110, Sem 0.0110 LR: 4.612e-02 Score 99.604 Data time: 0.2288, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.5249 [2021-07-08 14:33:24,966][root][INFO] - ===> Epoch[6923](34615/5): Loss 0.0107, Sem 0.0107 LR: 4.611e-02 Score 99.616 Data time: 0.2055, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7880 [2021-07-08 14:33:38,599][root][INFO] - ===> Epoch[6924](34620/5): Loss 0.0108, Sem 0.0108 LR: 4.610e-02 Score 99.610 Data time: 0.2756, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6646 [2021-07-08 14:33:53,163][root][INFO] - ===> Epoch[6925](34625/5): Loss 0.0099, Sem 0.0099 LR: 4.609e-02 Score 99.645 Data time: 0.2452, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8723 [2021-07-08 14:34:07,892][root][INFO] - ===> Epoch[6926](34630/5): Loss 0.0110, Sem 0.0110 LR: 4.609e-02 Score 99.605 Data time: 0.2568, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8975 [2021-07-08 14:34:22,324][root][INFO] - ===> Epoch[6927](34635/5): Loss 0.0106, Sem 0.0106 LR: 4.608e-02 Score 99.616 Data time: 0.2960, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8425 [2021-07-08 14:34:36,298][root][INFO] - ===> Epoch[6928](34640/5): Loss 0.0106, Sem 0.0106 LR: 4.607e-02 Score 99.618 Data time: 0.2373, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7510 [2021-07-08 14:34:50,067][root][INFO] - ===> Epoch[6929](34645/5): Loss 0.0102, Sem 0.0102 LR: 4.606e-02 Score 99.630 Data time: 0.2562, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7164 [2021-07-08 14:35:04,879][root][INFO] - ===> Epoch[6930](34650/5): Loss 0.0103, Sem 0.0103 LR: 4.605e-02 Score 99.633 Data time: 0.2759, Forward time: 1.2568, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.9189 [2021-07-08 14:35:19,194][root][INFO] - ===> Epoch[6931](34655/5): Loss 0.0102, Sem 0.0102 LR: 4.604e-02 Score 99.632 Data time: 0.2461, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8055 [2021-07-08 14:35:33,756][root][INFO] - ===> Epoch[6932](34660/5): Loss 0.0103, Sem 0.0103 LR: 4.604e-02 Score 99.630 Data time: 0.1933, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8737 [2021-07-08 14:35:47,512][root][INFO] - ===> Epoch[6933](34665/5): Loss 0.0100, Sem 0.0100 LR: 4.603e-02 Score 99.640 Data time: 0.2109, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7024 [2021-07-08 14:36:01,261][root][INFO] - ===> Epoch[6934](34670/5): Loss 0.0103, Sem 0.0103 LR: 4.602e-02 Score 99.625 Data time: 0.2069, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7076 [2021-07-08 14:36:14,793][root][INFO] - ===> Epoch[6935](34675/5): Loss 0.0100, Sem 0.0100 LR: 4.601e-02 Score 99.639 Data time: 0.2296, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6641 [2021-07-08 14:36:27,945][root][INFO] - ===> Epoch[6936](34680/5): Loss 0.0109, Sem 0.0109 LR: 4.600e-02 Score 99.609 Data time: 0.2288, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.5790 [2021-07-08 14:36:41,637][root][INFO] - ===> Epoch[6937](34685/5): Loss 0.0104, Sem 0.0104 LR: 4.600e-02 Score 99.624 Data time: 0.2144, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6944 [2021-07-08 14:36:55,103][root][INFO] - ===> Epoch[6938](34690/5): Loss 0.0102, Sem 0.0102 LR: 4.599e-02 Score 99.630 Data time: 0.1626, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6493 [2021-07-08 14:37:09,116][root][INFO] - ===> Epoch[6939](34695/5): Loss 0.0100, Sem 0.0100 LR: 4.598e-02 Score 99.638 Data time: 0.2508, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7605 [2021-07-08 14:37:23,355][root][INFO] - ===> Epoch[6940](34700/5): Loss 0.0101, Sem 0.0101 LR: 4.597e-02 Score 99.632 Data time: 0.2336, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7955 [2021-07-08 14:37:36,905][root][INFO] - ===> Epoch[6941](34705/5): Loss 0.0095, Sem 0.0095 LR: 4.596e-02 Score 99.657 Data time: 0.2300, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6659 [2021-07-08 14:37:51,080][root][INFO] - ===> Epoch[6942](34710/5): Loss 0.0102, Sem 0.0102 LR: 4.595e-02 Score 99.631 Data time: 0.1958, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7995 [2021-07-08 14:38:05,281][root][INFO] - ===> Epoch[6943](34715/5): Loss 0.0100, Sem 0.0100 LR: 4.595e-02 Score 99.636 Data time: 0.1454, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7998 [2021-07-08 14:38:19,174][root][INFO] - ===> Epoch[6944](34720/5): Loss 0.0103, Sem 0.0103 LR: 4.594e-02 Score 99.626 Data time: 0.2509, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7384 [2021-07-08 14:38:34,205][root][INFO] - ===> Epoch[6945](34725/5): Loss 0.0102, Sem 0.0102 LR: 4.593e-02 Score 99.637 Data time: 0.2278, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.9602 [2021-07-08 14:38:48,095][root][INFO] - ===> Epoch[6946](34730/5): Loss 0.0101, Sem 0.0101 LR: 4.592e-02 Score 99.631 Data time: 0.2328, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.7334 [2021-07-08 14:39:01,547][root][INFO] - ===> Epoch[6947](34735/5): Loss 0.0102, Sem 0.0102 LR: 4.591e-02 Score 99.632 Data time: 0.1991, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6503 [2021-07-08 14:39:15,067][root][INFO] - ===> Epoch[6948](34740/5): Loss 0.0100, Sem 0.0100 LR: 4.591e-02 Score 99.639 Data time: 0.2069, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6600 [2021-07-08 14:39:28,659][root][INFO] - ===> Epoch[6949](34745/5): Loss 0.0099, Sem 0.0099 LR: 4.590e-02 Score 99.642 Data time: 0.1714, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.6733 [2021-07-08 14:39:43,329][root][INFO] - ===> Epoch[6950](34750/5): Loss 0.0100, Sem 0.0100 LR: 4.589e-02 Score 99.637 Data time: 0.2091, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8994 [2021-07-08 14:39:57,667][root][INFO] - ===> Epoch[6951](34755/5): Loss 0.0095, Sem 0.0095 LR: 4.588e-02 Score 99.657 Data time: 0.2257, Forward time: 1.2567, Backward time: 1.2416, DDP time: 0.0042, Total iter time: 2.8189 [2021-07-08 14:40:12,014][root][INFO] - ===> Epoch[6952](34760/5): Loss 0.0104, Sem 0.0104 LR: 4.587e-02 Score 99.620 Data time: 0.1616, Forward time: 1.2566, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.8352 [2021-07-08 14:40:25,435][root][INFO] - ===> Epoch[6953](34765/5): Loss 0.0102, Sem 0.0102 LR: 4.586e-02 Score 99.632 Data time: 0.1897, Forward time: 1.2566, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.6420 [2021-07-08 14:40:38,900][root][INFO] - ===> Epoch[6954](34770/5): Loss 0.0102, Sem 0.0102 LR: 4.586e-02 Score 99.634 Data time: 0.1849, Forward time: 1.2566, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.6497 [2021-07-08 14:40:53,141][root][INFO] - ===> Epoch[6955](34775/5): Loss 0.0099, Sem 0.0099 LR: 4.585e-02 Score 99.643 Data time: 0.2237, Forward time: 1.2566, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7904 [2021-07-08 14:41:06,643][root][INFO] - ===> Epoch[6956](34780/5): Loss 0.0103, Sem 0.0103 LR: 4.584e-02 Score 99.627 Data time: 0.1759, Forward time: 1.2566, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.6658 [2021-07-08 14:41:20,584][root][INFO] - ===> Epoch[6957](34785/5): Loss 0.0096, Sem 0.0096 LR: 4.583e-02 Score 99.650 Data time: 0.2698, Forward time: 1.2566, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7436 [2021-07-08 14:41:34,515][root][INFO] - ===> Epoch[6958](34790/5): Loss 0.0100, Sem 0.0100 LR: 4.582e-02 Score 99.641 Data time: 0.1653, Forward time: 1.2566, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.7491 [2021-07-08 14:41:47,272][root][INFO] - ===> Epoch[6959](34795/5): Loss 0.0105, Sem 0.0105 LR: 4.582e-02 Score 99.621 Data time: 0.2026, Forward time: 1.2565, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.5017 [2021-07-08 14:42:01,488][root][INFO] - ===> Epoch[6960](34800/5): Loss 0.0096, Sem 0.0096 LR: 4.581e-02 Score 99.653 Data time: 0.2678, Forward time: 1.2565, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7809 [2021-07-08 14:42:01,490][root][INFO] - ===> Start testing [2021-07-08 14:42:35,625][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1221 Loss 0.325 (AVG: 0.580) Score 93.382 (AVG: 91.009) mIOU 71.656 mAP 83.878 mAcc 77.621 IOU: 63.967 0.000 85.554 75.184 94.608 92.188 34.103 83.139 98.513 81.051 79.637 86.622 56.958 mAP: 83.801 5.832 92.569 87.189 98.690 98.781 59.295 98.241 99.802 91.019 91.514 94.239 89.443 mAcc: 79.829 0.000 88.681 83.816 96.170 97.521 36.844 94.958 99.302 84.609 89.583 96.274 61.480 [2021-07-08 14:42:35,625][root][INFO] - Finished test. Elapsed time: 34.1344 [2021-07-08 14:42:36,694][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 14:42:50,445][root][INFO] - ===> Epoch[6961](34805/5): Loss 0.0101, Sem 0.0101 LR: 4.580e-02 Score 99.633 Data time: 0.2178, Forward time: 1.2565, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7190 [2021-07-08 14:43:04,748][root][INFO] - ===> Epoch[6962](34810/5): Loss 0.0102, Sem 0.0102 LR: 4.579e-02 Score 99.628 Data time: 0.2847, Forward time: 1.2565, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.8098 [2021-07-08 14:43:18,419][root][INFO] - ===> Epoch[6963](34815/5): Loss 0.0100, Sem 0.0100 LR: 4.578e-02 Score 99.636 Data time: 0.2577, Forward time: 1.2566, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.6896 [2021-07-08 14:43:33,288][root][INFO] - ===> Epoch[6964](34820/5): Loss 0.0102, Sem 0.0102 LR: 4.577e-02 Score 99.633 Data time: 0.2580, Forward time: 1.2566, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.9249 [2021-07-08 14:43:47,416][root][INFO] - ===> Epoch[6965](34825/5): Loss 0.0103, Sem 0.0103 LR: 4.577e-02 Score 99.630 Data time: 0.1832, Forward time: 1.2566, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7807 [2021-07-08 14:44:01,164][root][INFO] - ===> Epoch[6966](34830/5): Loss 0.0102, Sem 0.0102 LR: 4.576e-02 Score 99.632 Data time: 0.2122, Forward time: 1.2565, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7050 [2021-07-08 14:44:16,062][root][INFO] - ===> Epoch[6967](34835/5): Loss 0.0103, Sem 0.0103 LR: 4.575e-02 Score 99.636 Data time: 0.2585, Forward time: 1.2565, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.9337 [2021-07-08 14:44:30,655][root][INFO] - ===> Epoch[6968](34840/5): Loss 0.0098, Sem 0.0098 LR: 4.574e-02 Score 99.646 Data time: 0.2737, Forward time: 1.2565, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.8673 [2021-07-08 14:44:44,512][root][INFO] - ===> Epoch[6969](34845/5): Loss 0.0104, Sem 0.0104 LR: 4.573e-02 Score 99.628 Data time: 0.1956, Forward time: 1.2565, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.7230 [2021-07-08 14:44:57,638][root][INFO] - ===> Epoch[6970](34850/5): Loss 0.0107, Sem 0.0107 LR: 4.573e-02 Score 99.617 Data time: 0.2563, Forward time: 1.2565, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.5867 [2021-07-08 14:45:11,877][root][INFO] - ===> Epoch[6971](34855/5): Loss 0.0098, Sem 0.0098 LR: 4.572e-02 Score 99.651 Data time: 0.2434, Forward time: 1.2565, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.8041 [2021-07-08 14:45:25,686][root][INFO] - ===> Epoch[6972](34860/5): Loss 0.0105, Sem 0.0105 LR: 4.571e-02 Score 99.621 Data time: 0.2558, Forward time: 1.2565, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7155 [2021-07-08 14:45:39,449][root][INFO] - ===> Epoch[6973](34865/5): Loss 0.0107, Sem 0.0107 LR: 4.570e-02 Score 99.615 Data time: 0.2324, Forward time: 1.2565, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.7135 [2021-07-08 14:45:52,922][root][INFO] - ===> Epoch[6974](34870/5): Loss 0.0101, Sem 0.0101 LR: 4.569e-02 Score 99.634 Data time: 0.2540, Forward time: 1.2565, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.6447 [2021-07-08 14:46:06,902][root][INFO] - ===> Epoch[6975](34875/5): Loss 0.0101, Sem 0.0101 LR: 4.568e-02 Score 99.638 Data time: 0.2200, Forward time: 1.2565, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7462 [2021-07-08 14:46:20,717][root][INFO] - ===> Epoch[6976](34880/5): Loss 0.0102, Sem 0.0102 LR: 4.568e-02 Score 99.634 Data time: 0.1847, Forward time: 1.2565, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7217 [2021-07-08 14:46:34,507][root][INFO] - ===> Epoch[6977](34885/5): Loss 0.0102, Sem 0.0102 LR: 4.567e-02 Score 99.636 Data time: 0.1882, Forward time: 1.2565, Backward time: 1.2417, DDP time: 0.0042, Total iter time: 2.7208 [2021-07-08 14:46:48,612][root][INFO] - ===> Epoch[6978](34890/5): Loss 0.0103, Sem 0.0103 LR: 4.566e-02 Score 99.627 Data time: 0.1887, Forward time: 1.2565, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.7722 [2021-07-08 14:47:03,031][root][INFO] - ===> Epoch[6979](34895/5): Loss 0.0106, Sem 0.0106 LR: 4.565e-02 Score 99.619 Data time: 0.2267, Forward time: 1.2565, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.8435 [2021-07-08 14:47:16,528][root][INFO] - ===> Epoch[6980](34900/5): Loss 0.0108, Sem 0.0108 LR: 4.564e-02 Score 99.613 Data time: 0.1956, Forward time: 1.2565, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.6621 [2021-07-08 14:47:30,918][root][INFO] - ===> Epoch[6981](34905/5): Loss 0.0103, Sem 0.0103 LR: 4.564e-02 Score 99.631 Data time: 0.2649, Forward time: 1.2565, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.8318 [2021-07-08 14:47:44,760][root][INFO] - ===> Epoch[6982](34910/5): Loss 0.0103, Sem 0.0103 LR: 4.563e-02 Score 99.627 Data time: 0.2382, Forward time: 1.2565, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.7238 [2021-07-08 14:47:58,262][root][INFO] - ===> Epoch[6983](34915/5): Loss 0.0105, Sem 0.0105 LR: 4.562e-02 Score 99.621 Data time: 0.2507, Forward time: 1.2564, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.6581 [2021-07-08 14:48:12,785][root][INFO] - ===> Epoch[6984](34920/5): Loss 0.0100, Sem 0.0100 LR: 4.561e-02 Score 99.639 Data time: 0.1926, Forward time: 1.2564, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.8566 [2021-07-08 14:48:26,951][root][INFO] - ===> Epoch[6985](34925/5): Loss 0.0105, Sem 0.0105 LR: 4.560e-02 Score 99.620 Data time: 0.2472, Forward time: 1.2564, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.7870 [2021-07-08 14:48:40,564][root][INFO] - ===> Epoch[6986](34930/5): Loss 0.0102, Sem 0.0102 LR: 4.559e-02 Score 99.633 Data time: 0.2151, Forward time: 1.2564, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.6719 [2021-07-08 14:48:53,959][root][INFO] - ===> Epoch[6987](34935/5): Loss 0.0103, Sem 0.0103 LR: 4.559e-02 Score 99.632 Data time: 0.1920, Forward time: 1.2564, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.6434 [2021-07-08 14:49:07,627][root][INFO] - ===> Epoch[6988](34940/5): Loss 0.0107, Sem 0.0107 LR: 4.558e-02 Score 99.616 Data time: 0.2730, Forward time: 1.2564, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.6896 [2021-07-08 14:49:21,802][root][INFO] - ===> Epoch[6989](34945/5): Loss 0.0097, Sem 0.0097 LR: 4.557e-02 Score 99.647 Data time: 0.2038, Forward time: 1.2564, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.7847 [2021-07-08 14:49:35,725][root][INFO] - ===> Epoch[6990](34950/5): Loss 0.0100, Sem 0.0100 LR: 4.556e-02 Score 99.639 Data time: 0.2352, Forward time: 1.2564, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.7249 [2021-07-08 14:49:49,379][root][INFO] - ===> Epoch[6991](34955/5): Loss 0.0099, Sem 0.0099 LR: 4.555e-02 Score 99.641 Data time: 0.1895, Forward time: 1.2564, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.6895 [2021-07-08 14:50:03,104][root][INFO] - ===> Epoch[6992](34960/5): Loss 0.0096, Sem 0.0096 LR: 4.555e-02 Score 99.655 Data time: 0.1980, Forward time: 1.2564, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.6961 [2021-07-08 14:50:17,862][root][INFO] - ===> Epoch[6993](34965/5): Loss 0.0095, Sem 0.0095 LR: 4.554e-02 Score 99.658 Data time: 0.2258, Forward time: 1.2564, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.9041 [2021-07-08 14:50:31,578][root][INFO] - ===> Epoch[6994](34970/5): Loss 0.0097, Sem 0.0097 LR: 4.553e-02 Score 99.650 Data time: 0.1917, Forward time: 1.2564, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.6870 [2021-07-08 14:50:46,053][root][INFO] - ===> Epoch[6995](34975/5): Loss 0.0099, Sem 0.0099 LR: 4.552e-02 Score 99.638 Data time: 0.1992, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.8421 [2021-07-08 14:50:59,925][root][INFO] - ===> Epoch[6996](34980/5): Loss 0.0099, Sem 0.0099 LR: 4.551e-02 Score 99.646 Data time: 0.2262, Forward time: 1.2564, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7332 [2021-07-08 14:51:13,903][root][INFO] - ===> Epoch[6997](34985/5): Loss 0.0103, Sem 0.0103 LR: 4.550e-02 Score 99.629 Data time: 0.1612, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7525 [2021-07-08 14:51:27,828][root][INFO] - ===> Epoch[6998](34990/5): Loss 0.0101, Sem 0.0101 LR: 4.550e-02 Score 99.633 Data time: 0.2011, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7350 [2021-07-08 14:51:42,030][root][INFO] - ===> Epoch[6999](34995/5): Loss 0.0098, Sem 0.0098 LR: 4.549e-02 Score 99.648 Data time: 0.2056, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7893 [2021-07-08 14:51:56,365][root][INFO] - ===> Epoch[7000](35000/5): Loss 0.0100, Sem 0.0100 LR: 4.548e-02 Score 99.639 Data time: 0.1934, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.8235 [2021-07-08 14:51:56,367][root][INFO] - ===> Start testing [2021-07-08 14:52:35,541][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1308 Loss 0.410 (AVG: 0.582) Score 92.190 (AVG: 90.948) mIOU 71.732 mAP 83.909 mAcc 77.667 IOU: 64.336 0.000 84.144 75.148 94.583 92.214 30.044 85.315 98.693 83.796 80.693 86.070 57.478 mAP: 83.917 5.844 92.659 87.367 98.773 98.778 55.835 98.049 99.791 94.072 91.850 93.502 90.378 mAcc: 81.129 0.000 88.757 83.142 95.697 97.701 32.066 94.560 99.512 88.467 89.531 96.178 62.932 [2021-07-08 14:52:35,542][root][INFO] - Finished test. Elapsed time: 39.1732 [2021-07-08 14:52:36,674][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 14:52:49,971][root][INFO] - ===> Epoch[7001](35005/5): Loss 0.0095, Sem 0.0095 LR: 4.547e-02 Score 99.654 Data time: 0.2381, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.6167 [2021-07-08 14:53:04,722][root][INFO] - ===> Epoch[7002](35010/5): Loss 0.0094, Sem 0.0094 LR: 4.546e-02 Score 99.657 Data time: 0.2883, Forward time: 1.2564, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.8896 [2021-07-08 14:53:19,801][root][INFO] - ===> Epoch[7003](35015/5): Loss 0.0098, Sem 0.0098 LR: 4.546e-02 Score 99.649 Data time: 0.2549, Forward time: 1.2564, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.9677 [2021-07-08 14:53:33,380][root][INFO] - ===> Epoch[7004](35020/5): Loss 0.0100, Sem 0.0100 LR: 4.545e-02 Score 99.637 Data time: 0.2491, Forward time: 1.2564, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.6632 [2021-07-08 14:53:47,257][root][INFO] - ===> Epoch[7005](35025/5): Loss 0.0099, Sem 0.0099 LR: 4.544e-02 Score 99.642 Data time: 0.1946, Forward time: 1.2564, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7312 [2021-07-08 14:54:02,116][root][INFO] - ===> Epoch[7006](35030/5): Loss 0.0099, Sem 0.0099 LR: 4.543e-02 Score 99.645 Data time: 0.2127, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.9287 [2021-07-08 14:54:15,951][root][INFO] - ===> Epoch[7007](35035/5): Loss 0.0099, Sem 0.0099 LR: 4.542e-02 Score 99.643 Data time: 0.2155, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7264 [2021-07-08 14:54:29,994][root][INFO] - ===> Epoch[7008](35040/5): Loss 0.0097, Sem 0.0097 LR: 4.541e-02 Score 99.649 Data time: 0.2282, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7578 [2021-07-08 14:54:43,683][root][INFO] - ===> Epoch[7009](35045/5): Loss 0.0097, Sem 0.0097 LR: 4.541e-02 Score 99.645 Data time: 0.2513, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.6898 [2021-07-08 14:54:57,319][root][INFO] - ===> Epoch[7010](35050/5): Loss 0.0103, Sem 0.0103 LR: 4.540e-02 Score 99.628 Data time: 0.2342, Forward time: 1.2564, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.6679 [2021-07-08 14:55:11,647][root][INFO] - ===> Epoch[7011](35055/5): Loss 0.0097, Sem 0.0097 LR: 4.539e-02 Score 99.650 Data time: 0.2433, Forward time: 1.2564, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.8129 [2021-07-08 14:55:25,542][root][INFO] - ===> Epoch[7012](35060/5): Loss 0.0093, Sem 0.0093 LR: 4.538e-02 Score 99.663 Data time: 0.2018, Forward time: 1.2564, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7366 [2021-07-08 14:55:39,151][root][INFO] - ===> Epoch[7013](35065/5): Loss 0.0100, Sem 0.0100 LR: 4.537e-02 Score 99.639 Data time: 0.2665, Forward time: 1.2564, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.6783 [2021-07-08 14:55:53,610][root][INFO] - ===> Epoch[7014](35070/5): Loss 0.0097, Sem 0.0097 LR: 4.537e-02 Score 99.651 Data time: 0.1918, Forward time: 1.2564, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.8451 [2021-07-08 14:56:07,849][root][INFO] - ===> Epoch[7015](35075/5): Loss 0.0101, Sem 0.0101 LR: 4.536e-02 Score 99.639 Data time: 0.2583, Forward time: 1.2564, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.8051 [2021-07-08 14:56:21,554][root][INFO] - ===> Epoch[7016](35080/5): Loss 0.0102, Sem 0.0102 LR: 4.535e-02 Score 99.632 Data time: 0.2159, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7056 [2021-07-08 14:56:35,451][root][INFO] - ===> Epoch[7017](35085/5): Loss 0.0095, Sem 0.0095 LR: 4.534e-02 Score 99.660 Data time: 0.1998, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7307 [2021-07-08 14:56:49,567][root][INFO] - ===> Epoch[7018](35090/5): Loss 0.0101, Sem 0.0101 LR: 4.533e-02 Score 99.637 Data time: 0.2546, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7705 [2021-07-08 14:57:03,158][root][INFO] - ===> Epoch[7019](35095/5): Loss 0.0099, Sem 0.0099 LR: 4.532e-02 Score 99.642 Data time: 0.2065, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.6694 [2021-07-08 14:57:16,750][root][INFO] - ===> Epoch[7020](35100/5): Loss 0.0094, Sem 0.0094 LR: 4.532e-02 Score 99.658 Data time: 0.2388, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.6743 [2021-07-08 14:57:30,217][root][INFO] - ===> Epoch[7021](35105/5): Loss 0.0102, Sem 0.0102 LR: 4.531e-02 Score 99.636 Data time: 0.2289, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.6520 [2021-07-08 14:57:43,253][root][INFO] - ===> Epoch[7022](35110/5): Loss 0.0103, Sem 0.0103 LR: 4.530e-02 Score 99.631 Data time: 0.2367, Forward time: 1.2563, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.5489 [2021-07-08 14:57:56,589][root][INFO] - ===> Epoch[7023](35115/5): Loss 0.0105, Sem 0.0105 LR: 4.529e-02 Score 99.625 Data time: 0.1725, Forward time: 1.2563, Backward time: 1.2418, DDP time: 0.0042, Total iter time: 2.6208 [2021-07-08 14:58:10,736][root][INFO] - ===> Epoch[7024](35120/5): Loss 0.0102, Sem 0.0102 LR: 4.528e-02 Score 99.634 Data time: 0.1695, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7815 [2021-07-08 14:58:24,886][root][INFO] - ===> Epoch[7025](35125/5): Loss 0.0097, Sem 0.0097 LR: 4.528e-02 Score 99.651 Data time: 0.1836, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7840 [2021-07-08 14:58:38,963][root][INFO] - ===> Epoch[7026](35130/5): Loss 0.0115, Sem 0.0115 LR: 4.527e-02 Score 99.604 Data time: 0.1910, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7695 [2021-07-08 14:58:52,600][root][INFO] - ===> Epoch[7027](35135/5): Loss 0.0099, Sem 0.0099 LR: 4.526e-02 Score 99.645 Data time: 0.2603, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.6813 [2021-07-08 14:59:06,340][root][INFO] - ===> Epoch[7028](35140/5): Loss 0.0103, Sem 0.0103 LR: 4.525e-02 Score 99.632 Data time: 0.1972, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7026 [2021-07-08 14:59:20,728][root][INFO] - ===> Epoch[7029](35145/5): Loss 0.0105, Sem 0.0105 LR: 4.524e-02 Score 99.635 Data time: 0.1897, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.8317 [2021-07-08 14:59:34,315][root][INFO] - ===> Epoch[7030](35150/5): Loss 0.0104, Sem 0.0104 LR: 4.523e-02 Score 99.621 Data time: 0.1928, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.6707 [2021-07-08 14:59:48,947][root][INFO] - ===> Epoch[7031](35155/5): Loss 0.0102, Sem 0.0102 LR: 4.523e-02 Score 99.632 Data time: 0.2388, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.8704 [2021-07-08 15:00:02,489][root][INFO] - ===> Epoch[7032](35160/5): Loss 0.0107, Sem 0.0107 LR: 4.522e-02 Score 99.622 Data time: 0.1633, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.6587 [2021-07-08 15:00:16,189][root][INFO] - ===> Epoch[7033](35165/5): Loss 0.0099, Sem 0.0099 LR: 4.521e-02 Score 99.641 Data time: 0.2272, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7023 [2021-07-08 15:00:30,548][root][INFO] - ===> Epoch[7034](35170/5): Loss 0.0103, Sem 0.0103 LR: 4.520e-02 Score 99.627 Data time: 0.1648, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.8317 [2021-07-08 15:00:44,679][root][INFO] - ===> Epoch[7035](35175/5): Loss 0.0101, Sem 0.0101 LR: 4.519e-02 Score 99.635 Data time: 0.2381, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7805 [2021-07-08 15:00:59,563][root][INFO] - ===> Epoch[7036](35180/5): Loss 0.0100, Sem 0.0100 LR: 4.519e-02 Score 99.634 Data time: 0.2551, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.9275 [2021-07-08 15:01:13,768][root][INFO] - ===> Epoch[7037](35185/5): Loss 0.0097, Sem 0.0097 LR: 4.518e-02 Score 99.648 Data time: 0.2325, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7922 [2021-07-08 15:01:27,957][root][INFO] - ===> Epoch[7038](35190/5): Loss 0.0100, Sem 0.0100 LR: 4.517e-02 Score 99.646 Data time: 0.2902, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7862 [2021-07-08 15:01:42,112][root][INFO] - ===> Epoch[7039](35195/5): Loss 0.0099, Sem 0.0099 LR: 4.516e-02 Score 99.643 Data time: 0.2927, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7816 [2021-07-08 15:01:56,733][root][INFO] - ===> Epoch[7040](35200/5): Loss 0.0097, Sem 0.0097 LR: 4.515e-02 Score 99.650 Data time: 0.2985, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.8776 [2021-07-08 15:01:56,734][root][INFO] - ===> Start testing [2021-07-08 15:02:33,248][root][INFO] - 68/68: Data time: 0.0016, Iter time: 0.1230 Loss 0.356 (AVG: 0.575) Score 92.515 (AVG: 91.114) mIOU 71.844 mAP 83.889 mAcc 77.840 IOU: 64.579 0.000 84.245 75.350 95.129 91.705 33.610 82.655 98.675 84.000 79.589 86.709 57.727 mAP: 83.851 5.027 91.085 87.245 98.796 98.524 58.176 98.003 99.803 93.798 92.060 93.919 90.266 mAcc: 80.859 0.000 88.554 83.932 96.264 97.396 35.910 94.081 99.434 88.052 88.624 96.229 62.588 [2021-07-08 15:02:33,248][root][INFO] - Finished test. Elapsed time: 36.5129 [2021-07-08 15:02:34,274][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 15:02:47,139][root][INFO] - ===> Epoch[7041](35205/5): Loss 0.0097, Sem 0.0097 LR: 4.514e-02 Score 99.650 Data time: 0.1930, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.5365 [2021-07-08 15:03:01,561][root][INFO] - ===> Epoch[7042](35210/5): Loss 0.0094, Sem 0.0094 LR: 4.514e-02 Score 99.660 Data time: 0.2113, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.8315 [2021-07-08 15:03:14,838][root][INFO] - ===> Epoch[7043](35215/5): Loss 0.0099, Sem 0.0099 LR: 4.513e-02 Score 99.645 Data time: 0.1816, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.6117 [2021-07-08 15:03:28,973][root][INFO] - ===> Epoch[7044](35220/5): Loss 0.0098, Sem 0.0098 LR: 4.512e-02 Score 99.648 Data time: 0.2086, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7813 [2021-07-08 15:03:44,108][root][INFO] - ===> Epoch[7045](35225/5): Loss 0.0099, Sem 0.0099 LR: 4.511e-02 Score 99.642 Data time: 0.2568, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.9784 [2021-07-08 15:03:58,350][root][INFO] - ===> Epoch[7046](35230/5): Loss 0.0094, Sem 0.0094 LR: 4.510e-02 Score 99.663 Data time: 0.1918, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.8145 [2021-07-08 15:04:12,376][root][INFO] - ===> Epoch[7047](35235/5): Loss 0.0096, Sem 0.0096 LR: 4.509e-02 Score 99.655 Data time: 0.2223, Forward time: 1.2563, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7698 [2021-07-08 15:04:26,476][root][INFO] - ===> Epoch[7048](35240/5): Loss 0.0096, Sem 0.0096 LR: 4.509e-02 Score 99.655 Data time: 0.2322, Forward time: 1.2563, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.7766 [2021-07-08 15:04:40,398][root][INFO] - ===> Epoch[7049](35245/5): Loss 0.0095, Sem 0.0095 LR: 4.508e-02 Score 99.656 Data time: 0.2276, Forward time: 1.2563, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.7421 [2021-07-08 15:04:54,748][root][INFO] - ===> Epoch[7050](35250/5): Loss 0.0104, Sem 0.0104 LR: 4.507e-02 Score 99.627 Data time: 0.2129, Forward time: 1.2563, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.8301 [2021-07-08 15:05:09,024][root][INFO] - ===> Epoch[7051](35255/5): Loss 0.0094, Sem 0.0094 LR: 4.506e-02 Score 99.663 Data time: 0.2319, Forward time: 1.2562, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.8095 [2021-07-08 15:05:22,360][root][INFO] - ===> Epoch[7052](35260/5): Loss 0.0102, Sem 0.0102 LR: 4.505e-02 Score 99.630 Data time: 0.2683, Forward time: 1.2562, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.6209 [2021-07-08 15:05:36,096][root][INFO] - ===> Epoch[7053](35265/5): Loss 0.0101, Sem 0.0101 LR: 4.505e-02 Score 99.632 Data time: 0.1996, Forward time: 1.2562, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.6985 [2021-07-08 15:05:50,300][root][INFO] - ===> Epoch[7054](35270/5): Loss 0.0096, Sem 0.0096 LR: 4.504e-02 Score 99.655 Data time: 0.2564, Forward time: 1.2562, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.7874 [2021-07-08 15:06:03,699][root][INFO] - ===> Epoch[7055](35275/5): Loss 0.0098, Sem 0.0098 LR: 4.503e-02 Score 99.647 Data time: 0.2138, Forward time: 1.2562, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.6380 [2021-07-08 15:06:17,253][root][INFO] - ===> Epoch[7056](35280/5): Loss 0.0096, Sem 0.0096 LR: 4.502e-02 Score 99.654 Data time: 0.2659, Forward time: 1.2562, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.6605 [2021-07-08 15:06:31,425][root][INFO] - ===> Epoch[7057](35285/5): Loss 0.0101, Sem 0.0101 LR: 4.501e-02 Score 99.641 Data time: 0.2543, Forward time: 1.2562, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.7706 [2021-07-08 15:06:45,349][root][INFO] - ===> Epoch[7058](35290/5): Loss 0.0095, Sem 0.0095 LR: 4.500e-02 Score 99.659 Data time: 0.2337, Forward time: 1.2562, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7369 [2021-07-08 15:06:59,243][root][INFO] - ===> Epoch[7059](35295/5): Loss 0.0097, Sem 0.0097 LR: 4.500e-02 Score 99.649 Data time: 0.1827, Forward time: 1.2562, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.7374 [2021-07-08 15:07:13,150][root][INFO] - ===> Epoch[7060](35300/5): Loss 0.0099, Sem 0.0099 LR: 4.499e-02 Score 99.645 Data time: 0.2204, Forward time: 1.2562, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.7387 [2021-07-08 15:07:26,677][root][INFO] - ===> Epoch[7061](35305/5): Loss 0.0096, Sem 0.0096 LR: 4.498e-02 Score 99.654 Data time: 0.2268, Forward time: 1.2562, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.6595 [2021-07-08 15:07:40,304][root][INFO] - ===> Epoch[7062](35310/5): Loss 0.0103, Sem 0.0103 LR: 4.497e-02 Score 99.628 Data time: 0.2819, Forward time: 1.2562, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.6809 [2021-07-08 15:07:53,825][root][INFO] - ===> Epoch[7063](35315/5): Loss 0.0096, Sem 0.0096 LR: 4.496e-02 Score 99.652 Data time: 0.2222, Forward time: 1.2562, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.6626 [2021-07-08 15:08:08,036][root][INFO] - ===> Epoch[7064](35320/5): Loss 0.0103, Sem 0.0103 LR: 4.496e-02 Score 99.634 Data time: 0.2468, Forward time: 1.2562, Backward time: 1.2419, DDP time: 0.0042, Total iter time: 2.8020 [2021-07-08 15:08:22,421][root][INFO] - ===> Epoch[7065](35325/5): Loss 0.0095, Sem 0.0095 LR: 4.495e-02 Score 99.655 Data time: 0.1721, Forward time: 1.2562, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.8363 [2021-07-08 15:08:37,144][root][INFO] - ===> Epoch[7066](35330/5): Loss 0.0100, Sem 0.0100 LR: 4.494e-02 Score 99.640 Data time: 0.2309, Forward time: 1.2562, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.8996 [2021-07-08 15:08:51,271][root][INFO] - ===> Epoch[7067](35335/5): Loss 0.0095, Sem 0.0095 LR: 4.493e-02 Score 99.657 Data time: 0.2186, Forward time: 1.2562, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.7852 [2021-07-08 15:09:05,833][root][INFO] - ===> Epoch[7068](35340/5): Loss 0.0101, Sem 0.0101 LR: 4.492e-02 Score 99.636 Data time: 0.1727, Forward time: 1.2561, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.8698 [2021-07-08 15:09:19,172][root][INFO] - ===> Epoch[7069](35345/5): Loss 0.0098, Sem 0.0098 LR: 4.491e-02 Score 99.646 Data time: 0.2375, Forward time: 1.2561, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.6217 [2021-07-08 15:09:32,812][root][INFO] - ===> Epoch[7070](35350/5): Loss 0.0097, Sem 0.0097 LR: 4.491e-02 Score 99.647 Data time: 0.2634, Forward time: 1.2561, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.6756 [2021-07-08 15:09:47,756][root][INFO] - ===> Epoch[7071](35355/5): Loss 0.0096, Sem 0.0096 LR: 4.490e-02 Score 99.652 Data time: 0.2578, Forward time: 1.2561, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.9451 [2021-07-08 15:10:01,402][root][INFO] - ===> Epoch[7072](35360/5): Loss 0.0095, Sem 0.0095 LR: 4.489e-02 Score 99.657 Data time: 0.1882, Forward time: 1.2562, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.6852 [2021-07-08 15:10:15,081][root][INFO] - ===> Epoch[7073](35365/5): Loss 0.0099, Sem 0.0099 LR: 4.488e-02 Score 99.643 Data time: 0.2254, Forward time: 1.2562, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.6893 [2021-07-08 15:10:28,832][root][INFO] - ===> Epoch[7074](35370/5): Loss 0.0098, Sem 0.0098 LR: 4.487e-02 Score 99.645 Data time: 0.1861, Forward time: 1.2561, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.7011 [2021-07-08 15:10:43,663][root][INFO] - ===> Epoch[7075](35375/5): Loss 0.0100, Sem 0.0100 LR: 4.487e-02 Score 99.640 Data time: 0.2676, Forward time: 1.2561, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.9155 [2021-07-08 15:10:58,118][root][INFO] - ===> Epoch[7076](35380/5): Loss 0.0094, Sem 0.0094 LR: 4.486e-02 Score 99.661 Data time: 0.2122, Forward time: 1.2561, Backward time: 1.2420, DDP time: 0.0042, Total iter time: 2.8297 [2021-07-08 15:11:13,142][root][INFO] - ===> Epoch[7077](35385/5): Loss 0.0102, Sem 0.0102 LR: 4.485e-02 Score 99.631 Data time: 0.2268, Forward time: 1.2562, Backward time: 1.2421, DDP time: 0.0042, Total iter time: 2.9600 [2021-07-08 15:11:27,844][root][INFO] - ===> Epoch[7078](35390/5): Loss 0.0098, Sem 0.0098 LR: 4.484e-02 Score 99.651 Data time: 0.2021, Forward time: 1.2562, Backward time: 1.2421, DDP time: 0.0042, Total iter time: 2.8967 [2021-07-08 15:11:41,489][root][INFO] - ===> Epoch[7079](35395/5): Loss 0.0099, Sem 0.0099 LR: 4.483e-02 Score 99.645 Data time: 0.2199, Forward time: 1.2562, Backward time: 1.2421, DDP time: 0.0042, Total iter time: 2.6894 [2021-07-08 15:11:55,791][root][INFO] - ===> Epoch[7080](35400/5): Loss 0.0098, Sem 0.0098 LR: 4.482e-02 Score 99.646 Data time: 0.2298, Forward time: 1.2561, Backward time: 1.2421, DDP time: 0.0042, Total iter time: 2.8166 [2021-07-08 15:11:55,793][root][INFO] - ===> Start testing [2021-07-08 15:12:37,174][root][INFO] - 68/68: Data time: 0.0016, Iter time: 0.1323 Loss 0.332 (AVG: 0.603) Score 93.077 (AVG: 90.979) mIOU 71.019 mAP 83.123 mAcc 77.081 IOU: 65.132 0.000 84.356 75.030 95.011 91.728 27.498 83.642 98.603 79.854 78.675 86.289 57.424 mAP: 84.108 4.629 92.721 85.986 98.673 98.695 52.035 98.013 99.793 91.686 91.321 93.240 89.696 mAcc: 82.121 0.000 89.345 81.894 96.255 97.596 28.994 94.639 99.341 84.234 89.403 96.562 61.671 [2021-07-08 15:12:37,175][root][INFO] - Finished test. Elapsed time: 41.3808 [2021-07-08 15:12:38,349][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 15:12:52,479][root][INFO] - ===> Epoch[7081](35405/5): Loss 0.0094, Sem 0.0094 LR: 4.482e-02 Score 99.661 Data time: 0.2342, Forward time: 1.2562, Backward time: 1.2421, DDP time: 0.0042, Total iter time: 2.7841 [2021-07-08 15:13:06,438][root][INFO] - ===> Epoch[7082](35410/5): Loss 0.0093, Sem 0.0093 LR: 4.481e-02 Score 99.665 Data time: 0.1923, Forward time: 1.2561, Backward time: 1.2421, DDP time: 0.0042, Total iter time: 2.7574 [2021-07-08 15:13:21,763][root][INFO] - ===> Epoch[7083](35415/5): Loss 0.0101, Sem 0.0101 LR: 4.480e-02 Score 99.637 Data time: 0.2364, Forward time: 1.2562, Backward time: 1.2421, DDP time: 0.0042, Total iter time: 3.0239 [2021-07-08 15:13:35,510][root][INFO] - ===> Epoch[7084](35420/5): Loss 0.0095, Sem 0.0095 LR: 4.479e-02 Score 99.658 Data time: 0.1724, Forward time: 1.2561, Backward time: 1.2421, DDP time: 0.0042, Total iter time: 2.7202 [2021-07-08 15:13:49,400][root][INFO] - ===> Epoch[7085](35425/5): Loss 0.0096, Sem 0.0096 LR: 4.478e-02 Score 99.654 Data time: 0.1950, Forward time: 1.2561, Backward time: 1.2421, DDP time: 0.0042, Total iter time: 2.7371 [2021-07-08 15:14:03,937][root][INFO] - ===> Epoch[7086](35430/5): Loss 0.0096, Sem 0.0096 LR: 4.478e-02 Score 99.655 Data time: 0.2030, Forward time: 1.2562, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.8619 [2021-07-08 15:14:18,589][root][INFO] - ===> Epoch[7087](35435/5): Loss 0.0096, Sem 0.0096 LR: 4.477e-02 Score 99.657 Data time: 0.2811, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.8860 [2021-07-08 15:14:32,479][root][INFO] - ===> Epoch[7088](35440/5): Loss 0.0097, Sem 0.0097 LR: 4.476e-02 Score 99.651 Data time: 0.2257, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7375 [2021-07-08 15:14:46,477][root][INFO] - ===> Epoch[7089](35445/5): Loss 0.0099, Sem 0.0099 LR: 4.475e-02 Score 99.642 Data time: 0.2905, Forward time: 1.2562, Backward time: 1.2421, DDP time: 0.0042, Total iter time: 2.7492 [2021-07-08 15:15:01,473][root][INFO] - ===> Epoch[7090](35450/5): Loss 0.0097, Sem 0.0097 LR: 4.474e-02 Score 99.643 Data time: 0.2691, Forward time: 1.2562, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.9622 [2021-07-08 15:15:15,788][root][INFO] - ===> Epoch[7091](35455/5): Loss 0.0092, Sem 0.0092 LR: 4.473e-02 Score 99.664 Data time: 0.2206, Forward time: 1.2562, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.8199 [2021-07-08 15:15:29,927][root][INFO] - ===> Epoch[7092](35460/5): Loss 0.0099, Sem 0.0099 LR: 4.473e-02 Score 99.647 Data time: 0.2631, Forward time: 1.2562, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7866 [2021-07-08 15:15:44,412][root][INFO] - ===> Epoch[7093](35465/5): Loss 0.0094, Sem 0.0094 LR: 4.472e-02 Score 99.659 Data time: 0.2172, Forward time: 1.2562, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.8510 [2021-07-08 15:15:58,725][root][INFO] - ===> Epoch[7094](35470/5): Loss 0.0094, Sem 0.0094 LR: 4.471e-02 Score 99.664 Data time: 0.2564, Forward time: 1.2562, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.8151 [2021-07-08 15:16:13,072][root][INFO] - ===> Epoch[7095](35475/5): Loss 0.0097, Sem 0.0097 LR: 4.470e-02 Score 99.653 Data time: 0.1835, Forward time: 1.2562, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.8306 [2021-07-08 15:16:26,579][root][INFO] - ===> Epoch[7096](35480/5): Loss 0.0101, Sem 0.0101 LR: 4.469e-02 Score 99.635 Data time: 0.2445, Forward time: 1.2562, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.6562 [2021-07-08 15:16:40,668][root][INFO] - ===> Epoch[7097](35485/5): Loss 0.0100, Sem 0.0100 LR: 4.469e-02 Score 99.635 Data time: 0.2205, Forward time: 1.2562, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7649 [2021-07-08 15:16:54,657][root][INFO] - ===> Epoch[7098](35490/5): Loss 0.0104, Sem 0.0104 LR: 4.468e-02 Score 99.626 Data time: 0.1896, Forward time: 1.2562, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7538 [2021-07-08 15:17:08,392][root][INFO] - ===> Epoch[7099](35495/5): Loss 0.0098, Sem 0.0098 LR: 4.467e-02 Score 99.648 Data time: 0.2278, Forward time: 1.2562, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.6931 [2021-07-08 15:17:23,366][root][INFO] - ===> Epoch[7100](35500/5): Loss 0.0094, Sem 0.0094 LR: 4.466e-02 Score 99.659 Data time: 0.1621, Forward time: 1.2562, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.9583 [2021-07-08 15:17:37,445][root][INFO] - ===> Epoch[7101](35505/5): Loss 0.0097, Sem 0.0097 LR: 4.465e-02 Score 99.652 Data time: 0.2190, Forward time: 1.2562, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7746 [2021-07-08 15:17:51,875][root][INFO] - ===> Epoch[7102](35510/5): Loss 0.0104, Sem 0.0104 LR: 4.464e-02 Score 99.630 Data time: 0.1886, Forward time: 1.2562, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.8361 [2021-07-08 15:18:05,910][root][INFO] - ===> Epoch[7103](35515/5): Loss 0.0097, Sem 0.0097 LR: 4.464e-02 Score 99.653 Data time: 0.2108, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.7478 [2021-07-08 15:18:19,536][root][INFO] - ===> Epoch[7104](35520/5): Loss 0.0101, Sem 0.0101 LR: 4.463e-02 Score 99.640 Data time: 0.2468, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.6695 [2021-07-08 15:18:33,947][root][INFO] - ===> Epoch[7105](35525/5): Loss 0.0096, Sem 0.0096 LR: 4.462e-02 Score 99.654 Data time: 0.2059, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.8394 [2021-07-08 15:18:47,200][root][INFO] - ===> Epoch[7106](35530/5): Loss 0.0106, Sem 0.0106 LR: 4.461e-02 Score 99.634 Data time: 0.2359, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.6069 [2021-07-08 15:19:01,799][root][INFO] - ===> Epoch[7107](35535/5): Loss 0.0098, Sem 0.0098 LR: 4.460e-02 Score 99.645 Data time: 0.1395, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.8847 [2021-07-08 15:19:15,312][root][INFO] - ===> Epoch[7108](35540/5): Loss 0.0094, Sem 0.0094 LR: 4.459e-02 Score 99.659 Data time: 0.1818, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.6577 [2021-07-08 15:19:29,225][root][INFO] - ===> Epoch[7109](35545/5): Loss 0.0101, Sem 0.0101 LR: 4.459e-02 Score 99.633 Data time: 0.2021, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.7394 [2021-07-08 15:19:43,863][root][INFO] - ===> Epoch[7110](35550/5): Loss 0.0094, Sem 0.0094 LR: 4.458e-02 Score 99.664 Data time: 0.2526, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.8852 [2021-07-08 15:19:57,870][root][INFO] - ===> Epoch[7111](35555/5): Loss 0.0093, Sem 0.0093 LR: 4.457e-02 Score 99.666 Data time: 0.2963, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.7477 [2021-07-08 15:20:10,850][root][INFO] - ===> Epoch[7112](35560/5): Loss 0.0102, Sem 0.0102 LR: 4.456e-02 Score 99.633 Data time: 0.2775, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.5487 [2021-07-08 15:20:24,565][root][INFO] - ===> Epoch[7113](35565/5): Loss 0.0095, Sem 0.0095 LR: 4.455e-02 Score 99.659 Data time: 0.2168, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.6953 [2021-07-08 15:20:39,352][root][INFO] - ===> Epoch[7114](35570/5): Loss 0.0098, Sem 0.0098 LR: 4.455e-02 Score 99.648 Data time: 0.2941, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.9026 [2021-07-08 15:20:53,311][root][INFO] - ===> Epoch[7115](35575/5): Loss 0.0099, Sem 0.0099 LR: 4.454e-02 Score 99.642 Data time: 0.2380, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.7459 [2021-07-08 15:21:07,304][root][INFO] - ===> Epoch[7116](35580/5): Loss 0.0092, Sem 0.0092 LR: 4.453e-02 Score 99.665 Data time: 0.2441, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.7552 [2021-07-08 15:21:21,328][root][INFO] - ===> Epoch[7117](35585/5): Loss 0.0095, Sem 0.0095 LR: 4.452e-02 Score 99.658 Data time: 0.2345, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7621 [2021-07-08 15:21:35,037][root][INFO] - ===> Epoch[7118](35590/5): Loss 0.0099, Sem 0.0099 LR: 4.451e-02 Score 99.645 Data time: 0.1840, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.6966 [2021-07-08 15:21:48,712][root][INFO] - ===> Epoch[7119](35595/5): Loss 0.0098, Sem 0.0098 LR: 4.450e-02 Score 99.649 Data time: 0.1748, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.6819 [2021-07-08 15:22:02,506][root][INFO] - ===> Epoch[7120](35600/5): Loss 0.0094, Sem 0.0094 LR: 4.450e-02 Score 99.664 Data time: 0.2068, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7168 [2021-07-08 15:22:02,508][root][INFO] - ===> Start testing [2021-07-08 15:22:41,179][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1295 Loss 0.346 (AVG: 0.581) Score 92.669 (AVG: 91.121) mIOU 72.163 mAP 83.935 mAcc 78.377 IOU: 64.655 0.000 84.269 75.210 95.033 92.389 37.934 84.365 98.562 82.917 79.232 86.405 57.148 mAP: 83.956 4.088 92.590 86.184 98.673 98.756 59.646 97.624 99.796 94.101 91.393 94.272 90.078 mAcc: 81.565 0.000 89.526 82.244 96.393 97.005 41.102 93.519 99.366 90.361 89.873 96.306 61.635 [2021-07-08 15:22:41,179][root][INFO] - Finished test. Elapsed time: 38.6703 [2021-07-08 15:22:42,263][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 15:22:55,209][root][INFO] - ===> Epoch[7121](35605/5): Loss 0.0099, Sem 0.0099 LR: 4.449e-02 Score 99.643 Data time: 0.2182, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.5527 [2021-07-08 15:23:08,545][root][INFO] - ===> Epoch[7122](35610/5): Loss 0.0099, Sem 0.0099 LR: 4.448e-02 Score 99.642 Data time: 0.2307, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.6194 [2021-07-08 15:23:22,410][root][INFO] - ===> Epoch[7123](35615/5): Loss 0.0097, Sem 0.0097 LR: 4.447e-02 Score 99.650 Data time: 0.2197, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7306 [2021-07-08 15:23:36,665][root][INFO] - ===> Epoch[7124](35620/5): Loss 0.0095, Sem 0.0095 LR: 4.446e-02 Score 99.657 Data time: 0.2493, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.8080 [2021-07-08 15:23:51,149][root][INFO] - ===> Epoch[7125](35625/5): Loss 0.0092, Sem 0.0092 LR: 4.446e-02 Score 99.664 Data time: 0.1901, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.8592 [2021-07-08 15:24:05,046][root][INFO] - ===> Epoch[7126](35630/5): Loss 0.0094, Sem 0.0094 LR: 4.445e-02 Score 99.660 Data time: 0.2384, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7251 [2021-07-08 15:24:19,099][root][INFO] - ===> Epoch[7127](35635/5): Loss 0.0097, Sem 0.0097 LR: 4.444e-02 Score 99.650 Data time: 0.2552, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7573 [2021-07-08 15:24:33,005][root][INFO] - ===> Epoch[7128](35640/5): Loss 0.0096, Sem 0.0096 LR: 4.443e-02 Score 99.656 Data time: 0.2320, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7328 [2021-07-08 15:24:47,474][root][INFO] - ===> Epoch[7129](35645/5): Loss 0.0095, Sem 0.0095 LR: 4.442e-02 Score 99.656 Data time: 0.2088, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.8478 [2021-07-08 15:25:01,157][root][INFO] - ===> Epoch[7130](35650/5): Loss 0.0097, Sem 0.0097 LR: 4.441e-02 Score 99.648 Data time: 0.2022, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.6814 [2021-07-08 15:25:16,146][root][INFO] - ===> Epoch[7131](35655/5): Loss 0.0097, Sem 0.0097 LR: 4.441e-02 Score 99.652 Data time: 0.2093, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.9563 [2021-07-08 15:25:29,914][root][INFO] - ===> Epoch[7132](35660/5): Loss 0.0098, Sem 0.0098 LR: 4.440e-02 Score 99.649 Data time: 0.1975, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7030 [2021-07-08 15:25:44,507][root][INFO] - ===> Epoch[7133](35665/5): Loss 0.0101, Sem 0.0101 LR: 4.439e-02 Score 99.635 Data time: 0.2464, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.8765 [2021-07-08 15:25:58,230][root][INFO] - ===> Epoch[7134](35670/5): Loss 0.0098, Sem 0.0098 LR: 4.438e-02 Score 99.646 Data time: 0.2130, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.7058 [2021-07-08 15:26:11,842][root][INFO] - ===> Epoch[7135](35675/5): Loss 0.0097, Sem 0.0097 LR: 4.437e-02 Score 99.649 Data time: 0.2313, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.6827 [2021-07-08 15:26:25,704][root][INFO] - ===> Epoch[7136](35680/5): Loss 0.0093, Sem 0.0093 LR: 4.437e-02 Score 99.662 Data time: 0.1833, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7275 [2021-07-08 15:26:39,860][root][INFO] - ===> Epoch[7137](35685/5): Loss 0.0096, Sem 0.0096 LR: 4.436e-02 Score 99.653 Data time: 0.2356, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7841 [2021-07-08 15:26:53,302][root][INFO] - ===> Epoch[7138](35690/5): Loss 0.0097, Sem 0.0097 LR: 4.435e-02 Score 99.652 Data time: 0.1928, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.6380 [2021-07-08 15:27:07,415][root][INFO] - ===> Epoch[7139](35695/5): Loss 0.0094, Sem 0.0094 LR: 4.434e-02 Score 99.661 Data time: 0.2343, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7846 [2021-07-08 15:27:21,417][root][INFO] - ===> Epoch[7140](35700/5): Loss 0.0095, Sem 0.0095 LR: 4.433e-02 Score 99.656 Data time: 0.1754, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7519 [2021-07-08 15:27:34,706][root][INFO] - ===> Epoch[7141](35705/5): Loss 0.0098, Sem 0.0098 LR: 4.432e-02 Score 99.649 Data time: 0.2138, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.6123 [2021-07-08 15:27:48,938][root][INFO] - ===> Epoch[7142](35710/5): Loss 0.0098, Sem 0.0098 LR: 4.432e-02 Score 99.650 Data time: 0.2331, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7995 [2021-07-08 15:28:03,458][root][INFO] - ===> Epoch[7143](35715/5): Loss 0.0091, Sem 0.0091 LR: 4.431e-02 Score 99.671 Data time: 0.2484, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.8624 [2021-07-08 15:28:17,820][root][INFO] - ===> Epoch[7144](35720/5): Loss 0.0095, Sem 0.0095 LR: 4.430e-02 Score 99.658 Data time: 0.2572, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.8226 [2021-07-08 15:28:31,558][root][INFO] - ===> Epoch[7145](35725/5): Loss 0.0096, Sem 0.0096 LR: 4.429e-02 Score 99.652 Data time: 0.1929, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.7052 [2021-07-08 15:28:45,691][root][INFO] - ===> Epoch[7146](35730/5): Loss 0.0098, Sem 0.0098 LR: 4.428e-02 Score 99.648 Data time: 0.2541, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.7661 [2021-07-08 15:28:59,422][root][INFO] - ===> Epoch[7147](35735/5): Loss 0.0098, Sem 0.0098 LR: 4.427e-02 Score 99.646 Data time: 0.1838, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.7023 [2021-07-08 15:29:13,106][root][INFO] - ===> Epoch[7148](35740/5): Loss 0.0094, Sem 0.0094 LR: 4.427e-02 Score 99.660 Data time: 0.2364, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.6910 [2021-07-08 15:29:26,817][root][INFO] - ===> Epoch[7149](35745/5): Loss 0.0097, Sem 0.0097 LR: 4.426e-02 Score 99.647 Data time: 0.1944, Forward time: 1.2561, Backward time: 1.2422, DDP time: 0.0042, Total iter time: 2.7063 [2021-07-08 15:29:41,827][root][INFO] - ===> Epoch[7150](35750/5): Loss 0.0091, Sem 0.0091 LR: 4.425e-02 Score 99.667 Data time: 0.2805, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.9418 [2021-07-08 15:29:55,590][root][INFO] - ===> Epoch[7151](35755/5): Loss 0.0094, Sem 0.0094 LR: 4.424e-02 Score 99.664 Data time: 0.1817, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.7144 [2021-07-08 15:30:10,585][root][INFO] - ===> Epoch[7152](35760/5): Loss 0.0093, Sem 0.0093 LR: 4.423e-02 Score 99.664 Data time: 0.2885, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.9464 [2021-07-08 15:30:25,359][root][INFO] - ===> Epoch[7153](35765/5): Loss 0.0088, Sem 0.0088 LR: 4.423e-02 Score 99.683 Data time: 0.2651, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.9060 [2021-07-08 15:30:39,298][root][INFO] - ===> Epoch[7154](35770/5): Loss 0.0095, Sem 0.0095 LR: 4.422e-02 Score 99.658 Data time: 0.1925, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.7462 [2021-07-08 15:30:53,638][root][INFO] - ===> Epoch[7155](35775/5): Loss 0.0099, Sem 0.0099 LR: 4.421e-02 Score 99.642 Data time: 0.1996, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.8177 [2021-07-08 15:31:08,374][root][INFO] - ===> Epoch[7156](35780/5): Loss 0.0094, Sem 0.0094 LR: 4.420e-02 Score 99.663 Data time: 0.2711, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.8956 [2021-07-08 15:31:22,008][root][INFO] - ===> Epoch[7157](35785/5): Loss 0.0094, Sem 0.0094 LR: 4.419e-02 Score 99.660 Data time: 0.2104, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.6872 [2021-07-08 15:31:36,031][root][INFO] - ===> Epoch[7158](35790/5): Loss 0.0095, Sem 0.0095 LR: 4.418e-02 Score 99.658 Data time: 0.1698, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.7636 [2021-07-08 15:31:49,452][root][INFO] - ===> Epoch[7159](35795/5): Loss 0.0095, Sem 0.0095 LR: 4.418e-02 Score 99.652 Data time: 0.2241, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.6371 [2021-07-08 15:32:03,911][root][INFO] - ===> Epoch[7160](35800/5): Loss 0.0097, Sem 0.0097 LR: 4.417e-02 Score 99.654 Data time: 0.2064, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.8471 [2021-07-08 15:32:03,913][root][INFO] - ===> Start testing [2021-07-08 15:32:42,799][root][INFO] - 68/68: Data time: 0.0669, Iter time: 0.1304 Loss 0.397 (AVG: 0.577) Score 92.251 (AVG: 91.181) mIOU 71.891 mAP 84.136 mAcc 78.180 IOU: 64.724 0.000 85.121 75.555 95.082 91.594 35.213 83.425 98.561 81.628 79.127 86.515 58.038 mAP: 84.000 5.516 93.476 86.845 98.839 98.654 59.623 97.919 99.800 93.203 91.661 93.877 90.356 mAcc: 78.858 0.000 90.406 83.764 96.411 97.274 37.470 94.748 99.327 88.329 90.720 96.710 62.325 [2021-07-08 15:32:42,799][root][INFO] - Finished test. Elapsed time: 38.8855 [2021-07-08 15:32:43,877][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 15:32:57,422][root][INFO] - ===> Epoch[7161](35805/5): Loss 0.0095, Sem 0.0095 LR: 4.416e-02 Score 99.657 Data time: 0.1848, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.6730 [2021-07-08 15:33:11,268][root][INFO] - ===> Epoch[7162](35810/5): Loss 0.0096, Sem 0.0096 LR: 4.415e-02 Score 99.650 Data time: 0.1980, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.7220 [2021-07-08 15:33:25,754][root][INFO] - ===> Epoch[7163](35815/5): Loss 0.0092, Sem 0.0092 LR: 4.414e-02 Score 99.665 Data time: 0.2058, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.8505 [2021-07-08 15:33:39,907][root][INFO] - ===> Epoch[7164](35820/5): Loss 0.0096, Sem 0.0096 LR: 4.414e-02 Score 99.656 Data time: 0.2188, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.7894 [2021-07-08 15:33:53,535][root][INFO] - ===> Epoch[7165](35825/5): Loss 0.0097, Sem 0.0097 LR: 4.413e-02 Score 99.647 Data time: 0.2121, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.6863 [2021-07-08 15:34:08,467][root][INFO] - ===> Epoch[7166](35830/5): Loss 0.0094, Sem 0.0094 LR: 4.412e-02 Score 99.660 Data time: 0.2299, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.9394 [2021-07-08 15:34:22,114][root][INFO] - ===> Epoch[7167](35835/5): Loss 0.0091, Sem 0.0091 LR: 4.411e-02 Score 99.671 Data time: 0.2200, Forward time: 1.2561, Backward time: 1.2423, DDP time: 0.0042, Total iter time: 2.6716 [2021-07-08 15:34:37,141][root][INFO] - ===> Epoch[7168](35840/5): Loss 0.0095, Sem 0.0095 LR: 4.410e-02 Score 99.658 Data time: 0.1645, Forward time: 1.2561, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.9719 [2021-07-08 15:34:52,411][root][INFO] - ===> Epoch[7169](35845/5): Loss 0.0091, Sem 0.0091 LR: 4.409e-02 Score 99.673 Data time: 0.1846, Forward time: 1.2561, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.9926 [2021-07-08 15:35:06,442][root][INFO] - ===> Epoch[7170](35850/5): Loss 0.0094, Sem 0.0094 LR: 4.409e-02 Score 99.662 Data time: 0.2475, Forward time: 1.2561, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.7677 [2021-07-08 15:35:20,942][root][INFO] - ===> Epoch[7171](35855/5): Loss 0.0090, Sem 0.0090 LR: 4.408e-02 Score 99.674 Data time: 0.2371, Forward time: 1.2561, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.8560 [2021-07-08 15:35:35,189][root][INFO] - ===> Epoch[7172](35860/5): Loss 0.0098, Sem 0.0098 LR: 4.407e-02 Score 99.647 Data time: 0.2082, Forward time: 1.2561, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.8034 [2021-07-08 15:35:49,365][root][INFO] - ===> Epoch[7173](35865/5): Loss 0.0090, Sem 0.0090 LR: 4.406e-02 Score 99.675 Data time: 0.2337, Forward time: 1.2561, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.7929 [2021-07-08 15:36:02,923][root][INFO] - ===> Epoch[7174](35870/5): Loss 0.0099, Sem 0.0099 LR: 4.405e-02 Score 99.642 Data time: 0.1921, Forward time: 1.2561, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.6681 [2021-07-08 15:36:17,020][root][INFO] - ===> Epoch[7175](35875/5): Loss 0.0091, Sem 0.0091 LR: 4.404e-02 Score 99.677 Data time: 0.2266, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7779 [2021-07-08 15:36:30,906][root][INFO] - ===> Epoch[7176](35880/5): Loss 0.0096, Sem 0.0096 LR: 4.404e-02 Score 99.653 Data time: 0.1884, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7330 [2021-07-08 15:36:44,901][root][INFO] - ===> Epoch[7177](35885/5): Loss 0.0094, Sem 0.0094 LR: 4.403e-02 Score 99.663 Data time: 0.2909, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.7442 [2021-07-08 15:36:58,223][root][INFO] - ===> Epoch[7178](35890/5): Loss 0.0094, Sem 0.0094 LR: 4.402e-02 Score 99.661 Data time: 0.1884, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.6217 [2021-07-08 15:37:11,680][root][INFO] - ===> Epoch[7179](35895/5): Loss 0.0089, Sem 0.0089 LR: 4.401e-02 Score 99.676 Data time: 0.2187, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.6408 [2021-07-08 15:37:24,952][root][INFO] - ===> Epoch[7180](35900/5): Loss 0.0126, Sem 0.0126 LR: 4.400e-02 Score 99.601 Data time: 0.2312, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.6132 [2021-07-08 15:37:39,186][root][INFO] - ===> Epoch[7181](35905/5): Loss 0.0124, Sem 0.0124 LR: 4.400e-02 Score 99.561 Data time: 0.2053, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.8029 [2021-07-08 15:37:53,020][root][INFO] - ===> Epoch[7182](35910/5): Loss 0.0122, Sem 0.0122 LR: 4.399e-02 Score 99.570 Data time: 0.2003, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7145 [2021-07-08 15:38:06,008][root][INFO] - ===> Epoch[7183](35915/5): Loss 0.0126, Sem 0.0126 LR: 4.398e-02 Score 99.558 Data time: 0.1906, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.5493 [2021-07-08 15:38:20,109][root][INFO] - ===> Epoch[7184](35920/5): Loss 0.0124, Sem 0.0124 LR: 4.397e-02 Score 99.554 Data time: 0.2460, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.7584 [2021-07-08 15:38:34,203][root][INFO] - ===> Epoch[7185](35925/5): Loss 0.0119, Sem 0.0119 LR: 4.396e-02 Score 99.576 Data time: 0.1958, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.7723 [2021-07-08 15:38:48,537][root][INFO] - ===> Epoch[7186](35930/5): Loss 0.0120, Sem 0.0120 LR: 4.395e-02 Score 99.563 Data time: 0.2911, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.8239 [2021-07-08 15:39:03,017][root][INFO] - ===> Epoch[7187](35935/5): Loss 0.0115, Sem 0.0115 LR: 4.395e-02 Score 99.591 Data time: 0.2010, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.8393 [2021-07-08 15:39:17,012][root][INFO] - ===> Epoch[7188](35940/5): Loss 0.0104, Sem 0.0104 LR: 4.394e-02 Score 99.621 Data time: 0.2472, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.7573 [2021-07-08 15:39:31,662][root][INFO] - ===> Epoch[7189](35945/5): Loss 0.0110, Sem 0.0110 LR: 4.393e-02 Score 99.608 Data time: 0.2432, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.8835 [2021-07-08 15:39:46,365][root][INFO] - ===> Epoch[7190](35950/5): Loss 0.0105, Sem 0.0105 LR: 4.392e-02 Score 99.625 Data time: 0.2448, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.8859 [2021-07-08 15:40:00,054][root][INFO] - ===> Epoch[7191](35955/5): Loss 0.0104, Sem 0.0104 LR: 4.391e-02 Score 99.625 Data time: 0.2255, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.7046 [2021-07-08 15:40:14,105][root][INFO] - ===> Epoch[7192](35960/5): Loss 0.0102, Sem 0.0102 LR: 4.391e-02 Score 99.630 Data time: 0.2115, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.7626 [2021-07-08 15:40:28,324][root][INFO] - ===> Epoch[7193](35965/5): Loss 0.0099, Sem 0.0099 LR: 4.390e-02 Score 99.645 Data time: 0.1876, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.8025 [2021-07-08 15:40:42,134][root][INFO] - ===> Epoch[7194](35970/5): Loss 0.0104, Sem 0.0104 LR: 4.389e-02 Score 99.627 Data time: 0.2506, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.7161 [2021-07-08 15:40:56,330][root][INFO] - ===> Epoch[7195](35975/5): Loss 0.0106, Sem 0.0106 LR: 4.388e-02 Score 99.617 Data time: 0.2426, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.8062 [2021-07-08 15:41:10,558][root][INFO] - ===> Epoch[7196](35980/5): Loss 0.0103, Sem 0.0103 LR: 4.387e-02 Score 99.631 Data time: 0.2143, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.8062 [2021-07-08 15:41:25,520][root][INFO] - ===> Epoch[7197](35985/5): Loss 0.0098, Sem 0.0098 LR: 4.386e-02 Score 99.644 Data time: 0.1693, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.9487 [2021-07-08 15:41:38,332][root][INFO] - ===> Epoch[7198](35990/5): Loss 0.0107, Sem 0.0107 LR: 4.386e-02 Score 99.615 Data time: 0.2232, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.5202 [2021-07-08 15:41:52,750][root][INFO] - ===> Epoch[7199](35995/5): Loss 0.0103, Sem 0.0103 LR: 4.385e-02 Score 99.645 Data time: 0.2386, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.8384 [2021-07-08 15:42:07,013][root][INFO] - ===> Epoch[7200](36000/5): Loss 0.0102, Sem 0.0102 LR: 4.384e-02 Score 99.635 Data time: 0.2390, Forward time: 1.2561, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.8061 [2021-07-08 15:42:07,014][root][INFO] - ===> Start testing [2021-07-08 15:42:51,349][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1249 Loss 0.320 (AVG: 0.582) Score 93.581 (AVG: 90.813) mIOU 70.928 mAP 84.125 mAcc 77.348 IOU: 64.478 0.000 84.951 74.280 94.766 92.112 27.578 82.409 98.598 79.155 78.979 85.975 58.781 mAP: 84.394 8.123 94.284 86.273 98.694 98.831 57.170 97.529 99.770 93.191 91.397 93.343 90.620 mAcc: 81.071 0.000 90.695 81.585 96.044 97.711 30.333 93.850 99.445 84.442 89.926 96.073 64.343 [2021-07-08 15:42:51,349][root][INFO] - Finished test. Elapsed time: 44.3342 [2021-07-08 15:42:52,448][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 15:43:04,469][root][INFO] - ===> Epoch[7201](36005/5): Loss 0.0107, Sem 0.0107 LR: 4.383e-02 Score 99.621 Data time: 0.1494, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.3714 [2021-07-08 15:43:18,616][root][INFO] - ===> Epoch[7202](36010/5): Loss 0.0100, Sem 0.0100 LR: 4.382e-02 Score 99.640 Data time: 0.2254, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.7877 [2021-07-08 15:43:33,133][root][INFO] - ===> Epoch[7203](36015/5): Loss 0.0098, Sem 0.0098 LR: 4.381e-02 Score 99.648 Data time: 0.2267, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.8636 [2021-07-08 15:43:46,181][root][INFO] - ===> Epoch[7204](36020/5): Loss 0.0101, Sem 0.0101 LR: 4.381e-02 Score 99.640 Data time: 0.2504, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.5668 [2021-07-08 15:43:59,752][root][INFO] - ===> Epoch[7205](36025/5): Loss 0.0101, Sem 0.0101 LR: 4.380e-02 Score 99.641 Data time: 0.2498, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.6710 [2021-07-08 15:44:14,255][root][INFO] - ===> Epoch[7206](36030/5): Loss 0.0101, Sem 0.0101 LR: 4.379e-02 Score 99.633 Data time: 0.1914, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.8598 [2021-07-08 15:44:29,258][root][INFO] - ===> Epoch[7207](36035/5): Loss 0.0096, Sem 0.0096 LR: 4.378e-02 Score 99.658 Data time: 0.2272, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.9505 [2021-07-08 15:44:44,592][root][INFO] - ===> Epoch[7208](36040/5): Loss 0.0109, Sem 0.0109 LR: 4.377e-02 Score 99.631 Data time: 0.2113, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 3.0167 [2021-07-08 15:44:58,279][root][INFO] - ===> Epoch[7209](36045/5): Loss 0.0108, Sem 0.0108 LR: 4.377e-02 Score 99.615 Data time: 0.2039, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6978 [2021-07-08 15:45:11,729][root][INFO] - ===> Epoch[7210](36050/5): Loss 0.0130, Sem 0.0130 LR: 4.376e-02 Score 99.581 Data time: 0.2412, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6386 [2021-07-08 15:45:26,408][root][INFO] - ===> Epoch[7211](36055/5): Loss 0.0119, Sem 0.0119 LR: 4.375e-02 Score 99.575 Data time: 0.2529, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.8838 [2021-07-08 15:45:40,315][root][INFO] - ===> Epoch[7212](36060/5): Loss 0.0122, Sem 0.0122 LR: 4.374e-02 Score 99.574 Data time: 0.2403, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7248 [2021-07-08 15:45:54,897][root][INFO] - ===> Epoch[7213](36065/5): Loss 0.0117, Sem 0.0117 LR: 4.373e-02 Score 99.590 Data time: 0.2136, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.8707 [2021-07-08 15:46:08,620][root][INFO] - ===> Epoch[7214](36070/5): Loss 0.0127, Sem 0.0127 LR: 4.372e-02 Score 99.555 Data time: 0.2152, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6941 [2021-07-08 15:46:22,787][root][INFO] - ===> Epoch[7215](36075/5): Loss 0.0115, Sem 0.0115 LR: 4.372e-02 Score 99.587 Data time: 0.2100, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7856 [2021-07-08 15:46:35,969][root][INFO] - ===> Epoch[7216](36080/5): Loss 0.0136, Sem 0.0136 LR: 4.371e-02 Score 99.528 Data time: 0.1894, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.5967 [2021-07-08 15:46:49,308][root][INFO] - ===> Epoch[7217](36085/5): Loss 0.0134, Sem 0.0134 LR: 4.370e-02 Score 99.537 Data time: 0.2647, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6284 [2021-07-08 15:47:03,562][root][INFO] - ===> Epoch[7218](36090/5): Loss 0.0125, Sem 0.0125 LR: 4.369e-02 Score 99.554 Data time: 0.1742, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.8089 [2021-07-08 15:47:17,275][root][INFO] - ===> Epoch[7219](36095/5): Loss 0.0135, Sem 0.0135 LR: 4.368e-02 Score 99.516 Data time: 0.2542, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7012 [2021-07-08 15:47:31,044][root][INFO] - ===> Epoch[7220](36100/5): Loss 0.0156, Sem 0.0156 LR: 4.367e-02 Score 99.471 Data time: 0.2078, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7060 [2021-07-08 15:47:44,890][root][INFO] - ===> Epoch[7221](36105/5): Loss 0.0175, Sem 0.0175 LR: 4.367e-02 Score 99.426 Data time: 0.1904, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7268 [2021-07-08 15:47:59,951][root][INFO] - ===> Epoch[7222](36110/5): Loss 0.0177, Sem 0.0177 LR: 4.366e-02 Score 99.400 Data time: 0.2290, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.9710 [2021-07-08 15:48:14,317][root][INFO] - ===> Epoch[7223](36115/5): Loss 0.0163, Sem 0.0163 LR: 4.365e-02 Score 99.434 Data time: 0.3164, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.8258 [2021-07-08 15:48:29,743][root][INFO] - ===> Epoch[7224](36120/5): Loss 0.0155, Sem 0.0155 LR: 4.364e-02 Score 99.461 Data time: 0.2673, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 3.0385 [2021-07-08 15:48:43,252][root][INFO] - ===> Epoch[7225](36125/5): Loss 0.0147, Sem 0.0147 LR: 4.363e-02 Score 99.474 Data time: 0.2816, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6532 [2021-07-08 15:48:58,240][root][INFO] - ===> Epoch[7226](36130/5): Loss 0.0135, Sem 0.0135 LR: 4.363e-02 Score 99.515 Data time: 0.2403, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.9444 [2021-07-08 15:49:11,954][root][INFO] - ===> Epoch[7227](36135/5): Loss 0.0138, Sem 0.0138 LR: 4.362e-02 Score 99.516 Data time: 0.2723, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6955 [2021-07-08 15:49:26,907][root][INFO] - ===> Epoch[7228](36140/5): Loss 0.0137, Sem 0.0137 LR: 4.361e-02 Score 99.514 Data time: 0.2036, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.9417 [2021-07-08 15:49:40,432][root][INFO] - ===> Epoch[7229](36145/5): Loss 0.0132, Sem 0.0132 LR: 4.360e-02 Score 99.525 Data time: 0.2377, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6565 [2021-07-08 15:49:53,850][root][INFO] - ===> Epoch[7230](36150/5): Loss 0.0119, Sem 0.0119 LR: 4.359e-02 Score 99.574 Data time: 0.1760, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6267 [2021-07-08 15:50:07,980][root][INFO] - ===> Epoch[7231](36155/5): Loss 0.0123, Sem 0.0123 LR: 4.358e-02 Score 99.560 Data time: 0.2108, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7738 [2021-07-08 15:50:22,794][root][INFO] - ===> Epoch[7232](36160/5): Loss 0.0112, Sem 0.0112 LR: 4.358e-02 Score 99.599 Data time: 0.2041, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.9090 [2021-07-08 15:50:36,839][root][INFO] - ===> Epoch[7233](36165/5): Loss 0.0113, Sem 0.0113 LR: 4.357e-02 Score 99.597 Data time: 0.2277, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7535 [2021-07-08 15:50:50,626][root][INFO] - ===> Epoch[7234](36170/5): Loss 0.0117, Sem 0.0117 LR: 4.356e-02 Score 99.582 Data time: 0.2440, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7192 [2021-07-08 15:51:03,810][root][INFO] - ===> Epoch[7235](36175/5): Loss 0.0130, Sem 0.0130 LR: 4.355e-02 Score 99.548 Data time: 0.2339, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.5906 [2021-07-08 15:51:17,376][root][INFO] - ===> Epoch[7236](36180/5): Loss 0.0409, Sem 0.0409 LR: 4.354e-02 Score 98.909 Data time: 0.2152, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6748 [2021-07-08 15:51:31,059][root][INFO] - ===> Epoch[7237](36185/5): Loss 0.1067, Sem 0.1067 LR: 4.354e-02 Score 97.050 Data time: 0.2453, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6896 [2021-07-08 15:51:46,114][root][INFO] - ===> Epoch[7238](36190/5): Loss 0.1172, Sem 0.1172 LR: 4.353e-02 Score 96.738 Data time: 0.1937, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.9703 [2021-07-08 15:51:59,879][root][INFO] - ===> Epoch[7239](36195/5): Loss 0.1009, Sem 0.1009 LR: 4.352e-02 Score 96.887 Data time: 0.2304, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7146 [2021-07-08 15:52:14,651][root][INFO] - ===> Epoch[7240](36200/5): Loss 0.1038, Sem 0.1038 LR: 4.351e-02 Score 96.873 Data time: 0.2299, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.9139 [2021-07-08 15:52:14,652][root][INFO] - ===> Start testing [2021-07-08 15:52:53,187][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1274 Loss 0.647 (AVG: 0.667) Score 85.845 (AVG: 86.592) mIOU 60.376 mAP 76.323 mAcc 67.742 IOU: 53.667 0.015 54.140 66.933 91.692 79.858 15.916 62.417 97.439 63.666 70.004 81.173 47.963 mAP: 75.836 9.421 64.635 84.634 98.154 94.520 39.331 83.080 99.698 76.353 89.849 91.980 84.712 mAcc: 73.877 0.182 56.874 76.051 93.063 89.855 18.748 70.212 98.422 65.726 92.074 95.505 50.061 [2021-07-08 15:52:53,187][root][INFO] - Finished test. Elapsed time: 38.5343 [2021-07-08 15:52:54,249][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 15:53:07,251][root][INFO] - ===> Epoch[7241](36205/5): Loss 0.0898, Sem 0.0898 LR: 4.350e-02 Score 97.259 Data time: 0.1930, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.5574 [2021-07-08 15:53:21,055][root][INFO] - ===> Epoch[7242](36210/5): Loss 0.0932, Sem 0.0932 LR: 4.349e-02 Score 97.105 Data time: 0.2014, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7168 [2021-07-08 15:53:34,902][root][INFO] - ===> Epoch[7243](36215/5): Loss 0.0841, Sem 0.0841 LR: 4.349e-02 Score 97.391 Data time: 0.2958, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7258 [2021-07-08 15:53:49,047][root][INFO] - ===> Epoch[7244](36220/5): Loss 0.0773, Sem 0.0773 LR: 4.348e-02 Score 97.558 Data time: 0.1482, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7886 [2021-07-08 15:54:04,354][root][INFO] - ===> Epoch[7245](36225/5): Loss 0.0677, Sem 0.0677 LR: 4.347e-02 Score 97.777 Data time: 0.2595, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 3.0223 [2021-07-08 15:54:17,700][root][INFO] - ===> Epoch[7246](36230/5): Loss 0.0636, Sem 0.0636 LR: 4.346e-02 Score 97.937 Data time: 0.2080, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6245 [2021-07-08 15:54:33,144][root][INFO] - ===> Epoch[7247](36235/5): Loss 0.0566, Sem 0.0566 LR: 4.345e-02 Score 98.164 Data time: 0.2462, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 3.0429 [2021-07-08 15:54:47,112][root][INFO] - ===> Epoch[7248](36240/5): Loss 0.0574, Sem 0.0574 LR: 4.344e-02 Score 98.125 Data time: 0.2000, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7515 [2021-07-08 15:55:01,209][root][INFO] - ===> Epoch[7249](36245/5): Loss 0.0541, Sem 0.0541 LR: 4.344e-02 Score 98.226 Data time: 0.2365, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7739 [2021-07-08 15:55:14,186][root][INFO] - ===> Epoch[7250](36250/5): Loss 0.0559, Sem 0.0559 LR: 4.343e-02 Score 98.167 Data time: 0.2109, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.5610 [2021-07-08 15:55:28,461][root][INFO] - ===> Epoch[7251](36255/5): Loss 0.0519, Sem 0.0519 LR: 4.342e-02 Score 98.285 Data time: 0.2166, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.8085 [2021-07-08 15:55:41,881][root][INFO] - ===> Epoch[7252](36260/5): Loss 0.0471, Sem 0.0471 LR: 4.341e-02 Score 98.421 Data time: 0.2200, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6404 [2021-07-08 15:55:56,544][root][INFO] - ===> Epoch[7253](36265/5): Loss 0.0418, Sem 0.0418 LR: 4.340e-02 Score 98.620 Data time: 0.2733, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.8841 [2021-07-08 15:56:09,870][root][INFO] - ===> Epoch[7254](36270/5): Loss 0.0423, Sem 0.0423 LR: 4.340e-02 Score 98.586 Data time: 0.2487, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6191 [2021-07-08 15:56:23,440][root][INFO] - ===> Epoch[7255](36275/5): Loss 0.0377, Sem 0.0377 LR: 4.339e-02 Score 98.735 Data time: 0.2104, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6803 [2021-07-08 15:56:37,503][root][INFO] - ===> Epoch[7256](36280/5): Loss 0.0373, Sem 0.0373 LR: 4.338e-02 Score 98.769 Data time: 0.2294, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7662 [2021-07-08 15:56:51,261][root][INFO] - ===> Epoch[7257](36285/5): Loss 0.0331, Sem 0.0331 LR: 4.337e-02 Score 98.849 Data time: 0.2492, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7030 [2021-07-08 15:57:06,052][root][INFO] - ===> Epoch[7258](36290/5): Loss 0.0371, Sem 0.0371 LR: 4.336e-02 Score 98.830 Data time: 0.1786, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.9139 [2021-07-08 15:57:20,069][root][INFO] - ===> Epoch[7259](36295/5): Loss 0.0356, Sem 0.0356 LR: 4.335e-02 Score 98.873 Data time: 0.2958, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7491 [2021-07-08 15:57:34,556][root][INFO] - ===> Epoch[7260](36300/5): Loss 0.0338, Sem 0.0338 LR: 4.335e-02 Score 98.895 Data time: 0.3542, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.8562 [2021-07-08 15:57:48,428][root][INFO] - ===> Epoch[7261](36305/5): Loss 0.0383, Sem 0.0383 LR: 4.334e-02 Score 98.770 Data time: 0.2149, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7263 [2021-07-08 15:58:03,394][root][INFO] - ===> Epoch[7262](36310/5): Loss 0.0351, Sem 0.0351 LR: 4.333e-02 Score 98.887 Data time: 0.2487, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.9462 [2021-07-08 15:58:17,084][root][INFO] - ===> Epoch[7263](36315/5): Loss 0.0305, Sem 0.0305 LR: 4.332e-02 Score 98.968 Data time: 0.3488, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.6842 [2021-07-08 15:58:30,801][root][INFO] - ===> Epoch[7264](36320/5): Loss 0.0317, Sem 0.0317 LR: 4.331e-02 Score 98.915 Data time: 0.3333, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.6852 [2021-07-08 15:58:44,286][root][INFO] - ===> Epoch[7265](36325/5): Loss 0.0275, Sem 0.0275 LR: 4.330e-02 Score 99.042 Data time: 0.2991, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.6524 [2021-07-08 15:58:58,220][root][INFO] - ===> Epoch[7266](36330/5): Loss 0.0353, Sem 0.0353 LR: 4.330e-02 Score 98.921 Data time: 0.2112, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.7353 [2021-07-08 15:59:11,646][root][INFO] - ===> Epoch[7267](36335/5): Loss 0.0378, Sem 0.0378 LR: 4.329e-02 Score 98.816 Data time: 0.2040, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.6448 [2021-07-08 15:59:25,409][root][INFO] - ===> Epoch[7268](36340/5): Loss 0.0296, Sem 0.0296 LR: 4.328e-02 Score 98.997 Data time: 0.1708, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.7140 [2021-07-08 15:59:40,256][root][INFO] - ===> Epoch[7269](36345/5): Loss 0.0330, Sem 0.0330 LR: 4.327e-02 Score 98.918 Data time: 0.2573, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.9124 [2021-07-08 15:59:54,620][root][INFO] - ===> Epoch[7270](36350/5): Loss 0.0340, Sem 0.0340 LR: 4.326e-02 Score 98.876 Data time: 0.1838, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.8177 [2021-07-08 16:00:08,211][root][INFO] - ===> Epoch[7271](36355/5): Loss 0.0316, Sem 0.0316 LR: 4.326e-02 Score 98.937 Data time: 0.2371, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.6698 [2021-07-08 16:00:22,980][root][INFO] - ===> Epoch[7272](36360/5): Loss 0.0275, Sem 0.0275 LR: 4.325e-02 Score 99.063 Data time: 0.2367, Forward time: 1.2560, Backward time: 1.2424, DDP time: 0.0042, Total iter time: 2.9072 [2021-07-08 16:00:36,923][root][INFO] - ===> Epoch[7273](36365/5): Loss 0.0263, Sem 0.0263 LR: 4.324e-02 Score 99.089 Data time: 0.2056, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7463 [2021-07-08 16:00:51,637][root][INFO] - ===> Epoch[7274](36370/5): Loss 0.0268, Sem 0.0268 LR: 4.323e-02 Score 99.097 Data time: 0.2404, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.8918 [2021-07-08 16:01:05,472][root][INFO] - ===> Epoch[7275](36375/5): Loss 0.0236, Sem 0.0236 LR: 4.322e-02 Score 99.168 Data time: 0.2426, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7109 [2021-07-08 16:01:18,709][root][INFO] - ===> Epoch[7276](36380/5): Loss 0.0237, Sem 0.0237 LR: 4.321e-02 Score 99.171 Data time: 0.2015, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6084 [2021-07-08 16:01:34,129][root][INFO] - ===> Epoch[7277](36385/5): Loss 0.0220, Sem 0.0220 LR: 4.321e-02 Score 99.242 Data time: 0.2369, Forward time: 1.2560, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 3.0417 [2021-07-08 16:01:47,521][root][INFO] - ===> Epoch[7278](36390/5): Loss 0.0202, Sem 0.0202 LR: 4.320e-02 Score 99.291 Data time: 0.2001, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6417 [2021-07-08 16:02:01,893][root][INFO] - ===> Epoch[7279](36395/5): Loss 0.0202, Sem 0.0202 LR: 4.319e-02 Score 99.281 Data time: 0.2227, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.8256 [2021-07-08 16:02:15,053][root][INFO] - ===> Epoch[7280](36400/5): Loss 0.0194, Sem 0.0194 LR: 4.318e-02 Score 99.318 Data time: 0.2036, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.5888 [2021-07-08 16:02:15,055][root][INFO] - ===> Start testing [2021-07-08 16:02:58,711][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1280 Loss 0.283 (AVG: 0.505) Score 92.190 (AVG: 90.573) mIOU 70.202 mAP 83.177 mAcc 76.137 IOU: 63.383 0.000 85.651 73.751 95.367 92.048 23.774 81.410 98.503 79.474 79.485 85.501 54.278 mAP: 83.911 3.544 94.805 87.432 99.122 99.043 51.774 96.525 99.829 87.998 92.862 94.420 90.033 mAcc: 81.880 0.000 91.170 81.057 96.833 97.061 25.722 92.165 99.245 82.893 88.469 96.421 56.870 [2021-07-08 16:02:58,711][root][INFO] - Finished test. Elapsed time: 43.6558 [2021-07-08 16:02:59,811][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 16:03:12,744][root][INFO] - ===> Epoch[7281](36405/5): Loss 0.0204, Sem 0.0204 LR: 4.317e-02 Score 99.292 Data time: 0.1438, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.5518 [2021-07-08 16:03:27,085][root][INFO] - ===> Epoch[7282](36410/5): Loss 0.0198, Sem 0.0198 LR: 4.316e-02 Score 99.295 Data time: 0.2684, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.8167 [2021-07-08 16:03:41,368][root][INFO] - ===> Epoch[7283](36415/5): Loss 0.0206, Sem 0.0206 LR: 4.316e-02 Score 99.296 Data time: 0.2447, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.8006 [2021-07-08 16:03:55,585][root][INFO] - ===> Epoch[7284](36420/5): Loss 0.0198, Sem 0.0198 LR: 4.315e-02 Score 99.314 Data time: 0.2813, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7926 [2021-07-08 16:04:09,546][root][INFO] - ===> Epoch[7285](36425/5): Loss 0.0184, Sem 0.0184 LR: 4.314e-02 Score 99.343 Data time: 0.2036, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7378 [2021-07-08 16:04:23,503][root][INFO] - ===> Epoch[7286](36430/5): Loss 0.0189, Sem 0.0189 LR: 4.313e-02 Score 99.333 Data time: 0.2310, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7542 [2021-07-08 16:04:37,330][root][INFO] - ===> Epoch[7287](36435/5): Loss 0.0195, Sem 0.0195 LR: 4.312e-02 Score 99.311 Data time: 0.1714, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7209 [2021-07-08 16:04:50,832][root][INFO] - ===> Epoch[7288](36440/5): Loss 0.0202, Sem 0.0202 LR: 4.312e-02 Score 99.323 Data time: 0.1849, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6472 [2021-07-08 16:05:04,482][root][INFO] - ===> Epoch[7289](36445/5): Loss 0.0196, Sem 0.0196 LR: 4.311e-02 Score 99.310 Data time: 0.2478, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6824 [2021-07-08 16:05:17,932][root][INFO] - ===> Epoch[7290](36450/5): Loss 0.0201, Sem 0.0201 LR: 4.310e-02 Score 99.303 Data time: 0.2736, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6446 [2021-07-08 16:05:32,284][root][INFO] - ===> Epoch[7291](36455/5): Loss 0.0180, Sem 0.0180 LR: 4.309e-02 Score 99.367 Data time: 0.2243, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.8184 [2021-07-08 16:05:45,507][root][INFO] - ===> Epoch[7292](36460/5): Loss 0.0177, Sem 0.0177 LR: 4.308e-02 Score 99.368 Data time: 0.2185, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.5870 [2021-07-08 16:06:00,010][root][INFO] - ===> Epoch[7293](36465/5): Loss 0.0171, Sem 0.0171 LR: 4.307e-02 Score 99.406 Data time: 0.1600, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.8590 [2021-07-08 16:06:14,531][root][INFO] - ===> Epoch[7294](36470/5): Loss 0.0168, Sem 0.0168 LR: 4.307e-02 Score 99.397 Data time: 0.1832, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.8612 [2021-07-08 16:06:30,071][root][INFO] - ===> Epoch[7295](36475/5): Loss 0.0166, Sem 0.0166 LR: 4.306e-02 Score 99.410 Data time: 0.2835, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 3.0538 [2021-07-08 16:06:42,869][root][INFO] - ===> Epoch[7296](36480/5): Loss 0.0169, Sem 0.0169 LR: 4.305e-02 Score 99.401 Data time: 0.2605, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.5102 [2021-07-08 16:06:57,432][root][INFO] - ===> Epoch[7297](36485/5): Loss 0.0160, Sem 0.0160 LR: 4.304e-02 Score 99.432 Data time: 0.2381, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.8703 [2021-07-08 16:07:11,489][root][INFO] - ===> Epoch[7298](36490/5): Loss 0.0162, Sem 0.0162 LR: 4.303e-02 Score 99.422 Data time: 0.2414, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.7640 [2021-07-08 16:07:24,920][root][INFO] - ===> Epoch[7299](36495/5): Loss 0.0154, Sem 0.0154 LR: 4.302e-02 Score 99.454 Data time: 0.1911, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6351 [2021-07-08 16:07:39,468][root][INFO] - ===> Epoch[7300](36500/5): Loss 0.0157, Sem 0.0157 LR: 4.302e-02 Score 99.448 Data time: 0.2193, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.8506 [2021-07-08 16:07:54,253][root][INFO] - ===> Epoch[7301](36505/5): Loss 0.0153, Sem 0.0153 LR: 4.301e-02 Score 99.456 Data time: 0.1812, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.9149 [2021-07-08 16:08:07,669][root][INFO] - ===> Epoch[7302](36510/5): Loss 0.0155, Sem 0.0155 LR: 4.300e-02 Score 99.447 Data time: 0.2124, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.6343 [2021-07-08 16:08:22,553][root][INFO] - ===> Epoch[7303](36515/5): Loss 0.0150, Sem 0.0150 LR: 4.299e-02 Score 99.476 Data time: 0.2025, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.9386 [2021-07-08 16:08:36,400][root][INFO] - ===> Epoch[7304](36520/5): Loss 0.0153, Sem 0.0153 LR: 4.298e-02 Score 99.453 Data time: 0.2136, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.7166 [2021-07-08 16:08:50,707][root][INFO] - ===> Epoch[7305](36525/5): Loss 0.0146, Sem 0.0146 LR: 4.298e-02 Score 99.482 Data time: 0.2190, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.8191 [2021-07-08 16:09:04,540][root][INFO] - ===> Epoch[7306](36530/5): Loss 0.0151, Sem 0.0151 LR: 4.297e-02 Score 99.457 Data time: 0.2466, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.7248 [2021-07-08 16:09:18,005][root][INFO] - ===> Epoch[7307](36535/5): Loss 0.0145, Sem 0.0145 LR: 4.296e-02 Score 99.477 Data time: 0.2533, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.6542 [2021-07-08 16:09:31,347][root][INFO] - ===> Epoch[7308](36540/5): Loss 0.0145, Sem 0.0145 LR: 4.295e-02 Score 99.484 Data time: 0.2145, Forward time: 1.2559, Backward time: 1.2425, DDP time: 0.0042, Total iter time: 2.6211 [2021-07-08 16:09:46,381][root][INFO] - ===> Epoch[7309](36545/5): Loss 0.0151, Sem 0.0151 LR: 4.294e-02 Score 99.460 Data time: 0.1863, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.9669 [2021-07-08 16:09:59,245][root][INFO] - ===> Epoch[7310](36550/5): Loss 0.0135, Sem 0.0135 LR: 4.293e-02 Score 99.511 Data time: 0.2280, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.5343 [2021-07-08 16:10:13,496][root][INFO] - ===> Epoch[7311](36555/5): Loss 0.0142, Sem 0.0142 LR: 4.293e-02 Score 99.491 Data time: 0.2445, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.8043 [2021-07-08 16:10:27,085][root][INFO] - ===> Epoch[7312](36560/5): Loss 0.0137, Sem 0.0137 LR: 4.292e-02 Score 99.503 Data time: 0.2019, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.6789 [2021-07-08 16:10:41,506][root][INFO] - ===> Epoch[7313](36565/5): Loss 0.0138, Sem 0.0138 LR: 4.291e-02 Score 99.508 Data time: 0.1904, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.8470 [2021-07-08 16:10:55,263][root][INFO] - ===> Epoch[7314](36570/5): Loss 0.0141, Sem 0.0141 LR: 4.290e-02 Score 99.498 Data time: 0.2935, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.7007 [2021-07-08 16:11:09,488][root][INFO] - ===> Epoch[7315](36575/5): Loss 0.0141, Sem 0.0141 LR: 4.289e-02 Score 99.497 Data time: 0.1914, Forward time: 1.2558, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.8004 [2021-07-08 16:11:23,252][root][INFO] - ===> Epoch[7316](36580/5): Loss 0.0135, Sem 0.0135 LR: 4.288e-02 Score 99.521 Data time: 0.2249, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.7001 [2021-07-08 16:11:36,693][root][INFO] - ===> Epoch[7317](36585/5): Loss 0.0138, Sem 0.0138 LR: 4.288e-02 Score 99.505 Data time: 0.2275, Forward time: 1.2558, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.6442 [2021-07-08 16:11:50,948][root][INFO] - ===> Epoch[7318](36590/5): Loss 0.0133, Sem 0.0133 LR: 4.287e-02 Score 99.523 Data time: 0.2408, Forward time: 1.2558, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.8040 [2021-07-08 16:12:05,135][root][INFO] - ===> Epoch[7319](36595/5): Loss 0.0133, Sem 0.0133 LR: 4.286e-02 Score 99.524 Data time: 0.2056, Forward time: 1.2559, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.7855 [2021-07-08 16:12:18,875][root][INFO] - ===> Epoch[7320](36600/5): Loss 0.0135, Sem 0.0135 LR: 4.285e-02 Score 99.525 Data time: 0.2708, Forward time: 1.2558, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.6909 [2021-07-08 16:12:18,877][root][INFO] - ===> Start testing [2021-07-08 16:12:55,041][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1226 Loss 0.278 (AVG: 0.547) Score 93.633 (AVG: 90.719) mIOU 71.060 mAP 84.413 mAcc 76.950 IOU: 63.911 0.000 85.305 74.004 94.773 92.042 27.240 84.006 98.498 80.429 80.024 85.556 57.989 mAP: 84.606 9.384 95.018 86.192 98.862 98.813 55.571 98.169 99.830 92.806 91.885 94.620 91.610 mAcc: 82.078 0.000 90.525 80.846 95.853 97.487 30.234 94.123 99.489 82.143 90.519 96.170 60.878 [2021-07-08 16:12:55,041][root][INFO] - Finished test. Elapsed time: 36.1637 [2021-07-08 16:12:56,235][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 16:13:09,949][root][INFO] - ===> Epoch[7321](36605/5): Loss 0.0136, Sem 0.0136 LR: 4.284e-02 Score 99.514 Data time: 0.1478, Forward time: 1.2558, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.7061 [2021-07-08 16:13:24,921][root][INFO] - ===> Epoch[7322](36610/5): Loss 0.0132, Sem 0.0132 LR: 4.284e-02 Score 99.522 Data time: 0.2019, Forward time: 1.2558, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.9519 [2021-07-08 16:13:39,214][root][INFO] - ===> Epoch[7323](36615/5): Loss 0.0132, Sem 0.0132 LR: 4.283e-02 Score 99.536 Data time: 0.2167, Forward time: 1.2558, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.8140 [2021-07-08 16:13:52,955][root][INFO] - ===> Epoch[7324](36620/5): Loss 0.0140, Sem 0.0140 LR: 4.282e-02 Score 99.501 Data time: 0.1945, Forward time: 1.2558, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.6954 [2021-07-08 16:14:06,784][root][INFO] - ===> Epoch[7325](36625/5): Loss 0.0142, Sem 0.0142 LR: 4.281e-02 Score 99.492 Data time: 0.1697, Forward time: 1.2558, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.7202 [2021-07-08 16:14:20,525][root][INFO] - ===> Epoch[7326](36630/5): Loss 0.0129, Sem 0.0129 LR: 4.280e-02 Score 99.532 Data time: 0.1846, Forward time: 1.2558, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.7020 [2021-07-08 16:14:34,465][root][INFO] - ===> Epoch[7327](36635/5): Loss 0.0133, Sem 0.0133 LR: 4.279e-02 Score 99.522 Data time: 0.2098, Forward time: 1.2558, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.7523 [2021-07-08 16:14:48,775][root][INFO] - ===> Epoch[7328](36640/5): Loss 0.0127, Sem 0.0127 LR: 4.279e-02 Score 99.543 Data time: 0.2168, Forward time: 1.2558, Backward time: 1.2426, DDP time: 0.0042, Total iter time: 2.8142 [2021-07-08 16:15:03,744][root][INFO] - ===> Epoch[7329](36645/5): Loss 0.0128, Sem 0.0128 LR: 4.278e-02 Score 99.544 Data time: 0.1852, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.9437 [2021-07-08 16:15:18,767][root][INFO] - ===> Epoch[7330](36650/5): Loss 0.0130, Sem 0.0130 LR: 4.277e-02 Score 99.535 Data time: 0.2520, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.9449 [2021-07-08 16:15:32,468][root][INFO] - ===> Epoch[7331](36655/5): Loss 0.0125, Sem 0.0125 LR: 4.276e-02 Score 99.548 Data time: 0.2070, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.6981 [2021-07-08 16:15:46,413][root][INFO] - ===> Epoch[7332](36660/5): Loss 0.0126, Sem 0.0126 LR: 4.275e-02 Score 99.549 Data time: 0.2251, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.7498 [2021-07-08 16:16:00,588][root][INFO] - ===> Epoch[7333](36665/5): Loss 0.0134, Sem 0.0134 LR: 4.274e-02 Score 99.520 Data time: 0.2278, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.7801 [2021-07-08 16:16:14,754][root][INFO] - ===> Epoch[7334](36670/5): Loss 0.0126, Sem 0.0126 LR: 4.274e-02 Score 99.544 Data time: 0.1990, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.7882 [2021-07-08 16:16:29,426][root][INFO] - ===> Epoch[7335](36675/5): Loss 0.0133, Sem 0.0133 LR: 4.273e-02 Score 99.524 Data time: 0.2018, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.8870 [2021-07-08 16:16:43,331][root][INFO] - ===> Epoch[7336](36680/5): Loss 0.0130, Sem 0.0130 LR: 4.272e-02 Score 99.531 Data time: 0.2551, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.7328 [2021-07-08 16:16:58,080][root][INFO] - ===> Epoch[7337](36685/5): Loss 0.0135, Sem 0.0135 LR: 4.271e-02 Score 99.530 Data time: 0.3146, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.9096 [2021-07-08 16:17:12,337][root][INFO] - ===> Epoch[7338](36690/5): Loss 0.0131, Sem 0.0131 LR: 4.270e-02 Score 99.534 Data time: 0.2937, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.7993 [2021-07-08 16:17:27,185][root][INFO] - ===> Epoch[7339](36695/5): Loss 0.0127, Sem 0.0127 LR: 4.270e-02 Score 99.543 Data time: 0.3220, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.9251 [2021-07-08 16:17:42,020][root][INFO] - ===> Epoch[7340](36700/5): Loss 0.0130, Sem 0.0130 LR: 4.269e-02 Score 99.531 Data time: 0.2718, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.9171 [2021-07-08 16:17:55,788][root][INFO] - ===> Epoch[7341](36705/5): Loss 0.0129, Sem 0.0129 LR: 4.268e-02 Score 99.531 Data time: 0.2165, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.7138 [2021-07-08 16:18:09,226][root][INFO] - ===> Epoch[7342](36710/5): Loss 0.0134, Sem 0.0134 LR: 4.267e-02 Score 99.519 Data time: 0.2073, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.6447 [2021-07-08 16:18:23,710][root][INFO] - ===> Epoch[7343](36715/5): Loss 0.0126, Sem 0.0126 LR: 4.266e-02 Score 99.549 Data time: 0.2043, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.8522 [2021-07-08 16:18:37,901][root][INFO] - ===> Epoch[7344](36720/5): Loss 0.0130, Sem 0.0130 LR: 4.265e-02 Score 99.531 Data time: 0.2338, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.7912 [2021-07-08 16:18:52,135][root][INFO] - ===> Epoch[7345](36725/5): Loss 0.0128, Sem 0.0128 LR: 4.265e-02 Score 99.542 Data time: 0.2148, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.8069 [2021-07-08 16:19:06,353][root][INFO] - ===> Epoch[7346](36730/5): Loss 0.0128, Sem 0.0128 LR: 4.264e-02 Score 99.538 Data time: 0.1984, Forward time: 1.2558, Backward time: 1.2427, DDP time: 0.0042, Total iter time: 2.7972 [2021-07-08 16:19:21,473][root][INFO] - ===> Epoch[7347](36735/5): Loss 0.0123, Sem 0.0123 LR: 4.263e-02 Score 99.560 Data time: 0.2370, Forward time: 1.2558, Backward time: 1.2428, DDP time: 0.0042, Total iter time: 2.9805 [2021-07-08 16:19:35,893][root][INFO] - ===> Epoch[7348](36740/5): Loss 0.0126, Sem 0.0126 LR: 4.262e-02 Score 99.552 Data time: 0.1794, Forward time: 1.2558, Backward time: 1.2428, DDP time: 0.0042, Total iter time: 2.8422 [2021-07-08 16:19:49,943][root][INFO] - ===> Epoch[7349](36745/5): Loss 0.0128, Sem 0.0128 LR: 4.261e-02 Score 99.536 Data time: 0.2386, Forward time: 1.2558, Backward time: 1.2428, DDP time: 0.0042, Total iter time: 2.7600 [2021-07-08 16:20:03,883][root][INFO] - ===> Epoch[7350](36750/5): Loss 0.0136, Sem 0.0136 LR: 4.260e-02 Score 99.517 Data time: 0.2433, Forward time: 1.2558, Backward time: 1.2428, DDP time: 0.0042, Total iter time: 2.7247 [2021-07-08 16:20:18,091][root][INFO] - ===> Epoch[7351](36755/5): Loss 0.0132, Sem 0.0132 LR: 4.260e-02 Score 99.526 Data time: 0.2192, Forward time: 1.2558, Backward time: 1.2428, DDP time: 0.0042, Total iter time: 2.7965 [2021-07-08 16:20:32,132][root][INFO] - ===> Epoch[7352](36760/5): Loss 0.0129, Sem 0.0129 LR: 4.259e-02 Score 99.541 Data time: 0.1919, Forward time: 1.2558, Backward time: 1.2428, DDP time: 0.0042, Total iter time: 2.7595 [2021-07-08 16:20:46,383][root][INFO] - ===> Epoch[7353](36765/5): Loss 0.0127, Sem 0.0127 LR: 4.258e-02 Score 99.541 Data time: 0.2158, Forward time: 1.2558, Backward time: 1.2428, DDP time: 0.0042, Total iter time: 2.8024 [2021-07-08 16:21:00,464][root][INFO] - ===> Epoch[7354](36770/5): Loss 0.0125, Sem 0.0125 LR: 4.257e-02 Score 99.549 Data time: 0.1803, Forward time: 1.2558, Backward time: 1.2428, DDP time: 0.0042, Total iter time: 2.7764 [2021-07-08 16:21:14,843][root][INFO] - ===> Epoch[7355](36775/5): Loss 0.0126, Sem 0.0126 LR: 4.256e-02 Score 99.545 Data time: 0.2260, Forward time: 1.2558, Backward time: 1.2428, DDP time: 0.0042, Total iter time: 2.8287 [2021-07-08 16:21:28,809][root][INFO] - ===> Epoch[7356](36780/5): Loss 0.0133, Sem 0.0133 LR: 4.255e-02 Score 99.518 Data time: 0.2465, Forward time: 1.2558, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7572 [2021-07-08 16:21:41,823][root][INFO] - ===> Epoch[7357](36785/5): Loss 0.0133, Sem 0.0133 LR: 4.255e-02 Score 99.527 Data time: 0.2007, Forward time: 1.2558, Backward time: 1.2428, DDP time: 0.0042, Total iter time: 2.5574 [2021-07-08 16:21:56,012][root][INFO] - ===> Epoch[7358](36790/5): Loss 0.0129, Sem 0.0129 LR: 4.254e-02 Score 99.538 Data time: 0.2239, Forward time: 1.2558, Backward time: 1.2428, DDP time: 0.0042, Total iter time: 2.7819 [2021-07-08 16:22:09,752][root][INFO] - ===> Epoch[7359](36795/5): Loss 0.0140, Sem 0.0140 LR: 4.253e-02 Score 99.505 Data time: 0.2087, Forward time: 1.2558, Backward time: 1.2428, DDP time: 0.0042, Total iter time: 2.7085 [2021-07-08 16:22:23,622][root][INFO] - ===> Epoch[7360](36800/5): Loss 0.0128, Sem 0.0128 LR: 4.252e-02 Score 99.540 Data time: 0.1585, Forward time: 1.2558, Backward time: 1.2428, DDP time: 0.0042, Total iter time: 2.7312 [2021-07-08 16:22:23,623][root][INFO] - ===> Start testing [2021-07-08 16:23:04,871][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1690 Loss 0.339 (AVG: 0.512) Score 91.637 (AVG: 91.009) mIOU 71.081 mAP 84.703 mAcc 78.487 IOU: 64.043 0.000 73.484 75.766 94.664 91.188 40.813 76.279 98.487 83.492 80.523 87.254 58.063 mAP: 83.700 4.953 95.956 87.529 98.870 98.689 66.988 95.945 99.789 90.200 92.216 94.859 91.445 mAcc: 79.618 0.000 92.533 83.446 95.757 97.487 47.092 90.598 99.352 85.872 90.574 96.275 61.730 [2021-07-08 16:23:04,871][root][INFO] - Finished test. Elapsed time: 41.2473 [2021-07-08 16:23:05,971][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 16:23:19,873][root][INFO] - ===> Epoch[7361](36805/5): Loss 0.0125, Sem 0.0125 LR: 4.251e-02 Score 99.546 Data time: 0.2023, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7379 [2021-07-08 16:23:34,132][root][INFO] - ===> Epoch[7362](36810/5): Loss 0.0122, Sem 0.0122 LR: 4.251e-02 Score 99.559 Data time: 0.2198, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8095 [2021-07-08 16:23:48,226][root][INFO] - ===> Epoch[7363](36815/5): Loss 0.0125, Sem 0.0125 LR: 4.250e-02 Score 99.552 Data time: 0.1992, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7755 [2021-07-08 16:24:01,893][root][INFO] - ===> Epoch[7364](36820/5): Loss 0.0129, Sem 0.0129 LR: 4.249e-02 Score 99.534 Data time: 0.2268, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.6849 [2021-07-08 16:24:15,451][root][INFO] - ===> Epoch[7365](36825/5): Loss 0.0126, Sem 0.0126 LR: 4.248e-02 Score 99.545 Data time: 0.2419, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.6546 [2021-07-08 16:24:29,169][root][INFO] - ===> Epoch[7366](36830/5): Loss 0.0125, Sem 0.0125 LR: 4.247e-02 Score 99.547 Data time: 0.2001, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7033 [2021-07-08 16:24:43,938][root][INFO] - ===> Epoch[7367](36835/5): Loss 0.0122, Sem 0.0122 LR: 4.246e-02 Score 99.555 Data time: 0.2255, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.9018 [2021-07-08 16:24:57,902][root][INFO] - ===> Epoch[7368](36840/5): Loss 0.0124, Sem 0.0124 LR: 4.246e-02 Score 99.549 Data time: 0.2136, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7439 [2021-07-08 16:25:12,102][root][INFO] - ===> Epoch[7369](36845/5): Loss 0.0113, Sem 0.0113 LR: 4.245e-02 Score 99.593 Data time: 0.2261, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7889 [2021-07-08 16:25:26,086][root][INFO] - ===> Epoch[7370](36850/5): Loss 0.0130, Sem 0.0130 LR: 4.244e-02 Score 99.533 Data time: 0.2081, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7608 [2021-07-08 16:25:40,273][root][INFO] - ===> Epoch[7371](36855/5): Loss 0.0119, Sem 0.0119 LR: 4.243e-02 Score 99.575 Data time: 0.2516, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7883 [2021-07-08 16:25:54,253][root][INFO] - ===> Epoch[7372](36860/5): Loss 0.0122, Sem 0.0122 LR: 4.242e-02 Score 99.558 Data time: 0.2130, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7531 [2021-07-08 16:26:08,003][root][INFO] - ===> Epoch[7373](36865/5): Loss 0.0122, Sem 0.0122 LR: 4.241e-02 Score 99.562 Data time: 0.1891, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7026 [2021-07-08 16:26:22,691][root][INFO] - ===> Epoch[7374](36870/5): Loss 0.0121, Sem 0.0121 LR: 4.241e-02 Score 99.565 Data time: 0.2904, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8972 [2021-07-08 16:26:36,568][root][INFO] - ===> Epoch[7375](36875/5): Loss 0.0117, Sem 0.0117 LR: 4.240e-02 Score 99.574 Data time: 0.2517, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7313 [2021-07-08 16:26:50,524][root][INFO] - ===> Epoch[7376](36880/5): Loss 0.0123, Sem 0.0123 LR: 4.239e-02 Score 99.561 Data time: 0.2901, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7468 [2021-07-08 16:27:05,018][root][INFO] - ===> Epoch[7377](36885/5): Loss 0.0118, Sem 0.0118 LR: 4.238e-02 Score 99.571 Data time: 0.2293, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8520 [2021-07-08 16:27:18,906][root][INFO] - ===> Epoch[7378](36890/5): Loss 0.0121, Sem 0.0121 LR: 4.237e-02 Score 99.556 Data time: 0.3003, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7245 [2021-07-08 16:27:33,242][root][INFO] - ===> Epoch[7379](36895/5): Loss 0.0116, Sem 0.0116 LR: 4.237e-02 Score 99.580 Data time: 0.2099, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8213 [2021-07-08 16:27:47,438][root][INFO] - ===> Epoch[7380](36900/5): Loss 0.0115, Sem 0.0115 LR: 4.236e-02 Score 99.588 Data time: 0.2239, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7883 [2021-07-08 16:28:01,492][root][INFO] - ===> Epoch[7381](36905/5): Loss 0.0120, Sem 0.0120 LR: 4.235e-02 Score 99.569 Data time: 0.2149, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7595 [2021-07-08 16:28:15,651][root][INFO] - ===> Epoch[7382](36910/5): Loss 0.0112, Sem 0.0112 LR: 4.234e-02 Score 99.597 Data time: 0.2524, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7903 [2021-07-08 16:28:29,234][root][INFO] - ===> Epoch[7383](36915/5): Loss 0.0124, Sem 0.0124 LR: 4.233e-02 Score 99.556 Data time: 0.1809, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.6715 [2021-07-08 16:28:43,457][root][INFO] - ===> Epoch[7384](36920/5): Loss 0.0118, Sem 0.0118 LR: 4.232e-02 Score 99.572 Data time: 0.2218, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8068 [2021-07-08 16:28:57,477][root][INFO] - ===> Epoch[7385](36925/5): Loss 0.0126, Sem 0.0126 LR: 4.232e-02 Score 99.549 Data time: 0.2027, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7662 [2021-07-08 16:29:11,124][root][INFO] - ===> Epoch[7386](36930/5): Loss 0.0113, Sem 0.0113 LR: 4.231e-02 Score 99.588 Data time: 0.2528, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.6683 [2021-07-08 16:29:25,261][root][INFO] - ===> Epoch[7387](36935/5): Loss 0.0115, Sem 0.0115 LR: 4.230e-02 Score 99.583 Data time: 0.2365, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7851 [2021-07-08 16:29:39,491][root][INFO] - ===> Epoch[7388](36940/5): Loss 0.0121, Sem 0.0121 LR: 4.229e-02 Score 99.577 Data time: 0.2206, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7916 [2021-07-08 16:29:54,183][root][INFO] - ===> Epoch[7389](36945/5): Loss 0.0117, Sem 0.0117 LR: 4.228e-02 Score 99.574 Data time: 0.2740, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8894 [2021-07-08 16:30:08,738][root][INFO] - ===> Epoch[7390](36950/5): Loss 0.0120, Sem 0.0120 LR: 4.227e-02 Score 99.566 Data time: 0.2419, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8693 [2021-07-08 16:30:22,491][root][INFO] - ===> Epoch[7391](36955/5): Loss 0.0121, Sem 0.0121 LR: 4.227e-02 Score 99.569 Data time: 0.1929, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.6909 [2021-07-08 16:30:36,979][root][INFO] - ===> Epoch[7392](36960/5): Loss 0.0124, Sem 0.0124 LR: 4.226e-02 Score 99.557 Data time: 0.1823, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8546 [2021-07-08 16:30:51,735][root][INFO] - ===> Epoch[7393](36965/5): Loss 0.0120, Sem 0.0120 LR: 4.225e-02 Score 99.566 Data time: 0.2024, Forward time: 1.2557, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.9132 [2021-07-08 16:31:05,146][root][INFO] - ===> Epoch[7394](36970/5): Loss 0.0130, Sem 0.0130 LR: 4.224e-02 Score 99.542 Data time: 0.2322, Forward time: 1.2557, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6408 [2021-07-08 16:31:18,959][root][INFO] - ===> Epoch[7395](36975/5): Loss 0.0120, Sem 0.0120 LR: 4.223e-02 Score 99.566 Data time: 0.2190, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7209 [2021-07-08 16:31:32,570][root][INFO] - ===> Epoch[7396](36980/5): Loss 0.0117, Sem 0.0117 LR: 4.222e-02 Score 99.580 Data time: 0.2846, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.6800 [2021-07-08 16:31:46,558][root][INFO] - ===> Epoch[7397](36985/5): Loss 0.0121, Sem 0.0121 LR: 4.222e-02 Score 99.569 Data time: 0.2332, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7416 [2021-07-08 16:32:00,717][root][INFO] - ===> Epoch[7398](36990/5): Loss 0.0116, Sem 0.0116 LR: 4.221e-02 Score 99.582 Data time: 0.2096, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7841 [2021-07-08 16:32:14,919][root][INFO] - ===> Epoch[7399](36995/5): Loss 0.0115, Sem 0.0115 LR: 4.220e-02 Score 99.582 Data time: 0.2316, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7831 [2021-07-08 16:32:28,042][root][INFO] - ===> Epoch[7400](37000/5): Loss 0.0115, Sem 0.0115 LR: 4.219e-02 Score 99.582 Data time: 0.2296, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.5787 [2021-07-08 16:32:28,043][root][INFO] - ===> Start testing [2021-07-08 16:33:05,818][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1280 Loss 0.345 (AVG: 0.550) Score 92.013 (AVG: 91.120) mIOU 71.763 mAP 84.268 mAcc 77.637 IOU: 65.291 0.000 84.650 75.307 95.258 92.025 29.200 82.551 98.641 85.765 79.928 86.192 58.116 mAP: 85.343 7.339 94.598 87.277 98.843 98.609 53.945 97.656 99.801 94.478 91.828 94.789 90.978 mAcc: 81.470 0.000 89.718 82.655 96.353 96.685 31.793 94.072 99.395 88.447 90.670 96.656 61.372 [2021-07-08 16:33:05,818][root][INFO] - Finished test. Elapsed time: 37.7743 [2021-07-08 16:33:07,047][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 16:33:20,755][root][INFO] - ===> Epoch[7401](37005/5): Loss 0.0112, Sem 0.0112 LR: 4.218e-02 Score 99.593 Data time: 0.1886, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.6941 [2021-07-08 16:33:35,233][root][INFO] - ===> Epoch[7402](37010/5): Loss 0.0115, Sem 0.0115 LR: 4.218e-02 Score 99.584 Data time: 0.1935, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8481 [2021-07-08 16:33:49,541][root][INFO] - ===> Epoch[7403](37015/5): Loss 0.0114, Sem 0.0114 LR: 4.217e-02 Score 99.590 Data time: 0.2054, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8214 [2021-07-08 16:34:03,629][root][INFO] - ===> Epoch[7404](37020/5): Loss 0.0115, Sem 0.0115 LR: 4.216e-02 Score 99.585 Data time: 0.1771, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7792 [2021-07-08 16:34:18,369][root][INFO] - ===> Epoch[7405](37025/5): Loss 0.0116, Sem 0.0116 LR: 4.215e-02 Score 99.580 Data time: 0.2547, Forward time: 1.2557, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.9080 [2021-07-08 16:34:32,734][root][INFO] - ===> Epoch[7406](37030/5): Loss 0.0123, Sem 0.0123 LR: 4.214e-02 Score 99.561 Data time: 0.1572, Forward time: 1.2557, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.8394 [2021-07-08 16:34:45,698][root][INFO] - ===> Epoch[7407](37035/5): Loss 0.0120, Sem 0.0120 LR: 4.213e-02 Score 99.568 Data time: 0.1932, Forward time: 1.2557, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.5433 [2021-07-08 16:34:59,805][root][INFO] - ===> Epoch[7408](37040/5): Loss 0.0116, Sem 0.0116 LR: 4.213e-02 Score 99.578 Data time: 0.2374, Forward time: 1.2557, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7871 [2021-07-08 16:35:13,670][root][INFO] - ===> Epoch[7409](37045/5): Loss 0.0113, Sem 0.0113 LR: 4.212e-02 Score 99.590 Data time: 0.2289, Forward time: 1.2557, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7261 [2021-07-08 16:35:27,799][root][INFO] - ===> Epoch[7410](37050/5): Loss 0.0117, Sem 0.0117 LR: 4.211e-02 Score 99.577 Data time: 0.2635, Forward time: 1.2557, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7707 [2021-07-08 16:35:41,166][root][INFO] - ===> Epoch[7411](37055/5): Loss 0.0118, Sem 0.0118 LR: 4.210e-02 Score 99.577 Data time: 0.2396, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.6337 [2021-07-08 16:35:55,748][root][INFO] - ===> Epoch[7412](37060/5): Loss 0.0114, Sem 0.0114 LR: 4.209e-02 Score 99.590 Data time: 0.3022, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8621 [2021-07-08 16:36:09,701][root][INFO] - ===> Epoch[7413](37065/5): Loss 0.0119, Sem 0.0119 LR: 4.208e-02 Score 99.569 Data time: 0.2452, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7384 [2021-07-08 16:36:24,159][root][INFO] - ===> Epoch[7414](37070/5): Loss 0.0119, Sem 0.0119 LR: 4.208e-02 Score 99.571 Data time: 0.2589, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8411 [2021-07-08 16:36:38,015][root][INFO] - ===> Epoch[7415](37075/5): Loss 0.0112, Sem 0.0112 LR: 4.207e-02 Score 99.594 Data time: 0.2234, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7224 [2021-07-08 16:36:51,943][root][INFO] - ===> Epoch[7416](37080/5): Loss 0.0112, Sem 0.0112 LR: 4.206e-02 Score 99.595 Data time: 0.2286, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7349 [2021-07-08 16:37:05,428][root][INFO] - ===> Epoch[7417](37085/5): Loss 0.0110, Sem 0.0110 LR: 4.205e-02 Score 99.601 Data time: 0.2364, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.6516 [2021-07-08 16:37:19,623][root][INFO] - ===> Epoch[7418](37090/5): Loss 0.0108, Sem 0.0108 LR: 4.204e-02 Score 99.609 Data time: 0.1965, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7900 [2021-07-08 16:37:33,057][root][INFO] - ===> Epoch[7419](37095/5): Loss 0.0114, Sem 0.0114 LR: 4.204e-02 Score 99.589 Data time: 0.2530, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.6404 [2021-07-08 16:37:47,140][root][INFO] - ===> Epoch[7420](37100/5): Loss 0.0112, Sem 0.0112 LR: 4.203e-02 Score 99.598 Data time: 0.2306, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7678 [2021-07-08 16:38:01,691][root][INFO] - ===> Epoch[7421](37105/5): Loss 0.0109, Sem 0.0109 LR: 4.202e-02 Score 99.605 Data time: 0.2136, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8593 [2021-07-08 16:38:16,070][root][INFO] - ===> Epoch[7422](37110/5): Loss 0.0114, Sem 0.0114 LR: 4.201e-02 Score 99.585 Data time: 0.2374, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8335 [2021-07-08 16:38:30,044][root][INFO] - ===> Epoch[7423](37115/5): Loss 0.0110, Sem 0.0110 LR: 4.200e-02 Score 99.602 Data time: 0.2620, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7493 [2021-07-08 16:38:44,286][root][INFO] - ===> Epoch[7424](37120/5): Loss 0.0112, Sem 0.0112 LR: 4.199e-02 Score 99.596 Data time: 0.2021, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7980 [2021-07-08 16:38:57,515][root][INFO] - ===> Epoch[7425](37125/5): Loss 0.0113, Sem 0.0113 LR: 4.199e-02 Score 99.594 Data time: 0.1980, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.6089 [2021-07-08 16:39:11,129][root][INFO] - ===> Epoch[7426](37130/5): Loss 0.0116, Sem 0.0116 LR: 4.198e-02 Score 99.583 Data time: 0.2028, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.6776 [2021-07-08 16:39:25,169][root][INFO] - ===> Epoch[7427](37135/5): Loss 0.0115, Sem 0.0115 LR: 4.197e-02 Score 99.587 Data time: 0.1981, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.7626 [2021-07-08 16:39:38,879][root][INFO] - ===> Epoch[7428](37140/5): Loss 0.0106, Sem 0.0106 LR: 4.196e-02 Score 99.617 Data time: 0.1986, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6990 [2021-07-08 16:39:53,586][root][INFO] - ===> Epoch[7429](37145/5): Loss 0.0111, Sem 0.0111 LR: 4.195e-02 Score 99.601 Data time: 0.2582, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.8986 [2021-07-08 16:40:07,659][root][INFO] - ===> Epoch[7430](37150/5): Loss 0.0110, Sem 0.0110 LR: 4.194e-02 Score 99.606 Data time: 0.2302, Forward time: 1.2557, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7609 [2021-07-08 16:40:20,726][root][INFO] - ===> Epoch[7431](37155/5): Loss 0.0113, Sem 0.0113 LR: 4.194e-02 Score 99.593 Data time: 0.2284, Forward time: 1.2557, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.5713 [2021-07-08 16:40:33,955][root][INFO] - ===> Epoch[7432](37160/5): Loss 0.0114, Sem 0.0114 LR: 4.193e-02 Score 99.589 Data time: 0.1963, Forward time: 1.2556, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.5963 [2021-07-08 16:40:49,589][root][INFO] - ===> Epoch[7433](37165/5): Loss 0.0107, Sem 0.0107 LR: 4.192e-02 Score 99.614 Data time: 0.2549, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 3.0799 [2021-07-08 16:41:03,355][root][INFO] - ===> Epoch[7434](37170/5): Loss 0.0110, Sem 0.0110 LR: 4.191e-02 Score 99.600 Data time: 0.1797, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7071 [2021-07-08 16:41:16,955][root][INFO] - ===> Epoch[7435](37175/5): Loss 0.0113, Sem 0.0113 LR: 4.190e-02 Score 99.589 Data time: 0.1464, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6735 [2021-07-08 16:41:30,971][root][INFO] - ===> Epoch[7436](37180/5): Loss 0.0112, Sem 0.0112 LR: 4.189e-02 Score 99.600 Data time: 0.2280, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7583 [2021-07-08 16:41:44,915][root][INFO] - ===> Epoch[7437](37185/5): Loss 0.0112, Sem 0.0112 LR: 4.189e-02 Score 99.595 Data time: 0.1496, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7550 [2021-07-08 16:41:58,367][root][INFO] - ===> Epoch[7438](37190/5): Loss 0.0105, Sem 0.0105 LR: 4.188e-02 Score 99.619 Data time: 0.2068, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6571 [2021-07-08 16:42:12,869][root][INFO] - ===> Epoch[7439](37195/5): Loss 0.0114, Sem 0.0114 LR: 4.187e-02 Score 99.590 Data time: 0.2210, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.8568 [2021-07-08 16:42:27,305][root][INFO] - ===> Epoch[7440](37200/5): Loss 0.0108, Sem 0.0108 LR: 4.186e-02 Score 99.610 Data time: 0.2280, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.8366 [2021-07-08 16:42:27,307][root][INFO] - ===> Start testing [2021-07-08 16:43:04,319][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1255 Loss 0.380 (AVG: 0.580) Score 91.689 (AVG: 90.865) mIOU 71.765 mAP 83.960 mAcc 77.786 IOU: 63.801 0.000 84.126 74.944 94.861 91.522 28.628 83.978 98.566 88.738 80.436 85.849 57.494 mAP: 84.009 4.670 94.779 86.492 98.799 98.747 53.601 97.885 99.766 95.281 91.868 93.992 91.594 mAcc: 79.737 0.000 89.796 83.075 95.932 97.742 31.104 93.727 99.311 92.739 90.888 96.518 60.646 [2021-07-08 16:43:04,319][root][INFO] - Finished test. Elapsed time: 37.0110 [2021-07-08 16:43:05,408][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 16:43:17,546][root][INFO] - ===> Epoch[7441](37205/5): Loss 0.0105, Sem 0.0105 LR: 4.185e-02 Score 99.625 Data time: 0.1951, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.3880 [2021-07-08 16:43:32,414][root][INFO] - ===> Epoch[7442](37210/5): Loss 0.0109, Sem 0.0109 LR: 4.185e-02 Score 99.607 Data time: 0.1812, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.9176 [2021-07-08 16:43:46,189][root][INFO] - ===> Epoch[7443](37215/5): Loss 0.0111, Sem 0.0111 LR: 4.184e-02 Score 99.604 Data time: 0.2271, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7031 [2021-07-08 16:44:00,173][root][INFO] - ===> Epoch[7444](37220/5): Loss 0.0105, Sem 0.0105 LR: 4.183e-02 Score 99.621 Data time: 0.2438, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7507 [2021-07-08 16:44:13,837][root][INFO] - ===> Epoch[7445](37225/5): Loss 0.0117, Sem 0.0117 LR: 4.182e-02 Score 99.588 Data time: 0.2351, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6808 [2021-07-08 16:44:27,784][root][INFO] - ===> Epoch[7446](37230/5): Loss 0.0121, Sem 0.0121 LR: 4.181e-02 Score 99.571 Data time: 0.1884, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7411 [2021-07-08 16:44:40,572][root][INFO] - ===> Epoch[7447](37235/5): Loss 0.0120, Sem 0.0120 LR: 4.180e-02 Score 99.568 Data time: 0.1589, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.5150 [2021-07-08 16:44:54,744][root][INFO] - ===> Epoch[7448](37240/5): Loss 0.0121, Sem 0.0121 LR: 4.180e-02 Score 99.573 Data time: 0.2497, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7949 [2021-07-08 16:45:07,942][root][INFO] - ===> Epoch[7449](37245/5): Loss 0.0119, Sem 0.0119 LR: 4.179e-02 Score 99.572 Data time: 0.2349, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6001 [2021-07-08 16:45:22,310][root][INFO] - ===> Epoch[7450](37250/5): Loss 0.0114, Sem 0.0114 LR: 4.178e-02 Score 99.589 Data time: 0.2371, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.8312 [2021-07-08 16:45:36,803][root][INFO] - ===> Epoch[7451](37255/5): Loss 0.0109, Sem 0.0109 LR: 4.177e-02 Score 99.604 Data time: 0.2764, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.8475 [2021-07-08 16:45:51,203][root][INFO] - ===> Epoch[7452](37260/5): Loss 0.0109, Sem 0.0109 LR: 4.176e-02 Score 99.607 Data time: 0.2544, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.8358 [2021-07-08 16:46:05,292][root][INFO] - ===> Epoch[7453](37265/5): Loss 0.0113, Sem 0.0113 LR: 4.175e-02 Score 99.593 Data time: 0.2707, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7714 [2021-07-08 16:46:19,266][root][INFO] - ===> Epoch[7454](37270/5): Loss 0.0115, Sem 0.0115 LR: 4.175e-02 Score 99.587 Data time: 0.3498, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7489 [2021-07-08 16:46:33,761][root][INFO] - ===> Epoch[7455](37275/5): Loss 0.0110, Sem 0.0110 LR: 4.174e-02 Score 99.602 Data time: 0.3163, Forward time: 1.2556, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8533 [2021-07-08 16:46:48,036][root][INFO] - ===> Epoch[7456](37280/5): Loss 0.0114, Sem 0.0114 LR: 4.173e-02 Score 99.593 Data time: 0.2493, Forward time: 1.2556, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8101 [2021-07-08 16:47:02,345][root][INFO] - ===> Epoch[7457](37285/5): Loss 0.0110, Sem 0.0110 LR: 4.172e-02 Score 99.605 Data time: 0.2107, Forward time: 1.2556, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8174 [2021-07-08 16:47:16,638][root][INFO] - ===> Epoch[7458](37290/5): Loss 0.0110, Sem 0.0110 LR: 4.171e-02 Score 99.603 Data time: 0.2044, Forward time: 1.2556, Backward time: 1.2429, DDP time: 0.0042, Total iter time: 2.8148 [2021-07-08 16:47:31,145][root][INFO] - ===> Epoch[7459](37295/5): Loss 0.0106, Sem 0.0106 LR: 4.170e-02 Score 99.620 Data time: 0.2241, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.8614 [2021-07-08 16:47:46,037][root][INFO] - ===> Epoch[7460](37300/5): Loss 0.0107, Sem 0.0107 LR: 4.170e-02 Score 99.615 Data time: 0.2314, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.9384 [2021-07-08 16:48:00,078][root][INFO] - ===> Epoch[7461](37305/5): Loss 0.0109, Sem 0.0109 LR: 4.169e-02 Score 99.606 Data time: 0.2255, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7514 [2021-07-08 16:48:14,044][root][INFO] - ===> Epoch[7462](37310/5): Loss 0.0112, Sem 0.0112 LR: 4.168e-02 Score 99.597 Data time: 0.2280, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7449 [2021-07-08 16:48:27,743][root][INFO] - ===> Epoch[7463](37315/5): Loss 0.0111, Sem 0.0111 LR: 4.167e-02 Score 99.599 Data time: 0.2428, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6920 [2021-07-08 16:48:42,550][root][INFO] - ===> Epoch[7464](37320/5): Loss 0.0105, Sem 0.0105 LR: 4.166e-02 Score 99.618 Data time: 0.2056, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.9207 [2021-07-08 16:48:56,013][root][INFO] - ===> Epoch[7465](37325/5): Loss 0.0109, Sem 0.0109 LR: 4.166e-02 Score 99.607 Data time: 0.1905, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6596 [2021-07-08 16:49:09,915][root][INFO] - ===> Epoch[7466](37330/5): Loss 0.0108, Sem 0.0108 LR: 4.165e-02 Score 99.611 Data time: 0.2065, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7454 [2021-07-08 16:49:23,566][root][INFO] - ===> Epoch[7467](37335/5): Loss 0.0102, Sem 0.0102 LR: 4.164e-02 Score 99.629 Data time: 0.1819, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6940 [2021-07-08 16:49:38,007][root][INFO] - ===> Epoch[7468](37340/5): Loss 0.0106, Sem 0.0106 LR: 4.163e-02 Score 99.614 Data time: 0.2230, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.8444 [2021-07-08 16:49:51,953][root][INFO] - ===> Epoch[7469](37345/5): Loss 0.0113, Sem 0.0113 LR: 4.162e-02 Score 99.593 Data time: 0.2314, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7481 [2021-07-08 16:50:05,593][root][INFO] - ===> Epoch[7470](37350/5): Loss 0.0109, Sem 0.0109 LR: 4.161e-02 Score 99.605 Data time: 0.1531, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6864 [2021-07-08 16:50:18,587][root][INFO] - ===> Epoch[7471](37355/5): Loss 0.0111, Sem 0.0111 LR: 4.161e-02 Score 99.598 Data time: 0.2001, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.5534 [2021-07-08 16:50:33,635][root][INFO] - ===> Epoch[7472](37360/5): Loss 0.0107, Sem 0.0107 LR: 4.160e-02 Score 99.616 Data time: 0.2572, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.9621 [2021-07-08 16:50:46,844][root][INFO] - ===> Epoch[7473](37365/5): Loss 0.0110, Sem 0.0110 LR: 4.159e-02 Score 99.604 Data time: 0.1452, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6031 [2021-07-08 16:51:00,565][root][INFO] - ===> Epoch[7474](37370/5): Loss 0.0106, Sem 0.0106 LR: 4.158e-02 Score 99.622 Data time: 0.1936, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6937 [2021-07-08 16:51:14,608][root][INFO] - ===> Epoch[7475](37375/5): Loss 0.0104, Sem 0.0104 LR: 4.157e-02 Score 99.626 Data time: 0.2868, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7580 [2021-07-08 16:51:28,655][root][INFO] - ===> Epoch[7476](37380/5): Loss 0.0108, Sem 0.0108 LR: 4.156e-02 Score 99.607 Data time: 0.2115, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7546 [2021-07-08 16:51:43,692][root][INFO] - ===> Epoch[7477](37385/5): Loss 0.0105, Sem 0.0105 LR: 4.156e-02 Score 99.620 Data time: 0.1776, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.9468 [2021-07-08 16:51:57,765][root][INFO] - ===> Epoch[7478](37390/5): Loss 0.0107, Sem 0.0107 LR: 4.155e-02 Score 99.618 Data time: 0.2627, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.7705 [2021-07-08 16:52:11,001][root][INFO] - ===> Epoch[7479](37395/5): Loss 0.0103, Sem 0.0103 LR: 4.154e-02 Score 99.627 Data time: 0.2035, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.5991 [2021-07-08 16:52:25,245][root][INFO] - ===> Epoch[7480](37400/5): Loss 0.0109, Sem 0.0109 LR: 4.153e-02 Score 99.605 Data time: 0.2634, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.7860 [2021-07-08 16:52:25,246][root][INFO] - ===> Start testing [2021-07-08 16:53:08,270][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1208 Loss 0.424 (AVG: 0.558) Score 90.956 (AVG: 91.250) mIOU 72.007 mAP 84.435 mAcc 78.138 IOU: 65.190 0.000 83.336 75.829 95.240 92.561 31.448 81.414 98.586 87.967 80.170 86.740 57.610 mAP: 85.171 5.369 94.221 87.343 98.870 98.628 57.493 98.058 99.741 95.395 91.825 94.600 90.950 mAcc: 80.133 0.000 88.816 83.834 96.517 96.973 33.863 96.437 99.366 91.091 91.129 96.578 61.056 [2021-07-08 16:53:08,271][root][INFO] - Finished test. Elapsed time: 43.0237 [2021-07-08 16:53:09,338][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 16:53:22,327][root][INFO] - ===> Epoch[7481](37405/5): Loss 0.0108, Sem 0.0108 LR: 4.152e-02 Score 99.611 Data time: 0.1778, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.5635 [2021-07-08 16:53:35,991][root][INFO] - ===> Epoch[7482](37410/5): Loss 0.0107, Sem 0.0107 LR: 4.151e-02 Score 99.615 Data time: 0.2371, Forward time: 1.2555, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6758 [2021-07-08 16:53:51,188][root][INFO] - ===> Epoch[7483](37415/5): Loss 0.0104, Sem 0.0104 LR: 4.151e-02 Score 99.622 Data time: 0.1661, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.9971 [2021-07-08 16:54:05,479][root][INFO] - ===> Epoch[7484](37420/5): Loss 0.0101, Sem 0.0101 LR: 4.150e-02 Score 99.636 Data time: 0.2413, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.8080 [2021-07-08 16:54:19,863][root][INFO] - ===> Epoch[7485](37425/5): Loss 0.0106, Sem 0.0106 LR: 4.149e-02 Score 99.616 Data time: 0.2229, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.8268 [2021-07-08 16:54:33,077][root][INFO] - ===> Epoch[7486](37430/5): Loss 0.0110, Sem 0.0110 LR: 4.148e-02 Score 99.604 Data time: 0.2090, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.5955 [2021-07-08 16:54:47,360][root][INFO] - ===> Epoch[7487](37435/5): Loss 0.0107, Sem 0.0107 LR: 4.147e-02 Score 99.614 Data time: 0.1761, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.8256 [2021-07-08 16:55:02,345][root][INFO] - ===> Epoch[7488](37440/5): Loss 0.0102, Sem 0.0102 LR: 4.146e-02 Score 99.631 Data time: 0.2810, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.9418 [2021-07-08 16:55:16,403][root][INFO] - ===> Epoch[7489](37445/5): Loss 0.0105, Sem 0.0105 LR: 4.146e-02 Score 99.622 Data time: 0.2901, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.7567 [2021-07-08 16:55:30,688][root][INFO] - ===> Epoch[7490](37450/5): Loss 0.0100, Sem 0.0100 LR: 4.145e-02 Score 99.639 Data time: 0.2480, Forward time: 1.2556, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.8147 [2021-07-08 16:55:45,639][root][INFO] - ===> Epoch[7491](37455/5): Loss 0.0107, Sem 0.0107 LR: 4.144e-02 Score 99.610 Data time: 0.2578, Forward time: 1.2556, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.9477 [2021-07-08 16:55:59,986][root][INFO] - ===> Epoch[7492](37460/5): Loss 0.0099, Sem 0.0099 LR: 4.143e-02 Score 99.643 Data time: 0.2945, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.8283 [2021-07-08 16:56:14,812][root][INFO] - ===> Epoch[7493](37465/5): Loss 0.0102, Sem 0.0102 LR: 4.142e-02 Score 99.631 Data time: 0.2880, Forward time: 1.2556, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.9235 [2021-07-08 16:56:29,128][root][INFO] - ===> Epoch[7494](37470/5): Loss 0.0104, Sem 0.0104 LR: 4.142e-02 Score 99.624 Data time: 0.2584, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.8103 [2021-07-08 16:56:42,874][root][INFO] - ===> Epoch[7495](37475/5): Loss 0.0108, Sem 0.0108 LR: 4.141e-02 Score 99.608 Data time: 0.2155, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7053 [2021-07-08 16:56:55,988][root][INFO] - ===> Epoch[7496](37480/5): Loss 0.0106, Sem 0.0106 LR: 4.140e-02 Score 99.618 Data time: 0.2258, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.5787 [2021-07-08 16:57:10,553][root][INFO] - ===> Epoch[7497](37485/5): Loss 0.0102, Sem 0.0102 LR: 4.139e-02 Score 99.635 Data time: 0.1893, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.8678 [2021-07-08 16:57:23,876][root][INFO] - ===> Epoch[7498](37490/5): Loss 0.0112, Sem 0.0112 LR: 4.138e-02 Score 99.596 Data time: 0.2404, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6128 [2021-07-08 16:57:39,288][root][INFO] - ===> Epoch[7499](37495/5): Loss 0.0110, Sem 0.0110 LR: 4.137e-02 Score 99.613 Data time: 0.2608, Forward time: 1.2556, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 3.0299 [2021-07-08 16:57:52,704][root][INFO] - ===> Epoch[7500](37500/5): Loss 0.0122, Sem 0.0122 LR: 4.137e-02 Score 99.577 Data time: 0.2695, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6321 [2021-07-08 16:58:06,290][root][INFO] - ===> Epoch[7501](37505/5): Loss 0.0128, Sem 0.0128 LR: 4.136e-02 Score 99.570 Data time: 0.2432, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6615 [2021-07-08 16:58:20,720][root][INFO] - ===> Epoch[7502](37510/5): Loss 0.0116, Sem 0.0116 LR: 4.135e-02 Score 99.587 Data time: 0.2116, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.8419 [2021-07-08 16:58:35,123][root][INFO] - ===> Epoch[7503](37515/5): Loss 0.0110, Sem 0.0110 LR: 4.134e-02 Score 99.603 Data time: 0.2513, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.8297 [2021-07-08 16:58:49,090][root][INFO] - ===> Epoch[7504](37520/5): Loss 0.0122, Sem 0.0122 LR: 4.133e-02 Score 99.570 Data time: 0.1921, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7429 [2021-07-08 16:59:02,582][root][INFO] - ===> Epoch[7505](37525/5): Loss 0.0115, Sem 0.0115 LR: 4.132e-02 Score 99.580 Data time: 0.2218, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6464 [2021-07-08 16:59:16,839][root][INFO] - ===> Epoch[7506](37530/5): Loss 0.0110, Sem 0.0110 LR: 4.132e-02 Score 99.601 Data time: 0.2225, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.7964 [2021-07-08 16:59:30,496][root][INFO] - ===> Epoch[7507](37535/5): Loss 0.0112, Sem 0.0112 LR: 4.131e-02 Score 99.600 Data time: 0.1968, Forward time: 1.2556, Backward time: 1.2430, DDP time: 0.0042, Total iter time: 2.6931 [2021-07-08 16:59:45,265][root][INFO] - ===> Epoch[7508](37540/5): Loss 0.0108, Sem 0.0108 LR: 4.130e-02 Score 99.607 Data time: 0.2111, Forward time: 1.2556, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.9119 [2021-07-08 16:59:59,907][root][INFO] - ===> Epoch[7509](37545/5): Loss 0.0110, Sem 0.0110 LR: 4.129e-02 Score 99.605 Data time: 0.2386, Forward time: 1.2556, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.8690 [2021-07-08 17:00:14,149][root][INFO] - ===> Epoch[7510](37550/5): Loss 0.0110, Sem 0.0110 LR: 4.128e-02 Score 99.599 Data time: 0.2996, Forward time: 1.2556, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.8056 [2021-07-08 17:00:27,582][root][INFO] - ===> Epoch[7511](37555/5): Loss 0.0105, Sem 0.0105 LR: 4.127e-02 Score 99.621 Data time: 0.2345, Forward time: 1.2556, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.6488 [2021-07-08 17:00:41,844][root][INFO] - ===> Epoch[7512](37560/5): Loss 0.0107, Sem 0.0107 LR: 4.127e-02 Score 99.615 Data time: 0.1851, Forward time: 1.2556, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.8022 [2021-07-08 17:00:55,214][root][INFO] - ===> Epoch[7513](37565/5): Loss 0.0108, Sem 0.0108 LR: 4.126e-02 Score 99.611 Data time: 0.2245, Forward time: 1.2556, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.6256 [2021-07-08 17:01:08,604][root][INFO] - ===> Epoch[7514](37570/5): Loss 0.0108, Sem 0.0108 LR: 4.125e-02 Score 99.614 Data time: 0.2218, Forward time: 1.2556, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.6232 [2021-07-08 17:01:22,404][root][INFO] - ===> Epoch[7515](37575/5): Loss 0.0108, Sem 0.0108 LR: 4.124e-02 Score 99.612 Data time: 0.2146, Forward time: 1.2556, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.7162 [2021-07-08 17:01:36,803][root][INFO] - ===> Epoch[7516](37580/5): Loss 0.0105, Sem 0.0105 LR: 4.123e-02 Score 99.622 Data time: 0.1652, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.8437 [2021-07-08 17:01:50,941][root][INFO] - ===> Epoch[7517](37585/5): Loss 0.0104, Sem 0.0104 LR: 4.122e-02 Score 99.622 Data time: 0.2145, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.7820 [2021-07-08 17:02:05,238][root][INFO] - ===> Epoch[7518](37590/5): Loss 0.0104, Sem 0.0104 LR: 4.122e-02 Score 99.618 Data time: 0.2154, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.8212 [2021-07-08 17:02:20,031][root][INFO] - ===> Epoch[7519](37595/5): Loss 0.0101, Sem 0.0101 LR: 4.121e-02 Score 99.638 Data time: 0.1873, Forward time: 1.2555, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.9174 [2021-07-08 17:02:33,293][root][INFO] - ===> Epoch[7520](37600/5): Loss 0.0103, Sem 0.0103 LR: 4.120e-02 Score 99.628 Data time: 0.2768, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.5965 [2021-07-08 17:02:33,296][root][INFO] - ===> Start testing [2021-07-08 17:03:08,114][root][INFO] - 68/68: Data time: 0.1032, Iter time: 0.1264 Loss 0.360 (AVG: 0.563) Score 92.235 (AVG: 91.079) mIOU 71.972 mAP 84.057 mAcc 78.029 IOU: 64.279 0.000 84.419 74.834 95.176 92.212 29.580 83.415 98.619 87.402 80.556 86.472 58.670 mAP: 84.142 4.601 92.134 86.678 98.817 98.707 56.851 98.192 99.792 94.745 91.972 94.392 91.719 mAcc: 80.242 0.000 88.001 82.744 96.399 97.541 32.428 96.047 99.388 91.377 90.288 96.351 63.570 [2021-07-08 17:03:08,115][root][INFO] - Finished test. Elapsed time: 34.8170 [2021-07-08 17:03:09,185][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 17:03:22,193][root][INFO] - ===> Epoch[7521](37605/5): Loss 0.0106, Sem 0.0106 LR: 4.119e-02 Score 99.620 Data time: 0.1449, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.5680 [2021-07-08 17:03:35,515][root][INFO] - ===> Epoch[7522](37610/5): Loss 0.0102, Sem 0.0102 LR: 4.118e-02 Score 99.630 Data time: 0.2145, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.6151 [2021-07-08 17:03:49,749][root][INFO] - ===> Epoch[7523](37615/5): Loss 0.0111, Sem 0.0111 LR: 4.118e-02 Score 99.606 Data time: 0.2710, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.7994 [2021-07-08 17:04:04,033][root][INFO] - ===> Epoch[7524](37620/5): Loss 0.0109, Sem 0.0109 LR: 4.117e-02 Score 99.621 Data time: 0.1946, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.8163 [2021-07-08 17:04:18,348][root][INFO] - ===> Epoch[7525](37625/5): Loss 0.0109, Sem 0.0109 LR: 4.116e-02 Score 99.608 Data time: 0.1991, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.8136 [2021-07-08 17:04:31,860][root][INFO] - ===> Epoch[7526](37630/5): Loss 0.0105, Sem 0.0105 LR: 4.115e-02 Score 99.624 Data time: 0.1984, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.6639 [2021-07-08 17:04:45,228][root][INFO] - ===> Epoch[7527](37635/5): Loss 0.0108, Sem 0.0108 LR: 4.114e-02 Score 99.610 Data time: 0.2481, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.6246 [2021-07-08 17:04:59,708][root][INFO] - ===> Epoch[7528](37640/5): Loss 0.0101, Sem 0.0101 LR: 4.113e-02 Score 99.635 Data time: 0.3056, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.8505 [2021-07-08 17:05:13,729][root][INFO] - ===> Epoch[7529](37645/5): Loss 0.0104, Sem 0.0104 LR: 4.113e-02 Score 99.626 Data time: 0.2226, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.7623 [2021-07-08 17:05:28,310][root][INFO] - ===> Epoch[7530](37650/5): Loss 0.0105, Sem 0.0105 LR: 4.112e-02 Score 99.620 Data time: 0.2704, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.8613 [2021-07-08 17:05:43,385][root][INFO] - ===> Epoch[7531](37655/5): Loss 0.0104, Sem 0.0104 LR: 4.111e-02 Score 99.625 Data time: 0.2727, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.9752 [2021-07-08 17:05:57,901][root][INFO] - ===> Epoch[7532](37660/5): Loss 0.0100, Sem 0.0100 LR: 4.110e-02 Score 99.639 Data time: 0.3032, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.8528 [2021-07-08 17:06:11,726][root][INFO] - ===> Epoch[7533](37665/5): Loss 0.0102, Sem 0.0102 LR: 4.109e-02 Score 99.633 Data time: 0.2390, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.7044 [2021-07-08 17:06:25,242][root][INFO] - ===> Epoch[7534](37670/5): Loss 0.0103, Sem 0.0103 LR: 4.108e-02 Score 99.630 Data time: 0.2124, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.6636 [2021-07-08 17:06:39,605][root][INFO] - ===> Epoch[7535](37675/5): Loss 0.0101, Sem 0.0101 LR: 4.108e-02 Score 99.636 Data time: 0.2139, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.8219 [2021-07-08 17:06:53,194][root][INFO] - ===> Epoch[7536](37680/5): Loss 0.0099, Sem 0.0099 LR: 4.107e-02 Score 99.643 Data time: 0.2197, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.6696 [2021-07-08 17:07:07,392][root][INFO] - ===> Epoch[7537](37685/5): Loss 0.0108, Sem 0.0108 LR: 4.106e-02 Score 99.609 Data time: 0.2353, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.7876 [2021-07-08 17:07:20,282][root][INFO] - ===> Epoch[7538](37690/5): Loss 0.0101, Sem 0.0101 LR: 4.105e-02 Score 99.631 Data time: 0.2497, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.5377 [2021-07-08 17:07:34,565][root][INFO] - ===> Epoch[7539](37695/5): Loss 0.0103, Sem 0.0103 LR: 4.104e-02 Score 99.624 Data time: 0.1723, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.8081 [2021-07-08 17:07:48,859][root][INFO] - ===> Epoch[7540](37700/5): Loss 0.0106, Sem 0.0106 LR: 4.103e-02 Score 99.618 Data time: 0.2225, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.8128 [2021-07-08 17:08:02,796][root][INFO] - ===> Epoch[7541](37705/5): Loss 0.0104, Sem 0.0104 LR: 4.103e-02 Score 99.625 Data time: 0.2372, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.7385 [2021-07-08 17:08:16,669][root][INFO] - ===> Epoch[7542](37710/5): Loss 0.0103, Sem 0.0103 LR: 4.102e-02 Score 99.629 Data time: 0.2084, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.7270 [2021-07-08 17:08:31,775][root][INFO] - ===> Epoch[7543](37715/5): Loss 0.0106, Sem 0.0106 LR: 4.101e-02 Score 99.618 Data time: 0.2270, Forward time: 1.2555, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.9720 [2021-07-08 17:08:44,967][root][INFO] - ===> Epoch[7544](37720/5): Loss 0.0107, Sem 0.0107 LR: 4.100e-02 Score 99.609 Data time: 0.2763, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.5861 [2021-07-08 17:08:59,823][root][INFO] - ===> Epoch[7545](37725/5): Loss 0.0104, Sem 0.0104 LR: 4.099e-02 Score 99.625 Data time: 0.1733, Forward time: 1.2555, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.9308 [2021-07-08 17:09:13,588][root][INFO] - ===> Epoch[7546](37730/5): Loss 0.0099, Sem 0.0099 LR: 4.098e-02 Score 99.643 Data time: 0.2179, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.7021 [2021-07-08 17:09:27,396][root][INFO] - ===> Epoch[7547](37735/5): Loss 0.0108, Sem 0.0108 LR: 4.098e-02 Score 99.610 Data time: 0.2190, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.7193 [2021-07-08 17:09:42,081][root][INFO] - ===> Epoch[7548](37740/5): Loss 0.0097, Sem 0.0097 LR: 4.097e-02 Score 99.652 Data time: 0.2315, Forward time: 1.2555, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8814 [2021-07-08 17:09:55,795][root][INFO] - ===> Epoch[7549](37745/5): Loss 0.0100, Sem 0.0100 LR: 4.096e-02 Score 99.642 Data time: 0.2642, Forward time: 1.2555, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6929 [2021-07-08 17:10:09,174][root][INFO] - ===> Epoch[7550](37750/5): Loss 0.0105, Sem 0.0105 LR: 4.095e-02 Score 99.623 Data time: 0.2481, Forward time: 1.2555, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.6313 [2021-07-08 17:10:23,764][root][INFO] - ===> Epoch[7551](37755/5): Loss 0.0098, Sem 0.0098 LR: 4.094e-02 Score 99.650 Data time: 0.1628, Forward time: 1.2555, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8812 [2021-07-08 17:10:37,647][root][INFO] - ===> Epoch[7552](37760/5): Loss 0.0100, Sem 0.0100 LR: 4.094e-02 Score 99.638 Data time: 0.2200, Forward time: 1.2555, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7318 [2021-07-08 17:10:51,068][root][INFO] - ===> Epoch[7553](37765/5): Loss 0.0105, Sem 0.0105 LR: 4.093e-02 Score 99.624 Data time: 0.1782, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6418 [2021-07-08 17:11:04,477][root][INFO] - ===> Epoch[7554](37770/5): Loss 0.0107, Sem 0.0107 LR: 4.092e-02 Score 99.612 Data time: 0.2711, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6356 [2021-07-08 17:11:18,407][root][INFO] - ===> Epoch[7555](37775/5): Loss 0.0102, Sem 0.0102 LR: 4.091e-02 Score 99.634 Data time: 0.2292, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7400 [2021-07-08 17:11:32,203][root][INFO] - ===> Epoch[7556](37780/5): Loss 0.0104, Sem 0.0104 LR: 4.090e-02 Score 99.627 Data time: 0.2164, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7174 [2021-07-08 17:11:46,367][root][INFO] - ===> Epoch[7557](37785/5): Loss 0.0102, Sem 0.0102 LR: 4.089e-02 Score 99.634 Data time: 0.1994, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7794 [2021-07-08 17:12:00,895][root][INFO] - ===> Epoch[7558](37790/5): Loss 0.0104, Sem 0.0104 LR: 4.089e-02 Score 99.622 Data time: 0.2182, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8621 [2021-07-08 17:12:14,605][root][INFO] - ===> Epoch[7559](37795/5): Loss 0.0104, Sem 0.0104 LR: 4.088e-02 Score 99.626 Data time: 0.1967, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6892 [2021-07-08 17:12:29,019][root][INFO] - ===> Epoch[7560](37800/5): Loss 0.0099, Sem 0.0099 LR: 4.087e-02 Score 99.646 Data time: 0.2070, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8337 [2021-07-08 17:12:29,021][root][INFO] - ===> Start testing [2021-07-08 17:13:11,609][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1293 Loss 0.440 (AVG: 0.567) Score 91.091 (AVG: 91.044) mIOU 71.732 mAP 83.674 mAcc 77.508 IOU: 65.014 0.000 85.429 75.823 94.954 92.009 25.743 84.668 98.565 87.576 80.539 85.828 56.362 mAP: 84.514 4.329 93.916 87.121 98.768 98.697 51.972 98.468 99.794 95.212 91.108 94.459 89.406 mAcc: 80.343 0.000 89.707 83.473 96.137 97.596 27.583 95.697 99.256 90.479 90.789 96.851 59.699 [2021-07-08 17:13:11,610][root][INFO] - Finished test. Elapsed time: 42.5879 [2021-07-08 17:13:12,788][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 17:13:25,717][root][INFO] - ===> Epoch[7561](37805/5): Loss 0.0101, Sem 0.0101 LR: 4.086e-02 Score 99.638 Data time: 0.1962, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.5440 [2021-07-08 17:13:39,723][root][INFO] - ===> Epoch[7562](37810/5): Loss 0.0102, Sem 0.0102 LR: 4.085e-02 Score 99.630 Data time: 0.2285, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7471 [2021-07-08 17:13:55,490][root][INFO] - ===> Epoch[7563](37815/5): Loss 0.0099, Sem 0.0099 LR: 4.084e-02 Score 99.644 Data time: 0.2612, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 3.1032 [2021-07-08 17:14:08,604][root][INFO] - ===> Epoch[7564](37820/5): Loss 0.0107, Sem 0.0107 LR: 4.084e-02 Score 99.616 Data time: 0.2273, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.5779 [2021-07-08 17:14:22,969][root][INFO] - ===> Epoch[7565](37825/5): Loss 0.0102, Sem 0.0102 LR: 4.083e-02 Score 99.634 Data time: 0.2655, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8235 [2021-07-08 17:14:38,027][root][INFO] - ===> Epoch[7566](37830/5): Loss 0.0105, Sem 0.0105 LR: 4.082e-02 Score 99.620 Data time: 0.2562, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.9680 [2021-07-08 17:14:52,511][root][INFO] - ===> Epoch[7567](37835/5): Loss 0.0099, Sem 0.0099 LR: 4.081e-02 Score 99.645 Data time: 0.3483, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8458 [2021-07-08 17:15:06,449][root][INFO] - ===> Epoch[7568](37840/5): Loss 0.0102, Sem 0.0102 LR: 4.080e-02 Score 99.629 Data time: 0.3043, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7463 [2021-07-08 17:15:19,664][root][INFO] - ===> Epoch[7569](37845/5): Loss 0.0103, Sem 0.0103 LR: 4.079e-02 Score 99.628 Data time: 0.3005, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.5964 [2021-07-08 17:15:34,985][root][INFO] - ===> Epoch[7570](37850/5): Loss 0.0102, Sem 0.0102 LR: 4.079e-02 Score 99.631 Data time: 0.2582, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 3.0059 [2021-07-08 17:15:49,224][root][INFO] - ===> Epoch[7571](37855/5): Loss 0.0099, Sem 0.0099 LR: 4.078e-02 Score 99.641 Data time: 0.2642, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8014 [2021-07-08 17:16:03,712][root][INFO] - ===> Epoch[7572](37860/5): Loss 0.0095, Sem 0.0095 LR: 4.077e-02 Score 99.655 Data time: 0.2724, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8542 [2021-07-08 17:16:18,146][root][INFO] - ===> Epoch[7573](37865/5): Loss 0.0099, Sem 0.0099 LR: 4.076e-02 Score 99.645 Data time: 0.1661, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8469 [2021-07-08 17:16:32,287][root][INFO] - ===> Epoch[7574](37870/5): Loss 0.0098, Sem 0.0098 LR: 4.075e-02 Score 99.645 Data time: 0.1770, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7797 [2021-07-08 17:16:46,363][root][INFO] - ===> Epoch[7575](37875/5): Loss 0.0104, Sem 0.0104 LR: 4.074e-02 Score 99.622 Data time: 0.1949, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7683 [2021-07-08 17:17:00,236][root][INFO] - ===> Epoch[7576](37880/5): Loss 0.0097, Sem 0.0097 LR: 4.074e-02 Score 99.648 Data time: 0.1884, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7329 [2021-07-08 17:17:13,716][root][INFO] - ===> Epoch[7577](37885/5): Loss 0.0101, Sem 0.0101 LR: 4.073e-02 Score 99.633 Data time: 0.1664, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6534 [2021-07-08 17:17:28,494][root][INFO] - ===> Epoch[7578](37890/5): Loss 0.0098, Sem 0.0098 LR: 4.072e-02 Score 99.643 Data time: 0.2266, Forward time: 1.2554, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.8976 [2021-07-08 17:17:42,662][root][INFO] - ===> Epoch[7579](37895/5): Loss 0.0099, Sem 0.0099 LR: 4.071e-02 Score 99.642 Data time: 0.2673, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7908 [2021-07-08 17:17:55,890][root][INFO] - ===> Epoch[7580](37900/5): Loss 0.0100, Sem 0.0100 LR: 4.070e-02 Score 99.637 Data time: 0.2469, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6012 [2021-07-08 17:18:10,235][root][INFO] - ===> Epoch[7581](37905/5): Loss 0.0099, Sem 0.0099 LR: 4.069e-02 Score 99.638 Data time: 0.2223, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8176 [2021-07-08 17:18:25,212][root][INFO] - ===> Epoch[7582](37910/5): Loss 0.0096, Sem 0.0096 LR: 4.069e-02 Score 99.654 Data time: 0.2359, Forward time: 1.2555, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.9517 [2021-07-08 17:18:38,999][root][INFO] - ===> Epoch[7583](37915/5): Loss 0.0097, Sem 0.0097 LR: 4.068e-02 Score 99.647 Data time: 0.1994, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7227 [2021-07-08 17:18:52,525][root][INFO] - ===> Epoch[7584](37920/5): Loss 0.0098, Sem 0.0098 LR: 4.067e-02 Score 99.643 Data time: 0.2265, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6640 [2021-07-08 17:19:05,662][root][INFO] - ===> Epoch[7585](37925/5): Loss 0.0099, Sem 0.0099 LR: 4.066e-02 Score 99.645 Data time: 0.2206, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.5842 [2021-07-08 17:19:19,856][root][INFO] - ===> Epoch[7586](37930/5): Loss 0.0102, Sem 0.0102 LR: 4.065e-02 Score 99.632 Data time: 0.2522, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7928 [2021-07-08 17:19:33,999][root][INFO] - ===> Epoch[7587](37935/5): Loss 0.0096, Sem 0.0096 LR: 4.065e-02 Score 99.655 Data time: 0.2001, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7835 [2021-07-08 17:19:47,262][root][INFO] - ===> Epoch[7588](37940/5): Loss 0.0103, Sem 0.0103 LR: 4.064e-02 Score 99.627 Data time: 0.2424, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6033 [2021-07-08 17:20:01,053][root][INFO] - ===> Epoch[7589](37945/5): Loss 0.0101, Sem 0.0101 LR: 4.063e-02 Score 99.636 Data time: 0.2209, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7122 [2021-07-08 17:20:14,801][root][INFO] - ===> Epoch[7590](37950/5): Loss 0.0103, Sem 0.0103 LR: 4.062e-02 Score 99.629 Data time: 0.2124, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7116 [2021-07-08 17:20:28,705][root][INFO] - ===> Epoch[7591](37955/5): Loss 0.0103, Sem 0.0103 LR: 4.061e-02 Score 99.632 Data time: 0.2259, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7271 [2021-07-08 17:20:43,179][root][INFO] - ===> Epoch[7592](37960/5): Loss 0.0109, Sem 0.0109 LR: 4.060e-02 Score 99.607 Data time: 0.1587, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8637 [2021-07-08 17:20:56,602][root][INFO] - ===> Epoch[7593](37965/5): Loss 0.0101, Sem 0.0101 LR: 4.060e-02 Score 99.631 Data time: 0.2104, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6350 [2021-07-08 17:21:10,195][root][INFO] - ===> Epoch[7594](37970/5): Loss 0.0100, Sem 0.0100 LR: 4.059e-02 Score 99.641 Data time: 0.2085, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6759 [2021-07-08 17:21:24,348][root][INFO] - ===> Epoch[7595](37975/5): Loss 0.0100, Sem 0.0100 LR: 4.058e-02 Score 99.637 Data time: 0.1984, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7799 [2021-07-08 17:21:37,169][root][INFO] - ===> Epoch[7596](37980/5): Loss 0.0102, Sem 0.0102 LR: 4.057e-02 Score 99.630 Data time: 0.1925, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.5085 [2021-07-08 17:21:50,962][root][INFO] - ===> Epoch[7597](37985/5): Loss 0.0106, Sem 0.0106 LR: 4.056e-02 Score 99.622 Data time: 0.1864, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7106 [2021-07-08 17:22:04,697][root][INFO] - ===> Epoch[7598](37990/5): Loss 0.0104, Sem 0.0104 LR: 4.055e-02 Score 99.621 Data time: 0.2175, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7051 [2021-07-08 17:22:18,971][root][INFO] - ===> Epoch[7599](37995/5): Loss 0.0099, Sem 0.0099 LR: 4.055e-02 Score 99.638 Data time: 0.2330, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8080 [2021-07-08 17:22:33,388][root][INFO] - ===> Epoch[7600](38000/5): Loss 0.0102, Sem 0.0102 LR: 4.054e-02 Score 99.628 Data time: 0.2188, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8416 [2021-07-08 17:22:33,390][root][INFO] - ===> Start testing [2021-07-08 17:23:11,008][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1300 Loss 0.471 (AVG: 0.559) Score 91.307 (AVG: 91.251) mIOU 71.870 mAP 84.311 mAcc 77.620 IOU: 65.320 0.000 84.642 75.662 95.030 91.822 34.502 83.974 98.524 79.852 80.808 86.666 57.517 mAP: 85.211 5.182 92.857 86.462 98.716 98.542 61.845 98.162 99.753 93.324 91.485 94.273 90.228 mAcc: 81.732 0.000 87.045 82.803 96.385 97.755 37.622 96.269 99.319 81.955 90.088 96.534 61.546 [2021-07-08 17:23:11,009][root][INFO] - Finished test. Elapsed time: 37.6181 [2021-07-08 17:23:12,075][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 17:23:24,666][root][INFO] - ===> Epoch[7601](38005/5): Loss 0.0103, Sem 0.0103 LR: 4.053e-02 Score 99.629 Data time: 0.1963, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.4804 [2021-07-08 17:23:38,439][root][INFO] - ===> Epoch[7602](38010/5): Loss 0.0099, Sem 0.0099 LR: 4.052e-02 Score 99.641 Data time: 0.1941, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7031 [2021-07-08 17:23:52,227][root][INFO] - ===> Epoch[7603](38015/5): Loss 0.0107, Sem 0.0107 LR: 4.051e-02 Score 99.617 Data time: 0.1866, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7125 [2021-07-08 17:24:06,042][root][INFO] - ===> Epoch[7604](38020/5): Loss 0.0098, Sem 0.0098 LR: 4.050e-02 Score 99.645 Data time: 0.1550, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7273 [2021-07-08 17:24:20,237][root][INFO] - ===> Epoch[7605](38025/5): Loss 0.0100, Sem 0.0100 LR: 4.050e-02 Score 99.640 Data time: 0.2776, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7873 [2021-07-08 17:24:34,771][root][INFO] - ===> Epoch[7606](38030/5): Loss 0.0102, Sem 0.0102 LR: 4.049e-02 Score 99.631 Data time: 0.2679, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8622 [2021-07-08 17:24:49,080][root][INFO] - ===> Epoch[7607](38035/5): Loss 0.0099, Sem 0.0099 LR: 4.048e-02 Score 99.641 Data time: 0.3123, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8086 [2021-07-08 17:25:02,789][root][INFO] - ===> Epoch[7608](38040/5): Loss 0.0098, Sem 0.0098 LR: 4.047e-02 Score 99.644 Data time: 0.2779, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6987 [2021-07-08 17:25:17,189][root][INFO] - ===> Epoch[7609](38045/5): Loss 0.0102, Sem 0.0102 LR: 4.046e-02 Score 99.628 Data time: 0.3358, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8361 [2021-07-08 17:25:31,124][root][INFO] - ===> Epoch[7610](38050/5): Loss 0.0102, Sem 0.0102 LR: 4.045e-02 Score 99.630 Data time: 0.2671, Forward time: 1.2554, Backward time: 1.2431, DDP time: 0.0042, Total iter time: 2.7338 [2021-07-08 17:25:45,482][root][INFO] - ===> Epoch[7611](38055/5): Loss 0.0101, Sem 0.0101 LR: 4.045e-02 Score 99.632 Data time: 0.2104, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8311 [2021-07-08 17:26:00,056][root][INFO] - ===> Epoch[7612](38060/5): Loss 0.0100, Sem 0.0100 LR: 4.044e-02 Score 99.639 Data time: 0.2387, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8774 [2021-07-08 17:26:13,649][root][INFO] - ===> Epoch[7613](38065/5): Loss 0.0101, Sem 0.0101 LR: 4.043e-02 Score 99.635 Data time: 0.1656, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6730 [2021-07-08 17:26:27,805][root][INFO] - ===> Epoch[7614](38070/5): Loss 0.0101, Sem 0.0101 LR: 4.042e-02 Score 99.638 Data time: 0.2687, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7888 [2021-07-08 17:26:42,230][root][INFO] - ===> Epoch[7615](38075/5): Loss 0.0099, Sem 0.0099 LR: 4.041e-02 Score 99.643 Data time: 0.2352, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8425 [2021-07-08 17:26:56,102][root][INFO] - ===> Epoch[7616](38080/5): Loss 0.0095, Sem 0.0095 LR: 4.040e-02 Score 99.655 Data time: 0.2070, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7319 [2021-07-08 17:27:09,434][root][INFO] - ===> Epoch[7617](38085/5): Loss 0.0101, Sem 0.0101 LR: 4.040e-02 Score 99.638 Data time: 0.1971, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6293 [2021-07-08 17:27:23,505][root][INFO] - ===> Epoch[7618](38090/5): Loss 0.0099, Sem 0.0099 LR: 4.039e-02 Score 99.642 Data time: 0.2012, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7750 [2021-07-08 17:27:37,629][root][INFO] - ===> Epoch[7619](38095/5): Loss 0.0098, Sem 0.0098 LR: 4.038e-02 Score 99.646 Data time: 0.2033, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7820 [2021-07-08 17:27:51,073][root][INFO] - ===> Epoch[7620](38100/5): Loss 0.0093, Sem 0.0093 LR: 4.037e-02 Score 99.668 Data time: 0.2461, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6340 [2021-07-08 17:28:05,185][root][INFO] - ===> Epoch[7621](38105/5): Loss 0.0101, Sem 0.0101 LR: 4.036e-02 Score 99.636 Data time: 0.1850, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7894 [2021-07-08 17:28:19,643][root][INFO] - ===> Epoch[7622](38110/5): Loss 0.0101, Sem 0.0101 LR: 4.035e-02 Score 99.635 Data time: 0.2509, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8325 [2021-07-08 17:28:34,225][root][INFO] - ===> Epoch[7623](38115/5): Loss 0.0096, Sem 0.0096 LR: 4.035e-02 Score 99.653 Data time: 0.2299, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8667 [2021-07-08 17:28:48,205][root][INFO] - ===> Epoch[7624](38120/5): Loss 0.0097, Sem 0.0097 LR: 4.034e-02 Score 99.650 Data time: 0.2512, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7551 [2021-07-08 17:29:02,463][root][INFO] - ===> Epoch[7625](38125/5): Loss 0.0097, Sem 0.0097 LR: 4.033e-02 Score 99.652 Data time: 0.1850, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8054 [2021-07-08 17:29:16,663][root][INFO] - ===> Epoch[7626](38130/5): Loss 0.0099, Sem 0.0099 LR: 4.032e-02 Score 99.644 Data time: 0.1705, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8046 [2021-07-08 17:29:30,156][root][INFO] - ===> Epoch[7627](38135/5): Loss 0.0100, Sem 0.0100 LR: 4.031e-02 Score 99.639 Data time: 0.2506, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6608 [2021-07-08 17:29:43,710][root][INFO] - ===> Epoch[7628](38140/5): Loss 0.0098, Sem 0.0098 LR: 4.031e-02 Score 99.645 Data time: 0.2091, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6580 [2021-07-08 17:29:57,624][root][INFO] - ===> Epoch[7629](38145/5): Loss 0.0101, Sem 0.0101 LR: 4.030e-02 Score 99.637 Data time: 0.2209, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7435 [2021-07-08 17:30:11,889][root][INFO] - ===> Epoch[7630](38150/5): Loss 0.0097, Sem 0.0097 LR: 4.029e-02 Score 99.650 Data time: 0.2621, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8072 [2021-07-08 17:30:26,246][root][INFO] - ===> Epoch[7631](38155/5): Loss 0.0098, Sem 0.0098 LR: 4.028e-02 Score 99.642 Data time: 0.1718, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8409 [2021-07-08 17:30:40,219][root][INFO] - ===> Epoch[7632](38160/5): Loss 0.0100, Sem 0.0100 LR: 4.027e-02 Score 99.643 Data time: 0.1764, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7443 [2021-07-08 17:30:53,428][root][INFO] - ===> Epoch[7633](38165/5): Loss 0.0102, Sem 0.0102 LR: 4.026e-02 Score 99.628 Data time: 0.1946, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.5900 [2021-07-08 17:31:07,920][root][INFO] - ===> Epoch[7634](38170/5): Loss 0.0100, Sem 0.0100 LR: 4.026e-02 Score 99.640 Data time: 0.1960, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8656 [2021-07-08 17:31:21,017][root][INFO] - ===> Epoch[7635](38175/5): Loss 0.0098, Sem 0.0098 LR: 4.025e-02 Score 99.650 Data time: 0.1946, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.5795 [2021-07-08 17:31:34,317][root][INFO] - ===> Epoch[7636](38180/5): Loss 0.0098, Sem 0.0098 LR: 4.024e-02 Score 99.644 Data time: 0.2033, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6158 [2021-07-08 17:31:48,672][root][INFO] - ===> Epoch[7637](38185/5): Loss 0.0096, Sem 0.0096 LR: 4.023e-02 Score 99.653 Data time: 0.2497, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8263 [2021-07-08 17:32:02,555][root][INFO] - ===> Epoch[7638](38190/5): Loss 0.0100, Sem 0.0100 LR: 4.022e-02 Score 99.639 Data time: 0.1933, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7323 [2021-07-08 17:32:16,719][root][INFO] - ===> Epoch[7639](38195/5): Loss 0.0094, Sem 0.0094 LR: 4.021e-02 Score 99.660 Data time: 0.2167, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7926 [2021-07-08 17:32:30,725][root][INFO] - ===> Epoch[7640](38200/5): Loss 0.0098, Sem 0.0098 LR: 4.021e-02 Score 99.647 Data time: 0.2143, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7518 [2021-07-08 17:32:30,726][root][INFO] - ===> Start testing [2021-07-08 17:33:06,084][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1228 Loss 0.452 (AVG: 0.587) Score 90.998 (AVG: 91.077) mIOU 71.792 mAP 83.968 mAcc 77.683 IOU: 64.462 0.000 82.892 75.682 94.748 92.097 28.749 85.362 98.659 86.231 80.574 86.117 57.724 mAP: 84.887 4.762 93.254 87.218 98.548 98.504 55.915 98.321 99.746 94.969 90.943 93.619 90.897 mAcc: 80.331 0.000 88.217 83.056 96.056 97.400 30.379 95.915 99.411 90.578 90.305 96.841 61.385 [2021-07-08 17:33:06,084][root][INFO] - Finished test. Elapsed time: 35.3567 [2021-07-08 17:33:07,185][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 17:33:20,027][root][INFO] - ===> Epoch[7641](38205/5): Loss 0.0096, Sem 0.0096 LR: 4.020e-02 Score 99.651 Data time: 0.2221, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.5261 [2021-07-08 17:33:34,110][root][INFO] - ===> Epoch[7642](38210/5): Loss 0.0097, Sem 0.0097 LR: 4.019e-02 Score 99.653 Data time: 0.2050, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7563 [2021-07-08 17:33:48,612][root][INFO] - ===> Epoch[7643](38215/5): Loss 0.0091, Sem 0.0091 LR: 4.018e-02 Score 99.672 Data time: 0.2973, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8523 [2021-07-08 17:34:02,194][root][INFO] - ===> Epoch[7644](38220/5): Loss 0.0101, Sem 0.0101 LR: 4.017e-02 Score 99.638 Data time: 0.1756, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6726 [2021-07-08 17:34:17,116][root][INFO] - ===> Epoch[7645](38225/5): Loss 0.0096, Sem 0.0096 LR: 4.016e-02 Score 99.656 Data time: 0.2446, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.9343 [2021-07-08 17:34:31,396][root][INFO] - ===> Epoch[7646](38230/5): Loss 0.0096, Sem 0.0096 LR: 4.016e-02 Score 99.652 Data time: 0.2815, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8042 [2021-07-08 17:34:45,719][root][INFO] - ===> Epoch[7647](38235/5): Loss 0.0095, Sem 0.0095 LR: 4.015e-02 Score 99.661 Data time: 0.2852, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8134 [2021-07-08 17:35:00,345][root][INFO] - ===> Epoch[7648](38240/5): Loss 0.0098, Sem 0.0098 LR: 4.014e-02 Score 99.648 Data time: 0.2653, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8768 [2021-07-08 17:35:15,908][root][INFO] - ===> Epoch[7649](38245/5): Loss 0.0095, Sem 0.0095 LR: 4.013e-02 Score 99.659 Data time: 0.2812, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 3.0599 [2021-07-08 17:35:30,178][root][INFO] - ===> Epoch[7650](38250/5): Loss 0.0096, Sem 0.0096 LR: 4.012e-02 Score 99.652 Data time: 0.2714, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8090 [2021-07-08 17:35:44,488][root][INFO] - ===> Epoch[7651](38255/5): Loss 0.0093, Sem 0.0093 LR: 4.011e-02 Score 99.665 Data time: 0.2801, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8020 [2021-07-08 17:35:59,170][root][INFO] - ===> Epoch[7652](38260/5): Loss 0.0097, Sem 0.0097 LR: 4.011e-02 Score 99.650 Data time: 0.2345, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8859 [2021-07-08 17:36:13,787][root][INFO] - ===> Epoch[7653](38265/5): Loss 0.0091, Sem 0.0091 LR: 4.010e-02 Score 99.671 Data time: 0.2364, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8756 [2021-07-08 17:36:27,583][root][INFO] - ===> Epoch[7654](38270/5): Loss 0.0100, Sem 0.0100 LR: 4.009e-02 Score 99.636 Data time: 0.1684, Forward time: 1.2554, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7217 [2021-07-08 17:36:41,791][root][INFO] - ===> Epoch[7655](38275/5): Loss 0.0094, Sem 0.0094 LR: 4.008e-02 Score 99.664 Data time: 0.1482, Forward time: 1.2554, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.8076 [2021-07-08 17:36:55,889][root][INFO] - ===> Epoch[7656](38280/5): Loss 0.0098, Sem 0.0098 LR: 4.007e-02 Score 99.647 Data time: 0.2137, Forward time: 1.2554, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7814 [2021-07-08 17:37:09,772][root][INFO] - ===> Epoch[7657](38285/5): Loss 0.0099, Sem 0.0099 LR: 4.006e-02 Score 99.640 Data time: 0.2176, Forward time: 1.2554, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7281 [2021-07-08 17:37:23,752][root][INFO] - ===> Epoch[7658](38290/5): Loss 0.0091, Sem 0.0091 LR: 4.006e-02 Score 99.670 Data time: 0.2002, Forward time: 1.2554, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7535 [2021-07-08 17:37:37,949][root][INFO] - ===> Epoch[7659](38295/5): Loss 0.0092, Sem 0.0092 LR: 4.005e-02 Score 99.667 Data time: 0.1806, Forward time: 1.2554, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7952 [2021-07-08 17:37:51,235][root][INFO] - ===> Epoch[7660](38300/5): Loss 0.0097, Sem 0.0097 LR: 4.004e-02 Score 99.653 Data time: 0.2302, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.6043 [2021-07-08 17:38:05,478][root][INFO] - ===> Epoch[7661](38305/5): Loss 0.0100, Sem 0.0100 LR: 4.003e-02 Score 99.638 Data time: 0.1977, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7957 [2021-07-08 17:38:20,087][root][INFO] - ===> Epoch[7662](38310/5): Loss 0.0096, Sem 0.0096 LR: 4.002e-02 Score 99.653 Data time: 0.2134, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.8712 [2021-07-08 17:38:33,811][root][INFO] - ===> Epoch[7663](38315/5): Loss 0.0096, Sem 0.0096 LR: 4.001e-02 Score 99.654 Data time: 0.1830, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7026 [2021-07-08 17:38:48,516][root][INFO] - ===> Epoch[7664](38320/5): Loss 0.0094, Sem 0.0094 LR: 4.001e-02 Score 99.657 Data time: 0.1809, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.9008 [2021-07-08 17:39:02,360][root][INFO] - ===> Epoch[7665](38325/5): Loss 0.0098, Sem 0.0098 LR: 4.000e-02 Score 99.645 Data time: 0.2470, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7209 [2021-07-08 17:39:16,701][root][INFO] - ===> Epoch[7666](38330/5): Loss 0.0089, Sem 0.0089 LR: 3.999e-02 Score 99.676 Data time: 0.1981, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.8291 [2021-07-08 17:39:30,409][root][INFO] - ===> Epoch[7667](38335/5): Loss 0.0095, Sem 0.0095 LR: 3.998e-02 Score 99.657 Data time: 0.2109, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.6965 [2021-07-08 17:39:43,872][root][INFO] - ===> Epoch[7668](38340/5): Loss 0.0095, Sem 0.0095 LR: 3.997e-02 Score 99.656 Data time: 0.2345, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.6422 [2021-07-08 17:39:57,128][root][INFO] - ===> Epoch[7669](38345/5): Loss 0.0097, Sem 0.0097 LR: 3.996e-02 Score 99.648 Data time: 0.1926, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.6136 [2021-07-08 17:40:11,871][root][INFO] - ===> Epoch[7670](38350/5): Loss 0.0096, Sem 0.0096 LR: 3.996e-02 Score 99.654 Data time: 0.2703, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.9043 [2021-07-08 17:40:25,043][root][INFO] - ===> Epoch[7671](38355/5): Loss 0.0097, Sem 0.0097 LR: 3.995e-02 Score 99.648 Data time: 0.2367, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.5822 [2021-07-08 17:40:38,867][root][INFO] - ===> Epoch[7672](38360/5): Loss 0.0094, Sem 0.0094 LR: 3.994e-02 Score 99.657 Data time: 0.1943, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7172 [2021-07-08 17:40:53,004][root][INFO] - ===> Epoch[7673](38365/5): Loss 0.0096, Sem 0.0096 LR: 3.993e-02 Score 99.651 Data time: 0.2021, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7870 [2021-07-08 17:41:07,054][root][INFO] - ===> Epoch[7674](38370/5): Loss 0.0096, Sem 0.0096 LR: 3.992e-02 Score 99.652 Data time: 0.2346, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7525 [2021-07-08 17:41:21,189][root][INFO] - ===> Epoch[7675](38375/5): Loss 0.0099, Sem 0.0099 LR: 3.991e-02 Score 99.640 Data time: 0.2475, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7820 [2021-07-08 17:41:35,569][root][INFO] - ===> Epoch[7676](38380/5): Loss 0.0094, Sem 0.0094 LR: 3.991e-02 Score 99.663 Data time: 0.2240, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.8273 [2021-07-08 17:41:49,364][root][INFO] - ===> Epoch[7677](38385/5): Loss 0.0102, Sem 0.0102 LR: 3.990e-02 Score 99.637 Data time: 0.2457, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7181 [2021-07-08 17:42:02,480][root][INFO] - ===> Epoch[7678](38390/5): Loss 0.0099, Sem 0.0099 LR: 3.989e-02 Score 99.643 Data time: 0.1912, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.5773 [2021-07-08 17:42:16,629][root][INFO] - ===> Epoch[7679](38395/5): Loss 0.0098, Sem 0.0098 LR: 3.988e-02 Score 99.650 Data time: 0.2141, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7754 [2021-07-08 17:42:30,531][root][INFO] - ===> Epoch[7680](38400/5): Loss 0.0095, Sem 0.0095 LR: 3.987e-02 Score 99.657 Data time: 0.1998, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7372 [2021-07-08 17:42:30,532][root][INFO] - ===> Start testing [2021-07-08 17:43:09,206][root][INFO] - 68/68: Data time: 0.0017, Iter time: 0.1342 Loss 0.464 (AVG: 0.585) Score 91.201 (AVG: 91.084) mIOU 71.925 mAP 83.879 mAcc 77.443 IOU: 65.004 0.000 83.281 75.283 95.036 91.683 33.490 87.182 98.495 82.099 80.589 85.778 57.112 mAP: 84.994 4.140 92.417 86.427 98.653 98.486 58.872 98.086 99.766 92.608 91.143 93.870 90.968 mAcc: 81.242 0.000 87.318 83.016 96.213 97.376 36.575 94.876 99.161 83.889 89.629 96.604 60.856 [2021-07-08 17:43:09,206][root][INFO] - Finished test. Elapsed time: 38.6732 [2021-07-08 17:43:10,274][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 17:43:22,842][root][INFO] - ===> Epoch[7681](38405/5): Loss 0.0095, Sem 0.0095 LR: 3.987e-02 Score 99.655 Data time: 0.2008, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.4726 [2021-07-08 17:43:38,282][root][INFO] - ===> Epoch[7682](38410/5): Loss 0.0094, Sem 0.0094 LR: 3.986e-02 Score 99.665 Data time: 0.2452, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 3.0395 [2021-07-08 17:43:52,076][root][INFO] - ===> Epoch[7683](38415/5): Loss 0.0103, Sem 0.0103 LR: 3.985e-02 Score 99.630 Data time: 0.3016, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7181 [2021-07-08 17:44:05,987][root][INFO] - ===> Epoch[7684](38420/5): Loss 0.0097, Sem 0.0097 LR: 3.984e-02 Score 99.651 Data time: 0.2385, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7352 [2021-07-08 17:44:20,507][root][INFO] - ===> Epoch[7685](38425/5): Loss 0.0096, Sem 0.0096 LR: 3.983e-02 Score 99.652 Data time: 0.2170, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.8640 [2021-07-08 17:44:34,837][root][INFO] - ===> Epoch[7686](38430/5): Loss 0.0095, Sem 0.0095 LR: 3.982e-02 Score 99.658 Data time: 0.3319, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.8131 [2021-07-08 17:44:49,151][root][INFO] - ===> Epoch[7687](38435/5): Loss 0.0098, Sem 0.0098 LR: 3.982e-02 Score 99.645 Data time: 0.3583, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7993 [2021-07-08 17:45:03,942][root][INFO] - ===> Epoch[7688](38440/5): Loss 0.0100, Sem 0.0100 LR: 3.981e-02 Score 99.639 Data time: 0.2778, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.9068 [2021-07-08 17:45:18,441][root][INFO] - ===> Epoch[7689](38445/5): Loss 0.0099, Sem 0.0099 LR: 3.980e-02 Score 99.646 Data time: 0.2870, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.8556 [2021-07-08 17:45:32,141][root][INFO] - ===> Epoch[7690](38450/5): Loss 0.0102, Sem 0.0102 LR: 3.979e-02 Score 99.638 Data time: 0.2206, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.6924 [2021-07-08 17:45:45,822][root][INFO] - ===> Epoch[7691](38455/5): Loss 0.0096, Sem 0.0096 LR: 3.978e-02 Score 99.654 Data time: 0.2317, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.6848 [2021-07-08 17:45:59,854][root][INFO] - ===> Epoch[7692](38460/5): Loss 0.0096, Sem 0.0096 LR: 3.977e-02 Score 99.650 Data time: 0.2190, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7679 [2021-07-08 17:46:12,827][root][INFO] - ===> Epoch[7693](38465/5): Loss 0.0098, Sem 0.0098 LR: 3.977e-02 Score 99.648 Data time: 0.1796, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.5577 [2021-07-08 17:46:27,044][root][INFO] - ===> Epoch[7694](38470/5): Loss 0.0099, Sem 0.0099 LR: 3.976e-02 Score 99.645 Data time: 0.2591, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.8044 [2021-07-08 17:46:40,918][root][INFO] - ===> Epoch[7695](38475/5): Loss 0.0092, Sem 0.0092 LR: 3.975e-02 Score 99.667 Data time: 0.2330, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7307 [2021-07-08 17:46:55,203][root][INFO] - ===> Epoch[7696](38480/5): Loss 0.0097, Sem 0.0097 LR: 3.974e-02 Score 99.653 Data time: 0.2019, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.8065 [2021-07-08 17:47:08,406][root][INFO] - ===> Epoch[7697](38485/5): Loss 0.0096, Sem 0.0096 LR: 3.973e-02 Score 99.657 Data time: 0.2165, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.5926 [2021-07-08 17:47:21,609][root][INFO] - ===> Epoch[7698](38490/5): Loss 0.0100, Sem 0.0100 LR: 3.972e-02 Score 99.640 Data time: 0.1989, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6020 [2021-07-08 17:47:35,017][root][INFO] - ===> Epoch[7699](38495/5): Loss 0.0093, Sem 0.0093 LR: 3.972e-02 Score 99.664 Data time: 0.1879, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6406 [2021-07-08 17:47:49,031][root][INFO] - ===> Epoch[7700](38500/5): Loss 0.0093, Sem 0.0093 LR: 3.971e-02 Score 99.663 Data time: 0.2327, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7608 [2021-07-08 17:48:03,093][root][INFO] - ===> Epoch[7701](38505/5): Loss 0.0095, Sem 0.0095 LR: 3.970e-02 Score 99.658 Data time: 0.2388, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7680 [2021-07-08 17:48:17,133][root][INFO] - ===> Epoch[7702](38510/5): Loss 0.0092, Sem 0.0092 LR: 3.969e-02 Score 99.669 Data time: 0.2566, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7606 [2021-07-08 17:48:31,498][root][INFO] - ===> Epoch[7703](38515/5): Loss 0.0099, Sem 0.0099 LR: 3.968e-02 Score 99.644 Data time: 0.2050, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8312 [2021-07-08 17:48:45,132][root][INFO] - ===> Epoch[7704](38520/5): Loss 0.0100, Sem 0.0100 LR: 3.967e-02 Score 99.637 Data time: 0.1660, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.6911 [2021-07-08 17:48:58,966][root][INFO] - ===> Epoch[7705](38525/5): Loss 0.0092, Sem 0.0092 LR: 3.967e-02 Score 99.667 Data time: 0.1823, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.7223 [2021-07-08 17:49:13,674][root][INFO] - ===> Epoch[7706](38530/5): Loss 0.0099, Sem 0.0099 LR: 3.966e-02 Score 99.644 Data time: 0.2279, Forward time: 1.2553, Backward time: 1.2432, DDP time: 0.0042, Total iter time: 2.8889 [2021-07-08 17:49:27,472][root][INFO] - ===> Epoch[7707](38535/5): Loss 0.0094, Sem 0.0094 LR: 3.965e-02 Score 99.657 Data time: 0.2243, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7151 [2021-07-08 17:49:42,369][root][INFO] - ===> Epoch[7708](38540/5): Loss 0.0093, Sem 0.0093 LR: 3.964e-02 Score 99.660 Data time: 0.2188, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.9324 [2021-07-08 17:49:56,306][root][INFO] - ===> Epoch[7709](38545/5): Loss 0.0094, Sem 0.0094 LR: 3.963e-02 Score 99.661 Data time: 0.2025, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7507 [2021-07-08 17:50:09,687][root][INFO] - ===> Epoch[7710](38550/5): Loss 0.0097, Sem 0.0097 LR: 3.962e-02 Score 99.648 Data time: 0.1792, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.6279 [2021-07-08 17:50:23,391][root][INFO] - ===> Epoch[7711](38555/5): Loss 0.0098, Sem 0.0098 LR: 3.962e-02 Score 99.649 Data time: 0.1976, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.6974 [2021-07-08 17:50:37,604][root][INFO] - ===> Epoch[7712](38560/5): Loss 0.0094, Sem 0.0094 LR: 3.961e-02 Score 99.661 Data time: 0.2275, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7983 [2021-07-08 17:50:51,848][root][INFO] - ===> Epoch[7713](38565/5): Loss 0.0094, Sem 0.0094 LR: 3.960e-02 Score 99.658 Data time: 0.2296, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.8061 [2021-07-08 17:51:05,650][root][INFO] - ===> Epoch[7714](38570/5): Loss 0.0096, Sem 0.0096 LR: 3.959e-02 Score 99.650 Data time: 0.2106, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7088 [2021-07-08 17:51:19,784][root][INFO] - ===> Epoch[7715](38575/5): Loss 0.0102, Sem 0.0102 LR: 3.958e-02 Score 99.649 Data time: 0.2271, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7784 [2021-07-08 17:51:33,692][root][INFO] - ===> Epoch[7716](38580/5): Loss 0.0095, Sem 0.0095 LR: 3.957e-02 Score 99.659 Data time: 0.1875, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7315 [2021-07-08 17:51:48,311][root][INFO] - ===> Epoch[7717](38585/5): Loss 0.0097, Sem 0.0097 LR: 3.957e-02 Score 99.650 Data time: 0.2448, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.8766 [2021-07-08 17:52:02,260][root][INFO] - ===> Epoch[7718](38590/5): Loss 0.0096, Sem 0.0096 LR: 3.956e-02 Score 99.654 Data time: 0.1911, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7515 [2021-07-08 17:52:16,302][root][INFO] - ===> Epoch[7719](38595/5): Loss 0.0097, Sem 0.0097 LR: 3.955e-02 Score 99.651 Data time: 0.2026, Forward time: 1.2552, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7542 [2021-07-08 17:52:30,412][root][INFO] - ===> Epoch[7720](38600/5): Loss 0.0091, Sem 0.0091 LR: 3.954e-02 Score 99.673 Data time: 0.2493, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7705 [2021-07-08 17:52:30,413][root][INFO] - ===> Start testing [2021-07-08 17:53:10,087][root][INFO] - 68/68: Data time: 0.0327, Iter time: 0.1309 Loss 0.423 (AVG: 0.613) Score 91.490 (AVG: 90.842) mIOU 71.168 mAP 83.255 mAcc 77.263 IOU: 64.139 0.000 80.699 74.391 94.880 91.502 27.317 85.548 98.573 85.526 80.420 85.878 56.313 mAP: 83.947 4.692 89.753 86.128 98.660 98.552 52.658 98.091 99.782 95.122 91.348 93.487 90.093 mAcc: 82.035 0.000 84.921 81.307 96.117 98.001 28.698 95.496 99.243 92.344 90.102 96.650 59.507 [2021-07-08 17:53:10,087][root][INFO] - Finished test. Elapsed time: 39.6737 [2021-07-08 17:53:11,290][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 17:53:25,484][root][INFO] - ===> Epoch[7721](38605/5): Loss 0.0096, Sem 0.0096 LR: 3.953e-02 Score 99.652 Data time: 0.1967, Forward time: 1.2552, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.8032 [2021-07-08 17:53:39,271][root][INFO] - ===> Epoch[7722](38610/5): Loss 0.0097, Sem 0.0097 LR: 3.952e-02 Score 99.645 Data time: 0.2395, Forward time: 1.2552, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7059 [2021-07-08 17:53:53,202][root][INFO] - ===> Epoch[7723](38615/5): Loss 0.0096, Sem 0.0096 LR: 3.952e-02 Score 99.652 Data time: 0.1693, Forward time: 1.2552, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7390 [2021-07-08 17:54:07,449][root][INFO] - ===> Epoch[7724](38620/5): Loss 0.0096, Sem 0.0096 LR: 3.951e-02 Score 99.655 Data time: 0.1907, Forward time: 1.2552, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.8057 [2021-07-08 17:54:21,211][root][INFO] - ===> Epoch[7725](38625/5): Loss 0.0093, Sem 0.0093 LR: 3.950e-02 Score 99.665 Data time: 0.1916, Forward time: 1.2552, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7092 [2021-07-08 17:54:36,333][root][INFO] - ===> Epoch[7726](38630/5): Loss 0.0095, Sem 0.0095 LR: 3.949e-02 Score 99.660 Data time: 0.3305, Forward time: 1.2552, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.9645 [2021-07-08 17:54:50,615][root][INFO] - ===> Epoch[7727](38635/5): Loss 0.0092, Sem 0.0092 LR: 3.948e-02 Score 99.666 Data time: 0.2702, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.8095 [2021-07-08 17:55:04,346][root][INFO] - ===> Epoch[7728](38640/5): Loss 0.0094, Sem 0.0094 LR: 3.947e-02 Score 99.656 Data time: 0.2302, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.6951 [2021-07-08 17:55:18,537][root][INFO] - ===> Epoch[7729](38645/5): Loss 0.0095, Sem 0.0095 LR: 3.947e-02 Score 99.655 Data time: 0.2353, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7887 [2021-07-08 17:55:32,759][root][INFO] - ===> Epoch[7730](38650/5): Loss 0.0099, Sem 0.0099 LR: 3.946e-02 Score 99.645 Data time: 0.2244, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7973 [2021-07-08 17:55:46,859][root][INFO] - ===> Epoch[7731](38655/5): Loss 0.0095, Sem 0.0095 LR: 3.945e-02 Score 99.662 Data time: 0.2708, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7650 [2021-07-08 17:56:00,598][root][INFO] - ===> Epoch[7732](38660/5): Loss 0.0092, Sem 0.0092 LR: 3.944e-02 Score 99.665 Data time: 0.2589, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7044 [2021-07-08 17:56:14,819][root][INFO] - ===> Epoch[7733](38665/5): Loss 0.0093, Sem 0.0093 LR: 3.943e-02 Score 99.661 Data time: 0.2848, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7942 [2021-07-08 17:56:28,748][root][INFO] - ===> Epoch[7734](38670/5): Loss 0.0096, Sem 0.0096 LR: 3.942e-02 Score 99.653 Data time: 0.2146, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7411 [2021-07-08 17:56:43,569][root][INFO] - ===> Epoch[7735](38675/5): Loss 0.0096, Sem 0.0096 LR: 3.942e-02 Score 99.652 Data time: 0.2025, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.9246 [2021-07-08 17:56:57,640][root][INFO] - ===> Epoch[7736](38680/5): Loss 0.0094, Sem 0.0094 LR: 3.941e-02 Score 99.662 Data time: 0.2299, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7643 [2021-07-08 17:57:11,839][root][INFO] - ===> Epoch[7737](38685/5): Loss 0.0091, Sem 0.0091 LR: 3.940e-02 Score 99.671 Data time: 0.1922, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7995 [2021-07-08 17:57:25,676][root][INFO] - ===> Epoch[7738](38690/5): Loss 0.0091, Sem 0.0091 LR: 3.939e-02 Score 99.669 Data time: 0.1907, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.7302 [2021-07-08 17:57:39,102][root][INFO] - ===> Epoch[7739](38695/5): Loss 0.0099, Sem 0.0099 LR: 3.938e-02 Score 99.641 Data time: 0.2138, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.6357 [2021-07-08 17:57:54,543][root][INFO] - ===> Epoch[7740](38700/5): Loss 0.0093, Sem 0.0093 LR: 3.937e-02 Score 99.663 Data time: 0.1884, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 3.0438 [2021-07-08 17:58:09,370][root][INFO] - ===> Epoch[7741](38705/5): Loss 0.0096, Sem 0.0096 LR: 3.937e-02 Score 99.654 Data time: 0.1633, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.9181 [2021-07-08 17:58:22,885][root][INFO] - ===> Epoch[7742](38710/5): Loss 0.0093, Sem 0.0093 LR: 3.936e-02 Score 99.665 Data time: 0.2375, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.6566 [2021-07-08 17:58:36,273][root][INFO] - ===> Epoch[7743](38715/5): Loss 0.0091, Sem 0.0091 LR: 3.935e-02 Score 99.672 Data time: 0.2115, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.6252 [2021-07-08 17:58:50,967][root][INFO] - ===> Epoch[7744](38720/5): Loss 0.0095, Sem 0.0095 LR: 3.934e-02 Score 99.659 Data time: 0.1839, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.8962 [2021-07-08 17:59:04,643][root][INFO] - ===> Epoch[7745](38725/5): Loss 0.0096, Sem 0.0096 LR: 3.933e-02 Score 99.653 Data time: 0.2620, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.6915 [2021-07-08 17:59:19,474][root][INFO] - ===> Epoch[7746](38730/5): Loss 0.0095, Sem 0.0095 LR: 3.932e-02 Score 99.654 Data time: 0.2564, Forward time: 1.2553, Backward time: 1.2433, DDP time: 0.0042, Total iter time: 2.9112 [2021-07-08 17:59:32,964][root][INFO] - ===> Epoch[7747](38735/5): Loss 0.0095, Sem 0.0095 LR: 3.932e-02 Score 99.657 Data time: 0.1352, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.6717 [2021-07-08 17:59:47,309][root][INFO] - ===> Epoch[7748](38740/5): Loss 0.0094, Sem 0.0094 LR: 3.931e-02 Score 99.661 Data time: 0.2368, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.8293 [2021-07-08 18:00:01,019][root][INFO] - ===> Epoch[7749](38745/5): Loss 0.0099, Sem 0.0099 LR: 3.930e-02 Score 99.641 Data time: 0.2066, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.7054 [2021-07-08 18:00:15,530][root][INFO] - ===> Epoch[7750](38750/5): Loss 0.0092, Sem 0.0092 LR: 3.929e-02 Score 99.668 Data time: 0.2300, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.8653 [2021-07-08 18:00:29,093][root][INFO] - ===> Epoch[7751](38755/5): Loss 0.0092, Sem 0.0092 LR: 3.928e-02 Score 99.664 Data time: 0.2076, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.6712 [2021-07-08 18:00:43,190][root][INFO] - ===> Epoch[7752](38760/5): Loss 0.0095, Sem 0.0095 LR: 3.927e-02 Score 99.653 Data time: 0.1769, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.7752 [2021-07-08 18:00:57,629][root][INFO] - ===> Epoch[7753](38765/5): Loss 0.0092, Sem 0.0092 LR: 3.927e-02 Score 99.670 Data time: 0.2174, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.8417 [2021-07-08 18:01:12,185][root][INFO] - ===> Epoch[7754](38770/5): Loss 0.0096, Sem 0.0096 LR: 3.926e-02 Score 99.655 Data time: 0.1771, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.8724 [2021-07-08 18:01:26,054][root][INFO] - ===> Epoch[7755](38775/5): Loss 0.0095, Sem 0.0095 LR: 3.925e-02 Score 99.658 Data time: 0.2010, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.7311 [2021-07-08 18:01:40,161][root][INFO] - ===> Epoch[7756](38780/5): Loss 0.0095, Sem 0.0095 LR: 3.924e-02 Score 99.661 Data time: 0.2343, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.7780 [2021-07-08 18:01:54,185][root][INFO] - ===> Epoch[7757](38785/5): Loss 0.0090, Sem 0.0090 LR: 3.923e-02 Score 99.676 Data time: 0.1783, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.7608 [2021-07-08 18:02:08,042][root][INFO] - ===> Epoch[7758](38790/5): Loss 0.0094, Sem 0.0094 LR: 3.922e-02 Score 99.659 Data time: 0.1815, Forward time: 1.2553, Backward time: 1.2434, DDP time: 0.0042, Total iter time: 2.7330 [2021-07-08 18:02:22,893][root][INFO] - ===> Epoch[7759](38795/5): Loss 0.0094, Sem 0.0094 LR: 3.922e-02 Score 99.658 Data time: 0.1653, Forward time: 1.2553, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.9296 [2021-07-08 18:02:37,251][root][INFO] - ===> Epoch[7760](38800/5): Loss 0.0095, Sem 0.0095 LR: 3.921e-02 Score 99.659 Data time: 0.2319, Forward time: 1.2553, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.8272 [2021-07-08 18:02:37,252][root][INFO] - ===> Start testing [2021-07-08 18:03:17,795][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1277 Loss 0.352 (AVG: 0.583) Score 92.206 (AVG: 91.134) mIOU 72.136 mAP 83.704 mAcc 77.942 IOU: 64.971 0.000 84.596 75.531 95.154 92.095 30.180 86.212 98.504 85.588 80.308 85.792 58.839 mAP: 84.813 3.678 91.728 86.451 98.624 98.395 55.026 98.019 99.764 95.003 91.350 93.738 91.557 mAcc: 80.281 0.000 87.332 82.569 96.593 97.516 32.405 93.948 99.188 92.630 91.432 96.822 62.526 [2021-07-08 18:03:17,796][root][INFO] - Finished test. Elapsed time: 40.5426 [2021-07-08 18:03:18,844][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 18:03:32,133][root][INFO] - ===> Epoch[7761](38805/5): Loss 0.0095, Sem 0.0095 LR: 3.920e-02 Score 99.660 Data time: 0.1749, Forward time: 1.2553, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.6197 [2021-07-08 18:03:46,336][root][INFO] - ===> Epoch[7762](38810/5): Loss 0.0094, Sem 0.0094 LR: 3.919e-02 Score 99.665 Data time: 0.1970, Forward time: 1.2552, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.8048 [2021-07-08 18:04:00,773][root][INFO] - ===> Epoch[7763](38815/5): Loss 0.0090, Sem 0.0090 LR: 3.918e-02 Score 99.672 Data time: 0.2500, Forward time: 1.2552, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.8408 [2021-07-08 18:04:14,820][root][INFO] - ===> Epoch[7764](38820/5): Loss 0.0091, Sem 0.0091 LR: 3.918e-02 Score 99.669 Data time: 0.2123, Forward time: 1.2552, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.7551 [2021-07-08 18:04:29,368][root][INFO] - ===> Epoch[7765](38825/5): Loss 0.0090, Sem 0.0090 LR: 3.917e-02 Score 99.677 Data time: 0.2246, Forward time: 1.2553, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.8647 [2021-07-08 18:04:43,880][root][INFO] - ===> Epoch[7766](38830/5): Loss 0.0099, Sem 0.0099 LR: 3.916e-02 Score 99.646 Data time: 0.3299, Forward time: 1.2553, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.8609 [2021-07-08 18:04:58,044][root][INFO] - ===> Epoch[7767](38835/5): Loss 0.0090, Sem 0.0090 LR: 3.915e-02 Score 99.671 Data time: 0.2282, Forward time: 1.2553, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.7820 [2021-07-08 18:05:12,065][root][INFO] - ===> Epoch[7768](38840/5): Loss 0.0095, Sem 0.0095 LR: 3.914e-02 Score 99.656 Data time: 0.2148, Forward time: 1.2553, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.7585 [2021-07-08 18:05:26,284][root][INFO] - ===> Epoch[7769](38845/5): Loss 0.0096, Sem 0.0096 LR: 3.913e-02 Score 99.657 Data time: 0.2447, Forward time: 1.2553, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.8002 [2021-07-08 18:05:40,483][root][INFO] - ===> Epoch[7770](38850/5): Loss 0.0098, Sem 0.0098 LR: 3.913e-02 Score 99.649 Data time: 0.2142, Forward time: 1.2553, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.7819 [2021-07-08 18:05:55,099][root][INFO] - ===> Epoch[7771](38855/5): Loss 0.0091, Sem 0.0091 LR: 3.912e-02 Score 99.672 Data time: 0.3153, Forward time: 1.2553, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.8785 [2021-07-08 18:06:10,219][root][INFO] - ===> Epoch[7772](38860/5): Loss 0.0093, Sem 0.0093 LR: 3.911e-02 Score 99.665 Data time: 0.2063, Forward time: 1.2553, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.9782 [2021-07-08 18:06:23,593][root][INFO] - ===> Epoch[7773](38865/5): Loss 0.0092, Sem 0.0092 LR: 3.910e-02 Score 99.670 Data time: 0.2004, Forward time: 1.2553, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.6302 [2021-07-08 18:06:37,495][root][INFO] - ===> Epoch[7774](38870/5): Loss 0.0093, Sem 0.0093 LR: 3.909e-02 Score 99.666 Data time: 0.2467, Forward time: 1.2553, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.7379 [2021-07-08 18:06:51,770][root][INFO] - ===> Epoch[7775](38875/5): Loss 0.0093, Sem 0.0093 LR: 3.908e-02 Score 99.666 Data time: 0.2259, Forward time: 1.2553, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.8070 [2021-07-08 18:07:06,360][root][INFO] - ===> Epoch[7776](38880/5): Loss 0.0089, Sem 0.0089 LR: 3.908e-02 Score 99.678 Data time: 0.2047, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.8567 [2021-07-08 18:07:20,074][root][INFO] - ===> Epoch[7777](38885/5): Loss 0.0095, Sem 0.0095 LR: 3.907e-02 Score 99.658 Data time: 0.2464, Forward time: 1.2553, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.6918 [2021-07-08 18:07:34,549][root][INFO] - ===> Epoch[7778](38890/5): Loss 0.0092, Sem 0.0092 LR: 3.906e-02 Score 99.673 Data time: 0.2456, Forward time: 1.2553, Backward time: 1.2435, DDP time: 0.0042, Total iter time: 2.8389 [2021-07-08 18:07:48,256][root][INFO] - ===> Epoch[7779](38895/5): Loss 0.0091, Sem 0.0091 LR: 3.905e-02 Score 99.671 Data time: 0.1763, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.7008 [2021-07-08 18:08:02,831][root][INFO] - ===> Epoch[7780](38900/5): Loss 0.0092, Sem 0.0092 LR: 3.904e-02 Score 99.667 Data time: 0.2005, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.8791 [2021-07-08 18:08:17,217][root][INFO] - ===> Epoch[7781](38905/5): Loss 0.0089, Sem 0.0089 LR: 3.903e-02 Score 99.677 Data time: 0.2199, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.8257 [2021-07-08 18:08:31,929][root][INFO] - ===> Epoch[7782](38910/5): Loss 0.0092, Sem 0.0092 LR: 3.903e-02 Score 99.669 Data time: 0.1640, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.8995 [2021-07-08 18:08:46,020][root][INFO] - ===> Epoch[7783](38915/5): Loss 0.0091, Sem 0.0091 LR: 3.902e-02 Score 99.667 Data time: 0.1703, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.7697 [2021-07-08 18:09:00,463][root][INFO] - ===> Epoch[7784](38920/5): Loss 0.0092, Sem 0.0092 LR: 3.901e-02 Score 99.666 Data time: 0.2458, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.8416 [2021-07-08 18:09:13,642][root][INFO] - ===> Epoch[7785](38925/5): Loss 0.0092, Sem 0.0092 LR: 3.900e-02 Score 99.667 Data time: 0.2387, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.5735 [2021-07-08 18:09:27,390][root][INFO] - ===> Epoch[7786](38930/5): Loss 0.0094, Sem 0.0094 LR: 3.899e-02 Score 99.658 Data time: 0.2239, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.7062 [2021-07-08 18:09:41,255][root][INFO] - ===> Epoch[7787](38935/5): Loss 0.0093, Sem 0.0093 LR: 3.898e-02 Score 99.665 Data time: 0.1756, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.7285 [2021-07-08 18:09:55,308][root][INFO] - ===> Epoch[7788](38940/5): Loss 0.0092, Sem 0.0092 LR: 3.898e-02 Score 99.667 Data time: 0.1860, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.7730 [2021-07-08 18:10:08,918][root][INFO] - ===> Epoch[7789](38945/5): Loss 0.0093, Sem 0.0093 LR: 3.897e-02 Score 99.664 Data time: 0.2129, Forward time: 1.2552, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.6820 [2021-07-08 18:10:22,687][root][INFO] - ===> Epoch[7790](38950/5): Loss 0.0090, Sem 0.0090 LR: 3.896e-02 Score 99.671 Data time: 0.2347, Forward time: 1.2552, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.7073 [2021-07-08 18:10:36,808][root][INFO] - ===> Epoch[7791](38955/5): Loss 0.0091, Sem 0.0091 LR: 3.895e-02 Score 99.671 Data time: 0.2075, Forward time: 1.2552, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.7780 [2021-07-08 18:10:50,918][root][INFO] - ===> Epoch[7792](38960/5): Loss 0.0090, Sem 0.0090 LR: 3.894e-02 Score 99.671 Data time: 0.1967, Forward time: 1.2552, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.7857 [2021-07-08 18:11:05,268][root][INFO] - ===> Epoch[7793](38965/5): Loss 0.0094, Sem 0.0094 LR: 3.893e-02 Score 99.662 Data time: 0.2168, Forward time: 1.2552, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.8243 [2021-07-08 18:11:18,653][root][INFO] - ===> Epoch[7794](38970/5): Loss 0.0095, Sem 0.0095 LR: 3.893e-02 Score 99.657 Data time: 0.1825, Forward time: 1.2552, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.6340 [2021-07-08 18:11:33,290][root][INFO] - ===> Epoch[7795](38975/5): Loss 0.0091, Sem 0.0091 LR: 3.892e-02 Score 99.670 Data time: 0.1668, Forward time: 1.2552, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.8792 [2021-07-08 18:11:47,984][root][INFO] - ===> Epoch[7796](38980/5): Loss 0.0090, Sem 0.0090 LR: 3.891e-02 Score 99.674 Data time: 0.2128, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.9028 [2021-07-08 18:12:02,212][root][INFO] - ===> Epoch[7797](38985/5): Loss 0.0089, Sem 0.0089 LR: 3.890e-02 Score 99.681 Data time: 0.2107, Forward time: 1.2553, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.8067 [2021-07-08 18:12:16,069][root][INFO] - ===> Epoch[7798](38990/5): Loss 0.0094, Sem 0.0094 LR: 3.889e-02 Score 99.661 Data time: 0.2398, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.7284 [2021-07-08 18:12:30,916][root][INFO] - ===> Epoch[7799](38995/5): Loss 0.0090, Sem 0.0090 LR: 3.888e-02 Score 99.681 Data time: 0.2390, Forward time: 1.2553, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.9194 [2021-07-08 18:12:45,005][root][INFO] - ===> Epoch[7800](39000/5): Loss 0.0089, Sem 0.0089 LR: 3.888e-02 Score 99.678 Data time: 0.2184, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.7671 [2021-07-08 18:12:45,006][root][INFO] - ===> Start testing [2021-07-08 18:13:23,928][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1241 Loss 0.507 (AVG: 0.582) Score 90.388 (AVG: 91.055) mIOU 71.591 mAP 83.718 mAcc 77.376 IOU: 64.727 0.000 80.838 75.230 95.135 92.867 28.986 85.072 98.676 85.638 80.289 86.042 57.187 mAP: 84.720 4.008 91.328 86.960 98.756 98.296 55.706 98.409 99.750 94.044 91.306 93.846 91.200 mAcc: 81.350 0.000 85.102 82.279 96.509 97.324 31.062 96.486 99.494 89.187 90.361 96.540 60.192 [2021-07-08 18:13:23,928][root][INFO] - Finished test. Elapsed time: 38.9210 [2021-07-08 18:13:25,016][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 18:13:38,313][root][INFO] - ===> Epoch[7801](39005/5): Loss 0.0093, Sem 0.0093 LR: 3.887e-02 Score 99.666 Data time: 0.2460, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.6240 [2021-07-08 18:13:52,158][root][INFO] - ===> Epoch[7802](39010/5): Loss 0.0097, Sem 0.0097 LR: 3.886e-02 Score 99.653 Data time: 0.2391, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.7271 [2021-07-08 18:14:06,449][root][INFO] - ===> Epoch[7803](39015/5): Loss 0.0091, Sem 0.0091 LR: 3.885e-02 Score 99.671 Data time: 0.2427, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.8056 [2021-07-08 18:14:20,036][root][INFO] - ===> Epoch[7804](39020/5): Loss 0.0096, Sem 0.0096 LR: 3.884e-02 Score 99.655 Data time: 0.2266, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.6753 [2021-07-08 18:14:35,054][root][INFO] - ===> Epoch[7805](39025/5): Loss 0.0095, Sem 0.0095 LR: 3.883e-02 Score 99.658 Data time: 0.2676, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.9414 [2021-07-08 18:14:47,841][root][INFO] - ===> Epoch[7806](39030/5): Loss 0.0100, Sem 0.0100 LR: 3.883e-02 Score 99.638 Data time: 0.2404, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.5044 [2021-07-08 18:15:01,439][root][INFO] - ===> Epoch[7807](39035/5): Loss 0.0091, Sem 0.0091 LR: 3.882e-02 Score 99.670 Data time: 0.1869, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.6746 [2021-07-08 18:15:15,747][root][INFO] - ===> Epoch[7808](39040/5): Loss 0.0093, Sem 0.0093 LR: 3.881e-02 Score 99.663 Data time: 0.2912, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.8174 [2021-07-08 18:15:30,077][root][INFO] - ===> Epoch[7809](39045/5): Loss 0.0093, Sem 0.0093 LR: 3.880e-02 Score 99.664 Data time: 0.2686, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.8140 [2021-07-08 18:15:44,543][root][INFO] - ===> Epoch[7810](39050/5): Loss 0.0093, Sem 0.0093 LR: 3.879e-02 Score 99.664 Data time: 0.2030, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.8541 [2021-07-08 18:15:58,042][root][INFO] - ===> Epoch[7811](39055/5): Loss 0.0093, Sem 0.0093 LR: 3.878e-02 Score 99.665 Data time: 0.2220, Forward time: 1.2552, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.6624 [2021-07-08 18:16:12,028][root][INFO] - ===> Epoch[7812](39060/5): Loss 0.0090, Sem 0.0090 LR: 3.878e-02 Score 99.674 Data time: 0.2009, Forward time: 1.2552, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.7541 [2021-07-08 18:16:25,578][root][INFO] - ===> Epoch[7813](39065/5): Loss 0.0091, Sem 0.0091 LR: 3.877e-02 Score 99.673 Data time: 0.2440, Forward time: 1.2552, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.6682 [2021-07-08 18:16:39,022][root][INFO] - ===> Epoch[7814](39070/5): Loss 0.0094, Sem 0.0094 LR: 3.876e-02 Score 99.662 Data time: 0.2194, Forward time: 1.2552, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.6309 [2021-07-08 18:16:53,760][root][INFO] - ===> Epoch[7815](39075/5): Loss 0.0094, Sem 0.0094 LR: 3.875e-02 Score 99.664 Data time: 0.2397, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.8960 [2021-07-08 18:17:08,843][root][INFO] - ===> Epoch[7816](39080/5): Loss 0.0092, Sem 0.0092 LR: 3.874e-02 Score 99.668 Data time: 0.1699, Forward time: 1.2552, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.9835 [2021-07-08 18:17:22,471][root][INFO] - ===> Epoch[7817](39085/5): Loss 0.0090, Sem 0.0090 LR: 3.873e-02 Score 99.676 Data time: 0.2179, Forward time: 1.2552, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.6778 [2021-07-08 18:17:36,642][root][INFO] - ===> Epoch[7818](39090/5): Loss 0.0093, Sem 0.0093 LR: 3.873e-02 Score 99.664 Data time: 0.1951, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.7838 [2021-07-08 18:17:50,590][root][INFO] - ===> Epoch[7819](39095/5): Loss 0.0091, Sem 0.0091 LR: 3.872e-02 Score 99.672 Data time: 0.2916, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.7400 [2021-07-08 18:18:04,568][root][INFO] - ===> Epoch[7820](39100/5): Loss 0.0091, Sem 0.0091 LR: 3.871e-02 Score 99.672 Data time: 0.2697, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.7545 [2021-07-08 18:18:18,654][root][INFO] - ===> Epoch[7821](39105/5): Loss 0.0088, Sem 0.0088 LR: 3.870e-02 Score 99.679 Data time: 0.2066, Forward time: 1.2552, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.7695 [2021-07-08 18:18:32,928][root][INFO] - ===> Epoch[7822](39110/5): Loss 0.0090, Sem 0.0090 LR: 3.869e-02 Score 99.674 Data time: 0.2627, Forward time: 1.2553, Backward time: 1.2436, DDP time: 0.0042, Total iter time: 2.8032 [2021-07-08 18:18:46,881][root][INFO] - ===> Epoch[7823](39115/5): Loss 0.0103, Sem 0.0103 LR: 3.868e-02 Score 99.639 Data time: 0.1878, Forward time: 1.2552, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.7528 [2021-07-08 18:19:00,754][root][INFO] - ===> Epoch[7824](39120/5): Loss 0.0092, Sem 0.0092 LR: 3.868e-02 Score 99.665 Data time: 0.2107, Forward time: 1.2552, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.7391 [2021-07-08 18:19:14,456][root][INFO] - ===> Epoch[7825](39125/5): Loss 0.0093, Sem 0.0093 LR: 3.867e-02 Score 99.665 Data time: 0.1840, Forward time: 1.2552, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.6997 [2021-07-08 18:19:28,335][root][INFO] - ===> Epoch[7826](39130/5): Loss 0.0095, Sem 0.0095 LR: 3.866e-02 Score 99.660 Data time: 0.2112, Forward time: 1.2552, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.7327 [2021-07-08 18:19:42,396][root][INFO] - ===> Epoch[7827](39135/5): Loss 0.0096, Sem 0.0096 LR: 3.865e-02 Score 99.650 Data time: 0.3053, Forward time: 1.2552, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.7637 [2021-07-08 18:19:56,028][root][INFO] - ===> Epoch[7828](39140/5): Loss 0.0089, Sem 0.0089 LR: 3.864e-02 Score 99.676 Data time: 0.2307, Forward time: 1.2552, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.6785 [2021-07-08 18:20:09,970][root][INFO] - ===> Epoch[7829](39145/5): Loss 0.0093, Sem 0.0093 LR: 3.863e-02 Score 99.664 Data time: 0.2376, Forward time: 1.2552, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.7451 [2021-07-08 18:20:23,982][root][INFO] - ===> Epoch[7830](39150/5): Loss 0.0089, Sem 0.0089 LR: 3.863e-02 Score 99.677 Data time: 0.2361, Forward time: 1.2552, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.7506 [2021-07-08 18:20:38,483][root][INFO] - ===> Epoch[7831](39155/5): Loss 0.0090, Sem 0.0090 LR: 3.862e-02 Score 99.676 Data time: 0.2667, Forward time: 1.2552, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.8536 [2021-07-08 18:20:53,293][root][INFO] - ===> Epoch[7832](39160/5): Loss 0.0090, Sem 0.0090 LR: 3.861e-02 Score 99.674 Data time: 0.1526, Forward time: 1.2552, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.9232 [2021-07-08 18:21:07,396][root][INFO] - ===> Epoch[7833](39165/5): Loss 0.0094, Sem 0.0094 LR: 3.860e-02 Score 99.660 Data time: 0.2278, Forward time: 1.2551, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.7745 [2021-07-08 18:21:21,235][root][INFO] - ===> Epoch[7834](39170/5): Loss 0.0088, Sem 0.0088 LR: 3.859e-02 Score 99.683 Data time: 0.1677, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.7294 [2021-07-08 18:21:34,560][root][INFO] - ===> Epoch[7835](39175/5): Loss 0.0097, Sem 0.0097 LR: 3.858e-02 Score 99.651 Data time: 0.2118, Forward time: 1.2551, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.6243 [2021-07-08 18:21:48,178][root][INFO] - ===> Epoch[7836](39180/5): Loss 0.0094, Sem 0.0094 LR: 3.858e-02 Score 99.659 Data time: 0.2574, Forward time: 1.2551, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.6765 [2021-07-08 18:22:03,087][root][INFO] - ===> Epoch[7837](39185/5): Loss 0.0093, Sem 0.0093 LR: 3.857e-02 Score 99.666 Data time: 0.2357, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.9400 [2021-07-08 18:22:16,488][root][INFO] - ===> Epoch[7838](39190/5): Loss 0.0098, Sem 0.0098 LR: 3.856e-02 Score 99.649 Data time: 0.2668, Forward time: 1.2551, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.6266 [2021-07-08 18:22:30,679][root][INFO] - ===> Epoch[7839](39195/5): Loss 0.0093, Sem 0.0093 LR: 3.855e-02 Score 99.666 Data time: 0.2138, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.7903 [2021-07-08 18:22:44,500][root][INFO] - ===> Epoch[7840](39200/5): Loss 0.0090, Sem 0.0090 LR: 3.854e-02 Score 99.677 Data time: 0.1966, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.7138 [2021-07-08 18:22:44,501][root][INFO] - ===> Start testing [2021-07-08 18:23:26,687][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1269 Loss 0.433 (AVG: 0.596) Score 90.940 (AVG: 91.008) mIOU 71.569 mAP 83.336 mAcc 77.365 IOU: 64.946 0.000 82.491 75.015 95.255 92.064 25.081 87.808 98.588 86.365 80.115 85.642 57.028 mAP: 84.343 3.108 92.973 86.630 98.736 98.475 51.905 98.488 99.714 94.519 91.088 92.981 90.411 mAcc: 80.650 0.000 88.225 82.225 96.629 97.349 26.442 95.605 99.410 91.298 90.817 96.769 60.318 [2021-07-08 18:23:26,688][root][INFO] - Finished test. Elapsed time: 42.1859 [2021-07-08 18:23:27,789][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 18:23:41,487][root][INFO] - ===> Epoch[7841](39205/5): Loss 0.0091, Sem 0.0091 LR: 3.853e-02 Score 99.675 Data time: 0.2146, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.6873 [2021-07-08 18:23:55,037][root][INFO] - ===> Epoch[7842](39210/5): Loss 0.0096, Sem 0.0096 LR: 3.853e-02 Score 99.654 Data time: 0.2482, Forward time: 1.2551, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.6726 [2021-07-08 18:24:09,377][root][INFO] - ===> Epoch[7843](39215/5): Loss 0.0092, Sem 0.0092 LR: 3.852e-02 Score 99.672 Data time: 0.2185, Forward time: 1.2551, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.8176 [2021-07-08 18:24:24,448][root][INFO] - ===> Epoch[7844](39220/5): Loss 0.0091, Sem 0.0091 LR: 3.851e-02 Score 99.674 Data time: 0.1799, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.9682 [2021-07-08 18:24:38,954][root][INFO] - ===> Epoch[7845](39225/5): Loss 0.0090, Sem 0.0090 LR: 3.850e-02 Score 99.677 Data time: 0.2703, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.8553 [2021-07-08 18:24:53,065][root][INFO] - ===> Epoch[7846](39230/5): Loss 0.0092, Sem 0.0092 LR: 3.849e-02 Score 99.667 Data time: 0.2607, Forward time: 1.2551, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.7677 [2021-07-08 18:25:07,166][root][INFO] - ===> Epoch[7847](39235/5): Loss 0.0088, Sem 0.0088 LR: 3.848e-02 Score 99.684 Data time: 0.1947, Forward time: 1.2552, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.7797 [2021-07-08 18:25:21,832][root][INFO] - ===> Epoch[7848](39240/5): Loss 0.0087, Sem 0.0087 LR: 3.848e-02 Score 99.691 Data time: 0.2347, Forward time: 1.2552, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.8740 [2021-07-08 18:25:36,783][root][INFO] - ===> Epoch[7849](39245/5): Loss 0.0097, Sem 0.0097 LR: 3.847e-02 Score 99.655 Data time: 0.2408, Forward time: 1.2552, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.9398 [2021-07-08 18:25:50,624][root][INFO] - ===> Epoch[7850](39250/5): Loss 0.0090, Sem 0.0090 LR: 3.846e-02 Score 99.673 Data time: 0.2573, Forward time: 1.2552, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.7254 [2021-07-08 18:26:04,326][root][INFO] - ===> Epoch[7851](39255/5): Loss 0.0094, Sem 0.0094 LR: 3.845e-02 Score 99.662 Data time: 0.2339, Forward time: 1.2552, Backward time: 1.2437, DDP time: 0.0042, Total iter time: 2.6996 [2021-07-08 18:26:18,752][root][INFO] - ===> Epoch[7852](39260/5): Loss 0.0089, Sem 0.0089 LR: 3.844e-02 Score 99.678 Data time: 0.1855, Forward time: 1.2552, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.8490 [2021-07-08 18:26:32,367][root][INFO] - ===> Epoch[7853](39265/5): Loss 0.0092, Sem 0.0092 LR: 3.843e-02 Score 99.670 Data time: 0.2353, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.6707 [2021-07-08 18:26:47,043][root][INFO] - ===> Epoch[7854](39270/5): Loss 0.0088, Sem 0.0088 LR: 3.843e-02 Score 99.683 Data time: 0.1875, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.8908 [2021-07-08 18:27:00,086][root][INFO] - ===> Epoch[7855](39275/5): Loss 0.0094, Sem 0.0094 LR: 3.842e-02 Score 99.661 Data time: 0.2238, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.5673 [2021-07-08 18:27:14,459][root][INFO] - ===> Epoch[7856](39280/5): Loss 0.0090, Sem 0.0090 LR: 3.841e-02 Score 99.672 Data time: 0.2277, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.8308 [2021-07-08 18:27:27,708][root][INFO] - ===> Epoch[7857](39285/5): Loss 0.0090, Sem 0.0090 LR: 3.840e-02 Score 99.674 Data time: 0.2309, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.6034 [2021-07-08 18:27:41,798][root][INFO] - ===> Epoch[7858](39290/5): Loss 0.0087, Sem 0.0087 LR: 3.839e-02 Score 99.686 Data time: 0.1872, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.7770 [2021-07-08 18:27:55,864][root][INFO] - ===> Epoch[7859](39295/5): Loss 0.0090, Sem 0.0090 LR: 3.838e-02 Score 99.677 Data time: 0.2295, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.7654 [2021-07-08 18:28:09,926][root][INFO] - ===> Epoch[7860](39300/5): Loss 0.0090, Sem 0.0090 LR: 3.838e-02 Score 99.674 Data time: 0.2502, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.7650 [2021-07-08 18:28:24,668][root][INFO] - ===> Epoch[7861](39305/5): Loss 0.0091, Sem 0.0091 LR: 3.837e-02 Score 99.671 Data time: 0.1988, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.8996 [2021-07-08 18:28:39,099][root][INFO] - ===> Epoch[7862](39310/5): Loss 0.0088, Sem 0.0088 LR: 3.836e-02 Score 99.680 Data time: 0.2300, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.8418 [2021-07-08 18:28:54,367][root][INFO] - ===> Epoch[7863](39315/5): Loss 0.0086, Sem 0.0086 LR: 3.835e-02 Score 99.692 Data time: 0.2646, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.9943 [2021-07-08 18:29:08,161][root][INFO] - ===> Epoch[7864](39320/5): Loss 0.0097, Sem 0.0097 LR: 3.834e-02 Score 99.649 Data time: 0.2710, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.7138 [2021-07-08 18:29:21,955][root][INFO] - ===> Epoch[7865](39325/5): Loss 0.0098, Sem 0.0098 LR: 3.833e-02 Score 99.648 Data time: 0.2273, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.7098 [2021-07-08 18:29:35,561][root][INFO] - ===> Epoch[7866](39330/5): Loss 0.0094, Sem 0.0094 LR: 3.833e-02 Score 99.663 Data time: 0.2777, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.6841 [2021-07-08 18:29:49,951][root][INFO] - ===> Epoch[7867](39335/5): Loss 0.0093, Sem 0.0093 LR: 3.832e-02 Score 99.660 Data time: 0.1975, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.8370 [2021-07-08 18:30:04,621][root][INFO] - ===> Epoch[7868](39340/5): Loss 0.0087, Sem 0.0087 LR: 3.831e-02 Score 99.687 Data time: 0.2242, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.8908 [2021-07-08 18:30:18,892][root][INFO] - ===> Epoch[7869](39345/5): Loss 0.0094, Sem 0.0094 LR: 3.830e-02 Score 99.662 Data time: 0.2700, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.7974 [2021-07-08 18:30:32,870][root][INFO] - ===> Epoch[7870](39350/5): Loss 0.0091, Sem 0.0091 LR: 3.829e-02 Score 99.674 Data time: 0.2031, Forward time: 1.2551, Backward time: 1.2438, DDP time: 0.0042, Total iter time: 2.7576 [2021-07-08 18:30:47,509][root][INFO] - ===> Epoch[7871](39355/5): Loss 0.0087, Sem 0.0087 LR: 3.828e-02 Score 99.684 Data time: 0.2374, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.8735 [2021-07-08 18:31:01,649][root][INFO] - ===> Epoch[7872](39360/5): Loss 0.0088, Sem 0.0088 LR: 3.827e-02 Score 99.682 Data time: 0.1708, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7851 [2021-07-08 18:31:16,094][root][INFO] - ===> Epoch[7873](39365/5): Loss 0.0099, Sem 0.0099 LR: 3.827e-02 Score 99.644 Data time: 0.1865, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.8423 [2021-07-08 18:31:30,106][root][INFO] - ===> Epoch[7874](39370/5): Loss 0.0101, Sem 0.0101 LR: 3.826e-02 Score 99.652 Data time: 0.2029, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7617 [2021-07-08 18:31:44,008][root][INFO] - ===> Epoch[7875](39375/5): Loss 0.0104, Sem 0.0104 LR: 3.825e-02 Score 99.629 Data time: 0.2414, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7259 [2021-07-08 18:31:58,082][root][INFO] - ===> Epoch[7876](39380/5): Loss 0.0095, Sem 0.0095 LR: 3.824e-02 Score 99.659 Data time: 0.2075, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7739 [2021-07-08 18:32:12,248][root][INFO] - ===> Epoch[7877](39385/5): Loss 0.0092, Sem 0.0092 LR: 3.823e-02 Score 99.669 Data time: 0.2216, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7794 [2021-07-08 18:32:27,011][root][INFO] - ===> Epoch[7878](39390/5): Loss 0.0096, Sem 0.0096 LR: 3.822e-02 Score 99.654 Data time: 0.1945, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.9110 [2021-07-08 18:32:41,475][root][INFO] - ===> Epoch[7879](39395/5): Loss 0.0091, Sem 0.0091 LR: 3.822e-02 Score 99.668 Data time: 0.2095, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.8455 [2021-07-08 18:32:54,779][root][INFO] - ===> Epoch[7880](39400/5): Loss 0.0096, Sem 0.0096 LR: 3.821e-02 Score 99.655 Data time: 0.2156, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.6196 [2021-07-08 18:32:54,781][root][INFO] - ===> Start testing [2021-07-08 18:33:40,195][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1247 Loss 0.363 (AVG: 0.579) Score 91.997 (AVG: 91.154) mIOU 71.666 mAP 83.686 mAcc 77.758 IOU: 65.508 0.000 82.483 75.864 95.243 91.594 27.240 85.386 98.609 85.467 81.467 85.879 56.920 mAP: 84.402 3.553 93.384 87.929 98.693 98.318 53.104 98.540 99.715 94.784 91.809 93.486 90.207 mAcc: 79.622 0.000 89.397 84.064 96.685 96.579 29.754 95.462 99.423 91.150 91.457 96.551 60.715 [2021-07-08 18:33:40,195][root][INFO] - Finished test. Elapsed time: 45.4137 [2021-07-08 18:33:41,250][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 18:33:54,956][root][INFO] - ===> Epoch[7881](39405/5): Loss 0.0088, Sem 0.0088 LR: 3.820e-02 Score 99.681 Data time: 0.2052, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7037 [2021-07-08 18:34:09,066][root][INFO] - ===> Epoch[7882](39410/5): Loss 0.0094, Sem 0.0094 LR: 3.819e-02 Score 99.663 Data time: 0.1961, Forward time: 1.2551, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7836 [2021-07-08 18:34:23,870][root][INFO] - ===> Epoch[7883](39415/5): Loss 0.0094, Sem 0.0094 LR: 3.818e-02 Score 99.659 Data time: 0.2550, Forward time: 1.2551, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.9153 [2021-07-08 18:34:38,563][root][INFO] - ===> Epoch[7884](39420/5): Loss 0.0090, Sem 0.0090 LR: 3.817e-02 Score 99.675 Data time: 0.2497, Forward time: 1.2551, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.8872 [2021-07-08 18:34:52,705][root][INFO] - ===> Epoch[7885](39425/5): Loss 0.0093, Sem 0.0093 LR: 3.817e-02 Score 99.665 Data time: 0.2487, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7775 [2021-07-08 18:35:07,017][root][INFO] - ===> Epoch[7886](39430/5): Loss 0.0091, Sem 0.0091 LR: 3.816e-02 Score 99.674 Data time: 0.2543, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.8250 [2021-07-08 18:35:20,261][root][INFO] - ===> Epoch[7887](39435/5): Loss 0.0092, Sem 0.0092 LR: 3.815e-02 Score 99.668 Data time: 0.2468, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.6125 [2021-07-08 18:35:34,379][root][INFO] - ===> Epoch[7888](39440/5): Loss 0.0093, Sem 0.0093 LR: 3.814e-02 Score 99.663 Data time: 0.2568, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7707 [2021-07-08 18:35:48,079][root][INFO] - ===> Epoch[7889](39445/5): Loss 0.0089, Sem 0.0089 LR: 3.813e-02 Score 99.680 Data time: 0.2896, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7014 [2021-07-08 18:36:01,761][root][INFO] - ===> Epoch[7890](39450/5): Loss 0.0090, Sem 0.0090 LR: 3.812e-02 Score 99.673 Data time: 0.2790, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.6787 [2021-07-08 18:36:15,814][root][INFO] - ===> Epoch[7891](39455/5): Loss 0.0090, Sem 0.0090 LR: 3.812e-02 Score 99.676 Data time: 0.1839, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7720 [2021-07-08 18:36:29,645][root][INFO] - ===> Epoch[7892](39460/5): Loss 0.0091, Sem 0.0091 LR: 3.811e-02 Score 99.672 Data time: 0.2790, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7146 [2021-07-08 18:36:43,574][root][INFO] - ===> Epoch[7893](39465/5): Loss 0.0091, Sem 0.0091 LR: 3.810e-02 Score 99.671 Data time: 0.1662, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7518 [2021-07-08 18:36:57,202][root][INFO] - ===> Epoch[7894](39470/5): Loss 0.0092, Sem 0.0092 LR: 3.809e-02 Score 99.669 Data time: 0.1707, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.6900 [2021-07-08 18:37:11,073][root][INFO] - ===> Epoch[7895](39475/5): Loss 0.0088, Sem 0.0088 LR: 3.808e-02 Score 99.682 Data time: 0.2023, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7293 [2021-07-08 18:37:24,591][root][INFO] - ===> Epoch[7896](39480/5): Loss 0.0092, Sem 0.0092 LR: 3.807e-02 Score 99.670 Data time: 0.2211, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.6574 [2021-07-08 18:37:39,233][root][INFO] - ===> Epoch[7897](39485/5): Loss 0.0092, Sem 0.0092 LR: 3.807e-02 Score 99.671 Data time: 0.2470, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.8851 [2021-07-08 18:37:52,473][root][INFO] - ===> Epoch[7898](39490/5): Loss 0.0094, Sem 0.0094 LR: 3.806e-02 Score 99.658 Data time: 0.1974, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.5930 [2021-07-08 18:38:06,216][root][INFO] - ===> Epoch[7899](39495/5): Loss 0.0091, Sem 0.0091 LR: 3.805e-02 Score 99.673 Data time: 0.1812, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7042 [2021-07-08 18:38:21,593][root][INFO] - ===> Epoch[7900](39500/5): Loss 0.0089, Sem 0.0089 LR: 3.804e-02 Score 99.681 Data time: 0.2238, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 3.0338 [2021-07-08 18:38:35,743][root][INFO] - ===> Epoch[7901](39505/5): Loss 0.0093, Sem 0.0093 LR: 3.803e-02 Score 99.664 Data time: 0.2098, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7899 [2021-07-08 18:38:50,027][root][INFO] - ===> Epoch[7902](39510/5): Loss 0.0091, Sem 0.0091 LR: 3.802e-02 Score 99.673 Data time: 0.2274, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.8119 [2021-07-08 18:39:03,857][root][INFO] - ===> Epoch[7903](39515/5): Loss 0.0093, Sem 0.0093 LR: 3.802e-02 Score 99.665 Data time: 0.2082, Forward time: 1.2551, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7191 [2021-07-08 18:39:18,313][root][INFO] - ===> Epoch[7904](39520/5): Loss 0.0092, Sem 0.0092 LR: 3.801e-02 Score 99.668 Data time: 0.1869, Forward time: 1.2551, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.8574 [2021-07-08 18:39:31,851][root][INFO] - ===> Epoch[7905](39525/5): Loss 0.0092, Sem 0.0092 LR: 3.800e-02 Score 99.669 Data time: 0.2492, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.6566 [2021-07-08 18:39:45,805][root][INFO] - ===> Epoch[7906](39530/5): Loss 0.0090, Sem 0.0090 LR: 3.799e-02 Score 99.681 Data time: 0.2053, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7510 [2021-07-08 18:39:59,611][root][INFO] - ===> Epoch[7907](39535/5): Loss 0.0093, Sem 0.0093 LR: 3.798e-02 Score 99.667 Data time: 0.1724, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7136 [2021-07-08 18:40:13,645][root][INFO] - ===> Epoch[7908](39540/5): Loss 0.0088, Sem 0.0088 LR: 3.797e-02 Score 99.690 Data time: 0.2074, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7493 [2021-07-08 18:40:28,404][root][INFO] - ===> Epoch[7909](39545/5): Loss 0.0095, Sem 0.0095 LR: 3.797e-02 Score 99.656 Data time: 0.2173, Forward time: 1.2551, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.9054 [2021-07-08 18:40:42,446][root][INFO] - ===> Epoch[7910](39550/5): Loss 0.0091, Sem 0.0091 LR: 3.796e-02 Score 99.671 Data time: 0.1853, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7660 [2021-07-08 18:40:55,381][root][INFO] - ===> Epoch[7911](39555/5): Loss 0.0093, Sem 0.0093 LR: 3.795e-02 Score 99.667 Data time: 0.2473, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.5404 [2021-07-08 18:41:09,393][root][INFO] - ===> Epoch[7912](39560/5): Loss 0.0092, Sem 0.0092 LR: 3.794e-02 Score 99.666 Data time: 0.1881, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7579 [2021-07-08 18:41:22,687][root][INFO] - ===> Epoch[7913](39565/5): Loss 0.0088, Sem 0.0088 LR: 3.793e-02 Score 99.679 Data time: 0.1972, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.5954 [2021-07-08 18:41:36,677][root][INFO] - ===> Epoch[7914](39570/5): Loss 0.0088, Sem 0.0088 LR: 3.792e-02 Score 99.683 Data time: 0.1865, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7547 [2021-07-08 18:41:50,495][root][INFO] - ===> Epoch[7915](39575/5): Loss 0.0096, Sem 0.0096 LR: 3.792e-02 Score 99.656 Data time: 0.2408, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7213 [2021-07-08 18:42:04,144][root][INFO] - ===> Epoch[7916](39580/5): Loss 0.0093, Sem 0.0093 LR: 3.791e-02 Score 99.669 Data time: 0.2445, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.6831 [2021-07-08 18:42:17,906][root][INFO] - ===> Epoch[7917](39585/5): Loss 0.0090, Sem 0.0090 LR: 3.790e-02 Score 99.673 Data time: 0.2043, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7136 [2021-07-08 18:42:31,008][root][INFO] - ===> Epoch[7918](39590/5): Loss 0.0094, Sem 0.0094 LR: 3.789e-02 Score 99.660 Data time: 0.1986, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.5792 [2021-07-08 18:42:45,405][root][INFO] - ===> Epoch[7919](39595/5): Loss 0.0088, Sem 0.0088 LR: 3.788e-02 Score 99.685 Data time: 0.1869, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.8406 [2021-07-08 18:42:59,813][root][INFO] - ===> Epoch[7920](39600/5): Loss 0.0089, Sem 0.0089 LR: 3.787e-02 Score 99.681 Data time: 0.2563, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.8295 [2021-07-08 18:42:59,814][root][INFO] - ===> Start testing [2021-07-08 18:43:42,736][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1267 Loss 0.445 (AVG: 0.597) Score 90.934 (AVG: 91.043) mIOU 71.583 mAP 83.708 mAcc 77.533 IOU: 64.511 0.000 83.166 75.615 95.372 92.002 26.392 86.466 98.648 84.870 79.521 85.990 58.028 mAP: 84.274 4.228 92.318 86.998 98.548 98.489 55.940 98.482 99.684 94.196 90.959 93.775 90.315 mAcc: 80.088 0.000 88.765 82.914 96.765 97.348 28.728 96.210 99.472 88.822 90.426 96.304 62.094 [2021-07-08 18:43:42,736][root][INFO] - Finished test. Elapsed time: 42.9213 [2021-07-08 18:43:43,773][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 18:43:56,869][root][INFO] - ===> Epoch[7921](39605/5): Loss 0.0088, Sem 0.0088 LR: 3.787e-02 Score 99.682 Data time: 0.1482, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.5837 [2021-07-08 18:44:10,359][root][INFO] - ===> Epoch[7922](39610/5): Loss 0.0091, Sem 0.0091 LR: 3.786e-02 Score 99.671 Data time: 0.2055, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.6589 [2021-07-08 18:44:23,908][root][INFO] - ===> Epoch[7923](39615/5): Loss 0.0095, Sem 0.0095 LR: 3.785e-02 Score 99.659 Data time: 0.2196, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.6648 [2021-07-08 18:44:38,381][root][INFO] - ===> Epoch[7924](39620/5): Loss 0.0088, Sem 0.0088 LR: 3.784e-02 Score 99.682 Data time: 0.2991, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.8558 [2021-07-08 18:44:52,420][root][INFO] - ===> Epoch[7925](39625/5): Loss 0.0091, Sem 0.0091 LR: 3.783e-02 Score 99.673 Data time: 0.1657, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7749 [2021-07-08 18:45:06,634][root][INFO] - ===> Epoch[7926](39630/5): Loss 0.0092, Sem 0.0092 LR: 3.782e-02 Score 99.667 Data time: 0.2548, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7895 [2021-07-08 18:45:20,376][root][INFO] - ===> Epoch[7927](39635/5): Loss 0.0089, Sem 0.0089 LR: 3.782e-02 Score 99.676 Data time: 0.2181, Forward time: 1.2550, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7102 [2021-07-08 18:45:34,279][root][INFO] - ===> Epoch[7928](39640/5): Loss 0.0094, Sem 0.0094 LR: 3.781e-02 Score 99.661 Data time: 0.2638, Forward time: 1.2550, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7358 [2021-07-08 18:45:47,769][root][INFO] - ===> Epoch[7929](39645/5): Loss 0.0087, Sem 0.0087 LR: 3.780e-02 Score 99.687 Data time: 0.2691, Forward time: 1.2550, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.6459 [2021-07-08 18:46:01,631][root][INFO] - ===> Epoch[7930](39650/5): Loss 0.0091, Sem 0.0091 LR: 3.779e-02 Score 99.671 Data time: 0.2327, Forward time: 1.2550, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7220 [2021-07-08 18:46:15,684][root][INFO] - ===> Epoch[7931](39655/5): Loss 0.0091, Sem 0.0091 LR: 3.778e-02 Score 99.673 Data time: 0.1966, Forward time: 1.2550, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7626 [2021-07-08 18:46:29,875][root][INFO] - ===> Epoch[7932](39660/5): Loss 0.0089, Sem 0.0089 LR: 3.777e-02 Score 99.683 Data time: 0.2279, Forward time: 1.2550, Backward time: 1.2439, DDP time: 0.0042, Total iter time: 2.7997 [2021-07-08 18:46:45,009][root][INFO] - ===> Epoch[7933](39665/5): Loss 0.0091, Sem 0.0091 LR: 3.777e-02 Score 99.672 Data time: 0.2238, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.9785 [2021-07-08 18:46:59,943][root][INFO] - ===> Epoch[7934](39670/5): Loss 0.0096, Sem 0.0096 LR: 3.776e-02 Score 99.672 Data time: 0.1793, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.9294 [2021-07-08 18:47:13,231][root][INFO] - ===> Epoch[7935](39675/5): Loss 0.0125, Sem 0.0125 LR: 3.775e-02 Score 99.619 Data time: 0.2548, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.6145 [2021-07-08 18:47:26,908][root][INFO] - ===> Epoch[7936](39680/5): Loss 0.0109, Sem 0.0109 LR: 3.774e-02 Score 99.622 Data time: 0.2110, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.6968 [2021-07-08 18:47:41,344][root][INFO] - ===> Epoch[7937](39685/5): Loss 0.0109, Sem 0.0109 LR: 3.773e-02 Score 99.620 Data time: 0.2084, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.8418 [2021-07-08 18:47:54,658][root][INFO] - ===> Epoch[7938](39690/5): Loss 0.0111, Sem 0.0111 LR: 3.772e-02 Score 99.605 Data time: 0.1904, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.6223 [2021-07-08 18:48:09,446][root][INFO] - ===> Epoch[7939](39695/5): Loss 0.0116, Sem 0.0116 LR: 3.772e-02 Score 99.596 Data time: 0.2421, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.9129 [2021-07-08 18:48:23,425][root][INFO] - ===> Epoch[7940](39700/5): Loss 0.0114, Sem 0.0114 LR: 3.771e-02 Score 99.609 Data time: 0.2230, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7506 [2021-07-08 18:48:37,225][root][INFO] - ===> Epoch[7941](39705/5): Loss 0.0103, Sem 0.0103 LR: 3.770e-02 Score 99.630 Data time: 0.1705, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7233 [2021-07-08 18:48:51,220][root][INFO] - ===> Epoch[7942](39710/5): Loss 0.0105, Sem 0.0105 LR: 3.769e-02 Score 99.625 Data time: 0.1918, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7578 [2021-07-08 18:49:06,174][root][INFO] - ===> Epoch[7943](39715/5): Loss 0.0103, Sem 0.0103 LR: 3.768e-02 Score 99.634 Data time: 0.2387, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.9482 [2021-07-08 18:49:20,036][root][INFO] - ===> Epoch[7944](39720/5): Loss 0.0103, Sem 0.0103 LR: 3.767e-02 Score 99.629 Data time: 0.2231, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7245 [2021-07-08 18:49:34,468][root][INFO] - ===> Epoch[7945](39725/5): Loss 0.0098, Sem 0.0098 LR: 3.767e-02 Score 99.650 Data time: 0.2096, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.8390 [2021-07-08 18:49:48,273][root][INFO] - ===> Epoch[7946](39730/5): Loss 0.0094, Sem 0.0094 LR: 3.766e-02 Score 99.661 Data time: 0.1748, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7218 [2021-07-08 18:50:02,358][root][INFO] - ===> Epoch[7947](39735/5): Loss 0.0094, Sem 0.0094 LR: 3.765e-02 Score 99.663 Data time: 0.2314, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7636 [2021-07-08 18:50:17,323][root][INFO] - ===> Epoch[7948](39740/5): Loss 0.0097, Sem 0.0097 LR: 3.764e-02 Score 99.648 Data time: 0.2520, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.9433 [2021-07-08 18:50:31,673][root][INFO] - ===> Epoch[7949](39745/5): Loss 0.0091, Sem 0.0091 LR: 3.763e-02 Score 99.671 Data time: 0.2331, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.8215 [2021-07-08 18:50:45,407][root][INFO] - ===> Epoch[7950](39750/5): Loss 0.0093, Sem 0.0093 LR: 3.762e-02 Score 99.663 Data time: 0.1957, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7016 [2021-07-08 18:50:58,751][root][INFO] - ===> Epoch[7951](39755/5): Loss 0.0096, Sem 0.0096 LR: 3.762e-02 Score 99.654 Data time: 0.1704, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.6306 [2021-07-08 18:51:12,296][root][INFO] - ===> Epoch[7952](39760/5): Loss 0.0095, Sem 0.0095 LR: 3.761e-02 Score 99.655 Data time: 0.2499, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.6565 [2021-07-08 18:51:25,546][root][INFO] - ===> Epoch[7953](39765/5): Loss 0.0089, Sem 0.0089 LR: 3.760e-02 Score 99.676 Data time: 0.2356, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.6063 [2021-07-08 18:51:39,572][root][INFO] - ===> Epoch[7954](39770/5): Loss 0.0103, Sem 0.0103 LR: 3.759e-02 Score 99.632 Data time: 0.2401, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7583 [2021-07-08 18:51:53,441][root][INFO] - ===> Epoch[7955](39775/5): Loss 0.0098, Sem 0.0098 LR: 3.758e-02 Score 99.648 Data time: 0.2230, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7290 [2021-07-08 18:52:08,010][root][INFO] - ===> Epoch[7956](39780/5): Loss 0.0092, Sem 0.0092 LR: 3.757e-02 Score 99.670 Data time: 0.2423, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.8630 [2021-07-08 18:52:21,677][root][INFO] - ===> Epoch[7957](39785/5): Loss 0.0100, Sem 0.0100 LR: 3.756e-02 Score 99.644 Data time: 0.1348, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7005 [2021-07-08 18:52:35,976][root][INFO] - ===> Epoch[7958](39790/5): Loss 0.0093, Sem 0.0093 LR: 3.756e-02 Score 99.665 Data time: 0.1954, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.8229 [2021-07-08 18:52:49,639][root][INFO] - ===> Epoch[7959](39795/5): Loss 0.0093, Sem 0.0093 LR: 3.755e-02 Score 99.666 Data time: 0.2066, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7005 [2021-07-08 18:53:03,238][root][INFO] - ===> Epoch[7960](39800/5): Loss 0.0091, Sem 0.0091 LR: 3.754e-02 Score 99.672 Data time: 0.1993, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.6798 [2021-07-08 18:53:03,239][root][INFO] - ===> Start testing [2021-07-08 18:53:40,531][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1245 Loss 0.403 (AVG: 0.581) Score 91.618 (AVG: 91.178) mIOU 71.863 mAP 83.334 mAcc 77.985 IOU: 65.122 0.000 84.261 75.235 95.115 91.293 33.905 83.741 98.617 82.751 80.136 86.388 57.660 mAP: 85.864 3.534 91.064 87.693 98.417 98.667 57.882 98.221 99.685 85.587 91.305 93.960 91.462 mAcc: 80.674 0.000 87.272 82.554 96.424 97.357 36.432 95.643 99.513 89.079 90.670 96.620 61.573 [2021-07-08 18:53:40,531][root][INFO] - Finished test. Elapsed time: 37.2912 [2021-07-08 18:53:41,590][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 18:53:55,130][root][INFO] - ===> Epoch[7961](39805/5): Loss 0.0092, Sem 0.0092 LR: 3.753e-02 Score 99.674 Data time: 0.1734, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.6664 [2021-07-08 18:54:09,983][root][INFO] - ===> Epoch[7962](39810/5): Loss 0.0087, Sem 0.0087 LR: 3.752e-02 Score 99.682 Data time: 0.2470, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.9110 [2021-07-08 18:54:23,813][root][INFO] - ===> Epoch[7963](39815/5): Loss 0.0091, Sem 0.0091 LR: 3.751e-02 Score 99.671 Data time: 0.2340, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7169 [2021-07-08 18:54:37,899][root][INFO] - ===> Epoch[7964](39820/5): Loss 0.0095, Sem 0.0095 LR: 3.751e-02 Score 99.657 Data time: 0.2095, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7793 [2021-07-08 18:54:51,341][root][INFO] - ===> Epoch[7965](39825/5): Loss 0.0091, Sem 0.0091 LR: 3.750e-02 Score 99.667 Data time: 0.1614, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.6507 [2021-07-08 18:55:05,773][root][INFO] - ===> Epoch[7966](39830/5): Loss 0.0093, Sem 0.0093 LR: 3.749e-02 Score 99.668 Data time: 0.2906, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.8328 [2021-07-08 18:55:19,548][root][INFO] - ===> Epoch[7967](39835/5): Loss 0.0100, Sem 0.0100 LR: 3.748e-02 Score 99.643 Data time: 0.2036, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7042 [2021-07-08 18:55:33,932][root][INFO] - ===> Epoch[7968](39840/5): Loss 0.0094, Sem 0.0094 LR: 3.747e-02 Score 99.664 Data time: 0.2439, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.8365 [2021-07-08 18:55:48,925][root][INFO] - ===> Epoch[7969](39845/5): Loss 0.0092, Sem 0.0092 LR: 3.746e-02 Score 99.671 Data time: 0.2235, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.9526 [2021-07-08 18:56:03,321][root][INFO] - ===> Epoch[7970](39850/5): Loss 0.0095, Sem 0.0095 LR: 3.746e-02 Score 99.661 Data time: 0.3362, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.8254 [2021-07-08 18:56:17,556][root][INFO] - ===> Epoch[7971](39855/5): Loss 0.0090, Sem 0.0090 LR: 3.745e-02 Score 99.675 Data time: 0.2356, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.8003 [2021-07-08 18:56:31,707][root][INFO] - ===> Epoch[7972](39860/5): Loss 0.0094, Sem 0.0094 LR: 3.744e-02 Score 99.661 Data time: 0.3079, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7760 [2021-07-08 18:56:46,747][root][INFO] - ===> Epoch[7973](39865/5): Loss 0.0095, Sem 0.0095 LR: 3.743e-02 Score 99.658 Data time: 0.2282, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.9583 [2021-07-08 18:56:59,916][root][INFO] - ===> Epoch[7974](39870/5): Loss 0.0090, Sem 0.0090 LR: 3.742e-02 Score 99.673 Data time: 0.2101, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.5954 [2021-07-08 18:57:14,022][root][INFO] - ===> Epoch[7975](39875/5): Loss 0.0092, Sem 0.0092 LR: 3.741e-02 Score 99.671 Data time: 0.1916, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7829 [2021-07-08 18:57:27,496][root][INFO] - ===> Epoch[7976](39880/5): Loss 0.0094, Sem 0.0094 LR: 3.741e-02 Score 99.658 Data time: 0.2025, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.6556 [2021-07-08 18:57:42,199][root][INFO] - ===> Epoch[7977](39885/5): Loss 0.0091, Sem 0.0091 LR: 3.740e-02 Score 99.669 Data time: 0.2499, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.8851 [2021-07-08 18:57:56,849][root][INFO] - ===> Epoch[7978](39890/5): Loss 0.0097, Sem 0.0097 LR: 3.739e-02 Score 99.654 Data time: 0.2726, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.8711 [2021-07-08 18:58:10,649][root][INFO] - ===> Epoch[7979](39895/5): Loss 0.0089, Sem 0.0089 LR: 3.738e-02 Score 99.681 Data time: 0.2448, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7153 [2021-07-08 18:58:24,899][root][INFO] - ===> Epoch[7980](39900/5): Loss 0.0094, Sem 0.0094 LR: 3.737e-02 Score 99.663 Data time: 0.1911, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.8046 [2021-07-08 18:58:39,491][root][INFO] - ===> Epoch[7981](39905/5): Loss 0.0088, Sem 0.0088 LR: 3.736e-02 Score 99.685 Data time: 0.2275, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.8737 [2021-07-08 18:58:52,925][root][INFO] - ===> Epoch[7982](39910/5): Loss 0.0088, Sem 0.0088 LR: 3.736e-02 Score 99.685 Data time: 0.1778, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.6453 [2021-07-08 18:59:06,520][root][INFO] - ===> Epoch[7983](39915/5): Loss 0.0092, Sem 0.0092 LR: 3.735e-02 Score 99.667 Data time: 0.2181, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.6678 [2021-07-08 18:59:20,590][root][INFO] - ===> Epoch[7984](39920/5): Loss 0.0095, Sem 0.0095 LR: 3.734e-02 Score 99.660 Data time: 0.2642, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7732 [2021-07-08 18:59:33,726][root][INFO] - ===> Epoch[7985](39925/5): Loss 0.0093, Sem 0.0093 LR: 3.733e-02 Score 99.668 Data time: 0.1996, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.5792 [2021-07-08 18:59:48,048][root][INFO] - ===> Epoch[7986](39930/5): Loss 0.0092, Sem 0.0092 LR: 3.732e-02 Score 99.667 Data time: 0.2398, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.8197 [2021-07-08 19:00:02,427][root][INFO] - ===> Epoch[7987](39935/5): Loss 0.0095, Sem 0.0095 LR: 3.731e-02 Score 99.668 Data time: 0.3274, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.8294 [2021-07-08 19:00:16,450][root][INFO] - ===> Epoch[7988](39940/5): Loss 0.0096, Sem 0.0096 LR: 3.731e-02 Score 99.653 Data time: 0.2351, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.7568 [2021-07-08 19:00:30,761][root][INFO] - ===> Epoch[7989](39945/5): Loss 0.0092, Sem 0.0092 LR: 3.730e-02 Score 99.668 Data time: 0.2051, Forward time: 1.2550, Backward time: 1.2440, DDP time: 0.0042, Total iter time: 2.8205 [2021-07-08 19:00:44,799][root][INFO] - ===> Epoch[7990](39950/5): Loss 0.0093, Sem 0.0093 LR: 3.729e-02 Score 99.667 Data time: 0.2038, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7692 [2021-07-08 19:00:59,375][root][INFO] - ===> Epoch[7991](39955/5): Loss 0.0091, Sem 0.0091 LR: 3.728e-02 Score 99.669 Data time: 0.1900, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.8494 [2021-07-08 19:01:13,205][root][INFO] - ===> Epoch[7992](39960/5): Loss 0.0090, Sem 0.0090 LR: 3.727e-02 Score 99.676 Data time: 0.2226, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7126 [2021-07-08 19:01:26,657][root][INFO] - ===> Epoch[7993](39965/5): Loss 0.0094, Sem 0.0094 LR: 3.726e-02 Score 99.660 Data time: 0.2462, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.6484 [2021-07-08 19:01:40,711][root][INFO] - ===> Epoch[7994](39970/5): Loss 0.0090, Sem 0.0090 LR: 3.726e-02 Score 99.678 Data time: 0.2479, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7662 [2021-07-08 19:01:54,581][root][INFO] - ===> Epoch[7995](39975/5): Loss 0.0090, Sem 0.0090 LR: 3.725e-02 Score 99.674 Data time: 0.2549, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7269 [2021-07-08 19:02:08,312][root][INFO] - ===> Epoch[7996](39980/5): Loss 0.0089, Sem 0.0089 LR: 3.724e-02 Score 99.678 Data time: 0.1911, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7048 [2021-07-08 19:02:22,897][root][INFO] - ===> Epoch[7997](39985/5): Loss 0.0092, Sem 0.0092 LR: 3.723e-02 Score 99.671 Data time: 0.2167, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.8742 [2021-07-08 19:02:37,803][root][INFO] - ===> Epoch[7998](39990/5): Loss 0.0088, Sem 0.0088 LR: 3.722e-02 Score 99.682 Data time: 0.2247, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.9263 [2021-07-08 19:02:51,669][root][INFO] - ===> Epoch[7999](39995/5): Loss 0.0090, Sem 0.0090 LR: 3.721e-02 Score 99.674 Data time: 0.2181, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7294 [2021-07-08 19:03:05,517][root][INFO] - ===> Epoch[8000](40000/5): Loss 0.0089, Sem 0.0089 LR: 3.721e-02 Score 99.682 Data time: 0.2485, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7194 [2021-07-08 19:03:05,518][root][INFO] - ===> Start testing [2021-07-08 19:03:42,832][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1290 Loss 0.397 (AVG: 0.598) Score 91.631 (AVG: 91.089) mIOU 72.163 mAP 83.404 mAcc 77.951 IOU: 64.751 0.000 85.211 75.075 94.939 91.755 33.979 87.490 98.653 83.585 80.551 85.771 56.352 mAP: 84.963 2.755 92.637 87.636 98.457 98.559 56.384 98.572 99.728 90.642 90.989 92.940 89.990 mAcc: 79.817 0.000 88.530 82.728 96.281 97.420 36.409 96.056 99.440 89.976 90.494 96.868 59.347 [2021-07-08 19:03:42,832][root][INFO] - Finished test. Elapsed time: 37.3125 [2021-07-08 19:03:43,905][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 19:03:56,091][root][INFO] - ===> Epoch[8001](40005/5): Loss 0.0098, Sem 0.0098 LR: 3.720e-02 Score 99.646 Data time: 0.1934, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.3989 [2021-07-08 19:04:09,867][root][INFO] - ===> Epoch[8002](40010/5): Loss 0.0094, Sem 0.0094 LR: 3.719e-02 Score 99.658 Data time: 0.2357, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7198 [2021-07-08 19:04:25,199][root][INFO] - ===> Epoch[8003](40015/5): Loss 0.0093, Sem 0.0093 LR: 3.718e-02 Score 99.668 Data time: 0.2551, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 3.0207 [2021-07-08 19:04:40,045][root][INFO] - ===> Epoch[8004](40020/5): Loss 0.0092, Sem 0.0092 LR: 3.717e-02 Score 99.666 Data time: 0.2505, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.9167 [2021-07-08 19:04:53,559][root][INFO] - ===> Epoch[8005](40025/5): Loss 0.0088, Sem 0.0088 LR: 3.716e-02 Score 99.678 Data time: 0.1861, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.6571 [2021-07-08 19:05:07,992][root][INFO] - ===> Epoch[8006](40030/5): Loss 0.0091, Sem 0.0091 LR: 3.715e-02 Score 99.672 Data time: 0.2610, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.8426 [2021-07-08 19:05:22,293][root][INFO] - ===> Epoch[8007](40035/5): Loss 0.0095, Sem 0.0095 LR: 3.715e-02 Score 99.656 Data time: 0.2484, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.8176 [2021-07-08 19:05:36,670][root][INFO] - ===> Epoch[8008](40040/5): Loss 0.0091, Sem 0.0091 LR: 3.714e-02 Score 99.671 Data time: 0.2688, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.8190 [2021-07-08 19:05:50,747][root][INFO] - ===> Epoch[8009](40045/5): Loss 0.0095, Sem 0.0095 LR: 3.713e-02 Score 99.660 Data time: 0.2708, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7751 [2021-07-08 19:06:04,902][root][INFO] - ===> Epoch[8010](40050/5): Loss 0.0089, Sem 0.0089 LR: 3.712e-02 Score 99.679 Data time: 0.3115, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7806 [2021-07-08 19:06:19,527][root][INFO] - ===> Epoch[8011](40055/5): Loss 0.0088, Sem 0.0088 LR: 3.711e-02 Score 99.681 Data time: 0.2024, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.8773 [2021-07-08 19:06:33,496][root][INFO] - ===> Epoch[8012](40060/5): Loss 0.0092, Sem 0.0092 LR: 3.710e-02 Score 99.669 Data time: 0.2625, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7480 [2021-07-08 19:06:47,662][root][INFO] - ===> Epoch[8013](40065/5): Loss 0.0087, Sem 0.0087 LR: 3.710e-02 Score 99.684 Data time: 0.2833, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7923 [2021-07-08 19:07:01,534][root][INFO] - ===> Epoch[8014](40070/5): Loss 0.0091, Sem 0.0091 LR: 3.709e-02 Score 99.670 Data time: 0.2008, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7246 [2021-07-08 19:07:16,481][root][INFO] - ===> Epoch[8015](40075/5): Loss 0.0088, Sem 0.0088 LR: 3.708e-02 Score 99.682 Data time: 0.2627, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.9374 [2021-07-08 19:07:30,335][root][INFO] - ===> Epoch[8016](40080/5): Loss 0.0090, Sem 0.0090 LR: 3.707e-02 Score 99.674 Data time: 0.2418, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7344 [2021-07-08 19:07:44,226][root][INFO] - ===> Epoch[8017](40085/5): Loss 0.0090, Sem 0.0090 LR: 3.706e-02 Score 99.677 Data time: 0.2038, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7330 [2021-07-08 19:07:57,902][root][INFO] - ===> Epoch[8018](40090/5): Loss 0.0090, Sem 0.0090 LR: 3.705e-02 Score 99.676 Data time: 0.2345, Forward time: 1.2550, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.6870 [2021-07-08 19:08:12,947][root][INFO] - ===> Epoch[8019](40095/5): Loss 0.0092, Sem 0.0092 LR: 3.705e-02 Score 99.666 Data time: 0.1425, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.9694 [2021-07-08 19:08:27,079][root][INFO] - ===> Epoch[8020](40100/5): Loss 0.0087, Sem 0.0087 LR: 3.704e-02 Score 99.684 Data time: 0.2629, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7813 [2021-07-08 19:08:41,235][root][INFO] - ===> Epoch[8021](40105/5): Loss 0.0090, Sem 0.0090 LR: 3.703e-02 Score 99.676 Data time: 0.2085, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7754 [2021-07-08 19:08:55,724][root][INFO] - ===> Epoch[8022](40110/5): Loss 0.0089, Sem 0.0089 LR: 3.702e-02 Score 99.679 Data time: 0.2287, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.8547 [2021-07-08 19:09:09,910][root][INFO] - ===> Epoch[8023](40115/5): Loss 0.0090, Sem 0.0090 LR: 3.701e-02 Score 99.678 Data time: 0.2607, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7986 [2021-07-08 19:09:23,737][root][INFO] - ===> Epoch[8024](40120/5): Loss 0.0093, Sem 0.0093 LR: 3.700e-02 Score 99.666 Data time: 0.2307, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7185 [2021-07-08 19:09:36,863][root][INFO] - ===> Epoch[8025](40125/5): Loss 0.0091, Sem 0.0091 LR: 3.700e-02 Score 99.671 Data time: 0.1707, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.5946 [2021-07-08 19:09:50,977][root][INFO] - ===> Epoch[8026](40130/5): Loss 0.0089, Sem 0.0089 LR: 3.699e-02 Score 99.677 Data time: 0.2490, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7722 [2021-07-08 19:10:05,476][root][INFO] - ===> Epoch[8027](40135/5): Loss 0.0092, Sem 0.0092 LR: 3.698e-02 Score 99.665 Data time: 0.1982, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.8606 [2021-07-08 19:10:19,153][root][INFO] - ===> Epoch[8028](40140/5): Loss 0.0088, Sem 0.0088 LR: 3.697e-02 Score 99.684 Data time: 0.1630, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7088 [2021-07-08 19:10:33,240][root][INFO] - ===> Epoch[8029](40145/5): Loss 0.0091, Sem 0.0091 LR: 3.696e-02 Score 99.674 Data time: 0.1998, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.7742 [2021-07-08 19:10:46,623][root][INFO] - ===> Epoch[8030](40150/5): Loss 0.0094, Sem 0.0094 LR: 3.695e-02 Score 99.661 Data time: 0.2097, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.6380 [2021-07-08 19:11:00,663][root][INFO] - ===> Epoch[8031](40155/5): Loss 0.0084, Sem 0.0084 LR: 3.695e-02 Score 99.695 Data time: 0.2236, Forward time: 1.2549, Backward time: 1.2441, DDP time: 0.0042, Total iter time: 2.7640 [2021-07-08 19:11:14,929][root][INFO] - ===> Epoch[8032](40160/5): Loss 0.0092, Sem 0.0092 LR: 3.694e-02 Score 99.670 Data time: 0.2179, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.8089 [2021-07-08 19:11:29,578][root][INFO] - ===> Epoch[8033](40165/5): Loss 0.0089, Sem 0.0089 LR: 3.693e-02 Score 99.678 Data time: 0.1694, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.8922 [2021-07-08 19:11:43,394][root][INFO] - ===> Epoch[8034](40170/5): Loss 0.0093, Sem 0.0093 LR: 3.692e-02 Score 99.674 Data time: 0.2207, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.7172 [2021-07-08 19:11:57,804][root][INFO] - ===> Epoch[8035](40175/5): Loss 0.0092, Sem 0.0092 LR: 3.691e-02 Score 99.668 Data time: 0.1824, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.8382 [2021-07-08 19:12:12,500][root][INFO] - ===> Epoch[8036](40180/5): Loss 0.0090, Sem 0.0090 LR: 3.690e-02 Score 99.680 Data time: 0.2240, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.8915 [2021-07-08 19:12:26,167][root][INFO] - ===> Epoch[8037](40185/5): Loss 0.0091, Sem 0.0091 LR: 3.690e-02 Score 99.673 Data time: 0.2185, Forward time: 1.2549, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7035 [2021-07-08 19:12:40,487][root][INFO] - ===> Epoch[8038](40190/5): Loss 0.0088, Sem 0.0088 LR: 3.689e-02 Score 99.684 Data time: 0.1924, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.8244 [2021-07-08 19:12:54,705][root][INFO] - ===> Epoch[8039](40195/5): Loss 0.0091, Sem 0.0091 LR: 3.688e-02 Score 99.672 Data time: 0.1417, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.8093 [2021-07-08 19:13:09,637][root][INFO] - ===> Epoch[8040](40200/5): Loss 0.0089, Sem 0.0089 LR: 3.687e-02 Score 99.678 Data time: 0.2037, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.9388 [2021-07-08 19:13:09,638][root][INFO] - ===> Start testing [2021-07-08 19:13:51,445][root][INFO] - 68/68: Data time: 0.0460, Iter time: 0.1254 Loss 0.440 (AVG: 0.608) Score 91.808 (AVG: 91.008) mIOU 70.994 mAP 82.569 mAcc 77.010 IOU: 65.116 0.000 83.436 75.346 95.164 90.578 25.709 87.111 98.612 80.135 78.921 85.683 57.117 mAP: 84.952 2.681 91.013 87.318 98.407 98.503 51.431 98.498 99.691 87.526 90.758 92.824 89.797 mAcc: 81.072 0.000 87.186 82.341 96.457 97.138 26.746 96.358 99.367 86.563 90.899 96.712 60.288 [2021-07-08 19:13:51,445][root][INFO] - Finished test. Elapsed time: 41.8058 [2021-07-08 19:13:52,542][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 19:14:06,135][root][INFO] - ===> Epoch[8041](40205/5): Loss 0.0093, Sem 0.0093 LR: 3.686e-02 Score 99.672 Data time: 0.2417, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.6545 [2021-07-08 19:14:20,012][root][INFO] - ===> Epoch[8042](40210/5): Loss 0.0090, Sem 0.0090 LR: 3.685e-02 Score 99.674 Data time: 0.2134, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7312 [2021-07-08 19:14:34,698][root][INFO] - ===> Epoch[8043](40215/5): Loss 0.0092, Sem 0.0092 LR: 3.685e-02 Score 99.669 Data time: 0.1915, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.8754 [2021-07-08 19:14:48,647][root][INFO] - ===> Epoch[8044](40220/5): Loss 0.0087, Sem 0.0087 LR: 3.684e-02 Score 99.685 Data time: 0.2188, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7321 [2021-07-08 19:15:02,497][root][INFO] - ===> Epoch[8045](40225/5): Loss 0.0091, Sem 0.0091 LR: 3.683e-02 Score 99.671 Data time: 0.2717, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7177 [2021-07-08 19:15:17,191][root][INFO] - ===> Epoch[8046](40230/5): Loss 0.0089, Sem 0.0089 LR: 3.682e-02 Score 99.677 Data time: 0.2234, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.8901 [2021-07-08 19:15:31,037][root][INFO] - ===> Epoch[8047](40235/5): Loss 0.0088, Sem 0.0088 LR: 3.681e-02 Score 99.685 Data time: 0.2867, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7196 [2021-07-08 19:15:44,998][root][INFO] - ===> Epoch[8048](40240/5): Loss 0.0087, Sem 0.0087 LR: 3.680e-02 Score 99.688 Data time: 0.2645, Forward time: 1.2549, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7483 [2021-07-08 19:15:59,616][root][INFO] - ===> Epoch[8049](40245/5): Loss 0.0090, Sem 0.0090 LR: 3.679e-02 Score 99.674 Data time: 0.2161, Forward time: 1.2549, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.8777 [2021-07-08 19:16:13,222][root][INFO] - ===> Epoch[8050](40250/5): Loss 0.0089, Sem 0.0089 LR: 3.679e-02 Score 99.677 Data time: 0.2840, Forward time: 1.2549, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.6676 [2021-07-08 19:16:26,654][root][INFO] - ===> Epoch[8051](40255/5): Loss 0.0092, Sem 0.0092 LR: 3.678e-02 Score 99.671 Data time: 0.2623, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.6388 [2021-07-08 19:16:41,232][root][INFO] - ===> Epoch[8052](40260/5): Loss 0.0088, Sem 0.0088 LR: 3.677e-02 Score 99.685 Data time: 0.2491, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.8757 [2021-07-08 19:16:55,401][root][INFO] - ===> Epoch[8053](40265/5): Loss 0.0090, Sem 0.0090 LR: 3.676e-02 Score 99.679 Data time: 0.2872, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.7811 [2021-07-08 19:17:10,270][root][INFO] - ===> Epoch[8054](40270/5): Loss 0.0090, Sem 0.0090 LR: 3.675e-02 Score 99.675 Data time: 0.2863, Forward time: 1.2549, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.9136 [2021-07-08 19:17:24,270][root][INFO] - ===> Epoch[8055](40275/5): Loss 0.0092, Sem 0.0092 LR: 3.674e-02 Score 99.671 Data time: 0.2331, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.7528 [2021-07-08 19:17:37,153][root][INFO] - ===> Epoch[8056](40280/5): Loss 0.0092, Sem 0.0092 LR: 3.674e-02 Score 99.669 Data time: 0.2344, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.5379 [2021-07-08 19:17:51,886][root][INFO] - ===> Epoch[8057](40285/5): Loss 0.0089, Sem 0.0089 LR: 3.673e-02 Score 99.680 Data time: 0.2424, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.8973 [2021-07-08 19:18:05,739][root][INFO] - ===> Epoch[8058](40290/5): Loss 0.0087, Sem 0.0087 LR: 3.672e-02 Score 99.686 Data time: 0.2017, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.7149 [2021-07-08 19:18:20,123][root][INFO] - ===> Epoch[8059](40295/5): Loss 0.0087, Sem 0.0087 LR: 3.671e-02 Score 99.687 Data time: 0.1704, Forward time: 1.2549, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.8265 [2021-07-08 19:18:34,001][root][INFO] - ===> Epoch[8060](40300/5): Loss 0.0087, Sem 0.0087 LR: 3.670e-02 Score 99.683 Data time: 0.1561, Forward time: 1.2549, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7404 [2021-07-08 19:18:47,560][root][INFO] - ===> Epoch[8061](40305/5): Loss 0.0088, Sem 0.0088 LR: 3.669e-02 Score 99.684 Data time: 0.2532, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.6700 [2021-07-08 19:19:01,107][root][INFO] - ===> Epoch[8062](40310/5): Loss 0.0089, Sem 0.0089 LR: 3.669e-02 Score 99.679 Data time: 0.2269, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.6657 [2021-07-08 19:19:14,720][root][INFO] - ===> Epoch[8063](40315/5): Loss 0.0088, Sem 0.0088 LR: 3.668e-02 Score 99.685 Data time: 0.2010, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.6818 [2021-07-08 19:19:28,186][root][INFO] - ===> Epoch[8064](40320/5): Loss 0.0086, Sem 0.0086 LR: 3.667e-02 Score 99.687 Data time: 0.2528, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.6362 [2021-07-08 19:19:41,392][root][INFO] - ===> Epoch[8065](40325/5): Loss 0.0090, Sem 0.0090 LR: 3.666e-02 Score 99.676 Data time: 0.2031, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.5921 [2021-07-08 19:19:54,903][root][INFO] - ===> Epoch[8066](40330/5): Loss 0.0090, Sem 0.0090 LR: 3.665e-02 Score 99.678 Data time: 0.2136, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.6615 [2021-07-08 19:20:08,461][root][INFO] - ===> Epoch[8067](40335/5): Loss 0.0091, Sem 0.0091 LR: 3.664e-02 Score 99.674 Data time: 0.2335, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.6638 [2021-07-08 19:20:22,959][root][INFO] - ===> Epoch[8068](40340/5): Loss 0.0091, Sem 0.0091 LR: 3.664e-02 Score 99.674 Data time: 0.2044, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.8538 [2021-07-08 19:20:37,142][root][INFO] - ===> Epoch[8069](40345/5): Loss 0.0087, Sem 0.0087 LR: 3.663e-02 Score 99.686 Data time: 0.2237, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.7874 [2021-07-08 19:20:51,021][root][INFO] - ===> Epoch[8070](40350/5): Loss 0.0087, Sem 0.0087 LR: 3.662e-02 Score 99.688 Data time: 0.2496, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.7285 [2021-07-08 19:21:06,392][root][INFO] - ===> Epoch[8071](40355/5): Loss 0.0085, Sem 0.0085 LR: 3.661e-02 Score 99.694 Data time: 0.2593, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 3.0234 [2021-07-08 19:21:19,993][root][INFO] - ===> Epoch[8072](40360/5): Loss 0.0092, Sem 0.0092 LR: 3.660e-02 Score 99.671 Data time: 0.2570, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.6731 [2021-07-08 19:21:34,555][root][INFO] - ===> Epoch[8073](40365/5): Loss 0.0088, Sem 0.0088 LR: 3.659e-02 Score 99.683 Data time: 0.2125, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.8678 [2021-07-08 19:21:48,587][root][INFO] - ===> Epoch[8074](40370/5): Loss 0.0090, Sem 0.0090 LR: 3.659e-02 Score 99.677 Data time: 0.2405, Forward time: 1.2549, Backward time: 1.2442, DDP time: 0.0042, Total iter time: 2.7609 [2021-07-08 19:22:03,818][root][INFO] - ===> Epoch[8075](40375/5): Loss 0.0089, Sem 0.0089 LR: 3.658e-02 Score 99.679 Data time: 0.2234, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.9972 [2021-07-08 19:22:17,926][root][INFO] - ===> Epoch[8076](40380/5): Loss 0.0088, Sem 0.0088 LR: 3.657e-02 Score 99.681 Data time: 0.1620, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7801 [2021-07-08 19:22:31,907][root][INFO] - ===> Epoch[8077](40385/5): Loss 0.0091, Sem 0.0091 LR: 3.656e-02 Score 99.671 Data time: 0.2077, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7431 [2021-07-08 19:22:46,718][root][INFO] - ===> Epoch[8078](40390/5): Loss 0.0083, Sem 0.0083 LR: 3.655e-02 Score 99.700 Data time: 0.2614, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.9115 [2021-07-08 19:23:00,858][root][INFO] - ===> Epoch[8079](40395/5): Loss 0.0086, Sem 0.0086 LR: 3.654e-02 Score 99.692 Data time: 0.2223, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7807 [2021-07-08 19:23:14,863][root][INFO] - ===> Epoch[8080](40400/5): Loss 0.0091, Sem 0.0091 LR: 3.653e-02 Score 99.668 Data time: 0.2234, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7520 [2021-07-08 19:23:14,864][root][INFO] - ===> Start testing [2021-07-08 19:23:51,140][root][INFO] - 68/68: Data time: 0.1139, Iter time: 0.1298 Loss 0.408 (AVG: 0.580) Score 90.211 (AVG: 91.301) mIOU 72.403 mAP 83.118 mAcc 78.432 IOU: 65.792 0.000 83.234 76.022 95.093 91.732 36.603 86.665 98.657 82.141 79.867 86.356 59.080 mAP: 84.850 4.889 92.470 87.308 98.628 98.475 56.845 98.526 99.678 84.248 90.826 93.167 90.620 mAcc: 81.526 0.000 88.195 83.604 96.365 97.293 40.657 95.920 99.539 86.079 91.468 95.731 63.238 [2021-07-08 19:23:51,141][root][INFO] - Finished test. Elapsed time: 36.2759 [2021-07-08 19:23:52,205][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 19:24:05,511][root][INFO] - ===> Epoch[8081](40405/5): Loss 0.0092, Sem 0.0092 LR: 3.653e-02 Score 99.669 Data time: 0.1525, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.6224 [2021-07-08 19:24:18,969][root][INFO] - ===> Epoch[8082](40410/5): Loss 0.0089, Sem 0.0089 LR: 3.652e-02 Score 99.677 Data time: 0.2303, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.6488 [2021-07-08 19:24:32,606][root][INFO] - ===> Epoch[8083](40415/5): Loss 0.0087, Sem 0.0087 LR: 3.651e-02 Score 99.686 Data time: 0.2034, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.6802 [2021-07-08 19:24:46,763][root][INFO] - ===> Epoch[8084](40420/5): Loss 0.0087, Sem 0.0087 LR: 3.650e-02 Score 99.685 Data time: 0.2010, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7960 [2021-07-08 19:25:01,670][root][INFO] - ===> Epoch[8085](40425/5): Loss 0.0084, Sem 0.0084 LR: 3.649e-02 Score 99.698 Data time: 0.2702, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.9328 [2021-07-08 19:25:16,566][root][INFO] - ===> Epoch[8086](40430/5): Loss 0.0088, Sem 0.0088 LR: 3.648e-02 Score 99.683 Data time: 0.2226, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.9348 [2021-07-08 19:25:30,631][root][INFO] - ===> Epoch[8087](40435/5): Loss 0.0088, Sem 0.0088 LR: 3.648e-02 Score 99.679 Data time: 0.2251, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7602 [2021-07-08 19:25:44,630][root][INFO] - ===> Epoch[8088](40440/5): Loss 0.0085, Sem 0.0085 LR: 3.647e-02 Score 99.690 Data time: 0.2094, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7574 [2021-07-08 19:25:59,001][root][INFO] - ===> Epoch[8089](40445/5): Loss 0.0091, Sem 0.0091 LR: 3.646e-02 Score 99.672 Data time: 0.2068, Forward time: 1.2549, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.8265 [2021-07-08 19:26:13,882][root][INFO] - ===> Epoch[8090](40450/5): Loss 0.0087, Sem 0.0087 LR: 3.645e-02 Score 99.687 Data time: 0.1779, Forward time: 1.2549, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.9339 [2021-07-08 19:26:28,190][root][INFO] - ===> Epoch[8091](40455/5): Loss 0.0092, Sem 0.0092 LR: 3.644e-02 Score 99.668 Data time: 0.3241, Forward time: 1.2549, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.8138 [2021-07-08 19:26:42,169][root][INFO] - ===> Epoch[8092](40460/5): Loss 0.0089, Sem 0.0089 LR: 3.643e-02 Score 99.678 Data time: 0.2125, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7614 [2021-07-08 19:26:56,135][root][INFO] - ===> Epoch[8093](40465/5): Loss 0.0090, Sem 0.0090 LR: 3.643e-02 Score 99.678 Data time: 0.2783, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7456 [2021-07-08 19:27:09,782][root][INFO] - ===> Epoch[8094](40470/5): Loss 0.0091, Sem 0.0091 LR: 3.642e-02 Score 99.672 Data time: 0.2528, Forward time: 1.2549, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.6869 [2021-07-08 19:27:24,350][root][INFO] - ===> Epoch[8095](40475/5): Loss 0.0088, Sem 0.0088 LR: 3.641e-02 Score 99.683 Data time: 0.2556, Forward time: 1.2549, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.8712 [2021-07-08 19:27:38,653][root][INFO] - ===> Epoch[8096](40480/5): Loss 0.0089, Sem 0.0089 LR: 3.640e-02 Score 99.680 Data time: 0.2375, Forward time: 1.2549, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.8205 [2021-07-08 19:27:52,386][root][INFO] - ===> Epoch[8097](40485/5): Loss 0.0095, Sem 0.0095 LR: 3.639e-02 Score 99.659 Data time: 0.2666, Forward time: 1.2549, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.6921 [2021-07-08 19:28:06,402][root][INFO] - ===> Epoch[8098](40490/5): Loss 0.0085, Sem 0.0085 LR: 3.638e-02 Score 99.693 Data time: 0.2511, Forward time: 1.2549, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7663 [2021-07-08 19:28:20,254][root][INFO] - ===> Epoch[8099](40495/5): Loss 0.0086, Sem 0.0086 LR: 3.638e-02 Score 99.691 Data time: 0.1590, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7293 [2021-07-08 19:28:34,182][root][INFO] - ===> Epoch[8100](40500/5): Loss 0.0089, Sem 0.0089 LR: 3.637e-02 Score 99.678 Data time: 0.2174, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7490 [2021-07-08 19:28:47,911][root][INFO] - ===> Epoch[8101](40505/5): Loss 0.0088, Sem 0.0088 LR: 3.636e-02 Score 99.683 Data time: 0.1625, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7034 [2021-07-08 19:29:01,801][root][INFO] - ===> Epoch[8102](40510/5): Loss 0.0086, Sem 0.0086 LR: 3.635e-02 Score 99.694 Data time: 0.2036, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7351 [2021-07-08 19:29:15,379][root][INFO] - ===> Epoch[8103](40515/5): Loss 0.0089, Sem 0.0089 LR: 3.634e-02 Score 99.682 Data time: 0.1818, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.6690 [2021-07-08 19:29:29,445][root][INFO] - ===> Epoch[8104](40520/5): Loss 0.0094, Sem 0.0094 LR: 3.633e-02 Score 99.664 Data time: 0.2135, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7736 [2021-07-08 19:29:43,686][root][INFO] - ===> Epoch[8105](40525/5): Loss 0.0083, Sem 0.0083 LR: 3.633e-02 Score 99.702 Data time: 0.1949, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.8024 [2021-07-08 19:29:57,945][root][INFO] - ===> Epoch[8106](40530/5): Loss 0.0086, Sem 0.0086 LR: 3.632e-02 Score 99.688 Data time: 0.2106, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.8082 [2021-07-08 19:30:11,668][root][INFO] - ===> Epoch[8107](40535/5): Loss 0.0092, Sem 0.0092 LR: 3.631e-02 Score 99.669 Data time: 0.1801, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.6937 [2021-07-08 19:30:25,727][root][INFO] - ===> Epoch[8108](40540/5): Loss 0.0085, Sem 0.0085 LR: 3.630e-02 Score 99.690 Data time: 0.1861, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7667 [2021-07-08 19:30:39,515][root][INFO] - ===> Epoch[8109](40545/5): Loss 0.0088, Sem 0.0088 LR: 3.629e-02 Score 99.684 Data time: 0.2109, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7107 [2021-07-08 19:30:53,735][root][INFO] - ===> Epoch[8110](40550/5): Loss 0.0089, Sem 0.0089 LR: 3.628e-02 Score 99.682 Data time: 0.2686, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.8028 [2021-07-08 19:31:07,031][root][INFO] - ===> Epoch[8111](40555/5): Loss 0.0095, Sem 0.0095 LR: 3.627e-02 Score 99.663 Data time: 0.2707, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.6125 [2021-07-08 19:31:20,717][root][INFO] - ===> Epoch[8112](40560/5): Loss 0.0086, Sem 0.0086 LR: 3.627e-02 Score 99.693 Data time: 0.2460, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.6962 [2021-07-08 19:31:35,311][root][INFO] - ===> Epoch[8113](40565/5): Loss 0.0088, Sem 0.0088 LR: 3.626e-02 Score 99.685 Data time: 0.2571, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.8729 [2021-07-08 19:31:49,237][root][INFO] - ===> Epoch[8114](40570/5): Loss 0.0086, Sem 0.0086 LR: 3.625e-02 Score 99.690 Data time: 0.2007, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.7265 [2021-07-08 19:32:03,487][root][INFO] - ===> Epoch[8115](40575/5): Loss 0.0086, Sem 0.0086 LR: 3.624e-02 Score 99.690 Data time: 0.2352, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7979 [2021-07-08 19:32:16,899][root][INFO] - ===> Epoch[8116](40580/5): Loss 0.0089, Sem 0.0089 LR: 3.623e-02 Score 99.680 Data time: 0.2123, Forward time: 1.2548, Backward time: 1.2443, DDP time: 0.0042, Total iter time: 2.6431 [2021-07-08 19:32:32,082][root][INFO] - ===> Epoch[8117](40585/5): Loss 0.0091, Sem 0.0091 LR: 3.622e-02 Score 99.673 Data time: 0.1909, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.9903 [2021-07-08 19:32:46,660][root][INFO] - ===> Epoch[8118](40590/5): Loss 0.0084, Sem 0.0084 LR: 3.622e-02 Score 99.697 Data time: 0.2818, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.8701 [2021-07-08 19:33:01,010][root][INFO] - ===> Epoch[8119](40595/5): Loss 0.0090, Sem 0.0090 LR: 3.621e-02 Score 99.678 Data time: 0.1699, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.8281 [2021-07-08 19:33:14,849][root][INFO] - ===> Epoch[8120](40600/5): Loss 0.0086, Sem 0.0086 LR: 3.620e-02 Score 99.693 Data time: 0.2787, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7223 [2021-07-08 19:33:14,851][root][INFO] - ===> Start testing [2021-07-08 19:33:52,505][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1199 Loss 0.414 (AVG: 0.578) Score 91.242 (AVG: 91.256) mIOU 72.359 mAP 83.571 mAcc 78.326 IOU: 65.182 0.000 85.182 75.716 95.001 91.675 33.963 86.544 98.597 84.114 79.959 86.375 58.360 mAP: 85.041 4.385 93.408 87.804 98.469 98.373 54.091 98.475 99.740 91.593 91.127 92.706 91.205 mAcc: 80.078 0.000 90.217 83.401 96.378 97.439 36.726 96.080 99.342 88.546 91.232 96.540 62.260 [2021-07-08 19:33:52,505][root][INFO] - Finished test. Elapsed time: 37.6540 [2021-07-08 19:33:53,586][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 19:34:07,065][root][INFO] - ===> Epoch[8121](40605/5): Loss 0.0088, Sem 0.0088 LR: 3.619e-02 Score 99.685 Data time: 0.1799, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.6616 [2021-07-08 19:34:22,318][root][INFO] - ===> Epoch[8122](40610/5): Loss 0.0086, Sem 0.0086 LR: 3.618e-02 Score 99.691 Data time: 0.2326, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.9952 [2021-07-08 19:34:36,047][root][INFO] - ===> Epoch[8123](40615/5): Loss 0.0090, Sem 0.0090 LR: 3.617e-02 Score 99.679 Data time: 0.2589, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.6985 [2021-07-08 19:34:49,570][root][INFO] - ===> Epoch[8124](40620/5): Loss 0.0089, Sem 0.0089 LR: 3.617e-02 Score 99.679 Data time: 0.2190, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.6568 [2021-07-08 19:35:04,483][root][INFO] - ===> Epoch[8125](40625/5): Loss 0.0088, Sem 0.0088 LR: 3.616e-02 Score 99.684 Data time: 0.2342, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.9349 [2021-07-08 19:35:19,559][root][INFO] - ===> Epoch[8126](40630/5): Loss 0.0086, Sem 0.0086 LR: 3.615e-02 Score 99.690 Data time: 0.2317, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.9721 [2021-07-08 19:35:33,816][root][INFO] - ===> Epoch[8127](40635/5): Loss 0.0086, Sem 0.0086 LR: 3.614e-02 Score 99.690 Data time: 0.2330, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.8057 [2021-07-08 19:35:48,306][root][INFO] - ===> Epoch[8128](40640/5): Loss 0.0089, Sem 0.0089 LR: 3.613e-02 Score 99.686 Data time: 0.2137, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.8562 [2021-07-08 19:36:01,122][root][INFO] - ===> Epoch[8129](40645/5): Loss 0.0086, Sem 0.0086 LR: 3.612e-02 Score 99.693 Data time: 0.2460, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.5122 [2021-07-08 19:36:16,092][root][INFO] - ===> Epoch[8130](40650/5): Loss 0.0087, Sem 0.0087 LR: 3.612e-02 Score 99.690 Data time: 0.2144, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.9491 [2021-07-08 19:36:30,556][root][INFO] - ===> Epoch[8131](40655/5): Loss 0.0087, Sem 0.0087 LR: 3.611e-02 Score 99.682 Data time: 0.2273, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.8489 [2021-07-08 19:36:45,980][root][INFO] - ===> Epoch[8132](40660/5): Loss 0.0087, Sem 0.0087 LR: 3.610e-02 Score 99.685 Data time: 0.4068, Forward time: 1.2549, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 3.0407 [2021-07-08 19:37:00,638][root][INFO] - ===> Epoch[8133](40665/5): Loss 0.0085, Sem 0.0085 LR: 3.609e-02 Score 99.690 Data time: 0.2197, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.8934 [2021-07-08 19:37:14,633][root][INFO] - ===> Epoch[8134](40670/5): Loss 0.0086, Sem 0.0086 LR: 3.608e-02 Score 99.690 Data time: 0.2808, Forward time: 1.2549, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7418 [2021-07-08 19:37:29,297][root][INFO] - ===> Epoch[8135](40675/5): Loss 0.0087, Sem 0.0087 LR: 3.607e-02 Score 99.690 Data time: 0.1401, Forward time: 1.2549, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.8964 [2021-07-08 19:37:44,304][root][INFO] - ===> Epoch[8136](40680/5): Loss 0.0083, Sem 0.0083 LR: 3.606e-02 Score 99.700 Data time: 0.2527, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.9599 [2021-07-08 19:37:58,434][root][INFO] - ===> Epoch[8137](40685/5): Loss 0.0091, Sem 0.0091 LR: 3.606e-02 Score 99.671 Data time: 0.2465, Forward time: 1.2549, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7795 [2021-07-08 19:38:13,582][root][INFO] - ===> Epoch[8138](40690/5): Loss 0.0085, Sem 0.0085 LR: 3.605e-02 Score 99.695 Data time: 0.1949, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.9848 [2021-07-08 19:38:27,089][root][INFO] - ===> Epoch[8139](40695/5): Loss 0.0091, Sem 0.0091 LR: 3.604e-02 Score 99.674 Data time: 0.1843, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.6622 [2021-07-08 19:38:41,047][root][INFO] - ===> Epoch[8140](40700/5): Loss 0.0092, Sem 0.0092 LR: 3.603e-02 Score 99.671 Data time: 0.2049, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7501 [2021-07-08 19:38:56,163][root][INFO] - ===> Epoch[8141](40705/5): Loss 0.0093, Sem 0.0093 LR: 3.602e-02 Score 99.671 Data time: 0.2133, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.9794 [2021-07-08 19:39:10,196][root][INFO] - ===> Epoch[8142](40710/5): Loss 0.0091, Sem 0.0091 LR: 3.601e-02 Score 99.670 Data time: 0.2292, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7647 [2021-07-08 19:39:24,108][root][INFO] - ===> Epoch[8143](40715/5): Loss 0.0090, Sem 0.0090 LR: 3.601e-02 Score 99.675 Data time: 0.1695, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7356 [2021-07-08 19:39:38,667][root][INFO] - ===> Epoch[8144](40720/5): Loss 0.0089, Sem 0.0089 LR: 3.600e-02 Score 99.679 Data time: 0.2265, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.8674 [2021-07-08 19:39:52,509][root][INFO] - ===> Epoch[8145](40725/5): Loss 0.0090, Sem 0.0090 LR: 3.599e-02 Score 99.676 Data time: 0.1947, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7213 [2021-07-08 19:40:05,895][root][INFO] - ===> Epoch[8146](40730/5): Loss 0.0090, Sem 0.0090 LR: 3.598e-02 Score 99.677 Data time: 0.1606, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.6332 [2021-07-08 19:40:20,516][root][INFO] - ===> Epoch[8147](40735/5): Loss 0.0091, Sem 0.0091 LR: 3.597e-02 Score 99.672 Data time: 0.2349, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8711 [2021-07-08 19:40:35,698][root][INFO] - ===> Epoch[8148](40740/5): Loss 0.0089, Sem 0.0089 LR: 3.596e-02 Score 99.681 Data time: 0.2189, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9928 [2021-07-08 19:40:49,604][root][INFO] - ===> Epoch[8149](40745/5): Loss 0.0088, Sem 0.0088 LR: 3.596e-02 Score 99.681 Data time: 0.1550, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7436 [2021-07-08 19:41:03,749][root][INFO] - ===> Epoch[8150](40750/5): Loss 0.0086, Sem 0.0086 LR: 3.595e-02 Score 99.692 Data time: 0.2540, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7789 [2021-07-08 19:41:17,289][root][INFO] - ===> Epoch[8151](40755/5): Loss 0.0088, Sem 0.0088 LR: 3.594e-02 Score 99.680 Data time: 0.1899, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6712 [2021-07-08 19:41:31,105][root][INFO] - ===> Epoch[8152](40760/5): Loss 0.0093, Sem 0.0093 LR: 3.593e-02 Score 99.670 Data time: 0.2519, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7035 [2021-07-08 19:41:45,483][root][INFO] - ===> Epoch[8153](40765/5): Loss 0.0088, Sem 0.0088 LR: 3.592e-02 Score 99.684 Data time: 0.2250, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8300 [2021-07-08 19:41:59,507][root][INFO] - ===> Epoch[8154](40770/5): Loss 0.0088, Sem 0.0088 LR: 3.591e-02 Score 99.685 Data time: 0.2247, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7672 [2021-07-08 19:42:13,159][root][INFO] - ===> Epoch[8155](40775/5): Loss 0.0087, Sem 0.0087 LR: 3.591e-02 Score 99.687 Data time: 0.2180, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6568 [2021-07-08 19:42:27,399][root][INFO] - ===> Epoch[8156](40780/5): Loss 0.0083, Sem 0.0083 LR: 3.590e-02 Score 99.697 Data time: 0.2600, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7998 [2021-07-08 19:42:41,619][root][INFO] - ===> Epoch[8157](40785/5): Loss 0.0086, Sem 0.0086 LR: 3.589e-02 Score 99.686 Data time: 0.2662, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7929 [2021-07-08 19:42:55,179][root][INFO] - ===> Epoch[8158](40790/5): Loss 0.0091, Sem 0.0091 LR: 3.588e-02 Score 99.673 Data time: 0.1690, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6759 [2021-07-08 19:43:09,872][root][INFO] - ===> Epoch[8159](40795/5): Loss 0.0084, Sem 0.0084 LR: 3.587e-02 Score 99.696 Data time: 0.2417, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8905 [2021-07-08 19:43:24,062][root][INFO] - ===> Epoch[8160](40800/5): Loss 0.0089, Sem 0.0089 LR: 3.586e-02 Score 99.680 Data time: 0.1978, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7977 [2021-07-08 19:43:24,064][root][INFO] - ===> Start testing [2021-07-08 19:44:07,319][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1298 Loss 0.368 (AVG: 0.597) Score 92.052 (AVG: 91.091) mIOU 71.579 mAP 83.077 mAcc 77.614 IOU: 64.795 0.000 84.205 75.885 94.840 91.238 30.889 85.388 98.628 80.416 79.804 86.085 58.348 mAP: 84.547 2.706 91.643 87.476 98.370 98.437 52.096 98.148 99.736 92.373 91.110 93.275 90.082 mAcc: 80.841 0.000 87.710 83.581 96.025 97.228 32.920 95.431 99.365 86.533 90.111 96.352 62.883 [2021-07-08 19:44:07,320][root][INFO] - Finished test. Elapsed time: 43.2553 [2021-07-08 19:44:08,522][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 19:44:21,054][root][INFO] - ===> Epoch[8161](40805/5): Loss 0.0087, Sem 0.0087 LR: 3.585e-02 Score 99.689 Data time: 0.1804, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.4749 [2021-07-08 19:44:34,956][root][INFO] - ===> Epoch[8162](40810/5): Loss 0.0086, Sem 0.0086 LR: 3.585e-02 Score 99.691 Data time: 0.2364, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7369 [2021-07-08 19:44:49,591][root][INFO] - ===> Epoch[8163](40815/5): Loss 0.0086, Sem 0.0086 LR: 3.584e-02 Score 99.690 Data time: 0.2214, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8838 [2021-07-08 19:45:03,793][root][INFO] - ===> Epoch[8164](40820/5): Loss 0.0085, Sem 0.0085 LR: 3.583e-02 Score 99.694 Data time: 0.2390, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7860 [2021-07-08 19:45:18,152][root][INFO] - ===> Epoch[8165](40825/5): Loss 0.0085, Sem 0.0085 LR: 3.582e-02 Score 99.694 Data time: 0.2286, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8206 [2021-07-08 19:45:32,352][root][INFO] - ===> Epoch[8166](40830/5): Loss 0.0087, Sem 0.0087 LR: 3.581e-02 Score 99.685 Data time: 0.2779, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7802 [2021-07-08 19:45:45,828][root][INFO] - ===> Epoch[8167](40835/5): Loss 0.0083, Sem 0.0083 LR: 3.580e-02 Score 99.699 Data time: 0.2383, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6508 [2021-07-08 19:46:00,313][root][INFO] - ===> Epoch[8168](40840/5): Loss 0.0088, Sem 0.0088 LR: 3.580e-02 Score 99.683 Data time: 0.2540, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8541 [2021-07-08 19:46:14,129][root][INFO] - ===> Epoch[8169](40845/5): Loss 0.0083, Sem 0.0083 LR: 3.579e-02 Score 99.700 Data time: 0.2602, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7084 [2021-07-08 19:46:28,356][root][INFO] - ===> Epoch[8170](40850/5): Loss 0.0089, Sem 0.0089 LR: 3.578e-02 Score 99.685 Data time: 0.2031, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7892 [2021-07-08 19:46:42,715][root][INFO] - ===> Epoch[8171](40855/5): Loss 0.0086, Sem 0.0086 LR: 3.577e-02 Score 99.688 Data time: 0.2153, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8269 [2021-07-08 19:46:56,525][root][INFO] - ===> Epoch[8172](40860/5): Loss 0.0085, Sem 0.0085 LR: 3.576e-02 Score 99.690 Data time: 0.2604, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7157 [2021-07-08 19:47:11,711][root][INFO] - ===> Epoch[8173](40865/5): Loss 0.0082, Sem 0.0082 LR: 3.575e-02 Score 99.705 Data time: 0.2441, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9904 [2021-07-08 19:47:25,435][root][INFO] - ===> Epoch[8174](40870/5): Loss 0.0089, Sem 0.0089 LR: 3.575e-02 Score 99.679 Data time: 0.2631, Forward time: 1.2549, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7027 [2021-07-08 19:47:39,519][root][INFO] - ===> Epoch[8175](40875/5): Loss 0.0085, Sem 0.0085 LR: 3.574e-02 Score 99.689 Data time: 0.2351, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7678 [2021-07-08 19:47:54,261][root][INFO] - ===> Epoch[8176](40880/5): Loss 0.0086, Sem 0.0086 LR: 3.573e-02 Score 99.690 Data time: 0.2853, Forward time: 1.2549, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9011 [2021-07-08 19:48:08,438][root][INFO] - ===> Epoch[8177](40885/5): Loss 0.0087, Sem 0.0087 LR: 3.572e-02 Score 99.687 Data time: 0.2557, Forward time: 1.2549, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7894 [2021-07-08 19:48:22,622][root][INFO] - ===> Epoch[8178](40890/5): Loss 0.0084, Sem 0.0084 LR: 3.571e-02 Score 99.697 Data time: 0.2650, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7914 [2021-07-08 19:48:35,556][root][INFO] - ===> Epoch[8179](40895/5): Loss 0.0091, Sem 0.0091 LR: 3.570e-02 Score 99.671 Data time: 0.2380, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.5353 [2021-07-08 19:48:49,553][root][INFO] - ===> Epoch[8180](40900/5): Loss 0.0089, Sem 0.0089 LR: 3.570e-02 Score 99.686 Data time: 0.2425, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7463 [2021-07-08 19:49:03,626][root][INFO] - ===> Epoch[8181](40905/5): Loss 0.0087, Sem 0.0087 LR: 3.569e-02 Score 99.685 Data time: 0.2249, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7700 [2021-07-08 19:49:16,608][root][INFO] - ===> Epoch[8182](40910/5): Loss 0.0091, Sem 0.0091 LR: 3.568e-02 Score 99.673 Data time: 0.2416, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.5528 [2021-07-08 19:49:30,314][root][INFO] - ===> Epoch[8183](40915/5): Loss 0.0092, Sem 0.0092 LR: 3.567e-02 Score 99.669 Data time: 0.2089, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7051 [2021-07-08 19:49:44,544][root][INFO] - ===> Epoch[8184](40920/5): Loss 0.0086, Sem 0.0086 LR: 3.566e-02 Score 99.689 Data time: 0.2481, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7871 [2021-07-08 19:49:58,534][root][INFO] - ===> Epoch[8185](40925/5): Loss 0.0084, Sem 0.0084 LR: 3.565e-02 Score 99.698 Data time: 0.1967, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7606 [2021-07-08 19:50:12,428][root][INFO] - ===> Epoch[8186](40930/5): Loss 0.0090, Sem 0.0090 LR: 3.564e-02 Score 99.675 Data time: 0.2351, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7400 [2021-07-08 19:50:25,948][root][INFO] - ===> Epoch[8187](40935/5): Loss 0.0086, Sem 0.0086 LR: 3.564e-02 Score 99.688 Data time: 0.1826, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.6537 [2021-07-08 19:50:39,621][root][INFO] - ===> Epoch[8188](40940/5): Loss 0.0084, Sem 0.0084 LR: 3.563e-02 Score 99.696 Data time: 0.2410, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.6898 [2021-07-08 19:50:53,969][root][INFO] - ===> Epoch[8189](40945/5): Loss 0.0089, Sem 0.0089 LR: 3.562e-02 Score 99.676 Data time: 0.2247, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.8271 [2021-07-08 19:51:08,930][root][INFO] - ===> Epoch[8190](40950/5): Loss 0.0084, Sem 0.0084 LR: 3.561e-02 Score 99.697 Data time: 0.1815, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.9496 [2021-07-08 19:51:23,173][root][INFO] - ===> Epoch[8191](40955/5): Loss 0.0088, Sem 0.0088 LR: 3.560e-02 Score 99.682 Data time: 0.2166, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.8088 [2021-07-08 19:51:37,094][root][INFO] - ===> Epoch[8192](40960/5): Loss 0.0090, Sem 0.0090 LR: 3.559e-02 Score 99.680 Data time: 0.1941, Forward time: 1.2549, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7343 [2021-07-08 19:51:50,940][root][INFO] - ===> Epoch[8193](40965/5): Loss 0.0085, Sem 0.0085 LR: 3.559e-02 Score 99.693 Data time: 0.2111, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7262 [2021-07-08 19:52:04,609][root][INFO] - ===> Epoch[8194](40970/5): Loss 0.0086, Sem 0.0086 LR: 3.558e-02 Score 99.687 Data time: 0.2255, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6922 [2021-07-08 19:52:18,065][root][INFO] - ===> Epoch[8195](40975/5): Loss 0.0086, Sem 0.0086 LR: 3.557e-02 Score 99.693 Data time: 0.2532, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.6468 [2021-07-08 19:52:31,899][root][INFO] - ===> Epoch[8196](40980/5): Loss 0.0103, Sem 0.0103 LR: 3.556e-02 Score 99.651 Data time: 0.2159, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7156 [2021-07-08 19:52:45,896][root][INFO] - ===> Epoch[8197](40985/5): Loss 0.0137, Sem 0.0137 LR: 3.555e-02 Score 99.538 Data time: 0.2206, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7527 [2021-07-08 19:52:59,308][root][INFO] - ===> Epoch[8198](40990/5): Loss 0.0127, Sem 0.0127 LR: 3.554e-02 Score 99.542 Data time: 0.1994, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.6460 [2021-07-08 19:53:12,637][root][INFO] - ===> Epoch[8199](40995/5): Loss 0.0131, Sem 0.0131 LR: 3.554e-02 Score 99.536 Data time: 0.2523, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.6218 [2021-07-08 19:53:26,934][root][INFO] - ===> Epoch[8200](41000/5): Loss 0.0124, Sem 0.0124 LR: 3.553e-02 Score 99.552 Data time: 0.2125, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.8174 [2021-07-08 19:53:26,935][root][INFO] - ===> Start testing [2021-07-08 19:54:02,513][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1233 Loss 0.311 (AVG: 0.619) Score 94.086 (AVG: 90.533) mIOU 69.509 mAP 82.184 mAcc 76.049 IOU: 63.905 0.000 82.722 74.730 94.794 89.856 19.223 81.433 98.441 75.512 78.732 85.233 59.035 mAP: 82.635 3.536 91.854 87.705 98.610 98.147 43.100 96.624 99.748 90.812 90.952 93.583 91.094 mAcc: 78.919 0.000 85.804 82.685 95.730 96.974 20.237 93.414 99.186 84.451 89.929 96.643 64.665 [2021-07-08 19:54:02,513][root][INFO] - Finished test. Elapsed time: 35.5770 [2021-07-08 19:54:03,594][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 19:54:15,745][root][INFO] - ===> Epoch[8201](41005/5): Loss 0.0113, Sem 0.0113 LR: 3.552e-02 Score 99.591 Data time: 0.1190, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.4000 [2021-07-08 19:54:28,999][root][INFO] - ===> Epoch[8202](41010/5): Loss 0.0112, Sem 0.0112 LR: 3.551e-02 Score 99.600 Data time: 0.2470, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.6079 [2021-07-08 19:54:42,994][root][INFO] - ===> Epoch[8203](41015/5): Loss 0.0106, Sem 0.0106 LR: 3.550e-02 Score 99.625 Data time: 0.1937, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7641 [2021-07-08 19:54:56,783][root][INFO] - ===> Epoch[8204](41020/5): Loss 0.0103, Sem 0.0103 LR: 3.549e-02 Score 99.633 Data time: 0.1744, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7164 [2021-07-08 19:55:11,079][root][INFO] - ===> Epoch[8205](41025/5): Loss 0.0124, Sem 0.0124 LR: 3.548e-02 Score 99.608 Data time: 0.2399, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.8143 [2021-07-08 19:55:25,163][root][INFO] - ===> Epoch[8206](41030/5): Loss 0.0101, Sem 0.0101 LR: 3.548e-02 Score 99.639 Data time: 0.2242, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7763 [2021-07-08 19:55:38,977][root][INFO] - ===> Epoch[8207](41035/5): Loss 0.0114, Sem 0.0114 LR: 3.547e-02 Score 99.620 Data time: 0.1981, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7047 [2021-07-08 19:55:52,209][root][INFO] - ===> Epoch[8208](41040/5): Loss 0.0191, Sem 0.0191 LR: 3.546e-02 Score 99.432 Data time: 0.2016, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.5903 [2021-07-08 19:56:06,057][root][INFO] - ===> Epoch[8209](41045/5): Loss 0.0268, Sem 0.0268 LR: 3.545e-02 Score 99.066 Data time: 0.2136, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7229 [2021-07-08 19:56:19,919][root][INFO] - ===> Epoch[8210](41050/5): Loss 0.0259, Sem 0.0259 LR: 3.544e-02 Score 99.161 Data time: 0.1906, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7160 [2021-07-08 19:56:33,850][root][INFO] - ===> Epoch[8211](41055/5): Loss 0.0238, Sem 0.0238 LR: 3.543e-02 Score 99.223 Data time: 0.1597, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7430 [2021-07-08 19:56:46,929][root][INFO] - ===> Epoch[8212](41060/5): Loss 0.0221, Sem 0.0221 LR: 3.543e-02 Score 99.267 Data time: 0.2611, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.5690 [2021-07-08 19:57:00,794][root][INFO] - ===> Epoch[8213](41065/5): Loss 0.0202, Sem 0.0202 LR: 3.542e-02 Score 99.319 Data time: 0.2154, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7294 [2021-07-08 19:57:15,469][root][INFO] - ===> Epoch[8214](41070/5): Loss 0.0186, Sem 0.0186 LR: 3.541e-02 Score 99.387 Data time: 0.2555, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.8831 [2021-07-08 19:57:29,874][root][INFO] - ===> Epoch[8215](41075/5): Loss 0.0174, Sem 0.0174 LR: 3.540e-02 Score 99.400 Data time: 0.3192, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.8340 [2021-07-08 19:57:44,350][root][INFO] - ===> Epoch[8216](41080/5): Loss 0.0158, Sem 0.0158 LR: 3.539e-02 Score 99.445 Data time: 0.2307, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.8469 [2021-07-08 19:57:59,133][root][INFO] - ===> Epoch[8217](41085/5): Loss 0.0152, Sem 0.0152 LR: 3.538e-02 Score 99.468 Data time: 0.2448, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.9202 [2021-07-08 19:58:12,745][root][INFO] - ===> Epoch[8218](41090/5): Loss 0.0141, Sem 0.0141 LR: 3.538e-02 Score 99.512 Data time: 0.1943, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.6886 [2021-07-08 19:58:27,547][root][INFO] - ===> Epoch[8219](41095/5): Loss 0.0136, Sem 0.0136 LR: 3.537e-02 Score 99.525 Data time: 0.2703, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.9120 [2021-07-08 19:58:40,833][root][INFO] - ===> Epoch[8220](41100/5): Loss 0.0139, Sem 0.0139 LR: 3.536e-02 Score 99.515 Data time: 0.2146, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.6235 [2021-07-08 19:58:56,176][root][INFO] - ===> Epoch[8221](41105/5): Loss 0.0129, Sem 0.0129 LR: 3.535e-02 Score 99.548 Data time: 0.2955, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 3.0215 [2021-07-08 19:59:09,641][root][INFO] - ===> Epoch[8222](41110/5): Loss 0.0130, Sem 0.0130 LR: 3.534e-02 Score 99.538 Data time: 0.2033, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.6540 [2021-07-08 19:59:23,648][root][INFO] - ===> Epoch[8223](41115/5): Loss 0.0128, Sem 0.0128 LR: 3.533e-02 Score 99.549 Data time: 0.2125, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7595 [2021-07-08 19:59:37,084][root][INFO] - ===> Epoch[8224](41120/5): Loss 0.0121, Sem 0.0121 LR: 3.532e-02 Score 99.573 Data time: 0.1839, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.6428 [2021-07-08 19:59:50,476][root][INFO] - ===> Epoch[8225](41125/5): Loss 0.0125, Sem 0.0125 LR: 3.532e-02 Score 99.556 Data time: 0.2183, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.6399 [2021-07-08 20:00:05,218][root][INFO] - ===> Epoch[8226](41130/5): Loss 0.0116, Sem 0.0116 LR: 3.531e-02 Score 99.592 Data time: 0.2269, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.8958 [2021-07-08 20:00:19,030][root][INFO] - ===> Epoch[8227](41135/5): Loss 0.0115, Sem 0.0115 LR: 3.530e-02 Score 99.587 Data time: 0.2468, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7162 [2021-07-08 20:00:33,443][root][INFO] - ===> Epoch[8228](41140/5): Loss 0.0116, Sem 0.0116 LR: 3.529e-02 Score 99.595 Data time: 0.2400, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.8334 [2021-07-08 20:00:47,005][root][INFO] - ===> Epoch[8229](41145/5): Loss 0.0117, Sem 0.0117 LR: 3.528e-02 Score 99.586 Data time: 0.2234, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.6698 [2021-07-08 20:01:00,970][root][INFO] - ===> Epoch[8230](41150/5): Loss 0.0115, Sem 0.0115 LR: 3.527e-02 Score 99.594 Data time: 0.2330, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7495 [2021-07-08 20:01:14,646][root][INFO] - ===> Epoch[8231](41155/5): Loss 0.0110, Sem 0.0110 LR: 3.527e-02 Score 99.610 Data time: 0.2435, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.6829 [2021-07-08 20:01:28,392][root][INFO] - ===> Epoch[8232](41160/5): Loss 0.0114, Sem 0.0114 LR: 3.526e-02 Score 99.593 Data time: 0.2667, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7009 [2021-07-08 20:01:42,376][root][INFO] - ===> Epoch[8233](41165/5): Loss 0.0103, Sem 0.0103 LR: 3.525e-02 Score 99.631 Data time: 0.2200, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7507 [2021-07-08 20:01:56,848][root][INFO] - ===> Epoch[8234](41170/5): Loss 0.0106, Sem 0.0106 LR: 3.524e-02 Score 99.621 Data time: 0.2760, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.8509 [2021-07-08 20:02:10,686][root][INFO] - ===> Epoch[8235](41175/5): Loss 0.0108, Sem 0.0108 LR: 3.523e-02 Score 99.615 Data time: 0.2455, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7213 [2021-07-08 20:02:24,001][root][INFO] - ===> Epoch[8236](41180/5): Loss 0.0106, Sem 0.0106 LR: 3.522e-02 Score 99.622 Data time: 0.2587, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.6107 [2021-07-08 20:02:37,198][root][INFO] - ===> Epoch[8237](41185/5): Loss 0.0107, Sem 0.0107 LR: 3.522e-02 Score 99.619 Data time: 0.2510, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.5907 [2021-07-08 20:02:51,395][root][INFO] - ===> Epoch[8238](41190/5): Loss 0.0109, Sem 0.0109 LR: 3.521e-02 Score 99.614 Data time: 0.1771, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7975 [2021-07-08 20:03:04,765][root][INFO] - ===> Epoch[8239](41195/5): Loss 0.0101, Sem 0.0101 LR: 3.520e-02 Score 99.638 Data time: 0.2865, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.6298 [2021-07-08 20:03:19,176][root][INFO] - ===> Epoch[8240](41200/5): Loss 0.0106, Sem 0.0106 LR: 3.519e-02 Score 99.624 Data time: 0.2362, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.8392 [2021-07-08 20:03:19,178][root][INFO] - ===> Start testing [2021-07-08 20:03:56,498][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1724 Loss 0.375 (AVG: 0.578) Score 91.435 (AVG: 90.853) mIOU 71.748 mAP 84.015 mAcc 77.825 IOU: 62.529 0.000 85.712 74.992 94.849 92.270 32.897 84.013 98.566 84.657 78.791 86.082 57.366 mAP: 82.373 6.545 92.851 87.079 98.932 98.504 55.544 98.394 99.818 95.323 91.612 94.207 91.016 mAcc: 77.994 0.000 88.320 82.377 96.237 96.792 35.496 95.699 99.417 90.578 91.591 96.693 60.531 [2021-07-08 20:03:56,498][root][INFO] - Finished test. Elapsed time: 37.3202 [2021-07-08 20:03:57,549][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 20:04:10,838][root][INFO] - ===> Epoch[8241](41205/5): Loss 0.0103, Sem 0.0103 LR: 3.518e-02 Score 99.632 Data time: 0.1763, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.6133 [2021-07-08 20:04:25,102][root][INFO] - ===> Epoch[8242](41210/5): Loss 0.0098, Sem 0.0098 LR: 3.517e-02 Score 99.646 Data time: 0.2169, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.8057 [2021-07-08 20:04:39,369][root][INFO] - ===> Epoch[8243](41215/5): Loss 0.0100, Sem 0.0100 LR: 3.516e-02 Score 99.642 Data time: 0.3329, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7919 [2021-07-08 20:04:53,994][root][INFO] - ===> Epoch[8244](41220/5): Loss 0.0101, Sem 0.0101 LR: 3.516e-02 Score 99.637 Data time: 0.2237, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.8761 [2021-07-08 20:05:08,130][root][INFO] - ===> Epoch[8245](41225/5): Loss 0.0099, Sem 0.0099 LR: 3.515e-02 Score 99.648 Data time: 0.2466, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7827 [2021-07-08 20:05:21,881][root][INFO] - ===> Epoch[8246](41230/5): Loss 0.0100, Sem 0.0100 LR: 3.514e-02 Score 99.642 Data time: 0.2511, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7165 [2021-07-08 20:05:36,704][root][INFO] - ===> Epoch[8247](41235/5): Loss 0.0099, Sem 0.0099 LR: 3.513e-02 Score 99.642 Data time: 0.1717, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.9060 [2021-07-08 20:05:50,487][root][INFO] - ===> Epoch[8248](41240/5): Loss 0.0100, Sem 0.0100 LR: 3.512e-02 Score 99.646 Data time: 0.1840, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7112 [2021-07-08 20:06:04,174][root][INFO] - ===> Epoch[8249](41245/5): Loss 0.0105, Sem 0.0105 LR: 3.511e-02 Score 99.629 Data time: 0.2081, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.6884 [2021-07-08 20:06:18,007][root][INFO] - ===> Epoch[8250](41250/5): Loss 0.0097, Sem 0.0097 LR: 3.511e-02 Score 99.651 Data time: 0.2269, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7179 [2021-07-08 20:06:31,982][root][INFO] - ===> Epoch[8251](41255/5): Loss 0.0099, Sem 0.0099 LR: 3.510e-02 Score 99.647 Data time: 0.1927, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7532 [2021-07-08 20:06:46,034][root][INFO] - ===> Epoch[8252](41260/5): Loss 0.0099, Sem 0.0099 LR: 3.509e-02 Score 99.646 Data time: 0.2318, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.7675 [2021-07-08 20:06:59,900][root][INFO] - ===> Epoch[8253](41265/5): Loss 0.0096, Sem 0.0096 LR: 3.508e-02 Score 99.654 Data time: 0.1596, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7261 [2021-07-08 20:07:13,960][root][INFO] - ===> Epoch[8254](41270/5): Loss 0.0098, Sem 0.0098 LR: 3.507e-02 Score 99.649 Data time: 0.2225, Forward time: 1.2547, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7667 [2021-07-08 20:07:27,954][root][INFO] - ===> Epoch[8255](41275/5): Loss 0.0106, Sem 0.0106 LR: 3.506e-02 Score 99.636 Data time: 0.2424, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7524 [2021-07-08 20:07:42,009][root][INFO] - ===> Epoch[8256](41280/5): Loss 0.0123, Sem 0.0123 LR: 3.506e-02 Score 99.614 Data time: 0.1944, Forward time: 1.2547, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7635 [2021-07-08 20:07:56,062][root][INFO] - ===> Epoch[8257](41285/5): Loss 0.0106, Sem 0.0106 LR: 3.505e-02 Score 99.621 Data time: 0.2723, Forward time: 1.2547, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.7671 [2021-07-08 20:08:10,877][root][INFO] - ===> Epoch[8258](41290/5): Loss 0.0113, Sem 0.0113 LR: 3.504e-02 Score 99.620 Data time: 0.2890, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.9183 [2021-07-08 20:08:25,376][root][INFO] - ===> Epoch[8259](41295/5): Loss 0.0107, Sem 0.0107 LR: 3.503e-02 Score 99.619 Data time: 0.2647, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.8554 [2021-07-08 20:08:40,370][root][INFO] - ===> Epoch[8260](41300/5): Loss 0.0118, Sem 0.0118 LR: 3.502e-02 Score 99.598 Data time: 0.3033, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.9436 [2021-07-08 20:08:54,656][root][INFO] - ===> Epoch[8261](41305/5): Loss 0.0115, Sem 0.0115 LR: 3.501e-02 Score 99.597 Data time: 0.2105, Forward time: 1.2548, Backward time: 1.2444, DDP time: 0.0042, Total iter time: 2.8216 [2021-07-08 20:09:09,758][root][INFO] - ===> Epoch[8262](41310/5): Loss 0.0107, Sem 0.0107 LR: 3.500e-02 Score 99.617 Data time: 0.2416, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.9776 [2021-07-08 20:09:24,901][root][INFO] - ===> Epoch[8263](41315/5): Loss 0.0113, Sem 0.0113 LR: 3.500e-02 Score 99.599 Data time: 0.2659, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.9895 [2021-07-08 20:09:39,294][root][INFO] - ===> Epoch[8264](41320/5): Loss 0.0103, Sem 0.0103 LR: 3.499e-02 Score 99.631 Data time: 0.1482, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.8516 [2021-07-08 20:09:53,733][root][INFO] - ===> Epoch[8265](41325/5): Loss 0.0114, Sem 0.0114 LR: 3.498e-02 Score 99.594 Data time: 0.1711, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.8571 [2021-07-08 20:10:08,536][root][INFO] - ===> Epoch[8266](41330/5): Loss 0.0111, Sem 0.0111 LR: 3.497e-02 Score 99.613 Data time: 0.3103, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.9137 [2021-07-08 20:10:22,347][root][INFO] - ===> Epoch[8267](41335/5): Loss 0.0114, Sem 0.0114 LR: 3.496e-02 Score 99.599 Data time: 0.2207, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7209 [2021-07-08 20:10:35,899][root][INFO] - ===> Epoch[8268](41340/5): Loss 0.0105, Sem 0.0105 LR: 3.495e-02 Score 99.621 Data time: 0.2501, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.6613 [2021-07-08 20:10:50,554][root][INFO] - ===> Epoch[8269](41345/5): Loss 0.0108, Sem 0.0108 LR: 3.495e-02 Score 99.614 Data time: 0.1938, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8762 [2021-07-08 20:11:03,729][root][INFO] - ===> Epoch[8270](41350/5): Loss 0.0105, Sem 0.0105 LR: 3.494e-02 Score 99.625 Data time: 0.1956, Forward time: 1.2548, Backward time: 1.2445, DDP time: 0.0042, Total iter time: 2.5763 [2021-07-08 20:11:19,538][root][INFO] - ===> Epoch[8271](41355/5): Loss 0.0104, Sem 0.0104 LR: 3.493e-02 Score 99.626 Data time: 0.2063, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 3.1110 [2021-07-08 20:11:33,155][root][INFO] - ===> Epoch[8272](41360/5): Loss 0.0101, Sem 0.0101 LR: 3.492e-02 Score 99.638 Data time: 0.2255, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6799 [2021-07-08 20:11:46,714][root][INFO] - ===> Epoch[8273](41365/5): Loss 0.0101, Sem 0.0101 LR: 3.491e-02 Score 99.638 Data time: 0.2763, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6608 [2021-07-08 20:12:01,448][root][INFO] - ===> Epoch[8274](41370/5): Loss 0.0102, Sem 0.0102 LR: 3.490e-02 Score 99.634 Data time: 0.2368, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9005 [2021-07-08 20:12:14,970][root][INFO] - ===> Epoch[8275](41375/5): Loss 0.0101, Sem 0.0101 LR: 3.490e-02 Score 99.641 Data time: 0.1884, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6622 [2021-07-08 20:12:29,370][root][INFO] - ===> Epoch[8276](41380/5): Loss 0.0100, Sem 0.0100 LR: 3.489e-02 Score 99.642 Data time: 0.2281, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8304 [2021-07-08 20:12:43,535][root][INFO] - ===> Epoch[8277](41385/5): Loss 0.0099, Sem 0.0099 LR: 3.488e-02 Score 99.645 Data time: 0.2381, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7906 [2021-07-08 20:12:57,338][root][INFO] - ===> Epoch[8278](41390/5): Loss 0.0098, Sem 0.0098 LR: 3.487e-02 Score 99.651 Data time: 0.1881, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7247 [2021-07-08 20:13:11,716][root][INFO] - ===> Epoch[8279](41395/5): Loss 0.0093, Sem 0.0093 LR: 3.486e-02 Score 99.666 Data time: 0.2359, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8219 [2021-07-08 20:13:25,741][root][INFO] - ===> Epoch[8280](41400/5): Loss 0.0096, Sem 0.0096 LR: 3.485e-02 Score 99.657 Data time: 0.1803, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7682 [2021-07-08 20:13:25,744][root][INFO] - ===> Start testing [2021-07-08 20:14:09,875][root][INFO] - 68/68: Data time: 0.1046, Iter time: 0.1276 Loss 0.364 (AVG: 0.619) Score 92.274 (AVG: 90.676) mIOU 71.341 mAP 83.343 mAcc 77.598 IOU: 64.056 0.000 84.334 75.240 93.469 91.985 34.095 83.089 96.778 81.824 78.972 85.885 57.703 mAP: 83.113 3.825 89.350 86.684 98.280 98.523 60.115 97.745 99.147 91.401 91.104 94.190 89.984 mAcc: 79.847 0.000 86.668 82.260 94.894 97.151 37.820 93.246 99.346 88.605 91.077 96.551 61.305 [2021-07-08 20:14:09,875][root][INFO] - Finished test. Elapsed time: 44.1306 [2021-07-08 20:14:11,148][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 20:14:24,993][root][INFO] - ===> Epoch[8281](41405/5): Loss 0.0092, Sem 0.0092 LR: 3.484e-02 Score 99.667 Data time: 0.1835, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7302 [2021-07-08 20:14:38,949][root][INFO] - ===> Epoch[8282](41410/5): Loss 0.0091, Sem 0.0091 LR: 3.484e-02 Score 99.671 Data time: 0.1902, Forward time: 1.2548, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7452 [2021-07-08 20:14:52,908][root][INFO] - ===> Epoch[8283](41415/5): Loss 0.0092, Sem 0.0092 LR: 3.483e-02 Score 99.675 Data time: 0.1911, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7555 [2021-07-08 20:15:06,846][root][INFO] - ===> Epoch[8284](41420/5): Loss 0.0097, Sem 0.0097 LR: 3.482e-02 Score 99.651 Data time: 0.2464, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7262 [2021-07-08 20:15:21,504][root][INFO] - ===> Epoch[8285](41425/5): Loss 0.0094, Sem 0.0094 LR: 3.481e-02 Score 99.664 Data time: 0.2360, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8822 [2021-07-08 20:15:36,606][root][INFO] - ===> Epoch[8286](41430/5): Loss 0.0095, Sem 0.0095 LR: 3.480e-02 Score 99.657 Data time: 0.2566, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.9778 [2021-07-08 20:15:50,236][root][INFO] - ===> Epoch[8287](41435/5): Loss 0.0097, Sem 0.0097 LR: 3.479e-02 Score 99.653 Data time: 0.2345, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6769 [2021-07-08 20:16:03,733][root][INFO] - ===> Epoch[8288](41440/5): Loss 0.0097, Sem 0.0097 LR: 3.479e-02 Score 99.651 Data time: 0.2044, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6543 [2021-07-08 20:16:18,336][root][INFO] - ===> Epoch[8289](41445/5): Loss 0.0094, Sem 0.0094 LR: 3.478e-02 Score 99.660 Data time: 0.2713, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8654 [2021-07-08 20:16:31,811][root][INFO] - ===> Epoch[8290](41450/5): Loss 0.0092, Sem 0.0092 LR: 3.477e-02 Score 99.675 Data time: 0.2011, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6545 [2021-07-08 20:16:45,886][root][INFO] - ===> Epoch[8291](41455/5): Loss 0.0098, Sem 0.0098 LR: 3.476e-02 Score 99.651 Data time: 0.1529, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7837 [2021-07-08 20:16:59,875][root][INFO] - ===> Epoch[8292](41460/5): Loss 0.0098, Sem 0.0098 LR: 3.475e-02 Score 99.651 Data time: 0.1918, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7603 [2021-07-08 20:17:14,340][root][INFO] - ===> Epoch[8293](41465/5): Loss 0.0094, Sem 0.0094 LR: 3.474e-02 Score 99.665 Data time: 0.2191, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8543 [2021-07-08 20:17:27,719][root][INFO] - ===> Epoch[8294](41470/5): Loss 0.0098, Sem 0.0098 LR: 3.473e-02 Score 99.647 Data time: 0.2199, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6297 [2021-07-08 20:17:41,901][root][INFO] - ===> Epoch[8295](41475/5): Loss 0.0093, Sem 0.0093 LR: 3.473e-02 Score 99.670 Data time: 0.2156, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7905 [2021-07-08 20:17:55,804][root][INFO] - ===> Epoch[8296](41480/5): Loss 0.0095, Sem 0.0095 LR: 3.472e-02 Score 99.657 Data time: 0.2346, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7214 [2021-07-08 20:18:09,145][root][INFO] - ===> Epoch[8297](41485/5): Loss 0.0091, Sem 0.0091 LR: 3.471e-02 Score 99.674 Data time: 0.2549, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6185 [2021-07-08 20:18:23,491][root][INFO] - ===> Epoch[8298](41490/5): Loss 0.0093, Sem 0.0093 LR: 3.470e-02 Score 99.666 Data time: 0.2375, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8096 [2021-07-08 20:18:37,534][root][INFO] - ===> Epoch[8299](41495/5): Loss 0.0098, Sem 0.0098 LR: 3.469e-02 Score 99.650 Data time: 0.2660, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7573 [2021-07-08 20:18:51,739][root][INFO] - ===> Epoch[8300](41500/5): Loss 0.0095, Sem 0.0095 LR: 3.468e-02 Score 99.663 Data time: 0.2819, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7981 [2021-07-08 20:19:05,699][root][INFO] - ===> Epoch[8301](41505/5): Loss 0.0094, Sem 0.0094 LR: 3.468e-02 Score 99.664 Data time: 0.2542, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7479 [2021-07-08 20:19:19,413][root][INFO] - ===> Epoch[8302](41510/5): Loss 0.0094, Sem 0.0094 LR: 3.467e-02 Score 99.663 Data time: 0.2725, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6983 [2021-07-08 20:19:33,002][root][INFO] - ===> Epoch[8303](41515/5): Loss 0.0099, Sem 0.0099 LR: 3.466e-02 Score 99.647 Data time: 0.2517, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6796 [2021-07-08 20:19:46,447][root][INFO] - ===> Epoch[8304](41520/5): Loss 0.0102, Sem 0.0102 LR: 3.465e-02 Score 99.638 Data time: 0.2471, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6537 [2021-07-08 20:19:59,865][root][INFO] - ===> Epoch[8305](41525/5): Loss 0.0105, Sem 0.0105 LR: 3.464e-02 Score 99.624 Data time: 0.2971, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6352 [2021-07-08 20:20:14,417][root][INFO] - ===> Epoch[8306](41530/5): Loss 0.0097, Sem 0.0097 LR: 3.463e-02 Score 99.653 Data time: 0.2019, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8588 [2021-07-08 20:20:29,051][root][INFO] - ===> Epoch[8307](41535/5): Loss 0.0095, Sem 0.0095 LR: 3.463e-02 Score 99.656 Data time: 0.2266, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8688 [2021-07-08 20:20:42,522][root][INFO] - ===> Epoch[8308](41540/5): Loss 0.0100, Sem 0.0100 LR: 3.462e-02 Score 99.644 Data time: 0.2624, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6436 [2021-07-08 20:20:56,897][root][INFO] - ===> Epoch[8309](41545/5): Loss 0.0098, Sem 0.0098 LR: 3.461e-02 Score 99.647 Data time: 0.1715, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8400 [2021-07-08 20:21:10,824][root][INFO] - ===> Epoch[8310](41550/5): Loss 0.0095, Sem 0.0095 LR: 3.460e-02 Score 99.659 Data time: 0.2334, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7400 [2021-07-08 20:21:24,200][root][INFO] - ===> Epoch[8311](41555/5): Loss 0.0097, Sem 0.0097 LR: 3.459e-02 Score 99.653 Data time: 0.2425, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6232 [2021-07-08 20:21:37,978][root][INFO] - ===> Epoch[8312](41560/5): Loss 0.0096, Sem 0.0096 LR: 3.458e-02 Score 99.654 Data time: 0.2548, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7076 [2021-07-08 20:21:51,897][root][INFO] - ===> Epoch[8313](41565/5): Loss 0.0098, Sem 0.0098 LR: 3.457e-02 Score 99.653 Data time: 0.2142, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7333 [2021-07-08 20:22:06,157][root][INFO] - ===> Epoch[8314](41570/5): Loss 0.0108, Sem 0.0108 LR: 3.457e-02 Score 99.633 Data time: 0.2512, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8027 [2021-07-08 20:22:19,837][root][INFO] - ===> Epoch[8315](41575/5): Loss 0.0320, Sem 0.0320 LR: 3.456e-02 Score 99.049 Data time: 0.2442, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6904 [2021-07-08 20:22:34,406][root][INFO] - ===> Epoch[8316](41580/5): Loss 0.0360, Sem 0.0360 LR: 3.455e-02 Score 98.893 Data time: 0.2038, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8714 [2021-07-08 20:22:49,445][root][INFO] - ===> Epoch[8317](41585/5): Loss 0.0360, Sem 0.0360 LR: 3.454e-02 Score 98.909 Data time: 0.2236, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.9594 [2021-07-08 20:23:02,842][root][INFO] - ===> Epoch[8318](41590/5): Loss 0.0360, Sem 0.0360 LR: 3.453e-02 Score 98.841 Data time: 0.2539, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6285 [2021-07-08 20:23:16,916][root][INFO] - ===> Epoch[8319](41595/5): Loss 0.0336, Sem 0.0336 LR: 3.452e-02 Score 98.910 Data time: 0.1924, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7628 [2021-07-08 20:23:31,731][root][INFO] - ===> Epoch[8320](41600/5): Loss 0.0311, Sem 0.0311 LR: 3.452e-02 Score 98.979 Data time: 0.2640, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.9148 [2021-07-08 20:23:31,732][root][INFO] - ===> Start testing [2021-07-08 20:24:14,074][root][INFO] - 68/68: Data time: 0.0471, Iter time: 0.1252 Loss 0.752 (AVG: 0.580) Score 84.152 (AVG: 89.021) mIOU 68.153 mAP 84.129 mAcc 75.788 IOU: 57.353 0.126 80.323 69.491 90.265 88.086 32.262 73.914 97.461 75.650 77.943 85.556 57.559 mAP: 80.120 20.677 95.438 88.700 98.665 97.953 58.640 95.337 99.380 84.977 91.143 94.093 88.546 mAcc: 72.045 0.638 89.313 85.803 90.853 96.453 34.522 91.964 98.726 81.255 86.171 96.095 61.404 [2021-07-08 20:24:14,074][root][INFO] - Finished test. Elapsed time: 42.3409 [2021-07-08 20:24:15,115][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 20:24:28,173][root][INFO] - ===> Epoch[8321](41605/5): Loss 0.0269, Sem 0.0269 LR: 3.451e-02 Score 99.100 Data time: 0.1687, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.5795 [2021-07-08 20:24:42,683][root][INFO] - ===> Epoch[8322](41610/5): Loss 0.0236, Sem 0.0236 LR: 3.450e-02 Score 99.204 Data time: 0.2051, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8464 [2021-07-08 20:24:57,359][root][INFO] - ===> Epoch[8323](41615/5): Loss 0.0225, Sem 0.0225 LR: 3.449e-02 Score 99.251 Data time: 0.2686, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8906 [2021-07-08 20:25:11,716][root][INFO] - ===> Epoch[8324](41620/5): Loss 0.0201, Sem 0.0201 LR: 3.448e-02 Score 99.309 Data time: 0.1966, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8207 [2021-07-08 20:25:25,618][root][INFO] - ===> Epoch[8325](41625/5): Loss 0.0194, Sem 0.0194 LR: 3.447e-02 Score 99.330 Data time: 0.1659, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7312 [2021-07-08 20:25:39,384][root][INFO] - ===> Epoch[8326](41630/5): Loss 0.0196, Sem 0.0196 LR: 3.446e-02 Score 99.339 Data time: 0.2082, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7052 [2021-07-08 20:25:53,606][root][INFO] - ===> Epoch[8327](41635/5): Loss 0.0183, Sem 0.0183 LR: 3.446e-02 Score 99.369 Data time: 0.2190, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8038 [2021-07-08 20:26:07,654][root][INFO] - ===> Epoch[8328](41640/5): Loss 0.0174, Sem 0.0174 LR: 3.445e-02 Score 99.395 Data time: 0.1939, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7728 [2021-07-08 20:26:21,442][root][INFO] - ===> Epoch[8329](41645/5): Loss 0.0187, Sem 0.0187 LR: 3.444e-02 Score 99.361 Data time: 0.2219, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7111 [2021-07-08 20:26:35,060][root][INFO] - ===> Epoch[8330](41650/5): Loss 0.0195, Sem 0.0195 LR: 3.443e-02 Score 99.342 Data time: 0.2075, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6765 [2021-07-08 20:26:49,417][root][INFO] - ===> Epoch[8331](41655/5): Loss 0.0194, Sem 0.0194 LR: 3.442e-02 Score 99.332 Data time: 0.2150, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8240 [2021-07-08 20:27:03,592][root][INFO] - ===> Epoch[8332](41660/5): Loss 0.0180, Sem 0.0180 LR: 3.441e-02 Score 99.384 Data time: 0.2530, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7911 [2021-07-08 20:27:18,168][root][INFO] - ===> Epoch[8333](41665/5): Loss 0.0178, Sem 0.0178 LR: 3.441e-02 Score 99.392 Data time: 0.2767, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8680 [2021-07-08 20:27:32,071][root][INFO] - ===> Epoch[8334](41670/5): Loss 0.0167, Sem 0.0167 LR: 3.440e-02 Score 99.411 Data time: 0.2578, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7330 [2021-07-08 20:27:45,553][root][INFO] - ===> Epoch[8335](41675/5): Loss 0.0178, Sem 0.0178 LR: 3.439e-02 Score 99.396 Data time: 0.2268, Forward time: 1.2546, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6495 [2021-07-08 20:27:59,740][root][INFO] - ===> Epoch[8336](41680/5): Loss 0.0180, Sem 0.0180 LR: 3.438e-02 Score 99.388 Data time: 0.2436, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7959 [2021-07-08 20:28:13,524][root][INFO] - ===> Epoch[8337](41685/5): Loss 0.0171, Sem 0.0171 LR: 3.437e-02 Score 99.420 Data time: 0.2400, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6962 [2021-07-08 20:28:26,623][root][INFO] - ===> Epoch[8338](41690/5): Loss 0.0161, Sem 0.0161 LR: 3.436e-02 Score 99.431 Data time: 0.1881, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.5797 [2021-07-08 20:28:40,827][root][INFO] - ===> Epoch[8339](41695/5): Loss 0.0147, Sem 0.0147 LR: 3.436e-02 Score 99.483 Data time: 0.2098, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7910 [2021-07-08 20:28:55,208][root][INFO] - ===> Epoch[8340](41700/5): Loss 0.0160, Sem 0.0160 LR: 3.435e-02 Score 99.451 Data time: 0.2426, Forward time: 1.2547, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8335 [2021-07-08 20:29:09,097][root][INFO] - ===> Epoch[8341](41705/5): Loss 0.0149, Sem 0.0149 LR: 3.434e-02 Score 99.475 Data time: 0.2540, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7336 [2021-07-08 20:29:23,957][root][INFO] - ===> Epoch[8342](41710/5): Loss 0.0140, Sem 0.0140 LR: 3.433e-02 Score 99.499 Data time: 0.2583, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9229 [2021-07-08 20:29:38,026][root][INFO] - ===> Epoch[8343](41715/5): Loss 0.0140, Sem 0.0140 LR: 3.432e-02 Score 99.502 Data time: 0.2273, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7706 [2021-07-08 20:29:52,499][root][INFO] - ===> Epoch[8344](41720/5): Loss 0.0139, Sem 0.0139 LR: 3.431e-02 Score 99.503 Data time: 0.2498, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8406 [2021-07-08 20:30:06,406][root][INFO] - ===> Epoch[8345](41725/5): Loss 0.0131, Sem 0.0131 LR: 3.430e-02 Score 99.536 Data time: 0.2417, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7397 [2021-07-08 20:30:20,847][root][INFO] - ===> Epoch[8346](41730/5): Loss 0.0123, Sem 0.0123 LR: 3.430e-02 Score 99.561 Data time: 0.2588, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8448 [2021-07-08 20:30:34,849][root][INFO] - ===> Epoch[8347](41735/5): Loss 0.0133, Sem 0.0133 LR: 3.429e-02 Score 99.531 Data time: 0.2404, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7455 [2021-07-08 20:30:48,320][root][INFO] - ===> Epoch[8348](41740/5): Loss 0.0133, Sem 0.0133 LR: 3.428e-02 Score 99.531 Data time: 0.2319, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6418 [2021-07-08 20:31:02,195][root][INFO] - ===> Epoch[8349](41745/5): Loss 0.0132, Sem 0.0132 LR: 3.427e-02 Score 99.543 Data time: 0.2526, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7253 [2021-07-08 20:31:15,841][root][INFO] - ===> Epoch[8350](41750/5): Loss 0.0146, Sem 0.0146 LR: 3.426e-02 Score 99.517 Data time: 0.2361, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6827 [2021-07-08 20:31:29,342][root][INFO] - ===> Epoch[8351](41755/5): Loss 0.0138, Sem 0.0138 LR: 3.425e-02 Score 99.512 Data time: 0.2365, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6571 [2021-07-08 20:31:43,548][root][INFO] - ===> Epoch[8352](41760/5): Loss 0.0144, Sem 0.0144 LR: 3.425e-02 Score 99.497 Data time: 0.1804, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7882 [2021-07-08 20:31:57,958][root][INFO] - ===> Epoch[8353](41765/5): Loss 0.0142, Sem 0.0142 LR: 3.424e-02 Score 99.498 Data time: 0.2010, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8338 [2021-07-08 20:32:11,790][root][INFO] - ===> Epoch[8354](41770/5): Loss 0.0149, Sem 0.0149 LR: 3.423e-02 Score 99.490 Data time: 0.2039, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7343 [2021-07-08 20:32:25,499][root][INFO] - ===> Epoch[8355](41775/5): Loss 0.0164, Sem 0.0164 LR: 3.422e-02 Score 99.459 Data time: 0.2219, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6945 [2021-07-08 20:32:38,794][root][INFO] - ===> Epoch[8356](41780/5): Loss 0.0139, Sem 0.0139 LR: 3.421e-02 Score 99.513 Data time: 0.2184, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6163 [2021-07-08 20:32:53,095][root][INFO] - ===> Epoch[8357](41785/5): Loss 0.0134, Sem 0.0134 LR: 3.420e-02 Score 99.520 Data time: 0.2125, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8112 [2021-07-08 20:33:06,744][root][INFO] - ===> Epoch[8358](41790/5): Loss 0.0137, Sem 0.0137 LR: 3.419e-02 Score 99.511 Data time: 0.2403, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6825 [2021-07-08 20:33:21,147][root][INFO] - ===> Epoch[8359](41795/5): Loss 0.0125, Sem 0.0125 LR: 3.419e-02 Score 99.560 Data time: 0.2301, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8458 [2021-07-08 20:33:34,958][root][INFO] - ===> Epoch[8360](41800/5): Loss 0.0132, Sem 0.0132 LR: 3.418e-02 Score 99.535 Data time: 0.2235, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7138 [2021-07-08 20:33:34,959][root][INFO] - ===> Start testing [2021-07-08 20:34:17,687][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1201 Loss 0.355 (AVG: 0.599) Score 90.632 (AVG: 90.506) mIOU 70.061 mAP 82.419 mAcc 76.324 IOU: 64.381 0.000 79.116 74.769 94.518 91.443 29.456 78.435 98.395 80.344 80.304 85.242 54.386 mAP: 84.759 3.464 87.444 85.183 98.661 98.237 55.797 96.188 99.779 87.077 93.030 93.940 87.884 mAcc: 81.751 0.000 83.979 81.661 95.918 95.782 32.022 91.588 99.170 86.582 90.521 96.363 56.879 [2021-07-08 20:34:17,687][root][INFO] - Finished test. Elapsed time: 42.7269 [2021-07-08 20:34:18,754][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 20:34:31,852][root][INFO] - ===> Epoch[8361](41805/5): Loss 0.0127, Sem 0.0127 LR: 3.417e-02 Score 99.543 Data time: 0.1635, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.5869 [2021-07-08 20:34:45,983][root][INFO] - ===> Epoch[8362](41810/5): Loss 0.0163, Sem 0.0163 LR: 3.416e-02 Score 99.461 Data time: 0.2286, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7719 [2021-07-08 20:34:59,405][root][INFO] - ===> Epoch[8363](41815/5): Loss 0.0157, Sem 0.0157 LR: 3.415e-02 Score 99.447 Data time: 0.1918, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6386 [2021-07-08 20:35:13,637][root][INFO] - ===> Epoch[8364](41820/5): Loss 0.0179, Sem 0.0179 LR: 3.414e-02 Score 99.402 Data time: 0.2248, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7978 [2021-07-08 20:35:26,583][root][INFO] - ===> Epoch[8365](41825/5): Loss 0.0191, Sem 0.0191 LR: 3.414e-02 Score 99.343 Data time: 0.2302, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.5492 [2021-07-08 20:35:40,133][root][INFO] - ===> Epoch[8366](41830/5): Loss 0.0164, Sem 0.0164 LR: 3.413e-02 Score 99.431 Data time: 0.2169, Forward time: 1.2547, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6672 [2021-07-08 20:35:53,748][root][INFO] - ===> Epoch[8367](41835/5): Loss 0.0168, Sem 0.0168 LR: 3.412e-02 Score 99.414 Data time: 0.1601, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6704 [2021-07-08 20:36:08,025][root][INFO] - ===> Epoch[8368](41840/5): Loss 0.0169, Sem 0.0169 LR: 3.411e-02 Score 99.410 Data time: 0.2248, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8040 [2021-07-08 20:36:22,672][root][INFO] - ===> Epoch[8369](41845/5): Loss 0.0164, Sem 0.0164 LR: 3.410e-02 Score 99.417 Data time: 0.2364, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8698 [2021-07-08 20:36:36,750][root][INFO] - ===> Epoch[8370](41850/5): Loss 0.0158, Sem 0.0158 LR: 3.409e-02 Score 99.439 Data time: 0.2046, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7726 [2021-07-08 20:36:49,826][root][INFO] - ===> Epoch[8371](41855/5): Loss 0.0159, Sem 0.0159 LR: 3.408e-02 Score 99.435 Data time: 0.2000, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.5741 [2021-07-08 20:37:03,466][root][INFO] - ===> Epoch[8372](41860/5): Loss 0.0155, Sem 0.0155 LR: 3.408e-02 Score 99.449 Data time: 0.1919, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6859 [2021-07-08 20:37:17,227][root][INFO] - ===> Epoch[8373](41865/5): Loss 0.0137, Sem 0.0137 LR: 3.407e-02 Score 99.517 Data time: 0.2453, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7045 [2021-07-08 20:37:30,611][root][INFO] - ===> Epoch[8374](41870/5): Loss 0.0131, Sem 0.0131 LR: 3.406e-02 Score 99.528 Data time: 0.2266, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6290 [2021-07-08 20:37:44,417][root][INFO] - ===> Epoch[8375](41875/5): Loss 0.0141, Sem 0.0141 LR: 3.405e-02 Score 99.504 Data time: 0.1866, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7165 [2021-07-08 20:37:58,575][root][INFO] - ===> Epoch[8376](41880/5): Loss 0.0135, Sem 0.0135 LR: 3.404e-02 Score 99.528 Data time: 0.2210, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7890 [2021-07-08 20:38:12,801][root][INFO] - ===> Epoch[8377](41885/5): Loss 0.0137, Sem 0.0137 LR: 3.403e-02 Score 99.523 Data time: 0.2100, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8052 [2021-07-08 20:38:26,611][root][INFO] - ===> Epoch[8378](41890/5): Loss 0.0145, Sem 0.0145 LR: 3.403e-02 Score 99.520 Data time: 0.2066, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7144 [2021-07-08 20:38:40,772][root][INFO] - ===> Epoch[8379](41895/5): Loss 0.0138, Sem 0.0138 LR: 3.402e-02 Score 99.515 Data time: 0.1945, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7808 [2021-07-08 20:38:54,678][root][INFO] - ===> Epoch[8380](41900/5): Loss 0.0127, Sem 0.0127 LR: 3.401e-02 Score 99.548 Data time: 0.1843, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7381 [2021-07-08 20:39:08,661][root][INFO] - ===> Epoch[8381](41905/5): Loss 0.0135, Sem 0.0135 LR: 3.400e-02 Score 99.531 Data time: 0.2078, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7439 [2021-07-08 20:39:22,384][root][INFO] - ===> Epoch[8382](41910/5): Loss 0.0126, Sem 0.0126 LR: 3.399e-02 Score 99.548 Data time: 0.2090, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6940 [2021-07-08 20:39:35,618][root][INFO] - ===> Epoch[8383](41915/5): Loss 0.0143, Sem 0.0143 LR: 3.398e-02 Score 99.505 Data time: 0.2125, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.5992 [2021-07-08 20:39:48,838][root][INFO] - ===> Epoch[8384](41920/5): Loss 0.0133, Sem 0.0133 LR: 3.397e-02 Score 99.526 Data time: 0.2065, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6029 [2021-07-08 20:40:02,940][root][INFO] - ===> Epoch[8385](41925/5): Loss 0.0133, Sem 0.0133 LR: 3.397e-02 Score 99.545 Data time: 0.2461, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7790 [2021-07-08 20:40:17,232][root][INFO] - ===> Epoch[8386](41930/5): Loss 0.0136, Sem 0.0136 LR: 3.396e-02 Score 99.522 Data time: 0.2855, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8031 [2021-07-08 20:40:31,710][root][INFO] - ===> Epoch[8387](41935/5): Loss 0.0128, Sem 0.0128 LR: 3.395e-02 Score 99.550 Data time: 0.2280, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8464 [2021-07-08 20:40:46,546][root][INFO] - ===> Epoch[8388](41940/5): Loss 0.0134, Sem 0.0134 LR: 3.394e-02 Score 99.550 Data time: 0.2317, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9273 [2021-07-08 20:41:00,538][root][INFO] - ===> Epoch[8389](41945/5): Loss 0.0132, Sem 0.0132 LR: 3.393e-02 Score 99.530 Data time: 0.2580, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7519 [2021-07-08 20:41:14,619][root][INFO] - ===> Epoch[8390](41950/5): Loss 0.0126, Sem 0.0126 LR: 3.392e-02 Score 99.551 Data time: 0.2653, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7729 [2021-07-08 20:41:28,797][root][INFO] - ===> Epoch[8391](41955/5): Loss 0.0116, Sem 0.0116 LR: 3.392e-02 Score 99.583 Data time: 0.2694, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7903 [2021-07-08 20:41:44,033][root][INFO] - ===> Epoch[8392](41960/5): Loss 0.0119, Sem 0.0119 LR: 3.391e-02 Score 99.568 Data time: 0.2928, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9849 [2021-07-08 20:41:56,970][root][INFO] - ===> Epoch[8393](41965/5): Loss 0.0123, Sem 0.0123 LR: 3.390e-02 Score 99.558 Data time: 0.1871, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.5458 [2021-07-08 20:42:10,905][root][INFO] - ===> Epoch[8394](41970/5): Loss 0.0120, Sem 0.0120 LR: 3.389e-02 Score 99.567 Data time: 0.2295, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7444 [2021-07-08 20:42:24,251][root][INFO] - ===> Epoch[8395](41975/5): Loss 0.0114, Sem 0.0114 LR: 3.388e-02 Score 99.589 Data time: 0.2302, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6247 [2021-07-08 20:42:38,558][root][INFO] - ===> Epoch[8396](41980/5): Loss 0.0111, Sem 0.0111 LR: 3.387e-02 Score 99.597 Data time: 0.2324, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8206 [2021-07-08 20:42:51,910][root][INFO] - ===> Epoch[8397](41985/5): Loss 0.0117, Sem 0.0117 LR: 3.386e-02 Score 99.586 Data time: 0.2235, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6244 [2021-07-08 20:43:05,718][root][INFO] - ===> Epoch[8398](41990/5): Loss 0.0106, Sem 0.0106 LR: 3.386e-02 Score 99.619 Data time: 0.2054, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7102 [2021-07-08 20:43:19,350][root][INFO] - ===> Epoch[8399](41995/5): Loss 0.0113, Sem 0.0113 LR: 3.385e-02 Score 99.596 Data time: 0.2362, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6808 [2021-07-08 20:43:32,849][root][INFO] - ===> Epoch[8400](42000/5): Loss 0.0109, Sem 0.0109 LR: 3.384e-02 Score 99.611 Data time: 0.2291, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6440 [2021-07-08 20:43:32,851][root][INFO] - ===> Start testing [2021-07-08 20:44:11,332][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1330 Loss 0.334 (AVG: 0.576) Score 92.704 (AVG: 90.728) mIOU 70.014 mAP 83.375 mAcc 76.426 IOU: 64.525 0.000 80.003 74.814 94.841 90.936 20.805 81.184 98.484 79.745 81.446 85.397 58.008 mAP: 83.565 6.487 92.054 86.121 98.438 98.676 48.810 96.650 99.795 94.980 92.891 94.471 90.943 mAcc: 80.355 0.000 86.060 83.452 96.414 97.155 22.132 91.616 99.299 88.753 90.870 96.309 61.128 [2021-07-08 20:44:11,333][root][INFO] - Finished test. Elapsed time: 38.4812 [2021-07-08 20:44:12,468][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 20:44:26,205][root][INFO] - ===> Epoch[8401](42005/5): Loss 0.0108, Sem 0.0108 LR: 3.383e-02 Score 99.612 Data time: 0.2025, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7054 [2021-07-08 20:44:40,742][root][INFO] - ===> Epoch[8402](42010/5): Loss 0.0108, Sem 0.0108 LR: 3.382e-02 Score 99.612 Data time: 0.2094, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8691 [2021-07-08 20:44:55,493][root][INFO] - ===> Epoch[8403](42015/5): Loss 0.0108, Sem 0.0108 LR: 3.381e-02 Score 99.615 Data time: 0.2172, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8974 [2021-07-08 20:45:09,863][root][INFO] - ===> Epoch[8404](42020/5): Loss 0.0105, Sem 0.0105 LR: 3.381e-02 Score 99.621 Data time: 0.2431, Forward time: 1.2546, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8329 [2021-07-08 20:45:23,503][root][INFO] - ===> Epoch[8405](42025/5): Loss 0.0105, Sem 0.0105 LR: 3.380e-02 Score 99.624 Data time: 0.2310, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6747 [2021-07-08 20:45:38,335][root][INFO] - ===> Epoch[8406](42030/5): Loss 0.0106, Sem 0.0106 LR: 3.379e-02 Score 99.621 Data time: 0.2451, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9173 [2021-07-08 20:45:51,767][root][INFO] - ===> Epoch[8407](42035/5): Loss 0.0104, Sem 0.0104 LR: 3.378e-02 Score 99.626 Data time: 0.2098, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6451 [2021-07-08 20:46:06,126][root][INFO] - ===> Epoch[8408](42040/5): Loss 0.0100, Sem 0.0100 LR: 3.377e-02 Score 99.644 Data time: 0.2039, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8160 [2021-07-08 20:46:20,395][root][INFO] - ===> Epoch[8409](42045/5): Loss 0.0107, Sem 0.0107 LR: 3.376e-02 Score 99.619 Data time: 0.2802, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8006 [2021-07-08 20:46:34,392][root][INFO] - ===> Epoch[8410](42050/5): Loss 0.0104, Sem 0.0104 LR: 3.375e-02 Score 99.625 Data time: 0.1880, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7573 [2021-07-08 20:46:48,902][root][INFO] - ===> Epoch[8411](42055/5): Loss 0.0107, Sem 0.0107 LR: 3.375e-02 Score 99.629 Data time: 0.2331, Forward time: 1.2545, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8515 [2021-07-08 20:47:02,607][root][INFO] - ===> Epoch[8412](42060/5): Loss 0.0102, Sem 0.0102 LR: 3.374e-02 Score 99.633 Data time: 0.2085, Forward time: 1.2545, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6986 [2021-07-08 20:47:16,310][root][INFO] - ===> Epoch[8413](42065/5): Loss 0.0103, Sem 0.0103 LR: 3.373e-02 Score 99.631 Data time: 0.2455, Forward time: 1.2545, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6883 [2021-07-08 20:47:29,631][root][INFO] - ===> Epoch[8414](42070/5): Loss 0.0101, Sem 0.0101 LR: 3.372e-02 Score 99.637 Data time: 0.1821, Forward time: 1.2545, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6189 [2021-07-08 20:47:44,189][root][INFO] - ===> Epoch[8415](42075/5): Loss 0.0105, Sem 0.0105 LR: 3.371e-02 Score 99.621 Data time: 0.2073, Forward time: 1.2545, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8659 [2021-07-08 20:47:57,358][root][INFO] - ===> Epoch[8416](42080/5): Loss 0.0102, Sem 0.0102 LR: 3.370e-02 Score 99.636 Data time: 0.2035, Forward time: 1.2545, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.5906 [2021-07-08 20:48:11,684][root][INFO] - ===> Epoch[8417](42085/5): Loss 0.0102, Sem 0.0102 LR: 3.370e-02 Score 99.633 Data time: 0.2397, Forward time: 1.2545, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8067 [2021-07-08 20:48:24,833][root][INFO] - ===> Epoch[8418](42090/5): Loss 0.0099, Sem 0.0099 LR: 3.369e-02 Score 99.644 Data time: 0.2412, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.5832 [2021-07-08 20:48:38,865][root][INFO] - ===> Epoch[8419](42095/5): Loss 0.0101, Sem 0.0101 LR: 3.368e-02 Score 99.640 Data time: 0.2867, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7527 [2021-07-08 20:48:52,760][root][INFO] - ===> Epoch[8420](42100/5): Loss 0.0103, Sem 0.0103 LR: 3.367e-02 Score 99.632 Data time: 0.2852, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7264 [2021-07-08 20:49:06,136][root][INFO] - ===> Epoch[8421](42105/5): Loss 0.0119, Sem 0.0119 LR: 3.366e-02 Score 99.604 Data time: 0.2470, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6216 [2021-07-08 20:49:19,568][root][INFO] - ===> Epoch[8422](42110/5): Loss 0.0114, Sem 0.0114 LR: 3.365e-02 Score 99.600 Data time: 0.2020, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6459 [2021-07-08 20:49:33,700][root][INFO] - ===> Epoch[8423](42115/5): Loss 0.0117, Sem 0.0117 LR: 3.364e-02 Score 99.585 Data time: 0.2389, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7808 [2021-07-08 20:49:47,299][root][INFO] - ===> Epoch[8424](42120/5): Loss 0.0115, Sem 0.0115 LR: 3.364e-02 Score 99.586 Data time: 0.2099, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6777 [2021-07-08 20:50:00,641][root][INFO] - ===> Epoch[8425](42125/5): Loss 0.0109, Sem 0.0109 LR: 3.363e-02 Score 99.609 Data time: 0.1732, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6295 [2021-07-08 20:50:15,115][root][INFO] - ===> Epoch[8426](42130/5): Loss 0.0110, Sem 0.0110 LR: 3.362e-02 Score 99.605 Data time: 0.2263, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8540 [2021-07-08 20:50:30,129][root][INFO] - ===> Epoch[8427](42135/5): Loss 0.0109, Sem 0.0109 LR: 3.361e-02 Score 99.606 Data time: 0.2032, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9449 [2021-07-08 20:50:44,003][root][INFO] - ===> Epoch[8428](42140/5): Loss 0.0107, Sem 0.0107 LR: 3.360e-02 Score 99.617 Data time: 0.1819, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7293 [2021-07-08 20:50:58,778][root][INFO] - ===> Epoch[8429](42145/5): Loss 0.0108, Sem 0.0108 LR: 3.359e-02 Score 99.610 Data time: 0.2497, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9055 [2021-07-08 20:51:12,607][root][INFO] - ===> Epoch[8430](42150/5): Loss 0.0109, Sem 0.0109 LR: 3.359e-02 Score 99.608 Data time: 0.2825, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7150 [2021-07-08 20:51:27,099][root][INFO] - ===> Epoch[8431](42155/5): Loss 0.0105, Sem 0.0105 LR: 3.358e-02 Score 99.620 Data time: 0.2377, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8565 [2021-07-08 20:51:41,058][root][INFO] - ===> Epoch[8432](42160/5): Loss 0.0107, Sem 0.0107 LR: 3.357e-02 Score 99.613 Data time: 0.1776, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7544 [2021-07-08 20:51:55,231][root][INFO] - ===> Epoch[8433](42165/5): Loss 0.0101, Sem 0.0101 LR: 3.356e-02 Score 99.632 Data time: 0.3293, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7930 [2021-07-08 20:52:09,770][root][INFO] - ===> Epoch[8434](42170/5): Loss 0.0102, Sem 0.0102 LR: 3.355e-02 Score 99.632 Data time: 0.3133, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8536 [2021-07-08 20:52:24,136][root][INFO] - ===> Epoch[8435](42175/5): Loss 0.0103, Sem 0.0103 LR: 3.354e-02 Score 99.629 Data time: 0.3071, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8215 [2021-07-08 20:52:37,976][root][INFO] - ===> Epoch[8436](42180/5): Loss 0.0108, Sem 0.0108 LR: 3.353e-02 Score 99.613 Data time: 0.1970, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7153 [2021-07-08 20:52:52,126][root][INFO] - ===> Epoch[8437](42185/5): Loss 0.0113, Sem 0.0113 LR: 3.353e-02 Score 99.608 Data time: 0.1958, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7850 [2021-07-08 20:53:06,329][root][INFO] - ===> Epoch[8438](42190/5): Loss 0.0104, Sem 0.0104 LR: 3.352e-02 Score 99.626 Data time: 0.2650, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7792 [2021-07-08 20:53:20,989][root][INFO] - ===> Epoch[8439](42195/5): Loss 0.0115, Sem 0.0115 LR: 3.351e-02 Score 99.607 Data time: 0.1737, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8849 [2021-07-08 20:53:34,886][root][INFO] - ===> Epoch[8440](42200/5): Loss 0.0115, Sem 0.0115 LR: 3.350e-02 Score 99.598 Data time: 0.2683, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7320 [2021-07-08 20:53:34,888][root][INFO] - ===> Start testing [2021-07-08 20:54:17,536][root][INFO] - 68/68: Data time: 0.0907, Iter time: 0.1333 Loss 0.351 (AVG: 0.560) Score 92.203 (AVG: 90.938) mIOU 71.710 mAP 83.681 mAcc 77.598 IOU: 65.008 0.000 80.001 75.086 94.350 91.494 33.017 84.687 97.927 86.732 79.956 85.983 57.991 mAP: 83.545 4.638 90.325 88.086 98.226 98.615 55.008 97.520 99.356 95.106 91.686 94.449 91.299 mAcc: 80.782 0.000 83.785 83.526 95.895 96.368 35.814 93.578 99.542 91.190 88.309 96.171 63.816 [2021-07-08 20:54:17,536][root][INFO] - Finished test. Elapsed time: 42.6480 [2021-07-08 20:54:18,673][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 20:54:31,337][root][INFO] - ===> Epoch[8441](42205/5): Loss 0.0111, Sem 0.0111 LR: 3.349e-02 Score 99.613 Data time: 0.1813, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.4935 [2021-07-08 20:54:45,531][root][INFO] - ===> Epoch[8442](42210/5): Loss 0.0107, Sem 0.0107 LR: 3.348e-02 Score 99.617 Data time: 0.2077, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7860 [2021-07-08 20:54:59,554][root][INFO] - ===> Epoch[8443](42215/5): Loss 0.0101, Sem 0.0101 LR: 3.348e-02 Score 99.637 Data time: 0.2424, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7617 [2021-07-08 20:55:13,040][root][INFO] - ===> Epoch[8444](42220/5): Loss 0.0107, Sem 0.0107 LR: 3.347e-02 Score 99.617 Data time: 0.1874, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6526 [2021-07-08 20:55:27,101][root][INFO] - ===> Epoch[8445](42225/5): Loss 0.0105, Sem 0.0105 LR: 3.346e-02 Score 99.623 Data time: 0.1955, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7777 [2021-07-08 20:55:42,059][root][INFO] - ===> Epoch[8446](42230/5): Loss 0.0106, Sem 0.0106 LR: 3.345e-02 Score 99.626 Data time: 0.2994, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9526 [2021-07-08 20:55:56,127][root][INFO] - ===> Epoch[8447](42235/5): Loss 0.0103, Sem 0.0103 LR: 3.344e-02 Score 99.631 Data time: 0.2361, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7583 [2021-07-08 20:56:10,497][root][INFO] - ===> Epoch[8448](42240/5): Loss 0.0108, Sem 0.0108 LR: 3.343e-02 Score 99.612 Data time: 0.2321, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8236 [2021-07-08 20:56:24,679][root][INFO] - ===> Epoch[8449](42245/5): Loss 0.0100, Sem 0.0100 LR: 3.342e-02 Score 99.639 Data time: 0.2583, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7845 [2021-07-08 20:56:38,189][root][INFO] - ===> Epoch[8450](42250/5): Loss 0.0101, Sem 0.0101 LR: 3.342e-02 Score 99.634 Data time: 0.2621, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6535 [2021-07-08 20:56:52,332][root][INFO] - ===> Epoch[8451](42255/5): Loss 0.0096, Sem 0.0096 LR: 3.341e-02 Score 99.650 Data time: 0.2559, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7805 [2021-07-08 20:57:06,841][root][INFO] - ===> Epoch[8452](42260/5): Loss 0.0099, Sem 0.0099 LR: 3.340e-02 Score 99.642 Data time: 0.2638, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8554 [2021-07-08 20:57:21,715][root][INFO] - ===> Epoch[8453](42265/5): Loss 0.0099, Sem 0.0099 LR: 3.339e-02 Score 99.644 Data time: 0.2163, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9203 [2021-07-08 20:57:35,124][root][INFO] - ===> Epoch[8454](42270/5): Loss 0.0101, Sem 0.0101 LR: 3.338e-02 Score 99.636 Data time: 0.2559, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6410 [2021-07-08 20:57:49,287][root][INFO] - ===> Epoch[8455](42275/5): Loss 0.0095, Sem 0.0095 LR: 3.337e-02 Score 99.654 Data time: 0.2423, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7753 [2021-07-08 20:58:03,288][root][INFO] - ===> Epoch[8456](42280/5): Loss 0.0099, Sem 0.0099 LR: 3.337e-02 Score 99.648 Data time: 0.2147, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7366 [2021-07-08 20:58:16,870][root][INFO] - ===> Epoch[8457](42285/5): Loss 0.0099, Sem 0.0099 LR: 3.336e-02 Score 99.648 Data time: 0.2343, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6757 [2021-07-08 20:58:30,982][root][INFO] - ===> Epoch[8458](42290/5): Loss 0.0103, Sem 0.0103 LR: 3.335e-02 Score 99.630 Data time: 0.1901, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7761 [2021-07-08 20:58:45,038][root][INFO] - ===> Epoch[8459](42295/5): Loss 0.0101, Sem 0.0101 LR: 3.334e-02 Score 99.635 Data time: 0.2113, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7667 [2021-07-08 20:58:59,184][root][INFO] - ===> Epoch[8460](42300/5): Loss 0.0097, Sem 0.0097 LR: 3.333e-02 Score 99.655 Data time: 0.2071, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7840 [2021-07-08 20:59:13,956][root][INFO] - ===> Epoch[8461](42305/5): Loss 0.0102, Sem 0.0102 LR: 3.332e-02 Score 99.632 Data time: 0.2008, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9189 [2021-07-08 20:59:27,146][root][INFO] - ===> Epoch[8462](42310/5): Loss 0.0104, Sem 0.0104 LR: 3.331e-02 Score 99.631 Data time: 0.1893, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6025 [2021-07-08 20:59:40,094][root][INFO] - ===> Epoch[8463](42315/5): Loss 0.0109, Sem 0.0109 LR: 3.331e-02 Score 99.613 Data time: 0.2074, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.5469 [2021-07-08 20:59:54,563][root][INFO] - ===> Epoch[8464](42320/5): Loss 0.0098, Sem 0.0098 LR: 3.330e-02 Score 99.647 Data time: 0.2206, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8445 [2021-07-08 21:00:07,481][root][INFO] - ===> Epoch[8465](42325/5): Loss 0.0102, Sem 0.0102 LR: 3.329e-02 Score 99.632 Data time: 0.1896, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.5421 [2021-07-08 21:00:20,920][root][INFO] - ===> Epoch[8466](42330/5): Loss 0.0104, Sem 0.0104 LR: 3.328e-02 Score 99.637 Data time: 0.2214, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6400 [2021-07-08 21:00:35,282][root][INFO] - ===> Epoch[8467](42335/5): Loss 0.0097, Sem 0.0097 LR: 3.327e-02 Score 99.652 Data time: 0.1933, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8269 [2021-07-08 21:00:49,183][root][INFO] - ===> Epoch[8468](42340/5): Loss 0.0104, Sem 0.0104 LR: 3.326e-02 Score 99.629 Data time: 0.2424, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7284 [2021-07-08 21:01:02,266][root][INFO] - ===> Epoch[8469](42345/5): Loss 0.0112, Sem 0.0112 LR: 3.326e-02 Score 99.604 Data time: 0.2103, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.5773 [2021-07-08 21:01:16,723][root][INFO] - ===> Epoch[8470](42350/5): Loss 0.0108, Sem 0.0108 LR: 3.325e-02 Score 99.613 Data time: 0.1667, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8553 [2021-07-08 21:01:30,285][root][INFO] - ===> Epoch[8471](42355/5): Loss 0.0105, Sem 0.0105 LR: 3.324e-02 Score 99.624 Data time: 0.2113, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6671 [2021-07-08 21:01:44,209][root][INFO] - ===> Epoch[8472](42360/5): Loss 0.0107, Sem 0.0107 LR: 3.323e-02 Score 99.616 Data time: 0.2809, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7377 [2021-07-08 21:01:58,896][root][INFO] - ===> Epoch[8473](42365/5): Loss 0.0097, Sem 0.0097 LR: 3.322e-02 Score 99.654 Data time: 0.2721, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8934 [2021-07-08 21:02:12,895][root][INFO] - ===> Epoch[8474](42370/5): Loss 0.0102, Sem 0.0102 LR: 3.321e-02 Score 99.638 Data time: 0.2915, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7500 [2021-07-08 21:02:27,723][root][INFO] - ===> Epoch[8475](42375/5): Loss 0.0103, Sem 0.0103 LR: 3.320e-02 Score 99.627 Data time: 0.2352, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9223 [2021-07-08 21:02:42,003][root][INFO] - ===> Epoch[8476](42380/5): Loss 0.0101, Sem 0.0101 LR: 3.320e-02 Score 99.634 Data time: 0.2726, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8093 [2021-07-08 21:02:56,105][root][INFO] - ===> Epoch[8477](42385/5): Loss 0.0100, Sem 0.0100 LR: 3.319e-02 Score 99.641 Data time: 0.2768, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7720 [2021-07-08 21:03:09,972][root][INFO] - ===> Epoch[8478](42390/5): Loss 0.0093, Sem 0.0093 LR: 3.318e-02 Score 99.667 Data time: 0.2580, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7208 [2021-07-08 21:03:24,330][root][INFO] - ===> Epoch[8479](42395/5): Loss 0.0102, Sem 0.0102 LR: 3.317e-02 Score 99.639 Data time: 0.2642, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8228 [2021-07-08 21:03:37,682][root][INFO] - ===> Epoch[8480](42400/5): Loss 0.0094, Sem 0.0094 LR: 3.316e-02 Score 99.663 Data time: 0.2226, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6255 [2021-07-08 21:03:37,684][root][INFO] - ===> Start testing [2021-07-08 21:04:19,279][root][INFO] - 68/68: Data time: 0.0019, Iter time: 0.1224 Loss 0.341 (AVG: 0.595) Score 91.888 (AVG: 90.799) mIOU 71.088 mAP 83.018 mAcc 77.098 IOU: 65.249 0.000 79.832 75.118 94.371 91.960 24.634 84.386 98.182 87.922 79.679 85.503 57.316 mAP: 83.754 3.955 89.392 87.447 98.428 98.673 48.232 97.263 99.739 96.116 92.087 93.596 90.548 mAcc: 82.327 0.000 83.590 82.825 95.670 96.949 26.299 93.166 99.342 94.653 90.675 96.333 60.449 [2021-07-08 21:04:19,279][root][INFO] - Finished test. Elapsed time: 41.5937 [2021-07-08 21:04:20,317][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 21:04:33,212][root][INFO] - ===> Epoch[8481](42405/5): Loss 0.0099, Sem 0.0099 LR: 3.315e-02 Score 99.646 Data time: 0.1451, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.5446 [2021-07-08 21:04:47,217][root][INFO] - ===> Epoch[8482](42410/5): Loss 0.0096, Sem 0.0096 LR: 3.315e-02 Score 99.652 Data time: 0.2682, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7568 [2021-07-08 21:05:01,492][root][INFO] - ===> Epoch[8483](42415/5): Loss 0.0092, Sem 0.0092 LR: 3.314e-02 Score 99.669 Data time: 0.2226, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8092 [2021-07-08 21:05:15,669][root][INFO] - ===> Epoch[8484](42420/5): Loss 0.0098, Sem 0.0098 LR: 3.313e-02 Score 99.646 Data time: 0.2148, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7832 [2021-07-08 21:05:29,340][root][INFO] - ===> Epoch[8485](42425/5): Loss 0.0094, Sem 0.0094 LR: 3.312e-02 Score 99.659 Data time: 0.2122, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6899 [2021-07-08 21:05:42,893][root][INFO] - ===> Epoch[8486](42430/5): Loss 0.0099, Sem 0.0099 LR: 3.311e-02 Score 99.643 Data time: 0.2066, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6681 [2021-07-08 21:05:57,374][root][INFO] - ===> Epoch[8487](42435/5): Loss 0.0094, Sem 0.0094 LR: 3.310e-02 Score 99.664 Data time: 0.2202, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8516 [2021-07-08 21:06:11,146][root][INFO] - ===> Epoch[8488](42440/5): Loss 0.0092, Sem 0.0092 LR: 3.309e-02 Score 99.667 Data time: 0.1641, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7034 [2021-07-08 21:06:25,846][root][INFO] - ===> Epoch[8489](42445/5): Loss 0.0098, Sem 0.0098 LR: 3.309e-02 Score 99.648 Data time: 0.1671, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9041 [2021-07-08 21:06:39,590][root][INFO] - ===> Epoch[8490](42450/5): Loss 0.0093, Sem 0.0093 LR: 3.308e-02 Score 99.666 Data time: 0.2127, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7047 [2021-07-08 21:06:53,794][root][INFO] - ===> Epoch[8491](42455/5): Loss 0.0096, Sem 0.0096 LR: 3.307e-02 Score 99.657 Data time: 0.2558, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7978 [2021-07-08 21:07:08,120][root][INFO] - ===> Epoch[8492](42460/5): Loss 0.0097, Sem 0.0097 LR: 3.306e-02 Score 99.653 Data time: 0.2472, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8177 [2021-07-08 21:07:21,097][root][INFO] - ===> Epoch[8493](42465/5): Loss 0.0093, Sem 0.0093 LR: 3.305e-02 Score 99.665 Data time: 0.2175, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.5536 [2021-07-08 21:07:34,707][root][INFO] - ===> Epoch[8494](42470/5): Loss 0.0095, Sem 0.0095 LR: 3.304e-02 Score 99.660 Data time: 0.2339, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6795 [2021-07-08 21:07:48,315][root][INFO] - ===> Epoch[8495](42475/5): Loss 0.0100, Sem 0.0100 LR: 3.303e-02 Score 99.641 Data time: 0.2041, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6799 [2021-07-08 21:08:02,165][root][INFO] - ===> Epoch[8496](42480/5): Loss 0.0098, Sem 0.0098 LR: 3.303e-02 Score 99.650 Data time: 0.2041, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7280 [2021-07-08 21:08:16,336][root][INFO] - ===> Epoch[8497](42485/5): Loss 0.0095, Sem 0.0095 LR: 3.302e-02 Score 99.660 Data time: 0.1499, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7838 [2021-07-08 21:08:29,073][root][INFO] - ===> Epoch[8498](42490/5): Loss 0.0097, Sem 0.0097 LR: 3.301e-02 Score 99.652 Data time: 0.2147, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.5058 [2021-07-08 21:08:44,218][root][INFO] - ===> Epoch[8499](42495/5): Loss 0.0093, Sem 0.0093 LR: 3.300e-02 Score 99.666 Data time: 0.2560, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9765 [2021-07-08 21:08:58,478][root][INFO] - ===> Epoch[8500](42500/5): Loss 0.0089, Sem 0.0089 LR: 3.299e-02 Score 99.682 Data time: 0.2184, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8072 [2021-07-08 21:09:12,572][root][INFO] - ===> Epoch[8501](42505/5): Loss 0.0089, Sem 0.0089 LR: 3.298e-02 Score 99.677 Data time: 0.2598, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7735 [2021-07-08 21:09:26,495][root][INFO] - ===> Epoch[8502](42510/5): Loss 0.0094, Sem 0.0094 LR: 3.298e-02 Score 99.663 Data time: 0.2081, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7532 [2021-07-08 21:09:39,762][root][INFO] - ===> Epoch[8503](42515/5): Loss 0.0095, Sem 0.0095 LR: 3.297e-02 Score 99.656 Data time: 0.2329, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6083 [2021-07-08 21:09:53,389][root][INFO] - ===> Epoch[8504](42520/5): Loss 0.0095, Sem 0.0095 LR: 3.296e-02 Score 99.656 Data time: 0.1774, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6775 [2021-07-08 21:10:06,916][root][INFO] - ===> Epoch[8505](42525/5): Loss 0.0097, Sem 0.0097 LR: 3.295e-02 Score 99.651 Data time: 0.1705, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6739 [2021-07-08 21:10:20,826][root][INFO] - ===> Epoch[8506](42530/5): Loss 0.0091, Sem 0.0091 LR: 3.294e-02 Score 99.673 Data time: 0.1757, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7434 [2021-07-08 21:10:34,777][root][INFO] - ===> Epoch[8507](42535/5): Loss 0.0094, Sem 0.0094 LR: 3.293e-02 Score 99.661 Data time: 0.2343, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7283 [2021-07-08 21:10:48,553][root][INFO] - ===> Epoch[8508](42540/5): Loss 0.0092, Sem 0.0092 LR: 3.292e-02 Score 99.670 Data time: 0.2178, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7156 [2021-07-08 21:11:02,098][root][INFO] - ===> Epoch[8509](42545/5): Loss 0.0092, Sem 0.0092 LR: 3.292e-02 Score 99.668 Data time: 0.2242, Forward time: 1.2543, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6606 [2021-07-08 21:11:15,904][root][INFO] - ===> Epoch[8510](42550/5): Loss 0.0090, Sem 0.0090 LR: 3.291e-02 Score 99.673 Data time: 0.1700, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7140 [2021-07-08 21:11:30,121][root][INFO] - ===> Epoch[8511](42555/5): Loss 0.0089, Sem 0.0089 LR: 3.290e-02 Score 99.677 Data time: 0.2265, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7948 [2021-07-08 21:11:44,387][root][INFO] - ===> Epoch[8512](42560/5): Loss 0.0090, Sem 0.0090 LR: 3.289e-02 Score 99.673 Data time: 0.2228, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8027 [2021-07-08 21:11:59,390][root][INFO] - ===> Epoch[8513](42565/5): Loss 0.0094, Sem 0.0094 LR: 3.288e-02 Score 99.665 Data time: 0.1793, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.9521 [2021-07-08 21:12:13,909][root][INFO] - ===> Epoch[8514](42570/5): Loss 0.0091, Sem 0.0091 LR: 3.287e-02 Score 99.672 Data time: 0.1925, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8616 [2021-07-08 21:12:27,743][root][INFO] - ===> Epoch[8515](42575/5): Loss 0.0091, Sem 0.0091 LR: 3.287e-02 Score 99.669 Data time: 0.2413, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7238 [2021-07-08 21:12:42,370][root][INFO] - ===> Epoch[8516](42580/5): Loss 0.0090, Sem 0.0090 LR: 3.286e-02 Score 99.673 Data time: 0.2708, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8739 [2021-07-08 21:12:56,609][root][INFO] - ===> Epoch[8517](42585/5): Loss 0.0087, Sem 0.0087 LR: 3.285e-02 Score 99.687 Data time: 0.2880, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8018 [2021-07-08 21:13:11,267][root][INFO] - ===> Epoch[8518](42590/5): Loss 0.0092, Sem 0.0092 LR: 3.284e-02 Score 99.667 Data time: 0.3056, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8839 [2021-07-08 21:13:25,811][root][INFO] - ===> Epoch[8519](42595/5): Loss 0.0089, Sem 0.0089 LR: 3.283e-02 Score 99.680 Data time: 0.2591, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8517 [2021-07-08 21:13:40,056][root][INFO] - ===> Epoch[8520](42600/5): Loss 0.0092, Sem 0.0092 LR: 3.282e-02 Score 99.672 Data time: 0.2180, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8089 [2021-07-08 21:13:40,058][root][INFO] - ===> Start testing [2021-07-08 21:14:21,817][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1257 Loss 0.402 (AVG: 0.606) Score 91.567 (AVG: 90.843) mIOU 71.355 mAP 83.298 mAcc 77.523 IOU: 64.772 0.000 82.281 75.337 94.224 91.835 27.742 82.488 97.942 86.642 81.180 85.866 57.304 mAP: 83.171 3.312 92.161 86.121 98.568 98.639 51.836 97.380 99.429 96.002 92.193 93.509 90.555 mAcc: 81.009 0.000 86.076 82.756 95.564 97.209 30.128 93.607 99.391 93.942 90.947 96.571 60.597 [2021-07-08 21:14:21,817][root][INFO] - Finished test. Elapsed time: 41.7578 [2021-07-08 21:14:23,068][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 21:14:35,728][root][INFO] - ===> Epoch[8521](42605/5): Loss 0.0092, Sem 0.0092 LR: 3.281e-02 Score 99.670 Data time: 0.2394, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.4925 [2021-07-08 21:14:49,415][root][INFO] - ===> Epoch[8522](42610/5): Loss 0.0091, Sem 0.0091 LR: 3.281e-02 Score 99.670 Data time: 0.3275, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6854 [2021-07-08 21:15:03,049][root][INFO] - ===> Epoch[8523](42615/5): Loss 0.0093, Sem 0.0093 LR: 3.280e-02 Score 99.664 Data time: 0.2035, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6825 [2021-07-08 21:15:17,048][root][INFO] - ===> Epoch[8524](42620/5): Loss 0.0093, Sem 0.0093 LR: 3.279e-02 Score 99.665 Data time: 0.2865, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7464 [2021-07-08 21:15:31,669][root][INFO] - ===> Epoch[8525](42625/5): Loss 0.0092, Sem 0.0092 LR: 3.278e-02 Score 99.668 Data time: 0.2231, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8767 [2021-07-08 21:15:45,720][root][INFO] - ===> Epoch[8526](42630/5): Loss 0.0093, Sem 0.0093 LR: 3.277e-02 Score 99.662 Data time: 0.2583, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7665 [2021-07-08 21:15:59,941][root][INFO] - ===> Epoch[8527](42635/5): Loss 0.0092, Sem 0.0092 LR: 3.276e-02 Score 99.669 Data time: 0.2241, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7877 [2021-07-08 21:16:13,657][root][INFO] - ===> Epoch[8528](42640/5): Loss 0.0094, Sem 0.0094 LR: 3.275e-02 Score 99.661 Data time: 0.2476, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7001 [2021-07-08 21:16:27,600][root][INFO] - ===> Epoch[8529](42645/5): Loss 0.0091, Sem 0.0091 LR: 3.275e-02 Score 99.671 Data time: 0.1996, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7428 [2021-07-08 21:16:40,843][root][INFO] - ===> Epoch[8530](42650/5): Loss 0.0092, Sem 0.0092 LR: 3.274e-02 Score 99.669 Data time: 0.2142, Forward time: 1.2543, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6060 [2021-07-08 21:16:55,008][root][INFO] - ===> Epoch[8531](42655/5): Loss 0.0093, Sem 0.0093 LR: 3.273e-02 Score 99.667 Data time: 0.2573, Forward time: 1.2543, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7894 [2021-07-08 21:17:09,487][root][INFO] - ===> Epoch[8532](42660/5): Loss 0.0090, Sem 0.0090 LR: 3.272e-02 Score 99.677 Data time: 0.1494, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8528 [2021-07-08 21:17:22,390][root][INFO] - ===> Epoch[8533](42665/5): Loss 0.0092, Sem 0.0092 LR: 3.271e-02 Score 99.671 Data time: 0.2372, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.5346 [2021-07-08 21:17:36,986][root][INFO] - ===> Epoch[8534](42670/5): Loss 0.0090, Sem 0.0090 LR: 3.270e-02 Score 99.673 Data time: 0.2030, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8769 [2021-07-08 21:17:50,750][root][INFO] - ===> Epoch[8535](42675/5): Loss 0.0093, Sem 0.0093 LR: 3.270e-02 Score 99.665 Data time: 0.1985, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7124 [2021-07-08 21:18:04,956][root][INFO] - ===> Epoch[8536](42680/5): Loss 0.0086, Sem 0.0086 LR: 3.269e-02 Score 99.692 Data time: 0.2722, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7982 [2021-07-08 21:18:19,194][root][INFO] - ===> Epoch[8537](42685/5): Loss 0.0091, Sem 0.0091 LR: 3.268e-02 Score 99.673 Data time: 0.2155, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8081 [2021-07-08 21:18:33,228][root][INFO] - ===> Epoch[8538](42690/5): Loss 0.0094, Sem 0.0094 LR: 3.267e-02 Score 99.660 Data time: 0.2243, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7621 [2021-07-08 21:18:46,653][root][INFO] - ===> Epoch[8539](42695/5): Loss 0.0094, Sem 0.0094 LR: 3.266e-02 Score 99.661 Data time: 0.1757, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6331 [2021-07-08 21:19:00,752][root][INFO] - ===> Epoch[8540](42700/5): Loss 0.0093, Sem 0.0093 LR: 3.265e-02 Score 99.665 Data time: 0.2805, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7684 [2021-07-08 21:19:14,972][root][INFO] - ===> Epoch[8541](42705/5): Loss 0.0091, Sem 0.0091 LR: 3.264e-02 Score 99.674 Data time: 0.2138, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8033 [2021-07-08 21:19:28,154][root][INFO] - ===> Epoch[8542](42710/5): Loss 0.0096, Sem 0.0096 LR: 3.264e-02 Score 99.658 Data time: 0.2311, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.5759 [2021-07-08 21:19:41,648][root][INFO] - ===> Epoch[8543](42715/5): Loss 0.0106, Sem 0.0106 LR: 3.263e-02 Score 99.632 Data time: 0.2042, Forward time: 1.2543, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6597 [2021-07-08 21:19:55,855][root][INFO] - ===> Epoch[8544](42720/5): Loss 0.0103, Sem 0.0103 LR: 3.262e-02 Score 99.632 Data time: 0.2376, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7916 [2021-07-08 21:20:09,599][root][INFO] - ===> Epoch[8545](42725/5): Loss 0.0102, Sem 0.0102 LR: 3.261e-02 Score 99.632 Data time: 0.2366, Forward time: 1.2543, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7088 [2021-07-08 21:20:23,684][root][INFO] - ===> Epoch[8546](42730/5): Loss 0.0100, Sem 0.0100 LR: 3.260e-02 Score 99.636 Data time: 0.2221, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7725 [2021-07-08 21:20:37,144][root][INFO] - ===> Epoch[8547](42735/5): Loss 0.0099, Sem 0.0099 LR: 3.259e-02 Score 99.643 Data time: 0.2168, Forward time: 1.2543, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6554 [2021-07-08 21:20:50,251][root][INFO] - ===> Epoch[8548](42740/5): Loss 0.0099, Sem 0.0099 LR: 3.258e-02 Score 99.642 Data time: 0.2427, Forward time: 1.2543, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.5718 [2021-07-08 21:21:04,547][root][INFO] - ===> Epoch[8549](42745/5): Loss 0.0095, Sem 0.0095 LR: 3.258e-02 Score 99.657 Data time: 0.1968, Forward time: 1.2543, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8121 [2021-07-08 21:21:18,968][root][INFO] - ===> Epoch[8550](42750/5): Loss 0.0094, Sem 0.0094 LR: 3.257e-02 Score 99.663 Data time: 0.1998, Forward time: 1.2543, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8374 [2021-07-08 21:21:33,225][root][INFO] - ===> Epoch[8551](42755/5): Loss 0.0099, Sem 0.0099 LR: 3.256e-02 Score 99.646 Data time: 0.1703, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8079 [2021-07-08 21:21:47,367][root][INFO] - ===> Epoch[8552](42760/5): Loss 0.0092, Sem 0.0092 LR: 3.255e-02 Score 99.670 Data time: 0.1594, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7916 [2021-07-08 21:22:01,819][root][INFO] - ===> Epoch[8553](42765/5): Loss 0.0090, Sem 0.0090 LR: 3.254e-02 Score 99.677 Data time: 0.1972, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8326 [2021-07-08 21:22:15,517][root][INFO] - ===> Epoch[8554](42770/5): Loss 0.0094, Sem 0.0094 LR: 3.253e-02 Score 99.661 Data time: 0.2044, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6961 [2021-07-08 21:22:29,882][root][INFO] - ===> Epoch[8555](42775/5): Loss 0.0091, Sem 0.0091 LR: 3.253e-02 Score 99.672 Data time: 0.2116, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8313 [2021-07-08 21:22:43,846][root][INFO] - ===> Epoch[8556](42780/5): Loss 0.0090, Sem 0.0090 LR: 3.252e-02 Score 99.675 Data time: 0.2037, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7431 [2021-07-08 21:22:57,905][root][INFO] - ===> Epoch[8557](42785/5): Loss 0.0089, Sem 0.0089 LR: 3.251e-02 Score 99.675 Data time: 0.2313, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7665 [2021-07-08 21:23:12,094][root][INFO] - ===> Epoch[8558](42790/5): Loss 0.0094, Sem 0.0094 LR: 3.250e-02 Score 99.659 Data time: 0.2077, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7875 [2021-07-08 21:23:26,580][root][INFO] - ===> Epoch[8559](42795/5): Loss 0.0087, Sem 0.0087 LR: 3.249e-02 Score 99.687 Data time: 0.2804, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8486 [2021-07-08 21:23:40,758][root][INFO] - ===> Epoch[8560](42800/5): Loss 0.0092, Sem 0.0092 LR: 3.248e-02 Score 99.666 Data time: 0.2537, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7902 [2021-07-08 21:23:40,761][root][INFO] - ===> Start testing [2021-07-08 21:24:15,742][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1364 Loss 0.295 (AVG: 0.591) Score 93.449 (AVG: 91.075) mIOU 71.177 mAP 83.290 mAcc 77.320 IOU: 65.099 0.000 80.901 75.523 95.237 92.443 24.347 80.386 98.654 85.603 81.279 86.331 59.502 mAP: 83.393 3.513 90.232 86.417 98.568 98.656 52.447 97.398 99.772 96.295 92.325 93.438 90.321 mAcc: 82.111 0.000 84.030 82.484 96.742 96.950 26.120 93.125 99.410 93.508 90.588 96.378 63.718 [2021-07-08 21:24:15,742][root][INFO] - Finished test. Elapsed time: 34.9806 [2021-07-08 21:24:16,812][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 21:24:30,448][root][INFO] - ===> Epoch[8561](42805/5): Loss 0.0091, Sem 0.0091 LR: 3.247e-02 Score 99.674 Data time: 0.2038, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6924 [2021-07-08 21:24:44,179][root][INFO] - ===> Epoch[8562](42810/5): Loss 0.0091, Sem 0.0091 LR: 3.247e-02 Score 99.670 Data time: 0.2613, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6903 [2021-07-08 21:24:59,721][root][INFO] - ===> Epoch[8563](42815/5): Loss 0.0090, Sem 0.0090 LR: 3.246e-02 Score 99.678 Data time: 0.2248, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 3.0440 [2021-07-08 21:25:13,300][root][INFO] - ===> Epoch[8564](42820/5): Loss 0.0088, Sem 0.0088 LR: 3.245e-02 Score 99.685 Data time: 0.2429, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6639 [2021-07-08 21:25:27,011][root][INFO] - ===> Epoch[8565](42825/5): Loss 0.0091, Sem 0.0091 LR: 3.244e-02 Score 99.672 Data time: 0.2074, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7001 [2021-07-08 21:25:41,599][root][INFO] - ===> Epoch[8566](42830/5): Loss 0.0089, Sem 0.0089 LR: 3.243e-02 Score 99.683 Data time: 0.2304, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8640 [2021-07-08 21:25:56,298][root][INFO] - ===> Epoch[8567](42835/5): Loss 0.0092, Sem 0.0092 LR: 3.242e-02 Score 99.667 Data time: 0.2043, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8951 [2021-07-08 21:26:10,371][root][INFO] - ===> Epoch[8568](42840/5): Loss 0.0091, Sem 0.0091 LR: 3.241e-02 Score 99.672 Data time: 0.2937, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7714 [2021-07-08 21:26:24,504][root][INFO] - ===> Epoch[8569](42845/5): Loss 0.0091, Sem 0.0091 LR: 3.241e-02 Score 99.672 Data time: 0.2466, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7740 [2021-07-08 21:26:38,240][root][INFO] - ===> Epoch[8570](42850/5): Loss 0.0087, Sem 0.0087 LR: 3.240e-02 Score 99.686 Data time: 0.2734, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6947 [2021-07-08 21:26:51,897][root][INFO] - ===> Epoch[8571](42855/5): Loss 0.0092, Sem 0.0092 LR: 3.239e-02 Score 99.669 Data time: 0.2241, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6897 [2021-07-08 21:27:05,954][root][INFO] - ===> Epoch[8572](42860/5): Loss 0.0091, Sem 0.0091 LR: 3.238e-02 Score 99.674 Data time: 0.1991, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7688 [2021-07-08 21:27:19,534][root][INFO] - ===> Epoch[8573](42865/5): Loss 0.0087, Sem 0.0087 LR: 3.237e-02 Score 99.684 Data time: 0.2139, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6696 [2021-07-08 21:27:32,720][root][INFO] - ===> Epoch[8574](42870/5): Loss 0.0092, Sem 0.0092 LR: 3.236e-02 Score 99.670 Data time: 0.2284, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.5961 [2021-07-08 21:27:46,439][root][INFO] - ===> Epoch[8575](42875/5): Loss 0.0091, Sem 0.0091 LR: 3.236e-02 Score 99.673 Data time: 0.1956, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7009 [2021-07-08 21:28:00,199][root][INFO] - ===> Epoch[8576](42880/5): Loss 0.0090, Sem 0.0090 LR: 3.235e-02 Score 99.673 Data time: 0.2160, Forward time: 1.2543, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7120 [2021-07-08 21:28:14,716][root][INFO] - ===> Epoch[8577](42885/5): Loss 0.0091, Sem 0.0091 LR: 3.234e-02 Score 99.674 Data time: 0.2052, Forward time: 1.2543, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8608 [2021-07-08 21:28:29,304][root][INFO] - ===> Epoch[8578](42890/5): Loss 0.0089, Sem 0.0089 LR: 3.233e-02 Score 99.678 Data time: 0.2365, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8736 [2021-07-08 21:28:43,437][root][INFO] - ===> Epoch[8579](42895/5): Loss 0.0091, Sem 0.0091 LR: 3.232e-02 Score 99.675 Data time: 0.1586, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7937 [2021-07-08 21:28:57,498][root][INFO] - ===> Epoch[8580](42900/5): Loss 0.0089, Sem 0.0089 LR: 3.231e-02 Score 99.678 Data time: 0.1852, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7600 [2021-07-08 21:29:11,062][root][INFO] - ===> Epoch[8581](42905/5): Loss 0.0088, Sem 0.0088 LR: 3.230e-02 Score 99.681 Data time: 0.2360, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6721 [2021-07-08 21:29:25,912][root][INFO] - ===> Epoch[8582](42910/5): Loss 0.0085, Sem 0.0085 LR: 3.230e-02 Score 99.696 Data time: 0.2414, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.9271 [2021-07-08 21:29:40,134][root][INFO] - ===> Epoch[8583](42915/5): Loss 0.0087, Sem 0.0087 LR: 3.229e-02 Score 99.685 Data time: 0.2424, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7982 [2021-07-08 21:29:53,533][root][INFO] - ===> Epoch[8584](42920/5): Loss 0.0084, Sem 0.0084 LR: 3.228e-02 Score 99.698 Data time: 0.2379, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6324 [2021-07-08 21:30:07,207][root][INFO] - ===> Epoch[8585](42925/5): Loss 0.0086, Sem 0.0086 LR: 3.227e-02 Score 99.691 Data time: 0.2101, Forward time: 1.2543, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6979 [2021-07-08 21:30:21,654][root][INFO] - ===> Epoch[8586](42930/5): Loss 0.0088, Sem 0.0088 LR: 3.226e-02 Score 99.682 Data time: 0.2405, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8496 [2021-07-08 21:30:35,319][root][INFO] - ===> Epoch[8587](42935/5): Loss 0.0088, Sem 0.0088 LR: 3.225e-02 Score 99.683 Data time: 0.1866, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6813 [2021-07-08 21:30:48,958][root][INFO] - ===> Epoch[8588](42940/5): Loss 0.0089, Sem 0.0089 LR: 3.224e-02 Score 99.677 Data time: 0.2136, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6842 [2021-07-08 21:31:02,833][root][INFO] - ===> Epoch[8589](42945/5): Loss 0.0090, Sem 0.0090 LR: 3.224e-02 Score 99.674 Data time: 0.2467, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7214 [2021-07-08 21:31:16,183][root][INFO] - ===> Epoch[8590](42950/5): Loss 0.0086, Sem 0.0086 LR: 3.223e-02 Score 99.692 Data time: 0.1843, Forward time: 1.2543, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6349 [2021-07-08 21:31:30,456][root][INFO] - ===> Epoch[8591](42955/5): Loss 0.0088, Sem 0.0088 LR: 3.222e-02 Score 99.686 Data time: 0.2754, Forward time: 1.2543, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8049 [2021-07-08 21:31:44,874][root][INFO] - ===> Epoch[8592](42960/5): Loss 0.0086, Sem 0.0086 LR: 3.221e-02 Score 99.689 Data time: 0.2761, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8305 [2021-07-08 21:31:58,537][root][INFO] - ===> Epoch[8593](42965/5): Loss 0.0086, Sem 0.0086 LR: 3.220e-02 Score 99.690 Data time: 0.1940, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6871 [2021-07-08 21:32:12,930][root][INFO] - ===> Epoch[8594](42970/5): Loss 0.0090, Sem 0.0090 LR: 3.219e-02 Score 99.677 Data time: 0.1925, Forward time: 1.2543, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8303 [2021-07-08 21:32:26,536][root][INFO] - ===> Epoch[8595](42975/5): Loss 0.0092, Sem 0.0092 LR: 3.219e-02 Score 99.668 Data time: 0.1850, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6821 [2021-07-08 21:32:40,896][root][INFO] - ===> Epoch[8596](42980/5): Loss 0.0093, Sem 0.0093 LR: 3.218e-02 Score 99.665 Data time: 0.2143, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8129 [2021-07-08 21:32:54,847][root][INFO] - ===> Epoch[8597](42985/5): Loss 0.0088, Sem 0.0088 LR: 3.217e-02 Score 99.684 Data time: 0.2513, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7435 [2021-07-08 21:33:08,923][root][INFO] - ===> Epoch[8598](42990/5): Loss 0.0083, Sem 0.0083 LR: 3.216e-02 Score 99.699 Data time: 0.1655, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7794 [2021-07-08 21:33:23,481][root][INFO] - ===> Epoch[8599](42995/5): Loss 0.0091, Sem 0.0091 LR: 3.215e-02 Score 99.673 Data time: 0.1984, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8597 [2021-07-08 21:33:37,498][root][INFO] - ===> Epoch[8600](43000/5): Loss 0.0090, Sem 0.0090 LR: 3.214e-02 Score 99.675 Data time: 0.2454, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7434 [2021-07-08 21:33:37,501][root][INFO] - ===> Start testing [2021-07-08 21:34:15,587][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1297 Loss 0.340 (AVG: 0.592) Score 92.617 (AVG: 91.063) mIOU 71.126 mAP 83.235 mAcc 77.386 IOU: 65.565 0.000 82.054 75.437 95.269 91.684 25.715 83.832 98.587 81.264 81.888 85.679 57.669 mAP: 84.810 2.968 90.880 87.312 98.623 98.463 50.404 97.266 99.800 95.645 92.268 93.171 90.452 mAcc: 81.584 0.000 85.210 83.262 96.584 97.507 27.381 92.552 99.267 93.844 91.388 96.531 60.904 [2021-07-08 21:34:15,588][root][INFO] - Finished test. Elapsed time: 38.0855 [2021-07-08 21:34:16,573][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 21:34:29,971][root][INFO] - ===> Epoch[8601](43005/5): Loss 0.0086, Sem 0.0086 LR: 3.213e-02 Score 99.690 Data time: 0.1612, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6384 [2021-07-08 21:34:44,462][root][INFO] - ===> Epoch[8602](43010/5): Loss 0.0093, Sem 0.0093 LR: 3.213e-02 Score 99.666 Data time: 0.2808, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8516 [2021-07-08 21:34:59,544][root][INFO] - ===> Epoch[8603](43015/5): Loss 0.0085, Sem 0.0085 LR: 3.212e-02 Score 99.693 Data time: 0.2891, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.9651 [2021-07-08 21:35:13,582][root][INFO] - ===> Epoch[8604](43020/5): Loss 0.0088, Sem 0.0088 LR: 3.211e-02 Score 99.682 Data time: 0.3272, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7642 [2021-07-08 21:35:27,760][root][INFO] - ===> Epoch[8605](43025/5): Loss 0.0092, Sem 0.0092 LR: 3.210e-02 Score 99.671 Data time: 0.2488, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7810 [2021-07-08 21:35:42,238][root][INFO] - ===> Epoch[8606](43030/5): Loss 0.0084, Sem 0.0084 LR: 3.209e-02 Score 99.700 Data time: 0.2665, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8419 [2021-07-08 21:35:56,596][root][INFO] - ===> Epoch[8607](43035/5): Loss 0.0091, Sem 0.0091 LR: 3.208e-02 Score 99.668 Data time: 0.2223, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8296 [2021-07-08 21:36:10,842][root][INFO] - ===> Epoch[8608](43040/5): Loss 0.0084, Sem 0.0084 LR: 3.207e-02 Score 99.698 Data time: 0.1869, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8059 [2021-07-08 21:36:24,927][root][INFO] - ===> Epoch[8609](43045/5): Loss 0.0086, Sem 0.0086 LR: 3.207e-02 Score 99.688 Data time: 0.2594, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7730 [2021-07-08 21:36:38,516][root][INFO] - ===> Epoch[8610](43050/5): Loss 0.0091, Sem 0.0091 LR: 3.206e-02 Score 99.669 Data time: 0.2329, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6846 [2021-07-08 21:36:52,596][root][INFO] - ===> Epoch[8611](43055/5): Loss 0.0090, Sem 0.0090 LR: 3.205e-02 Score 99.678 Data time: 0.2561, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7814 [2021-07-08 21:37:06,199][root][INFO] - ===> Epoch[8612](43060/5): Loss 0.0086, Sem 0.0086 LR: 3.204e-02 Score 99.692 Data time: 0.2157, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6808 [2021-07-08 21:37:19,840][root][INFO] - ===> Epoch[8613](43065/5): Loss 0.0092, Sem 0.0092 LR: 3.203e-02 Score 99.671 Data time: 0.2248, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6848 [2021-07-08 21:37:34,310][root][INFO] - ===> Epoch[8614](43070/5): Loss 0.0090, Sem 0.0090 LR: 3.202e-02 Score 99.671 Data time: 0.2111, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8531 [2021-07-08 21:37:48,005][root][INFO] - ===> Epoch[8615](43075/5): Loss 0.0090, Sem 0.0090 LR: 3.202e-02 Score 99.679 Data time: 0.2031, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6977 [2021-07-08 21:38:01,648][root][INFO] - ===> Epoch[8616](43080/5): Loss 0.0091, Sem 0.0091 LR: 3.201e-02 Score 99.671 Data time: 0.2216, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6732 [2021-07-08 21:38:15,716][root][INFO] - ===> Epoch[8617](43085/5): Loss 0.0090, Sem 0.0090 LR: 3.200e-02 Score 99.683 Data time: 0.1892, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7609 [2021-07-08 21:38:30,639][root][INFO] - ===> Epoch[8618](43090/5): Loss 0.0088, Sem 0.0088 LR: 3.199e-02 Score 99.681 Data time: 0.2312, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9307 [2021-07-08 21:38:45,301][root][INFO] - ===> Epoch[8619](43095/5): Loss 0.0086, Sem 0.0086 LR: 3.198e-02 Score 99.690 Data time: 0.1538, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8968 [2021-07-08 21:38:59,320][root][INFO] - ===> Epoch[8620](43100/5): Loss 0.0084, Sem 0.0084 LR: 3.197e-02 Score 99.697 Data time: 0.2496, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7551 [2021-07-08 21:39:13,685][root][INFO] - ===> Epoch[8621](43105/5): Loss 0.0084, Sem 0.0084 LR: 3.196e-02 Score 99.697 Data time: 0.1677, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8358 [2021-07-08 21:39:28,118][root][INFO] - ===> Epoch[8622](43110/5): Loss 0.0089, Sem 0.0089 LR: 3.196e-02 Score 99.679 Data time: 0.2120, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8472 [2021-07-08 21:39:41,846][root][INFO] - ===> Epoch[8623](43115/5): Loss 0.0085, Sem 0.0085 LR: 3.195e-02 Score 99.692 Data time: 0.2269, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7006 [2021-07-08 21:39:56,033][root][INFO] - ===> Epoch[8624](43120/5): Loss 0.0094, Sem 0.0094 LR: 3.194e-02 Score 99.664 Data time: 0.1896, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7995 [2021-07-08 21:40:10,034][root][INFO] - ===> Epoch[8625](43125/5): Loss 0.0086, Sem 0.0086 LR: 3.193e-02 Score 99.689 Data time: 0.2273, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7589 [2021-07-08 21:40:23,636][root][INFO] - ===> Epoch[8626](43130/5): Loss 0.0091, Sem 0.0091 LR: 3.192e-02 Score 99.674 Data time: 0.2394, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6791 [2021-07-08 21:40:37,816][root][INFO] - ===> Epoch[8627](43135/5): Loss 0.0086, Sem 0.0086 LR: 3.191e-02 Score 99.687 Data time: 0.2178, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7878 [2021-07-08 21:40:51,043][root][INFO] - ===> Epoch[8628](43140/5): Loss 0.0088, Sem 0.0088 LR: 3.190e-02 Score 99.681 Data time: 0.1860, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6094 [2021-07-08 21:41:05,441][root][INFO] - ===> Epoch[8629](43145/5): Loss 0.0091, Sem 0.0091 LR: 3.190e-02 Score 99.671 Data time: 0.2165, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8386 [2021-07-08 21:41:19,208][root][INFO] - ===> Epoch[8630](43150/5): Loss 0.0084, Sem 0.0084 LR: 3.189e-02 Score 99.696 Data time: 0.2346, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6989 [2021-07-08 21:41:32,235][root][INFO] - ===> Epoch[8631](43155/5): Loss 0.0093, Sem 0.0093 LR: 3.188e-02 Score 99.669 Data time: 0.1984, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.5652 [2021-07-08 21:41:46,786][root][INFO] - ===> Epoch[8632](43160/5): Loss 0.0089, Sem 0.0089 LR: 3.187e-02 Score 99.678 Data time: 0.2240, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8590 [2021-07-08 21:42:00,716][root][INFO] - ===> Epoch[8633](43165/5): Loss 0.0087, Sem 0.0087 LR: 3.186e-02 Score 99.686 Data time: 0.2153, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7349 [2021-07-08 21:42:14,298][root][INFO] - ===> Epoch[8634](43170/5): Loss 0.0087, Sem 0.0087 LR: 3.185e-02 Score 99.685 Data time: 0.2716, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6636 [2021-07-08 21:42:28,260][root][INFO] - ===> Epoch[8635](43175/5): Loss 0.0089, Sem 0.0089 LR: 3.184e-02 Score 99.680 Data time: 0.2704, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7373 [2021-07-08 21:42:42,814][root][INFO] - ===> Epoch[8636](43180/5): Loss 0.0088, Sem 0.0088 LR: 3.184e-02 Score 99.682 Data time: 0.2365, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8499 [2021-07-08 21:42:56,110][root][INFO] - ===> Epoch[8637](43185/5): Loss 0.0090, Sem 0.0090 LR: 3.183e-02 Score 99.677 Data time: 0.1810, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6141 [2021-07-08 21:43:10,400][root][INFO] - ===> Epoch[8638](43190/5): Loss 0.0084, Sem 0.0084 LR: 3.182e-02 Score 99.696 Data time: 0.2099, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8120 [2021-07-08 21:43:23,956][root][INFO] - ===> Epoch[8639](43195/5): Loss 0.0084, Sem 0.0084 LR: 3.181e-02 Score 99.698 Data time: 0.2344, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6696 [2021-07-08 21:43:37,410][root][INFO] - ===> Epoch[8640](43200/5): Loss 0.0091, Sem 0.0091 LR: 3.180e-02 Score 99.673 Data time: 0.2154, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6484 [2021-07-08 21:43:37,412][root][INFO] - ===> Start testing [2021-07-08 21:44:17,318][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1238 Loss 0.350 (AVG: 0.590) Score 92.283 (AVG: 91.008) mIOU 71.092 mAP 83.045 mAcc 77.316 IOU: 65.008 0.000 82.118 75.381 95.157 91.371 24.023 82.569 98.539 84.653 81.154 85.988 58.235 mAP: 83.551 4.241 90.508 86.764 98.665 98.476 48.610 97.483 99.773 95.393 92.021 93.249 90.849 mAcc: 80.113 0.000 85.671 83.247 96.580 97.131 26.345 94.495 99.183 92.512 91.535 96.676 61.621 [2021-07-08 21:44:17,318][root][INFO] - Finished test. Elapsed time: 39.9054 [2021-07-08 21:44:18,348][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 21:44:31,683][root][INFO] - ===> Epoch[8641](43205/5): Loss 0.0087, Sem 0.0087 LR: 3.179e-02 Score 99.685 Data time: 0.1618, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6349 [2021-07-08 21:44:45,347][root][INFO] - ===> Epoch[8642](43210/5): Loss 0.0093, Sem 0.0093 LR: 3.179e-02 Score 99.666 Data time: 0.2550, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6792 [2021-07-08 21:44:58,989][root][INFO] - ===> Epoch[8643](43215/5): Loss 0.0087, Sem 0.0087 LR: 3.178e-02 Score 99.689 Data time: 0.1805, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6822 [2021-07-08 21:45:12,716][root][INFO] - ===> Epoch[8644](43220/5): Loss 0.0092, Sem 0.0092 LR: 3.177e-02 Score 99.671 Data time: 0.1901, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6974 [2021-07-08 21:45:27,104][root][INFO] - ===> Epoch[8645](43225/5): Loss 0.0088, Sem 0.0088 LR: 3.176e-02 Score 99.684 Data time: 0.2848, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8270 [2021-07-08 21:45:41,063][root][INFO] - ===> Epoch[8646](43230/5): Loss 0.0093, Sem 0.0093 LR: 3.175e-02 Score 99.668 Data time: 0.2119, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7421 [2021-07-08 21:45:55,661][root][INFO] - ===> Epoch[8647](43235/5): Loss 0.0087, Sem 0.0087 LR: 3.174e-02 Score 99.685 Data time: 0.2599, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8786 [2021-07-08 21:46:10,688][root][INFO] - ===> Epoch[8648](43240/5): Loss 0.0087, Sem 0.0087 LR: 3.173e-02 Score 99.686 Data time: 0.2900, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9555 [2021-07-08 21:46:24,842][root][INFO] - ===> Epoch[8649](43245/5): Loss 0.0088, Sem 0.0088 LR: 3.173e-02 Score 99.681 Data time: 0.2301, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7914 [2021-07-08 21:46:39,255][root][INFO] - ===> Epoch[8650](43250/5): Loss 0.0089, Sem 0.0089 LR: 3.172e-02 Score 99.684 Data time: 0.2310, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8319 [2021-07-08 21:46:53,917][root][INFO] - ===> Epoch[8651](43255/5): Loss 0.0090, Sem 0.0090 LR: 3.171e-02 Score 99.676 Data time: 0.2858, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8773 [2021-07-08 21:47:07,607][root][INFO] - ===> Epoch[8652](43260/5): Loss 0.0086, Sem 0.0086 LR: 3.170e-02 Score 99.692 Data time: 0.2244, Forward time: 1.2545, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.6959 [2021-07-08 21:47:21,638][root][INFO] - ===> Epoch[8653](43265/5): Loss 0.0087, Sem 0.0087 LR: 3.169e-02 Score 99.686 Data time: 0.2022, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7604 [2021-07-08 21:47:35,815][root][INFO] - ===> Epoch[8654](43270/5): Loss 0.0088, Sem 0.0088 LR: 3.168e-02 Score 99.684 Data time: 0.1980, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.7897 [2021-07-08 21:47:50,752][root][INFO] - ===> Epoch[8655](43275/5): Loss 0.0086, Sem 0.0086 LR: 3.167e-02 Score 99.689 Data time: 0.2373, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.9416 [2021-07-08 21:48:04,993][root][INFO] - ===> Epoch[8656](43280/5): Loss 0.0087, Sem 0.0087 LR: 3.167e-02 Score 99.693 Data time: 0.2343, Forward time: 1.2544, Backward time: 1.2446, DDP time: 0.0042, Total iter time: 2.8053 [2021-07-08 21:48:19,075][root][INFO] - ===> Epoch[8657](43285/5): Loss 0.0087, Sem 0.0087 LR: 3.166e-02 Score 99.690 Data time: 0.1938, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7734 [2021-07-08 21:48:33,523][root][INFO] - ===> Epoch[8658](43290/5): Loss 0.0089, Sem 0.0089 LR: 3.165e-02 Score 99.683 Data time: 0.1898, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8380 [2021-07-08 21:48:48,173][root][INFO] - ===> Epoch[8659](43295/5): Loss 0.0087, Sem 0.0087 LR: 3.164e-02 Score 99.681 Data time: 0.2741, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8749 [2021-07-08 21:49:01,752][root][INFO] - ===> Epoch[8660](43300/5): Loss 0.0089, Sem 0.0089 LR: 3.163e-02 Score 99.679 Data time: 0.2099, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6772 [2021-07-08 21:49:15,581][root][INFO] - ===> Epoch[8661](43305/5): Loss 0.0084, Sem 0.0084 LR: 3.162e-02 Score 99.694 Data time: 0.1823, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7268 [2021-07-08 21:49:30,514][root][INFO] - ===> Epoch[8662](43310/5): Loss 0.0084, Sem 0.0084 LR: 3.161e-02 Score 99.697 Data time: 0.3023, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.9303 [2021-07-08 21:49:45,162][root][INFO] - ===> Epoch[8663](43315/5): Loss 0.0092, Sem 0.0092 LR: 3.161e-02 Score 99.668 Data time: 0.1918, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8922 [2021-07-08 21:49:59,933][root][INFO] - ===> Epoch[8664](43320/5): Loss 0.0090, Sem 0.0090 LR: 3.160e-02 Score 99.675 Data time: 0.2237, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.9111 [2021-07-08 21:50:13,001][root][INFO] - ===> Epoch[8665](43325/5): Loss 0.0088, Sem 0.0088 LR: 3.159e-02 Score 99.682 Data time: 0.1696, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.5736 [2021-07-08 21:50:27,387][root][INFO] - ===> Epoch[8666](43330/5): Loss 0.0095, Sem 0.0095 LR: 3.158e-02 Score 99.664 Data time: 0.2450, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8365 [2021-07-08 21:50:41,451][root][INFO] - ===> Epoch[8667](43335/5): Loss 0.0089, Sem 0.0089 LR: 3.157e-02 Score 99.678 Data time: 0.1796, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7699 [2021-07-08 21:50:55,967][root][INFO] - ===> Epoch[8668](43340/5): Loss 0.0088, Sem 0.0088 LR: 3.156e-02 Score 99.688 Data time: 0.2346, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8636 [2021-07-08 21:51:09,604][root][INFO] - ===> Epoch[8669](43345/5): Loss 0.0090, Sem 0.0090 LR: 3.156e-02 Score 99.678 Data time: 0.2503, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6616 [2021-07-08 21:51:23,573][root][INFO] - ===> Epoch[8670](43350/5): Loss 0.0091, Sem 0.0091 LR: 3.155e-02 Score 99.672 Data time: 0.2269, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7515 [2021-07-08 21:51:37,542][root][INFO] - ===> Epoch[8671](43355/5): Loss 0.0090, Sem 0.0090 LR: 3.154e-02 Score 99.676 Data time: 0.2103, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7458 [2021-07-08 21:51:51,443][root][INFO] - ===> Epoch[8672](43360/5): Loss 0.0088, Sem 0.0088 LR: 3.153e-02 Score 99.685 Data time: 0.2205, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7378 [2021-07-08 21:52:04,774][root][INFO] - ===> Epoch[8673](43365/5): Loss 0.0099, Sem 0.0099 LR: 3.152e-02 Score 99.644 Data time: 0.2447, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6268 [2021-07-08 21:52:18,514][root][INFO] - ===> Epoch[8674](43370/5): Loss 0.0108, Sem 0.0108 LR: 3.151e-02 Score 99.640 Data time: 0.1828, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6912 [2021-07-08 21:52:32,754][root][INFO] - ===> Epoch[8675](43375/5): Loss 0.0106, Sem 0.0106 LR: 3.150e-02 Score 99.627 Data time: 0.2069, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7992 [2021-07-08 21:52:46,792][root][INFO] - ===> Epoch[8676](43380/5): Loss 0.0121, Sem 0.0121 LR: 3.150e-02 Score 99.569 Data time: 0.2000, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7582 [2021-07-08 21:52:59,463][root][INFO] - ===> Epoch[8677](43385/5): Loss 0.0107, Sem 0.0107 LR: 3.149e-02 Score 99.615 Data time: 0.1867, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.4974 [2021-07-08 21:53:13,810][root][INFO] - ===> Epoch[8678](43390/5): Loss 0.0127, Sem 0.0127 LR: 3.148e-02 Score 99.600 Data time: 0.2278, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8221 [2021-07-08 21:53:27,471][root][INFO] - ===> Epoch[8679](43395/5): Loss 0.0102, Sem 0.0102 LR: 3.147e-02 Score 99.630 Data time: 0.2156, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6947 [2021-07-08 21:53:41,503][root][INFO] - ===> Epoch[8680](43400/5): Loss 0.0109, Sem 0.0109 LR: 3.146e-02 Score 99.626 Data time: 0.2294, Forward time: 1.2544, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7382 [2021-07-08 21:53:41,505][root][INFO] - ===> Start testing [2021-07-08 21:54:16,660][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1264 Loss 0.300 (AVG: 0.597) Score 93.514 (AVG: 90.668) mIOU 70.416 mAP 82.557 mAcc 76.556 IOU: 64.047 0.000 80.514 74.768 94.638 91.986 25.082 82.266 98.202 80.882 79.185 85.572 58.268 mAP: 83.676 3.779 87.398 86.042 98.511 98.371 49.384 97.369 99.720 92.521 91.556 93.623 91.290 mAcc: 81.821 0.000 82.483 81.210 96.070 97.319 27.007 93.340 99.387 86.484 90.747 96.071 63.294 [2021-07-08 21:54:16,660][root][INFO] - Finished test. Elapsed time: 35.1551 [2021-07-08 21:54:17,816][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 21:54:30,410][root][INFO] - ===> Epoch[8681](43405/5): Loss 0.0099, Sem 0.0099 LR: 3.145e-02 Score 99.642 Data time: 0.1730, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.4821 [2021-07-08 21:54:45,198][root][INFO] - ===> Epoch[8682](43410/5): Loss 0.0102, Sem 0.0102 LR: 3.144e-02 Score 99.642 Data time: 0.2169, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.9103 [2021-07-08 21:54:59,352][root][INFO] - ===> Epoch[8683](43415/5): Loss 0.0100, Sem 0.0100 LR: 3.144e-02 Score 99.645 Data time: 0.2498, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7851 [2021-07-08 21:55:13,208][root][INFO] - ===> Epoch[8684](43420/5): Loss 0.0095, Sem 0.0095 LR: 3.143e-02 Score 99.661 Data time: 0.2730, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7221 [2021-07-08 21:55:27,690][root][INFO] - ===> Epoch[8685](43425/5): Loss 0.0091, Sem 0.0091 LR: 3.142e-02 Score 99.670 Data time: 0.1941, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.8551 [2021-07-08 21:55:41,702][root][INFO] - ===> Epoch[8686](43430/5): Loss 0.0090, Sem 0.0090 LR: 3.141e-02 Score 99.679 Data time: 0.2102, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.7533 [2021-07-08 21:55:56,726][root][INFO] - ===> Epoch[8687](43435/5): Loss 0.0094, Sem 0.0094 LR: 3.140e-02 Score 99.662 Data time: 0.2333, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.9554 [2021-07-08 21:56:10,378][root][INFO] - ===> Epoch[8688](43440/5): Loss 0.0087, Sem 0.0087 LR: 3.139e-02 Score 99.686 Data time: 0.2299, Forward time: 1.2543, Backward time: 1.2447, DDP time: 0.0042, Total iter time: 2.6868 [2021-07-08 21:56:26,059][root][INFO] - ===> Epoch[8689](43445/5): Loss 0.0089, Sem 0.0089 LR: 3.138e-02 Score 99.684 Data time: 0.2251, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 3.0852 [2021-07-08 21:56:41,269][root][INFO] - ===> Epoch[8690](43450/5): Loss 0.0095, Sem 0.0095 LR: 3.138e-02 Score 99.658 Data time: 0.2473, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.9953 [2021-07-08 21:56:55,480][root][INFO] - ===> Epoch[8691](43455/5): Loss 0.0088, Sem 0.0088 LR: 3.137e-02 Score 99.685 Data time: 0.2885, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.7884 [2021-07-08 21:57:10,470][root][INFO] - ===> Epoch[8692](43460/5): Loss 0.0086, Sem 0.0086 LR: 3.136e-02 Score 99.689 Data time: 0.2441, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.9510 [2021-07-08 21:57:24,595][root][INFO] - ===> Epoch[8693](43465/5): Loss 0.0089, Sem 0.0089 LR: 3.135e-02 Score 99.680 Data time: 0.2675, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.7587 [2021-07-08 21:57:38,875][root][INFO] - ===> Epoch[8694](43470/5): Loss 0.0093, Sem 0.0093 LR: 3.134e-02 Score 99.666 Data time: 0.2490, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.8154 [2021-07-08 21:57:51,833][root][INFO] - ===> Epoch[8695](43475/5): Loss 0.0091, Sem 0.0091 LR: 3.133e-02 Score 99.676 Data time: 0.2391, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.5503 [2021-07-08 21:58:05,764][root][INFO] - ===> Epoch[8696](43480/5): Loss 0.0092, Sem 0.0092 LR: 3.132e-02 Score 99.667 Data time: 0.2016, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.7390 [2021-07-08 21:58:19,777][root][INFO] - ===> Epoch[8697](43485/5): Loss 0.0085, Sem 0.0085 LR: 3.132e-02 Score 99.691 Data time: 0.2487, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.7577 [2021-07-08 21:58:34,401][root][INFO] - ===> Epoch[8698](43490/5): Loss 0.0088, Sem 0.0088 LR: 3.131e-02 Score 99.685 Data time: 0.2517, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.8803 [2021-07-08 21:58:47,951][root][INFO] - ===> Epoch[8699](43495/5): Loss 0.0089, Sem 0.0089 LR: 3.130e-02 Score 99.679 Data time: 0.1884, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.6702 [2021-07-08 21:59:01,507][root][INFO] - ===> Epoch[8700](43500/5): Loss 0.0089, Sem 0.0089 LR: 3.129e-02 Score 99.680 Data time: 0.2105, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.6700 [2021-07-08 21:59:16,311][root][INFO] - ===> Epoch[8701](43505/5): Loss 0.0086, Sem 0.0086 LR: 3.128e-02 Score 99.686 Data time: 0.2111, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.9124 [2021-07-08 21:59:30,624][root][INFO] - ===> Epoch[8702](43510/5): Loss 0.0086, Sem 0.0086 LR: 3.127e-02 Score 99.691 Data time: 0.2200, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.8034 [2021-07-08 21:59:45,317][root][INFO] - ===> Epoch[8703](43515/5): Loss 0.0088, Sem 0.0088 LR: 3.127e-02 Score 99.683 Data time: 0.2085, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.8980 [2021-07-08 22:00:00,016][root][INFO] - ===> Epoch[8704](43520/5): Loss 0.0087, Sem 0.0087 LR: 3.126e-02 Score 99.689 Data time: 0.2227, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8942 [2021-07-08 22:00:13,960][root][INFO] - ===> Epoch[8705](43525/5): Loss 0.0088, Sem 0.0088 LR: 3.125e-02 Score 99.683 Data time: 0.2376, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.7427 [2021-07-08 22:00:27,350][root][INFO] - ===> Epoch[8706](43530/5): Loss 0.0088, Sem 0.0088 LR: 3.124e-02 Score 99.686 Data time: 0.1879, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.6304 [2021-07-08 22:00:41,264][root][INFO] - ===> Epoch[8707](43535/5): Loss 0.0087, Sem 0.0087 LR: 3.123e-02 Score 99.688 Data time: 0.2371, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.7323 [2021-07-08 22:00:55,167][root][INFO] - ===> Epoch[8708](43540/5): Loss 0.0088, Sem 0.0088 LR: 3.122e-02 Score 99.684 Data time: 0.2654, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.7343 [2021-07-08 22:01:08,310][root][INFO] - ===> Epoch[8709](43545/5): Loss 0.0087, Sem 0.0087 LR: 3.121e-02 Score 99.686 Data time: 0.2115, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.5850 [2021-07-08 22:01:21,289][root][INFO] - ===> Epoch[8710](43550/5): Loss 0.0086, Sem 0.0086 LR: 3.121e-02 Score 99.691 Data time: 0.1639, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.5639 [2021-07-08 22:01:35,054][root][INFO] - ===> Epoch[8711](43555/5): Loss 0.0091, Sem 0.0091 LR: 3.120e-02 Score 99.674 Data time: 0.2112, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.7145 [2021-07-08 22:01:49,251][root][INFO] - ===> Epoch[8712](43560/5): Loss 0.0083, Sem 0.0083 LR: 3.119e-02 Score 99.702 Data time: 0.2391, Forward time: 1.2544, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.8049 [2021-07-08 22:02:02,763][root][INFO] - ===> Epoch[8713](43565/5): Loss 0.0090, Sem 0.0090 LR: 3.118e-02 Score 99.675 Data time: 0.2434, Forward time: 1.2544, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.6684 [2021-07-08 22:02:17,018][root][INFO] - ===> Epoch[8714](43570/5): Loss 0.0085, Sem 0.0085 LR: 3.117e-02 Score 99.694 Data time: 0.1964, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.8052 [2021-07-08 22:02:30,911][root][INFO] - ===> Epoch[8715](43575/5): Loss 0.0087, Sem 0.0087 LR: 3.116e-02 Score 99.685 Data time: 0.2091, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.7389 [2021-07-08 22:02:45,075][root][INFO] - ===> Epoch[8716](43580/5): Loss 0.0088, Sem 0.0088 LR: 3.115e-02 Score 99.680 Data time: 0.2095, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.7868 [2021-07-08 22:02:58,676][root][INFO] - ===> Epoch[8717](43585/5): Loss 0.0085, Sem 0.0085 LR: 3.115e-02 Score 99.693 Data time: 0.1988, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.6765 [2021-07-08 22:03:13,262][root][INFO] - ===> Epoch[8718](43590/5): Loss 0.0087, Sem 0.0087 LR: 3.114e-02 Score 99.688 Data time: 0.1980, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.8682 [2021-07-08 22:03:27,583][root][INFO] - ===> Epoch[8719](43595/5): Loss 0.0084, Sem 0.0084 LR: 3.113e-02 Score 99.694 Data time: 0.2278, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.8038 [2021-07-08 22:03:41,849][root][INFO] - ===> Epoch[8720](43600/5): Loss 0.0083, Sem 0.0083 LR: 3.112e-02 Score 99.702 Data time: 0.2742, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.8047 [2021-07-08 22:03:41,851][root][INFO] - ===> Start testing [2021-07-08 22:04:18,316][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1311 Loss 0.357 (AVG: 0.590) Score 91.888 (AVG: 91.105) mIOU 70.929 mAP 82.800 mAcc 77.099 IOU: 65.661 0.000 80.905 76.240 95.327 92.152 24.024 80.980 98.657 83.088 81.056 86.112 57.881 mAP: 84.018 3.825 90.286 86.793 98.683 98.553 48.087 97.407 99.735 93.349 91.845 93.024 90.794 mAcc: 81.523 0.000 84.168 83.799 96.566 97.360 25.958 94.254 99.439 90.637 90.581 96.403 61.594 [2021-07-08 22:04:18,316][root][INFO] - Finished test. Elapsed time: 36.4644 [2021-07-08 22:04:19,484][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 22:04:32,657][root][INFO] - ===> Epoch[8721](43605/5): Loss 0.0086, Sem 0.0086 LR: 3.111e-02 Score 99.689 Data time: 0.1392, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.6096 [2021-07-08 22:04:46,538][root][INFO] - ===> Epoch[8722](43610/5): Loss 0.0084, Sem 0.0084 LR: 3.110e-02 Score 99.698 Data time: 0.2047, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.7258 [2021-07-08 22:05:00,569][root][INFO] - ===> Epoch[8723](43615/5): Loss 0.0086, Sem 0.0086 LR: 3.109e-02 Score 99.692 Data time: 0.1702, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.7527 [2021-07-08 22:05:14,640][root][INFO] - ===> Epoch[8724](43620/5): Loss 0.0087, Sem 0.0087 LR: 3.109e-02 Score 99.686 Data time: 0.1974, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.7634 [2021-07-08 22:05:28,578][root][INFO] - ===> Epoch[8725](43625/5): Loss 0.0087, Sem 0.0087 LR: 3.108e-02 Score 99.686 Data time: 0.2023, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.7330 [2021-07-08 22:05:43,136][root][INFO] - ===> Epoch[8726](43630/5): Loss 0.0086, Sem 0.0086 LR: 3.107e-02 Score 99.691 Data time: 0.2446, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.8573 [2021-07-08 22:05:57,395][root][INFO] - ===> Epoch[8727](43635/5): Loss 0.0084, Sem 0.0084 LR: 3.106e-02 Score 99.696 Data time: 0.1841, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.8131 [2021-07-08 22:06:10,441][root][INFO] - ===> Epoch[8728](43640/5): Loss 0.0087, Sem 0.0087 LR: 3.105e-02 Score 99.688 Data time: 0.2023, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.5660 [2021-07-08 22:06:24,468][root][INFO] - ===> Epoch[8729](43645/5): Loss 0.0088, Sem 0.0088 LR: 3.104e-02 Score 99.682 Data time: 0.1938, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.7515 [2021-07-08 22:06:38,147][root][INFO] - ===> Epoch[8730](43650/5): Loss 0.0088, Sem 0.0088 LR: 3.103e-02 Score 99.684 Data time: 0.2115, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.6938 [2021-07-08 22:06:52,412][root][INFO] - ===> Epoch[8731](43655/5): Loss 0.0086, Sem 0.0086 LR: 3.103e-02 Score 99.690 Data time: 0.2281, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.8132 [2021-07-08 22:07:06,823][root][INFO] - ===> Epoch[8732](43660/5): Loss 0.0091, Sem 0.0091 LR: 3.102e-02 Score 99.685 Data time: 0.2137, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.8356 [2021-07-08 22:07:21,383][root][INFO] - ===> Epoch[8733](43665/5): Loss 0.0086, Sem 0.0086 LR: 3.101e-02 Score 99.688 Data time: 0.2336, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8683 [2021-07-08 22:07:35,569][root][INFO] - ===> Epoch[8734](43670/5): Loss 0.0089, Sem 0.0089 LR: 3.100e-02 Score 99.681 Data time: 0.1566, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8017 [2021-07-08 22:07:50,042][root][INFO] - ===> Epoch[8735](43675/5): Loss 0.0091, Sem 0.0091 LR: 3.099e-02 Score 99.674 Data time: 0.3409, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8357 [2021-07-08 22:08:04,522][root][INFO] - ===> Epoch[8736](43680/5): Loss 0.0085, Sem 0.0085 LR: 3.098e-02 Score 99.691 Data time: 0.3027, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.8418 [2021-07-08 22:08:18,858][root][INFO] - ===> Epoch[8737](43685/5): Loss 0.0087, Sem 0.0087 LR: 3.097e-02 Score 99.685 Data time: 0.2123, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8174 [2021-07-08 22:08:33,864][root][INFO] - ===> Epoch[8738](43690/5): Loss 0.0087, Sem 0.0087 LR: 3.097e-02 Score 99.687 Data time: 0.2067, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.9551 [2021-07-08 22:08:47,092][root][INFO] - ===> Epoch[8739](43695/5): Loss 0.0089, Sem 0.0089 LR: 3.096e-02 Score 99.677 Data time: 0.2366, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.5933 [2021-07-08 22:09:02,858][root][INFO] - ===> Epoch[8740](43700/5): Loss 0.0086, Sem 0.0086 LR: 3.095e-02 Score 99.689 Data time: 0.3315, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 3.0997 [2021-07-08 22:09:17,056][root][INFO] - ===> Epoch[8741](43705/5): Loss 0.0086, Sem 0.0086 LR: 3.094e-02 Score 99.693 Data time: 0.2518, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7839 [2021-07-08 22:09:31,748][root][INFO] - ===> Epoch[8742](43710/5): Loss 0.0087, Sem 0.0087 LR: 3.093e-02 Score 99.684 Data time: 0.2859, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8947 [2021-07-08 22:09:46,032][root][INFO] - ===> Epoch[8743](43715/5): Loss 0.0084, Sem 0.0084 LR: 3.092e-02 Score 99.695 Data time: 0.1518, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8264 [2021-07-08 22:10:00,552][root][INFO] - ===> Epoch[8744](43720/5): Loss 0.0086, Sem 0.0086 LR: 3.092e-02 Score 99.692 Data time: 0.2201, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8422 [2021-07-08 22:10:13,695][root][INFO] - ===> Epoch[8745](43725/5): Loss 0.0087, Sem 0.0087 LR: 3.091e-02 Score 99.689 Data time: 0.1447, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.5921 [2021-07-08 22:10:27,191][root][INFO] - ===> Epoch[8746](43730/5): Loss 0.0085, Sem 0.0085 LR: 3.090e-02 Score 99.692 Data time: 0.2254, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.6413 [2021-07-08 22:10:41,509][root][INFO] - ===> Epoch[8747](43735/5): Loss 0.0084, Sem 0.0084 LR: 3.089e-02 Score 99.700 Data time: 0.1778, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8227 [2021-07-08 22:10:55,914][root][INFO] - ===> Epoch[8748](43740/5): Loss 0.0083, Sem 0.0083 LR: 3.088e-02 Score 99.701 Data time: 0.2074, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8325 [2021-07-08 22:11:10,015][root][INFO] - ===> Epoch[8749](43745/5): Loss 0.0085, Sem 0.0085 LR: 3.087e-02 Score 99.693 Data time: 0.2678, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7798 [2021-07-08 22:11:24,481][root][INFO] - ===> Epoch[8750](43750/5): Loss 0.0085, Sem 0.0085 LR: 3.086e-02 Score 99.693 Data time: 0.2292, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8396 [2021-07-08 22:11:37,350][root][INFO] - ===> Epoch[8751](43755/5): Loss 0.0088, Sem 0.0088 LR: 3.086e-02 Score 99.682 Data time: 0.2886, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.5282 [2021-07-08 22:11:52,347][root][INFO] - ===> Epoch[8752](43760/5): Loss 0.0086, Sem 0.0086 LR: 3.085e-02 Score 99.688 Data time: 0.1964, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.9538 [2021-07-08 22:12:06,084][root][INFO] - ===> Epoch[8753](43765/5): Loss 0.0089, Sem 0.0089 LR: 3.084e-02 Score 99.687 Data time: 0.2554, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7005 [2021-07-08 22:12:20,228][root][INFO] - ===> Epoch[8754](43770/5): Loss 0.0083, Sem 0.0083 LR: 3.083e-02 Score 99.700 Data time: 0.2370, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7798 [2021-07-08 22:12:33,989][root][INFO] - ===> Epoch[8755](43775/5): Loss 0.0085, Sem 0.0085 LR: 3.082e-02 Score 99.694 Data time: 0.2261, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7028 [2021-07-08 22:12:47,636][root][INFO] - ===> Epoch[8756](43780/5): Loss 0.0087, Sem 0.0087 LR: 3.081e-02 Score 99.686 Data time: 0.2364, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.6874 [2021-07-08 22:13:01,510][root][INFO] - ===> Epoch[8757](43785/5): Loss 0.0086, Sem 0.0086 LR: 3.080e-02 Score 99.690 Data time: 0.1965, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7269 [2021-07-08 22:13:15,387][root][INFO] - ===> Epoch[8758](43790/5): Loss 0.0087, Sem 0.0087 LR: 3.080e-02 Score 99.687 Data time: 0.2193, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7352 [2021-07-08 22:13:28,797][root][INFO] - ===> Epoch[8759](43795/5): Loss 0.0084, Sem 0.0084 LR: 3.079e-02 Score 99.699 Data time: 0.2177, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.6423 [2021-07-08 22:13:42,417][root][INFO] - ===> Epoch[8760](43800/5): Loss 0.0082, Sem 0.0082 LR: 3.078e-02 Score 99.704 Data time: 0.2381, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.6851 [2021-07-08 22:13:42,418][root][INFO] - ===> Start testing [2021-07-08 22:14:22,288][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1409 Loss 0.346 (AVG: 0.588) Score 93.080 (AVG: 91.193) mIOU 71.397 mAP 82.961 mAcc 77.548 IOU: 66.210 0.000 82.297 76.472 95.202 92.380 26.976 80.840 98.605 83.831 81.161 86.085 58.099 mAP: 83.764 3.874 89.173 86.730 98.654 98.421 51.152 97.190 99.752 94.380 91.782 92.717 90.910 mAcc: 82.301 0.000 84.619 83.692 96.510 97.348 29.185 92.862 99.384 92.788 91.280 96.350 61.807 [2021-07-08 22:14:22,289][root][INFO] - Finished test. Elapsed time: 39.8695 [2021-07-08 22:14:23,379][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 22:14:36,206][root][INFO] - ===> Epoch[8761](43805/5): Loss 0.0087, Sem 0.0087 LR: 3.077e-02 Score 99.686 Data time: 0.1655, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.5234 [2021-07-08 22:14:49,556][root][INFO] - ===> Epoch[8762](43810/5): Loss 0.0086, Sem 0.0086 LR: 3.076e-02 Score 99.690 Data time: 0.2363, Forward time: 1.2542, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.6292 [2021-07-08 22:15:03,919][root][INFO] - ===> Epoch[8763](43815/5): Loss 0.0083, Sem 0.0083 LR: 3.075e-02 Score 99.701 Data time: 0.2648, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8286 [2021-07-08 22:15:17,121][root][INFO] - ===> Epoch[8764](43820/5): Loss 0.0086, Sem 0.0086 LR: 3.074e-02 Score 99.690 Data time: 0.2501, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.6011 [2021-07-08 22:15:31,683][root][INFO] - ===> Epoch[8765](43825/5): Loss 0.0084, Sem 0.0084 LR: 3.074e-02 Score 99.699 Data time: 0.1917, Forward time: 1.2542, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8689 [2021-07-08 22:15:45,569][root][INFO] - ===> Epoch[8766](43830/5): Loss 0.0083, Sem 0.0083 LR: 3.073e-02 Score 99.703 Data time: 0.2320, Forward time: 1.2542, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7188 [2021-07-08 22:15:59,704][root][INFO] - ===> Epoch[8767](43835/5): Loss 0.0083, Sem 0.0083 LR: 3.072e-02 Score 99.702 Data time: 0.2192, Forward time: 1.2542, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7742 [2021-07-08 22:16:14,029][root][INFO] - ===> Epoch[8768](43840/5): Loss 0.0086, Sem 0.0086 LR: 3.071e-02 Score 99.691 Data time: 0.1971, Forward time: 1.2542, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8182 [2021-07-08 22:16:27,886][root][INFO] - ===> Epoch[8769](43845/5): Loss 0.0086, Sem 0.0086 LR: 3.070e-02 Score 99.693 Data time: 0.1771, Forward time: 1.2542, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7375 [2021-07-08 22:16:42,507][root][INFO] - ===> Epoch[8770](43850/5): Loss 0.0084, Sem 0.0084 LR: 3.069e-02 Score 99.694 Data time: 0.2728, Forward time: 1.2542, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8730 [2021-07-08 22:16:56,423][root][INFO] - ===> Epoch[8771](43855/5): Loss 0.0084, Sem 0.0084 LR: 3.068e-02 Score 99.700 Data time: 0.2100, Forward time: 1.2542, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7250 [2021-07-08 22:17:09,622][root][INFO] - ===> Epoch[8772](43860/5): Loss 0.0085, Sem 0.0085 LR: 3.068e-02 Score 99.694 Data time: 0.2174, Forward time: 1.2542, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.5931 [2021-07-08 22:17:24,055][root][INFO] - ===> Epoch[8773](43865/5): Loss 0.0085, Sem 0.0085 LR: 3.067e-02 Score 99.695 Data time: 0.2436, Forward time: 1.2542, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8441 [2021-07-08 22:17:37,880][root][INFO] - ===> Epoch[8774](43870/5): Loss 0.0083, Sem 0.0083 LR: 3.066e-02 Score 99.699 Data time: 0.1901, Forward time: 1.2542, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7221 [2021-07-08 22:17:52,179][root][INFO] - ===> Epoch[8775](43875/5): Loss 0.0084, Sem 0.0084 LR: 3.065e-02 Score 99.701 Data time: 0.1860, Forward time: 1.2542, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8205 [2021-07-08 22:18:06,825][root][INFO] - ===> Epoch[8776](43880/5): Loss 0.0084, Sem 0.0084 LR: 3.064e-02 Score 99.695 Data time: 0.1581, Forward time: 1.2542, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.8967 [2021-07-08 22:18:20,690][root][INFO] - ===> Epoch[8777](43885/5): Loss 0.0088, Sem 0.0088 LR: 3.063e-02 Score 99.687 Data time: 0.2075, Forward time: 1.2542, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7244 [2021-07-08 22:18:34,532][root][INFO] - ===> Epoch[8778](43890/5): Loss 0.0081, Sem 0.0081 LR: 3.062e-02 Score 99.709 Data time: 0.2161, Forward time: 1.2542, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7217 [2021-07-08 22:18:48,754][root][INFO] - ===> Epoch[8779](43895/5): Loss 0.0086, Sem 0.0086 LR: 3.062e-02 Score 99.692 Data time: 0.2357, Forward time: 1.2542, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.8035 [2021-07-08 22:19:02,988][root][INFO] - ===> Epoch[8780](43900/5): Loss 0.0087, Sem 0.0087 LR: 3.061e-02 Score 99.688 Data time: 0.2062, Forward time: 1.2542, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7965 [2021-07-08 22:19:16,804][root][INFO] - ===> Epoch[8781](43905/5): Loss 0.0086, Sem 0.0086 LR: 3.060e-02 Score 99.691 Data time: 0.2555, Forward time: 1.2542, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7200 [2021-07-08 22:19:31,178][root][INFO] - ===> Epoch[8782](43910/5): Loss 0.0082, Sem 0.0082 LR: 3.059e-02 Score 99.707 Data time: 0.2810, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8325 [2021-07-08 22:19:45,057][root][INFO] - ===> Epoch[8783](43915/5): Loss 0.0081, Sem 0.0081 LR: 3.058e-02 Score 99.707 Data time: 0.2587, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7313 [2021-07-08 22:19:59,571][root][INFO] - ===> Epoch[8784](43920/5): Loss 0.0090, Sem 0.0090 LR: 3.057e-02 Score 99.675 Data time: 0.2488, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8656 [2021-07-08 22:20:13,012][root][INFO] - ===> Epoch[8785](43925/5): Loss 0.0086, Sem 0.0086 LR: 3.056e-02 Score 99.690 Data time: 0.2656, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.6421 [2021-07-08 22:20:26,810][root][INFO] - ===> Epoch[8786](43930/5): Loss 0.0086, Sem 0.0086 LR: 3.056e-02 Score 99.691 Data time: 0.3002, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7109 [2021-07-08 22:20:40,727][root][INFO] - ===> Epoch[8787](43935/5): Loss 0.0087, Sem 0.0087 LR: 3.055e-02 Score 99.693 Data time: 0.2809, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7280 [2021-07-08 22:20:54,839][root][INFO] - ===> Epoch[8788](43940/5): Loss 0.0086, Sem 0.0086 LR: 3.054e-02 Score 99.690 Data time: 0.2354, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7798 [2021-07-08 22:21:09,318][root][INFO] - ===> Epoch[8789](43945/5): Loss 0.0082, Sem 0.0082 LR: 3.053e-02 Score 99.705 Data time: 0.1935, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8508 [2021-07-08 22:21:23,072][root][INFO] - ===> Epoch[8790](43950/5): Loss 0.0083, Sem 0.0083 LR: 3.052e-02 Score 99.701 Data time: 0.2270, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7008 [2021-07-08 22:21:36,853][root][INFO] - ===> Epoch[8791](43955/5): Loss 0.0083, Sem 0.0083 LR: 3.051e-02 Score 99.702 Data time: 0.2340, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7104 [2021-07-08 22:21:50,072][root][INFO] - ===> Epoch[8792](43960/5): Loss 0.0086, Sem 0.0086 LR: 3.050e-02 Score 99.688 Data time: 0.2195, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.5894 [2021-07-08 22:22:04,087][root][INFO] - ===> Epoch[8793](43965/5): Loss 0.0082, Sem 0.0082 LR: 3.050e-02 Score 99.705 Data time: 0.2506, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.7609 [2021-07-08 22:22:17,001][root][INFO] - ===> Epoch[8794](43970/5): Loss 0.0086, Sem 0.0086 LR: 3.049e-02 Score 99.696 Data time: 0.1820, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.5418 [2021-07-08 22:22:30,521][root][INFO] - ===> Epoch[8795](43975/5): Loss 0.0090, Sem 0.0090 LR: 3.048e-02 Score 99.679 Data time: 0.1918, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.6546 [2021-07-08 22:22:46,095][root][INFO] - ===> Epoch[8796](43980/5): Loss 0.0087, Sem 0.0087 LR: 3.047e-02 Score 99.687 Data time: 0.2248, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 3.0601 [2021-07-08 22:22:59,807][root][INFO] - ===> Epoch[8797](43985/5): Loss 0.0084, Sem 0.0084 LR: 3.046e-02 Score 99.695 Data time: 0.2209, Forward time: 1.2543, Backward time: 1.2448, DDP time: 0.0042, Total iter time: 2.7035 [2021-07-08 22:23:15,351][root][INFO] - ===> Epoch[8798](43990/5): Loss 0.0084, Sem 0.0084 LR: 3.045e-02 Score 99.693 Data time: 0.2453, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 3.0691 [2021-07-08 22:23:29,722][root][INFO] - ===> Epoch[8799](43995/5): Loss 0.0087, Sem 0.0087 LR: 3.044e-02 Score 99.687 Data time: 0.2263, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8272 [2021-07-08 22:23:44,231][root][INFO] - ===> Epoch[8800](44000/5): Loss 0.0084, Sem 0.0084 LR: 3.044e-02 Score 99.700 Data time: 0.1862, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8605 [2021-07-08 22:23:44,232][root][INFO] - ===> Start testing [2021-07-08 22:24:18,067][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1248 Loss 0.386 (AVG: 0.598) Score 90.960 (AVG: 91.055) mIOU 71.368 mAP 82.744 mAcc 77.507 IOU: 65.184 0.000 81.534 75.751 95.114 91.784 28.544 83.368 98.610 83.784 80.117 85.949 58.039 mAP: 83.285 3.448 87.293 86.174 98.486 98.414 51.546 97.681 99.776 94.198 91.253 93.214 90.906 mAcc: 81.573 0.000 84.791 82.545 96.565 97.239 30.706 95.072 99.259 91.190 90.594 96.410 61.651 [2021-07-08 22:24:18,068][root][INFO] - Finished test. Elapsed time: 33.8343 [2021-07-08 22:24:19,258][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 22:24:32,630][root][INFO] - ===> Epoch[8801](44005/5): Loss 0.0083, Sem 0.0083 LR: 3.043e-02 Score 99.705 Data time: 0.1961, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.6343 [2021-07-08 22:24:47,486][root][INFO] - ===> Epoch[8802](44010/5): Loss 0.0083, Sem 0.0083 LR: 3.042e-02 Score 99.701 Data time: 0.2348, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.9292 [2021-07-08 22:25:01,060][root][INFO] - ===> Epoch[8803](44015/5): Loss 0.0087, Sem 0.0087 LR: 3.041e-02 Score 99.683 Data time: 0.1999, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.6751 [2021-07-08 22:25:15,186][root][INFO] - ===> Epoch[8804](44020/5): Loss 0.0082, Sem 0.0082 LR: 3.040e-02 Score 99.706 Data time: 0.2419, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7844 [2021-07-08 22:25:28,791][root][INFO] - ===> Epoch[8805](44025/5): Loss 0.0082, Sem 0.0082 LR: 3.039e-02 Score 99.705 Data time: 0.1720, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.6821 [2021-07-08 22:25:42,532][root][INFO] - ===> Epoch[8806](44030/5): Loss 0.0085, Sem 0.0085 LR: 3.038e-02 Score 99.696 Data time: 0.2379, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.6976 [2021-07-08 22:25:57,127][root][INFO] - ===> Epoch[8807](44035/5): Loss 0.0082, Sem 0.0082 LR: 3.038e-02 Score 99.704 Data time: 0.2638, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8749 [2021-07-08 22:26:10,876][root][INFO] - ===> Epoch[8808](44040/5): Loss 0.0084, Sem 0.0084 LR: 3.037e-02 Score 99.696 Data time: 0.2202, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7051 [2021-07-08 22:26:25,643][root][INFO] - ===> Epoch[8809](44045/5): Loss 0.0084, Sem 0.0084 LR: 3.036e-02 Score 99.698 Data time: 0.1903, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8987 [2021-07-08 22:26:39,596][root][INFO] - ===> Epoch[8810](44050/5): Loss 0.0082, Sem 0.0082 LR: 3.035e-02 Score 99.707 Data time: 0.1818, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7558 [2021-07-08 22:26:54,910][root][INFO] - ===> Epoch[8811](44055/5): Loss 0.0083, Sem 0.0083 LR: 3.034e-02 Score 99.699 Data time: 0.2315, Forward time: 1.2542, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 3.0205 [2021-07-08 22:27:08,874][root][INFO] - ===> Epoch[8812](44060/5): Loss 0.0089, Sem 0.0089 LR: 3.033e-02 Score 99.688 Data time: 0.1576, Forward time: 1.2542, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7454 [2021-07-08 22:27:22,653][root][INFO] - ===> Epoch[8813](44065/5): Loss 0.0079, Sem 0.0079 LR: 3.032e-02 Score 99.718 Data time: 0.1896, Forward time: 1.2542, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7063 [2021-07-08 22:27:36,694][root][INFO] - ===> Epoch[8814](44070/5): Loss 0.0085, Sem 0.0085 LR: 3.032e-02 Score 99.691 Data time: 0.2097, Forward time: 1.2542, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7665 [2021-07-08 22:27:51,768][root][INFO] - ===> Epoch[8815](44075/5): Loss 0.0080, Sem 0.0080 LR: 3.031e-02 Score 99.712 Data time: 0.2631, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.9717 [2021-07-08 22:28:05,488][root][INFO] - ===> Epoch[8816](44080/5): Loss 0.0088, Sem 0.0088 LR: 3.030e-02 Score 99.685 Data time: 0.1754, Forward time: 1.2542, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7032 [2021-07-08 22:28:19,612][root][INFO] - ===> Epoch[8817](44085/5): Loss 0.0086, Sem 0.0086 LR: 3.029e-02 Score 99.689 Data time: 0.2065, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7704 [2021-07-08 22:28:33,275][root][INFO] - ===> Epoch[8818](44090/5): Loss 0.0084, Sem 0.0084 LR: 3.028e-02 Score 99.695 Data time: 0.2362, Forward time: 1.2542, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.6797 [2021-07-08 22:28:47,316][root][INFO] - ===> Epoch[8819](44095/5): Loss 0.0083, Sem 0.0083 LR: 3.027e-02 Score 99.698 Data time: 0.2212, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7702 [2021-07-08 22:29:01,858][root][INFO] - ===> Epoch[8820](44100/5): Loss 0.0081, Sem 0.0081 LR: 3.026e-02 Score 99.708 Data time: 0.2703, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.8620 [2021-07-08 22:29:16,196][root][INFO] - ===> Epoch[8821](44105/5): Loss 0.0086, Sem 0.0086 LR: 3.026e-02 Score 99.693 Data time: 0.2481, Forward time: 1.2542, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.8143 [2021-07-08 22:29:30,398][root][INFO] - ===> Epoch[8822](44110/5): Loss 0.0084, Sem 0.0084 LR: 3.025e-02 Score 99.701 Data time: 0.2482, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7877 [2021-07-08 22:29:44,712][root][INFO] - ===> Epoch[8823](44115/5): Loss 0.0082, Sem 0.0082 LR: 3.024e-02 Score 99.705 Data time: 0.2498, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.8158 [2021-07-08 22:29:57,874][root][INFO] - ===> Epoch[8824](44120/5): Loss 0.0090, Sem 0.0090 LR: 3.023e-02 Score 99.675 Data time: 0.1904, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.5807 [2021-07-08 22:30:11,944][root][INFO] - ===> Epoch[8825](44125/5): Loss 0.0081, Sem 0.0081 LR: 3.022e-02 Score 99.710 Data time: 0.1999, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7655 [2021-07-08 22:30:26,063][root][INFO] - ===> Epoch[8826](44130/5): Loss 0.0084, Sem 0.0084 LR: 3.021e-02 Score 99.693 Data time: 0.2330, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7777 [2021-07-08 22:30:40,385][root][INFO] - ===> Epoch[8827](44135/5): Loss 0.0081, Sem 0.0081 LR: 3.020e-02 Score 99.713 Data time: 0.2973, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8134 [2021-07-08 22:30:55,429][root][INFO] - ===> Epoch[8828](44140/5): Loss 0.0085, Sem 0.0085 LR: 3.020e-02 Score 99.694 Data time: 0.2492, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.9616 [2021-07-08 22:31:09,802][root][INFO] - ===> Epoch[8829](44145/5): Loss 0.0082, Sem 0.0082 LR: 3.019e-02 Score 99.706 Data time: 0.3045, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8247 [2021-07-08 22:31:24,084][root][INFO] - ===> Epoch[8830](44150/5): Loss 0.0082, Sem 0.0082 LR: 3.018e-02 Score 99.703 Data time: 0.2319, Forward time: 1.2544, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8106 [2021-07-08 22:31:38,081][root][INFO] - ===> Epoch[8831](44155/5): Loss 0.0084, Sem 0.0084 LR: 3.017e-02 Score 99.696 Data time: 0.2741, Forward time: 1.2544, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7576 [2021-07-08 22:31:51,859][root][INFO] - ===> Epoch[8832](44160/5): Loss 0.0086, Sem 0.0086 LR: 3.016e-02 Score 99.688 Data time: 0.1988, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7132 [2021-07-08 22:32:06,481][root][INFO] - ===> Epoch[8833](44165/5): Loss 0.0082, Sem 0.0082 LR: 3.015e-02 Score 99.704 Data time: 0.2349, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8692 [2021-07-08 22:32:20,289][root][INFO] - ===> Epoch[8834](44170/5): Loss 0.0082, Sem 0.0082 LR: 3.014e-02 Score 99.708 Data time: 0.1926, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7210 [2021-07-08 22:32:34,931][root][INFO] - ===> Epoch[8835](44175/5): Loss 0.0082, Sem 0.0082 LR: 3.014e-02 Score 99.705 Data time: 0.2323, Forward time: 1.2543, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8885 [2021-07-08 22:32:48,997][root][INFO] - ===> Epoch[8836](44180/5): Loss 0.0082, Sem 0.0082 LR: 3.013e-02 Score 99.708 Data time: 0.1781, Forward time: 1.2544, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7739 [2021-07-08 22:33:03,101][root][INFO] - ===> Epoch[8837](44185/5): Loss 0.0081, Sem 0.0081 LR: 3.012e-02 Score 99.714 Data time: 0.2128, Forward time: 1.2544, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.7722 [2021-07-08 22:33:17,500][root][INFO] - ===> Epoch[8838](44190/5): Loss 0.0081, Sem 0.0081 LR: 3.011e-02 Score 99.708 Data time: 0.2445, Forward time: 1.2544, Backward time: 1.2449, DDP time: 0.0042, Total iter time: 2.8377 [2021-07-08 22:33:31,465][root][INFO] - ===> Epoch[8839](44195/5): Loss 0.0083, Sem 0.0083 LR: 3.010e-02 Score 99.702 Data time: 0.2113, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7533 [2021-07-08 22:33:46,421][root][INFO] - ===> Epoch[8840](44200/5): Loss 0.0081, Sem 0.0081 LR: 3.009e-02 Score 99.706 Data time: 0.2169, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.9480 [2021-07-08 22:33:46,422][root][INFO] - ===> Start testing [2021-07-08 22:34:26,141][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1203 Loss 0.340 (AVG: 0.596) Score 92.910 (AVG: 91.179) mIOU 71.914 mAP 82.987 mAcc 77.870 IOU: 65.382 0.000 81.683 76.179 95.143 92.034 30.351 85.469 98.616 84.458 81.014 85.846 58.703 mAP: 83.099 3.014 90.196 86.937 98.499 98.377 51.333 97.995 99.755 95.080 91.455 92.699 90.387 mAcc: 80.614 0.000 85.083 83.513 96.821 97.299 32.970 94.812 99.334 92.699 90.006 96.374 62.786 [2021-07-08 22:34:26,142][root][INFO] - Finished test. Elapsed time: 39.7185 [2021-07-08 22:34:27,333][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 22:34:40,727][root][INFO] - ===> Epoch[8841](44205/5): Loss 0.0083, Sem 0.0083 LR: 3.008e-02 Score 99.702 Data time: 0.1907, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.6331 [2021-07-08 22:34:56,347][root][INFO] - ===> Epoch[8842](44210/5): Loss 0.0079, Sem 0.0079 LR: 3.008e-02 Score 99.718 Data time: 0.1819, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 3.0813 [2021-07-08 22:35:10,408][root][INFO] - ===> Epoch[8843](44215/5): Loss 0.0086, Sem 0.0086 LR: 3.007e-02 Score 99.692 Data time: 0.2394, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7637 [2021-07-08 22:35:24,304][root][INFO] - ===> Epoch[8844](44220/5): Loss 0.0082, Sem 0.0082 LR: 3.006e-02 Score 99.707 Data time: 0.2364, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7185 [2021-07-08 22:35:38,047][root][INFO] - ===> Epoch[8845](44225/5): Loss 0.0086, Sem 0.0086 LR: 3.005e-02 Score 99.690 Data time: 0.1998, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7099 [2021-07-08 22:35:51,481][root][INFO] - ===> Epoch[8846](44230/5): Loss 0.0083, Sem 0.0083 LR: 3.004e-02 Score 99.705 Data time: 0.2534, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.6359 [2021-07-08 22:36:05,811][root][INFO] - ===> Epoch[8847](44235/5): Loss 0.0083, Sem 0.0083 LR: 3.003e-02 Score 99.704 Data time: 0.1661, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.8272 [2021-07-08 22:36:19,856][root][INFO] - ===> Epoch[8848](44240/5): Loss 0.0085, Sem 0.0085 LR: 3.002e-02 Score 99.696 Data time: 0.2911, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7594 [2021-07-08 22:36:34,369][root][INFO] - ===> Epoch[8849](44245/5): Loss 0.0083, Sem 0.0083 LR: 3.002e-02 Score 99.701 Data time: 0.2343, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.8574 [2021-07-08 22:36:48,551][root][INFO] - ===> Epoch[8850](44250/5): Loss 0.0087, Sem 0.0087 LR: 3.001e-02 Score 99.689 Data time: 0.2076, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7890 [2021-07-08 22:37:03,322][root][INFO] - ===> Epoch[8851](44255/5): Loss 0.0081, Sem 0.0081 LR: 3.000e-02 Score 99.708 Data time: 0.2121, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.9144 [2021-07-08 22:37:17,537][root][INFO] - ===> Epoch[8852](44260/5): Loss 0.0081, Sem 0.0081 LR: 2.999e-02 Score 99.708 Data time: 0.2597, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.7988 [2021-07-08 22:37:31,821][root][INFO] - ===> Epoch[8853](44265/5): Loss 0.0083, Sem 0.0083 LR: 2.998e-02 Score 99.701 Data time: 0.2664, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.8136 [2021-07-08 22:37:45,432][root][INFO] - ===> Epoch[8854](44270/5): Loss 0.0086, Sem 0.0086 LR: 2.997e-02 Score 99.695 Data time: 0.2265, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.6762 [2021-07-08 22:37:59,676][root][INFO] - ===> Epoch[8855](44275/5): Loss 0.0081, Sem 0.0081 LR: 2.996e-02 Score 99.708 Data time: 0.1894, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.8046 [2021-07-08 22:38:13,830][root][INFO] - ===> Epoch[8856](44280/5): Loss 0.0081, Sem 0.0081 LR: 2.996e-02 Score 99.710 Data time: 0.2584, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7846 [2021-07-08 22:38:27,547][root][INFO] - ===> Epoch[8857](44285/5): Loss 0.0088, Sem 0.0088 LR: 2.995e-02 Score 99.690 Data time: 0.2614, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.6958 [2021-07-08 22:38:40,592][root][INFO] - ===> Epoch[8858](44290/5): Loss 0.0087, Sem 0.0087 LR: 2.994e-02 Score 99.689 Data time: 0.2686, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.5671 [2021-07-08 22:38:54,319][root][INFO] - ===> Epoch[8859](44295/5): Loss 0.0081, Sem 0.0081 LR: 2.993e-02 Score 99.709 Data time: 0.2177, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7050 [2021-07-08 22:39:08,933][root][INFO] - ===> Epoch[8860](44300/5): Loss 0.0080, Sem 0.0080 LR: 2.992e-02 Score 99.709 Data time: 0.2080, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.8855 [2021-07-08 22:39:23,292][root][INFO] - ===> Epoch[8861](44305/5): Loss 0.0082, Sem 0.0082 LR: 2.991e-02 Score 99.701 Data time: 0.2371, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.8175 [2021-07-08 22:39:36,836][root][INFO] - ===> Epoch[8862](44310/5): Loss 0.0081, Sem 0.0081 LR: 2.990e-02 Score 99.708 Data time: 0.2493, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.6622 [2021-07-08 22:39:50,727][root][INFO] - ===> Epoch[8863](44315/5): Loss 0.0085, Sem 0.0085 LR: 2.990e-02 Score 99.696 Data time: 0.2293, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7160 [2021-07-08 22:40:05,455][root][INFO] - ===> Epoch[8864](44320/5): Loss 0.0080, Sem 0.0080 LR: 2.989e-02 Score 99.716 Data time: 0.2392, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.8877 [2021-07-08 22:40:19,334][root][INFO] - ===> Epoch[8865](44325/5): Loss 0.0085, Sem 0.0085 LR: 2.988e-02 Score 99.696 Data time: 0.1875, Forward time: 1.2543, Backward time: 1.2450, DDP time: 0.0042, Total iter time: 2.7171 [2021-07-08 22:40:34,042][root][INFO] - ===> Epoch[8866](44330/5): Loss 0.0082, Sem 0.0082 LR: 2.987e-02 Score 99.706 Data time: 0.2306, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.8987 [2021-07-08 22:40:49,013][root][INFO] - ===> Epoch[8867](44335/5): Loss 0.0083, Sem 0.0083 LR: 2.986e-02 Score 99.701 Data time: 0.2184, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.9536 [2021-07-08 22:41:02,949][root][INFO] - ===> Epoch[8868](44340/5): Loss 0.0081, Sem 0.0081 LR: 2.985e-02 Score 99.708 Data time: 0.2102, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.7395 [2021-07-08 22:41:16,981][root][INFO] - ===> Epoch[8869](44345/5): Loss 0.0086, Sem 0.0086 LR: 2.984e-02 Score 99.689 Data time: 0.2369, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.7570 [2021-07-08 22:41:31,541][root][INFO] - ===> Epoch[8870](44350/5): Loss 0.0085, Sem 0.0085 LR: 2.984e-02 Score 99.699 Data time: 0.2637, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.8705 [2021-07-08 22:41:45,004][root][INFO] - ===> Epoch[8871](44355/5): Loss 0.0083, Sem 0.0083 LR: 2.983e-02 Score 99.703 Data time: 0.2072, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.6583 [2021-07-08 22:41:59,357][root][INFO] - ===> Epoch[8872](44360/5): Loss 0.0080, Sem 0.0080 LR: 2.982e-02 Score 99.713 Data time: 0.1811, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.8284 [2021-07-08 22:42:14,712][root][INFO] - ===> Epoch[8873](44365/5): Loss 0.0081, Sem 0.0081 LR: 2.981e-02 Score 99.712 Data time: 0.2843, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 3.0263 [2021-07-08 22:42:28,750][root][INFO] - ===> Epoch[8874](44370/5): Loss 0.0083, Sem 0.0083 LR: 2.980e-02 Score 99.699 Data time: 0.2573, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.7598 [2021-07-08 22:42:42,952][root][INFO] - ===> Epoch[8875](44375/5): Loss 0.0079, Sem 0.0079 LR: 2.979e-02 Score 99.716 Data time: 0.2456, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.8038 [2021-07-08 22:42:56,903][root][INFO] - ===> Epoch[8876](44380/5): Loss 0.0084, Sem 0.0084 LR: 2.978e-02 Score 99.698 Data time: 0.2783, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.7524 [2021-07-08 22:43:10,710][root][INFO] - ===> Epoch[8877](44385/5): Loss 0.0081, Sem 0.0081 LR: 2.978e-02 Score 99.712 Data time: 0.2098, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.7110 [2021-07-08 22:43:24,600][root][INFO] - ===> Epoch[8878](44390/5): Loss 0.0083, Sem 0.0083 LR: 2.977e-02 Score 99.702 Data time: 0.2625, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.7271 [2021-07-08 22:43:38,880][root][INFO] - ===> Epoch[8879](44395/5): Loss 0.0080, Sem 0.0080 LR: 2.976e-02 Score 99.712 Data time: 0.1882, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.8214 [2021-07-08 22:43:53,900][root][INFO] - ===> Epoch[8880](44400/5): Loss 0.0080, Sem 0.0080 LR: 2.975e-02 Score 99.713 Data time: 0.2688, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.9633 [2021-07-08 22:43:53,903][root][INFO] - ===> Start testing [2021-07-08 22:44:29,750][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1980 Loss 0.350 (AVG: 0.603) Score 92.800 (AVG: 91.110) mIOU 71.287 mAP 82.754 mAcc 77.354 IOU: 65.583 0.000 80.509 75.787 95.162 91.550 26.235 83.216 98.583 85.691 80.030 86.083 58.302 mAP: 83.696 3.222 87.948 86.372 98.476 98.408 49.957 97.807 99.728 94.776 91.531 93.206 90.678 mAcc: 81.691 0.000 82.859 82.304 96.595 97.583 27.928 94.482 99.324 92.936 91.634 96.744 61.521 [2021-07-08 22:44:29,750][root][INFO] - Finished test. Elapsed time: 35.8466 [2021-07-08 22:44:30,956][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 22:44:43,689][root][INFO] - ===> Epoch[8881](44405/5): Loss 0.0081, Sem 0.0081 LR: 2.974e-02 Score 99.707 Data time: 0.1776, Forward time: 1.2543, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.5033 [2021-07-08 22:44:58,175][root][INFO] - ===> Epoch[8882](44410/5): Loss 0.0084, Sem 0.0084 LR: 2.973e-02 Score 99.703 Data time: 0.2364, Forward time: 1.2542, Backward time: 1.2451, DDP time: 0.0042, Total iter time: 2.8520 [2021-07-08 22:45:13,081][root][INFO] - ===> Epoch[8883](44415/5): Loss 0.0082, Sem 0.0082 LR: 2.972e-02 Score 99.704 Data time: 0.2275, Forward time: 1.2542, Backward time: 1.2452, DDP time: 0.0042, Total iter time: 2.9414 [2021-07-08 22:45:27,623][root][INFO] - ===> Epoch[8884](44420/5): Loss 0.0081, Sem 0.0081 LR: 2.972e-02 Score 99.714 Data time: 0.2193, Forward time: 1.2542, Backward time: 1.2452, DDP time: 0.0042, Total iter time: 2.8673 [2021-07-08 22:45:41,593][root][INFO] - ===> Epoch[8885](44425/5): Loss 0.0080, Sem 0.0080 LR: 2.971e-02 Score 99.711 Data time: 0.1845, Forward time: 1.2542, Backward time: 1.2452, DDP time: 0.0042, Total iter time: 2.7474 [2021-07-08 22:45:55,691][root][INFO] - ===> Epoch[8886](44430/5): Loss 0.0085, Sem 0.0085 LR: 2.970e-02 Score 99.694 Data time: 0.1881, Forward time: 1.2542, Backward time: 1.2452, DDP time: 0.0042, Total iter time: 2.7744 [2021-07-08 22:46:10,159][root][INFO] - ===> Epoch[8887](44435/5): Loss 0.0080, Sem 0.0080 LR: 2.969e-02 Score 99.711 Data time: 0.2043, Forward time: 1.2542, Backward time: 1.2452, DDP time: 0.0042, Total iter time: 2.8535 [2021-07-08 22:46:24,369][root][INFO] - ===> Epoch[8888](44440/5): Loss 0.0079, Sem 0.0079 LR: 2.968e-02 Score 99.717 Data time: 0.3036, Forward time: 1.2542, Backward time: 1.2452, DDP time: 0.0042, Total iter time: 2.7912 [2021-07-08 22:46:39,165][root][INFO] - ===> Epoch[8889](44445/5): Loss 0.0084, Sem 0.0084 LR: 2.967e-02 Score 99.697 Data time: 0.1722, Forward time: 1.2542, Backward time: 1.2452, DDP time: 0.0042, Total iter time: 2.9244 [2021-07-08 22:46:52,310][root][INFO] - ===> Epoch[8890](44450/5): Loss 0.0082, Sem 0.0082 LR: 2.966e-02 Score 99.703 Data time: 0.1978, Forward time: 1.2542, Backward time: 1.2452, DDP time: 0.0042, Total iter time: 2.5910 [2021-07-08 22:47:06,295][root][INFO] - ===> Epoch[8891](44455/5): Loss 0.0080, Sem 0.0080 LR: 2.966e-02 Score 99.709 Data time: 0.2310, Forward time: 1.2542, Backward time: 1.2452, DDP time: 0.0042, Total iter time: 2.7450 [2021-07-08 22:47:20,135][root][INFO] - ===> Epoch[8892](44460/5): Loss 0.0084, Sem 0.0084 LR: 2.965e-02 Score 99.695 Data time: 0.2055, Forward time: 1.2542, Backward time: 1.2452, DDP time: 0.0042, Total iter time: 2.7145 [2021-07-08 22:47:34,801][root][INFO] - ===> Epoch[8893](44465/5): Loss 0.0081, Sem 0.0081 LR: 2.964e-02 Score 99.711 Data time: 0.1890, Forward time: 1.2542, Backward time: 1.2452, DDP time: 0.0042, Total iter time: 2.8909 [2021-07-08 22:47:48,558][root][INFO] - ===> Epoch[8894](44470/5): Loss 0.0079, Sem 0.0079 LR: 2.963e-02 Score 99.714 Data time: 0.2037, Forward time: 1.2542, Backward time: 1.2452, DDP time: 0.0042, Total iter time: 2.7119 [2021-07-08 22:48:01,758][root][INFO] - ===> Epoch[8895](44475/5): Loss 0.0084, Sem 0.0084 LR: 2.962e-02 Score 99.696 Data time: 0.2211, Forward time: 1.2542, Backward time: 1.2452, DDP time: 0.0042, Total iter time: 2.5873 [2021-07-08 22:48:15,404][root][INFO] - ===> Epoch[8896](44480/5): Loss 0.0081, Sem 0.0081 LR: 2.961e-02 Score 99.707 Data time: 0.1919, Forward time: 1.2542, Backward time: 1.2452, DDP time: 0.0042, Total iter time: 2.6846 [2021-07-08 22:48:29,722][root][INFO] - ===> Epoch[8897](44485/5): Loss 0.0085, Sem 0.0085 LR: 2.960e-02 Score 99.697 Data time: 0.2431, Forward time: 1.2542, Backward time: 1.2452, DDP time: 0.0042, Total iter time: 2.8231 [2021-07-08 22:48:43,282][root][INFO] - ===> Epoch[8898](44490/5): Loss 0.0088, Sem 0.0088 LR: 2.960e-02 Score 99.687 Data time: 0.1821, Forward time: 1.2542, Backward time: 1.2452, DDP time: 0.0042, Total iter time: 2.6591 [2021-07-08 22:48:57,384][root][INFO] - ===> Epoch[8899](44495/5): Loss 0.0081, Sem 0.0081 LR: 2.959e-02 Score 99.706 Data time: 0.2106, Forward time: 1.2542, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7760 [2021-07-08 22:49:12,499][root][INFO] - ===> Epoch[8900](44500/5): Loss 0.0082, Sem 0.0082 LR: 2.958e-02 Score 99.706 Data time: 0.1762, Forward time: 1.2542, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.9790 [2021-07-08 22:49:26,216][root][INFO] - ===> Epoch[8901](44505/5): Loss 0.0080, Sem 0.0080 LR: 2.957e-02 Score 99.712 Data time: 0.2427, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.6977 [2021-07-08 22:49:40,127][root][INFO] - ===> Epoch[8902](44510/5): Loss 0.0078, Sem 0.0078 LR: 2.956e-02 Score 99.716 Data time: 0.2124, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7345 [2021-07-08 22:49:54,150][root][INFO] - ===> Epoch[8903](44515/5): Loss 0.0082, Sem 0.0082 LR: 2.955e-02 Score 99.703 Data time: 0.1796, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7658 [2021-07-08 22:50:08,140][root][INFO] - ===> Epoch[8904](44520/5): Loss 0.0081, Sem 0.0081 LR: 2.954e-02 Score 99.711 Data time: 0.2708, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7467 [2021-07-08 22:50:21,942][root][INFO] - ===> Epoch[8905](44525/5): Loss 0.0087, Sem 0.0087 LR: 2.954e-02 Score 99.689 Data time: 0.1610, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7223 [2021-07-08 22:50:36,744][root][INFO] - ===> Epoch[8906](44530/5): Loss 0.0084, Sem 0.0084 LR: 2.953e-02 Score 99.699 Data time: 0.2415, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.9114 [2021-07-08 22:50:50,925][root][INFO] - ===> Epoch[8907](44535/5): Loss 0.0079, Sem 0.0079 LR: 2.952e-02 Score 99.714 Data time: 0.2499, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.7863 [2021-07-08 22:51:04,273][root][INFO] - ===> Epoch[8908](44540/5): Loss 0.0084, Sem 0.0084 LR: 2.951e-02 Score 99.698 Data time: 0.2073, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.6301 [2021-07-08 22:51:17,887][root][INFO] - ===> Epoch[8909](44545/5): Loss 0.0081, Sem 0.0081 LR: 2.950e-02 Score 99.710 Data time: 0.2050, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.6830 [2021-07-08 22:51:31,990][root][INFO] - ===> Epoch[8910](44550/5): Loss 0.0083, Sem 0.0083 LR: 2.949e-02 Score 99.702 Data time: 0.2080, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7751 [2021-07-08 22:51:45,793][root][INFO] - ===> Epoch[8911](44555/5): Loss 0.0083, Sem 0.0083 LR: 2.948e-02 Score 99.704 Data time: 0.2075, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7145 [2021-07-08 22:51:59,940][root][INFO] - ===> Epoch[8912](44560/5): Loss 0.0083, Sem 0.0083 LR: 2.948e-02 Score 99.699 Data time: 0.1899, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.7916 [2021-07-08 22:52:13,697][root][INFO] - ===> Epoch[8913](44565/5): Loss 0.0081, Sem 0.0081 LR: 2.947e-02 Score 99.707 Data time: 0.2125, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.6954 [2021-07-08 22:52:27,583][root][INFO] - ===> Epoch[8914](44570/5): Loss 0.0089, Sem 0.0089 LR: 2.946e-02 Score 99.688 Data time: 0.1772, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7312 [2021-07-08 22:52:42,422][root][INFO] - ===> Epoch[8915](44575/5): Loss 0.0081, Sem 0.0081 LR: 2.945e-02 Score 99.708 Data time: 0.2899, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.9134 [2021-07-08 22:52:56,618][root][INFO] - ===> Epoch[8916](44580/5): Loss 0.0083, Sem 0.0083 LR: 2.944e-02 Score 99.701 Data time: 0.2720, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.7917 [2021-07-08 22:53:10,402][root][INFO] - ===> Epoch[8917](44585/5): Loss 0.0092, Sem 0.0092 LR: 2.943e-02 Score 99.674 Data time: 0.2248, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7028 [2021-07-08 22:53:24,149][root][INFO] - ===> Epoch[8918](44590/5): Loss 0.0088, Sem 0.0088 LR: 2.942e-02 Score 99.685 Data time: 0.2448, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7027 [2021-07-08 22:53:38,135][root][INFO] - ===> Epoch[8919](44595/5): Loss 0.0087, Sem 0.0087 LR: 2.942e-02 Score 99.689 Data time: 0.2339, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7586 [2021-07-08 22:53:52,265][root][INFO] - ===> Epoch[8920](44600/5): Loss 0.0083, Sem 0.0083 LR: 2.941e-02 Score 99.701 Data time: 0.2677, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7769 [2021-07-08 22:53:52,268][root][INFO] - ===> Start testing [2021-07-08 22:54:28,052][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1328 Loss 0.367 (AVG: 0.615) Score 92.521 (AVG: 91.045) mIOU 70.885 mAP 82.529 mAcc 76.990 IOU: 65.393 0.000 83.179 76.134 95.211 92.051 23.304 80.680 98.686 83.414 79.594 85.962 57.890 mAP: 83.511 3.363 91.091 86.408 98.473 98.434 47.053 97.347 99.694 92.806 91.407 92.772 90.518 mAcc: 80.205 0.000 86.581 83.223 96.534 96.760 24.948 93.883 99.451 89.907 91.124 96.935 61.322 [2021-07-08 22:54:28,052][root][INFO] - Finished test. Elapsed time: 35.7836 [2021-07-08 22:54:29,165][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 22:54:42,158][root][INFO] - ===> Epoch[8921](44605/5): Loss 0.0086, Sem 0.0086 LR: 2.940e-02 Score 99.689 Data time: 0.1880, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.5623 [2021-07-08 22:54:55,766][root][INFO] - ===> Epoch[8922](44610/5): Loss 0.0082, Sem 0.0082 LR: 2.939e-02 Score 99.702 Data time: 0.2138, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.6718 [2021-07-08 22:55:09,423][root][INFO] - ===> Epoch[8923](44615/5): Loss 0.0088, Sem 0.0088 LR: 2.938e-02 Score 99.686 Data time: 0.2615, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.6745 [2021-07-08 22:55:23,391][root][INFO] - ===> Epoch[8924](44620/5): Loss 0.0083, Sem 0.0083 LR: 2.937e-02 Score 99.701 Data time: 0.2592, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7469 [2021-07-08 22:55:37,342][root][INFO] - ===> Epoch[8925](44625/5): Loss 0.0085, Sem 0.0085 LR: 2.936e-02 Score 99.695 Data time: 0.2104, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7410 [2021-07-08 22:55:51,932][root][INFO] - ===> Epoch[8926](44630/5): Loss 0.0083, Sem 0.0083 LR: 2.936e-02 Score 99.699 Data time: 0.2651, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.8627 [2021-07-08 22:56:05,848][root][INFO] - ===> Epoch[8927](44635/5): Loss 0.0087, Sem 0.0087 LR: 2.935e-02 Score 99.696 Data time: 0.2061, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7335 [2021-07-08 22:56:19,736][root][INFO] - ===> Epoch[8928](44640/5): Loss 0.0084, Sem 0.0084 LR: 2.934e-02 Score 99.701 Data time: 0.2183, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7394 [2021-07-08 22:56:34,304][root][INFO] - ===> Epoch[8929](44645/5): Loss 0.0081, Sem 0.0081 LR: 2.933e-02 Score 99.713 Data time: 0.2218, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.8679 [2021-07-08 22:56:47,636][root][INFO] - ===> Epoch[8930](44650/5): Loss 0.0086, Sem 0.0086 LR: 2.932e-02 Score 99.688 Data time: 0.1985, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.6208 [2021-07-08 22:57:01,434][root][INFO] - ===> Epoch[8931](44655/5): Loss 0.0082, Sem 0.0082 LR: 2.931e-02 Score 99.709 Data time: 0.2242, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7103 [2021-07-08 22:57:15,722][root][INFO] - ===> Epoch[8932](44660/5): Loss 0.0081, Sem 0.0081 LR: 2.930e-02 Score 99.709 Data time: 0.2125, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.8098 [2021-07-08 22:57:29,861][root][INFO] - ===> Epoch[8933](44665/5): Loss 0.0081, Sem 0.0081 LR: 2.930e-02 Score 99.710 Data time: 0.1972, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7745 [2021-07-08 22:57:44,163][root][INFO] - ===> Epoch[8934](44670/5): Loss 0.0081, Sem 0.0081 LR: 2.929e-02 Score 99.709 Data time: 0.2411, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.8119 [2021-07-08 22:57:58,497][root][INFO] - ===> Epoch[8935](44675/5): Loss 0.0084, Sem 0.0084 LR: 2.928e-02 Score 99.700 Data time: 0.1190, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8360 [2021-07-08 22:58:12,755][root][INFO] - ===> Epoch[8936](44680/5): Loss 0.0081, Sem 0.0081 LR: 2.927e-02 Score 99.710 Data time: 0.2311, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8030 [2021-07-08 22:58:26,762][root][INFO] - ===> Epoch[8937](44685/5): Loss 0.0082, Sem 0.0082 LR: 2.926e-02 Score 99.706 Data time: 0.1994, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.7548 [2021-07-08 22:58:41,374][root][INFO] - ===> Epoch[8938](44690/5): Loss 0.0080, Sem 0.0080 LR: 2.925e-02 Score 99.711 Data time: 0.2491, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8778 [2021-07-08 22:58:55,784][root][INFO] - ===> Epoch[8939](44695/5): Loss 0.0079, Sem 0.0079 LR: 2.924e-02 Score 99.717 Data time: 0.2514, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8309 [2021-07-08 22:59:09,624][root][INFO] - ===> Epoch[8940](44700/5): Loss 0.0086, Sem 0.0086 LR: 2.924e-02 Score 99.691 Data time: 0.2606, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.7146 [2021-07-08 22:59:23,515][root][INFO] - ===> Epoch[8941](44705/5): Loss 0.0082, Sem 0.0082 LR: 2.923e-02 Score 99.709 Data time: 0.2431, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.7162 [2021-07-08 22:59:37,419][root][INFO] - ===> Epoch[8942](44710/5): Loss 0.0083, Sem 0.0083 LR: 2.922e-02 Score 99.704 Data time: 0.2277, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.7425 [2021-07-08 22:59:51,130][root][INFO] - ===> Epoch[8943](44715/5): Loss 0.0078, Sem 0.0078 LR: 2.921e-02 Score 99.720 Data time: 0.1956, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7003 [2021-07-08 23:00:04,958][root][INFO] - ===> Epoch[8944](44720/5): Loss 0.0081, Sem 0.0081 LR: 2.920e-02 Score 99.708 Data time: 0.2615, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.7170 [2021-07-08 23:00:19,478][root][INFO] - ===> Epoch[8945](44725/5): Loss 0.0082, Sem 0.0082 LR: 2.919e-02 Score 99.705 Data time: 0.2284, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8489 [2021-07-08 23:00:33,244][root][INFO] - ===> Epoch[8946](44730/5): Loss 0.0081, Sem 0.0081 LR: 2.918e-02 Score 99.707 Data time: 0.2675, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.7085 [2021-07-08 23:00:46,982][root][INFO] - ===> Epoch[8947](44735/5): Loss 0.0088, Sem 0.0088 LR: 2.917e-02 Score 99.684 Data time: 0.2193, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.7054 [2021-07-08 23:01:00,343][root][INFO] - ===> Epoch[8948](44740/5): Loss 0.0079, Sem 0.0079 LR: 2.917e-02 Score 99.713 Data time: 0.2821, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.6195 [2021-07-08 23:01:14,598][root][INFO] - ===> Epoch[8949](44745/5): Loss 0.0083, Sem 0.0083 LR: 2.916e-02 Score 99.703 Data time: 0.2015, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.8027 [2021-07-08 23:01:28,484][root][INFO] - ===> Epoch[8950](44750/5): Loss 0.0089, Sem 0.0089 LR: 2.915e-02 Score 99.683 Data time: 0.1959, Forward time: 1.2541, Backward time: 1.2453, DDP time: 0.0042, Total iter time: 2.7347 [2021-07-08 23:01:44,050][root][INFO] - ===> Epoch[8951](44755/5): Loss 0.0083, Sem 0.0083 LR: 2.914e-02 Score 99.702 Data time: 0.2106, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 3.0740 [2021-07-08 23:01:57,954][root][INFO] - ===> Epoch[8952](44760/5): Loss 0.0087, Sem 0.0087 LR: 2.913e-02 Score 99.686 Data time: 0.2567, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.7303 [2021-07-08 23:02:11,366][root][INFO] - ===> Epoch[8953](44765/5): Loss 0.0084, Sem 0.0084 LR: 2.912e-02 Score 99.694 Data time: 0.1838, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.6368 [2021-07-08 23:02:25,715][root][INFO] - ===> Epoch[8954](44770/5): Loss 0.0087, Sem 0.0087 LR: 2.911e-02 Score 99.689 Data time: 0.2167, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8230 [2021-07-08 23:02:39,962][root][INFO] - ===> Epoch[8955](44775/5): Loss 0.0085, Sem 0.0085 LR: 2.911e-02 Score 99.696 Data time: 0.2618, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8069 [2021-07-08 23:02:54,458][root][INFO] - ===> Epoch[8956](44780/5): Loss 0.0082, Sem 0.0082 LR: 2.910e-02 Score 99.703 Data time: 0.2049, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8554 [2021-07-08 23:03:08,764][root][INFO] - ===> Epoch[8957](44785/5): Loss 0.0086, Sem 0.0086 LR: 2.909e-02 Score 99.692 Data time: 0.1924, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8103 [2021-07-08 23:03:23,055][root][INFO] - ===> Epoch[8958](44790/5): Loss 0.0083, Sem 0.0083 LR: 2.908e-02 Score 99.700 Data time: 0.2073, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8037 [2021-07-08 23:03:37,033][root][INFO] - ===> Epoch[8959](44795/5): Loss 0.0085, Sem 0.0085 LR: 2.907e-02 Score 99.695 Data time: 0.2536, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.7461 [2021-07-08 23:03:53,031][root][INFO] - ===> Epoch[8960](44800/5): Loss 0.0082, Sem 0.0082 LR: 2.906e-02 Score 99.708 Data time: 0.3033, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 3.1482 [2021-07-08 23:03:53,032][root][INFO] - ===> Start testing [2021-07-08 23:04:32,344][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1951 Loss 0.353 (AVG: 0.582) Score 92.788 (AVG: 91.193) mIOU 71.189 mAP 83.173 mAcc 77.423 IOU: 65.296 0.000 81.252 75.971 95.220 91.404 27.466 84.933 98.632 79.579 79.820 86.312 59.571 mAP: 83.645 3.497 89.258 88.109 98.416 98.233 51.108 98.353 99.749 94.673 91.306 93.549 91.359 mAcc: 80.876 0.000 83.496 83.831 96.568 97.167 29.408 95.793 99.368 89.542 90.328 96.409 63.712 [2021-07-08 23:04:32,345][root][INFO] - Finished test. Elapsed time: 39.3117 [2021-07-08 23:04:33,469][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 23:04:46,170][root][INFO] - ===> Epoch[8961](44805/5): Loss 0.0090, Sem 0.0090 LR: 2.905e-02 Score 99.684 Data time: 0.2478, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.4970 [2021-07-08 23:05:00,611][root][INFO] - ===> Epoch[8962](44810/5): Loss 0.0089, Sem 0.0089 LR: 2.905e-02 Score 99.685 Data time: 0.2856, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8343 [2021-07-08 23:05:15,301][root][INFO] - ===> Epoch[8963](44815/5): Loss 0.0095, Sem 0.0095 LR: 2.904e-02 Score 99.668 Data time: 0.2805, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8760 [2021-07-08 23:05:30,472][root][INFO] - ===> Epoch[8964](44820/5): Loss 0.0089, Sem 0.0089 LR: 2.903e-02 Score 99.686 Data time: 0.2723, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.9863 [2021-07-08 23:05:45,258][root][INFO] - ===> Epoch[8965](44825/5): Loss 0.0088, Sem 0.0088 LR: 2.902e-02 Score 99.682 Data time: 0.2065, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.9148 [2021-07-08 23:05:59,810][root][INFO] - ===> Epoch[8966](44830/5): Loss 0.0084, Sem 0.0084 LR: 2.901e-02 Score 99.697 Data time: 0.2625, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8696 [2021-07-08 23:06:13,037][root][INFO] - ===> Epoch[8967](44835/5): Loss 0.0083, Sem 0.0083 LR: 2.900e-02 Score 99.702 Data time: 0.2700, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.6041 [2021-07-08 23:06:26,080][root][INFO] - ===> Epoch[8968](44840/5): Loss 0.0084, Sem 0.0084 LR: 2.899e-02 Score 99.696 Data time: 0.2021, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.5644 [2021-07-08 23:06:39,769][root][INFO] - ===> Epoch[8969](44845/5): Loss 0.0085, Sem 0.0085 LR: 2.899e-02 Score 99.692 Data time: 0.1765, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.6923 [2021-07-08 23:06:53,736][root][INFO] - ===> Epoch[8970](44850/5): Loss 0.0085, Sem 0.0085 LR: 2.898e-02 Score 99.694 Data time: 0.2405, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.7423 [2021-07-08 23:07:08,145][root][INFO] - ===> Epoch[8971](44855/5): Loss 0.0086, Sem 0.0086 LR: 2.897e-02 Score 99.694 Data time: 0.2315, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8409 [2021-07-08 23:07:23,021][root][INFO] - ===> Epoch[8972](44860/5): Loss 0.0083, Sem 0.0083 LR: 2.896e-02 Score 99.702 Data time: 0.2085, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.9338 [2021-07-08 23:07:37,475][root][INFO] - ===> Epoch[8973](44865/5): Loss 0.0079, Sem 0.0079 LR: 2.895e-02 Score 99.718 Data time: 0.1845, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8383 [2021-07-08 23:07:51,699][root][INFO] - ===> Epoch[8974](44870/5): Loss 0.0086, Sem 0.0086 LR: 2.894e-02 Score 99.689 Data time: 0.2532, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.7992 [2021-07-08 23:08:05,959][root][INFO] - ===> Epoch[8975](44875/5): Loss 0.0082, Sem 0.0082 LR: 2.893e-02 Score 99.704 Data time: 0.2049, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8018 [2021-07-08 23:08:19,836][root][INFO] - ===> Epoch[8976](44880/5): Loss 0.0087, Sem 0.0087 LR: 2.893e-02 Score 99.691 Data time: 0.1556, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.7418 [2021-07-08 23:08:34,363][root][INFO] - ===> Epoch[8977](44885/5): Loss 0.0083, Sem 0.0083 LR: 2.892e-02 Score 99.699 Data time: 0.2001, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8594 [2021-07-08 23:08:48,390][root][INFO] - ===> Epoch[8978](44890/5): Loss 0.0084, Sem 0.0084 LR: 2.891e-02 Score 99.698 Data time: 0.2107, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.7597 [2021-07-08 23:09:01,602][root][INFO] - ===> Epoch[8979](44895/5): Loss 0.0084, Sem 0.0084 LR: 2.890e-02 Score 99.699 Data time: 0.1982, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.6063 [2021-07-08 23:09:16,038][root][INFO] - ===> Epoch[8980](44900/5): Loss 0.0084, Sem 0.0084 LR: 2.889e-02 Score 99.700 Data time: 0.2379, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8410 [2021-07-08 23:09:30,772][root][INFO] - ===> Epoch[8981](44905/5): Loss 0.0087, Sem 0.0087 LR: 2.888e-02 Score 99.694 Data time: 0.2067, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.9022 [2021-07-08 23:09:45,105][root][INFO] - ===> Epoch[8982](44910/5): Loss 0.0091, Sem 0.0091 LR: 2.887e-02 Score 99.671 Data time: 0.2291, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8237 [2021-07-08 23:09:59,428][root][INFO] - ===> Epoch[8983](44915/5): Loss 0.0087, Sem 0.0087 LR: 2.887e-02 Score 99.688 Data time: 0.2017, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.8276 [2021-07-08 23:10:14,198][root][INFO] - ===> Epoch[8984](44920/5): Loss 0.0088, Sem 0.0088 LR: 2.886e-02 Score 99.685 Data time: 0.2673, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.9005 [2021-07-08 23:10:27,884][root][INFO] - ===> Epoch[8985](44925/5): Loss 0.0086, Sem 0.0086 LR: 2.885e-02 Score 99.690 Data time: 0.1923, Forward time: 1.2541, Backward time: 1.2454, DDP time: 0.0042, Total iter time: 2.6928 [2021-07-08 23:10:42,067][root][INFO] - ===> Epoch[8986](44930/5): Loss 0.0086, Sem 0.0086 LR: 2.884e-02 Score 99.693 Data time: 0.1476, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7911 [2021-07-08 23:10:56,272][root][INFO] - ===> Epoch[8987](44935/5): Loss 0.0087, Sem 0.0087 LR: 2.883e-02 Score 99.687 Data time: 0.2217, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7868 [2021-07-08 23:11:10,993][root][INFO] - ===> Epoch[8988](44940/5): Loss 0.0084, Sem 0.0084 LR: 2.882e-02 Score 99.700 Data time: 0.2124, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.8997 [2021-07-08 23:11:24,593][root][INFO] - ===> Epoch[8989](44945/5): Loss 0.0086, Sem 0.0086 LR: 2.881e-02 Score 99.695 Data time: 0.1898, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.6765 [2021-07-08 23:11:38,560][root][INFO] - ===> Epoch[8990](44950/5): Loss 0.0080, Sem 0.0080 LR: 2.880e-02 Score 99.709 Data time: 0.2110, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7443 [2021-07-08 23:11:53,163][root][INFO] - ===> Epoch[8991](44955/5): Loss 0.0085, Sem 0.0085 LR: 2.880e-02 Score 99.694 Data time: 0.2031, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.8670 [2021-07-08 23:12:06,657][root][INFO] - ===> Epoch[8992](44960/5): Loss 0.0085, Sem 0.0085 LR: 2.879e-02 Score 99.697 Data time: 0.2145, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.6586 [2021-07-08 23:12:20,696][root][INFO] - ===> Epoch[8993](44965/5): Loss 0.0083, Sem 0.0083 LR: 2.878e-02 Score 99.705 Data time: 0.2581, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7641 [2021-07-08 23:12:34,639][root][INFO] - ===> Epoch[8994](44970/5): Loss 0.0083, Sem 0.0083 LR: 2.877e-02 Score 99.699 Data time: 0.2255, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7377 [2021-07-08 23:12:48,867][root][INFO] - ===> Epoch[8995](44975/5): Loss 0.0082, Sem 0.0082 LR: 2.876e-02 Score 99.703 Data time: 0.1946, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7962 [2021-07-08 23:13:02,990][root][INFO] - ===> Epoch[8996](44980/5): Loss 0.0084, Sem 0.0084 LR: 2.875e-02 Score 99.699 Data time: 0.2933, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7653 [2021-07-08 23:13:16,845][root][INFO] - ===> Epoch[8997](44985/5): Loss 0.0080, Sem 0.0080 LR: 2.874e-02 Score 99.711 Data time: 0.2339, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7270 [2021-07-08 23:13:31,172][root][INFO] - ===> Epoch[8998](44990/5): Loss 0.0085, Sem 0.0085 LR: 2.874e-02 Score 99.692 Data time: 0.1889, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.8189 [2021-07-08 23:13:45,202][root][INFO] - ===> Epoch[8999](44995/5): Loss 0.0085, Sem 0.0085 LR: 2.873e-02 Score 99.696 Data time: 0.2214, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7615 [2021-07-08 23:13:59,026][root][INFO] - ===> Epoch[9000](45000/5): Loss 0.0082, Sem 0.0082 LR: 2.872e-02 Score 99.706 Data time: 0.1525, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7283 [2021-07-08 23:13:59,028][root][INFO] - ===> Start testing [2021-07-08 23:14:40,370][root][INFO] - 68/68: Data time: 0.0807, Iter time: 0.1290 Loss 0.359 (AVG: 0.594) Score 91.821 (AVG: 91.233) mIOU 71.697 mAP 83.199 mAcc 77.807 IOU: 65.313 0.000 80.389 75.927 95.189 91.565 29.987 83.737 98.677 84.150 81.054 86.413 59.661 mAP: 83.304 3.987 91.628 86.694 98.643 98.264 51.388 97.666 99.720 95.031 91.546 92.709 91.001 mAcc: 81.673 0.000 85.223 83.089 96.705 96.801 32.442 93.855 99.391 91.456 90.327 96.385 64.145 [2021-07-08 23:14:40,370][root][INFO] - Finished test. Elapsed time: 41.3420 [2021-07-08 23:14:41,425][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 23:14:54,371][root][INFO] - ===> Epoch[9001](45005/5): Loss 0.0081, Sem 0.0081 LR: 2.871e-02 Score 99.709 Data time: 0.1448, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.5613 [2021-07-08 23:15:08,166][root][INFO] - ===> Epoch[9002](45010/5): Loss 0.0087, Sem 0.0087 LR: 2.870e-02 Score 99.687 Data time: 0.2966, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7114 [2021-07-08 23:15:22,271][root][INFO] - ===> Epoch[9003](45015/5): Loss 0.0081, Sem 0.0081 LR: 2.869e-02 Score 99.707 Data time: 0.3120, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7702 [2021-07-08 23:15:37,302][root][INFO] - ===> Epoch[9004](45020/5): Loss 0.0081, Sem 0.0081 LR: 2.868e-02 Score 99.708 Data time: 0.2616, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.9625 [2021-07-08 23:15:51,087][root][INFO] - ===> Epoch[9005](45025/5): Loss 0.0081, Sem 0.0081 LR: 2.868e-02 Score 99.706 Data time: 0.2629, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7054 [2021-07-08 23:16:07,087][root][INFO] - ===> Epoch[9006](45030/5): Loss 0.0084, Sem 0.0084 LR: 2.867e-02 Score 99.701 Data time: 0.3217, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 3.1469 [2021-07-08 23:16:20,951][root][INFO] - ===> Epoch[9007](45035/5): Loss 0.0086, Sem 0.0086 LR: 2.866e-02 Score 99.692 Data time: 0.2377, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7313 [2021-07-08 23:16:35,076][root][INFO] - ===> Epoch[9008](45040/5): Loss 0.0083, Sem 0.0083 LR: 2.865e-02 Score 99.700 Data time: 0.2446, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7795 [2021-07-08 23:16:49,259][root][INFO] - ===> Epoch[9009](45045/5): Loss 0.0080, Sem 0.0080 LR: 2.864e-02 Score 99.710 Data time: 0.2007, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7856 [2021-07-08 23:17:03,247][root][INFO] - ===> Epoch[9010](45050/5): Loss 0.0085, Sem 0.0085 LR: 2.863e-02 Score 99.698 Data time: 0.2951, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7511 [2021-07-08 23:17:17,496][root][INFO] - ===> Epoch[9011](45055/5): Loss 0.0083, Sem 0.0083 LR: 2.862e-02 Score 99.699 Data time: 0.2613, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8051 [2021-07-08 23:17:31,195][root][INFO] - ===> Epoch[9012](45060/5): Loss 0.0083, Sem 0.0083 LR: 2.862e-02 Score 99.703 Data time: 0.2214, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6962 [2021-07-08 23:17:44,557][root][INFO] - ===> Epoch[9013](45065/5): Loss 0.0089, Sem 0.0089 LR: 2.861e-02 Score 99.677 Data time: 0.2425, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.6311 [2021-07-08 23:17:57,883][root][INFO] - ===> Epoch[9014](45070/5): Loss 0.0083, Sem 0.0083 LR: 2.860e-02 Score 99.702 Data time: 0.1823, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.6215 [2021-07-08 23:18:12,082][root][INFO] - ===> Epoch[9015](45075/5): Loss 0.0082, Sem 0.0082 LR: 2.859e-02 Score 99.706 Data time: 0.1991, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7932 [2021-07-08 23:18:26,065][root][INFO] - ===> Epoch[9016](45080/5): Loss 0.0095, Sem 0.0095 LR: 2.858e-02 Score 99.685 Data time: 0.2473, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7451 [2021-07-08 23:18:40,242][root][INFO] - ===> Epoch[9017](45085/5): Loss 0.0096, Sem 0.0096 LR: 2.857e-02 Score 99.667 Data time: 0.2254, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7832 [2021-07-08 23:18:53,788][root][INFO] - ===> Epoch[9018](45090/5): Loss 0.0086, Sem 0.0086 LR: 2.856e-02 Score 99.689 Data time: 0.2208, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.6638 [2021-07-08 23:19:07,295][root][INFO] - ===> Epoch[9019](45095/5): Loss 0.0132, Sem 0.0132 LR: 2.856e-02 Score 99.620 Data time: 0.2175, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.6517 [2021-07-08 23:19:21,230][root][INFO] - ===> Epoch[9020](45100/5): Loss 0.0148, Sem 0.0148 LR: 2.855e-02 Score 99.558 Data time: 0.1972, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7348 [2021-07-08 23:19:35,241][root][INFO] - ===> Epoch[9021](45105/5): Loss 0.0124, Sem 0.0124 LR: 2.854e-02 Score 99.575 Data time: 0.2247, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7524 [2021-07-08 23:19:48,343][root][INFO] - ===> Epoch[9022](45110/5): Loss 0.0113, Sem 0.0113 LR: 2.853e-02 Score 99.601 Data time: 0.1990, Forward time: 1.2541, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.5801 [2021-07-08 23:20:02,868][root][INFO] - ===> Epoch[9023](45115/5): Loss 0.0115, Sem 0.0115 LR: 2.852e-02 Score 99.610 Data time: 0.2008, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.8618 [2021-07-08 23:20:16,997][root][INFO] - ===> Epoch[9024](45120/5): Loss 0.0107, Sem 0.0107 LR: 2.851e-02 Score 99.622 Data time: 0.2228, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7807 [2021-07-08 23:20:30,247][root][INFO] - ===> Epoch[9025](45125/5): Loss 0.0102, Sem 0.0102 LR: 2.850e-02 Score 99.640 Data time: 0.2875, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.5986 [2021-07-08 23:20:45,393][root][INFO] - ===> Epoch[9026](45130/5): Loss 0.0096, Sem 0.0096 LR: 2.849e-02 Score 99.653 Data time: 0.2233, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.9823 [2021-07-08 23:20:58,875][root][INFO] - ===> Epoch[9027](45135/5): Loss 0.0098, Sem 0.0098 LR: 2.849e-02 Score 99.655 Data time: 0.2476, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.6507 [2021-07-08 23:21:13,054][root][INFO] - ===> Epoch[9028](45140/5): Loss 0.0092, Sem 0.0092 LR: 2.848e-02 Score 99.668 Data time: 0.2519, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7965 [2021-07-08 23:21:27,651][root][INFO] - ===> Epoch[9029](45145/5): Loss 0.0091, Sem 0.0091 LR: 2.847e-02 Score 99.675 Data time: 0.1628, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.8790 [2021-07-08 23:21:41,216][root][INFO] - ===> Epoch[9030](45150/5): Loss 0.0096, Sem 0.0096 LR: 2.846e-02 Score 99.658 Data time: 0.2177, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.6671 [2021-07-08 23:21:54,142][root][INFO] - ===> Epoch[9031](45155/5): Loss 0.0090, Sem 0.0090 LR: 2.845e-02 Score 99.672 Data time: 0.1558, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.5440 [2021-07-08 23:22:08,516][root][INFO] - ===> Epoch[9032](45160/5): Loss 0.0083, Sem 0.0083 LR: 2.844e-02 Score 99.706 Data time: 0.2360, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.8175 [2021-07-08 23:22:22,266][root][INFO] - ===> Epoch[9033](45165/5): Loss 0.0088, Sem 0.0088 LR: 2.843e-02 Score 99.685 Data time: 0.1860, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7119 [2021-07-08 23:22:36,189][root][INFO] - ===> Epoch[9034](45170/5): Loss 0.0084, Sem 0.0084 LR: 2.843e-02 Score 99.698 Data time: 0.2250, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7383 [2021-07-08 23:22:49,763][root][INFO] - ===> Epoch[9035](45175/5): Loss 0.0087, Sem 0.0087 LR: 2.842e-02 Score 99.687 Data time: 0.2414, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.6679 [2021-07-08 23:23:04,471][root][INFO] - ===> Epoch[9036](45180/5): Loss 0.0091, Sem 0.0091 LR: 2.841e-02 Score 99.672 Data time: 0.2031, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.9008 [2021-07-08 23:23:18,362][root][INFO] - ===> Epoch[9037](45185/5): Loss 0.0087, Sem 0.0087 LR: 2.840e-02 Score 99.687 Data time: 0.2302, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7283 [2021-07-08 23:23:33,112][root][INFO] - ===> Epoch[9038](45190/5): Loss 0.0087, Sem 0.0087 LR: 2.839e-02 Score 99.683 Data time: 0.2042, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.9010 [2021-07-08 23:23:47,247][root][INFO] - ===> Epoch[9039](45195/5): Loss 0.0081, Sem 0.0081 LR: 2.838e-02 Score 99.709 Data time: 0.1793, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7800 [2021-07-08 23:24:01,057][root][INFO] - ===> Epoch[9040](45200/5): Loss 0.0090, Sem 0.0090 LR: 2.837e-02 Score 99.678 Data time: 0.1836, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.7178 [2021-07-08 23:24:01,059][root][INFO] - ===> Start testing [2021-07-08 23:24:41,368][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1225 Loss 0.438 (AVG: 0.630) Score 90.683 (AVG: 90.698) mIOU 70.778 mAP 82.865 mAcc 77.311 IOU: 63.257 0.000 81.871 73.265 95.098 90.207 29.521 85.970 98.731 80.579 78.383 85.572 57.659 mAP: 82.687 2.834 91.085 84.993 98.694 98.184 52.828 97.889 99.720 94.746 90.936 92.728 89.921 mAcc: 80.382 0.000 86.187 80.315 96.534 97.373 31.638 95.058 99.573 90.588 90.781 96.333 60.283 [2021-07-08 23:24:41,368][root][INFO] - Finished test. Elapsed time: 40.3081 [2021-07-08 23:24:42,612][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 23:24:55,970][root][INFO] - ===> Epoch[9041](45205/5): Loss 0.0088, Sem 0.0088 LR: 2.837e-02 Score 99.685 Data time: 0.1586, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6345 [2021-07-08 23:25:10,151][root][INFO] - ===> Epoch[9042](45210/5): Loss 0.0087, Sem 0.0087 LR: 2.836e-02 Score 99.685 Data time: 0.2412, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7830 [2021-07-08 23:25:24,144][root][INFO] - ===> Epoch[9043](45215/5): Loss 0.0082, Sem 0.0082 LR: 2.835e-02 Score 99.704 Data time: 0.2220, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7411 [2021-07-08 23:25:38,139][root][INFO] - ===> Epoch[9044](45220/5): Loss 0.0083, Sem 0.0083 LR: 2.834e-02 Score 99.700 Data time: 0.1996, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7518 [2021-07-08 23:25:52,051][root][INFO] - ===> Epoch[9045](45225/5): Loss 0.0081, Sem 0.0081 LR: 2.833e-02 Score 99.714 Data time: 0.1950, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7353 [2021-07-08 23:26:06,762][root][INFO] - ===> Epoch[9046](45230/5): Loss 0.0090, Sem 0.0090 LR: 2.832e-02 Score 99.674 Data time: 0.1700, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.9065 [2021-07-08 23:26:20,370][root][INFO] - ===> Epoch[9047](45235/5): Loss 0.0084, Sem 0.0084 LR: 2.831e-02 Score 99.699 Data time: 0.1915, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6729 [2021-07-08 23:26:34,666][root][INFO] - ===> Epoch[9048](45240/5): Loss 0.0079, Sem 0.0079 LR: 2.830e-02 Score 99.714 Data time: 0.3033, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8116 [2021-07-08 23:26:48,666][root][INFO] - ===> Epoch[9049](45245/5): Loss 0.0086, Sem 0.0086 LR: 2.830e-02 Score 99.695 Data time: 0.2168, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7446 [2021-07-08 23:27:02,922][root][INFO] - ===> Epoch[9050](45250/5): Loss 0.0084, Sem 0.0084 LR: 2.829e-02 Score 99.698 Data time: 0.2728, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7926 [2021-07-08 23:27:17,423][root][INFO] - ===> Epoch[9051](45255/5): Loss 0.0081, Sem 0.0081 LR: 2.828e-02 Score 99.711 Data time: 0.2601, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8558 [2021-07-08 23:27:32,003][root][INFO] - ===> Epoch[9052](45260/5): Loss 0.0082, Sem 0.0082 LR: 2.827e-02 Score 99.708 Data time: 0.2713, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8663 [2021-07-08 23:27:46,748][root][INFO] - ===> Epoch[9053](45265/5): Loss 0.0084, Sem 0.0084 LR: 2.826e-02 Score 99.696 Data time: 0.2540, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.9104 [2021-07-08 23:28:01,177][root][INFO] - ===> Epoch[9054](45270/5): Loss 0.0081, Sem 0.0081 LR: 2.825e-02 Score 99.708 Data time: 0.2241, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8425 [2021-07-08 23:28:15,127][root][INFO] - ===> Epoch[9055](45275/5): Loss 0.0082, Sem 0.0082 LR: 2.824e-02 Score 99.706 Data time: 0.2504, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7480 [2021-07-08 23:28:29,702][root][INFO] - ===> Epoch[9056](45280/5): Loss 0.0081, Sem 0.0081 LR: 2.824e-02 Score 99.707 Data time: 0.2115, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8664 [2021-07-08 23:28:43,501][root][INFO] - ===> Epoch[9057](45285/5): Loss 0.0082, Sem 0.0082 LR: 2.823e-02 Score 99.707 Data time: 0.2462, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7121 [2021-07-08 23:28:57,362][root][INFO] - ===> Epoch[9058](45290/5): Loss 0.0083, Sem 0.0083 LR: 2.822e-02 Score 99.700 Data time: 0.2503, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7242 [2021-07-08 23:29:10,884][root][INFO] - ===> Epoch[9059](45295/5): Loss 0.0080, Sem 0.0080 LR: 2.821e-02 Score 99.712 Data time: 0.2410, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6585 [2021-07-08 23:29:24,834][root][INFO] - ===> Epoch[9060](45300/5): Loss 0.0080, Sem 0.0080 LR: 2.820e-02 Score 99.711 Data time: 0.2449, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7408 [2021-07-08 23:29:39,096][root][INFO] - ===> Epoch[9061](45305/5): Loss 0.0080, Sem 0.0080 LR: 2.819e-02 Score 99.710 Data time: 0.2236, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8025 [2021-07-08 23:29:52,739][root][INFO] - ===> Epoch[9062](45310/5): Loss 0.0080, Sem 0.0080 LR: 2.818e-02 Score 99.711 Data time: 0.2443, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6900 [2021-07-08 23:30:07,383][root][INFO] - ===> Epoch[9063](45315/5): Loss 0.0083, Sem 0.0083 LR: 2.818e-02 Score 99.703 Data time: 0.2383, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8891 [2021-07-08 23:30:20,937][root][INFO] - ===> Epoch[9064](45320/5): Loss 0.0081, Sem 0.0081 LR: 2.817e-02 Score 99.709 Data time: 0.2024, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6688 [2021-07-08 23:30:35,049][root][INFO] - ===> Epoch[9065](45325/5): Loss 0.0083, Sem 0.0083 LR: 2.816e-02 Score 99.701 Data time: 0.2483, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7756 [2021-07-08 23:30:48,988][root][INFO] - ===> Epoch[9066](45330/5): Loss 0.0084, Sem 0.0084 LR: 2.815e-02 Score 99.698 Data time: 0.2677, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7438 [2021-07-08 23:31:02,799][root][INFO] - ===> Epoch[9067](45335/5): Loss 0.0083, Sem 0.0083 LR: 2.814e-02 Score 99.702 Data time: 0.2460, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7222 [2021-07-08 23:31:17,485][root][INFO] - ===> Epoch[9068](45340/5): Loss 0.0078, Sem 0.0078 LR: 2.813e-02 Score 99.720 Data time: 0.3022, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8937 [2021-07-08 23:31:31,391][root][INFO] - ===> Epoch[9069](45345/5): Loss 0.0081, Sem 0.0081 LR: 2.812e-02 Score 99.710 Data time: 0.2217, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7399 [2021-07-08 23:31:44,924][root][INFO] - ===> Epoch[9070](45350/5): Loss 0.0081, Sem 0.0081 LR: 2.811e-02 Score 99.709 Data time: 0.2158, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6675 [2021-07-08 23:31:59,107][root][INFO] - ===> Epoch[9071](45355/5): Loss 0.0080, Sem 0.0080 LR: 2.811e-02 Score 99.710 Data time: 0.2119, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7753 [2021-07-08 23:32:13,055][root][INFO] - ===> Epoch[9072](45360/5): Loss 0.0084, Sem 0.0084 LR: 2.810e-02 Score 99.701 Data time: 0.2014, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7396 [2021-07-08 23:32:26,709][root][INFO] - ===> Epoch[9073](45365/5): Loss 0.0081, Sem 0.0081 LR: 2.809e-02 Score 99.711 Data time: 0.2398, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6805 [2021-07-08 23:32:41,141][root][INFO] - ===> Epoch[9074](45370/5): Loss 0.0084, Sem 0.0084 LR: 2.808e-02 Score 99.696 Data time: 0.2449, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8210 [2021-07-08 23:32:55,324][root][INFO] - ===> Epoch[9075](45375/5): Loss 0.0083, Sem 0.0083 LR: 2.807e-02 Score 99.701 Data time: 0.2521, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7933 [2021-07-08 23:33:08,612][root][INFO] - ===> Epoch[9076](45380/5): Loss 0.0086, Sem 0.0086 LR: 2.806e-02 Score 99.695 Data time: 0.2179, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6153 [2021-07-08 23:33:23,198][root][INFO] - ===> Epoch[9077](45385/5): Loss 0.0082, Sem 0.0082 LR: 2.805e-02 Score 99.707 Data time: 0.2031, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8628 [2021-07-08 23:33:36,368][root][INFO] - ===> Epoch[9078](45390/5): Loss 0.0084, Sem 0.0084 LR: 2.805e-02 Score 99.699 Data time: 0.2855, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.5835 [2021-07-08 23:33:50,787][root][INFO] - ===> Epoch[9079](45395/5): Loss 0.0080, Sem 0.0080 LR: 2.804e-02 Score 99.713 Data time: 0.1905, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8417 [2021-07-08 23:34:04,149][root][INFO] - ===> Epoch[9080](45400/5): Loss 0.0086, Sem 0.0086 LR: 2.803e-02 Score 99.694 Data time: 0.1644, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6342 [2021-07-08 23:34:04,150][root][INFO] - ===> Start testing [2021-07-08 23:34:45,296][root][INFO] - 68/68: Data time: 0.0517, Iter time: 0.1245 Loss 0.388 (AVG: 0.623) Score 92.152 (AVG: 90.826) mIOU 70.773 mAP 82.622 mAcc 76.767 IOU: 64.236 0.000 80.679 74.741 94.917 91.569 24.731 87.339 98.664 80.890 79.599 85.316 57.369 mAP: 83.371 2.717 88.602 86.234 98.501 98.404 49.980 97.730 99.757 94.688 91.061 92.599 90.443 mAcc: 79.729 0.000 83.185 81.960 96.677 97.114 26.334 94.252 99.389 90.578 91.053 96.811 60.889 [2021-07-08 23:34:45,296][root][INFO] - Finished test. Elapsed time: 41.1446 [2021-07-08 23:34:46,379][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 23:34:59,118][root][INFO] - ===> Epoch[9081](45405/5): Loss 0.0086, Sem 0.0086 LR: 2.802e-02 Score 99.692 Data time: 0.1520, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.5006 [2021-07-08 23:35:13,077][root][INFO] - ===> Epoch[9082](45410/5): Loss 0.0080, Sem 0.0080 LR: 2.801e-02 Score 99.712 Data time: 0.1949, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7357 [2021-07-08 23:35:26,861][root][INFO] - ===> Epoch[9083](45415/5): Loss 0.0086, Sem 0.0086 LR: 2.800e-02 Score 99.686 Data time: 0.1779, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7172 [2021-07-08 23:35:40,161][root][INFO] - ===> Epoch[9084](45420/5): Loss 0.0083, Sem 0.0083 LR: 2.799e-02 Score 99.699 Data time: 0.2059, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6002 [2021-07-08 23:35:53,959][root][INFO] - ===> Epoch[9085](45425/5): Loss 0.0080, Sem 0.0080 LR: 2.799e-02 Score 99.712 Data time: 0.2296, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7049 [2021-07-08 23:36:07,876][root][INFO] - ===> Epoch[9086](45430/5): Loss 0.0084, Sem 0.0084 LR: 2.798e-02 Score 99.697 Data time: 0.2024, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7400 [2021-07-08 23:36:22,172][root][INFO] - ===> Epoch[9087](45435/5): Loss 0.0086, Sem 0.0086 LR: 2.797e-02 Score 99.690 Data time: 0.2126, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8217 [2021-07-08 23:36:36,549][root][INFO] - ===> Epoch[9088](45440/5): Loss 0.0083, Sem 0.0083 LR: 2.796e-02 Score 99.701 Data time: 0.2250, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8193 [2021-07-08 23:36:51,350][root][INFO] - ===> Epoch[9089](45445/5): Loss 0.0082, Sem 0.0082 LR: 2.795e-02 Score 99.708 Data time: 0.1214, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.9242 [2021-07-08 23:37:04,849][root][INFO] - ===> Epoch[9090](45450/5): Loss 0.0081, Sem 0.0081 LR: 2.794e-02 Score 99.710 Data time: 0.2367, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6491 [2021-07-08 23:37:19,139][root][INFO] - ===> Epoch[9091](45455/5): Loss 0.0079, Sem 0.0079 LR: 2.793e-02 Score 99.717 Data time: 0.2562, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8025 [2021-07-08 23:37:33,182][root][INFO] - ===> Epoch[9092](45460/5): Loss 0.0085, Sem 0.0085 LR: 2.792e-02 Score 99.697 Data time: 0.2686, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7585 [2021-07-08 23:37:47,520][root][INFO] - ===> Epoch[9093](45465/5): Loss 0.0083, Sem 0.0083 LR: 2.792e-02 Score 99.702 Data time: 0.2235, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8226 [2021-07-08 23:38:01,762][root][INFO] - ===> Epoch[9094](45470/5): Loss 0.0080, Sem 0.0080 LR: 2.791e-02 Score 99.716 Data time: 0.2733, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8013 [2021-07-08 23:38:16,301][root][INFO] - ===> Epoch[9095](45475/5): Loss 0.0083, Sem 0.0083 LR: 2.790e-02 Score 99.704 Data time: 0.2677, Forward time: 1.2540, Backward time: 1.2455, DDP time: 0.0042, Total iter time: 2.8620 [2021-07-08 23:38:31,809][root][INFO] - ===> Epoch[9096](45480/5): Loss 0.0080, Sem 0.0080 LR: 2.789e-02 Score 99.711 Data time: 0.1812, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 3.0558 [2021-07-08 23:38:46,866][root][INFO] - ===> Epoch[9097](45485/5): Loss 0.0083, Sem 0.0083 LR: 2.788e-02 Score 99.699 Data time: 0.2776, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.9510 [2021-07-08 23:39:00,905][root][INFO] - ===> Epoch[9098](45490/5): Loss 0.0080, Sem 0.0080 LR: 2.787e-02 Score 99.713 Data time: 0.2505, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7640 [2021-07-08 23:39:15,158][root][INFO] - ===> Epoch[9099](45495/5): Loss 0.0084, Sem 0.0084 LR: 2.786e-02 Score 99.698 Data time: 0.2470, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8112 [2021-07-08 23:39:29,048][root][INFO] - ===> Epoch[9100](45500/5): Loss 0.0082, Sem 0.0082 LR: 2.786e-02 Score 99.709 Data time: 0.2128, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7301 [2021-07-08 23:39:43,154][root][INFO] - ===> Epoch[9101](45505/5): Loss 0.0082, Sem 0.0082 LR: 2.785e-02 Score 99.702 Data time: 0.2778, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7746 [2021-07-08 23:39:57,649][root][INFO] - ===> Epoch[9102](45510/5): Loss 0.0085, Sem 0.0085 LR: 2.784e-02 Score 99.698 Data time: 0.2337, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8633 [2021-07-08 23:40:11,471][root][INFO] - ===> Epoch[9103](45515/5): Loss 0.0078, Sem 0.0078 LR: 2.783e-02 Score 99.715 Data time: 0.2207, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7180 [2021-07-08 23:40:25,476][root][INFO] - ===> Epoch[9104](45520/5): Loss 0.0084, Sem 0.0084 LR: 2.782e-02 Score 99.697 Data time: 0.2413, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7507 [2021-07-08 23:40:39,024][root][INFO] - ===> Epoch[9105](45525/5): Loss 0.0082, Sem 0.0082 LR: 2.781e-02 Score 99.711 Data time: 0.2061, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6716 [2021-07-08 23:40:52,769][root][INFO] - ===> Epoch[9106](45530/5): Loss 0.0078, Sem 0.0078 LR: 2.780e-02 Score 99.720 Data time: 0.2252, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6918 [2021-07-08 23:41:07,251][root][INFO] - ===> Epoch[9107](45535/5): Loss 0.0080, Sem 0.0080 LR: 2.780e-02 Score 99.712 Data time: 0.2194, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8580 [2021-07-08 23:41:21,264][root][INFO] - ===> Epoch[9108](45540/5): Loss 0.0078, Sem 0.0078 LR: 2.779e-02 Score 99.716 Data time: 0.2126, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7483 [2021-07-08 23:41:34,855][root][INFO] - ===> Epoch[9109](45545/5): Loss 0.0076, Sem 0.0076 LR: 2.778e-02 Score 99.724 Data time: 0.2410, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6695 [2021-07-08 23:41:49,479][root][INFO] - ===> Epoch[9110](45550/5): Loss 0.0080, Sem 0.0080 LR: 2.777e-02 Score 99.714 Data time: 0.2232, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8883 [2021-07-08 23:42:02,964][root][INFO] - ===> Epoch[9111](45555/5): Loss 0.0077, Sem 0.0077 LR: 2.776e-02 Score 99.725 Data time: 0.2058, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6510 [2021-07-08 23:42:17,842][root][INFO] - ===> Epoch[9112](45560/5): Loss 0.0082, Sem 0.0082 LR: 2.775e-02 Score 99.708 Data time: 0.2355, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.9341 [2021-07-08 23:42:31,780][root][INFO] - ===> Epoch[9113](45565/5): Loss 0.0082, Sem 0.0082 LR: 2.774e-02 Score 99.706 Data time: 0.2174, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7369 [2021-07-08 23:42:45,587][root][INFO] - ===> Epoch[9114](45570/5): Loss 0.0080, Sem 0.0080 LR: 2.773e-02 Score 99.712 Data time: 0.2688, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7087 [2021-07-08 23:42:59,545][root][INFO] - ===> Epoch[9115](45575/5): Loss 0.0082, Sem 0.0082 LR: 2.773e-02 Score 99.708 Data time: 0.2245, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7507 [2021-07-08 23:43:14,550][root][INFO] - ===> Epoch[9116](45580/5): Loss 0.0082, Sem 0.0082 LR: 2.772e-02 Score 99.706 Data time: 0.2356, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.9555 [2021-07-08 23:43:27,963][root][INFO] - ===> Epoch[9117](45585/5): Loss 0.0081, Sem 0.0081 LR: 2.771e-02 Score 99.708 Data time: 0.2417, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6439 [2021-07-08 23:43:41,327][root][INFO] - ===> Epoch[9118](45590/5): Loss 0.0082, Sem 0.0082 LR: 2.770e-02 Score 99.704 Data time: 0.2589, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6266 [2021-07-08 23:43:55,349][root][INFO] - ===> Epoch[9119](45595/5): Loss 0.0081, Sem 0.0081 LR: 2.769e-02 Score 99.708 Data time: 0.2116, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7551 [2021-07-08 23:44:09,651][root][INFO] - ===> Epoch[9120](45600/5): Loss 0.0079, Sem 0.0079 LR: 2.768e-02 Score 99.718 Data time: 0.2444, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8129 [2021-07-08 23:44:09,652][root][INFO] - ===> Start testing [2021-07-08 23:44:44,985][root][INFO] - 68/68: Data time: 0.0511, Iter time: 0.1321 Loss 0.439 (AVG: 0.608) Score 91.637 (AVG: 90.974) mIOU 71.575 mAP 83.302 mAcc 77.824 IOU: 64.117 0.000 81.939 74.678 95.090 91.834 33.071 83.673 98.637 83.996 78.809 86.341 58.286 mAP: 82.536 3.410 89.945 86.184 98.622 98.301 56.911 97.515 99.744 94.694 91.030 93.249 90.789 mAcc: 80.177 0.000 85.558 82.551 96.632 97.217 36.261 94.261 99.399 91.032 90.326 96.182 62.118 [2021-07-08 23:44:44,985][root][INFO] - Finished test. Elapsed time: 35.3320 [2021-07-08 23:44:46,117][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 23:44:59,756][root][INFO] - ===> Epoch[9121](45605/5): Loss 0.0081, Sem 0.0081 LR: 2.767e-02 Score 99.708 Data time: 0.1648, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6894 [2021-07-08 23:45:13,516][root][INFO] - ===> Epoch[9122](45610/5): Loss 0.0079, Sem 0.0079 LR: 2.767e-02 Score 99.715 Data time: 0.2095, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6986 [2021-07-08 23:45:27,512][root][INFO] - ===> Epoch[9123](45615/5): Loss 0.0078, Sem 0.0078 LR: 2.766e-02 Score 99.717 Data time: 0.2396, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7300 [2021-07-08 23:45:42,541][root][INFO] - ===> Epoch[9124](45620/5): Loss 0.0080, Sem 0.0080 LR: 2.765e-02 Score 99.716 Data time: 0.2013, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.9470 [2021-07-08 23:45:56,393][root][INFO] - ===> Epoch[9125](45625/5): Loss 0.0079, Sem 0.0079 LR: 2.764e-02 Score 99.713 Data time: 0.2309, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7324 [2021-07-08 23:46:09,965][root][INFO] - ===> Epoch[9126](45630/5): Loss 0.0083, Sem 0.0083 LR: 2.763e-02 Score 99.706 Data time: 0.2232, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6711 [2021-07-08 23:46:23,995][root][INFO] - ===> Epoch[9127](45635/5): Loss 0.0081, Sem 0.0081 LR: 2.762e-02 Score 99.708 Data time: 0.2302, Forward time: 1.2540, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7532 [2021-07-08 23:46:37,921][root][INFO] - ===> Epoch[9128](45640/5): Loss 0.0077, Sem 0.0077 LR: 2.761e-02 Score 99.721 Data time: 0.2004, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7396 [2021-07-08 23:46:52,434][root][INFO] - ===> Epoch[9129](45645/5): Loss 0.0080, Sem 0.0080 LR: 2.760e-02 Score 99.711 Data time: 0.1591, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8487 [2021-07-08 23:47:06,135][root][INFO] - ===> Epoch[9130](45650/5): Loss 0.0079, Sem 0.0079 LR: 2.760e-02 Score 99.714 Data time: 0.2452, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6914 [2021-07-08 23:47:20,480][root][INFO] - ===> Epoch[9131](45655/5): Loss 0.0081, Sem 0.0081 LR: 2.759e-02 Score 99.707 Data time: 0.1862, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.8243 [2021-07-08 23:47:33,928][root][INFO] - ===> Epoch[9132](45660/5): Loss 0.0078, Sem 0.0078 LR: 2.758e-02 Score 99.720 Data time: 0.2046, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.6516 [2021-07-08 23:47:47,898][root][INFO] - ===> Epoch[9133](45665/5): Loss 0.0087, Sem 0.0087 LR: 2.757e-02 Score 99.691 Data time: 0.2436, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7421 [2021-07-08 23:48:02,069][root][INFO] - ===> Epoch[9134](45670/5): Loss 0.0078, Sem 0.0078 LR: 2.756e-02 Score 99.716 Data time: 0.1504, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7904 [2021-07-08 23:48:14,835][root][INFO] - ===> Epoch[9135](45675/5): Loss 0.0082, Sem 0.0082 LR: 2.755e-02 Score 99.705 Data time: 0.1889, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.5144 [2021-07-08 23:48:27,862][root][INFO] - ===> Epoch[9136](45680/5): Loss 0.0084, Sem 0.0084 LR: 2.754e-02 Score 99.697 Data time: 0.1941, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.5688 [2021-07-08 23:48:42,557][root][INFO] - ===> Epoch[9137](45685/5): Loss 0.0084, Sem 0.0084 LR: 2.754e-02 Score 99.699 Data time: 0.1839, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8947 [2021-07-08 23:48:56,651][root][INFO] - ===> Epoch[9138](45690/5): Loss 0.0081, Sem 0.0081 LR: 2.753e-02 Score 99.709 Data time: 0.2565, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7705 [2021-07-08 23:49:11,080][root][INFO] - ===> Epoch[9139](45695/5): Loss 0.0079, Sem 0.0079 LR: 2.752e-02 Score 99.715 Data time: 0.2903, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8442 [2021-07-08 23:49:25,333][root][INFO] - ===> Epoch[9140](45700/5): Loss 0.0077, Sem 0.0077 LR: 2.751e-02 Score 99.724 Data time: 0.2510, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8104 [2021-07-08 23:49:40,576][root][INFO] - ===> Epoch[9141](45705/5): Loss 0.0080, Sem 0.0080 LR: 2.750e-02 Score 99.712 Data time: 0.2959, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.9957 [2021-07-08 23:49:54,392][root][INFO] - ===> Epoch[9142](45710/5): Loss 0.0078, Sem 0.0078 LR: 2.749e-02 Score 99.718 Data time: 0.2272, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.7147 [2021-07-08 23:50:08,826][root][INFO] - ===> Epoch[9143](45715/5): Loss 0.0079, Sem 0.0079 LR: 2.748e-02 Score 99.718 Data time: 0.2278, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.8497 [2021-07-08 23:50:22,307][root][INFO] - ===> Epoch[9144](45720/5): Loss 0.0083, Sem 0.0083 LR: 2.748e-02 Score 99.702 Data time: 0.2690, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.6539 [2021-07-08 23:50:37,459][root][INFO] - ===> Epoch[9145](45725/5): Loss 0.0078, Sem 0.0078 LR: 2.747e-02 Score 99.718 Data time: 0.2729, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.9804 [2021-07-08 23:50:52,646][root][INFO] - ===> Epoch[9146](45730/5): Loss 0.0083, Sem 0.0083 LR: 2.746e-02 Score 99.703 Data time: 0.2807, Forward time: 1.2539, Backward time: 1.2456, DDP time: 0.0042, Total iter time: 2.9835 [2021-07-08 23:51:07,106][root][INFO] - ===> Epoch[9147](45735/5): Loss 0.0079, Sem 0.0079 LR: 2.745e-02 Score 99.715 Data time: 0.2093, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.8524 [2021-07-08 23:51:20,526][root][INFO] - ===> Epoch[9148](45740/5): Loss 0.0083, Sem 0.0083 LR: 2.744e-02 Score 99.699 Data time: 0.1888, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.6511 [2021-07-08 23:51:34,304][root][INFO] - ===> Epoch[9149](45745/5): Loss 0.0082, Sem 0.0082 LR: 2.743e-02 Score 99.706 Data time: 0.2758, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7050 [2021-07-08 23:51:48,582][root][INFO] - ===> Epoch[9150](45750/5): Loss 0.0080, Sem 0.0080 LR: 2.742e-02 Score 99.710 Data time: 0.2182, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.8132 [2021-07-08 23:52:03,486][root][INFO] - ===> Epoch[9151](45755/5): Loss 0.0080, Sem 0.0080 LR: 2.741e-02 Score 99.712 Data time: 0.2128, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.9505 [2021-07-08 23:52:17,616][root][INFO] - ===> Epoch[9152](45760/5): Loss 0.0079, Sem 0.0079 LR: 2.741e-02 Score 99.716 Data time: 0.2374, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7869 [2021-07-08 23:52:32,053][root][INFO] - ===> Epoch[9153](45765/5): Loss 0.0078, Sem 0.0078 LR: 2.740e-02 Score 99.717 Data time: 0.2584, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.8404 [2021-07-08 23:52:45,706][root][INFO] - ===> Epoch[9154](45770/5): Loss 0.0081, Sem 0.0081 LR: 2.739e-02 Score 99.708 Data time: 0.2018, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.6873 [2021-07-08 23:52:59,234][root][INFO] - ===> Epoch[9155](45775/5): Loss 0.0082, Sem 0.0082 LR: 2.738e-02 Score 99.704 Data time: 0.2440, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.6566 [2021-07-08 23:53:13,061][root][INFO] - ===> Epoch[9156](45780/5): Loss 0.0083, Sem 0.0083 LR: 2.737e-02 Score 99.702 Data time: 0.2168, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7217 [2021-07-08 23:53:26,658][root][INFO] - ===> Epoch[9157](45785/5): Loss 0.0080, Sem 0.0080 LR: 2.736e-02 Score 99.714 Data time: 0.1899, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.6730 [2021-07-08 23:53:40,534][root][INFO] - ===> Epoch[9158](45790/5): Loss 0.0085, Sem 0.0085 LR: 2.735e-02 Score 99.700 Data time: 0.2440, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7336 [2021-07-08 23:53:54,154][root][INFO] - ===> Epoch[9159](45795/5): Loss 0.0083, Sem 0.0083 LR: 2.735e-02 Score 99.700 Data time: 0.2039, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.6849 [2021-07-08 23:54:08,122][root][INFO] - ===> Epoch[9160](45800/5): Loss 0.0098, Sem 0.0098 LR: 2.734e-02 Score 99.677 Data time: 0.1999, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7487 [2021-07-08 23:54:08,124][root][INFO] - ===> Start testing [2021-07-08 23:54:43,683][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1880 Loss 0.427 (AVG: 0.607) Score 91.927 (AVG: 90.938) mIOU 71.097 mAP 82.968 mAcc 76.858 IOU: 64.913 0.000 81.365 75.766 94.957 92.534 24.171 84.203 98.677 84.479 79.825 85.386 57.983 mAP: 83.569 2.744 89.931 86.622 98.653 98.325 52.014 97.562 99.743 94.428 91.646 92.786 90.562 mAcc: 79.884 0.000 83.496 83.505 96.426 97.577 25.736 93.070 99.441 91.022 90.105 96.751 62.138 [2021-07-08 23:54:43,683][root][INFO] - Finished test. Elapsed time: 35.5579 [2021-07-08 23:54:44,798][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-08 23:54:57,783][root][INFO] - ===> Epoch[9161](45805/5): Loss 0.0085, Sem 0.0085 LR: 2.733e-02 Score 99.699 Data time: 0.1897, Forward time: 1.2538, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.5605 [2021-07-08 23:55:11,702][root][INFO] - ===> Epoch[9162](45810/5): Loss 0.0085, Sem 0.0085 LR: 2.732e-02 Score 99.696 Data time: 0.2632, Forward time: 1.2538, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7350 [2021-07-08 23:55:25,710][root][INFO] - ===> Epoch[9163](45815/5): Loss 0.0085, Sem 0.0085 LR: 2.731e-02 Score 99.699 Data time: 0.2322, Forward time: 1.2538, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7562 [2021-07-08 23:55:40,539][root][INFO] - ===> Epoch[9164](45820/5): Loss 0.0087, Sem 0.0087 LR: 2.730e-02 Score 99.691 Data time: 0.2148, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.9245 [2021-07-08 23:55:54,301][root][INFO] - ===> Epoch[9165](45825/5): Loss 0.0086, Sem 0.0086 LR: 2.729e-02 Score 99.690 Data time: 0.2269, Forward time: 1.2538, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7051 [2021-07-08 23:56:08,004][root][INFO] - ===> Epoch[9166](45830/5): Loss 0.0085, Sem 0.0085 LR: 2.728e-02 Score 99.695 Data time: 0.2640, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.6913 [2021-07-08 23:56:22,437][root][INFO] - ===> Epoch[9167](45835/5): Loss 0.0083, Sem 0.0083 LR: 2.728e-02 Score 99.702 Data time: 0.2371, Forward time: 1.2538, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.8392 [2021-07-08 23:56:37,119][root][INFO] - ===> Epoch[9168](45840/5): Loss 0.0081, Sem 0.0081 LR: 2.727e-02 Score 99.708 Data time: 0.2902, Forward time: 1.2539, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.8905 [2021-07-08 23:56:51,455][root][INFO] - ===> Epoch[9169](45845/5): Loss 0.0079, Sem 0.0079 LR: 2.726e-02 Score 99.715 Data time: 0.1764, Forward time: 1.2538, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.8202 [2021-07-08 23:57:05,011][root][INFO] - ===> Epoch[9170](45850/5): Loss 0.0086, Sem 0.0086 LR: 2.725e-02 Score 99.692 Data time: 0.2081, Forward time: 1.2538, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.6707 [2021-07-08 23:57:19,288][root][INFO] - ===> Epoch[9171](45855/5): Loss 0.0084, Sem 0.0084 LR: 2.724e-02 Score 99.702 Data time: 0.2495, Forward time: 1.2538, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7877 [2021-07-08 23:57:33,464][root][INFO] - ===> Epoch[9172](45860/5): Loss 0.0079, Sem 0.0079 LR: 2.723e-02 Score 99.713 Data time: 0.1897, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7804 [2021-07-08 23:57:48,172][root][INFO] - ===> Epoch[9173](45865/5): Loss 0.0079, Sem 0.0079 LR: 2.722e-02 Score 99.716 Data time: 0.1808, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.9048 [2021-07-08 23:58:01,226][root][INFO] - ===> Epoch[9174](45870/5): Loss 0.0078, Sem 0.0078 LR: 2.722e-02 Score 99.719 Data time: 0.2075, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.5634 [2021-07-08 23:58:14,539][root][INFO] - ===> Epoch[9175](45875/5): Loss 0.0084, Sem 0.0084 LR: 2.721e-02 Score 99.698 Data time: 0.2656, Forward time: 1.2538, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.6022 [2021-07-08 23:58:28,114][root][INFO] - ===> Epoch[9176](45880/5): Loss 0.0078, Sem 0.0078 LR: 2.720e-02 Score 99.718 Data time: 0.1633, Forward time: 1.2538, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.6683 [2021-07-08 23:58:42,244][root][INFO] - ===> Epoch[9177](45885/5): Loss 0.0082, Sem 0.0082 LR: 2.719e-02 Score 99.705 Data time: 0.1898, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7838 [2021-07-08 23:58:55,789][root][INFO] - ===> Epoch[9178](45890/5): Loss 0.0084, Sem 0.0084 LR: 2.718e-02 Score 99.701 Data time: 0.2176, Forward time: 1.2538, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.6514 [2021-07-08 23:59:09,485][root][INFO] - ===> Epoch[9179](45895/5): Loss 0.0082, Sem 0.0082 LR: 2.717e-02 Score 99.704 Data time: 0.2005, Forward time: 1.2538, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.6993 [2021-07-08 23:59:22,630][root][INFO] - ===> Epoch[9180](45900/5): Loss 0.0082, Sem 0.0082 LR: 2.716e-02 Score 99.705 Data time: 0.2187, Forward time: 1.2538, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.5814 [2021-07-08 23:59:36,626][root][INFO] - ===> Epoch[9181](45905/5): Loss 0.0080, Sem 0.0080 LR: 2.715e-02 Score 99.716 Data time: 0.1831, Forward time: 1.2538, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7550 [2021-07-08 23:59:51,246][root][INFO] - ===> Epoch[9182](45910/5): Loss 0.0080, Sem 0.0080 LR: 2.715e-02 Score 99.713 Data time: 0.2588, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.8827 [2021-07-09 00:00:05,493][root][INFO] - ===> Epoch[9183](45915/5): Loss 0.0079, Sem 0.0079 LR: 2.714e-02 Score 99.713 Data time: 0.2197, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.8036 [2021-07-09 00:00:19,476][root][INFO] - ===> Epoch[9184](45920/5): Loss 0.0077, Sem 0.0077 LR: 2.713e-02 Score 99.719 Data time: 0.2098, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7480 [2021-07-09 00:00:33,554][root][INFO] - ===> Epoch[9185](45925/5): Loss 0.0083, Sem 0.0083 LR: 2.712e-02 Score 99.701 Data time: 0.2344, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7656 [2021-07-09 00:00:47,734][root][INFO] - ===> Epoch[9186](45930/5): Loss 0.0080, Sem 0.0080 LR: 2.711e-02 Score 99.713 Data time: 0.2525, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7950 [2021-07-09 00:01:01,471][root][INFO] - ===> Epoch[9187](45935/5): Loss 0.0083, Sem 0.0083 LR: 2.710e-02 Score 99.701 Data time: 0.2970, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7009 [2021-07-09 00:01:15,733][root][INFO] - ===> Epoch[9188](45940/5): Loss 0.0079, Sem 0.0079 LR: 2.709e-02 Score 99.716 Data time: 0.2046, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.8129 [2021-07-09 00:01:30,063][root][INFO] - ===> Epoch[9189](45945/5): Loss 0.0082, Sem 0.0082 LR: 2.709e-02 Score 99.705 Data time: 0.3055, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.8195 [2021-07-09 00:01:45,407][root][INFO] - ===> Epoch[9190](45950/5): Loss 0.0080, Sem 0.0080 LR: 2.708e-02 Score 99.713 Data time: 0.3411, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 3.0199 [2021-07-09 00:01:59,676][root][INFO] - ===> Epoch[9191](45955/5): Loss 0.0081, Sem 0.0081 LR: 2.707e-02 Score 99.714 Data time: 0.2447, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.8024 [2021-07-09 00:02:13,463][root][INFO] - ===> Epoch[9192](45960/5): Loss 0.0080, Sem 0.0080 LR: 2.706e-02 Score 99.711 Data time: 0.2344, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7013 [2021-07-09 00:02:29,087][root][INFO] - ===> Epoch[9193](45965/5): Loss 0.0078, Sem 0.0078 LR: 2.705e-02 Score 99.720 Data time: 0.2171, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 3.0828 [2021-07-09 00:02:42,983][root][INFO] - ===> Epoch[9194](45970/5): Loss 0.0081, Sem 0.0081 LR: 2.704e-02 Score 99.708 Data time: 0.2216, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7270 [2021-07-09 00:02:56,949][root][INFO] - ===> Epoch[9195](45975/5): Loss 0.0082, Sem 0.0082 LR: 2.703e-02 Score 99.706 Data time: 0.2479, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7498 [2021-07-09 00:03:10,697][root][INFO] - ===> Epoch[9196](45980/5): Loss 0.0079, Sem 0.0079 LR: 2.702e-02 Score 99.718 Data time: 0.1801, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7052 [2021-07-09 00:03:24,393][root][INFO] - ===> Epoch[9197](45985/5): Loss 0.0082, Sem 0.0082 LR: 2.702e-02 Score 99.707 Data time: 0.2353, Forward time: 1.2538, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.6946 [2021-07-09 00:03:37,780][root][INFO] - ===> Epoch[9198](45990/5): Loss 0.0084, Sem 0.0084 LR: 2.701e-02 Score 99.700 Data time: 0.2458, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.6345 [2021-07-09 00:03:51,374][root][INFO] - ===> Epoch[9199](45995/5): Loss 0.0082, Sem 0.0082 LR: 2.700e-02 Score 99.707 Data time: 0.2578, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.6643 [2021-07-09 00:04:05,292][root][INFO] - ===> Epoch[9200](46000/5): Loss 0.0082, Sem 0.0082 LR: 2.699e-02 Score 99.705 Data time: 0.2521, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7369 [2021-07-09 00:04:05,294][root][INFO] - ===> Start testing [2021-07-09 00:04:41,467][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1246 Loss 0.418 (AVG: 0.604) Score 91.358 (AVG: 90.948) mIOU 71.814 mAP 83.440 mAcc 78.214 IOU: 63.430 0.000 80.606 74.396 95.001 91.883 35.528 83.782 98.598 86.340 79.444 86.478 58.101 mAP: 82.902 4.230 90.161 86.200 98.529 98.347 56.397 98.146 99.702 95.489 90.701 93.205 90.716 mAcc: 79.935 0.000 84.530 82.120 96.485 97.209 39.436 95.866 99.432 93.538 90.257 96.109 61.861 [2021-07-09 00:04:41,467][root][INFO] - Finished test. Elapsed time: 36.1728 [2021-07-09 00:04:42,627][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 00:04:55,985][root][INFO] - ===> Epoch[9201](46005/5): Loss 0.0080, Sem 0.0080 LR: 2.698e-02 Score 99.715 Data time: 0.2080, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.6289 [2021-07-09 00:05:10,006][root][INFO] - ===> Epoch[9202](46010/5): Loss 0.0081, Sem 0.0081 LR: 2.697e-02 Score 99.708 Data time: 0.2178, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7678 [2021-07-09 00:05:23,351][root][INFO] - ===> Epoch[9203](46015/5): Loss 0.0078, Sem 0.0078 LR: 2.696e-02 Score 99.718 Data time: 0.2179, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.6164 [2021-07-09 00:05:37,396][root][INFO] - ===> Epoch[9204](46020/5): Loss 0.0078, Sem 0.0078 LR: 2.696e-02 Score 99.722 Data time: 0.1831, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7684 [2021-07-09 00:05:51,164][root][INFO] - ===> Epoch[9205](46025/5): Loss 0.0076, Sem 0.0076 LR: 2.695e-02 Score 99.724 Data time: 0.1844, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7124 [2021-07-09 00:06:04,609][root][INFO] - ===> Epoch[9206](46030/5): Loss 0.0078, Sem 0.0078 LR: 2.694e-02 Score 99.720 Data time: 0.1979, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.6456 [2021-07-09 00:06:18,898][root][INFO] - ===> Epoch[9207](46035/5): Loss 0.0083, Sem 0.0083 LR: 2.693e-02 Score 99.701 Data time: 0.2387, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.8014 [2021-07-09 00:06:33,363][root][INFO] - ===> Epoch[9208](46040/5): Loss 0.0074, Sem 0.0074 LR: 2.692e-02 Score 99.735 Data time: 0.2216, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.8406 [2021-07-09 00:06:47,847][root][INFO] - ===> Epoch[9209](46045/5): Loss 0.0078, Sem 0.0078 LR: 2.691e-02 Score 99.719 Data time: 0.2408, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.8521 [2021-07-09 00:07:01,071][root][INFO] - ===> Epoch[9210](46050/5): Loss 0.0079, Sem 0.0079 LR: 2.690e-02 Score 99.716 Data time: 0.2333, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.5968 [2021-07-09 00:07:14,993][root][INFO] - ===> Epoch[9211](46055/5): Loss 0.0082, Sem 0.0082 LR: 2.689e-02 Score 99.704 Data time: 0.2019, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7461 [2021-07-09 00:07:28,860][root][INFO] - ===> Epoch[9212](46060/5): Loss 0.0083, Sem 0.0083 LR: 2.689e-02 Score 99.705 Data time: 0.2261, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7285 [2021-07-09 00:07:43,559][root][INFO] - ===> Epoch[9213](46065/5): Loss 0.0085, Sem 0.0085 LR: 2.688e-02 Score 99.692 Data time: 0.1956, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.8867 [2021-07-09 00:07:57,579][root][INFO] - ===> Epoch[9214](46070/5): Loss 0.0080, Sem 0.0080 LR: 2.687e-02 Score 99.713 Data time: 0.2010, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7443 [2021-07-09 00:08:11,292][root][INFO] - ===> Epoch[9215](46075/5): Loss 0.0084, Sem 0.0084 LR: 2.686e-02 Score 99.699 Data time: 0.1514, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.6964 [2021-07-09 00:08:25,296][root][INFO] - ===> Epoch[9216](46080/5): Loss 0.0080, Sem 0.0080 LR: 2.685e-02 Score 99.710 Data time: 0.2069, Forward time: 1.2537, Backward time: 1.2459, DDP time: 0.0042, Total iter time: 2.7643 [2021-07-09 00:08:38,973][root][INFO] - ===> Epoch[9217](46085/5): Loss 0.0079, Sem 0.0079 LR: 2.684e-02 Score 99.719 Data time: 0.1811, Forward time: 1.2537, Backward time: 1.2459, DDP time: 0.0042, Total iter time: 2.6921 [2021-07-09 00:08:52,782][root][INFO] - ===> Epoch[9218](46090/5): Loss 0.0079, Sem 0.0079 LR: 2.683e-02 Score 99.718 Data time: 0.2567, Forward time: 1.2537, Backward time: 1.2459, DDP time: 0.0042, Total iter time: 2.7155 [2021-07-09 00:09:06,932][root][INFO] - ===> Epoch[9219](46095/5): Loss 0.0079, Sem 0.0079 LR: 2.682e-02 Score 99.711 Data time: 0.1869, Forward time: 1.2536, Backward time: 1.2459, DDP time: 0.0042, Total iter time: 2.7882 [2021-07-09 00:09:20,693][root][INFO] - ===> Epoch[9220](46100/5): Loss 0.0085, Sem 0.0085 LR: 2.682e-02 Score 99.698 Data time: 0.2380, Forward time: 1.2537, Backward time: 1.2459, DDP time: 0.0042, Total iter time: 2.7050 [2021-07-09 00:09:33,998][root][INFO] - ===> Epoch[9221](46105/5): Loss 0.0085, Sem 0.0085 LR: 2.681e-02 Score 99.693 Data time: 0.2500, Forward time: 1.2536, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.6040 [2021-07-09 00:09:47,987][root][INFO] - ===> Epoch[9222](46110/5): Loss 0.0086, Sem 0.0086 LR: 2.680e-02 Score 99.712 Data time: 0.2028, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7487 [2021-07-09 00:10:01,663][root][INFO] - ===> Epoch[9223](46115/5): Loss 0.0078, Sem 0.0078 LR: 2.679e-02 Score 99.718 Data time: 0.1920, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.6853 [2021-07-09 00:10:15,823][root][INFO] - ===> Epoch[9224](46120/5): Loss 0.0086, Sem 0.0086 LR: 2.678e-02 Score 99.695 Data time: 0.2340, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7862 [2021-07-09 00:10:29,927][root][INFO] - ===> Epoch[9225](46125/5): Loss 0.0081, Sem 0.0081 LR: 2.677e-02 Score 99.714 Data time: 0.2142, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7793 [2021-07-09 00:10:43,621][root][INFO] - ===> Epoch[9226](46130/5): Loss 0.0085, Sem 0.0085 LR: 2.676e-02 Score 99.693 Data time: 0.2280, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.6921 [2021-07-09 00:10:57,597][root][INFO] - ===> Epoch[9227](46135/5): Loss 0.0088, Sem 0.0088 LR: 2.676e-02 Score 99.686 Data time: 0.2221, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7455 [2021-07-09 00:11:12,080][root][INFO] - ===> Epoch[9228](46140/5): Loss 0.0080, Sem 0.0080 LR: 2.675e-02 Score 99.711 Data time: 0.2180, Forward time: 1.2536, Backward time: 1.2459, DDP time: 0.0042, Total iter time: 2.8593 [2021-07-09 00:11:25,961][root][INFO] - ===> Epoch[9229](46145/5): Loss 0.0083, Sem 0.0083 LR: 2.674e-02 Score 99.702 Data time: 0.2325, Forward time: 1.2536, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7379 [2021-07-09 00:11:39,768][root][INFO] - ===> Epoch[9230](46150/5): Loss 0.0084, Sem 0.0084 LR: 2.673e-02 Score 99.699 Data time: 0.1958, Forward time: 1.2536, Backward time: 1.2459, DDP time: 0.0042, Total iter time: 2.7225 [2021-07-09 00:11:53,206][root][INFO] - ===> Epoch[9231](46155/5): Loss 0.0081, Sem 0.0081 LR: 2.672e-02 Score 99.709 Data time: 0.2229, Forward time: 1.2536, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.6436 [2021-07-09 00:12:06,171][root][INFO] - ===> Epoch[9232](46160/5): Loss 0.0079, Sem 0.0079 LR: 2.671e-02 Score 99.713 Data time: 0.2148, Forward time: 1.2536, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.5400 [2021-07-09 00:12:19,676][root][INFO] - ===> Epoch[9233](46165/5): Loss 0.0077, Sem 0.0077 LR: 2.670e-02 Score 99.723 Data time: 0.2309, Forward time: 1.2536, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.6565 [2021-07-09 00:12:33,321][root][INFO] - ===> Epoch[9234](46170/5): Loss 0.0079, Sem 0.0079 LR: 2.669e-02 Score 99.714 Data time: 0.2374, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.6855 [2021-07-09 00:12:47,974][root][INFO] - ===> Epoch[9235](46175/5): Loss 0.0082, Sem 0.0082 LR: 2.669e-02 Score 99.709 Data time: 0.2670, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.8852 [2021-07-09 00:13:02,536][root][INFO] - ===> Epoch[9236](46180/5): Loss 0.0081, Sem 0.0081 LR: 2.668e-02 Score 99.710 Data time: 0.2702, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.8681 [2021-07-09 00:13:16,492][root][INFO] - ===> Epoch[9237](46185/5): Loss 0.0079, Sem 0.0079 LR: 2.667e-02 Score 99.719 Data time: 0.2589, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7436 [2021-07-09 00:13:30,581][root][INFO] - ===> Epoch[9238](46190/5): Loss 0.0080, Sem 0.0080 LR: 2.666e-02 Score 99.712 Data time: 0.3277, Forward time: 1.2537, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7604 [2021-07-09 00:13:45,691][root][INFO] - ===> Epoch[9239](46195/5): Loss 0.0077, Sem 0.0077 LR: 2.665e-02 Score 99.719 Data time: 0.2779, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.9736 [2021-07-09 00:13:59,364][root][INFO] - ===> Epoch[9240](46200/5): Loss 0.0085, Sem 0.0085 LR: 2.664e-02 Score 99.698 Data time: 0.3395, Forward time: 1.2537, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.6745 [2021-07-09 00:13:59,366][root][INFO] - ===> Start testing [2021-07-09 00:14:35,887][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1209 Loss 0.433 (AVG: 0.621) Score 90.741 (AVG: 90.841) mIOU 71.074 mAP 82.824 mAcc 77.433 IOU: 63.727 0.000 80.250 74.793 94.993 91.981 29.174 82.785 98.508 85.213 79.278 85.986 57.276 mAP: 82.930 2.618 91.141 85.791 98.453 98.236 51.702 97.838 99.750 94.987 90.733 93.013 89.517 mAcc: 79.243 0.000 84.492 82.079 96.537 97.467 31.333 95.228 99.191 92.670 90.714 96.704 60.973 [2021-07-09 00:14:35,887][root][INFO] - Finished test. Elapsed time: 36.5205 [2021-07-09 00:14:37,008][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 00:14:50,272][root][INFO] - ===> Epoch[9241](46205/5): Loss 0.0078, Sem 0.0078 LR: 2.663e-02 Score 99.719 Data time: 0.1961, Forward time: 1.2537, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.6143 [2021-07-09 00:15:04,216][root][INFO] - ===> Epoch[9242](46210/5): Loss 0.0078, Sem 0.0078 LR: 2.663e-02 Score 99.721 Data time: 0.1965, Forward time: 1.2537, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7427 [2021-07-09 00:15:18,446][root][INFO] - ===> Epoch[9243](46215/5): Loss 0.0078, Sem 0.0078 LR: 2.662e-02 Score 99.722 Data time: 0.2284, Forward time: 1.2537, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.8030 [2021-07-09 00:15:32,297][root][INFO] - ===> Epoch[9244](46220/5): Loss 0.0076, Sem 0.0076 LR: 2.661e-02 Score 99.722 Data time: 0.2288, Forward time: 1.2537, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7250 [2021-07-09 00:15:46,119][root][INFO] - ===> Epoch[9245](46225/5): Loss 0.0077, Sem 0.0077 LR: 2.660e-02 Score 99.726 Data time: 0.1821, Forward time: 1.2537, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7169 [2021-07-09 00:16:00,190][root][INFO] - ===> Epoch[9246](46230/5): Loss 0.0081, Sem 0.0081 LR: 2.659e-02 Score 99.709 Data time: 0.2183, Forward time: 1.2537, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7652 [2021-07-09 00:16:14,394][root][INFO] - ===> Epoch[9247](46235/5): Loss 0.0080, Sem 0.0080 LR: 2.658e-02 Score 99.713 Data time: 0.1592, Forward time: 1.2537, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.8061 [2021-07-09 00:16:28,601][root][INFO] - ===> Epoch[9248](46240/5): Loss 0.0078, Sem 0.0078 LR: 2.657e-02 Score 99.719 Data time: 0.2508, Forward time: 1.2537, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.7954 [2021-07-09 00:16:42,039][root][INFO] - ===> Epoch[9249](46245/5): Loss 0.0081, Sem 0.0081 LR: 2.656e-02 Score 99.709 Data time: 0.2101, Forward time: 1.2537, Backward time: 1.2457, DDP time: 0.0042, Total iter time: 2.6303 [2021-07-09 00:16:56,443][root][INFO] - ===> Epoch[9250](46250/5): Loss 0.0082, Sem 0.0082 LR: 2.656e-02 Score 99.708 Data time: 0.2094, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.8270 [2021-07-09 00:17:10,980][root][INFO] - ===> Epoch[9251](46255/5): Loss 0.0081, Sem 0.0081 LR: 2.655e-02 Score 99.711 Data time: 0.2270, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.8621 [2021-07-09 00:17:24,876][root][INFO] - ===> Epoch[9252](46260/5): Loss 0.0082, Sem 0.0082 LR: 2.654e-02 Score 99.707 Data time: 0.2202, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7377 [2021-07-09 00:17:39,606][root][INFO] - ===> Epoch[9253](46265/5): Loss 0.0079, Sem 0.0079 LR: 2.653e-02 Score 99.718 Data time: 0.2072, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.9020 [2021-07-09 00:17:53,527][root][INFO] - ===> Epoch[9254](46270/5): Loss 0.0080, Sem 0.0080 LR: 2.652e-02 Score 99.714 Data time: 0.2392, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7442 [2021-07-09 00:18:06,840][root][INFO] - ===> Epoch[9255](46275/5): Loss 0.0081, Sem 0.0081 LR: 2.651e-02 Score 99.705 Data time: 0.2414, Forward time: 1.2537, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.6177 [2021-07-09 00:18:21,144][root][INFO] - ===> Epoch[9256](46280/5): Loss 0.0078, Sem 0.0078 LR: 2.650e-02 Score 99.722 Data time: 0.2145, Forward time: 1.2536, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.8153 [2021-07-09 00:18:34,636][root][INFO] - ===> Epoch[9257](46285/5): Loss 0.0080, Sem 0.0080 LR: 2.649e-02 Score 99.711 Data time: 0.1657, Forward time: 1.2536, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.6537 [2021-07-09 00:18:48,463][root][INFO] - ===> Epoch[9258](46290/5): Loss 0.0079, Sem 0.0079 LR: 2.649e-02 Score 99.716 Data time: 0.2331, Forward time: 1.2536, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7196 [2021-07-09 00:19:02,305][root][INFO] - ===> Epoch[9259](46295/5): Loss 0.0079, Sem 0.0079 LR: 2.648e-02 Score 99.719 Data time: 0.2123, Forward time: 1.2536, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7304 [2021-07-09 00:19:16,928][root][INFO] - ===> Epoch[9260](46300/5): Loss 0.0082, Sem 0.0082 LR: 2.647e-02 Score 99.707 Data time: 0.1881, Forward time: 1.2536, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.8866 [2021-07-09 00:19:30,602][root][INFO] - ===> Epoch[9261](46305/5): Loss 0.0080, Sem 0.0080 LR: 2.646e-02 Score 99.710 Data time: 0.3110, Forward time: 1.2536, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.6807 [2021-07-09 00:19:44,239][root][INFO] - ===> Epoch[9262](46310/5): Loss 0.0081, Sem 0.0081 LR: 2.645e-02 Score 99.712 Data time: 0.1719, Forward time: 1.2536, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.6799 [2021-07-09 00:19:58,615][root][INFO] - ===> Epoch[9263](46315/5): Loss 0.0078, Sem 0.0078 LR: 2.644e-02 Score 99.717 Data time: 0.2007, Forward time: 1.2536, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.8369 [2021-07-09 00:20:12,562][root][INFO] - ===> Epoch[9264](46320/5): Loss 0.0078, Sem 0.0078 LR: 2.643e-02 Score 99.717 Data time: 0.2029, Forward time: 1.2536, Backward time: 1.2458, DDP time: 0.0042, Total iter time: 2.7435 [2021-07-09 00:20:26,822][root][INFO] - ===> Epoch[9265](46325/5): Loss 0.0081, Sem 0.0081 LR: 2.643e-02 Score 99.710 Data time: 0.1654, Forward time: 1.2536, Backward time: 1.2459, DDP time: 0.0042, Total iter time: 2.8180 [2021-07-09 00:20:41,588][root][INFO] - ===> Epoch[9266](46330/5): Loss 0.0076, Sem 0.0076 LR: 2.642e-02 Score 99.730 Data time: 0.2311, Forward time: 1.2536, Backward time: 1.2459, DDP time: 0.0042, Total iter time: 2.9030 [2021-07-09 00:20:55,432][root][INFO] - ===> Epoch[9267](46335/5): Loss 0.0077, Sem 0.0077 LR: 2.641e-02 Score 99.720 Data time: 0.1925, Forward time: 1.2536, Backward time: 1.2459, DDP time: 0.0042, Total iter time: 2.7345 [2021-07-09 00:21:09,181][root][INFO] - ===> Epoch[9268](46340/5): Loss 0.0075, Sem 0.0075 LR: 2.640e-02 Score 99.731 Data time: 0.1594, Forward time: 1.2536, Backward time: 1.2459, DDP time: 0.0042, Total iter time: 2.7023 [2021-07-09 00:21:23,666][root][INFO] - ===> Epoch[9269](46345/5): Loss 0.0078, Sem 0.0078 LR: 2.639e-02 Score 99.718 Data time: 0.2097, Forward time: 1.2536, Backward time: 1.2459, DDP time: 0.0042, Total iter time: 2.8563 [2021-07-09 00:21:37,737][root][INFO] - ===> Epoch[9270](46350/5): Loss 0.0077, Sem 0.0077 LR: 2.638e-02 Score 99.724 Data time: 0.2333, Forward time: 1.2536, Backward time: 1.2459, DDP time: 0.0042, Total iter time: 2.7628 [2021-07-09 00:21:51,909][root][INFO] - ===> Epoch[9271](46355/5): Loss 0.0077, Sem 0.0077 LR: 2.637e-02 Score 99.724 Data time: 0.2396, Forward time: 1.2536, Backward time: 1.2459, DDP time: 0.0042, Total iter time: 2.7839 [2021-07-09 00:22:05,808][root][INFO] - ===> Epoch[9272](46360/5): Loss 0.0081, Sem 0.0081 LR: 2.636e-02 Score 99.706 Data time: 0.2199, Forward time: 1.2536, Backward time: 1.2459, DDP time: 0.0042, Total iter time: 2.7266 [2021-07-09 00:22:20,403][root][INFO] - ===> Epoch[9273](46365/5): Loss 0.0079, Sem 0.0079 LR: 2.636e-02 Score 99.714 Data time: 0.2055, Forward time: 1.2536, Backward time: 1.2459, DDP time: 0.0042, Total iter time: 2.8671 [2021-07-09 00:22:34,673][root][INFO] - ===> Epoch[9274](46370/5): Loss 0.0077, Sem 0.0077 LR: 2.635e-02 Score 99.720 Data time: 0.1820, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.8073 [2021-07-09 00:22:48,774][root][INFO] - ===> Epoch[9275](46375/5): Loss 0.0076, Sem 0.0076 LR: 2.634e-02 Score 99.726 Data time: 0.2535, Forward time: 1.2536, Backward time: 1.2459, DDP time: 0.0042, Total iter time: 2.7668 [2021-07-09 00:23:04,313][root][INFO] - ===> Epoch[9276](46380/5): Loss 0.0077, Sem 0.0077 LR: 2.633e-02 Score 99.724 Data time: 0.1946, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 3.0599 [2021-07-09 00:23:18,565][root][INFO] - ===> Epoch[9277](46385/5): Loss 0.0078, Sem 0.0078 LR: 2.632e-02 Score 99.717 Data time: 0.2323, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.7937 [2021-07-09 00:23:32,610][root][INFO] - ===> Epoch[9278](46390/5): Loss 0.0077, Sem 0.0077 LR: 2.631e-02 Score 99.726 Data time: 0.2067, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.7615 [2021-07-09 00:23:47,285][root][INFO] - ===> Epoch[9279](46395/5): Loss 0.0080, Sem 0.0080 LR: 2.630e-02 Score 99.714 Data time: 0.1887, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.8855 [2021-07-09 00:24:01,322][root][INFO] - ===> Epoch[9280](46400/5): Loss 0.0073, Sem 0.0073 LR: 2.629e-02 Score 99.734 Data time: 0.2236, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.7648 [2021-07-09 00:24:01,324][root][INFO] - ===> Start testing [2021-07-09 00:24:36,300][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1260 Loss 0.416 (AVG: 0.619) Score 91.743 (AVG: 91.030) mIOU 71.483 mAP 82.960 mAcc 77.860 IOU: 64.361 0.000 80.367 75.167 95.207 91.851 33.043 82.607 98.614 84.055 79.038 86.453 58.515 mAP: 83.604 2.174 89.263 85.799 98.484 98.184 55.580 97.764 99.741 94.580 90.573 92.354 90.381 mAcc: 80.136 0.000 83.517 82.942 96.650 97.526 36.569 94.761 99.316 92.680 90.114 96.319 61.650 [2021-07-09 00:24:36,300][root][INFO] - Finished test. Elapsed time: 34.9758 [2021-07-09 00:24:37,401][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 00:24:51,294][root][INFO] - ===> Epoch[9281](46405/5): Loss 0.0079, Sem 0.0079 LR: 2.629e-02 Score 99.717 Data time: 0.2570, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.7380 [2021-07-09 00:25:06,388][root][INFO] - ===> Epoch[9282](46410/5): Loss 0.0075, Sem 0.0075 LR: 2.628e-02 Score 99.729 Data time: 0.2249, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.9669 [2021-07-09 00:25:20,145][root][INFO] - ===> Epoch[9283](46415/5): Loss 0.0079, Sem 0.0079 LR: 2.627e-02 Score 99.712 Data time: 0.2854, Forward time: 1.2536, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.7039 [2021-07-09 00:25:33,992][root][INFO] - ===> Epoch[9284](46420/5): Loss 0.0079, Sem 0.0079 LR: 2.626e-02 Score 99.713 Data time: 0.2279, Forward time: 1.2536, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.7231 [2021-07-09 00:25:48,137][root][INFO] - ===> Epoch[9285](46425/5): Loss 0.0075, Sem 0.0075 LR: 2.625e-02 Score 99.730 Data time: 0.2789, Forward time: 1.2536, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.7748 [2021-07-09 00:26:02,009][root][INFO] - ===> Epoch[9286](46430/5): Loss 0.0078, Sem 0.0078 LR: 2.624e-02 Score 99.723 Data time: 0.2217, Forward time: 1.2536, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.7259 [2021-07-09 00:26:16,915][root][INFO] - ===> Epoch[9287](46435/5): Loss 0.0081, Sem 0.0081 LR: 2.623e-02 Score 99.711 Data time: 0.1662, Forward time: 1.2536, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.9462 [2021-07-09 00:26:31,207][root][INFO] - ===> Epoch[9288](46440/5): Loss 0.0077, Sem 0.0077 LR: 2.623e-02 Score 99.720 Data time: 0.1827, Forward time: 1.2536, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.8224 [2021-07-09 00:26:44,862][root][INFO] - ===> Epoch[9289](46445/5): Loss 0.0075, Sem 0.0075 LR: 2.622e-02 Score 99.729 Data time: 0.3049, Forward time: 1.2536, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.6878 [2021-07-09 00:26:58,351][root][INFO] - ===> Epoch[9290](46450/5): Loss 0.0077, Sem 0.0077 LR: 2.621e-02 Score 99.724 Data time: 0.2538, Forward time: 1.2536, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.6489 [2021-07-09 00:27:12,410][root][INFO] - ===> Epoch[9291](46455/5): Loss 0.0077, Sem 0.0077 LR: 2.620e-02 Score 99.724 Data time: 0.2489, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.7731 [2021-07-09 00:27:25,683][root][INFO] - ===> Epoch[9292](46460/5): Loss 0.0080, Sem 0.0080 LR: 2.619e-02 Score 99.712 Data time: 0.2045, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.6095 [2021-07-09 00:27:39,918][root][INFO] - ===> Epoch[9293](46465/5): Loss 0.0078, Sem 0.0078 LR: 2.618e-02 Score 99.722 Data time: 0.2159, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.8062 [2021-07-09 00:27:54,156][root][INFO] - ===> Epoch[9294](46470/5): Loss 0.0077, Sem 0.0077 LR: 2.617e-02 Score 99.720 Data time: 0.2048, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.8043 [2021-07-09 00:28:08,215][root][INFO] - ===> Epoch[9295](46475/5): Loss 0.0077, Sem 0.0077 LR: 2.616e-02 Score 99.723 Data time: 0.2002, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.7726 [2021-07-09 00:28:21,784][root][INFO] - ===> Epoch[9296](46480/5): Loss 0.0078, Sem 0.0078 LR: 2.616e-02 Score 99.721 Data time: 0.2194, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.6716 [2021-07-09 00:28:35,901][root][INFO] - ===> Epoch[9297](46485/5): Loss 0.0077, Sem 0.0077 LR: 2.615e-02 Score 99.722 Data time: 0.2403, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.7700 [2021-07-09 00:28:49,680][root][INFO] - ===> Epoch[9298](46490/5): Loss 0.0078, Sem 0.0078 LR: 2.614e-02 Score 99.721 Data time: 0.1827, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.7210 [2021-07-09 00:29:04,479][root][INFO] - ===> Epoch[9299](46495/5): Loss 0.0078, Sem 0.0078 LR: 2.613e-02 Score 99.717 Data time: 0.2262, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.9156 [2021-07-09 00:29:17,526][root][INFO] - ===> Epoch[9300](46500/5): Loss 0.0082, Sem 0.0082 LR: 2.612e-02 Score 99.706 Data time: 0.1845, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.5698 [2021-07-09 00:29:32,659][root][INFO] - ===> Epoch[9301](46505/5): Loss 0.0072, Sem 0.0072 LR: 2.611e-02 Score 99.741 Data time: 0.2683, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.9835 [2021-07-09 00:29:46,183][root][INFO] - ===> Epoch[9302](46510/5): Loss 0.0080, Sem 0.0080 LR: 2.610e-02 Score 99.713 Data time: 0.2160, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.6608 [2021-07-09 00:30:00,238][root][INFO] - ===> Epoch[9303](46515/5): Loss 0.0076, Sem 0.0076 LR: 2.609e-02 Score 99.727 Data time: 0.1850, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.7556 [2021-07-09 00:30:14,476][root][INFO] - ===> Epoch[9304](46520/5): Loss 0.0079, Sem 0.0079 LR: 2.609e-02 Score 99.713 Data time: 0.2167, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.8055 [2021-07-09 00:30:28,470][root][INFO] - ===> Epoch[9305](46525/5): Loss 0.0078, Sem 0.0078 LR: 2.608e-02 Score 99.719 Data time: 0.2044, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.7494 [2021-07-09 00:30:42,923][root][INFO] - ===> Epoch[9306](46530/5): Loss 0.0077, Sem 0.0077 LR: 2.607e-02 Score 99.725 Data time: 0.2147, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.8361 [2021-07-09 00:30:56,770][root][INFO] - ===> Epoch[9307](46535/5): Loss 0.0079, Sem 0.0079 LR: 2.606e-02 Score 99.718 Data time: 0.2660, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.7179 [2021-07-09 00:31:09,695][root][INFO] - ===> Epoch[9308](46540/5): Loss 0.0080, Sem 0.0080 LR: 2.605e-02 Score 99.713 Data time: 0.1872, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.5477 [2021-07-09 00:31:23,769][root][INFO] - ===> Epoch[9309](46545/5): Loss 0.0079, Sem 0.0079 LR: 2.604e-02 Score 99.714 Data time: 0.2039, Forward time: 1.2535, Backward time: 1.2460, DDP time: 0.0042, Total iter time: 2.7673 [2021-07-09 00:31:37,731][root][INFO] - ===> Epoch[9310](46550/5): Loss 0.0075, Sem 0.0075 LR: 2.603e-02 Score 99.730 Data time: 0.1237, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.7576 [2021-07-09 00:31:52,145][root][INFO] - ===> Epoch[9311](46555/5): Loss 0.0082, Sem 0.0082 LR: 2.603e-02 Score 99.706 Data time: 0.2020, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.8383 [2021-07-09 00:32:06,174][root][INFO] - ===> Epoch[9312](46560/5): Loss 0.0075, Sem 0.0075 LR: 2.602e-02 Score 99.729 Data time: 0.1982, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.7549 [2021-07-09 00:32:20,060][root][INFO] - ===> Epoch[9313](46565/5): Loss 0.0081, Sem 0.0081 LR: 2.601e-02 Score 99.711 Data time: 0.1650, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.7384 [2021-07-09 00:32:33,844][root][INFO] - ===> Epoch[9314](46570/5): Loss 0.0080, Sem 0.0080 LR: 2.600e-02 Score 99.714 Data time: 0.2049, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.7022 [2021-07-09 00:32:48,483][root][INFO] - ===> Epoch[9315](46575/5): Loss 0.0079, Sem 0.0079 LR: 2.599e-02 Score 99.716 Data time: 0.1900, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.8887 [2021-07-09 00:33:02,779][root][INFO] - ===> Epoch[9316](46580/5): Loss 0.0079, Sem 0.0079 LR: 2.598e-02 Score 99.714 Data time: 0.2628, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.8111 [2021-07-09 00:33:17,316][root][INFO] - ===> Epoch[9317](46585/5): Loss 0.0075, Sem 0.0075 LR: 2.597e-02 Score 99.729 Data time: 0.2059, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.8647 [2021-07-09 00:33:31,157][root][INFO] - ===> Epoch[9318](46590/5): Loss 0.0075, Sem 0.0075 LR: 2.596e-02 Score 99.732 Data time: 0.2456, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.7179 [2021-07-09 00:33:44,712][root][INFO] - ===> Epoch[9319](46595/5): Loss 0.0080, Sem 0.0080 LR: 2.596e-02 Score 99.715 Data time: 0.1707, Forward time: 1.2535, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.6714 [2021-07-09 00:33:58,622][root][INFO] - ===> Epoch[9320](46600/5): Loss 0.0081, Sem 0.0081 LR: 2.595e-02 Score 99.710 Data time: 0.1603, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.7383 [2021-07-09 00:33:58,624][root][INFO] - ===> Start testing [2021-07-09 00:34:36,486][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1206 Loss 0.387 (AVG: 0.613) Score 92.119 (AVG: 91.046) mIOU 71.664 mAP 83.001 mAcc 77.839 IOU: 64.368 0.000 80.405 74.778 95.237 92.223 33.086 84.115 98.639 85.488 78.960 86.315 58.011 mAP: 82.791 3.040 89.957 85.910 98.507 98.182 54.486 97.757 99.702 94.701 90.525 93.057 90.396 mAcc: 79.841 0.000 84.327 82.270 96.917 97.290 36.559 94.947 99.485 91.713 90.559 96.402 61.602 [2021-07-09 00:34:36,486][root][INFO] - Finished test. Elapsed time: 37.8612 [2021-07-09 00:34:37,590][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 00:34:50,406][root][INFO] - ===> Epoch[9321](46605/5): Loss 0.0079, Sem 0.0079 LR: 2.594e-02 Score 99.715 Data time: 0.1980, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.5227 [2021-07-09 00:35:04,159][root][INFO] - ===> Epoch[9322](46610/5): Loss 0.0077, Sem 0.0077 LR: 2.593e-02 Score 99.721 Data time: 0.2270, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.7129 [2021-07-09 00:35:18,373][root][INFO] - ===> Epoch[9323](46615/5): Loss 0.0080, Sem 0.0080 LR: 2.592e-02 Score 99.716 Data time: 0.2094, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.7987 [2021-07-09 00:35:32,524][root][INFO] - ===> Epoch[9324](46620/5): Loss 0.0078, Sem 0.0078 LR: 2.591e-02 Score 99.724 Data time: 0.2360, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.7861 [2021-07-09 00:35:45,774][root][INFO] - ===> Epoch[9325](46625/5): Loss 0.0080, Sem 0.0080 LR: 2.590e-02 Score 99.713 Data time: 0.1811, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.6028 [2021-07-09 00:36:00,344][root][INFO] - ===> Epoch[9326](46630/5): Loss 0.0082, Sem 0.0082 LR: 2.589e-02 Score 99.709 Data time: 0.2561, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.8721 [2021-07-09 00:36:14,282][root][INFO] - ===> Epoch[9327](46635/5): Loss 0.0077, Sem 0.0077 LR: 2.589e-02 Score 99.723 Data time: 0.2953, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.7370 [2021-07-09 00:36:28,911][root][INFO] - ===> Epoch[9328](46640/5): Loss 0.0077, Sem 0.0077 LR: 2.588e-02 Score 99.721 Data time: 0.1757, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.8956 [2021-07-09 00:36:42,821][root][INFO] - ===> Epoch[9329](46645/5): Loss 0.0077, Sem 0.0077 LR: 2.587e-02 Score 99.722 Data time: 0.2237, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.7390 [2021-07-09 00:36:57,091][root][INFO] - ===> Epoch[9330](46650/5): Loss 0.0077, Sem 0.0077 LR: 2.586e-02 Score 99.726 Data time: 0.2830, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.8114 [2021-07-09 00:37:11,457][root][INFO] - ===> Epoch[9331](46655/5): Loss 0.0078, Sem 0.0078 LR: 2.585e-02 Score 99.720 Data time: 0.2626, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.8099 [2021-07-09 00:37:25,891][root][INFO] - ===> Epoch[9332](46660/5): Loss 0.0083, Sem 0.0083 LR: 2.584e-02 Score 99.701 Data time: 0.2506, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.8420 [2021-07-09 00:37:39,980][root][INFO] - ===> Epoch[9333](46665/5): Loss 0.0075, Sem 0.0075 LR: 2.583e-02 Score 99.729 Data time: 0.2681, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.7723 [2021-07-09 00:37:53,284][root][INFO] - ===> Epoch[9334](46670/5): Loss 0.0081, Sem 0.0081 LR: 2.582e-02 Score 99.709 Data time: 0.2334, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.6044 [2021-07-09 00:38:07,676][root][INFO] - ===> Epoch[9335](46675/5): Loss 0.0075, Sem 0.0075 LR: 2.582e-02 Score 99.728 Data time: 0.2109, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.8452 [2021-07-09 00:38:21,014][root][INFO] - ===> Epoch[9336](46680/5): Loss 0.0078, Sem 0.0078 LR: 2.581e-02 Score 99.718 Data time: 0.2497, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.6247 [2021-07-09 00:38:35,357][root][INFO] - ===> Epoch[9337](46685/5): Loss 0.0078, Sem 0.0078 LR: 2.580e-02 Score 99.719 Data time: 0.2547, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.8125 [2021-07-09 00:38:49,100][root][INFO] - ===> Epoch[9338](46690/5): Loss 0.0078, Sem 0.0078 LR: 2.579e-02 Score 99.721 Data time: 0.2791, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.7037 [2021-07-09 00:39:03,314][root][INFO] - ===> Epoch[9339](46695/5): Loss 0.0076, Sem 0.0076 LR: 2.578e-02 Score 99.727 Data time: 0.2148, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.8050 [2021-07-09 00:39:18,089][root][INFO] - ===> Epoch[9340](46700/5): Loss 0.0075, Sem 0.0075 LR: 2.577e-02 Score 99.729 Data time: 0.2238, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.9080 [2021-07-09 00:39:31,481][root][INFO] - ===> Epoch[9341](46705/5): Loss 0.0080, Sem 0.0080 LR: 2.576e-02 Score 99.714 Data time: 0.2338, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.6387 [2021-07-09 00:39:45,202][root][INFO] - ===> Epoch[9342](46710/5): Loss 0.0073, Sem 0.0073 LR: 2.575e-02 Score 99.738 Data time: 0.2484, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.6953 [2021-07-09 00:39:59,325][root][INFO] - ===> Epoch[9343](46715/5): Loss 0.0078, Sem 0.0078 LR: 2.575e-02 Score 99.719 Data time: 0.2504, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.7778 [2021-07-09 00:40:13,531][root][INFO] - ===> Epoch[9344](46720/5): Loss 0.0074, Sem 0.0074 LR: 2.574e-02 Score 99.732 Data time: 0.1928, Forward time: 1.2534, Backward time: 1.2461, DDP time: 0.0042, Total iter time: 2.7942 [2021-07-09 00:40:28,144][root][INFO] - ===> Epoch[9345](46725/5): Loss 0.0076, Sem 0.0076 LR: 2.573e-02 Score 99.731 Data time: 0.1849, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.8840 [2021-07-09 00:40:42,199][root][INFO] - ===> Epoch[9346](46730/5): Loss 0.0077, Sem 0.0077 LR: 2.572e-02 Score 99.726 Data time: 0.1958, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.7653 [2021-07-09 00:40:55,903][root][INFO] - ===> Epoch[9347](46735/5): Loss 0.0079, Sem 0.0079 LR: 2.571e-02 Score 99.714 Data time: 0.2627, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.6810 [2021-07-09 00:41:10,061][root][INFO] - ===> Epoch[9348](46740/5): Loss 0.0079, Sem 0.0079 LR: 2.570e-02 Score 99.718 Data time: 0.2240, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.7943 [2021-07-09 00:41:24,652][root][INFO] - ===> Epoch[9349](46745/5): Loss 0.0075, Sem 0.0075 LR: 2.569e-02 Score 99.729 Data time: 0.1793, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.8790 [2021-07-09 00:41:38,382][root][INFO] - ===> Epoch[9350](46750/5): Loss 0.0079, Sem 0.0079 LR: 2.569e-02 Score 99.717 Data time: 0.1948, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.7122 [2021-07-09 00:41:51,986][root][INFO] - ===> Epoch[9351](46755/5): Loss 0.0083, Sem 0.0083 LR: 2.568e-02 Score 99.704 Data time: 0.2272, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.6774 [2021-07-09 00:42:06,539][root][INFO] - ===> Epoch[9352](46760/5): Loss 0.0078, Sem 0.0078 LR: 2.567e-02 Score 99.722 Data time: 0.1849, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.8728 [2021-07-09 00:42:20,882][root][INFO] - ===> Epoch[9353](46765/5): Loss 0.0077, Sem 0.0077 LR: 2.566e-02 Score 99.723 Data time: 0.2400, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.8183 [2021-07-09 00:42:34,625][root][INFO] - ===> Epoch[9354](46770/5): Loss 0.0084, Sem 0.0084 LR: 2.565e-02 Score 99.704 Data time: 0.2445, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.7050 [2021-07-09 00:42:48,806][root][INFO] - ===> Epoch[9355](46775/5): Loss 0.0080, Sem 0.0080 LR: 2.564e-02 Score 99.711 Data time: 0.2420, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.7880 [2021-07-09 00:43:03,312][root][INFO] - ===> Epoch[9356](46780/5): Loss 0.0075, Sem 0.0075 LR: 2.563e-02 Score 99.731 Data time: 0.2120, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.8629 [2021-07-09 00:43:17,929][root][INFO] - ===> Epoch[9357](46785/5): Loss 0.0083, Sem 0.0083 LR: 2.562e-02 Score 99.703 Data time: 0.2609, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.8746 [2021-07-09 00:43:31,037][root][INFO] - ===> Epoch[9358](46790/5): Loss 0.0079, Sem 0.0079 LR: 2.562e-02 Score 99.720 Data time: 0.1987, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.5698 [2021-07-09 00:43:45,106][root][INFO] - ===> Epoch[9359](46795/5): Loss 0.0077, Sem 0.0077 LR: 2.561e-02 Score 99.724 Data time: 0.2080, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.7708 [2021-07-09 00:43:58,803][root][INFO] - ===> Epoch[9360](46800/5): Loss 0.0080, Sem 0.0080 LR: 2.560e-02 Score 99.714 Data time: 0.1938, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.6894 [2021-07-09 00:43:58,804][root][INFO] - ===> Start testing [2021-07-09 00:44:40,552][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1278 Loss 0.341 (AVG: 0.607) Score 93.215 (AVG: 91.040) mIOU 71.365 mAP 82.608 mAcc 77.547 IOU: 64.765 0.000 80.990 74.697 95.195 91.928 31.682 83.819 98.502 83.495 78.476 86.252 57.944 mAP: 82.979 4.047 87.814 85.841 98.659 98.149 52.315 97.701 99.744 93.103 90.544 92.744 90.268 mAcc: 80.378 0.000 83.517 81.589 96.846 97.470 33.991 94.612 99.268 91.683 90.724 96.899 61.134 [2021-07-09 00:44:40,552][root][INFO] - Finished test. Elapsed time: 41.7472 [2021-07-09 00:44:41,718][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 00:44:54,320][root][INFO] - ===> Epoch[9361](46805/5): Loss 0.0078, Sem 0.0078 LR: 2.559e-02 Score 99.720 Data time: 0.1672, Forward time: 1.2533, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.4783 [2021-07-09 00:45:08,793][root][INFO] - ===> Epoch[9362](46810/5): Loss 0.0076, Sem 0.0076 LR: 2.558e-02 Score 99.728 Data time: 0.2402, Forward time: 1.2534, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.8430 [2021-07-09 00:45:22,456][root][INFO] - ===> Epoch[9363](46815/5): Loss 0.0076, Sem 0.0076 LR: 2.557e-02 Score 99.725 Data time: 0.2116, Forward time: 1.2533, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.6847 [2021-07-09 00:45:37,264][root][INFO] - ===> Epoch[9364](46820/5): Loss 0.0081, Sem 0.0081 LR: 2.556e-02 Score 99.711 Data time: 0.2511, Forward time: 1.2533, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.9166 [2021-07-09 00:45:51,068][root][INFO] - ===> Epoch[9365](46825/5): Loss 0.0075, Sem 0.0075 LR: 2.555e-02 Score 99.730 Data time: 0.2204, Forward time: 1.2533, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.7194 [2021-07-09 00:46:05,147][root][INFO] - ===> Epoch[9366](46830/5): Loss 0.0075, Sem 0.0075 LR: 2.555e-02 Score 99.731 Data time: 0.2363, Forward time: 1.2533, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.7673 [2021-07-09 00:46:18,816][root][INFO] - ===> Epoch[9367](46835/5): Loss 0.0079, Sem 0.0079 LR: 2.554e-02 Score 99.717 Data time: 0.1724, Forward time: 1.2533, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.6862 [2021-07-09 00:46:32,857][root][INFO] - ===> Epoch[9368](46840/5): Loss 0.0081, Sem 0.0081 LR: 2.553e-02 Score 99.712 Data time: 0.2208, Forward time: 1.2533, Backward time: 1.2463, DDP time: 0.0042, Total iter time: 2.7683 [2021-07-09 00:46:46,244][root][INFO] - ===> Epoch[9369](46845/5): Loss 0.0077, Sem 0.0077 LR: 2.552e-02 Score 99.724 Data time: 0.1986, Forward time: 1.2533, Backward time: 1.2463, DDP time: 0.0042, Total iter time: 2.6342 [2021-07-09 00:46:59,477][root][INFO] - ===> Epoch[9370](46850/5): Loss 0.0080, Sem 0.0080 LR: 2.551e-02 Score 99.716 Data time: 0.1979, Forward time: 1.2533, Backward time: 1.2463, DDP time: 0.0042, Total iter time: 2.6085 [2021-07-09 00:47:13,163][root][INFO] - ===> Epoch[9371](46855/5): Loss 0.0082, Sem 0.0082 LR: 2.550e-02 Score 99.702 Data time: 0.2376, Forward time: 1.2533, Backward time: 1.2462, DDP time: 0.0042, Total iter time: 2.6871 [2021-07-09 00:47:27,022][root][INFO] - ===> Epoch[9372](46860/5): Loss 0.0081, Sem 0.0081 LR: 2.549e-02 Score 99.707 Data time: 0.1672, Forward time: 1.2533, Backward time: 1.2463, DDP time: 0.0042, Total iter time: 2.7345 [2021-07-09 00:47:42,082][root][INFO] - ===> Epoch[9373](46865/5): Loss 0.0079, Sem 0.0079 LR: 2.548e-02 Score 99.716 Data time: 0.3185, Forward time: 1.2533, Backward time: 1.2463, DDP time: 0.0042, Total iter time: 2.9599 [2021-07-09 00:47:56,345][root][INFO] - ===> Epoch[9374](46870/5): Loss 0.0079, Sem 0.0079 LR: 2.548e-02 Score 99.718 Data time: 0.2686, Forward time: 1.2533, Backward time: 1.2463, DDP time: 0.0042, Total iter time: 2.8096 [2021-07-09 00:48:12,175][root][INFO] - ===> Epoch[9375](46875/5): Loss 0.0077, Sem 0.0077 LR: 2.547e-02 Score 99.724 Data time: 0.2969, Forward time: 1.2533, Backward time: 1.2463, DDP time: 0.0042, Total iter time: 3.1297 [2021-07-09 00:48:27,007][root][INFO] - ===> Epoch[9376](46880/5): Loss 0.0079, Sem 0.0079 LR: 2.546e-02 Score 99.718 Data time: 0.2775, Forward time: 1.2533, Backward time: 1.2463, DDP time: 0.0042, Total iter time: 2.9214 [2021-07-09 00:48:40,496][root][INFO] - ===> Epoch[9377](46885/5): Loss 0.0083, Sem 0.0083 LR: 2.545e-02 Score 99.706 Data time: 0.2507, Forward time: 1.2533, Backward time: 1.2463, DDP time: 0.0042, Total iter time: 2.6559 [2021-07-09 00:48:54,665][root][INFO] - ===> Epoch[9378](46890/5): Loss 0.0075, Sem 0.0075 LR: 2.544e-02 Score 99.733 Data time: 0.3067, Forward time: 1.2533, Backward time: 1.2463, DDP time: 0.0042, Total iter time: 2.7871 [2021-07-09 00:49:08,614][root][INFO] - ===> Epoch[9379](46895/5): Loss 0.0080, Sem 0.0080 LR: 2.543e-02 Score 99.710 Data time: 0.2435, Forward time: 1.2533, Backward time: 1.2463, DDP time: 0.0042, Total iter time: 2.7370 [2021-07-09 00:49:22,876][root][INFO] - ===> Epoch[9380](46900/5): Loss 0.0074, Sem 0.0074 LR: 2.542e-02 Score 99.733 Data time: 0.2143, Forward time: 1.2533, Backward time: 1.2463, DDP time: 0.0042, Total iter time: 2.8059 [2021-07-09 00:49:36,400][root][INFO] - ===> Epoch[9381](46905/5): Loss 0.0078, Sem 0.0078 LR: 2.541e-02 Score 99.720 Data time: 0.2222, Forward time: 1.2533, Backward time: 1.2463, DDP time: 0.0042, Total iter time: 2.6585 [2021-07-09 00:49:50,962][root][INFO] - ===> Epoch[9382](46910/5): Loss 0.0077, Sem 0.0077 LR: 2.541e-02 Score 99.726 Data time: 0.2034, Forward time: 1.2533, Backward time: 1.2463, DDP time: 0.0042, Total iter time: 2.8639 [2021-07-09 00:50:04,483][root][INFO] - ===> Epoch[9383](46915/5): Loss 0.0077, Sem 0.0077 LR: 2.540e-02 Score 99.721 Data time: 0.1688, Forward time: 1.2533, Backward time: 1.2463, DDP time: 0.0042, Total iter time: 2.6652 [2021-07-09 00:50:19,248][root][INFO] - ===> Epoch[9384](46920/5): Loss 0.0072, Sem 0.0072 LR: 2.539e-02 Score 99.743 Data time: 0.2026, Forward time: 1.2533, Backward time: 1.2463, DDP time: 0.0042, Total iter time: 2.9090 [2021-07-09 00:50:33,379][root][INFO] - ===> Epoch[9385](46925/5): Loss 0.0080, Sem 0.0080 LR: 2.538e-02 Score 99.715 Data time: 0.1838, Forward time: 1.2533, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7911 [2021-07-09 00:50:47,037][root][INFO] - ===> Epoch[9386](46930/5): Loss 0.0077, Sem 0.0077 LR: 2.537e-02 Score 99.725 Data time: 0.2022, Forward time: 1.2533, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.6939 [2021-07-09 00:51:02,034][root][INFO] - ===> Epoch[9387](46935/5): Loss 0.0076, Sem 0.0076 LR: 2.536e-02 Score 99.724 Data time: 0.2332, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.9508 [2021-07-09 00:51:15,650][root][INFO] - ===> Epoch[9388](46940/5): Loss 0.0080, Sem 0.0080 LR: 2.535e-02 Score 99.715 Data time: 0.2072, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.6662 [2021-07-09 00:51:29,503][root][INFO] - ===> Epoch[9389](46945/5): Loss 0.0078, Sem 0.0078 LR: 2.534e-02 Score 99.722 Data time: 0.2487, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7280 [2021-07-09 00:51:43,862][root][INFO] - ===> Epoch[9390](46950/5): Loss 0.0078, Sem 0.0078 LR: 2.534e-02 Score 99.719 Data time: 0.2410, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8309 [2021-07-09 00:51:57,226][root][INFO] - ===> Epoch[9391](46955/5): Loss 0.0078, Sem 0.0078 LR: 2.533e-02 Score 99.720 Data time: 0.1879, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.6343 [2021-07-09 00:52:11,406][root][INFO] - ===> Epoch[9392](46960/5): Loss 0.0076, Sem 0.0076 LR: 2.532e-02 Score 99.729 Data time: 0.1759, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7912 [2021-07-09 00:52:24,590][root][INFO] - ===> Epoch[9393](46965/5): Loss 0.0081, Sem 0.0081 LR: 2.531e-02 Score 99.710 Data time: 0.2234, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.5986 [2021-07-09 00:52:38,086][root][INFO] - ===> Epoch[9394](46970/5): Loss 0.0079, Sem 0.0079 LR: 2.530e-02 Score 99.716 Data time: 0.2206, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.6546 [2021-07-09 00:52:52,419][root][INFO] - ===> Epoch[9395](46975/5): Loss 0.0077, Sem 0.0077 LR: 2.529e-02 Score 99.722 Data time: 0.2389, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8245 [2021-07-09 00:53:06,545][root][INFO] - ===> Epoch[9396](46980/5): Loss 0.0075, Sem 0.0075 LR: 2.528e-02 Score 99.731 Data time: 0.2631, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7731 [2021-07-09 00:53:20,761][root][INFO] - ===> Epoch[9397](46985/5): Loss 0.0081, Sem 0.0081 LR: 2.527e-02 Score 99.709 Data time: 0.2105, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8009 [2021-07-09 00:53:34,172][root][INFO] - ===> Epoch[9398](46990/5): Loss 0.0078, Sem 0.0078 LR: 2.527e-02 Score 99.719 Data time: 0.2216, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.6135 [2021-07-09 00:53:48,794][root][INFO] - ===> Epoch[9399](46995/5): Loss 0.0079, Sem 0.0079 LR: 2.526e-02 Score 99.718 Data time: 0.2012, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8755 [2021-07-09 00:54:02,737][root][INFO] - ===> Epoch[9400](47000/5): Loss 0.0079, Sem 0.0079 LR: 2.525e-02 Score 99.715 Data time: 0.2467, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7418 [2021-07-09 00:54:02,740][root][INFO] - ===> Start testing [2021-07-09 00:54:41,157][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1234 Loss 0.385 (AVG: 0.602) Score 92.103 (AVG: 91.153) mIOU 71.836 mAP 83.056 mAcc 78.076 IOU: 64.438 0.000 81.397 75.748 95.079 91.881 32.974 83.402 98.663 85.451 80.422 86.392 58.015 mAP: 82.925 2.636 89.736 86.529 98.445 98.087 54.972 97.625 99.709 94.329 90.931 93.102 90.704 mAcc: 79.857 0.000 84.333 83.404 96.989 97.448 36.524 94.605 99.432 93.696 90.629 96.267 61.809 [2021-07-09 00:54:41,157][root][INFO] - Finished test. Elapsed time: 38.4159 [2021-07-09 00:54:42,228][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 00:54:55,615][root][INFO] - ===> Epoch[9401](47005/5): Loss 0.0076, Sem 0.0076 LR: 2.524e-02 Score 99.727 Data time: 0.2118, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.6420 [2021-07-09 00:55:08,481][root][INFO] - ===> Epoch[9402](47010/5): Loss 0.0080, Sem 0.0080 LR: 2.523e-02 Score 99.712 Data time: 0.2147, Forward time: 1.2532, Backward time: 1.2463, DDP time: 0.0042, Total iter time: 2.5291 [2021-07-09 00:55:22,955][root][INFO] - ===> Epoch[9403](47015/5): Loss 0.0078, Sem 0.0078 LR: 2.522e-02 Score 99.721 Data time: 0.1730, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8430 [2021-07-09 00:55:37,034][root][INFO] - ===> Epoch[9404](47020/5): Loss 0.0074, Sem 0.0074 LR: 2.521e-02 Score 99.729 Data time: 0.2218, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7671 [2021-07-09 00:55:50,985][root][INFO] - ===> Epoch[9405](47025/5): Loss 0.0079, Sem 0.0079 LR: 2.520e-02 Score 99.718 Data time: 0.2253, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7443 [2021-07-09 00:56:04,783][root][INFO] - ===> Epoch[9406](47030/5): Loss 0.0078, Sem 0.0078 LR: 2.520e-02 Score 99.722 Data time: 0.2465, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7181 [2021-07-09 00:56:19,257][root][INFO] - ===> Epoch[9407](47035/5): Loss 0.0078, Sem 0.0078 LR: 2.519e-02 Score 99.721 Data time: 0.2686, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8473 [2021-07-09 00:56:33,325][root][INFO] - ===> Epoch[9408](47040/5): Loss 0.0076, Sem 0.0076 LR: 2.518e-02 Score 99.730 Data time: 0.1790, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7687 [2021-07-09 00:56:48,215][root][INFO] - ===> Epoch[9409](47045/5): Loss 0.0081, Sem 0.0081 LR: 2.517e-02 Score 99.707 Data time: 0.2200, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.9301 [2021-07-09 00:57:02,180][root][INFO] - ===> Epoch[9410](47050/5): Loss 0.0076, Sem 0.0076 LR: 2.516e-02 Score 99.728 Data time: 0.2300, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7517 [2021-07-09 00:57:16,931][root][INFO] - ===> Epoch[9411](47055/5): Loss 0.0075, Sem 0.0075 LR: 2.515e-02 Score 99.732 Data time: 0.2392, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8910 [2021-07-09 00:57:31,145][root][INFO] - ===> Epoch[9412](47060/5): Loss 0.0079, Sem 0.0079 LR: 2.514e-02 Score 99.715 Data time: 0.2670, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7997 [2021-07-09 00:57:44,796][root][INFO] - ===> Epoch[9413](47065/5): Loss 0.0082, Sem 0.0082 LR: 2.513e-02 Score 99.705 Data time: 0.2120, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.6918 [2021-07-09 00:57:57,780][root][INFO] - ===> Epoch[9414](47070/5): Loss 0.0077, Sem 0.0077 LR: 2.513e-02 Score 99.723 Data time: 0.1795, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.5529 [2021-07-09 00:58:12,122][root][INFO] - ===> Epoch[9415](47075/5): Loss 0.0076, Sem 0.0076 LR: 2.512e-02 Score 99.726 Data time: 0.2150, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8262 [2021-07-09 00:58:26,937][root][INFO] - ===> Epoch[9416](47080/5): Loss 0.0078, Sem 0.0078 LR: 2.511e-02 Score 99.721 Data time: 0.2514, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.9242 [2021-07-09 00:58:41,014][root][INFO] - ===> Epoch[9417](47085/5): Loss 0.0078, Sem 0.0078 LR: 2.510e-02 Score 99.728 Data time: 0.1948, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7698 [2021-07-09 00:58:54,238][root][INFO] - ===> Epoch[9418](47090/5): Loss 0.0086, Sem 0.0086 LR: 2.509e-02 Score 99.693 Data time: 0.2053, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.6067 [2021-07-09 00:59:08,977][root][INFO] - ===> Epoch[9419](47095/5): Loss 0.0084, Sem 0.0084 LR: 2.508e-02 Score 99.706 Data time: 0.2266, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8997 [2021-07-09 00:59:23,472][root][INFO] - ===> Epoch[9420](47100/5): Loss 0.0081, Sem 0.0081 LR: 2.507e-02 Score 99.711 Data time: 0.2049, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8586 [2021-07-09 00:59:37,579][root][INFO] - ===> Epoch[9421](47105/5): Loss 0.0082, Sem 0.0082 LR: 2.506e-02 Score 99.706 Data time: 0.2593, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7811 [2021-07-09 00:59:51,662][root][INFO] - ===> Epoch[9422](47110/5): Loss 0.0082, Sem 0.0082 LR: 2.506e-02 Score 99.709 Data time: 0.3007, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7683 [2021-07-09 01:00:05,342][root][INFO] - ===> Epoch[9423](47115/5): Loss 0.0083, Sem 0.0083 LR: 2.505e-02 Score 99.701 Data time: 0.2437, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.6802 [2021-07-09 01:00:20,072][root][INFO] - ===> Epoch[9424](47120/5): Loss 0.0079, Sem 0.0079 LR: 2.504e-02 Score 99.715 Data time: 0.2774, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8937 [2021-07-09 01:00:34,579][root][INFO] - ===> Epoch[9425](47125/5): Loss 0.0079, Sem 0.0079 LR: 2.503e-02 Score 99.715 Data time: 0.3299, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8616 [2021-07-09 01:00:48,747][root][INFO] - ===> Epoch[9426](47130/5): Loss 0.0078, Sem 0.0078 LR: 2.502e-02 Score 99.721 Data time: 0.2684, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7894 [2021-07-09 01:01:03,570][root][INFO] - ===> Epoch[9427](47135/5): Loss 0.0081, Sem 0.0081 LR: 2.501e-02 Score 99.708 Data time: 0.2666, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.9215 [2021-07-09 01:01:17,413][root][INFO] - ===> Epoch[9428](47140/5): Loss 0.0083, Sem 0.0083 LR: 2.500e-02 Score 99.706 Data time: 0.2748, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7311 [2021-07-09 01:01:30,922][root][INFO] - ===> Epoch[9429](47145/5): Loss 0.0076, Sem 0.0076 LR: 2.499e-02 Score 99.725 Data time: 0.2656, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.6595 [2021-07-09 01:01:44,761][root][INFO] - ===> Epoch[9430](47150/5): Loss 0.0075, Sem 0.0075 LR: 2.499e-02 Score 99.730 Data time: 0.1994, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7197 [2021-07-09 01:01:59,147][root][INFO] - ===> Epoch[9431](47155/5): Loss 0.0076, Sem 0.0076 LR: 2.498e-02 Score 99.728 Data time: 0.2691, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8363 [2021-07-09 01:02:14,047][root][INFO] - ===> Epoch[9432](47160/5): Loss 0.0076, Sem 0.0076 LR: 2.497e-02 Score 99.725 Data time: 0.2783, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.9213 [2021-07-09 01:02:27,729][root][INFO] - ===> Epoch[9433](47165/5): Loss 0.0078, Sem 0.0078 LR: 2.496e-02 Score 99.721 Data time: 0.2396, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.6839 [2021-07-09 01:02:42,696][root][INFO] - ===> Epoch[9434](47170/5): Loss 0.0077, Sem 0.0077 LR: 2.495e-02 Score 99.724 Data time: 0.2423, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.9383 [2021-07-09 01:02:56,811][root][INFO] - ===> Epoch[9435](47175/5): Loss 0.0080, Sem 0.0080 LR: 2.494e-02 Score 99.717 Data time: 0.2174, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7703 [2021-07-09 01:03:11,036][root][INFO] - ===> Epoch[9436](47180/5): Loss 0.0080, Sem 0.0080 LR: 2.493e-02 Score 99.710 Data time: 0.2300, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8108 [2021-07-09 01:03:25,528][root][INFO] - ===> Epoch[9437](47185/5): Loss 0.0077, Sem 0.0077 LR: 2.492e-02 Score 99.724 Data time: 0.2028, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8543 [2021-07-09 01:03:39,367][root][INFO] - ===> Epoch[9438](47190/5): Loss 0.0083, Sem 0.0083 LR: 2.492e-02 Score 99.705 Data time: 0.2215, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7268 [2021-07-09 01:03:53,109][root][INFO] - ===> Epoch[9439](47195/5): Loss 0.0077, Sem 0.0077 LR: 2.491e-02 Score 99.724 Data time: 0.2320, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6982 [2021-07-09 01:04:06,717][root][INFO] - ===> Epoch[9440](47200/5): Loss 0.0078, Sem 0.0078 LR: 2.490e-02 Score 99.721 Data time: 0.2346, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6777 [2021-07-09 01:04:06,719][root][INFO] - ===> Start testing [2021-07-09 01:04:44,300][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1586 Loss 0.335 (AVG: 0.613) Score 93.510 (AVG: 91.066) mIOU 71.733 mAP 82.934 mAcc 77.715 IOU: 64.358 0.000 81.691 74.853 95.027 92.025 32.125 86.544 98.584 83.574 80.203 86.057 57.483 mAP: 83.082 2.864 89.895 86.104 98.443 98.242 53.323 97.900 99.749 95.399 90.737 92.838 89.563 mAcc: 79.851 0.000 84.381 82.622 96.911 97.322 34.776 95.211 99.281 92.107 90.528 96.648 60.661 [2021-07-09 01:04:44,301][root][INFO] - Finished test. Elapsed time: 37.5808 [2021-07-09 01:04:45,530][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 01:04:58,621][root][INFO] - ===> Epoch[9441](47205/5): Loss 0.0078, Sem 0.0078 LR: 2.489e-02 Score 99.722 Data time: 0.1633, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.5835 [2021-07-09 01:05:11,965][root][INFO] - ===> Epoch[9442](47210/5): Loss 0.0079, Sem 0.0079 LR: 2.488e-02 Score 99.716 Data time: 0.2390, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6249 [2021-07-09 01:05:26,191][root][INFO] - ===> Epoch[9443](47215/5): Loss 0.0074, Sem 0.0074 LR: 2.487e-02 Score 99.735 Data time: 0.2364, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7988 [2021-07-09 01:05:40,199][root][INFO] - ===> Epoch[9444](47220/5): Loss 0.0081, Sem 0.0081 LR: 2.486e-02 Score 99.708 Data time: 0.2454, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7588 [2021-07-09 01:05:54,097][root][INFO] - ===> Epoch[9445](47225/5): Loss 0.0077, Sem 0.0077 LR: 2.485e-02 Score 99.724 Data time: 0.2238, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7362 [2021-07-09 01:06:08,327][root][INFO] - ===> Epoch[9446](47230/5): Loss 0.0078, Sem 0.0078 LR: 2.485e-02 Score 99.718 Data time: 0.2343, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7976 [2021-07-09 01:06:22,007][root][INFO] - ===> Epoch[9447](47235/5): Loss 0.0078, Sem 0.0078 LR: 2.484e-02 Score 99.720 Data time: 0.1901, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6910 [2021-07-09 01:06:37,067][root][INFO] - ===> Epoch[9448](47240/5): Loss 0.0075, Sem 0.0075 LR: 2.483e-02 Score 99.730 Data time: 0.2140, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.9591 [2021-07-09 01:06:50,489][root][INFO] - ===> Epoch[9449](47245/5): Loss 0.0080, Sem 0.0080 LR: 2.482e-02 Score 99.712 Data time: 0.1939, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6264 [2021-07-09 01:07:04,551][root][INFO] - ===> Epoch[9450](47250/5): Loss 0.0074, Sem 0.0074 LR: 2.481e-02 Score 99.736 Data time: 0.2277, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7611 [2021-07-09 01:07:18,390][root][INFO] - ===> Epoch[9451](47255/5): Loss 0.0077, Sem 0.0077 LR: 2.480e-02 Score 99.725 Data time: 0.2070, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7237 [2021-07-09 01:07:32,692][root][INFO] - ===> Epoch[9452](47260/5): Loss 0.0076, Sem 0.0076 LR: 2.479e-02 Score 99.726 Data time: 0.2220, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8153 [2021-07-09 01:07:46,623][root][INFO] - ===> Epoch[9453](47265/5): Loss 0.0079, Sem 0.0079 LR: 2.478e-02 Score 99.716 Data time: 0.2214, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7298 [2021-07-09 01:08:00,139][root][INFO] - ===> Epoch[9454](47270/5): Loss 0.0074, Sem 0.0074 LR: 2.478e-02 Score 99.736 Data time: 0.2129, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6543 [2021-07-09 01:08:14,030][root][INFO] - ===> Epoch[9455](47275/5): Loss 0.0075, Sem 0.0075 LR: 2.477e-02 Score 99.730 Data time: 0.2049, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7263 [2021-07-09 01:08:28,114][root][INFO] - ===> Epoch[9456](47280/5): Loss 0.0079, Sem 0.0079 LR: 2.476e-02 Score 99.719 Data time: 0.1661, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7757 [2021-07-09 01:08:42,283][root][INFO] - ===> Epoch[9457](47285/5): Loss 0.0073, Sem 0.0073 LR: 2.475e-02 Score 99.740 Data time: 0.2296, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7924 [2021-07-09 01:08:55,727][root][INFO] - ===> Epoch[9458](47290/5): Loss 0.0077, Sem 0.0077 LR: 2.474e-02 Score 99.721 Data time: 0.2062, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6380 [2021-07-09 01:09:09,773][root][INFO] - ===> Epoch[9459](47295/5): Loss 0.0075, Sem 0.0075 LR: 2.473e-02 Score 99.728 Data time: 0.1987, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7747 [2021-07-09 01:09:23,992][root][INFO] - ===> Epoch[9460](47300/5): Loss 0.0078, Sem 0.0078 LR: 2.472e-02 Score 99.721 Data time: 0.1495, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8113 [2021-07-09 01:09:37,233][root][INFO] - ===> Epoch[9461](47305/5): Loss 0.0076, Sem 0.0076 LR: 2.471e-02 Score 99.727 Data time: 0.1903, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6095 [2021-07-09 01:09:51,788][root][INFO] - ===> Epoch[9462](47310/5): Loss 0.0078, Sem 0.0078 LR: 2.471e-02 Score 99.719 Data time: 0.2302, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8622 [2021-07-09 01:10:06,022][root][INFO] - ===> Epoch[9463](47315/5): Loss 0.0076, Sem 0.0076 LR: 2.470e-02 Score 99.727 Data time: 0.2413, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8001 [2021-07-09 01:10:20,966][root][INFO] - ===> Epoch[9464](47320/5): Loss 0.0078, Sem 0.0078 LR: 2.469e-02 Score 99.720 Data time: 0.2544, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.9358 [2021-07-09 01:10:35,299][root][INFO] - ===> Epoch[9465](47325/5): Loss 0.0077, Sem 0.0077 LR: 2.468e-02 Score 99.726 Data time: 0.1939, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8259 [2021-07-09 01:10:48,837][root][INFO] - ===> Epoch[9466](47330/5): Loss 0.0073, Sem 0.0073 LR: 2.467e-02 Score 99.736 Data time: 0.1857, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6671 [2021-07-09 01:11:02,985][root][INFO] - ===> Epoch[9467](47335/5): Loss 0.0079, Sem 0.0079 LR: 2.466e-02 Score 99.715 Data time: 0.2535, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7850 [2021-07-09 01:11:16,767][root][INFO] - ===> Epoch[9468](47340/5): Loss 0.0076, Sem 0.0076 LR: 2.465e-02 Score 99.730 Data time: 0.1976, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7158 [2021-07-09 01:11:31,554][root][INFO] - ===> Epoch[9469](47345/5): Loss 0.0078, Sem 0.0078 LR: 2.464e-02 Score 99.721 Data time: 0.2516, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.9180 [2021-07-09 01:11:45,199][root][INFO] - ===> Epoch[9470](47350/5): Loss 0.0075, Sem 0.0075 LR: 2.464e-02 Score 99.730 Data time: 0.2320, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6812 [2021-07-09 01:11:59,609][root][INFO] - ===> Epoch[9471](47355/5): Loss 0.0076, Sem 0.0076 LR: 2.463e-02 Score 99.724 Data time: 0.2579, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8283 [2021-07-09 01:12:13,272][root][INFO] - ===> Epoch[9472](47360/5): Loss 0.0082, Sem 0.0082 LR: 2.462e-02 Score 99.706 Data time: 0.2556, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6898 [2021-07-09 01:12:28,496][root][INFO] - ===> Epoch[9473](47365/5): Loss 0.0075, Sem 0.0075 LR: 2.461e-02 Score 99.730 Data time: 0.2622, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.9725 [2021-07-09 01:12:42,352][root][INFO] - ===> Epoch[9474](47370/5): Loss 0.0075, Sem 0.0075 LR: 2.460e-02 Score 99.731 Data time: 0.1889, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7240 [2021-07-09 01:12:56,653][root][INFO] - ===> Epoch[9475](47375/5): Loss 0.0073, Sem 0.0073 LR: 2.459e-02 Score 99.737 Data time: 0.1607, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8274 [2021-07-09 01:13:10,839][root][INFO] - ===> Epoch[9476](47380/5): Loss 0.0078, Sem 0.0078 LR: 2.458e-02 Score 99.722 Data time: 0.2664, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7863 [2021-07-09 01:13:23,908][root][INFO] - ===> Epoch[9477](47385/5): Loss 0.0079, Sem 0.0079 LR: 2.457e-02 Score 99.716 Data time: 0.2015, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.5710 [2021-07-09 01:13:38,043][root][INFO] - ===> Epoch[9478](47390/5): Loss 0.0077, Sem 0.0077 LR: 2.457e-02 Score 99.726 Data time: 0.2487, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7697 [2021-07-09 01:13:52,029][root][INFO] - ===> Epoch[9479](47395/5): Loss 0.0073, Sem 0.0073 LR: 2.456e-02 Score 99.739 Data time: 0.2272, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7474 [2021-07-09 01:14:06,435][root][INFO] - ===> Epoch[9480](47400/5): Loss 0.0076, Sem 0.0076 LR: 2.455e-02 Score 99.724 Data time: 0.2406, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8365 [2021-07-09 01:14:06,440][root][INFO] - ===> Start testing [2021-07-09 01:14:44,844][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1305 Loss 0.359 (AVG: 0.617) Score 92.862 (AVG: 91.106) mIOU 71.684 mAP 82.779 mAcc 77.669 IOU: 64.515 0.000 81.289 75.588 95.156 92.179 29.843 85.939 98.644 84.627 80.833 85.996 57.277 mAP: 83.032 3.371 88.854 86.064 98.487 98.229 52.600 97.629 99.683 95.095 91.174 92.627 89.289 mAcc: 79.519 0.000 84.872 83.725 96.845 97.117 32.170 94.313 99.410 93.252 91.393 96.573 60.512 [2021-07-09 01:14:44,844][root][INFO] - Finished test. Elapsed time: 38.4035 [2021-07-09 01:14:45,923][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 01:14:59,284][root][INFO] - ===> Epoch[9481](47405/5): Loss 0.0078, Sem 0.0078 LR: 2.454e-02 Score 99.721 Data time: 0.1843, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6372 [2021-07-09 01:15:13,697][root][INFO] - ===> Epoch[9482](47410/5): Loss 0.0078, Sem 0.0078 LR: 2.453e-02 Score 99.718 Data time: 0.2229, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8373 [2021-07-09 01:15:27,788][root][INFO] - ===> Epoch[9483](47415/5): Loss 0.0076, Sem 0.0076 LR: 2.452e-02 Score 99.725 Data time: 0.2647, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7732 [2021-07-09 01:15:41,217][root][INFO] - ===> Epoch[9484](47420/5): Loss 0.0078, Sem 0.0078 LR: 2.451e-02 Score 99.721 Data time: 0.2650, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6296 [2021-07-09 01:15:55,252][root][INFO] - ===> Epoch[9485](47425/5): Loss 0.0075, Sem 0.0075 LR: 2.450e-02 Score 99.733 Data time: 0.2961, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7599 [2021-07-09 01:16:09,127][root][INFO] - ===> Epoch[9486](47430/5): Loss 0.0077, Sem 0.0077 LR: 2.450e-02 Score 99.724 Data time: 0.2002, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7262 [2021-07-09 01:16:23,157][root][INFO] - ===> Epoch[9487](47435/5): Loss 0.0076, Sem 0.0076 LR: 2.449e-02 Score 99.727 Data time: 0.2574, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7672 [2021-07-09 01:16:38,350][root][INFO] - ===> Epoch[9488](47440/5): Loss 0.0078, Sem 0.0078 LR: 2.448e-02 Score 99.720 Data time: 0.1991, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.9993 [2021-07-09 01:16:52,650][root][INFO] - ===> Epoch[9489](47445/5): Loss 0.0077, Sem 0.0077 LR: 2.447e-02 Score 99.724 Data time: 0.2443, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8167 [2021-07-09 01:17:06,355][root][INFO] - ===> Epoch[9490](47450/5): Loss 0.0076, Sem 0.0076 LR: 2.446e-02 Score 99.725 Data time: 0.2644, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6915 [2021-07-09 01:17:20,608][root][INFO] - ===> Epoch[9491](47455/5): Loss 0.0072, Sem 0.0072 LR: 2.445e-02 Score 99.741 Data time: 0.2194, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8155 [2021-07-09 01:17:34,565][root][INFO] - ===> Epoch[9492](47460/5): Loss 0.0078, Sem 0.0078 LR: 2.444e-02 Score 99.718 Data time: 0.2091, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7548 [2021-07-09 01:17:48,192][root][INFO] - ===> Epoch[9493](47465/5): Loss 0.0076, Sem 0.0076 LR: 2.443e-02 Score 99.728 Data time: 0.2244, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6760 [2021-07-09 01:18:01,911][root][INFO] - ===> Epoch[9494](47470/5): Loss 0.0078, Sem 0.0078 LR: 2.443e-02 Score 99.721 Data time: 0.2513, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6927 [2021-07-09 01:18:16,193][root][INFO] - ===> Epoch[9495](47475/5): Loss 0.0074, Sem 0.0074 LR: 2.442e-02 Score 99.733 Data time: 0.2624, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8073 [2021-07-09 01:18:30,159][root][INFO] - ===> Epoch[9496](47480/5): Loss 0.0074, Sem 0.0074 LR: 2.441e-02 Score 99.732 Data time: 0.2633, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7367 [2021-07-09 01:18:44,300][root][INFO] - ===> Epoch[9497](47485/5): Loss 0.0077, Sem 0.0077 LR: 2.440e-02 Score 99.723 Data time: 0.1493, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7930 [2021-07-09 01:18:58,310][root][INFO] - ===> Epoch[9498](47490/5): Loss 0.0071, Sem 0.0071 LR: 2.439e-02 Score 99.742 Data time: 0.2464, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7447 [2021-07-09 01:19:12,085][root][INFO] - ===> Epoch[9499](47495/5): Loss 0.0075, Sem 0.0075 LR: 2.438e-02 Score 99.730 Data time: 0.2299, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7055 [2021-07-09 01:19:26,417][root][INFO] - ===> Epoch[9500](47500/5): Loss 0.0074, Sem 0.0074 LR: 2.437e-02 Score 99.736 Data time: 0.2006, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8226 [2021-07-09 01:19:39,628][root][INFO] - ===> Epoch[9501](47505/5): Loss 0.0071, Sem 0.0071 LR: 2.436e-02 Score 99.743 Data time: 0.2230, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6028 [2021-07-09 01:19:54,411][root][INFO] - ===> Epoch[9502](47510/5): Loss 0.0080, Sem 0.0080 LR: 2.436e-02 Score 99.716 Data time: 0.2663, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.9160 [2021-07-09 01:20:07,779][root][INFO] - ===> Epoch[9503](47515/5): Loss 0.0077, Sem 0.0077 LR: 2.435e-02 Score 99.727 Data time: 0.1584, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6434 [2021-07-09 01:20:22,145][root][INFO] - ===> Epoch[9504](47520/5): Loss 0.0076, Sem 0.0076 LR: 2.434e-02 Score 99.730 Data time: 0.2489, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8225 [2021-07-09 01:20:36,340][root][INFO] - ===> Epoch[9505](47525/5): Loss 0.0076, Sem 0.0076 LR: 2.433e-02 Score 99.728 Data time: 0.1846, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7945 [2021-07-09 01:20:50,247][root][INFO] - ===> Epoch[9506](47530/5): Loss 0.0077, Sem 0.0077 LR: 2.432e-02 Score 99.720 Data time: 0.2677, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7373 [2021-07-09 01:21:03,794][root][INFO] - ===> Epoch[9507](47535/5): Loss 0.0081, Sem 0.0081 LR: 2.431e-02 Score 99.713 Data time: 0.1849, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6769 [2021-07-09 01:21:18,371][root][INFO] - ===> Epoch[9508](47540/5): Loss 0.0083, Sem 0.0083 LR: 2.430e-02 Score 99.710 Data time: 0.1943, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8683 [2021-07-09 01:21:31,758][root][INFO] - ===> Epoch[9509](47545/5): Loss 0.0085, Sem 0.0085 LR: 2.429e-02 Score 99.694 Data time: 0.2661, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6333 [2021-07-09 01:21:45,941][root][INFO] - ===> Epoch[9510](47550/5): Loss 0.0086, Sem 0.0086 LR: 2.429e-02 Score 99.692 Data time: 0.2265, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7890 [2021-07-09 01:21:59,859][root][INFO] - ===> Epoch[9511](47555/5): Loss 0.0101, Sem 0.0101 LR: 2.428e-02 Score 99.672 Data time: 0.2303, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7251 [2021-07-09 01:22:13,660][root][INFO] - ===> Epoch[9512](47560/5): Loss 0.0087, Sem 0.0087 LR: 2.427e-02 Score 99.693 Data time: 0.2156, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7034 [2021-07-09 01:22:28,353][root][INFO] - ===> Epoch[9513](47565/5): Loss 0.0089, Sem 0.0089 LR: 2.426e-02 Score 99.680 Data time: 0.1428, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8949 [2021-07-09 01:22:41,880][root][INFO] - ===> Epoch[9514](47570/5): Loss 0.0089, Sem 0.0089 LR: 2.425e-02 Score 99.678 Data time: 0.2223, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6662 [2021-07-09 01:22:56,105][root][INFO] - ===> Epoch[9515](47575/5): Loss 0.0093, Sem 0.0093 LR: 2.424e-02 Score 99.679 Data time: 0.2153, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8026 [2021-07-09 01:23:10,294][root][INFO] - ===> Epoch[9516](47580/5): Loss 0.0087, Sem 0.0087 LR: 2.423e-02 Score 99.688 Data time: 0.2253, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7946 [2021-07-09 01:23:24,351][root][INFO] - ===> Epoch[9517](47585/5): Loss 0.0081, Sem 0.0081 LR: 2.422e-02 Score 99.709 Data time: 0.2339, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7611 [2021-07-09 01:23:38,659][root][INFO] - ===> Epoch[9518](47590/5): Loss 0.0083, Sem 0.0083 LR: 2.421e-02 Score 99.701 Data time: 0.2549, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8176 [2021-07-09 01:23:53,636][root][INFO] - ===> Epoch[9519](47595/5): Loss 0.0088, Sem 0.0088 LR: 2.421e-02 Score 99.696 Data time: 0.2955, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.9307 [2021-07-09 01:24:07,497][root][INFO] - ===> Epoch[9520](47600/5): Loss 0.0083, Sem 0.0083 LR: 2.420e-02 Score 99.704 Data time: 0.2938, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7274 [2021-07-09 01:24:07,499][root][INFO] - ===> Start testing [2021-07-09 01:24:45,929][root][INFO] - 68/68: Data time: 0.0016, Iter time: 0.1298 Loss 0.349 (AVG: 0.623) Score 92.788 (AVG: 90.944) mIOU 71.069 mAP 82.740 mAcc 77.135 IOU: 64.574 0.000 81.259 74.261 95.143 91.665 27.166 85.395 98.651 82.453 79.801 86.049 57.477 mAP: 83.197 4.196 88.857 85.663 98.579 98.389 51.586 97.777 99.693 94.373 90.791 93.451 89.067 mAcc: 81.725 0.000 84.087 81.442 96.613 97.109 28.935 94.717 99.452 90.914 89.676 96.617 61.475 [2021-07-09 01:24:45,930][root][INFO] - Finished test. Elapsed time: 38.4294 [2021-07-09 01:24:47,011][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 01:25:00,207][root][INFO] - ===> Epoch[9521](47605/5): Loss 0.0084, Sem 0.0084 LR: 2.419e-02 Score 99.700 Data time: 0.2230, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6017 [2021-07-09 01:25:15,042][root][INFO] - ===> Epoch[9522](47610/5): Loss 0.0088, Sem 0.0088 LR: 2.418e-02 Score 99.690 Data time: 0.2385, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.9161 [2021-07-09 01:25:29,366][root][INFO] - ===> Epoch[9523](47615/5): Loss 0.0084, Sem 0.0084 LR: 2.417e-02 Score 99.700 Data time: 0.2048, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8187 [2021-07-09 01:25:43,175][root][INFO] - ===> Epoch[9524](47620/5): Loss 0.0087, Sem 0.0087 LR: 2.416e-02 Score 99.696 Data time: 0.2582, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7170 [2021-07-09 01:25:56,740][root][INFO] - ===> Epoch[9525](47625/5): Loss 0.0086, Sem 0.0086 LR: 2.415e-02 Score 99.692 Data time: 0.1983, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6705 [2021-07-09 01:26:10,692][root][INFO] - ===> Epoch[9526](47630/5): Loss 0.0086, Sem 0.0086 LR: 2.414e-02 Score 99.692 Data time: 0.2271, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7486 [2021-07-09 01:26:24,436][root][INFO] - ===> Epoch[9527](47635/5): Loss 0.0080, Sem 0.0080 LR: 2.414e-02 Score 99.712 Data time: 0.2496, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7016 [2021-07-09 01:26:38,977][root][INFO] - ===> Epoch[9528](47640/5): Loss 0.0081, Sem 0.0081 LR: 2.413e-02 Score 99.713 Data time: 0.2133, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8660 [2021-07-09 01:26:52,654][root][INFO] - ===> Epoch[9529](47645/5): Loss 0.0084, Sem 0.0084 LR: 2.412e-02 Score 99.698 Data time: 0.1362, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7050 [2021-07-09 01:27:06,394][root][INFO] - ===> Epoch[9530](47650/5): Loss 0.0082, Sem 0.0082 LR: 2.411e-02 Score 99.707 Data time: 0.2781, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7016 [2021-07-09 01:27:20,492][root][INFO] - ===> Epoch[9531](47655/5): Loss 0.0084, Sem 0.0084 LR: 2.410e-02 Score 99.697 Data time: 0.2062, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7702 [2021-07-09 01:27:34,225][root][INFO] - ===> Epoch[9532](47660/5): Loss 0.0078, Sem 0.0078 LR: 2.409e-02 Score 99.723 Data time: 0.1443, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6931 [2021-07-09 01:27:48,020][root][INFO] - ===> Epoch[9533](47665/5): Loss 0.0083, Sem 0.0083 LR: 2.408e-02 Score 99.700 Data time: 0.1981, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7164 [2021-07-09 01:28:01,962][root][INFO] - ===> Epoch[9534](47670/5): Loss 0.0079, Sem 0.0079 LR: 2.407e-02 Score 99.714 Data time: 0.2371, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7364 [2021-07-09 01:28:16,105][root][INFO] - ===> Epoch[9535](47675/5): Loss 0.0079, Sem 0.0079 LR: 2.407e-02 Score 99.716 Data time: 0.2158, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7823 [2021-07-09 01:28:29,650][root][INFO] - ===> Epoch[9536](47680/5): Loss 0.0081, Sem 0.0081 LR: 2.406e-02 Score 99.709 Data time: 0.2419, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6620 [2021-07-09 01:28:43,976][root][INFO] - ===> Epoch[9537](47685/5): Loss 0.0078, Sem 0.0078 LR: 2.405e-02 Score 99.719 Data time: 0.2553, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8234 [2021-07-09 01:28:57,470][root][INFO] - ===> Epoch[9538](47690/5): Loss 0.0077, Sem 0.0077 LR: 2.404e-02 Score 99.725 Data time: 0.2511, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6506 [2021-07-09 01:29:11,523][root][INFO] - ===> Epoch[9539](47695/5): Loss 0.0073, Sem 0.0073 LR: 2.403e-02 Score 99.737 Data time: 0.2236, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7540 [2021-07-09 01:29:25,551][root][INFO] - ===> Epoch[9540](47700/5): Loss 0.0079, Sem 0.0079 LR: 2.402e-02 Score 99.716 Data time: 0.2096, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7643 [2021-07-09 01:29:39,979][root][INFO] - ===> Epoch[9541](47705/5): Loss 0.0078, Sem 0.0078 LR: 2.401e-02 Score 99.720 Data time: 0.2209, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8383 [2021-07-09 01:29:55,078][root][INFO] - ===> Epoch[9542](47710/5): Loss 0.0078, Sem 0.0078 LR: 2.400e-02 Score 99.720 Data time: 0.2566, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.9688 [2021-07-09 01:30:09,029][root][INFO] - ===> Epoch[9543](47715/5): Loss 0.0077, Sem 0.0077 LR: 2.400e-02 Score 99.726 Data time: 0.1841, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7421 [2021-07-09 01:30:22,899][root][INFO] - ===> Epoch[9544](47720/5): Loss 0.0076, Sem 0.0076 LR: 2.399e-02 Score 99.729 Data time: 0.1981, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7263 [2021-07-09 01:30:36,433][root][INFO] - ===> Epoch[9545](47725/5): Loss 0.0078, Sem 0.0078 LR: 2.398e-02 Score 99.722 Data time: 0.2330, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6586 [2021-07-09 01:30:51,359][root][INFO] - ===> Epoch[9546](47730/5): Loss 0.0079, Sem 0.0079 LR: 2.397e-02 Score 99.716 Data time: 0.2043, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.9335 [2021-07-09 01:31:05,740][root][INFO] - ===> Epoch[9547](47735/5): Loss 0.0079, Sem 0.0079 LR: 2.396e-02 Score 99.717 Data time: 0.2443, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8137 [2021-07-09 01:31:19,498][root][INFO] - ===> Epoch[9548](47740/5): Loss 0.0076, Sem 0.0076 LR: 2.395e-02 Score 99.727 Data time: 0.2379, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7040 [2021-07-09 01:31:33,652][root][INFO] - ===> Epoch[9549](47745/5): Loss 0.0080, Sem 0.0080 LR: 2.394e-02 Score 99.713 Data time: 0.2194, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7924 [2021-07-09 01:31:47,066][root][INFO] - ===> Epoch[9550](47750/5): Loss 0.0075, Sem 0.0075 LR: 2.393e-02 Score 99.733 Data time: 0.1893, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.6394 [2021-07-09 01:32:00,523][root][INFO] - ===> Epoch[9551](47755/5): Loss 0.0079, Sem 0.0079 LR: 2.393e-02 Score 99.720 Data time: 0.2026, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.6480 [2021-07-09 01:32:14,603][root][INFO] - ===> Epoch[9552](47760/5): Loss 0.0080, Sem 0.0080 LR: 2.392e-02 Score 99.714 Data time: 0.2544, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7588 [2021-07-09 01:32:28,329][root][INFO] - ===> Epoch[9553](47765/5): Loss 0.0079, Sem 0.0079 LR: 2.391e-02 Score 99.717 Data time: 0.3060, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6914 [2021-07-09 01:32:41,968][root][INFO] - ===> Epoch[9554](47770/5): Loss 0.0078, Sem 0.0078 LR: 2.390e-02 Score 99.717 Data time: 0.2102, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6946 [2021-07-09 01:32:56,053][root][INFO] - ===> Epoch[9555](47775/5): Loss 0.0077, Sem 0.0077 LR: 2.389e-02 Score 99.726 Data time: 0.2217, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7714 [2021-07-09 01:33:09,638][root][INFO] - ===> Epoch[9556](47780/5): Loss 0.0079, Sem 0.0079 LR: 2.388e-02 Score 99.714 Data time: 0.1566, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6781 [2021-07-09 01:33:23,653][root][INFO] - ===> Epoch[9557](47785/5): Loss 0.0080, Sem 0.0080 LR: 2.387e-02 Score 99.713 Data time: 0.2014, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7553 [2021-07-09 01:33:37,245][root][INFO] - ===> Epoch[9558](47790/5): Loss 0.0076, Sem 0.0076 LR: 2.386e-02 Score 99.728 Data time: 0.2428, Forward time: 1.2528, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6744 [2021-07-09 01:33:51,390][root][INFO] - ===> Epoch[9559](47795/5): Loss 0.0073, Sem 0.0073 LR: 2.385e-02 Score 99.738 Data time: 0.1815, Forward time: 1.2528, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7901 [2021-07-09 01:34:05,582][root][INFO] - ===> Epoch[9560](47800/5): Loss 0.0076, Sem 0.0076 LR: 2.385e-02 Score 99.729 Data time: 0.2250, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7943 [2021-07-09 01:34:05,583][root][INFO] - ===> Start testing [2021-07-09 01:34:44,051][root][INFO] - 68/68: Data time: 0.0017, Iter time: 0.1255 Loss 0.384 (AVG: 0.606) Score 92.453 (AVG: 91.142) mIOU 71.770 mAP 83.170 mAcc 78.226 IOU: 64.638 0.000 80.074 75.178 95.372 91.733 35.069 84.548 98.604 83.759 80.453 86.449 57.126 mAP: 83.886 4.053 88.886 86.027 98.542 98.252 54.839 97.857 99.679 95.525 90.856 92.982 89.830 mAcc: 79.637 0.000 83.617 83.829 96.812 97.489 39.779 94.728 99.435 94.386 90.735 96.240 60.244 [2021-07-09 01:34:44,051][root][INFO] - Finished test. Elapsed time: 38.4674 [2021-07-09 01:34:45,109][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 01:34:57,648][root][INFO] - ===> Epoch[9561](47805/5): Loss 0.0074, Sem 0.0074 LR: 2.384e-02 Score 99.731 Data time: 0.1711, Forward time: 1.2528, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.4744 [2021-07-09 01:35:11,376][root][INFO] - ===> Epoch[9562](47810/5): Loss 0.0079, Sem 0.0079 LR: 2.383e-02 Score 99.720 Data time: 0.2734, Forward time: 1.2528, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7074 [2021-07-09 01:35:25,760][root][INFO] - ===> Epoch[9563](47815/5): Loss 0.0075, Sem 0.0075 LR: 2.382e-02 Score 99.729 Data time: 0.2692, Forward time: 1.2528, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8304 [2021-07-09 01:35:40,030][root][INFO] - ===> Epoch[9564](47820/5): Loss 0.0079, Sem 0.0079 LR: 2.381e-02 Score 99.717 Data time: 0.2282, Forward time: 1.2528, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8093 [2021-07-09 01:35:54,663][root][INFO] - ===> Epoch[9565](47825/5): Loss 0.0076, Sem 0.0076 LR: 2.380e-02 Score 99.727 Data time: 0.2131, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8845 [2021-07-09 01:36:08,764][root][INFO] - ===> Epoch[9566](47830/5): Loss 0.0076, Sem 0.0076 LR: 2.379e-02 Score 99.726 Data time: 0.2187, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7758 [2021-07-09 01:36:22,775][root][INFO] - ===> Epoch[9567](47835/5): Loss 0.0078, Sem 0.0078 LR: 2.378e-02 Score 99.717 Data time: 0.2359, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7527 [2021-07-09 01:36:36,428][root][INFO] - ===> Epoch[9568](47840/5): Loss 0.0080, Sem 0.0080 LR: 2.378e-02 Score 99.717 Data time: 0.2179, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6875 [2021-07-09 01:36:50,332][root][INFO] - ===> Epoch[9569](47845/5): Loss 0.0078, Sem 0.0078 LR: 2.377e-02 Score 99.723 Data time: 0.2582, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7384 [2021-07-09 01:37:04,701][root][INFO] - ===> Epoch[9570](47850/5): Loss 0.0078, Sem 0.0078 LR: 2.376e-02 Score 99.718 Data time: 0.2798, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8204 [2021-07-09 01:37:20,124][root][INFO] - ===> Epoch[9571](47855/5): Loss 0.0076, Sem 0.0076 LR: 2.375e-02 Score 99.724 Data time: 0.2014, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 3.0437 [2021-07-09 01:37:33,398][root][INFO] - ===> Epoch[9572](47860/5): Loss 0.0076, Sem 0.0076 LR: 2.374e-02 Score 99.725 Data time: 0.2101, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6072 [2021-07-09 01:37:48,703][root][INFO] - ===> Epoch[9573](47865/5): Loss 0.0078, Sem 0.0078 LR: 2.373e-02 Score 99.719 Data time: 0.2510, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 3.0248 [2021-07-09 01:38:02,471][root][INFO] - ===> Epoch[9574](47870/5): Loss 0.0077, Sem 0.0077 LR: 2.372e-02 Score 99.721 Data time: 0.1765, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7013 [2021-07-09 01:38:16,435][root][INFO] - ===> Epoch[9575](47875/5): Loss 0.0075, Sem 0.0075 LR: 2.371e-02 Score 99.731 Data time: 0.1998, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7462 [2021-07-09 01:38:30,832][root][INFO] - ===> Epoch[9576](47880/5): Loss 0.0076, Sem 0.0076 LR: 2.371e-02 Score 99.732 Data time: 0.2425, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8395 [2021-07-09 01:38:44,084][root][INFO] - ===> Epoch[9577](47885/5): Loss 0.0075, Sem 0.0075 LR: 2.370e-02 Score 99.732 Data time: 0.2298, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6050 [2021-07-09 01:38:58,369][root][INFO] - ===> Epoch[9578](47890/5): Loss 0.0074, Sem 0.0074 LR: 2.369e-02 Score 99.732 Data time: 0.2005, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8092 [2021-07-09 01:39:12,308][root][INFO] - ===> Epoch[9579](47895/5): Loss 0.0076, Sem 0.0076 LR: 2.368e-02 Score 99.729 Data time: 0.2243, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7433 [2021-07-09 01:39:25,972][root][INFO] - ===> Epoch[9580](47900/5): Loss 0.0077, Sem 0.0077 LR: 2.367e-02 Score 99.723 Data time: 0.2042, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6791 [2021-07-09 01:39:39,806][root][INFO] - ===> Epoch[9581](47905/5): Loss 0.0075, Sem 0.0075 LR: 2.366e-02 Score 99.732 Data time: 0.2376, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7137 [2021-07-09 01:39:53,645][root][INFO] - ===> Epoch[9582](47910/5): Loss 0.0072, Sem 0.0072 LR: 2.365e-02 Score 99.739 Data time: 0.1685, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7268 [2021-07-09 01:40:07,474][root][INFO] - ===> Epoch[9583](47915/5): Loss 0.0074, Sem 0.0074 LR: 2.364e-02 Score 99.732 Data time: 0.2265, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7209 [2021-07-09 01:40:20,851][root][INFO] - ===> Epoch[9584](47920/5): Loss 0.0077, Sem 0.0077 LR: 2.363e-02 Score 99.723 Data time: 0.2489, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6350 [2021-07-09 01:40:35,156][root][INFO] - ===> Epoch[9585](47925/5): Loss 0.0076, Sem 0.0076 LR: 2.363e-02 Score 99.727 Data time: 0.1704, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8204 [2021-07-09 01:40:49,061][root][INFO] - ===> Epoch[9586](47930/5): Loss 0.0077, Sem 0.0077 LR: 2.362e-02 Score 99.727 Data time: 0.2171, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7410 [2021-07-09 01:41:03,318][root][INFO] - ===> Epoch[9587](47935/5): Loss 0.0076, Sem 0.0076 LR: 2.361e-02 Score 99.725 Data time: 0.2395, Forward time: 1.2528, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8065 [2021-07-09 01:41:17,870][root][INFO] - ===> Epoch[9588](47940/5): Loss 0.0075, Sem 0.0075 LR: 2.360e-02 Score 99.730 Data time: 0.2383, Forward time: 1.2528, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8629 [2021-07-09 01:41:31,848][root][INFO] - ===> Epoch[9589](47945/5): Loss 0.0075, Sem 0.0075 LR: 2.359e-02 Score 99.728 Data time: 0.2439, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7514 [2021-07-09 01:41:46,192][root][INFO] - ===> Epoch[9590](47950/5): Loss 0.0073, Sem 0.0073 LR: 2.358e-02 Score 99.739 Data time: 0.2028, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8270 [2021-07-09 01:42:00,062][root][INFO] - ===> Epoch[9591](47955/5): Loss 0.0075, Sem 0.0075 LR: 2.357e-02 Score 99.733 Data time: 0.2120, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7339 [2021-07-09 01:42:13,435][root][INFO] - ===> Epoch[9592](47960/5): Loss 0.0080, Sem 0.0080 LR: 2.356e-02 Score 99.713 Data time: 0.2406, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.6333 [2021-07-09 01:42:27,779][root][INFO] - ===> Epoch[9593](47965/5): Loss 0.0076, Sem 0.0076 LR: 2.356e-02 Score 99.726 Data time: 0.2148, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8299 [2021-07-09 01:42:41,592][root][INFO] - ===> Epoch[9594](47970/5): Loss 0.0079, Sem 0.0079 LR: 2.355e-02 Score 99.716 Data time: 0.2701, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7095 [2021-07-09 01:42:54,725][root][INFO] - ===> Epoch[9595](47975/5): Loss 0.0074, Sem 0.0074 LR: 2.354e-02 Score 99.732 Data time: 0.2422, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.5705 [2021-07-09 01:43:09,795][root][INFO] - ===> Epoch[9596](47980/5): Loss 0.0072, Sem 0.0072 LR: 2.353e-02 Score 99.742 Data time: 0.2712, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.9759 [2021-07-09 01:43:23,695][root][INFO] - ===> Epoch[9597](47985/5): Loss 0.0074, Sem 0.0074 LR: 2.352e-02 Score 99.733 Data time: 0.1576, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7340 [2021-07-09 01:43:38,148][root][INFO] - ===> Epoch[9598](47990/5): Loss 0.0079, Sem 0.0079 LR: 2.351e-02 Score 99.718 Data time: 0.2108, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8362 [2021-07-09 01:43:51,311][root][INFO] - ===> Epoch[9599](47995/5): Loss 0.0076, Sem 0.0076 LR: 2.350e-02 Score 99.730 Data time: 0.2217, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.5943 [2021-07-09 01:44:04,752][root][INFO] - ===> Epoch[9600](48000/5): Loss 0.0077, Sem 0.0077 LR: 2.349e-02 Score 99.724 Data time: 0.2238, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.6426 [2021-07-09 01:44:04,753][root][INFO] - ===> Start testing [2021-07-09 01:44:40,613][root][INFO] - 68/68: Data time: 0.0016, Iter time: 0.1346 Loss 0.455 (AVG: 0.613) Score 91.763 (AVG: 91.063) mIOU 71.685 mAP 82.944 mAcc 77.696 IOU: 64.819 0.000 80.466 74.820 95.172 92.078 32.526 85.209 98.599 84.587 80.583 85.985 57.058 mAP: 83.084 4.166 89.547 85.939 98.510 98.090 53.477 97.805 99.703 94.221 90.794 92.974 89.963 mAcc: 79.969 0.000 83.007 82.855 96.677 97.477 35.687 94.835 99.384 92.857 90.166 96.701 60.427 [2021-07-09 01:44:40,614][root][INFO] - Finished test. Elapsed time: 35.8595 [2021-07-09 01:44:41,702][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 01:44:54,384][root][INFO] - ===> Epoch[9601](48005/5): Loss 0.0077, Sem 0.0077 LR: 2.348e-02 Score 99.726 Data time: 0.1871, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.5011 [2021-07-09 01:45:07,854][root][INFO] - ===> Epoch[9602](48010/5): Loss 0.0077, Sem 0.0077 LR: 2.348e-02 Score 99.722 Data time: 0.2039, Forward time: 1.2528, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6405 [2021-07-09 01:45:21,222][root][INFO] - ===> Epoch[9603](48015/5): Loss 0.0077, Sem 0.0077 LR: 2.347e-02 Score 99.726 Data time: 0.2552, Forward time: 1.2528, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6176 [2021-07-09 01:45:35,518][root][INFO] - ===> Epoch[9604](48020/5): Loss 0.0081, Sem 0.0081 LR: 2.346e-02 Score 99.711 Data time: 0.2763, Forward time: 1.2528, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8058 [2021-07-09 01:45:49,932][root][INFO] - ===> Epoch[9605](48025/5): Loss 0.0073, Sem 0.0073 LR: 2.345e-02 Score 99.739 Data time: 0.2096, Forward time: 1.2528, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8389 [2021-07-09 01:46:03,551][root][INFO] - ===> Epoch[9606](48030/5): Loss 0.0079, Sem 0.0079 LR: 2.344e-02 Score 99.721 Data time: 0.1988, Forward time: 1.2528, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6885 [2021-07-09 01:46:16,792][root][INFO] - ===> Epoch[9607](48035/5): Loss 0.0077, Sem 0.0077 LR: 2.343e-02 Score 99.723 Data time: 0.1973, Forward time: 1.2528, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6120 [2021-07-09 01:46:30,420][root][INFO] - ===> Epoch[9608](48040/5): Loss 0.0075, Sem 0.0075 LR: 2.342e-02 Score 99.732 Data time: 0.1984, Forward time: 1.2528, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6865 [2021-07-09 01:46:44,620][root][INFO] - ===> Epoch[9609](48045/5): Loss 0.0075, Sem 0.0075 LR: 2.341e-02 Score 99.732 Data time: 0.1743, Forward time: 1.2528, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8022 [2021-07-09 01:46:58,858][root][INFO] - ===> Epoch[9610](48050/5): Loss 0.0075, Sem 0.0075 LR: 2.341e-02 Score 99.730 Data time: 0.2282, Forward time: 1.2528, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8062 [2021-07-09 01:47:13,936][root][INFO] - ===> Epoch[9611](48055/5): Loss 0.0075, Sem 0.0075 LR: 2.340e-02 Score 99.730 Data time: 0.2349, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.9723 [2021-07-09 01:47:28,258][root][INFO] - ===> Epoch[9612](48060/5): Loss 0.0077, Sem 0.0077 LR: 2.339e-02 Score 99.723 Data time: 0.3464, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8134 [2021-07-09 01:47:42,743][root][INFO] - ===> Epoch[9613](48065/5): Loss 0.0075, Sem 0.0075 LR: 2.338e-02 Score 99.730 Data time: 0.2248, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8613 [2021-07-09 01:47:57,308][root][INFO] - ===> Epoch[9614](48070/5): Loss 0.0073, Sem 0.0073 LR: 2.337e-02 Score 99.737 Data time: 0.2756, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8668 [2021-07-09 01:48:11,091][root][INFO] - ===> Epoch[9615](48075/5): Loss 0.0075, Sem 0.0075 LR: 2.336e-02 Score 99.732 Data time: 0.2556, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7048 [2021-07-09 01:48:25,575][root][INFO] - ===> Epoch[9616](48080/5): Loss 0.0075, Sem 0.0075 LR: 2.335e-02 Score 99.733 Data time: 0.2957, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8477 [2021-07-09 01:48:39,867][root][INFO] - ===> Epoch[9617](48085/5): Loss 0.0077, Sem 0.0077 LR: 2.334e-02 Score 99.723 Data time: 0.2735, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7983 [2021-07-09 01:48:55,125][root][INFO] - ===> Epoch[9618](48090/5): Loss 0.0077, Sem 0.0077 LR: 2.334e-02 Score 99.721 Data time: 0.2407, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 3.0124 [2021-07-09 01:49:09,561][root][INFO] - ===> Epoch[9619](48095/5): Loss 0.0074, Sem 0.0074 LR: 2.333e-02 Score 99.733 Data time: 0.2439, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8477 [2021-07-09 01:49:24,147][root][INFO] - ===> Epoch[9620](48100/5): Loss 0.0072, Sem 0.0072 LR: 2.332e-02 Score 99.742 Data time: 0.2289, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8832 [2021-07-09 01:49:37,419][root][INFO] - ===> Epoch[9621](48105/5): Loss 0.0079, Sem 0.0079 LR: 2.331e-02 Score 99.717 Data time: 0.1870, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.6217 [2021-07-09 01:49:51,306][root][INFO] - ===> Epoch[9622](48110/5): Loss 0.0077, Sem 0.0077 LR: 2.330e-02 Score 99.718 Data time: 0.2239, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7319 [2021-07-09 01:50:05,001][root][INFO] - ===> Epoch[9623](48115/5): Loss 0.0076, Sem 0.0076 LR: 2.329e-02 Score 99.727 Data time: 0.2334, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.6908 [2021-07-09 01:50:19,132][root][INFO] - ===> Epoch[9624](48120/5): Loss 0.0078, Sem 0.0078 LR: 2.328e-02 Score 99.724 Data time: 0.2031, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7860 [2021-07-09 01:50:33,853][root][INFO] - ===> Epoch[9625](48125/5): Loss 0.0079, Sem 0.0079 LR: 2.327e-02 Score 99.723 Data time: 0.1996, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.9041 [2021-07-09 01:50:47,755][root][INFO] - ===> Epoch[9626](48130/5): Loss 0.0082, Sem 0.0082 LR: 2.326e-02 Score 99.709 Data time: 0.2243, Forward time: 1.2527, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7340 [2021-07-09 01:51:01,946][root][INFO] - ===> Epoch[9627](48135/5): Loss 0.0077, Sem 0.0077 LR: 2.326e-02 Score 99.723 Data time: 0.2094, Forward time: 1.2527, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7947 [2021-07-09 01:51:16,220][root][INFO] - ===> Epoch[9628](48140/5): Loss 0.0076, Sem 0.0076 LR: 2.325e-02 Score 99.727 Data time: 0.2004, Forward time: 1.2527, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8131 [2021-07-09 01:51:30,402][root][INFO] - ===> Epoch[9629](48145/5): Loss 0.0080, Sem 0.0080 LR: 2.324e-02 Score 99.714 Data time: 0.2516, Forward time: 1.2527, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7947 [2021-07-09 01:51:44,832][root][INFO] - ===> Epoch[9630](48150/5): Loss 0.0077, Sem 0.0077 LR: 2.323e-02 Score 99.721 Data time: 0.2398, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8411 [2021-07-09 01:51:59,418][root][INFO] - ===> Epoch[9631](48155/5): Loss 0.0077, Sem 0.0077 LR: 2.322e-02 Score 99.725 Data time: 0.1902, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8858 [2021-07-09 01:52:13,386][root][INFO] - ===> Epoch[9632](48160/5): Loss 0.0080, Sem 0.0080 LR: 2.321e-02 Score 99.711 Data time: 0.1625, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7538 [2021-07-09 01:52:27,473][root][INFO] - ===> Epoch[9633](48165/5): Loss 0.0077, Sem 0.0077 LR: 2.320e-02 Score 99.724 Data time: 0.2433, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7698 [2021-07-09 01:52:42,498][root][INFO] - ===> Epoch[9634](48170/5): Loss 0.0075, Sem 0.0075 LR: 2.319e-02 Score 99.728 Data time: 0.1893, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.9614 [2021-07-09 01:52:56,361][root][INFO] - ===> Epoch[9635](48175/5): Loss 0.0076, Sem 0.0076 LR: 2.319e-02 Score 99.728 Data time: 0.2162, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7231 [2021-07-09 01:53:10,276][root][INFO] - ===> Epoch[9636](48180/5): Loss 0.0073, Sem 0.0073 LR: 2.318e-02 Score 99.737 Data time: 0.2517, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7328 [2021-07-09 01:53:25,337][root][INFO] - ===> Epoch[9637](48185/5): Loss 0.0075, Sem 0.0075 LR: 2.317e-02 Score 99.733 Data time: 0.2187, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.9452 [2021-07-09 01:53:39,591][root][INFO] - ===> Epoch[9638](48190/5): Loss 0.0077, Sem 0.0077 LR: 2.316e-02 Score 99.729 Data time: 0.1911, Forward time: 1.2527, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8067 [2021-07-09 01:53:53,570][root][INFO] - ===> Epoch[9639](48195/5): Loss 0.0073, Sem 0.0073 LR: 2.315e-02 Score 99.738 Data time: 0.2345, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7584 [2021-07-09 01:54:07,923][root][INFO] - ===> Epoch[9640](48200/5): Loss 0.0075, Sem 0.0075 LR: 2.314e-02 Score 99.732 Data time: 0.2002, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8321 [2021-07-09 01:54:07,924][root][INFO] - ===> Start testing [2021-07-09 01:54:47,982][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1220 Loss 0.416 (AVG: 0.592) Score 91.753 (AVG: 91.171) mIOU 72.050 mAP 83.465 mAcc 78.570 IOU: 64.803 0.000 80.869 75.226 95.202 91.683 39.888 84.116 98.525 82.722 79.508 86.556 57.557 mAP: 84.335 3.017 89.575 85.691 98.431 98.115 60.382 97.820 99.718 94.524 90.658 92.593 90.181 mAcc: 79.910 0.000 83.741 83.181 96.655 97.525 46.149 94.511 99.329 92.581 90.681 96.246 60.897 [2021-07-09 01:54:47,982][root][INFO] - Finished test. Elapsed time: 40.0575 [2021-07-09 01:54:49,376][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 01:55:02,181][root][INFO] - ===> Epoch[9641](48205/5): Loss 0.0077, Sem 0.0077 LR: 2.313e-02 Score 99.723 Data time: 0.1878, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.5204 [2021-07-09 01:55:16,705][root][INFO] - ===> Epoch[9642](48210/5): Loss 0.0079, Sem 0.0079 LR: 2.312e-02 Score 99.716 Data time: 0.1774, Forward time: 1.2527, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8630 [2021-07-09 01:55:30,346][root][INFO] - ===> Epoch[9643](48215/5): Loss 0.0077, Sem 0.0077 LR: 2.311e-02 Score 99.724 Data time: 0.2246, Forward time: 1.2527, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6825 [2021-07-09 01:55:43,742][root][INFO] - ===> Epoch[9644](48220/5): Loss 0.0080, Sem 0.0080 LR: 2.311e-02 Score 99.715 Data time: 0.2238, Forward time: 1.2527, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6299 [2021-07-09 01:55:58,289][root][INFO] - ===> Epoch[9645](48225/5): Loss 0.0073, Sem 0.0073 LR: 2.310e-02 Score 99.737 Data time: 0.2440, Forward time: 1.2527, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8633 [2021-07-09 01:56:12,451][root][INFO] - ===> Epoch[9646](48230/5): Loss 0.0076, Sem 0.0076 LR: 2.309e-02 Score 99.728 Data time: 0.2567, Forward time: 1.2527, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7850 [2021-07-09 01:56:26,073][root][INFO] - ===> Epoch[9647](48235/5): Loss 0.0075, Sem 0.0075 LR: 2.308e-02 Score 99.733 Data time: 0.2270, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6778 [2021-07-09 01:56:39,630][root][INFO] - ===> Epoch[9648](48240/5): Loss 0.0079, Sem 0.0079 LR: 2.307e-02 Score 99.717 Data time: 0.2233, Forward time: 1.2527, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6662 [2021-07-09 01:56:53,531][root][INFO] - ===> Epoch[9649](48245/5): Loss 0.0078, Sem 0.0078 LR: 2.306e-02 Score 99.725 Data time: 0.1935, Forward time: 1.2527, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7316 [2021-07-09 01:57:07,454][root][INFO] - ===> Epoch[9650](48250/5): Loss 0.0077, Sem 0.0077 LR: 2.305e-02 Score 99.721 Data time: 0.1442, Forward time: 1.2527, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7388 [2021-07-09 01:57:21,610][root][INFO] - ===> Epoch[9651](48255/5): Loss 0.0079, Sem 0.0079 LR: 2.304e-02 Score 99.715 Data time: 0.1769, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7848 [2021-07-09 01:57:35,780][root][INFO] - ===> Epoch[9652](48260/5): Loss 0.0080, Sem 0.0080 LR: 2.304e-02 Score 99.714 Data time: 0.2024, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7856 [2021-07-09 01:57:49,888][root][INFO] - ===> Epoch[9653](48265/5): Loss 0.0078, Sem 0.0078 LR: 2.303e-02 Score 99.718 Data time: 0.2283, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7848 [2021-07-09 01:58:03,367][root][INFO] - ===> Epoch[9654](48270/5): Loss 0.0075, Sem 0.0075 LR: 2.302e-02 Score 99.731 Data time: 0.2510, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6424 [2021-07-09 01:58:17,710][root][INFO] - ===> Epoch[9655](48275/5): Loss 0.0076, Sem 0.0076 LR: 2.301e-02 Score 99.723 Data time: 0.2021, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8293 [2021-07-09 01:58:31,779][root][INFO] - ===> Epoch[9656](48280/5): Loss 0.0075, Sem 0.0075 LR: 2.300e-02 Score 99.729 Data time: 0.1939, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7756 [2021-07-09 01:58:44,579][root][INFO] - ===> Epoch[9657](48285/5): Loss 0.0079, Sem 0.0079 LR: 2.299e-02 Score 99.714 Data time: 0.1638, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.5206 [2021-07-09 01:58:58,278][root][INFO] - ===> Epoch[9658](48290/5): Loss 0.0073, Sem 0.0073 LR: 2.298e-02 Score 99.738 Data time: 0.2084, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6847 [2021-07-09 01:59:12,275][root][INFO] - ===> Epoch[9659](48295/5): Loss 0.0074, Sem 0.0074 LR: 2.297e-02 Score 99.735 Data time: 0.1829, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7567 [2021-07-09 01:59:26,403][root][INFO] - ===> Epoch[9660](48300/5): Loss 0.0075, Sem 0.0075 LR: 2.296e-02 Score 99.730 Data time: 0.2065, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7923 [2021-07-09 01:59:40,609][root][INFO] - ===> Epoch[9661](48305/5): Loss 0.0075, Sem 0.0075 LR: 2.296e-02 Score 99.731 Data time: 0.2146, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8011 [2021-07-09 01:59:55,338][root][INFO] - ===> Epoch[9662](48310/5): Loss 0.0076, Sem 0.0076 LR: 2.295e-02 Score 99.729 Data time: 0.2667, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.9001 [2021-07-09 02:00:10,156][root][INFO] - ===> Epoch[9663](48315/5): Loss 0.0077, Sem 0.0077 LR: 2.294e-02 Score 99.729 Data time: 0.2416, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.9004 [2021-07-09 02:00:25,161][root][INFO] - ===> Epoch[9664](48320/5): Loss 0.0077, Sem 0.0077 LR: 2.293e-02 Score 99.722 Data time: 0.2440, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.9570 [2021-07-09 02:00:39,583][root][INFO] - ===> Epoch[9665](48325/5): Loss 0.0073, Sem 0.0073 LR: 2.292e-02 Score 99.739 Data time: 0.2618, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8434 [2021-07-09 02:00:53,681][root][INFO] - ===> Epoch[9666](48330/5): Loss 0.0071, Sem 0.0071 LR: 2.291e-02 Score 99.745 Data time: 0.2671, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7627 [2021-07-09 02:01:07,766][root][INFO] - ===> Epoch[9667](48335/5): Loss 0.0077, Sem 0.0077 LR: 2.290e-02 Score 99.725 Data time: 0.2308, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7795 [2021-07-09 02:01:21,713][root][INFO] - ===> Epoch[9668](48340/5): Loss 0.0078, Sem 0.0078 LR: 2.289e-02 Score 99.719 Data time: 0.2225, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7416 [2021-07-09 02:01:35,855][root][INFO] - ===> Epoch[9669](48345/5): Loss 0.0078, Sem 0.0078 LR: 2.289e-02 Score 99.720 Data time: 0.2521, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7818 [2021-07-09 02:01:50,162][root][INFO] - ===> Epoch[9670](48350/5): Loss 0.0090, Sem 0.0090 LR: 2.288e-02 Score 99.699 Data time: 0.2401, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8057 [2021-07-09 02:02:04,113][root][INFO] - ===> Epoch[9671](48355/5): Loss 0.0082, Sem 0.0082 LR: 2.287e-02 Score 99.708 Data time: 0.2246, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7402 [2021-07-09 02:02:18,206][root][INFO] - ===> Epoch[9672](48360/5): Loss 0.0084, Sem 0.0084 LR: 2.286e-02 Score 99.700 Data time: 0.1923, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7857 [2021-07-09 02:02:32,137][root][INFO] - ===> Epoch[9673](48365/5): Loss 0.0080, Sem 0.0080 LR: 2.285e-02 Score 99.709 Data time: 0.2163, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7442 [2021-07-09 02:02:46,936][root][INFO] - ===> Epoch[9674](48370/5): Loss 0.0081, Sem 0.0081 LR: 2.284e-02 Score 99.712 Data time: 0.1999, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.9198 [2021-07-09 02:03:00,756][root][INFO] - ===> Epoch[9675](48375/5): Loss 0.0078, Sem 0.0078 LR: 2.283e-02 Score 99.718 Data time: 0.2125, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7082 [2021-07-09 02:03:14,839][root][INFO] - ===> Epoch[9676](48380/5): Loss 0.0077, Sem 0.0077 LR: 2.282e-02 Score 99.723 Data time: 0.2625, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7725 [2021-07-09 02:03:28,830][root][INFO] - ===> Epoch[9677](48385/5): Loss 0.0077, Sem 0.0077 LR: 2.281e-02 Score 99.725 Data time: 0.2310, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7576 [2021-07-09 02:03:42,315][root][INFO] - ===> Epoch[9678](48390/5): Loss 0.0080, Sem 0.0080 LR: 2.281e-02 Score 99.716 Data time: 0.1947, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6555 [2021-07-09 02:03:56,193][root][INFO] - ===> Epoch[9679](48395/5): Loss 0.0079, Sem 0.0079 LR: 2.280e-02 Score 99.716 Data time: 0.2267, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7276 [2021-07-09 02:04:09,931][root][INFO] - ===> Epoch[9680](48400/5): Loss 0.0075, Sem 0.0075 LR: 2.279e-02 Score 99.732 Data time: 0.2149, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7049 [2021-07-09 02:04:09,932][root][INFO] - ===> Start testing [2021-07-09 02:04:52,412][root][INFO] - 68/68: Data time: 0.0360, Iter time: 0.1287 Loss 0.404 (AVG: 0.609) Score 91.461 (AVG: 91.119) mIOU 71.652 mAP 82.952 mAcc 78.106 IOU: 64.368 0.000 82.927 75.194 95.190 91.608 33.029 84.429 98.545 82.458 79.062 86.434 58.231 mAP: 83.673 2.740 89.280 85.691 98.333 98.102 54.859 98.023 99.696 94.742 90.779 92.728 89.725 mAcc: 79.682 0.000 85.971 83.303 96.786 97.495 36.401 94.902 99.433 93.074 90.644 96.360 61.320 [2021-07-09 02:04:52,412][root][INFO] - Finished test. Elapsed time: 42.4786 [2021-07-09 02:04:53,424][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 02:05:06,732][root][INFO] - ===> Epoch[9681](48405/5): Loss 0.0075, Sem 0.0075 LR: 2.278e-02 Score 99.729 Data time: 0.2181, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6191 [2021-07-09 02:05:20,281][root][INFO] - ===> Epoch[9682](48410/5): Loss 0.0073, Sem 0.0073 LR: 2.277e-02 Score 99.738 Data time: 0.2173, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.6679 [2021-07-09 02:05:35,025][root][INFO] - ===> Epoch[9683](48415/5): Loss 0.0078, Sem 0.0078 LR: 2.276e-02 Score 99.720 Data time: 0.2042, Forward time: 1.2528, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.9085 [2021-07-09 02:05:49,395][root][INFO] - ===> Epoch[9684](48420/5): Loss 0.0075, Sem 0.0075 LR: 2.275e-02 Score 99.731 Data time: 0.1978, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8378 [2021-07-09 02:06:03,173][root][INFO] - ===> Epoch[9685](48425/5): Loss 0.0079, Sem 0.0079 LR: 2.274e-02 Score 99.716 Data time: 0.1792, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7045 [2021-07-09 02:06:17,907][root][INFO] - ===> Epoch[9686](48430/5): Loss 0.0077, Sem 0.0077 LR: 2.273e-02 Score 99.726 Data time: 0.2895, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.9057 [2021-07-09 02:06:32,971][root][INFO] - ===> Epoch[9687](48435/5): Loss 0.0074, Sem 0.0074 LR: 2.273e-02 Score 99.736 Data time: 0.1817, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.9638 [2021-07-09 02:06:46,783][root][INFO] - ===> Epoch[9688](48440/5): Loss 0.0073, Sem 0.0073 LR: 2.272e-02 Score 99.739 Data time: 0.1573, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7305 [2021-07-09 02:07:00,827][root][INFO] - ===> Epoch[9689](48445/5): Loss 0.0077, Sem 0.0077 LR: 2.271e-02 Score 99.725 Data time: 0.2568, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7709 [2021-07-09 02:07:14,707][root][INFO] - ===> Epoch[9690](48450/5): Loss 0.0075, Sem 0.0075 LR: 2.270e-02 Score 99.729 Data time: 0.2280, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7263 [2021-07-09 02:07:28,757][root][INFO] - ===> Epoch[9691](48455/5): Loss 0.0076, Sem 0.0076 LR: 2.269e-02 Score 99.728 Data time: 0.2232, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7574 [2021-07-09 02:07:43,552][root][INFO] - ===> Epoch[9692](48460/5): Loss 0.0076, Sem 0.0076 LR: 2.268e-02 Score 99.730 Data time: 0.2097, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.9213 [2021-07-09 02:07:57,466][root][INFO] - ===> Epoch[9693](48465/5): Loss 0.0075, Sem 0.0075 LR: 2.267e-02 Score 99.731 Data time: 0.2460, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7295 [2021-07-09 02:08:11,811][root][INFO] - ===> Epoch[9694](48470/5): Loss 0.0076, Sem 0.0076 LR: 2.266e-02 Score 99.726 Data time: 0.2215, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8284 [2021-07-09 02:08:25,423][root][INFO] - ===> Epoch[9695](48475/5): Loss 0.0076, Sem 0.0076 LR: 2.266e-02 Score 99.728 Data time: 0.2209, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6805 [2021-07-09 02:08:40,335][root][INFO] - ===> Epoch[9696](48480/5): Loss 0.0073, Sem 0.0073 LR: 2.265e-02 Score 99.739 Data time: 0.2386, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.9268 [2021-07-09 02:08:54,612][root][INFO] - ===> Epoch[9697](48485/5): Loss 0.0071, Sem 0.0071 LR: 2.264e-02 Score 99.745 Data time: 0.1358, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8120 [2021-07-09 02:09:09,308][root][INFO] - ===> Epoch[9698](48490/5): Loss 0.0074, Sem 0.0074 LR: 2.263e-02 Score 99.737 Data time: 0.2042, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8982 [2021-07-09 02:09:23,455][root][INFO] - ===> Epoch[9699](48495/5): Loss 0.0075, Sem 0.0075 LR: 2.262e-02 Score 99.731 Data time: 0.2307, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7861 [2021-07-09 02:09:37,310][root][INFO] - ===> Epoch[9700](48500/5): Loss 0.0079, Sem 0.0079 LR: 2.261e-02 Score 99.716 Data time: 0.2081, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7371 [2021-07-09 02:09:51,936][root][INFO] - ===> Epoch[9701](48505/5): Loss 0.0076, Sem 0.0076 LR: 2.260e-02 Score 99.728 Data time: 0.2493, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8826 [2021-07-09 02:10:06,066][root][INFO] - ===> Epoch[9702](48510/5): Loss 0.0074, Sem 0.0074 LR: 2.259e-02 Score 99.733 Data time: 0.1750, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7798 [2021-07-09 02:10:21,279][root][INFO] - ===> Epoch[9703](48515/5): Loss 0.0074, Sem 0.0074 LR: 2.258e-02 Score 99.734 Data time: 0.2506, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.9831 [2021-07-09 02:10:35,752][root][INFO] - ===> Epoch[9704](48520/5): Loss 0.0075, Sem 0.0075 LR: 2.258e-02 Score 99.731 Data time: 0.1328, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8584 [2021-07-09 02:10:50,516][root][INFO] - ===> Epoch[9705](48525/5): Loss 0.0073, Sem 0.0073 LR: 2.257e-02 Score 99.740 Data time: 0.2118, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.8954 [2021-07-09 02:11:04,850][root][INFO] - ===> Epoch[9706](48530/5): Loss 0.0076, Sem 0.0076 LR: 2.256e-02 Score 99.728 Data time: 0.2251, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.8180 [2021-07-09 02:11:18,883][root][INFO] - ===> Epoch[9707](48535/5): Loss 0.0076, Sem 0.0076 LR: 2.255e-02 Score 99.726 Data time: 0.1786, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.7590 [2021-07-09 02:11:32,819][root][INFO] - ===> Epoch[9708](48540/5): Loss 0.0079, Sem 0.0079 LR: 2.254e-02 Score 99.718 Data time: 0.2341, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.7429 [2021-07-09 02:11:45,898][root][INFO] - ===> Epoch[9709](48545/5): Loss 0.0075, Sem 0.0075 LR: 2.253e-02 Score 99.730 Data time: 0.2135, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.5606 [2021-07-09 02:11:59,773][root][INFO] - ===> Epoch[9710](48550/5): Loss 0.0077, Sem 0.0077 LR: 2.252e-02 Score 99.724 Data time: 0.1485, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7444 [2021-07-09 02:12:15,258][root][INFO] - ===> Epoch[9711](48555/5): Loss 0.0076, Sem 0.0076 LR: 2.251e-02 Score 99.727 Data time: 0.3399, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 3.0392 [2021-07-09 02:12:30,171][root][INFO] - ===> Epoch[9712](48560/5): Loss 0.0074, Sem 0.0074 LR: 2.250e-02 Score 99.734 Data time: 0.2588, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.9364 [2021-07-09 02:12:44,210][root][INFO] - ===> Epoch[9713](48565/5): Loss 0.0074, Sem 0.0074 LR: 2.250e-02 Score 99.736 Data time: 0.1972, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7751 [2021-07-09 02:12:58,305][root][INFO] - ===> Epoch[9714](48570/5): Loss 0.0076, Sem 0.0076 LR: 2.249e-02 Score 99.725 Data time: 0.2133, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7722 [2021-07-09 02:13:11,295][root][INFO] - ===> Epoch[9715](48575/5): Loss 0.0077, Sem 0.0077 LR: 2.248e-02 Score 99.725 Data time: 0.2788, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.5485 [2021-07-09 02:13:26,016][root][INFO] - ===> Epoch[9716](48580/5): Loss 0.0071, Sem 0.0071 LR: 2.247e-02 Score 99.747 Data time: 0.2496, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8983 [2021-07-09 02:13:40,222][root][INFO] - ===> Epoch[9717](48585/5): Loss 0.0076, Sem 0.0076 LR: 2.246e-02 Score 99.725 Data time: 0.2740, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7808 [2021-07-09 02:13:54,345][root][INFO] - ===> Epoch[9718](48590/5): Loss 0.0073, Sem 0.0073 LR: 2.245e-02 Score 99.738 Data time: 0.2908, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7671 [2021-07-09 02:14:07,732][root][INFO] - ===> Epoch[9719](48595/5): Loss 0.0077, Sem 0.0077 LR: 2.244e-02 Score 99.724 Data time: 0.2063, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6267 [2021-07-09 02:14:20,901][root][INFO] - ===> Epoch[9720](48600/5): Loss 0.0079, Sem 0.0079 LR: 2.243e-02 Score 99.717 Data time: 0.2245, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.5909 [2021-07-09 02:14:20,903][root][INFO] - ===> Start testing [2021-07-09 02:15:00,570][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1249 Loss 0.435 (AVG: 0.608) Score 91.072 (AVG: 91.149) mIOU 71.490 mAP 82.772 mAcc 77.767 IOU: 64.995 0.000 82.371 75.378 95.191 91.954 28.097 84.190 98.584 83.327 80.185 86.211 58.888 mAP: 84.381 3.325 89.439 86.476 98.360 98.093 50.340 97.930 99.645 94.535 90.999 92.551 89.964 mAcc: 79.443 0.000 86.295 83.715 96.800 97.501 30.368 95.228 99.489 92.946 90.313 96.594 62.283 [2021-07-09 02:15:00,570][root][INFO] - Finished test. Elapsed time: 39.6659 [2021-07-09 02:15:01,610][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 02:15:14,167][root][INFO] - ===> Epoch[9721](48605/5): Loss 0.0073, Sem 0.0073 LR: 2.243e-02 Score 99.736 Data time: 0.1998, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.4703 [2021-07-09 02:15:27,971][root][INFO] - ===> Epoch[9722](48610/5): Loss 0.0077, Sem 0.0077 LR: 2.242e-02 Score 99.723 Data time: 0.2161, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7116 [2021-07-09 02:15:42,363][root][INFO] - ===> Epoch[9723](48615/5): Loss 0.0079, Sem 0.0079 LR: 2.241e-02 Score 99.717 Data time: 0.2378, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8336 [2021-07-09 02:15:56,567][root][INFO] - ===> Epoch[9724](48620/5): Loss 0.0075, Sem 0.0075 LR: 2.240e-02 Score 99.733 Data time: 0.2581, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7830 [2021-07-09 02:16:10,872][root][INFO] - ===> Epoch[9725](48625/5): Loss 0.0075, Sem 0.0075 LR: 2.239e-02 Score 99.732 Data time: 0.2217, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8265 [2021-07-09 02:16:25,000][root][INFO] - ===> Epoch[9726](48630/5): Loss 0.0072, Sem 0.0072 LR: 2.238e-02 Score 99.743 Data time: 0.2071, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7728 [2021-07-09 02:16:38,464][root][INFO] - ===> Epoch[9727](48635/5): Loss 0.0071, Sem 0.0071 LR: 2.237e-02 Score 99.745 Data time: 0.2035, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6484 [2021-07-09 02:16:52,810][root][INFO] - ===> Epoch[9728](48640/5): Loss 0.0074, Sem 0.0074 LR: 2.236e-02 Score 99.735 Data time: 0.2162, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8310 [2021-07-09 02:17:06,560][root][INFO] - ===> Epoch[9729](48645/5): Loss 0.0078, Sem 0.0078 LR: 2.235e-02 Score 99.721 Data time: 0.2231, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7017 [2021-07-09 02:17:20,432][root][INFO] - ===> Epoch[9730](48650/5): Loss 0.0074, Sem 0.0074 LR: 2.235e-02 Score 99.738 Data time: 0.1995, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7269 [2021-07-09 02:17:34,226][root][INFO] - ===> Epoch[9731](48655/5): Loss 0.0079, Sem 0.0079 LR: 2.234e-02 Score 99.716 Data time: 0.2115, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7181 [2021-07-09 02:17:48,236][root][INFO] - ===> Epoch[9732](48660/5): Loss 0.0076, Sem 0.0076 LR: 2.233e-02 Score 99.725 Data time: 0.2117, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7662 [2021-07-09 02:18:01,552][root][INFO] - ===> Epoch[9733](48665/5): Loss 0.0078, Sem 0.0078 LR: 2.232e-02 Score 99.723 Data time: 0.2196, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6188 [2021-07-09 02:18:14,792][root][INFO] - ===> Epoch[9734](48670/5): Loss 0.0073, Sem 0.0073 LR: 2.231e-02 Score 99.738 Data time: 0.1844, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6129 [2021-07-09 02:18:30,414][root][INFO] - ===> Epoch[9735](48675/5): Loss 0.0075, Sem 0.0075 LR: 2.230e-02 Score 99.732 Data time: 0.2358, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 3.0711 [2021-07-09 02:18:43,522][root][INFO] - ===> Epoch[9736](48680/5): Loss 0.0075, Sem 0.0075 LR: 2.229e-02 Score 99.733 Data time: 0.2643, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.5798 [2021-07-09 02:18:58,358][root][INFO] - ===> Epoch[9737](48685/5): Loss 0.0072, Sem 0.0072 LR: 2.228e-02 Score 99.742 Data time: 0.2029, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.9216 [2021-07-09 02:19:12,681][root][INFO] - ===> Epoch[9738](48690/5): Loss 0.0073, Sem 0.0073 LR: 2.227e-02 Score 99.737 Data time: 0.2332, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8127 [2021-07-09 02:19:26,434][root][INFO] - ===> Epoch[9739](48695/5): Loss 0.0075, Sem 0.0075 LR: 2.227e-02 Score 99.730 Data time: 0.2233, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7133 [2021-07-09 02:19:40,956][root][INFO] - ===> Epoch[9740](48700/5): Loss 0.0079, Sem 0.0079 LR: 2.226e-02 Score 99.720 Data time: 0.1976, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8552 [2021-07-09 02:19:54,643][root][INFO] - ===> Epoch[9741](48705/5): Loss 0.0072, Sem 0.0072 LR: 2.225e-02 Score 99.739 Data time: 0.2290, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6932 [2021-07-09 02:20:08,256][root][INFO] - ===> Epoch[9742](48710/5): Loss 0.0075, Sem 0.0075 LR: 2.224e-02 Score 99.731 Data time: 0.2351, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6826 [2021-07-09 02:20:21,971][root][INFO] - ===> Epoch[9743](48715/5): Loss 0.0075, Sem 0.0075 LR: 2.223e-02 Score 99.729 Data time: 0.1778, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6973 [2021-07-09 02:20:36,400][root][INFO] - ===> Epoch[9744](48720/5): Loss 0.0079, Sem 0.0079 LR: 2.222e-02 Score 99.716 Data time: 0.2057, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8385 [2021-07-09 02:20:50,720][root][INFO] - ===> Epoch[9745](48725/5): Loss 0.0074, Sem 0.0074 LR: 2.221e-02 Score 99.734 Data time: 0.2750, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8178 [2021-07-09 02:21:04,504][root][INFO] - ===> Epoch[9746](48730/5): Loss 0.0075, Sem 0.0075 LR: 2.220e-02 Score 99.731 Data time: 0.2135, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7183 [2021-07-09 02:21:17,855][root][INFO] - ===> Epoch[9747](48735/5): Loss 0.0073, Sem 0.0073 LR: 2.219e-02 Score 99.740 Data time: 0.2228, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6245 [2021-07-09 02:21:32,381][root][INFO] - ===> Epoch[9748](48740/5): Loss 0.0081, Sem 0.0081 LR: 2.219e-02 Score 99.711 Data time: 0.2669, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8585 [2021-07-09 02:21:45,865][root][INFO] - ===> Epoch[9749](48745/5): Loss 0.0077, Sem 0.0077 LR: 2.218e-02 Score 99.729 Data time: 0.2375, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6552 [2021-07-09 02:21:59,658][root][INFO] - ===> Epoch[9750](48750/5): Loss 0.0075, Sem 0.0075 LR: 2.217e-02 Score 99.731 Data time: 0.2341, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7109 [2021-07-09 02:22:13,784][root][INFO] - ===> Epoch[9751](48755/5): Loss 0.0074, Sem 0.0074 LR: 2.216e-02 Score 99.735 Data time: 0.2169, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7818 [2021-07-09 02:22:28,270][root][INFO] - ===> Epoch[9752](48760/5): Loss 0.0071, Sem 0.0071 LR: 2.215e-02 Score 99.743 Data time: 0.1987, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8402 [2021-07-09 02:22:42,271][root][INFO] - ===> Epoch[9753](48765/5): Loss 0.0078, Sem 0.0078 LR: 2.214e-02 Score 99.724 Data time: 0.2471, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7609 [2021-07-09 02:22:55,895][root][INFO] - ===> Epoch[9754](48770/5): Loss 0.0075, Sem 0.0075 LR: 2.213e-02 Score 99.731 Data time: 0.1770, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6746 [2021-07-09 02:23:08,946][root][INFO] - ===> Epoch[9755](48775/5): Loss 0.0076, Sem 0.0076 LR: 2.212e-02 Score 99.727 Data time: 0.1906, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.5660 [2021-07-09 02:23:22,617][root][INFO] - ===> Epoch[9756](48780/5): Loss 0.0073, Sem 0.0073 LR: 2.211e-02 Score 99.738 Data time: 0.2245, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6702 [2021-07-09 02:23:36,326][root][INFO] - ===> Epoch[9757](48785/5): Loss 0.0074, Sem 0.0074 LR: 2.211e-02 Score 99.735 Data time: 0.2068, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6907 [2021-07-09 02:23:51,377][root][INFO] - ===> Epoch[9758](48790/5): Loss 0.0073, Sem 0.0073 LR: 2.210e-02 Score 99.735 Data time: 0.2398, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.9676 [2021-07-09 02:24:05,577][root][INFO] - ===> Epoch[9759](48795/5): Loss 0.0075, Sem 0.0075 LR: 2.209e-02 Score 99.733 Data time: 0.2286, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7855 [2021-07-09 02:24:18,784][root][INFO] - ===> Epoch[9760](48800/5): Loss 0.0073, Sem 0.0073 LR: 2.208e-02 Score 99.737 Data time: 0.2172, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.5900 [2021-07-09 02:24:18,786][root][INFO] - ===> Start testing [2021-07-09 02:24:57,497][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1253 Loss 0.389 (AVG: 0.618) Score 92.640 (AVG: 91.070) mIOU 71.248 mAP 82.444 mAcc 77.324 IOU: 64.721 0.000 82.337 75.300 95.034 91.778 25.737 84.187 98.581 83.460 80.215 86.153 58.719 mAP: 83.829 4.187 89.812 85.864 98.386 98.160 49.131 97.565 99.709 92.448 90.830 92.389 89.458 mAcc: 79.796 0.000 85.655 83.629 96.752 97.674 27.654 93.662 99.413 91.900 90.436 96.692 61.946 [2021-07-09 02:24:57,497][root][INFO] - Finished test. Elapsed time: 38.7109 [2021-07-09 02:24:58,612][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 02:25:11,951][root][INFO] - ===> Epoch[9761](48805/5): Loss 0.0076, Sem 0.0076 LR: 2.207e-02 Score 99.727 Data time: 0.2320, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6169 [2021-07-09 02:25:27,157][root][INFO] - ===> Epoch[9762](48810/5): Loss 0.0074, Sem 0.0074 LR: 2.206e-02 Score 99.733 Data time: 0.3250, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.9797 [2021-07-09 02:25:41,915][root][INFO] - ===> Epoch[9763](48815/5): Loss 0.0076, Sem 0.0076 LR: 2.205e-02 Score 99.727 Data time: 0.2435, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.9035 [2021-07-09 02:25:56,417][root][INFO] - ===> Epoch[9764](48820/5): Loss 0.0077, Sem 0.0077 LR: 2.204e-02 Score 99.722 Data time: 0.2200, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8595 [2021-07-09 02:26:10,213][root][INFO] - ===> Epoch[9765](48825/5): Loss 0.0072, Sem 0.0072 LR: 2.204e-02 Score 99.738 Data time: 0.2115, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7037 [2021-07-09 02:26:25,234][root][INFO] - ===> Epoch[9766](48830/5): Loss 0.0075, Sem 0.0075 LR: 2.203e-02 Score 99.731 Data time: 0.2643, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.9565 [2021-07-09 02:26:39,810][root][INFO] - ===> Epoch[9767](48835/5): Loss 0.0072, Sem 0.0072 LR: 2.202e-02 Score 99.741 Data time: 0.2391, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8717 [2021-07-09 02:26:53,816][root][INFO] - ===> Epoch[9768](48840/5): Loss 0.0076, Sem 0.0076 LR: 2.201e-02 Score 99.731 Data time: 0.2639, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7591 [2021-07-09 02:27:07,652][root][INFO] - ===> Epoch[9769](48845/5): Loss 0.0076, Sem 0.0076 LR: 2.200e-02 Score 99.731 Data time: 0.2619, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7189 [2021-07-09 02:27:21,298][root][INFO] - ===> Epoch[9770](48850/5): Loss 0.0077, Sem 0.0077 LR: 2.199e-02 Score 99.723 Data time: 0.2726, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6782 [2021-07-09 02:27:35,180][root][INFO] - ===> Epoch[9771](48855/5): Loss 0.0076, Sem 0.0076 LR: 2.198e-02 Score 99.732 Data time: 0.2771, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7291 [2021-07-09 02:27:49,049][root][INFO] - ===> Epoch[9772](48860/5): Loss 0.0076, Sem 0.0076 LR: 2.197e-02 Score 99.726 Data time: 0.2082, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7343 [2021-07-09 02:28:03,337][root][INFO] - ===> Epoch[9773](48865/5): Loss 0.0072, Sem 0.0072 LR: 2.196e-02 Score 99.744 Data time: 0.2013, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8009 [2021-07-09 02:28:17,144][root][INFO] - ===> Epoch[9774](48870/5): Loss 0.0076, Sem 0.0076 LR: 2.196e-02 Score 99.725 Data time: 0.2250, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7172 [2021-07-09 02:28:31,538][root][INFO] - ===> Epoch[9775](48875/5): Loss 0.0070, Sem 0.0070 LR: 2.195e-02 Score 99.747 Data time: 0.2258, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8362 [2021-07-09 02:28:45,835][root][INFO] - ===> Epoch[9776](48880/5): Loss 0.0077, Sem 0.0077 LR: 2.194e-02 Score 99.724 Data time: 0.2492, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8098 [2021-07-09 02:28:59,383][root][INFO] - ===> Epoch[9777](48885/5): Loss 0.0074, Sem 0.0074 LR: 2.193e-02 Score 99.734 Data time: 0.2462, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6640 [2021-07-09 02:29:13,054][root][INFO] - ===> Epoch[9778](48890/5): Loss 0.0070, Sem 0.0070 LR: 2.192e-02 Score 99.749 Data time: 0.2555, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6920 [2021-07-09 02:29:27,265][root][INFO] - ===> Epoch[9779](48895/5): Loss 0.0075, Sem 0.0075 LR: 2.191e-02 Score 99.732 Data time: 0.2827, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8015 [2021-07-09 02:29:40,807][root][INFO] - ===> Epoch[9780](48900/5): Loss 0.0072, Sem 0.0072 LR: 2.190e-02 Score 99.741 Data time: 0.2575, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6574 [2021-07-09 02:29:54,470][root][INFO] - ===> Epoch[9781](48905/5): Loss 0.0075, Sem 0.0075 LR: 2.189e-02 Score 99.728 Data time: 0.1929, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6890 [2021-07-09 02:30:09,220][root][INFO] - ===> Epoch[9782](48910/5): Loss 0.0069, Sem 0.0069 LR: 2.188e-02 Score 99.750 Data time: 0.2774, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.9040 [2021-07-09 02:30:23,410][root][INFO] - ===> Epoch[9783](48915/5): Loss 0.0073, Sem 0.0073 LR: 2.188e-02 Score 99.738 Data time: 0.2813, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7925 [2021-07-09 02:30:37,414][root][INFO] - ===> Epoch[9784](48920/5): Loss 0.0075, Sem 0.0075 LR: 2.187e-02 Score 99.733 Data time: 0.1537, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7616 [2021-07-09 02:30:51,179][root][INFO] - ===> Epoch[9785](48925/5): Loss 0.0068, Sem 0.0068 LR: 2.186e-02 Score 99.759 Data time: 0.2707, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7101 [2021-07-09 02:31:04,784][root][INFO] - ===> Epoch[9786](48930/5): Loss 0.0074, Sem 0.0074 LR: 2.185e-02 Score 99.733 Data time: 0.2651, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6687 [2021-07-09 02:31:19,023][root][INFO] - ===> Epoch[9787](48935/5): Loss 0.0071, Sem 0.0071 LR: 2.184e-02 Score 99.745 Data time: 0.2218, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7914 [2021-07-09 02:31:32,207][root][INFO] - ===> Epoch[9788](48940/5): Loss 0.0079, Sem 0.0079 LR: 2.183e-02 Score 99.716 Data time: 0.2177, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6007 [2021-07-09 02:31:46,261][root][INFO] - ===> Epoch[9789](48945/5): Loss 0.0074, Sem 0.0074 LR: 2.182e-02 Score 99.735 Data time: 0.2147, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7592 [2021-07-09 02:32:00,022][root][INFO] - ===> Epoch[9790](48950/5): Loss 0.0075, Sem 0.0075 LR: 2.181e-02 Score 99.732 Data time: 0.1879, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7121 [2021-07-09 02:32:13,661][root][INFO] - ===> Epoch[9791](48955/5): Loss 0.0076, Sem 0.0076 LR: 2.180e-02 Score 99.727 Data time: 0.2269, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6701 [2021-07-09 02:32:27,330][root][INFO] - ===> Epoch[9792](48960/5): Loss 0.0075, Sem 0.0075 LR: 2.180e-02 Score 99.730 Data time: 0.2420, Forward time: 1.2528, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6886 [2021-07-09 02:32:41,301][root][INFO] - ===> Epoch[9793](48965/5): Loss 0.0069, Sem 0.0069 LR: 2.179e-02 Score 99.751 Data time: 0.2218, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7431 [2021-07-09 02:32:55,497][root][INFO] - ===> Epoch[9794](48970/5): Loss 0.0074, Sem 0.0074 LR: 2.178e-02 Score 99.737 Data time: 0.1950, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7951 [2021-07-09 02:33:09,644][root][INFO] - ===> Epoch[9795](48975/5): Loss 0.0074, Sem 0.0074 LR: 2.177e-02 Score 99.735 Data time: 0.2158, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7702 [2021-07-09 02:33:23,472][root][INFO] - ===> Epoch[9796](48980/5): Loss 0.0075, Sem 0.0075 LR: 2.176e-02 Score 99.731 Data time: 0.2504, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7164 [2021-07-09 02:33:37,194][root][INFO] - ===> Epoch[9797](48985/5): Loss 0.0075, Sem 0.0075 LR: 2.175e-02 Score 99.732 Data time: 0.2950, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7027 [2021-07-09 02:33:50,417][root][INFO] - ===> Epoch[9798](48990/5): Loss 0.0073, Sem 0.0073 LR: 2.174e-02 Score 99.739 Data time: 0.1531, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.5967 [2021-07-09 02:34:04,092][root][INFO] - ===> Epoch[9799](48995/5): Loss 0.0075, Sem 0.0075 LR: 2.173e-02 Score 99.734 Data time: 0.2183, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.6857 [2021-07-09 02:34:18,265][root][INFO] - ===> Epoch[9800](49000/5): Loss 0.0072, Sem 0.0072 LR: 2.172e-02 Score 99.741 Data time: 0.2847, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7831 [2021-07-09 02:34:18,266][root][INFO] - ===> Start testing [2021-07-09 02:34:58,017][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1364 Loss 0.457 (AVG: 0.620) Score 90.844 (AVG: 91.070) mIOU 71.359 mAP 82.799 mAcc 77.943 IOU: 64.153 0.000 82.667 75.221 95.104 91.314 30.326 83.885 98.591 81.503 80.020 86.296 58.587 mAP: 83.587 2.893 91.209 86.058 98.413 98.070 53.045 98.099 99.633 92.317 90.961 92.795 89.310 mAcc: 78.498 0.000 86.541 82.967 96.890 97.782 33.136 95.284 99.448 92.985 90.916 96.627 62.188 [2021-07-09 02:34:58,017][root][INFO] - Finished test. Elapsed time: 39.7500 [2021-07-09 02:34:59,078][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 02:35:13,061][root][INFO] - ===> Epoch[9801](49005/5): Loss 0.0075, Sem 0.0075 LR: 2.172e-02 Score 99.733 Data time: 0.1965, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7574 [2021-07-09 02:35:26,014][root][INFO] - ===> Epoch[9802](49010/5): Loss 0.0072, Sem 0.0072 LR: 2.171e-02 Score 99.741 Data time: 0.1678, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.5540 [2021-07-09 02:35:40,059][root][INFO] - ===> Epoch[9803](49015/5): Loss 0.0075, Sem 0.0075 LR: 2.170e-02 Score 99.729 Data time: 0.2039, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7709 [2021-07-09 02:35:54,241][root][INFO] - ===> Epoch[9804](49020/5): Loss 0.0074, Sem 0.0074 LR: 2.169e-02 Score 99.737 Data time: 0.2792, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7852 [2021-07-09 02:36:08,056][root][INFO] - ===> Epoch[9805](49025/5): Loss 0.0073, Sem 0.0073 LR: 2.168e-02 Score 99.741 Data time: 0.1927, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7130 [2021-07-09 02:36:21,879][root][INFO] - ===> Epoch[9806](49030/5): Loss 0.0077, Sem 0.0077 LR: 2.167e-02 Score 99.725 Data time: 0.2200, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7154 [2021-07-09 02:36:35,085][root][INFO] - ===> Epoch[9807](49035/5): Loss 0.0073, Sem 0.0073 LR: 2.166e-02 Score 99.740 Data time: 0.2127, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.6023 [2021-07-09 02:36:49,698][root][INFO] - ===> Epoch[9808](49040/5): Loss 0.0073, Sem 0.0073 LR: 2.165e-02 Score 99.739 Data time: 0.2123, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8617 [2021-07-09 02:37:04,024][root][INFO] - ===> Epoch[9809](49045/5): Loss 0.0073, Sem 0.0073 LR: 2.164e-02 Score 99.741 Data time: 0.2262, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8122 [2021-07-09 02:37:19,102][root][INFO] - ===> Epoch[9810](49050/5): Loss 0.0076, Sem 0.0076 LR: 2.164e-02 Score 99.729 Data time: 0.2564, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.9623 [2021-07-09 02:37:33,400][root][INFO] - ===> Epoch[9811](49055/5): Loss 0.0071, Sem 0.0071 LR: 2.163e-02 Score 99.743 Data time: 0.3006, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8146 [2021-07-09 02:37:47,567][root][INFO] - ===> Epoch[9812](49060/5): Loss 0.0076, Sem 0.0076 LR: 2.162e-02 Score 99.727 Data time: 0.2923, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7895 [2021-07-09 02:38:02,882][root][INFO] - ===> Epoch[9813](49065/5): Loss 0.0071, Sem 0.0071 LR: 2.161e-02 Score 99.746 Data time: 0.2209, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 3.0233 [2021-07-09 02:38:17,202][root][INFO] - ===> Epoch[9814](49070/5): Loss 0.0077, Sem 0.0077 LR: 2.160e-02 Score 99.725 Data time: 0.3300, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8126 [2021-07-09 02:38:31,097][root][INFO] - ===> Epoch[9815](49075/5): Loss 0.0075, Sem 0.0075 LR: 2.159e-02 Score 99.733 Data time: 0.3163, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7381 [2021-07-09 02:38:45,785][root][INFO] - ===> Epoch[9816](49080/5): Loss 0.0074, Sem 0.0074 LR: 2.158e-02 Score 99.736 Data time: 0.2399, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8982 [2021-07-09 02:39:00,787][root][INFO] - ===> Epoch[9817](49085/5): Loss 0.0073, Sem 0.0073 LR: 2.157e-02 Score 99.734 Data time: 0.2554, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.9525 [2021-07-09 02:39:16,374][root][INFO] - ===> Epoch[9818](49090/5): Loss 0.0073, Sem 0.0073 LR: 2.156e-02 Score 99.737 Data time: 0.2714, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 3.0698 [2021-07-09 02:39:30,896][root][INFO] - ===> Epoch[9819](49095/5): Loss 0.0071, Sem 0.0071 LR: 2.156e-02 Score 99.742 Data time: 0.2329, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8556 [2021-07-09 02:39:44,641][root][INFO] - ===> Epoch[9820](49100/5): Loss 0.0074, Sem 0.0074 LR: 2.155e-02 Score 99.736 Data time: 0.2654, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.6944 [2021-07-09 02:39:59,171][root][INFO] - ===> Epoch[9821](49105/5): Loss 0.0072, Sem 0.0072 LR: 2.154e-02 Score 99.739 Data time: 0.2240, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8585 [2021-07-09 02:40:13,135][root][INFO] - ===> Epoch[9822](49110/5): Loss 0.0074, Sem 0.0074 LR: 2.153e-02 Score 99.732 Data time: 0.3003, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7483 [2021-07-09 02:40:26,968][root][INFO] - ===> Epoch[9823](49115/5): Loss 0.0073, Sem 0.0073 LR: 2.152e-02 Score 99.736 Data time: 0.2583, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7254 [2021-07-09 02:40:40,683][root][INFO] - ===> Epoch[9824](49120/5): Loss 0.0077, Sem 0.0077 LR: 2.151e-02 Score 99.728 Data time: 0.1969, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.6980 [2021-07-09 02:40:53,851][root][INFO] - ===> Epoch[9825](49125/5): Loss 0.0073, Sem 0.0073 LR: 2.150e-02 Score 99.735 Data time: 0.2282, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.5866 [2021-07-09 02:41:07,395][root][INFO] - ===> Epoch[9826](49130/5): Loss 0.0077, Sem 0.0077 LR: 2.149e-02 Score 99.725 Data time: 0.1862, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.6730 [2021-07-09 02:41:21,265][root][INFO] - ===> Epoch[9827](49135/5): Loss 0.0070, Sem 0.0070 LR: 2.148e-02 Score 99.749 Data time: 0.1508, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7347 [2021-07-09 02:41:35,152][root][INFO] - ===> Epoch[9828](49140/5): Loss 0.0076, Sem 0.0076 LR: 2.148e-02 Score 99.729 Data time: 0.2169, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7356 [2021-07-09 02:41:49,335][root][INFO] - ===> Epoch[9829](49145/5): Loss 0.0074, Sem 0.0074 LR: 2.147e-02 Score 99.736 Data time: 0.2832, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7758 [2021-07-09 02:42:03,121][root][INFO] - ===> Epoch[9830](49150/5): Loss 0.0070, Sem 0.0070 LR: 2.146e-02 Score 99.748 Data time: 0.1858, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7087 [2021-07-09 02:42:17,561][root][INFO] - ===> Epoch[9831](49155/5): Loss 0.0072, Sem 0.0072 LR: 2.145e-02 Score 99.742 Data time: 0.2523, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8278 [2021-07-09 02:42:31,749][root][INFO] - ===> Epoch[9832](49160/5): Loss 0.0074, Sem 0.0074 LR: 2.144e-02 Score 99.735 Data time: 0.2640, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7924 [2021-07-09 02:42:45,432][root][INFO] - ===> Epoch[9833](49165/5): Loss 0.0077, Sem 0.0077 LR: 2.143e-02 Score 99.724 Data time: 0.2509, Forward time: 1.2529, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.6791 [2021-07-09 02:42:58,817][root][INFO] - ===> Epoch[9834](49170/5): Loss 0.0076, Sem 0.0076 LR: 2.142e-02 Score 99.726 Data time: 0.2279, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6391 [2021-07-09 02:43:12,121][root][INFO] - ===> Epoch[9835](49175/5): Loss 0.0075, Sem 0.0075 LR: 2.141e-02 Score 99.737 Data time: 0.2388, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6061 [2021-07-09 02:43:26,119][root][INFO] - ===> Epoch[9836](49180/5): Loss 0.0074, Sem 0.0074 LR: 2.140e-02 Score 99.735 Data time: 0.2028, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7577 [2021-07-09 02:43:39,874][root][INFO] - ===> Epoch[9837](49185/5): Loss 0.0073, Sem 0.0073 LR: 2.140e-02 Score 99.735 Data time: 0.2281, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6997 [2021-07-09 02:43:53,824][root][INFO] - ===> Epoch[9838](49190/5): Loss 0.0080, Sem 0.0080 LR: 2.139e-02 Score 99.718 Data time: 0.2045, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7370 [2021-07-09 02:44:07,189][root][INFO] - ===> Epoch[9839](49195/5): Loss 0.0075, Sem 0.0075 LR: 2.138e-02 Score 99.732 Data time: 0.2456, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6312 [2021-07-09 02:44:20,959][root][INFO] - ===> Epoch[9840](49200/5): Loss 0.0074, Sem 0.0074 LR: 2.137e-02 Score 99.736 Data time: 0.2225, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7052 [2021-07-09 02:44:20,962][root][INFO] - ===> Start testing [2021-07-09 02:44:58,181][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1573 Loss 0.423 (AVG: 0.616) Score 91.101 (AVG: 91.096) mIOU 71.610 mAP 82.861 mAcc 77.846 IOU: 64.581 0.000 81.745 74.686 95.134 91.940 31.225 83.552 98.606 84.864 80.090 86.322 58.181 mAP: 84.252 3.045 89.504 85.915 98.354 98.147 53.430 97.909 99.686 93.517 90.891 92.798 89.749 mAcc: 79.071 0.000 84.999 83.963 96.664 97.723 34.049 95.937 99.385 92.433 89.885 96.480 61.415 [2021-07-09 02:44:58,182][root][INFO] - Finished test. Elapsed time: 37.2195 [2021-07-09 02:44:59,259][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 02:45:12,496][root][INFO] - ===> Epoch[9841](49205/5): Loss 0.0074, Sem 0.0074 LR: 2.136e-02 Score 99.734 Data time: 0.1500, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6166 [2021-07-09 02:45:26,307][root][INFO] - ===> Epoch[9842](49210/5): Loss 0.0073, Sem 0.0073 LR: 2.135e-02 Score 99.737 Data time: 0.1891, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7195 [2021-07-09 02:45:40,817][root][INFO] - ===> Epoch[9843](49215/5): Loss 0.0075, Sem 0.0075 LR: 2.134e-02 Score 99.731 Data time: 0.2514, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8593 [2021-07-09 02:45:54,580][root][INFO] - ===> Epoch[9844](49220/5): Loss 0.0078, Sem 0.0078 LR: 2.133e-02 Score 99.722 Data time: 0.2320, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7061 [2021-07-09 02:46:08,892][root][INFO] - ===> Epoch[9845](49225/5): Loss 0.0072, Sem 0.0072 LR: 2.132e-02 Score 99.741 Data time: 0.2264, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8186 [2021-07-09 02:46:22,059][root][INFO] - ===> Epoch[9846](49230/5): Loss 0.0073, Sem 0.0073 LR: 2.132e-02 Score 99.736 Data time: 0.2313, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.5980 [2021-07-09 02:46:36,703][root][INFO] - ===> Epoch[9847](49235/5): Loss 0.0072, Sem 0.0072 LR: 2.131e-02 Score 99.743 Data time: 0.3092, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8706 [2021-07-09 02:46:51,009][root][INFO] - ===> Epoch[9848](49240/5): Loss 0.0074, Sem 0.0074 LR: 2.130e-02 Score 99.736 Data time: 0.2410, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8096 [2021-07-09 02:47:05,174][root][INFO] - ===> Epoch[9849](49245/5): Loss 0.0078, Sem 0.0078 LR: 2.129e-02 Score 99.721 Data time: 0.2386, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7858 [2021-07-09 02:47:19,053][root][INFO] - ===> Epoch[9850](49250/5): Loss 0.0078, Sem 0.0078 LR: 2.128e-02 Score 99.719 Data time: 0.2903, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7335 [2021-07-09 02:47:33,306][root][INFO] - ===> Epoch[9851](49255/5): Loss 0.0074, Sem 0.0074 LR: 2.127e-02 Score 99.737 Data time: 0.2412, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7907 [2021-07-09 02:47:48,140][root][INFO] - ===> Epoch[9852](49260/5): Loss 0.0072, Sem 0.0072 LR: 2.126e-02 Score 99.741 Data time: 0.2337, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.9178 [2021-07-09 02:48:02,401][root][INFO] - ===> Epoch[9853](49265/5): Loss 0.0073, Sem 0.0073 LR: 2.125e-02 Score 99.739 Data time: 0.2304, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7999 [2021-07-09 02:48:16,143][root][INFO] - ===> Epoch[9854](49270/5): Loss 0.0075, Sem 0.0075 LR: 2.124e-02 Score 99.732 Data time: 0.2418, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6952 [2021-07-09 02:48:29,125][root][INFO] - ===> Epoch[9855](49275/5): Loss 0.0074, Sem 0.0074 LR: 2.123e-02 Score 99.731 Data time: 0.1920, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.5622 [2021-07-09 02:48:44,379][root][INFO] - ===> Epoch[9856](49280/5): Loss 0.0070, Sem 0.0070 LR: 2.123e-02 Score 99.749 Data time: 0.2419, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 3.0062 [2021-07-09 02:48:59,123][root][INFO] - ===> Epoch[9857](49285/5): Loss 0.0074, Sem 0.0074 LR: 2.122e-02 Score 99.734 Data time: 0.1958, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.9032 [2021-07-09 02:49:12,505][root][INFO] - ===> Epoch[9858](49290/5): Loss 0.0076, Sem 0.0076 LR: 2.121e-02 Score 99.727 Data time: 0.2564, Forward time: 1.2529, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6220 [2021-07-09 02:49:26,796][root][INFO] - ===> Epoch[9859](49295/5): Loss 0.0076, Sem 0.0076 LR: 2.120e-02 Score 99.728 Data time: 0.2335, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8185 [2021-07-09 02:49:40,824][root][INFO] - ===> Epoch[9860](49300/5): Loss 0.0075, Sem 0.0075 LR: 2.119e-02 Score 99.734 Data time: 0.2060, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7615 [2021-07-09 02:49:55,051][root][INFO] - ===> Epoch[9861](49305/5): Loss 0.0072, Sem 0.0072 LR: 2.118e-02 Score 99.741 Data time: 0.2176, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7949 [2021-07-09 02:50:08,970][root][INFO] - ===> Epoch[9862](49310/5): Loss 0.0072, Sem 0.0072 LR: 2.117e-02 Score 99.741 Data time: 0.2201, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7315 [2021-07-09 02:50:23,107][root][INFO] - ===> Epoch[9863](49315/5): Loss 0.0074, Sem 0.0074 LR: 2.116e-02 Score 99.736 Data time: 0.2680, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7759 [2021-07-09 02:50:38,100][root][INFO] - ===> Epoch[9864](49320/5): Loss 0.0074, Sem 0.0074 LR: 2.115e-02 Score 99.736 Data time: 0.2301, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.9501 [2021-07-09 02:50:52,202][root][INFO] - ===> Epoch[9865](49325/5): Loss 0.0074, Sem 0.0074 LR: 2.115e-02 Score 99.735 Data time: 0.2586, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7843 [2021-07-09 02:51:06,227][root][INFO] - ===> Epoch[9866](49330/5): Loss 0.0073, Sem 0.0073 LR: 2.114e-02 Score 99.738 Data time: 0.2900, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7584 [2021-07-09 02:51:21,620][root][INFO] - ===> Epoch[9867](49335/5): Loss 0.0071, Sem 0.0071 LR: 2.113e-02 Score 99.747 Data time: 0.3056, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 3.0176 [2021-07-09 02:51:35,644][root][INFO] - ===> Epoch[9868](49340/5): Loss 0.0075, Sem 0.0075 LR: 2.112e-02 Score 99.734 Data time: 0.2741, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7492 [2021-07-09 02:51:50,032][root][INFO] - ===> Epoch[9869](49345/5): Loss 0.0076, Sem 0.0076 LR: 2.111e-02 Score 99.731 Data time: 0.1553, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8379 [2021-07-09 02:52:04,115][root][INFO] - ===> Epoch[9870](49350/5): Loss 0.0073, Sem 0.0073 LR: 2.110e-02 Score 99.738 Data time: 0.2600, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7726 [2021-07-09 02:52:17,236][root][INFO] - ===> Epoch[9871](49355/5): Loss 0.0072, Sem 0.0072 LR: 2.109e-02 Score 99.742 Data time: 0.2234, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.5806 [2021-07-09 02:52:31,771][root][INFO] - ===> Epoch[9872](49360/5): Loss 0.0073, Sem 0.0073 LR: 2.108e-02 Score 99.741 Data time: 0.2498, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8582 [2021-07-09 02:52:45,224][root][INFO] - ===> Epoch[9873](49365/5): Loss 0.0071, Sem 0.0071 LR: 2.107e-02 Score 99.745 Data time: 0.3055, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6435 [2021-07-09 02:52:58,999][root][INFO] - ===> Epoch[9874](49370/5): Loss 0.0071, Sem 0.0071 LR: 2.107e-02 Score 99.748 Data time: 0.2748, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7063 [2021-07-09 02:53:12,924][root][INFO] - ===> Epoch[9875](49375/5): Loss 0.0079, Sem 0.0079 LR: 2.106e-02 Score 99.722 Data time: 0.2548, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7303 [2021-07-09 02:53:26,941][root][INFO] - ===> Epoch[9876](49380/5): Loss 0.0077, Sem 0.0077 LR: 2.105e-02 Score 99.725 Data time: 0.2797, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7475 [2021-07-09 02:53:40,867][root][INFO] - ===> Epoch[9877](49385/5): Loss 0.0072, Sem 0.0072 LR: 2.104e-02 Score 99.742 Data time: 0.2219, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7438 [2021-07-09 02:53:54,783][root][INFO] - ===> Epoch[9878](49390/5): Loss 0.0071, Sem 0.0071 LR: 2.103e-02 Score 99.744 Data time: 0.1993, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7335 [2021-07-09 02:54:08,299][root][INFO] - ===> Epoch[9879](49395/5): Loss 0.0075, Sem 0.0075 LR: 2.102e-02 Score 99.731 Data time: 0.2406, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6577 [2021-07-09 02:54:21,953][root][INFO] - ===> Epoch[9880](49400/5): Loss 0.0073, Sem 0.0073 LR: 2.101e-02 Score 99.740 Data time: 0.2490, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6878 [2021-07-09 02:54:21,954][root][INFO] - ===> Start testing [2021-07-09 02:55:00,692][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1226 Loss 0.380 (AVG: 0.621) Score 92.502 (AVG: 91.011) mIOU 71.613 mAP 82.914 mAcc 77.987 IOU: 64.204 0.000 81.968 75.066 94.923 91.498 31.903 84.576 98.552 84.764 80.213 86.180 57.125 mAP: 83.433 3.255 90.153 85.892 98.319 98.063 53.490 97.709 99.690 95.497 90.888 92.599 88.893 mAcc: 79.769 0.000 85.844 83.044 96.485 97.541 34.666 95.082 99.333 94.515 90.573 96.498 60.474 [2021-07-09 02:55:00,692][root][INFO] - Finished test. Elapsed time: 38.7372 [2021-07-09 02:55:01,845][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 02:55:14,887][root][INFO] - ===> Epoch[9881](49405/5): Loss 0.0073, Sem 0.0073 LR: 2.100e-02 Score 99.737 Data time: 0.1822, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.5693 [2021-07-09 02:55:29,223][root][INFO] - ===> Epoch[9882](49410/5): Loss 0.0072, Sem 0.0072 LR: 2.099e-02 Score 99.742 Data time: 0.2639, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8202 [2021-07-09 02:55:43,987][root][INFO] - ===> Epoch[9883](49415/5): Loss 0.0071, Sem 0.0071 LR: 2.099e-02 Score 99.747 Data time: 0.1968, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.9113 [2021-07-09 02:55:58,128][root][INFO] - ===> Epoch[9884](49420/5): Loss 0.0070, Sem 0.0070 LR: 2.098e-02 Score 99.751 Data time: 0.2610, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7750 [2021-07-09 02:56:12,225][root][INFO] - ===> Epoch[9885](49425/5): Loss 0.0075, Sem 0.0075 LR: 2.097e-02 Score 99.736 Data time: 0.2358, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7739 [2021-07-09 02:56:26,205][root][INFO] - ===> Epoch[9886](49430/5): Loss 0.0073, Sem 0.0073 LR: 2.096e-02 Score 99.737 Data time: 0.2696, Forward time: 1.2530, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7538 [2021-07-09 02:56:40,119][root][INFO] - ===> Epoch[9887](49435/5): Loss 0.0074, Sem 0.0074 LR: 2.095e-02 Score 99.735 Data time: 0.2612, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7298 [2021-07-09 02:56:54,080][root][INFO] - ===> Epoch[9888](49440/5): Loss 0.0072, Sem 0.0072 LR: 2.094e-02 Score 99.740 Data time: 0.2655, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7528 [2021-07-09 02:57:07,790][root][INFO] - ===> Epoch[9889](49445/5): Loss 0.0074, Sem 0.0074 LR: 2.093e-02 Score 99.735 Data time: 0.2011, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7050 [2021-07-09 02:57:21,703][root][INFO] - ===> Epoch[9890](49450/5): Loss 0.0073, Sem 0.0073 LR: 2.092e-02 Score 99.738 Data time: 0.2269, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7407 [2021-07-09 02:57:35,942][root][INFO] - ===> Epoch[9891](49455/5): Loss 0.0072, Sem 0.0072 LR: 2.091e-02 Score 99.744 Data time: 0.2089, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8111 [2021-07-09 02:57:48,793][root][INFO] - ===> Epoch[9892](49460/5): Loss 0.0076, Sem 0.0076 LR: 2.091e-02 Score 99.726 Data time: 0.2052, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.5287 [2021-07-09 02:58:02,952][root][INFO] - ===> Epoch[9893](49465/5): Loss 0.0074, Sem 0.0074 LR: 2.090e-02 Score 99.736 Data time: 0.1807, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7842 [2021-07-09 02:58:16,936][root][INFO] - ===> Epoch[9894](49470/5): Loss 0.0072, Sem 0.0072 LR: 2.089e-02 Score 99.744 Data time: 0.2595, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7533 [2021-07-09 02:58:30,851][root][INFO] - ===> Epoch[9895](49475/5): Loss 0.0073, Sem 0.0073 LR: 2.088e-02 Score 99.738 Data time: 0.1908, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7405 [2021-07-09 02:58:44,935][root][INFO] - ===> Epoch[9896](49480/5): Loss 0.0072, Sem 0.0072 LR: 2.087e-02 Score 99.744 Data time: 0.1899, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7759 [2021-07-09 02:58:58,839][root][INFO] - ===> Epoch[9897](49485/5): Loss 0.0077, Sem 0.0077 LR: 2.086e-02 Score 99.726 Data time: 0.2842, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7188 [2021-07-09 02:59:12,399][root][INFO] - ===> Epoch[9898](49490/5): Loss 0.0075, Sem 0.0075 LR: 2.085e-02 Score 99.731 Data time: 0.2083, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6609 [2021-07-09 02:59:26,211][root][INFO] - ===> Epoch[9899](49495/5): Loss 0.0076, Sem 0.0076 LR: 2.084e-02 Score 99.728 Data time: 0.3270, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7107 [2021-07-09 02:59:40,792][root][INFO] - ===> Epoch[9900](49500/5): Loss 0.0071, Sem 0.0071 LR: 2.083e-02 Score 99.746 Data time: 0.2000, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8693 [2021-07-09 02:59:54,491][root][INFO] - ===> Epoch[9901](49505/5): Loss 0.0076, Sem 0.0076 LR: 2.082e-02 Score 99.727 Data time: 0.2203, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6938 [2021-07-09 03:00:08,274][root][INFO] - ===> Epoch[9902](49510/5): Loss 0.0072, Sem 0.0072 LR: 2.082e-02 Score 99.742 Data time: 0.2484, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7073 [2021-07-09 03:00:22,362][root][INFO] - ===> Epoch[9903](49515/5): Loss 0.0074, Sem 0.0074 LR: 2.081e-02 Score 99.735 Data time: 0.2429, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7726 [2021-07-09 03:00:36,488][root][INFO] - ===> Epoch[9904](49520/5): Loss 0.0071, Sem 0.0071 LR: 2.080e-02 Score 99.745 Data time: 0.2463, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7723 [2021-07-09 03:00:50,572][root][INFO] - ===> Epoch[9905](49525/5): Loss 0.0073, Sem 0.0073 LR: 2.079e-02 Score 99.740 Data time: 0.2061, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7667 [2021-07-09 03:01:04,678][root][INFO] - ===> Epoch[9906](49530/5): Loss 0.0071, Sem 0.0071 LR: 2.078e-02 Score 99.747 Data time: 0.1600, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7834 [2021-07-09 03:01:18,008][root][INFO] - ===> Epoch[9907](49535/5): Loss 0.0074, Sem 0.0074 LR: 2.077e-02 Score 99.734 Data time: 0.1815, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6215 [2021-07-09 03:01:32,209][root][INFO] - ===> Epoch[9908](49540/5): Loss 0.0074, Sem 0.0074 LR: 2.076e-02 Score 99.732 Data time: 0.1976, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8003 [2021-07-09 03:01:45,480][root][INFO] - ===> Epoch[9909](49545/5): Loss 0.0070, Sem 0.0070 LR: 2.075e-02 Score 99.750 Data time: 0.2553, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.5996 [2021-07-09 03:02:00,044][root][INFO] - ===> Epoch[9910](49550/5): Loss 0.0072, Sem 0.0072 LR: 2.074e-02 Score 99.742 Data time: 0.1948, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8712 [2021-07-09 03:02:14,228][root][INFO] - ===> Epoch[9911](49555/5): Loss 0.0074, Sem 0.0074 LR: 2.074e-02 Score 99.736 Data time: 0.2142, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7805 [2021-07-09 03:02:28,790][root][INFO] - ===> Epoch[9912](49560/5): Loss 0.0075, Sem 0.0075 LR: 2.073e-02 Score 99.733 Data time: 0.2544, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8493 [2021-07-09 03:02:42,525][root][INFO] - ===> Epoch[9913](49565/5): Loss 0.0069, Sem 0.0069 LR: 2.072e-02 Score 99.755 Data time: 0.2366, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6989 [2021-07-09 03:02:56,776][root][INFO] - ===> Epoch[9914](49570/5): Loss 0.0076, Sem 0.0076 LR: 2.071e-02 Score 99.729 Data time: 0.2730, Forward time: 1.2530, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8101 [2021-07-09 03:03:11,486][root][INFO] - ===> Epoch[9915](49575/5): Loss 0.0072, Sem 0.0072 LR: 2.070e-02 Score 99.740 Data time: 0.2112, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.9000 [2021-07-09 03:03:26,358][root][INFO] - ===> Epoch[9916](49580/5): Loss 0.0072, Sem 0.0072 LR: 2.069e-02 Score 99.743 Data time: 0.2476, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.9156 [2021-07-09 03:03:40,265][root][INFO] - ===> Epoch[9917](49585/5): Loss 0.0072, Sem 0.0072 LR: 2.068e-02 Score 99.744 Data time: 0.3305, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7350 [2021-07-09 03:03:54,538][root][INFO] - ===> Epoch[9918](49590/5): Loss 0.0072, Sem 0.0072 LR: 2.067e-02 Score 99.742 Data time: 0.2185, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8040 [2021-07-09 03:04:08,832][root][INFO] - ===> Epoch[9919](49595/5): Loss 0.0075, Sem 0.0075 LR: 2.066e-02 Score 99.731 Data time: 0.3055, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8075 [2021-07-09 03:04:23,116][root][INFO] - ===> Epoch[9920](49600/5): Loss 0.0071, Sem 0.0071 LR: 2.065e-02 Score 99.744 Data time: 0.2358, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7998 [2021-07-09 03:04:23,118][root][INFO] - ===> Start testing [2021-07-09 03:05:00,385][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1327 Loss 0.393 (AVG: 0.615) Score 92.293 (AVG: 91.080) mIOU 71.477 mAP 82.728 mAcc 77.659 IOU: 64.644 0.000 83.060 74.637 95.154 92.126 29.640 82.916 98.608 83.776 80.090 86.413 58.141 mAP: 83.837 3.110 90.900 85.808 98.408 98.091 51.558 97.785 99.681 93.131 91.070 92.333 89.756 mAcc: 80.055 0.000 85.731 82.273 96.869 97.638 31.892 95.080 99.430 91.900 90.379 96.731 61.588 [2021-07-09 03:05:00,385][root][INFO] - Finished test. Elapsed time: 37.2657 [2021-07-09 03:05:01,487][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 03:05:13,306][root][INFO] - ===> Epoch[9921](49605/5): Loss 0.0072, Sem 0.0072 LR: 2.065e-02 Score 99.742 Data time: 0.1819, Forward time: 1.2531, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.3231 [2021-07-09 03:05:28,553][root][INFO] - ===> Epoch[9922](49610/5): Loss 0.0075, Sem 0.0075 LR: 2.064e-02 Score 99.730 Data time: 0.2654, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 3.0000 [2021-07-09 03:05:42,499][root][INFO] - ===> Epoch[9923](49615/5): Loss 0.0073, Sem 0.0073 LR: 2.063e-02 Score 99.737 Data time: 0.2253, Forward time: 1.2531, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7427 [2021-07-09 03:05:56,975][root][INFO] - ===> Epoch[9924](49620/5): Loss 0.0070, Sem 0.0070 LR: 2.062e-02 Score 99.753 Data time: 0.2293, Forward time: 1.2531, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8515 [2021-07-09 03:06:10,679][root][INFO] - ===> Epoch[9925](49625/5): Loss 0.0075, Sem 0.0075 LR: 2.061e-02 Score 99.731 Data time: 0.2160, Forward time: 1.2531, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7011 [2021-07-09 03:06:25,097][root][INFO] - ===> Epoch[9926](49630/5): Loss 0.0076, Sem 0.0076 LR: 2.060e-02 Score 99.726 Data time: 0.2560, Forward time: 1.2531, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8370 [2021-07-09 03:06:39,204][root][INFO] - ===> Epoch[9927](49635/5): Loss 0.0071, Sem 0.0071 LR: 2.059e-02 Score 99.744 Data time: 0.2406, Forward time: 1.2531, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7700 [2021-07-09 03:06:53,382][root][INFO] - ===> Epoch[9928](49640/5): Loss 0.0081, Sem 0.0081 LR: 2.058e-02 Score 99.716 Data time: 0.1754, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7980 [2021-07-09 03:07:07,615][root][INFO] - ===> Epoch[9929](49645/5): Loss 0.0073, Sem 0.0073 LR: 2.057e-02 Score 99.738 Data time: 0.2098, Forward time: 1.2531, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8029 [2021-07-09 03:07:21,153][root][INFO] - ===> Epoch[9930](49650/5): Loss 0.0077, Sem 0.0077 LR: 2.057e-02 Score 99.728 Data time: 0.2635, Forward time: 1.2531, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.6633 [2021-07-09 03:07:35,556][root][INFO] - ===> Epoch[9931](49655/5): Loss 0.0074, Sem 0.0074 LR: 2.056e-02 Score 99.733 Data time: 0.1978, Forward time: 1.2531, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8347 [2021-07-09 03:07:50,311][root][INFO] - ===> Epoch[9932](49660/5): Loss 0.0076, Sem 0.0076 LR: 2.055e-02 Score 99.730 Data time: 0.1948, Forward time: 1.2531, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.9064 [2021-07-09 03:08:04,596][root][INFO] - ===> Epoch[9933](49665/5): Loss 0.0074, Sem 0.0074 LR: 2.054e-02 Score 99.734 Data time: 0.1874, Forward time: 1.2531, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8191 [2021-07-09 03:08:19,080][root][INFO] - ===> Epoch[9934](49670/5): Loss 0.0072, Sem 0.0072 LR: 2.053e-02 Score 99.739 Data time: 0.1643, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8694 [2021-07-09 03:08:33,839][root][INFO] - ===> Epoch[9935](49675/5): Loss 0.0075, Sem 0.0075 LR: 2.052e-02 Score 99.732 Data time: 0.2458, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8973 [2021-07-09 03:08:48,066][root][INFO] - ===> Epoch[9936](49680/5): Loss 0.0075, Sem 0.0075 LR: 2.051e-02 Score 99.731 Data time: 0.1949, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8085 [2021-07-09 03:09:02,054][root][INFO] - ===> Epoch[9937](49685/5): Loss 0.0077, Sem 0.0077 LR: 2.050e-02 Score 99.723 Data time: 0.2282, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7607 [2021-07-09 03:09:16,682][root][INFO] - ===> Epoch[9938](49690/5): Loss 0.0071, Sem 0.0071 LR: 2.049e-02 Score 99.750 Data time: 0.2256, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8786 [2021-07-09 03:09:30,390][root][INFO] - ===> Epoch[9939](49695/5): Loss 0.0080, Sem 0.0080 LR: 2.049e-02 Score 99.712 Data time: 0.2517, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6998 [2021-07-09 03:09:44,718][root][INFO] - ===> Epoch[9940](49700/5): Loss 0.0075, Sem 0.0075 LR: 2.048e-02 Score 99.728 Data time: 0.1826, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8189 [2021-07-09 03:09:58,341][root][INFO] - ===> Epoch[9941](49705/5): Loss 0.0076, Sem 0.0076 LR: 2.047e-02 Score 99.728 Data time: 0.2142, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6793 [2021-07-09 03:10:12,424][root][INFO] - ===> Epoch[9942](49710/5): Loss 0.0076, Sem 0.0076 LR: 2.046e-02 Score 99.726 Data time: 0.1764, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7797 [2021-07-09 03:10:26,403][root][INFO] - ===> Epoch[9943](49715/5): Loss 0.0076, Sem 0.0076 LR: 2.045e-02 Score 99.728 Data time: 0.2036, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7516 [2021-07-09 03:10:40,473][root][INFO] - ===> Epoch[9944](49720/5): Loss 0.0072, Sem 0.0072 LR: 2.044e-02 Score 99.742 Data time: 0.2415, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7634 [2021-07-09 03:10:54,193][root][INFO] - ===> Epoch[9945](49725/5): Loss 0.0073, Sem 0.0073 LR: 2.043e-02 Score 99.738 Data time: 0.2515, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.6949 [2021-07-09 03:11:08,636][root][INFO] - ===> Epoch[9946](49730/5): Loss 0.0072, Sem 0.0072 LR: 2.042e-02 Score 99.742 Data time: 0.2395, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.8436 [2021-07-09 03:11:21,640][root][INFO] - ===> Epoch[9947](49735/5): Loss 0.0072, Sem 0.0072 LR: 2.041e-02 Score 99.741 Data time: 0.2137, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.5605 [2021-07-09 03:11:35,940][root][INFO] - ===> Epoch[9948](49740/5): Loss 0.0073, Sem 0.0073 LR: 2.040e-02 Score 99.737 Data time: 0.2637, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7963 [2021-07-09 03:11:49,870][root][INFO] - ===> Epoch[9949](49745/5): Loss 0.0072, Sem 0.0072 LR: 2.040e-02 Score 99.745 Data time: 0.1765, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.7570 [2021-07-09 03:12:03,519][root][INFO] - ===> Epoch[9950](49750/5): Loss 0.0076, Sem 0.0076 LR: 2.039e-02 Score 99.726 Data time: 0.2298, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.6822 [2021-07-09 03:12:18,467][root][INFO] - ===> Epoch[9951](49755/5): Loss 0.0074, Sem 0.0074 LR: 2.038e-02 Score 99.736 Data time: 0.2328, Forward time: 1.2532, Backward time: 1.2464, DDP time: 0.0042, Total iter time: 2.9482 [2021-07-09 03:12:32,824][root][INFO] - ===> Epoch[9952](49760/5): Loss 0.0073, Sem 0.0073 LR: 2.037e-02 Score 99.739 Data time: 0.1615, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8351 [2021-07-09 03:12:46,851][root][INFO] - ===> Epoch[9953](49765/5): Loss 0.0074, Sem 0.0074 LR: 2.036e-02 Score 99.736 Data time: 0.2339, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7615 [2021-07-09 03:13:00,991][root][INFO] - ===> Epoch[9954](49770/5): Loss 0.0075, Sem 0.0075 LR: 2.035e-02 Score 99.731 Data time: 0.2298, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7802 [2021-07-09 03:13:15,092][root][INFO] - ===> Epoch[9955](49775/5): Loss 0.0072, Sem 0.0072 LR: 2.034e-02 Score 99.741 Data time: 0.1932, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7788 [2021-07-09 03:13:28,931][root][INFO] - ===> Epoch[9956](49780/5): Loss 0.0073, Sem 0.0073 LR: 2.033e-02 Score 99.742 Data time: 0.2169, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7092 [2021-07-09 03:13:42,931][root][INFO] - ===> Epoch[9957](49785/5): Loss 0.0073, Sem 0.0073 LR: 2.032e-02 Score 99.737 Data time: 0.1862, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7443 [2021-07-09 03:13:56,839][root][INFO] - ===> Epoch[9958](49790/5): Loss 0.0070, Sem 0.0070 LR: 2.032e-02 Score 99.749 Data time: 0.2252, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7406 [2021-07-09 03:14:10,858][root][INFO] - ===> Epoch[9959](49795/5): Loss 0.0073, Sem 0.0073 LR: 2.031e-02 Score 99.740 Data time: 0.2033, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7532 [2021-07-09 03:14:25,769][root][INFO] - ===> Epoch[9960](49800/5): Loss 0.0072, Sem 0.0072 LR: 2.030e-02 Score 99.740 Data time: 0.2347, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.9365 [2021-07-09 03:14:25,772][root][INFO] - ===> Start testing [2021-07-09 03:15:01,831][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1258 Loss 0.465 (AVG: 0.621) Score 91.914 (AVG: 91.053) mIOU 71.723 mAP 82.644 mAcc 78.054 IOU: 64.378 0.000 82.980 75.040 95.001 91.398 33.446 83.731 98.557 84.265 79.833 86.338 57.429 mAP: 83.358 3.062 91.061 85.707 98.578 98.139 52.299 97.647 99.703 93.643 90.381 92.612 88.186 mAcc: 78.581 0.000 86.255 83.718 96.547 97.642 36.562 95.329 99.319 93.094 90.707 96.607 60.338 [2021-07-09 03:15:01,831][root][INFO] - Finished test. Elapsed time: 36.0593 [2021-07-09 03:15:02,873][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 03:15:17,221][root][INFO] - ===> Epoch[9961](49805/5): Loss 0.0072, Sem 0.0072 LR: 2.029e-02 Score 99.744 Data time: 0.1872, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8171 [2021-07-09 03:15:32,050][root][INFO] - ===> Epoch[9962](49810/5): Loss 0.0078, Sem 0.0078 LR: 2.028e-02 Score 99.721 Data time: 0.1945, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.9136 [2021-07-09 03:15:45,453][root][INFO] - ===> Epoch[9963](49815/5): Loss 0.0071, Sem 0.0071 LR: 2.027e-02 Score 99.743 Data time: 0.2405, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6343 [2021-07-09 03:15:59,514][root][INFO] - ===> Epoch[9964](49820/5): Loss 0.0075, Sem 0.0075 LR: 2.026e-02 Score 99.732 Data time: 0.2574, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7727 [2021-07-09 03:16:13,434][root][INFO] - ===> Epoch[9965](49825/5): Loss 0.0074, Sem 0.0074 LR: 2.025e-02 Score 99.733 Data time: 0.2593, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7326 [2021-07-09 03:16:27,886][root][INFO] - ===> Epoch[9966](49830/5): Loss 0.0073, Sem 0.0073 LR: 2.024e-02 Score 99.741 Data time: 0.3277, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8498 [2021-07-09 03:16:41,717][root][INFO] - ===> Epoch[9967](49835/5): Loss 0.0076, Sem 0.0076 LR: 2.023e-02 Score 99.730 Data time: 0.2773, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7210 [2021-07-09 03:16:56,266][root][INFO] - ===> Epoch[9968](49840/5): Loss 0.0071, Sem 0.0071 LR: 2.023e-02 Score 99.746 Data time: 0.2446, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8572 [2021-07-09 03:17:10,943][root][INFO] - ===> Epoch[9969](49845/5): Loss 0.0073, Sem 0.0073 LR: 2.022e-02 Score 99.740 Data time: 0.2621, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8981 [2021-07-09 03:17:26,050][root][INFO] - ===> Epoch[9970](49850/5): Loss 0.0071, Sem 0.0071 LR: 2.021e-02 Score 99.746 Data time: 0.2363, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.9743 [2021-07-09 03:17:39,887][root][INFO] - ===> Epoch[9971](49855/5): Loss 0.0074, Sem 0.0074 LR: 2.020e-02 Score 99.736 Data time: 0.2716, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7144 [2021-07-09 03:17:53,928][root][INFO] - ===> Epoch[9972](49860/5): Loss 0.0070, Sem 0.0070 LR: 2.019e-02 Score 99.749 Data time: 0.2288, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7619 [2021-07-09 03:18:07,571][root][INFO] - ===> Epoch[9973](49865/5): Loss 0.0072, Sem 0.0072 LR: 2.018e-02 Score 99.742 Data time: 0.1996, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6855 [2021-07-09 03:18:21,553][root][INFO] - ===> Epoch[9974](49870/5): Loss 0.0072, Sem 0.0072 LR: 2.017e-02 Score 99.744 Data time: 0.2393, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7528 [2021-07-09 03:18:35,194][root][INFO] - ===> Epoch[9975](49875/5): Loss 0.0071, Sem 0.0071 LR: 2.016e-02 Score 99.748 Data time: 0.2067, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6851 [2021-07-09 03:18:49,108][root][INFO] - ===> Epoch[9976](49880/5): Loss 0.0075, Sem 0.0075 LR: 2.015e-02 Score 99.734 Data time: 0.1892, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7464 [2021-07-09 03:19:02,627][root][INFO] - ===> Epoch[9977](49885/5): Loss 0.0071, Sem 0.0071 LR: 2.014e-02 Score 99.745 Data time: 0.2800, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6567 [2021-07-09 03:19:16,846][root][INFO] - ===> Epoch[9978](49890/5): Loss 0.0073, Sem 0.0073 LR: 2.014e-02 Score 99.740 Data time: 0.2123, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7976 [2021-07-09 03:19:30,570][root][INFO] - ===> Epoch[9979](49895/5): Loss 0.0075, Sem 0.0075 LR: 2.013e-02 Score 99.732 Data time: 0.2400, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6931 [2021-07-09 03:19:44,062][root][INFO] - ===> Epoch[9980](49900/5): Loss 0.0069, Sem 0.0069 LR: 2.012e-02 Score 99.753 Data time: 0.2382, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6533 [2021-07-09 03:19:58,410][root][INFO] - ===> Epoch[9981](49905/5): Loss 0.0073, Sem 0.0073 LR: 2.011e-02 Score 99.742 Data time: 0.2083, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8274 [2021-07-09 03:20:11,868][root][INFO] - ===> Epoch[9982](49910/5): Loss 0.0076, Sem 0.0076 LR: 2.010e-02 Score 99.730 Data time: 0.2225, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6487 [2021-07-09 03:20:26,194][root][INFO] - ===> Epoch[9983](49915/5): Loss 0.0072, Sem 0.0072 LR: 2.009e-02 Score 99.744 Data time: 0.2206, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8261 [2021-07-09 03:20:40,690][root][INFO] - ===> Epoch[9984](49920/5): Loss 0.0072, Sem 0.0072 LR: 2.008e-02 Score 99.741 Data time: 0.2740, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8571 [2021-07-09 03:20:55,039][root][INFO] - ===> Epoch[9985](49925/5): Loss 0.0072, Sem 0.0072 LR: 2.007e-02 Score 99.740 Data time: 0.1840, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8341 [2021-07-09 03:21:08,752][root][INFO] - ===> Epoch[9986](49930/5): Loss 0.0074, Sem 0.0074 LR: 2.006e-02 Score 99.735 Data time: 0.2638, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7076 [2021-07-09 03:21:23,560][root][INFO] - ===> Epoch[9987](49935/5): Loss 0.0071, Sem 0.0071 LR: 2.006e-02 Score 99.747 Data time: 0.1681, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.9201 [2021-07-09 03:21:37,001][root][INFO] - ===> Epoch[9988](49940/5): Loss 0.0079, Sem 0.0079 LR: 2.005e-02 Score 99.718 Data time: 0.2046, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6528 [2021-07-09 03:21:50,872][root][INFO] - ===> Epoch[9989](49945/5): Loss 0.0074, Sem 0.0074 LR: 2.004e-02 Score 99.736 Data time: 0.1885, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.7335 [2021-07-09 03:22:05,811][root][INFO] - ===> Epoch[9990](49950/5): Loss 0.0073, Sem 0.0073 LR: 2.003e-02 Score 99.736 Data time: 0.2434, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.9412 [2021-07-09 03:22:19,478][root][INFO] - ===> Epoch[9991](49955/5): Loss 0.0074, Sem 0.0074 LR: 2.002e-02 Score 99.736 Data time: 0.1945, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6959 [2021-07-09 03:22:34,196][root][INFO] - ===> Epoch[9992](49960/5): Loss 0.0073, Sem 0.0073 LR: 2.001e-02 Score 99.740 Data time: 0.2834, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8895 [2021-07-09 03:22:48,433][root][INFO] - ===> Epoch[9993](49965/5): Loss 0.0070, Sem 0.0070 LR: 2.000e-02 Score 99.748 Data time: 0.2545, Forward time: 1.2532, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.8041 [2021-07-09 03:23:02,985][root][INFO] - ===> Epoch[9994](49970/5): Loss 0.0072, Sem 0.0072 LR: 1.999e-02 Score 99.740 Data time: 0.1812, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8644 [2021-07-09 03:23:17,692][root][INFO] - ===> Epoch[9995](49975/5): Loss 0.0070, Sem 0.0070 LR: 1.998e-02 Score 99.748 Data time: 0.2251, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8897 [2021-07-09 03:23:32,066][root][INFO] - ===> Epoch[9996](49980/5): Loss 0.0071, Sem 0.0071 LR: 1.997e-02 Score 99.746 Data time: 0.2259, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8135 [2021-07-09 03:23:46,186][root][INFO] - ===> Epoch[9997](49985/5): Loss 0.0073, Sem 0.0073 LR: 1.997e-02 Score 99.737 Data time: 0.2034, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7889 [2021-07-09 03:23:59,654][root][INFO] - ===> Epoch[9998](49990/5): Loss 0.0074, Sem 0.0074 LR: 1.996e-02 Score 99.735 Data time: 0.2060, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6456 [2021-07-09 03:24:13,992][root][INFO] - ===> Epoch[9999](49995/5): Loss 0.0070, Sem 0.0070 LR: 1.995e-02 Score 99.749 Data time: 0.1805, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8267 [2021-07-09 03:24:27,082][root][INFO] - ===> Epoch[10000](50000/5): Loss 0.0074, Sem 0.0074 LR: 1.994e-02 Score 99.732 Data time: 0.2395, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.5779 [2021-07-09 03:24:27,083][root][INFO] - ===> Start testing [2021-07-09 03:25:11,833][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1253 Loss 0.452 (AVG: 0.627) Score 91.519 (AVG: 90.966) mIOU 71.465 mAP 82.650 mAcc 77.737 IOU: 64.018 0.000 81.244 74.786 95.016 91.996 29.463 86.275 98.603 84.364 78.754 85.933 58.598 mAP: 82.941 3.390 90.822 85.593 98.511 98.195 51.291 98.050 99.646 93.492 90.561 92.374 89.584 mAcc: 78.732 0.000 85.545 83.287 96.684 97.405 31.878 95.334 99.488 93.262 90.753 96.364 61.850 [2021-07-09 03:25:11,833][root][INFO] - Finished test. Elapsed time: 44.7484 [2021-07-09 03:25:12,960][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 03:25:26,155][root][INFO] - ===> Epoch[10001](50005/5): Loss 0.0072, Sem 0.0072 LR: 1.993e-02 Score 99.742 Data time: 0.1603, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6072 [2021-07-09 03:25:40,608][root][INFO] - ===> Epoch[10002](50010/5): Loss 0.0069, Sem 0.0069 LR: 1.992e-02 Score 99.753 Data time: 0.2028, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8369 [2021-07-09 03:25:54,380][root][INFO] - ===> Epoch[10003](50015/5): Loss 0.0074, Sem 0.0074 LR: 1.991e-02 Score 99.737 Data time: 0.2221, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7028 [2021-07-09 03:26:08,262][root][INFO] - ===> Epoch[10004](50020/5): Loss 0.0073, Sem 0.0073 LR: 1.990e-02 Score 99.741 Data time: 0.2295, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7236 [2021-07-09 03:26:22,687][root][INFO] - ===> Epoch[10005](50025/5): Loss 0.0071, Sem 0.0071 LR: 1.989e-02 Score 99.747 Data time: 0.2509, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8323 [2021-07-09 03:26:36,095][root][INFO] - ===> Epoch[10006](50030/5): Loss 0.0075, Sem 0.0075 LR: 1.988e-02 Score 99.727 Data time: 0.2117, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6376 [2021-07-09 03:26:49,817][root][INFO] - ===> Epoch[10007](50035/5): Loss 0.0068, Sem 0.0068 LR: 1.988e-02 Score 99.759 Data time: 0.1951, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7065 [2021-07-09 03:27:03,803][root][INFO] - ===> Epoch[10008](50040/5): Loss 0.0073, Sem 0.0073 LR: 1.987e-02 Score 99.738 Data time: 0.1902, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7483 [2021-07-09 03:27:17,362][root][INFO] - ===> Epoch[10009](50045/5): Loss 0.0072, Sem 0.0072 LR: 1.986e-02 Score 99.741 Data time: 0.2255, Forward time: 1.2531, Backward time: 1.2465, DDP time: 0.0042, Total iter time: 2.6688 [2021-07-09 03:27:31,430][root][INFO] - ===> Epoch[10010](50050/5): Loss 0.0071, Sem 0.0071 LR: 1.985e-02 Score 99.745 Data time: 0.1982, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7593 [2021-07-09 03:27:45,714][root][INFO] - ===> Epoch[10011](50055/5): Loss 0.0071, Sem 0.0071 LR: 1.984e-02 Score 99.745 Data time: 0.1919, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8077 [2021-07-09 03:27:59,646][root][INFO] - ===> Epoch[10012](50060/5): Loss 0.0075, Sem 0.0075 LR: 1.983e-02 Score 99.731 Data time: 0.1972, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7354 [2021-07-09 03:28:14,044][root][INFO] - ===> Epoch[10013](50065/5): Loss 0.0070, Sem 0.0070 LR: 1.982e-02 Score 99.750 Data time: 0.2561, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8287 [2021-07-09 03:28:26,973][root][INFO] - ===> Epoch[10014](50070/5): Loss 0.0074, Sem 0.0074 LR: 1.981e-02 Score 99.736 Data time: 0.1591, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.5489 [2021-07-09 03:28:42,027][root][INFO] - ===> Epoch[10015](50075/5): Loss 0.0074, Sem 0.0074 LR: 1.980e-02 Score 99.732 Data time: 0.3177, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.9553 [2021-07-09 03:28:56,041][root][INFO] - ===> Epoch[10016](50080/5): Loss 0.0073, Sem 0.0073 LR: 1.980e-02 Score 99.739 Data time: 0.3029, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7569 [2021-07-09 03:29:09,932][root][INFO] - ===> Epoch[10017](50085/5): Loss 0.0075, Sem 0.0075 LR: 1.979e-02 Score 99.734 Data time: 0.2276, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7322 [2021-07-09 03:29:25,033][root][INFO] - ===> Epoch[10018](50090/5): Loss 0.0073, Sem 0.0073 LR: 1.978e-02 Score 99.741 Data time: 0.2331, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.9723 [2021-07-09 03:29:39,926][root][INFO] - ===> Epoch[10019](50095/5): Loss 0.0073, Sem 0.0073 LR: 1.977e-02 Score 99.742 Data time: 0.2793, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.9365 [2021-07-09 03:29:54,308][root][INFO] - ===> Epoch[10020](50100/5): Loss 0.0074, Sem 0.0074 LR: 1.976e-02 Score 99.734 Data time: 0.2848, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8232 [2021-07-09 03:30:08,714][root][INFO] - ===> Epoch[10021](50105/5): Loss 0.0073, Sem 0.0073 LR: 1.975e-02 Score 99.740 Data time: 0.2714, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8234 [2021-07-09 03:30:24,046][root][INFO] - ===> Epoch[10022](50110/5): Loss 0.0071, Sem 0.0071 LR: 1.974e-02 Score 99.745 Data time: 0.2723, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 3.0160 [2021-07-09 03:30:38,039][root][INFO] - ===> Epoch[10023](50115/5): Loss 0.0071, Sem 0.0071 LR: 1.973e-02 Score 99.746 Data time: 0.2222, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7553 [2021-07-09 03:30:52,019][root][INFO] - ===> Epoch[10024](50120/5): Loss 0.0071, Sem 0.0071 LR: 1.972e-02 Score 99.748 Data time: 0.2813, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7422 [2021-07-09 03:31:05,840][root][INFO] - ===> Epoch[10025](50125/5): Loss 0.0076, Sem 0.0076 LR: 1.971e-02 Score 99.728 Data time: 0.2734, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7049 [2021-07-09 03:31:19,349][root][INFO] - ===> Epoch[10026](50130/5): Loss 0.0073, Sem 0.0073 LR: 1.971e-02 Score 99.736 Data time: 0.1845, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6518 [2021-07-09 03:31:34,560][root][INFO] - ===> Epoch[10027](50135/5): Loss 0.0070, Sem 0.0070 LR: 1.970e-02 Score 99.750 Data time: 0.2564, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.9947 [2021-07-09 03:31:48,080][root][INFO] - ===> Epoch[10028](50140/5): Loss 0.0075, Sem 0.0075 LR: 1.969e-02 Score 99.732 Data time: 0.1996, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6604 [2021-07-09 03:32:01,704][root][INFO] - ===> Epoch[10029](50145/5): Loss 0.0068, Sem 0.0068 LR: 1.968e-02 Score 99.754 Data time: 0.2094, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6746 [2021-07-09 03:32:15,701][root][INFO] - ===> Epoch[10030](50150/5): Loss 0.0074, Sem 0.0074 LR: 1.967e-02 Score 99.738 Data time: 0.1836, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7513 [2021-07-09 03:32:29,528][root][INFO] - ===> Epoch[10031](50155/5): Loss 0.0072, Sem 0.0072 LR: 1.966e-02 Score 99.743 Data time: 0.2274, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7182 [2021-07-09 03:32:43,736][root][INFO] - ===> Epoch[10032](50160/5): Loss 0.0071, Sem 0.0071 LR: 1.965e-02 Score 99.746 Data time: 0.2437, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7871 [2021-07-09 03:32:58,541][root][INFO] - ===> Epoch[10033](50165/5): Loss 0.0071, Sem 0.0071 LR: 1.964e-02 Score 99.747 Data time: 0.1896, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.9247 [2021-07-09 03:33:13,191][root][INFO] - ===> Epoch[10034](50170/5): Loss 0.0074, Sem 0.0074 LR: 1.963e-02 Score 99.735 Data time: 0.2383, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8927 [2021-07-09 03:33:27,784][root][INFO] - ===> Epoch[10035](50175/5): Loss 0.0071, Sem 0.0071 LR: 1.962e-02 Score 99.746 Data time: 0.2138, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8708 [2021-07-09 03:33:42,430][root][INFO] - ===> Epoch[10036](50180/5): Loss 0.0075, Sem 0.0075 LR: 1.962e-02 Score 99.733 Data time: 0.1886, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8883 [2021-07-09 03:33:56,790][root][INFO] - ===> Epoch[10037](50185/5): Loss 0.0071, Sem 0.0071 LR: 1.961e-02 Score 99.745 Data time: 0.2200, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8313 [2021-07-09 03:34:10,267][root][INFO] - ===> Epoch[10038](50190/5): Loss 0.0074, Sem 0.0074 LR: 1.960e-02 Score 99.735 Data time: 0.2729, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6483 [2021-07-09 03:34:24,574][root][INFO] - ===> Epoch[10039](50195/5): Loss 0.0071, Sem 0.0071 LR: 1.959e-02 Score 99.744 Data time: 0.2188, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8160 [2021-07-09 03:34:38,308][root][INFO] - ===> Epoch[10040](50200/5): Loss 0.0071, Sem 0.0071 LR: 1.958e-02 Score 99.745 Data time: 0.2475, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7078 [2021-07-09 03:34:38,309][root][INFO] - ===> Start testing [2021-07-09 03:35:21,203][root][INFO] - 68/68: Data time: 0.1346, Iter time: 0.1254 Loss 0.363 (AVG: 0.639) Score 93.096 (AVG: 90.929) mIOU 71.325 mAP 82.565 mAcc 77.770 IOU: 63.977 0.000 81.900 74.642 95.154 91.729 33.771 82.611 98.512 83.426 78.765 86.302 56.440 mAP: 83.203 3.283 89.340 85.345 98.358 98.203 53.636 97.681 99.663 93.990 90.125 92.411 88.112 mAcc: 80.015 0.000 84.476 82.733 96.670 97.784 37.107 94.925 99.352 92.768 90.116 96.311 58.752 [2021-07-09 03:35:21,203][root][INFO] - Finished test. Elapsed time: 42.8934 [2021-07-09 03:35:22,441][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 03:35:35,912][root][INFO] - ===> Epoch[10041](50205/5): Loss 0.0071, Sem 0.0071 LR: 1.957e-02 Score 99.743 Data time: 0.2310, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6511 [2021-07-09 03:35:50,048][root][INFO] - ===> Epoch[10042](50210/5): Loss 0.0072, Sem 0.0072 LR: 1.956e-02 Score 99.742 Data time: 0.1724, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7862 [2021-07-09 03:36:03,819][root][INFO] - ===> Epoch[10043](50215/5): Loss 0.0074, Sem 0.0074 LR: 1.955e-02 Score 99.739 Data time: 0.2368, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7132 [2021-07-09 03:36:18,225][root][INFO] - ===> Epoch[10044](50220/5): Loss 0.0075, Sem 0.0075 LR: 1.954e-02 Score 99.733 Data time: 0.1803, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8428 [2021-07-09 03:36:32,202][root][INFO] - ===> Epoch[10045](50225/5): Loss 0.0073, Sem 0.0073 LR: 1.953e-02 Score 99.739 Data time: 0.2714, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7410 [2021-07-09 03:36:46,356][root][INFO] - ===> Epoch[10046](50230/5): Loss 0.0073, Sem 0.0073 LR: 1.953e-02 Score 99.738 Data time: 0.2357, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7823 [2021-07-09 03:37:00,480][root][INFO] - ===> Epoch[10047](50235/5): Loss 0.0072, Sem 0.0072 LR: 1.952e-02 Score 99.743 Data time: 0.1890, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7861 [2021-07-09 03:37:14,208][root][INFO] - ===> Epoch[10048](50240/5): Loss 0.0070, Sem 0.0070 LR: 1.951e-02 Score 99.750 Data time: 0.2424, Forward time: 1.2532, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6934 [2021-07-09 03:37:27,895][root][INFO] - ===> Epoch[10049](50245/5): Loss 0.0074, Sem 0.0074 LR: 1.950e-02 Score 99.736 Data time: 0.1826, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6928 [2021-07-09 03:37:42,257][root][INFO] - ===> Epoch[10050](50250/5): Loss 0.0070, Sem 0.0070 LR: 1.949e-02 Score 99.750 Data time: 0.1626, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8268 [2021-07-09 03:37:56,046][root][INFO] - ===> Epoch[10051](50255/5): Loss 0.0074, Sem 0.0074 LR: 1.948e-02 Score 99.732 Data time: 0.2211, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7066 [2021-07-09 03:38:08,506][root][INFO] - ===> Epoch[10052](50260/5): Loss 0.0075, Sem 0.0075 LR: 1.947e-02 Score 99.731 Data time: 0.2082, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.4433 [2021-07-09 03:38:22,078][root][INFO] - ===> Epoch[10053](50265/5): Loss 0.0073, Sem 0.0073 LR: 1.946e-02 Score 99.743 Data time: 0.2015, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6617 [2021-07-09 03:38:35,961][root][INFO] - ===> Epoch[10054](50270/5): Loss 0.0076, Sem 0.0076 LR: 1.945e-02 Score 99.725 Data time: 0.2244, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7120 [2021-07-09 03:38:49,624][root][INFO] - ===> Epoch[10055](50275/5): Loss 0.0074, Sem 0.0074 LR: 1.944e-02 Score 99.733 Data time: 0.1963, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6908 [2021-07-09 03:39:02,692][root][INFO] - ===> Epoch[10056](50280/5): Loss 0.0073, Sem 0.0073 LR: 1.944e-02 Score 99.739 Data time: 0.2093, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.5678 [2021-07-09 03:39:17,984][root][INFO] - ===> Epoch[10057](50285/5): Loss 0.0071, Sem 0.0071 LR: 1.943e-02 Score 99.745 Data time: 0.2404, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 3.0092 [2021-07-09 03:39:31,972][root][INFO] - ===> Epoch[10058](50290/5): Loss 0.0080, Sem 0.0080 LR: 1.942e-02 Score 99.715 Data time: 0.2035, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7517 [2021-07-09 03:39:46,206][root][INFO] - ===> Epoch[10059](50295/5): Loss 0.0071, Sem 0.0071 LR: 1.941e-02 Score 99.744 Data time: 0.2534, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7992 [2021-07-09 03:40:00,162][root][INFO] - ===> Epoch[10060](50300/5): Loss 0.0073, Sem 0.0073 LR: 1.940e-02 Score 99.741 Data time: 0.2143, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7453 [2021-07-09 03:40:14,135][root][INFO] - ===> Epoch[10061](50305/5): Loss 0.0074, Sem 0.0074 LR: 1.939e-02 Score 99.731 Data time: 0.2367, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7424 [2021-07-09 03:40:28,750][root][INFO] - ===> Epoch[10062](50310/5): Loss 0.0072, Sem 0.0072 LR: 1.938e-02 Score 99.739 Data time: 0.2137, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8757 [2021-07-09 03:40:42,758][root][INFO] - ===> Epoch[10063](50315/5): Loss 0.0075, Sem 0.0075 LR: 1.937e-02 Score 99.734 Data time: 0.2211, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7593 [2021-07-09 03:40:56,066][root][INFO] - ===> Epoch[10064](50320/5): Loss 0.0075, Sem 0.0075 LR: 1.936e-02 Score 99.732 Data time: 0.1908, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6081 [2021-07-09 03:41:10,274][root][INFO] - ===> Epoch[10065](50325/5): Loss 0.0073, Sem 0.0073 LR: 1.935e-02 Score 99.740 Data time: 0.2253, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7845 [2021-07-09 03:41:24,794][root][INFO] - ===> Epoch[10066](50330/5): Loss 0.0075, Sem 0.0075 LR: 1.935e-02 Score 99.731 Data time: 0.2135, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8637 [2021-07-09 03:41:38,070][root][INFO] - ===> Epoch[10067](50335/5): Loss 0.0073, Sem 0.0073 LR: 1.934e-02 Score 99.735 Data time: 0.1490, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6095 [2021-07-09 03:41:53,046][root][INFO] - ===> Epoch[10068](50340/5): Loss 0.0073, Sem 0.0073 LR: 1.933e-02 Score 99.740 Data time: 0.2901, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.9556 [2021-07-09 03:42:07,466][root][INFO] - ===> Epoch[10069](50345/5): Loss 0.0074, Sem 0.0074 LR: 1.932e-02 Score 99.738 Data time: 0.3057, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8284 [2021-07-09 03:42:21,127][root][INFO] - ===> Epoch[10070](50350/5): Loss 0.0074, Sem 0.0074 LR: 1.931e-02 Score 99.736 Data time: 0.2762, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6805 [2021-07-09 03:42:36,304][root][INFO] - ===> Epoch[10071](50355/5): Loss 0.0073, Sem 0.0073 LR: 1.930e-02 Score 99.735 Data time: 0.3056, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.9763 [2021-07-09 03:42:51,060][root][INFO] - ===> Epoch[10072](50360/5): Loss 0.0073, Sem 0.0073 LR: 1.929e-02 Score 99.737 Data time: 0.2751, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.9024 [2021-07-09 03:43:05,486][root][INFO] - ===> Epoch[10073](50365/5): Loss 0.0074, Sem 0.0074 LR: 1.928e-02 Score 99.735 Data time: 0.2335, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8441 [2021-07-09 03:43:19,739][root][INFO] - ===> Epoch[10074](50370/5): Loss 0.0072, Sem 0.0072 LR: 1.927e-02 Score 99.742 Data time: 0.2824, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8037 [2021-07-09 03:43:34,172][root][INFO] - ===> Epoch[10075](50375/5): Loss 0.0071, Sem 0.0071 LR: 1.926e-02 Score 99.743 Data time: 0.2315, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8417 [2021-07-09 03:43:48,123][root][INFO] - ===> Epoch[10076](50380/5): Loss 0.0073, Sem 0.0073 LR: 1.926e-02 Score 99.735 Data time: 0.2114, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7401 [2021-07-09 03:44:01,115][root][INFO] - ===> Epoch[10077](50385/5): Loss 0.0073, Sem 0.0073 LR: 1.925e-02 Score 99.738 Data time: 0.2908, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.5461 [2021-07-09 03:44:14,860][root][INFO] - ===> Epoch[10078](50390/5): Loss 0.0076, Sem 0.0076 LR: 1.924e-02 Score 99.733 Data time: 0.1669, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7117 [2021-07-09 03:44:28,554][root][INFO] - ===> Epoch[10079](50395/5): Loss 0.0072, Sem 0.0072 LR: 1.923e-02 Score 99.741 Data time: 0.2253, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6873 [2021-07-09 03:44:41,836][root][INFO] - ===> Epoch[10080](50400/5): Loss 0.0070, Sem 0.0070 LR: 1.922e-02 Score 99.750 Data time: 0.2342, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6115 [2021-07-09 03:44:41,838][root][INFO] - ===> Start testing [2021-07-09 03:45:22,174][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1345 Loss 0.472 (AVG: 0.626) Score 91.682 (AVG: 91.082) mIOU 71.667 mAP 82.811 mAcc 78.181 IOU: 64.152 0.000 80.752 75.366 94.962 91.802 33.910 82.946 98.450 84.437 79.894 86.522 58.474 mAP: 82.978 3.282 91.038 85.718 98.221 98.163 53.666 97.704 99.705 94.478 90.619 92.256 88.713 mAcc: 79.381 0.000 85.833 83.673 96.508 97.631 36.895 95.339 99.283 93.193 90.431 96.331 61.852 [2021-07-09 03:45:22,174][root][INFO] - Finished test. Elapsed time: 40.3353 [2021-07-09 03:45:23,273][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 03:45:36,315][root][INFO] - ===> Epoch[10081](50405/5): Loss 0.0074, Sem 0.0074 LR: 1.921e-02 Score 99.735 Data time: 0.1670, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.5723 [2021-07-09 03:45:50,551][root][INFO] - ===> Epoch[10082](50410/5): Loss 0.0073, Sem 0.0073 LR: 1.920e-02 Score 99.736 Data time: 0.2098, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8080 [2021-07-09 03:46:04,877][root][INFO] - ===> Epoch[10083](50415/5): Loss 0.0068, Sem 0.0068 LR: 1.919e-02 Score 99.757 Data time: 0.2482, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.8195 [2021-07-09 03:46:18,853][root][INFO] - ===> Epoch[10084](50420/5): Loss 0.0070, Sem 0.0070 LR: 1.918e-02 Score 99.751 Data time: 0.2067, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7495 [2021-07-09 03:46:32,481][root][INFO] - ===> Epoch[10085](50425/5): Loss 0.0076, Sem 0.0076 LR: 1.917e-02 Score 99.730 Data time: 0.2400, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6857 [2021-07-09 03:46:46,168][root][INFO] - ===> Epoch[10086](50430/5): Loss 0.0074, Sem 0.0074 LR: 1.917e-02 Score 99.737 Data time: 0.1909, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.6896 [2021-07-09 03:47:00,374][root][INFO] - ===> Epoch[10087](50435/5): Loss 0.0073, Sem 0.0073 LR: 1.916e-02 Score 99.743 Data time: 0.1845, Forward time: 1.2531, Backward time: 1.2466, DDP time: 0.0042, Total iter time: 2.7958 [2021-07-09 03:47:14,831][root][INFO] - ===> Epoch[10088](50440/5): Loss 0.0070, Sem 0.0070 LR: 1.915e-02 Score 99.746 Data time: 0.2299, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8459 [2021-07-09 03:47:28,690][root][INFO] - ===> Epoch[10089](50445/5): Loss 0.0071, Sem 0.0071 LR: 1.914e-02 Score 99.748 Data time: 0.1791, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7203 [2021-07-09 03:47:42,282][root][INFO] - ===> Epoch[10090](50450/5): Loss 0.0070, Sem 0.0070 LR: 1.913e-02 Score 99.746 Data time: 0.2386, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6801 [2021-07-09 03:47:56,679][root][INFO] - ===> Epoch[10091](50455/5): Loss 0.0074, Sem 0.0074 LR: 1.912e-02 Score 99.735 Data time: 0.2325, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8201 [2021-07-09 03:48:10,849][root][INFO] - ===> Epoch[10092](50460/5): Loss 0.0070, Sem 0.0070 LR: 1.911e-02 Score 99.751 Data time: 0.2152, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7932 [2021-07-09 03:48:25,087][root][INFO] - ===> Epoch[10093](50465/5): Loss 0.0076, Sem 0.0076 LR: 1.910e-02 Score 99.729 Data time: 0.2147, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8024 [2021-07-09 03:48:39,113][root][INFO] - ===> Epoch[10094](50470/5): Loss 0.0075, Sem 0.0075 LR: 1.909e-02 Score 99.732 Data time: 0.1950, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7613 [2021-07-09 03:48:53,658][root][INFO] - ===> Epoch[10095](50475/5): Loss 0.0071, Sem 0.0071 LR: 1.908e-02 Score 99.747 Data time: 0.1758, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8739 [2021-07-09 03:49:07,042][root][INFO] - ===> Epoch[10096](50480/5): Loss 0.0076, Sem 0.0076 LR: 1.908e-02 Score 99.731 Data time: 0.2555, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6311 [2021-07-09 03:49:21,059][root][INFO] - ===> Epoch[10097](50485/5): Loss 0.0076, Sem 0.0076 LR: 1.907e-02 Score 99.728 Data time: 0.2026, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7631 [2021-07-09 03:49:34,779][root][INFO] - ===> Epoch[10098](50490/5): Loss 0.0069, Sem 0.0069 LR: 1.906e-02 Score 99.752 Data time: 0.1657, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7046 [2021-07-09 03:49:48,314][root][INFO] - ===> Epoch[10099](50495/5): Loss 0.0073, Sem 0.0073 LR: 1.905e-02 Score 99.741 Data time: 0.2572, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6595 [2021-07-09 03:50:02,480][root][INFO] - ===> Epoch[10100](50500/5): Loss 0.0074, Sem 0.0074 LR: 1.904e-02 Score 99.734 Data time: 0.2205, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7817 [2021-07-09 03:50:16,245][root][INFO] - ===> Epoch[10101](50505/5): Loss 0.0072, Sem 0.0072 LR: 1.903e-02 Score 99.741 Data time: 0.1813, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7252 [2021-07-09 03:50:30,915][root][INFO] - ===> Epoch[10102](50510/5): Loss 0.0072, Sem 0.0072 LR: 1.902e-02 Score 99.747 Data time: 0.1847, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8765 [2021-07-09 03:50:44,802][root][INFO] - ===> Epoch[10103](50515/5): Loss 0.0071, Sem 0.0071 LR: 1.901e-02 Score 99.747 Data time: 0.2272, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7244 [2021-07-09 03:50:59,100][root][INFO] - ===> Epoch[10104](50520/5): Loss 0.0071, Sem 0.0071 LR: 1.900e-02 Score 99.745 Data time: 0.2008, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8189 [2021-07-09 03:51:12,745][root][INFO] - ===> Epoch[10105](50525/5): Loss 0.0074, Sem 0.0074 LR: 1.899e-02 Score 99.736 Data time: 0.2437, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6758 [2021-07-09 03:51:26,849][root][INFO] - ===> Epoch[10106](50530/5): Loss 0.0074, Sem 0.0074 LR: 1.899e-02 Score 99.735 Data time: 0.1819, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7851 [2021-07-09 03:51:40,408][root][INFO] - ===> Epoch[10107](50535/5): Loss 0.0069, Sem 0.0069 LR: 1.898e-02 Score 99.750 Data time: 0.2419, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6588 [2021-07-09 03:51:54,639][root][INFO] - ===> Epoch[10108](50540/5): Loss 0.0074, Sem 0.0074 LR: 1.897e-02 Score 99.735 Data time: 0.1987, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8060 [2021-07-09 03:52:08,345][root][INFO] - ===> Epoch[10109](50545/5): Loss 0.0072, Sem 0.0072 LR: 1.896e-02 Score 99.741 Data time: 0.2882, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6850 [2021-07-09 03:52:22,375][root][INFO] - ===> Epoch[10110](50550/5): Loss 0.0073, Sem 0.0073 LR: 1.895e-02 Score 99.737 Data time: 0.2067, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7634 [2021-07-09 03:52:36,176][root][INFO] - ===> Epoch[10111](50555/5): Loss 0.0069, Sem 0.0069 LR: 1.894e-02 Score 99.756 Data time: 0.2594, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7211 [2021-07-09 03:52:50,720][root][INFO] - ===> Epoch[10112](50560/5): Loss 0.0069, Sem 0.0069 LR: 1.893e-02 Score 99.755 Data time: 0.2366, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8545 [2021-07-09 03:53:05,101][root][INFO] - ===> Epoch[10113](50565/5): Loss 0.0075, Sem 0.0075 LR: 1.892e-02 Score 99.734 Data time: 0.2287, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8307 [2021-07-09 03:53:18,597][root][INFO] - ===> Epoch[10114](50570/5): Loss 0.0071, Sem 0.0071 LR: 1.891e-02 Score 99.746 Data time: 0.1885, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6531 [2021-07-09 03:53:32,680][root][INFO] - ===> Epoch[10115](50575/5): Loss 0.0073, Sem 0.0073 LR: 1.890e-02 Score 99.740 Data time: 0.2572, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7769 [2021-07-09 03:53:46,932][root][INFO] - ===> Epoch[10116](50580/5): Loss 0.0072, Sem 0.0072 LR: 1.889e-02 Score 99.743 Data time: 0.2423, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8092 [2021-07-09 03:54:00,658][root][INFO] - ===> Epoch[10117](50585/5): Loss 0.0073, Sem 0.0073 LR: 1.889e-02 Score 99.737 Data time: 0.2040, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7022 [2021-07-09 03:54:14,387][root][INFO] - ===> Epoch[10118](50590/5): Loss 0.0066, Sem 0.0066 LR: 1.888e-02 Score 99.765 Data time: 0.1890, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7012 [2021-07-09 03:54:28,287][root][INFO] - ===> Epoch[10119](50595/5): Loss 0.0076, Sem 0.0076 LR: 1.887e-02 Score 99.728 Data time: 0.2667, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7292 [2021-07-09 03:54:42,271][root][INFO] - ===> Epoch[10120](50600/5): Loss 0.0071, Sem 0.0071 LR: 1.886e-02 Score 99.745 Data time: 0.2896, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7578 [2021-07-09 03:54:42,273][root][INFO] - ===> Start testing [2021-07-09 03:55:21,712][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1310 Loss 0.398 (AVG: 0.621) Score 92.492 (AVG: 91.051) mIOU 71.926 mAP 82.770 mAcc 78.042 IOU: 64.152 0.000 81.952 74.964 95.076 92.036 33.618 84.552 98.480 85.504 79.553 86.181 58.971 mAP: 83.347 2.586 89.870 85.530 98.258 98.095 54.047 97.967 99.667 94.520 90.417 92.363 89.346 mAcc: 79.721 0.000 84.419 83.215 96.561 97.626 37.159 95.478 99.362 92.176 90.439 96.196 62.200 [2021-07-09 03:55:21,713][root][INFO] - Finished test. Elapsed time: 39.4394 [2021-07-09 03:55:23,003][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 03:55:36,451][root][INFO] - ===> Epoch[10121](50605/5): Loss 0.0073, Sem 0.0073 LR: 1.885e-02 Score 99.739 Data time: 0.1742, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6533 [2021-07-09 03:55:50,819][root][INFO] - ===> Epoch[10122](50610/5): Loss 0.0073, Sem 0.0073 LR: 1.884e-02 Score 99.740 Data time: 0.1999, Forward time: 1.2530, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8333 [2021-07-09 03:56:05,604][root][INFO] - ===> Epoch[10123](50615/5): Loss 0.0071, Sem 0.0071 LR: 1.883e-02 Score 99.744 Data time: 0.2657, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.9044 [2021-07-09 03:56:20,784][root][INFO] - ===> Epoch[10124](50620/5): Loss 0.0068, Sem 0.0068 LR: 1.882e-02 Score 99.757 Data time: 0.2969, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.9790 [2021-07-09 03:56:36,006][root][INFO] - ===> Epoch[10125](50625/5): Loss 0.0074, Sem 0.0074 LR: 1.881e-02 Score 99.737 Data time: 0.2335, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 3.0004 [2021-07-09 03:56:51,338][root][INFO] - ===> Epoch[10126](50630/5): Loss 0.0068, Sem 0.0068 LR: 1.880e-02 Score 99.758 Data time: 0.2299, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 3.0299 [2021-07-09 03:57:04,644][root][INFO] - ===> Epoch[10127](50635/5): Loss 0.0073, Sem 0.0073 LR: 1.880e-02 Score 99.739 Data time: 0.2121, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6195 [2021-07-09 03:57:19,041][root][INFO] - ===> Epoch[10128](50640/5): Loss 0.0070, Sem 0.0070 LR: 1.879e-02 Score 99.751 Data time: 0.2137, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8263 [2021-07-09 03:57:32,462][root][INFO] - ===> Epoch[10129](50645/5): Loss 0.0075, Sem 0.0075 LR: 1.878e-02 Score 99.732 Data time: 0.1541, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6480 [2021-07-09 03:57:46,465][root][INFO] - ===> Epoch[10130](50650/5): Loss 0.0074, Sem 0.0074 LR: 1.877e-02 Score 99.735 Data time: 0.2561, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7626 [2021-07-09 03:58:00,385][root][INFO] - ===> Epoch[10131](50655/5): Loss 0.0072, Sem 0.0072 LR: 1.876e-02 Score 99.743 Data time: 0.1973, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7453 [2021-07-09 03:58:14,796][root][INFO] - ===> Epoch[10132](50660/5): Loss 0.0069, Sem 0.0069 LR: 1.875e-02 Score 99.751 Data time: 0.2064, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8296 [2021-07-09 03:58:29,201][root][INFO] - ===> Epoch[10133](50665/5): Loss 0.0071, Sem 0.0071 LR: 1.874e-02 Score 99.745 Data time: 0.2075, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8344 [2021-07-09 03:58:43,440][root][INFO] - ===> Epoch[10134](50670/5): Loss 0.0075, Sem 0.0075 LR: 1.873e-02 Score 99.730 Data time: 0.2032, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7990 [2021-07-09 03:58:57,791][root][INFO] - ===> Epoch[10135](50675/5): Loss 0.0073, Sem 0.0073 LR: 1.872e-02 Score 99.738 Data time: 0.2429, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8269 [2021-07-09 03:59:11,892][root][INFO] - ===> Epoch[10136](50680/5): Loss 0.0070, Sem 0.0070 LR: 1.871e-02 Score 99.750 Data time: 0.2100, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7798 [2021-07-09 03:59:26,910][root][INFO] - ===> Epoch[10137](50685/5): Loss 0.0068, Sem 0.0068 LR: 1.871e-02 Score 99.755 Data time: 0.2271, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.9522 [2021-07-09 03:59:40,826][root][INFO] - ===> Epoch[10138](50690/5): Loss 0.0069, Sem 0.0069 LR: 1.870e-02 Score 99.751 Data time: 0.1798, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7438 [2021-07-09 03:59:54,816][root][INFO] - ===> Epoch[10139](50695/5): Loss 0.0070, Sem 0.0070 LR: 1.869e-02 Score 99.750 Data time: 0.2212, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7532 [2021-07-09 04:00:08,848][root][INFO] - ===> Epoch[10140](50700/5): Loss 0.0070, Sem 0.0070 LR: 1.868e-02 Score 99.750 Data time: 0.1882, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7715 [2021-07-09 04:00:23,287][root][INFO] - ===> Epoch[10141](50705/5): Loss 0.0069, Sem 0.0069 LR: 1.867e-02 Score 99.751 Data time: 0.2595, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.8206 [2021-07-09 04:00:37,442][root][INFO] - ===> Epoch[10142](50710/5): Loss 0.0071, Sem 0.0071 LR: 1.866e-02 Score 99.745 Data time: 0.2587, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.7740 [2021-07-09 04:00:52,403][root][INFO] - ===> Epoch[10143](50715/5): Loss 0.0068, Sem 0.0068 LR: 1.865e-02 Score 99.754 Data time: 0.2410, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.9314 [2021-07-09 04:01:05,823][root][INFO] - ===> Epoch[10144](50720/5): Loss 0.0071, Sem 0.0071 LR: 1.864e-02 Score 99.744 Data time: 0.2005, Forward time: 1.2531, Backward time: 1.2467, DDP time: 0.0042, Total iter time: 2.6476 [2021-07-09 04:01:20,045][root][INFO] - ===> Epoch[10145](50725/5): Loss 0.0072, Sem 0.0072 LR: 1.863e-02 Score 99.741 Data time: 0.2509, Forward time: 1.2531, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7959 [2021-07-09 04:01:33,986][root][INFO] - ===> Epoch[10146](50730/5): Loss 0.0073, Sem 0.0073 LR: 1.862e-02 Score 99.736 Data time: 0.2018, Forward time: 1.2531, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7498 [2021-07-09 04:01:47,827][root][INFO] - ===> Epoch[10147](50735/5): Loss 0.0067, Sem 0.0067 LR: 1.861e-02 Score 99.761 Data time: 0.2077, Forward time: 1.2531, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7258 [2021-07-09 04:02:01,861][root][INFO] - ===> Epoch[10148](50740/5): Loss 0.0069, Sem 0.0069 LR: 1.861e-02 Score 99.755 Data time: 0.1886, Forward time: 1.2531, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7571 [2021-07-09 04:02:15,667][root][INFO] - ===> Epoch[10149](50745/5): Loss 0.0071, Sem 0.0071 LR: 1.860e-02 Score 99.749 Data time: 0.1735, Forward time: 1.2531, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7137 [2021-07-09 04:02:29,433][root][INFO] - ===> Epoch[10150](50750/5): Loss 0.0070, Sem 0.0070 LR: 1.859e-02 Score 99.749 Data time: 0.2235, Forward time: 1.2531, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7089 [2021-07-09 04:02:44,406][root][INFO] - ===> Epoch[10151](50755/5): Loss 0.0072, Sem 0.0072 LR: 1.858e-02 Score 99.741 Data time: 0.2015, Forward time: 1.2531, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.9545 [2021-07-09 04:02:58,114][root][INFO] - ===> Epoch[10152](50760/5): Loss 0.0069, Sem 0.0069 LR: 1.857e-02 Score 99.751 Data time: 0.1683, Forward time: 1.2531, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7078 [2021-07-09 04:03:11,982][root][INFO] - ===> Epoch[10153](50765/5): Loss 0.0068, Sem 0.0068 LR: 1.856e-02 Score 99.754 Data time: 0.1885, Forward time: 1.2531, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7313 [2021-07-09 04:03:25,748][root][INFO] - ===> Epoch[10154](50770/5): Loss 0.0072, Sem 0.0072 LR: 1.855e-02 Score 99.743 Data time: 0.2229, Forward time: 1.2531, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.7102 [2021-07-09 04:03:40,080][root][INFO] - ===> Epoch[10155](50775/5): Loss 0.0071, Sem 0.0071 LR: 1.854e-02 Score 99.746 Data time: 0.1936, Forward time: 1.2531, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8212 [2021-07-09 04:03:53,756][root][INFO] - ===> Epoch[10156](50780/5): Loss 0.0069, Sem 0.0069 LR: 1.853e-02 Score 99.751 Data time: 0.1640, Forward time: 1.2531, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.6883 [2021-07-09 04:04:08,057][root][INFO] - ===> Epoch[10157](50785/5): Loss 0.0071, Sem 0.0071 LR: 1.852e-02 Score 99.746 Data time: 0.1808, Forward time: 1.2531, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8243 [2021-07-09 04:04:23,209][root][INFO] - ===> Epoch[10158](50790/5): Loss 0.0071, Sem 0.0071 LR: 1.852e-02 Score 99.749 Data time: 0.1976, Forward time: 1.2531, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.9672 [2021-07-09 04:04:36,795][root][INFO] - ===> Epoch[10159](50795/5): Loss 0.0074, Sem 0.0074 LR: 1.851e-02 Score 99.736 Data time: 0.2215, Forward time: 1.2531, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.6603 [2021-07-09 04:04:50,345][root][INFO] - ===> Epoch[10160](50800/5): Loss 0.0073, Sem 0.0073 LR: 1.850e-02 Score 99.739 Data time: 0.2395, Forward time: 1.2530, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.6583 [2021-07-09 04:04:50,346][root][INFO] - ===> Start testing [2021-07-09 04:05:29,992][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.2278 Loss 0.566 (AVG: 0.626) Score 90.320 (AVG: 90.971) mIOU 71.661 mAP 82.781 mAcc 77.893 IOU: 63.576 0.000 80.785 74.700 95.028 92.063 32.437 84.382 98.473 85.918 80.031 86.119 58.086 mAP: 81.755 3.811 91.319 85.242 98.320 98.065 53.044 97.739 99.661 94.587 90.393 92.618 89.597 mAcc: 77.537 0.000 85.310 84.171 96.764 97.470 35.171 95.706 99.291 92.699 90.258 96.324 61.908 [2021-07-09 04:05:29,992][root][INFO] - Finished test. Elapsed time: 39.6450 [2021-07-09 04:05:31,065][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 04:05:44,319][root][INFO] - ===> Epoch[10161](50805/5): Loss 0.0070, Sem 0.0070 LR: 1.849e-02 Score 99.748 Data time: 0.1938, Forward time: 1.2530, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.6077 [2021-07-09 04:05:58,876][root][INFO] - ===> Epoch[10162](50810/5): Loss 0.0072, Sem 0.0072 LR: 1.848e-02 Score 99.745 Data time: 0.2781, Forward time: 1.2530, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.8674 [2021-07-09 04:06:13,804][root][INFO] - ===> Epoch[10163](50815/5): Loss 0.0071, Sem 0.0071 LR: 1.847e-02 Score 99.748 Data time: 0.1973, Forward time: 1.2530, Backward time: 1.2468, DDP time: 0.0042, Total iter time: 2.9207 [2021-07-09 04:06:27,721][root][INFO] - ===> Epoch[10164](50820/5): Loss 0.0073, Sem 0.0073 LR: 1.846e-02 Score 99.737 Data time: 0.1468, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7556 [2021-07-09 04:06:42,305][root][INFO] - ===> Epoch[10165](50825/5): Loss 0.0072, Sem 0.0072 LR: 1.845e-02 Score 99.741 Data time: 0.2371, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8693 [2021-07-09 04:06:55,931][root][INFO] - ===> Epoch[10166](50830/5): Loss 0.0073, Sem 0.0073 LR: 1.844e-02 Score 99.739 Data time: 0.2018, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6895 [2021-07-09 04:07:10,215][root][INFO] - ===> Epoch[10167](50835/5): Loss 0.0073, Sem 0.0073 LR: 1.843e-02 Score 99.740 Data time: 0.2344, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8124 [2021-07-09 04:07:24,941][root][INFO] - ===> Epoch[10168](50840/5): Loss 0.0079, Sem 0.0079 LR: 1.842e-02 Score 99.727 Data time: 0.2059, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8923 [2021-07-09 04:07:38,893][root][INFO] - ===> Epoch[10169](50845/5): Loss 0.0072, Sem 0.0072 LR: 1.842e-02 Score 99.748 Data time: 0.2039, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7576 [2021-07-09 04:07:53,141][root][INFO] - ===> Epoch[10170](50850/5): Loss 0.0069, Sem 0.0069 LR: 1.841e-02 Score 99.753 Data time: 0.3350, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8106 [2021-07-09 04:08:07,413][root][INFO] - ===> Epoch[10171](50855/5): Loss 0.0070, Sem 0.0070 LR: 1.840e-02 Score 99.750 Data time: 0.2240, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8086 [2021-07-09 04:08:21,358][root][INFO] - ===> Epoch[10172](50860/5): Loss 0.0074, Sem 0.0074 LR: 1.839e-02 Score 99.734 Data time: 0.2657, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7484 [2021-07-09 04:08:35,403][root][INFO] - ===> Epoch[10173](50865/5): Loss 0.0071, Sem 0.0071 LR: 1.838e-02 Score 99.746 Data time: 0.2511, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7585 [2021-07-09 04:08:49,712][root][INFO] - ===> Epoch[10174](50870/5): Loss 0.0076, Sem 0.0076 LR: 1.837e-02 Score 99.731 Data time: 0.2316, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8186 [2021-07-09 04:09:02,961][root][INFO] - ===> Epoch[10175](50875/5): Loss 0.0070, Sem 0.0070 LR: 1.836e-02 Score 99.745 Data time: 0.2472, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.5936 [2021-07-09 04:09:17,466][root][INFO] - ===> Epoch[10176](50880/5): Loss 0.0074, Sem 0.0074 LR: 1.835e-02 Score 99.732 Data time: 0.1935, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8639 [2021-07-09 04:09:31,142][root][INFO] - ===> Epoch[10177](50885/5): Loss 0.0070, Sem 0.0070 LR: 1.834e-02 Score 99.750 Data time: 0.1922, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6937 [2021-07-09 04:09:45,104][root][INFO] - ===> Epoch[10178](50890/5): Loss 0.0074, Sem 0.0074 LR: 1.833e-02 Score 99.736 Data time: 0.2280, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7506 [2021-07-09 04:09:58,652][root][INFO] - ===> Epoch[10179](50895/5): Loss 0.0070, Sem 0.0070 LR: 1.833e-02 Score 99.748 Data time: 0.2242, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6635 [2021-07-09 04:10:12,233][root][INFO] - ===> Epoch[10180](50900/5): Loss 0.0072, Sem 0.0072 LR: 1.832e-02 Score 99.743 Data time: 0.1919, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6681 [2021-07-09 04:10:26,259][root][INFO] - ===> Epoch[10181](50905/5): Loss 0.0073, Sem 0.0073 LR: 1.831e-02 Score 99.737 Data time: 0.2083, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7622 [2021-07-09 04:10:41,221][root][INFO] - ===> Epoch[10182](50910/5): Loss 0.0069, Sem 0.0069 LR: 1.830e-02 Score 99.753 Data time: 0.1941, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.9469 [2021-07-09 04:10:55,054][root][INFO] - ===> Epoch[10183](50915/5): Loss 0.0072, Sem 0.0072 LR: 1.829e-02 Score 99.741 Data time: 0.2347, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7209 [2021-07-09 04:11:08,467][root][INFO] - ===> Epoch[10184](50920/5): Loss 0.0073, Sem 0.0073 LR: 1.828e-02 Score 99.742 Data time: 0.2000, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6273 [2021-07-09 04:11:22,163][root][INFO] - ===> Epoch[10185](50925/5): Loss 0.0073, Sem 0.0073 LR: 1.827e-02 Score 99.739 Data time: 0.1915, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6989 [2021-07-09 04:11:36,023][root][INFO] - ===> Epoch[10186](50930/5): Loss 0.0072, Sem 0.0072 LR: 1.826e-02 Score 99.745 Data time: 0.2088, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7258 [2021-07-09 04:11:49,954][root][INFO] - ===> Epoch[10187](50935/5): Loss 0.0070, Sem 0.0070 LR: 1.825e-02 Score 99.748 Data time: 0.2253, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7287 [2021-07-09 04:12:03,977][root][INFO] - ===> Epoch[10188](50940/5): Loss 0.0068, Sem 0.0068 LR: 1.824e-02 Score 99.755 Data time: 0.2203, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7599 [2021-07-09 04:12:18,064][root][INFO] - ===> Epoch[10189](50945/5): Loss 0.0078, Sem 0.0078 LR: 1.823e-02 Score 99.721 Data time: 0.1940, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7808 [2021-07-09 04:12:31,949][root][INFO] - ===> Epoch[10190](50950/5): Loss 0.0073, Sem 0.0073 LR: 1.823e-02 Score 99.739 Data time: 0.2464, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7329 [2021-07-09 04:12:45,951][root][INFO] - ===> Epoch[10191](50955/5): Loss 0.0068, Sem 0.0068 LR: 1.822e-02 Score 99.755 Data time: 0.2505, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7475 [2021-07-09 04:12:59,674][root][INFO] - ===> Epoch[10192](50960/5): Loss 0.0074, Sem 0.0074 LR: 1.821e-02 Score 99.735 Data time: 0.2100, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6955 [2021-07-09 04:13:13,429][root][INFO] - ===> Epoch[10193](50965/5): Loss 0.0073, Sem 0.0073 LR: 1.820e-02 Score 99.737 Data time: 0.2077, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7087 [2021-07-09 04:13:27,836][root][INFO] - ===> Epoch[10194](50970/5): Loss 0.0069, Sem 0.0069 LR: 1.819e-02 Score 99.754 Data time: 0.2476, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8247 [2021-07-09 04:13:41,893][root][INFO] - ===> Epoch[10195](50975/5): Loss 0.0070, Sem 0.0070 LR: 1.818e-02 Score 99.748 Data time: 0.2828, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7634 [2021-07-09 04:13:55,927][root][INFO] - ===> Epoch[10196](50980/5): Loss 0.0071, Sem 0.0071 LR: 1.817e-02 Score 99.742 Data time: 0.1974, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7576 [2021-07-09 04:14:10,086][root][INFO] - ===> Epoch[10197](50985/5): Loss 0.0072, Sem 0.0072 LR: 1.816e-02 Score 99.743 Data time: 0.2344, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7874 [2021-07-09 04:14:23,802][root][INFO] - ===> Epoch[10198](50990/5): Loss 0.0072, Sem 0.0072 LR: 1.815e-02 Score 99.743 Data time: 0.1795, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7028 [2021-07-09 04:14:38,042][root][INFO] - ===> Epoch[10199](50995/5): Loss 0.0071, Sem 0.0071 LR: 1.814e-02 Score 99.747 Data time: 0.2239, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8052 [2021-07-09 04:14:51,353][root][INFO] - ===> Epoch[10200](51000/5): Loss 0.0073, Sem 0.0073 LR: 1.814e-02 Score 99.740 Data time: 0.1878, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6224 [2021-07-09 04:14:51,354][root][INFO] - ===> Start testing [2021-07-09 04:15:28,163][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1906 Loss 0.430 (AVG: 0.619) Score 91.628 (AVG: 90.971) mIOU 71.346 mAP 82.468 mAcc 77.752 IOU: 63.893 0.000 80.035 74.608 95.062 91.662 31.837 83.774 98.424 83.878 79.224 86.352 58.753 mAP: 83.184 2.711 88.241 85.874 98.349 98.068 52.726 97.660 99.714 93.386 90.364 92.203 89.606 mAcc: 78.823 0.000 83.747 83.295 96.808 97.620 34.623 94.719 99.231 92.492 90.560 96.306 62.557 [2021-07-09 04:15:28,163][root][INFO] - Finished test. Elapsed time: 36.8085 [2021-07-09 04:15:29,432][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 04:15:43,049][root][INFO] - ===> Epoch[10201](51005/5): Loss 0.0070, Sem 0.0070 LR: 1.813e-02 Score 99.751 Data time: 0.1791, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6809 [2021-07-09 04:15:56,484][root][INFO] - ===> Epoch[10202](51010/5): Loss 0.0072, Sem 0.0072 LR: 1.812e-02 Score 99.743 Data time: 0.1752, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6383 [2021-07-09 04:16:09,793][root][INFO] - ===> Epoch[10203](51015/5): Loss 0.0073, Sem 0.0073 LR: 1.811e-02 Score 99.739 Data time: 0.2279, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6228 [2021-07-09 04:16:23,799][root][INFO] - ===> Epoch[10204](51020/5): Loss 0.0073, Sem 0.0073 LR: 1.810e-02 Score 99.741 Data time: 0.2259, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7435 [2021-07-09 04:16:37,947][root][INFO] - ===> Epoch[10205](51025/5): Loss 0.0073, Sem 0.0073 LR: 1.809e-02 Score 99.738 Data time: 0.2510, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7885 [2021-07-09 04:16:52,319][root][INFO] - ===> Epoch[10206](51030/5): Loss 0.0070, Sem 0.0070 LR: 1.808e-02 Score 99.750 Data time: 0.1692, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8242 [2021-07-09 04:17:06,069][root][INFO] - ===> Epoch[10207](51035/5): Loss 0.0071, Sem 0.0071 LR: 1.807e-02 Score 99.745 Data time: 0.1685, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7163 [2021-07-09 04:17:20,704][root][INFO] - ===> Epoch[10208](51040/5): Loss 0.0069, Sem 0.0069 LR: 1.806e-02 Score 99.752 Data time: 0.2203, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8833 [2021-07-09 04:17:34,940][root][INFO] - ===> Epoch[10209](51045/5): Loss 0.0072, Sem 0.0072 LR: 1.805e-02 Score 99.741 Data time: 0.2204, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8035 [2021-07-09 04:17:49,031][root][INFO] - ===> Epoch[10210](51050/5): Loss 0.0071, Sem 0.0071 LR: 1.804e-02 Score 99.743 Data time: 0.2523, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7701 [2021-07-09 04:18:02,616][root][INFO] - ===> Epoch[10211](51055/5): Loss 0.0069, Sem 0.0069 LR: 1.804e-02 Score 99.751 Data time: 0.2236, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6727 [2021-07-09 04:18:16,620][root][INFO] - ===> Epoch[10212](51060/5): Loss 0.0073, Sem 0.0073 LR: 1.803e-02 Score 99.739 Data time: 0.1981, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7519 [2021-07-09 04:18:30,475][root][INFO] - ===> Epoch[10213](51065/5): Loss 0.0070, Sem 0.0070 LR: 1.802e-02 Score 99.750 Data time: 0.2198, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7246 [2021-07-09 04:18:44,828][root][INFO] - ===> Epoch[10214](51070/5): Loss 0.0070, Sem 0.0070 LR: 1.801e-02 Score 99.753 Data time: 0.2312, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8325 [2021-07-09 04:18:58,771][root][INFO] - ===> Epoch[10215](51075/5): Loss 0.0071, Sem 0.0071 LR: 1.800e-02 Score 99.744 Data time: 0.2041, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7456 [2021-07-09 04:19:12,381][root][INFO] - ===> Epoch[10216](51080/5): Loss 0.0073, Sem 0.0073 LR: 1.799e-02 Score 99.740 Data time: 0.1848, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6854 [2021-07-09 04:19:26,724][root][INFO] - ===> Epoch[10217](51085/5): Loss 0.0073, Sem 0.0073 LR: 1.798e-02 Score 99.737 Data time: 0.2182, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8271 [2021-07-09 04:19:40,760][root][INFO] - ===> Epoch[10218](51090/5): Loss 0.0068, Sem 0.0068 LR: 1.797e-02 Score 99.753 Data time: 0.2009, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7611 [2021-07-09 04:19:55,156][root][INFO] - ===> Epoch[10219](51095/5): Loss 0.0071, Sem 0.0071 LR: 1.796e-02 Score 99.746 Data time: 0.1904, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8305 [2021-07-09 04:20:10,048][root][INFO] - ===> Epoch[10220](51100/5): Loss 0.0070, Sem 0.0070 LR: 1.795e-02 Score 99.749 Data time: 0.2242, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.9179 [2021-07-09 04:20:22,934][root][INFO] - ===> Epoch[10221](51105/5): Loss 0.0074, Sem 0.0074 LR: 1.794e-02 Score 99.736 Data time: 0.1600, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.5326 [2021-07-09 04:20:37,013][root][INFO] - ===> Epoch[10222](51110/5): Loss 0.0078, Sem 0.0078 LR: 1.794e-02 Score 99.728 Data time: 0.1814, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7653 [2021-07-09 04:20:51,233][root][INFO] - ===> Epoch[10223](51115/5): Loss 0.0073, Sem 0.0073 LR: 1.793e-02 Score 99.741 Data time: 0.2658, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7895 [2021-07-09 04:21:06,141][root][INFO] - ===> Epoch[10224](51120/5): Loss 0.0070, Sem 0.0070 LR: 1.792e-02 Score 99.750 Data time: 0.2571, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.9320 [2021-07-09 04:21:20,926][root][INFO] - ===> Epoch[10225](51125/5): Loss 0.0069, Sem 0.0069 LR: 1.791e-02 Score 99.751 Data time: 0.2624, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.9028 [2021-07-09 04:21:33,813][root][INFO] - ===> Epoch[10226](51130/5): Loss 0.0074, Sem 0.0074 LR: 1.790e-02 Score 99.737 Data time: 0.2569, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.5253 [2021-07-09 04:21:47,053][root][INFO] - ===> Epoch[10227](51135/5): Loss 0.0072, Sem 0.0072 LR: 1.789e-02 Score 99.742 Data time: 0.2622, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.5972 [2021-07-09 04:22:01,427][root][INFO] - ===> Epoch[10228](51140/5): Loss 0.0074, Sem 0.0074 LR: 1.788e-02 Score 99.739 Data time: 0.2426, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8258 [2021-07-09 04:22:16,476][root][INFO] - ===> Epoch[10229](51145/5): Loss 0.0069, Sem 0.0069 LR: 1.787e-02 Score 99.753 Data time: 0.2875, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.9561 [2021-07-09 04:22:30,691][root][INFO] - ===> Epoch[10230](51150/5): Loss 0.0072, Sem 0.0072 LR: 1.786e-02 Score 99.737 Data time: 0.2239, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8050 [2021-07-09 04:22:44,307][root][INFO] - ===> Epoch[10231](51155/5): Loss 0.0072, Sem 0.0072 LR: 1.785e-02 Score 99.743 Data time: 0.2195, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6800 [2021-07-09 04:22:58,311][root][INFO] - ===> Epoch[10232](51160/5): Loss 0.0077, Sem 0.0077 LR: 1.784e-02 Score 99.727 Data time: 0.2629, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7617 [2021-07-09 04:23:12,531][root][INFO] - ===> Epoch[10233](51165/5): Loss 0.0074, Sem 0.0074 LR: 1.784e-02 Score 99.733 Data time: 0.2538, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8047 [2021-07-09 04:23:26,701][root][INFO] - ===> Epoch[10234](51170/5): Loss 0.0069, Sem 0.0069 LR: 1.783e-02 Score 99.753 Data time: 0.2341, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7850 [2021-07-09 04:23:41,443][root][INFO] - ===> Epoch[10235](51175/5): Loss 0.0070, Sem 0.0070 LR: 1.782e-02 Score 99.750 Data time: 0.2463, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8990 [2021-07-09 04:23:55,097][root][INFO] - ===> Epoch[10236](51180/5): Loss 0.0071, Sem 0.0071 LR: 1.781e-02 Score 99.745 Data time: 0.2202, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6747 [2021-07-09 04:24:08,667][root][INFO] - ===> Epoch[10237](51185/5): Loss 0.0073, Sem 0.0073 LR: 1.780e-02 Score 99.742 Data time: 0.1770, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6659 [2021-07-09 04:24:22,493][root][INFO] - ===> Epoch[10238](51190/5): Loss 0.0071, Sem 0.0071 LR: 1.779e-02 Score 99.745 Data time: 0.2596, Forward time: 1.2530, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7235 [2021-07-09 04:24:36,466][root][INFO] - ===> Epoch[10239](51195/5): Loss 0.0069, Sem 0.0069 LR: 1.778e-02 Score 99.754 Data time: 0.2482, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7447 [2021-07-09 04:24:51,583][root][INFO] - ===> Epoch[10240](51200/5): Loss 0.0068, Sem 0.0068 LR: 1.777e-02 Score 99.755 Data time: 0.2046, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.9795 [2021-07-09 04:24:51,584][root][INFO] - ===> Start testing [2021-07-09 04:25:30,624][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1278 Loss 0.410 (AVG: 0.617) Score 91.625 (AVG: 90.980) mIOU 71.536 mAP 82.653 mAcc 77.934 IOU: 64.028 0.000 80.142 74.996 94.941 92.050 33.777 84.787 98.507 82.782 79.565 86.084 58.306 mAP: 83.805 2.589 87.258 85.702 98.334 98.123 53.615 97.831 99.630 95.256 90.664 92.374 89.306 mAcc: 80.048 0.000 83.569 82.835 96.738 97.620 36.834 93.762 99.338 94.248 89.640 96.199 62.311 [2021-07-09 04:25:30,624][root][INFO] - Finished test. Elapsed time: 39.0390 [2021-07-09 04:25:31,717][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 04:25:44,561][root][INFO] - ===> Epoch[10241](51205/5): Loss 0.0072, Sem 0.0072 LR: 1.776e-02 Score 99.742 Data time: 0.2104, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.5264 [2021-07-09 04:25:59,045][root][INFO] - ===> Epoch[10242](51210/5): Loss 0.0076, Sem 0.0076 LR: 1.775e-02 Score 99.729 Data time: 0.1905, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8526 [2021-07-09 04:26:12,725][root][INFO] - ===> Epoch[10243](51215/5): Loss 0.0069, Sem 0.0069 LR: 1.774e-02 Score 99.756 Data time: 0.2124, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6843 [2021-07-09 04:26:25,749][root][INFO] - ===> Epoch[10244](51220/5): Loss 0.0071, Sem 0.0071 LR: 1.774e-02 Score 99.746 Data time: 0.2247, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.5589 [2021-07-09 04:26:40,063][root][INFO] - ===> Epoch[10245](51225/5): Loss 0.0068, Sem 0.0068 LR: 1.773e-02 Score 99.756 Data time: 0.2067, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.8177 [2021-07-09 04:26:53,633][root][INFO] - ===> Epoch[10246](51230/5): Loss 0.0074, Sem 0.0074 LR: 1.772e-02 Score 99.731 Data time: 0.2035, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6658 [2021-07-09 04:27:07,367][root][INFO] - ===> Epoch[10247](51235/5): Loss 0.0069, Sem 0.0069 LR: 1.771e-02 Score 99.753 Data time: 0.2015, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7111 [2021-07-09 04:27:21,378][root][INFO] - ===> Epoch[10248](51240/5): Loss 0.0070, Sem 0.0070 LR: 1.770e-02 Score 99.749 Data time: 0.2006, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7658 [2021-07-09 04:27:34,865][root][INFO] - ===> Epoch[10249](51245/5): Loss 0.0072, Sem 0.0072 LR: 1.769e-02 Score 99.743 Data time: 0.2337, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6559 [2021-07-09 04:27:48,873][root][INFO] - ===> Epoch[10250](51250/5): Loss 0.0072, Sem 0.0072 LR: 1.768e-02 Score 99.743 Data time: 0.1986, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7564 [2021-07-09 04:28:02,711][root][INFO] - ===> Epoch[10251](51255/5): Loss 0.0070, Sem 0.0070 LR: 1.767e-02 Score 99.748 Data time: 0.2402, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7158 [2021-07-09 04:28:16,722][root][INFO] - ===> Epoch[10252](51260/5): Loss 0.0073, Sem 0.0073 LR: 1.766e-02 Score 99.738 Data time: 0.2309, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7589 [2021-07-09 04:28:30,655][root][INFO] - ===> Epoch[10253](51265/5): Loss 0.0069, Sem 0.0069 LR: 1.765e-02 Score 99.748 Data time: 0.2124, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7348 [2021-07-09 04:28:44,420][root][INFO] - ===> Epoch[10254](51270/5): Loss 0.0074, Sem 0.0074 LR: 1.764e-02 Score 99.737 Data time: 0.2511, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.7069 [2021-07-09 04:28:58,002][root][INFO] - ===> Epoch[10255](51275/5): Loss 0.0072, Sem 0.0072 LR: 1.764e-02 Score 99.742 Data time: 0.1830, Forward time: 1.2529, Backward time: 1.2469, DDP time: 0.0042, Total iter time: 2.6780 [2021-07-09 04:29:12,007][root][INFO] - ===> Epoch[10256](51280/5): Loss 0.0069, Sem 0.0069 LR: 1.763e-02 Score 99.753 Data time: 0.1651, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7593 [2021-07-09 04:29:25,530][root][INFO] - ===> Epoch[10257](51285/5): Loss 0.0074, Sem 0.0074 LR: 1.762e-02 Score 99.735 Data time: 0.2420, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6579 [2021-07-09 04:29:39,613][root][INFO] - ===> Epoch[10258](51290/5): Loss 0.0071, Sem 0.0071 LR: 1.761e-02 Score 99.746 Data time: 0.1974, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7642 [2021-07-09 04:29:53,518][root][INFO] - ===> Epoch[10259](51295/5): Loss 0.0072, Sem 0.0072 LR: 1.760e-02 Score 99.741 Data time: 0.2089, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7390 [2021-07-09 04:30:08,480][root][INFO] - ===> Epoch[10260](51300/5): Loss 0.0067, Sem 0.0067 LR: 1.759e-02 Score 99.760 Data time: 0.2205, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.9429 [2021-07-09 04:30:22,617][root][INFO] - ===> Epoch[10261](51305/5): Loss 0.0070, Sem 0.0070 LR: 1.758e-02 Score 99.749 Data time: 0.2218, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7788 [2021-07-09 04:30:37,029][root][INFO] - ===> Epoch[10262](51310/5): Loss 0.0071, Sem 0.0071 LR: 1.757e-02 Score 99.745 Data time: 0.2122, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8352 [2021-07-09 04:30:50,625][root][INFO] - ===> Epoch[10263](51315/5): Loss 0.0067, Sem 0.0067 LR: 1.756e-02 Score 99.762 Data time: 0.2024, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6747 [2021-07-09 04:31:04,770][root][INFO] - ===> Epoch[10264](51320/5): Loss 0.0071, Sem 0.0071 LR: 1.755e-02 Score 99.747 Data time: 0.1556, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7920 [2021-07-09 04:31:18,590][root][INFO] - ===> Epoch[10265](51325/5): Loss 0.0070, Sem 0.0070 LR: 1.754e-02 Score 99.750 Data time: 0.1678, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7237 [2021-07-09 04:31:32,528][root][INFO] - ===> Epoch[10266](51330/5): Loss 0.0073, Sem 0.0073 LR: 1.754e-02 Score 99.737 Data time: 0.2195, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7467 [2021-07-09 04:31:47,442][root][INFO] - ===> Epoch[10267](51335/5): Loss 0.0072, Sem 0.0072 LR: 1.753e-02 Score 99.746 Data time: 0.1867, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.9456 [2021-07-09 04:32:01,810][root][INFO] - ===> Epoch[10268](51340/5): Loss 0.0072, Sem 0.0072 LR: 1.752e-02 Score 99.742 Data time: 0.2356, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.8285 [2021-07-09 04:32:15,670][root][INFO] - ===> Epoch[10269](51345/5): Loss 0.0071, Sem 0.0071 LR: 1.751e-02 Score 99.745 Data time: 0.1957, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7325 [2021-07-09 04:32:29,237][root][INFO] - ===> Epoch[10270](51350/5): Loss 0.0075, Sem 0.0075 LR: 1.750e-02 Score 99.729 Data time: 0.1978, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6716 [2021-07-09 04:32:42,656][root][INFO] - ===> Epoch[10271](51355/5): Loss 0.0072, Sem 0.0072 LR: 1.749e-02 Score 99.743 Data time: 0.2168, Forward time: 1.2529, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.6258 [2021-07-09 04:32:56,759][root][INFO] - ===> Epoch[10272](51360/5): Loss 0.0071, Sem 0.0071 LR: 1.748e-02 Score 99.747 Data time: 0.1542, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.7734 [2021-07-09 04:33:09,854][root][INFO] - ===> Epoch[10273](51365/5): Loss 0.0068, Sem 0.0068 LR: 1.747e-02 Score 99.755 Data time: 0.2114, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.5749 [2021-07-09 04:33:23,658][root][INFO] - ===> Epoch[10274](51370/5): Loss 0.0069, Sem 0.0069 LR: 1.746e-02 Score 99.755 Data time: 0.2464, Forward time: 1.2528, Backward time: 1.2470, DDP time: 0.0042, Total iter time: 2.7073 [2021-07-09 04:33:39,061][root][INFO] - ===> Epoch[10275](51375/5): Loss 0.0072, Sem 0.0072 LR: 1.745e-02 Score 99.743 Data time: 0.1723, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 3.0397 [2021-07-09 04:33:52,460][root][INFO] - ===> Epoch[10276](51380/5): Loss 0.0069, Sem 0.0069 LR: 1.744e-02 Score 99.752 Data time: 0.1920, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.6399 [2021-07-09 04:34:06,856][root][INFO] - ===> Epoch[10277](51385/5): Loss 0.0068, Sem 0.0068 LR: 1.744e-02 Score 99.758 Data time: 0.2038, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.8330 [2021-07-09 04:34:21,152][root][INFO] - ===> Epoch[10278](51390/5): Loss 0.0069, Sem 0.0069 LR: 1.743e-02 Score 99.753 Data time: 0.2347, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.8167 [2021-07-09 04:34:35,290][root][INFO] - ===> Epoch[10279](51395/5): Loss 0.0070, Sem 0.0070 LR: 1.742e-02 Score 99.750 Data time: 0.2275, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.7817 [2021-07-09 04:34:49,933][root][INFO] - ===> Epoch[10280](51400/5): Loss 0.0068, Sem 0.0068 LR: 1.741e-02 Score 99.758 Data time: 0.3073, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.8774 [2021-07-09 04:34:49,934][root][INFO] - ===> Start testing [2021-07-09 04:35:24,465][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1278 Loss 0.399 (AVG: 0.626) Score 92.868 (AVG: 91.076) mIOU 71.640 mAP 82.435 mAcc 77.760 IOU: 64.920 0.000 81.241 75.142 95.095 91.601 32.479 84.038 98.492 84.981 79.141 86.178 58.012 mAP: 83.843 2.764 87.464 85.876 98.323 98.014 52.021 98.032 99.670 93.725 90.454 92.094 89.375 mAcc: 80.692 0.000 83.714 82.875 96.691 97.653 35.286 95.308 99.329 91.940 89.654 96.400 61.334 [2021-07-09 04:35:24,465][root][INFO] - Finished test. Elapsed time: 34.5304 [2021-07-09 04:35:25,544][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 04:35:37,960][root][INFO] - ===> Epoch[10281](51405/5): Loss 0.0073, Sem 0.0073 LR: 1.740e-02 Score 99.740 Data time: 0.2389, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.4436 [2021-07-09 04:35:52,575][root][INFO] - ===> Epoch[10282](51410/5): Loss 0.0070, Sem 0.0070 LR: 1.739e-02 Score 99.749 Data time: 0.2927, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.8749 [2021-07-09 04:36:06,960][root][INFO] - ===> Epoch[10283](51415/5): Loss 0.0071, Sem 0.0071 LR: 1.738e-02 Score 99.746 Data time: 0.2490, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.8368 [2021-07-09 04:36:21,338][root][INFO] - ===> Epoch[10284](51420/5): Loss 0.0072, Sem 0.0072 LR: 1.737e-02 Score 99.744 Data time: 0.2679, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.8246 [2021-07-09 04:36:35,248][root][INFO] - ===> Epoch[10285](51425/5): Loss 0.0069, Sem 0.0069 LR: 1.736e-02 Score 99.751 Data time: 0.2270, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.7336 [2021-07-09 04:36:48,984][root][INFO] - ===> Epoch[10286](51430/5): Loss 0.0073, Sem 0.0073 LR: 1.735e-02 Score 99.739 Data time: 0.2169, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.7069 [2021-07-09 04:37:03,117][root][INFO] - ===> Epoch[10287](51435/5): Loss 0.0070, Sem 0.0070 LR: 1.734e-02 Score 99.753 Data time: 0.1856, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.7773 [2021-07-09 04:37:16,965][root][INFO] - ===> Epoch[10288](51440/5): Loss 0.0069, Sem 0.0069 LR: 1.734e-02 Score 99.755 Data time: 0.2284, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.7251 [2021-07-09 04:37:31,973][root][INFO] - ===> Epoch[10289](51445/5): Loss 0.0067, Sem 0.0067 LR: 1.733e-02 Score 99.761 Data time: 0.2647, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.9510 [2021-07-09 04:37:46,669][root][INFO] - ===> Epoch[10290](51450/5): Loss 0.0069, Sem 0.0069 LR: 1.732e-02 Score 99.752 Data time: 0.2193, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.8985 [2021-07-09 04:38:00,271][root][INFO] - ===> Epoch[10291](51455/5): Loss 0.0069, Sem 0.0069 LR: 1.731e-02 Score 99.754 Data time: 0.2359, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.6799 [2021-07-09 04:38:14,176][root][INFO] - ===> Epoch[10292](51460/5): Loss 0.0068, Sem 0.0068 LR: 1.730e-02 Score 99.756 Data time: 0.1933, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.7324 [2021-07-09 04:38:28,062][root][INFO] - ===> Epoch[10293](51465/5): Loss 0.0068, Sem 0.0068 LR: 1.729e-02 Score 99.756 Data time: 0.2229, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.7292 [2021-07-09 04:38:41,594][root][INFO] - ===> Epoch[10294](51470/5): Loss 0.0074, Sem 0.0074 LR: 1.728e-02 Score 99.737 Data time: 0.2551, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.6629 [2021-07-09 04:38:55,972][root][INFO] - ===> Epoch[10295](51475/5): Loss 0.0075, Sem 0.0075 LR: 1.727e-02 Score 99.733 Data time: 0.2165, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.8345 [2021-07-09 04:39:10,129][root][INFO] - ===> Epoch[10296](51480/5): Loss 0.0070, Sem 0.0070 LR: 1.726e-02 Score 99.748 Data time: 0.2179, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.7960 [2021-07-09 04:39:24,443][root][INFO] - ===> Epoch[10297](51485/5): Loss 0.0070, Sem 0.0070 LR: 1.725e-02 Score 99.746 Data time: 0.1960, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.8117 [2021-07-09 04:39:38,389][root][INFO] - ===> Epoch[10298](51490/5): Loss 0.0073, Sem 0.0073 LR: 1.724e-02 Score 99.739 Data time: 0.2148, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.7450 [2021-07-09 04:39:52,347][root][INFO] - ===> Epoch[10299](51495/5): Loss 0.0073, Sem 0.0073 LR: 1.723e-02 Score 99.740 Data time: 0.2295, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.7479 [2021-07-09 04:40:07,130][root][INFO] - ===> Epoch[10300](51500/5): Loss 0.0067, Sem 0.0067 LR: 1.723e-02 Score 99.762 Data time: 0.2117, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.9078 [2021-07-09 04:40:21,822][root][INFO] - ===> Epoch[10301](51505/5): Loss 0.0068, Sem 0.0068 LR: 1.722e-02 Score 99.758 Data time: 0.2418, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.8959 [2021-07-09 04:40:35,630][root][INFO] - ===> Epoch[10302](51510/5): Loss 0.0073, Sem 0.0073 LR: 1.721e-02 Score 99.739 Data time: 0.2263, Forward time: 1.2528, Backward time: 1.2471, DDP time: 0.0042, Total iter time: 2.7106 [2021-07-09 04:40:49,763][root][INFO] - ===> Epoch[10303](51515/5): Loss 0.0068, Sem 0.0068 LR: 1.720e-02 Score 99.756 Data time: 0.2077, Forward time: 1.2528, Backward time: 1.2472, DDP time: 0.0042, Total iter time: 2.7791 [2021-07-09 04:41:04,521][root][INFO] - ===> Epoch[10304](51520/5): Loss 0.0073, Sem 0.0073 LR: 1.719e-02 Score 99.741 Data time: 0.1842, Forward time: 1.2528, Backward time: 1.2472, DDP time: 0.0042, Total iter time: 2.9147 [2021-07-09 04:41:19,181][root][INFO] - ===> Epoch[10305](51525/5): Loss 0.0070, Sem 0.0070 LR: 1.718e-02 Score 99.747 Data time: 0.2120, Forward time: 1.2528, Backward time: 1.2472, DDP time: 0.0042, Total iter time: 2.8897 [2021-07-09 04:41:34,047][root][INFO] - ===> Epoch[10306](51530/5): Loss 0.0072, Sem 0.0072 LR: 1.717e-02 Score 99.739 Data time: 0.1606, Forward time: 1.2528, Backward time: 1.2472, DDP time: 0.0042, Total iter time: 2.9331 [2021-07-09 04:41:48,036][root][INFO] - ===> Epoch[10307](51535/5): Loss 0.0069, Sem 0.0069 LR: 1.716e-02 Score 99.754 Data time: 0.2294, Forward time: 1.2528, Backward time: 1.2472, DDP time: 0.0042, Total iter time: 2.7531 [2021-07-09 04:42:02,118][root][INFO] - ===> Epoch[10308](51540/5): Loss 0.0073, Sem 0.0073 LR: 1.715e-02 Score 99.741 Data time: 0.1898, Forward time: 1.2528, Backward time: 1.2472, DDP time: 0.0042, Total iter time: 2.7715 [2021-07-09 04:42:16,642][root][INFO] - ===> Epoch[10309](51545/5): Loss 0.0072, Sem 0.0072 LR: 1.714e-02 Score 99.737 Data time: 0.2143, Forward time: 1.2528, Backward time: 1.2472, DDP time: 0.0042, Total iter time: 2.8551 [2021-07-09 04:42:30,846][root][INFO] - ===> Epoch[10310](51550/5): Loss 0.0070, Sem 0.0070 LR: 1.713e-02 Score 99.748 Data time: 0.2159, Forward time: 1.2528, Backward time: 1.2472, DDP time: 0.0042, Total iter time: 2.8000 [2021-07-09 04:42:44,854][root][INFO] - ===> Epoch[10311](51555/5): Loss 0.0067, Sem 0.0067 LR: 1.713e-02 Score 99.762 Data time: 0.2455, Forward time: 1.2528, Backward time: 1.2472, DDP time: 0.0042, Total iter time: 2.7499 [2021-07-09 04:42:59,177][root][INFO] - ===> Epoch[10312](51560/5): Loss 0.0071, Sem 0.0071 LR: 1.712e-02 Score 99.746 Data time: 0.2292, Forward time: 1.2528, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.8192 [2021-07-09 04:43:13,384][root][INFO] - ===> Epoch[10313](51565/5): Loss 0.0071, Sem 0.0071 LR: 1.711e-02 Score 99.745 Data time: 0.1975, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7927 [2021-07-09 04:43:27,393][root][INFO] - ===> Epoch[10314](51570/5): Loss 0.0072, Sem 0.0072 LR: 1.710e-02 Score 99.742 Data time: 0.1898, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7603 [2021-07-09 04:43:41,343][root][INFO] - ===> Epoch[10315](51575/5): Loss 0.0071, Sem 0.0071 LR: 1.709e-02 Score 99.745 Data time: 0.1929, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7385 [2021-07-09 04:43:54,493][root][INFO] - ===> Epoch[10316](51580/5): Loss 0.0071, Sem 0.0071 LR: 1.708e-02 Score 99.746 Data time: 0.2482, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.5858 [2021-07-09 04:44:08,235][root][INFO] - ===> Epoch[10317](51585/5): Loss 0.0075, Sem 0.0075 LR: 1.707e-02 Score 99.733 Data time: 0.1808, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7014 [2021-07-09 04:44:23,172][root][INFO] - ===> Epoch[10318](51590/5): Loss 0.0071, Sem 0.0071 LR: 1.706e-02 Score 99.745 Data time: 0.2139, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.9424 [2021-07-09 04:44:37,502][root][INFO] - ===> Epoch[10319](51595/5): Loss 0.0069, Sem 0.0069 LR: 1.705e-02 Score 99.755 Data time: 0.1667, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.8216 [2021-07-09 04:44:51,907][root][INFO] - ===> Epoch[10320](51600/5): Loss 0.0070, Sem 0.0070 LR: 1.704e-02 Score 99.749 Data time: 0.2069, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.8331 [2021-07-09 04:44:51,908][root][INFO] - ===> Start testing [2021-07-09 04:45:29,007][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1216 Loss 0.437 (AVG: 0.619) Score 91.393 (AVG: 91.059) mIOU 71.663 mAP 82.832 mAcc 78.372 IOU: 64.772 0.000 80.381 75.060 94.871 91.426 35.426 82.573 98.432 84.678 79.470 86.517 58.015 mAP: 82.241 3.507 88.806 86.158 98.386 98.060 55.778 97.634 99.661 94.485 90.359 92.373 89.368 mAcc: 79.916 0.000 85.061 83.136 96.276 97.752 39.553 95.782 99.377 93.676 90.108 96.214 61.980 [2021-07-09 04:45:29,007][root][INFO] - Finished test. Elapsed time: 37.0982 [2021-07-09 04:45:30,130][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 04:45:43,137][root][INFO] - ===> Epoch[10321](51605/5): Loss 0.0069, Sem 0.0069 LR: 1.703e-02 Score 99.752 Data time: 0.1847, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.5644 [2021-07-09 04:45:56,992][root][INFO] - ===> Epoch[10322](51610/5): Loss 0.0071, Sem 0.0071 LR: 1.703e-02 Score 99.745 Data time: 0.2391, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7174 [2021-07-09 04:46:11,227][root][INFO] - ===> Epoch[10323](51615/5): Loss 0.0071, Sem 0.0071 LR: 1.702e-02 Score 99.744 Data time: 0.2307, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7911 [2021-07-09 04:46:25,501][root][INFO] - ===> Epoch[10324](51620/5): Loss 0.0069, Sem 0.0069 LR: 1.701e-02 Score 99.751 Data time: 0.2393, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7995 [2021-07-09 04:46:39,284][root][INFO] - ===> Epoch[10325](51625/5): Loss 0.0068, Sem 0.0068 LR: 1.700e-02 Score 99.758 Data time: 0.2175, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7088 [2021-07-09 04:46:52,755][root][INFO] - ===> Epoch[10326](51630/5): Loss 0.0070, Sem 0.0070 LR: 1.699e-02 Score 99.748 Data time: 0.2459, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.6447 [2021-07-09 04:47:07,511][root][INFO] - ===> Epoch[10327](51635/5): Loss 0.0071, Sem 0.0071 LR: 1.698e-02 Score 99.743 Data time: 0.2205, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.9084 [2021-07-09 04:47:21,487][root][INFO] - ===> Epoch[10328](51640/5): Loss 0.0070, Sem 0.0070 LR: 1.697e-02 Score 99.749 Data time: 0.2099, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7433 [2021-07-09 04:47:35,253][root][INFO] - ===> Epoch[10329](51645/5): Loss 0.0072, Sem 0.0072 LR: 1.696e-02 Score 99.745 Data time: 0.2995, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7086 [2021-07-09 04:47:49,403][root][INFO] - ===> Epoch[10330](51650/5): Loss 0.0073, Sem 0.0073 LR: 1.695e-02 Score 99.741 Data time: 0.2961, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7758 [2021-07-09 04:48:03,684][root][INFO] - ===> Epoch[10331](51655/5): Loss 0.0069, Sem 0.0069 LR: 1.694e-02 Score 99.755 Data time: 0.2549, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.8151 [2021-07-09 04:48:17,739][root][INFO] - ===> Epoch[10332](51660/5): Loss 0.0069, Sem 0.0069 LR: 1.693e-02 Score 99.754 Data time: 0.2590, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7613 [2021-07-09 04:48:31,351][root][INFO] - ===> Epoch[10333](51665/5): Loss 0.0071, Sem 0.0071 LR: 1.692e-02 Score 99.743 Data time: 0.2094, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.6715 [2021-07-09 04:48:45,988][root][INFO] - ===> Epoch[10334](51670/5): Loss 0.0070, Sem 0.0070 LR: 1.692e-02 Score 99.748 Data time: 0.2511, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.8758 [2021-07-09 04:49:00,542][root][INFO] - ===> Epoch[10335](51675/5): Loss 0.0068, Sem 0.0068 LR: 1.691e-02 Score 99.758 Data time: 0.3073, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.8644 [2021-07-09 04:49:13,861][root][INFO] - ===> Epoch[10336](51680/5): Loss 0.0071, Sem 0.0071 LR: 1.690e-02 Score 99.749 Data time: 0.2206, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.6167 [2021-07-09 04:49:27,822][root][INFO] - ===> Epoch[10337](51685/5): Loss 0.0071, Sem 0.0071 LR: 1.689e-02 Score 99.747 Data time: 0.3114, Forward time: 1.2527, Backward time: 1.2472, DDP time: 0.0042, Total iter time: 2.7409 [2021-07-09 04:49:41,272][root][INFO] - ===> Epoch[10338](51690/5): Loss 0.0073, Sem 0.0073 LR: 1.688e-02 Score 99.736 Data time: 0.2217, Forward time: 1.2527, Backward time: 1.2472, DDP time: 0.0042, Total iter time: 2.6531 [2021-07-09 04:49:54,844][root][INFO] - ===> Epoch[10339](51695/5): Loss 0.0073, Sem 0.0073 LR: 1.687e-02 Score 99.738 Data time: 0.2188, Forward time: 1.2527, Backward time: 1.2472, DDP time: 0.0042, Total iter time: 2.6722 [2021-07-09 04:50:08,936][root][INFO] - ===> Epoch[10340](51700/5): Loss 0.0068, Sem 0.0068 LR: 1.686e-02 Score 99.754 Data time: 0.2140, Forward time: 1.2527, Backward time: 1.2472, DDP time: 0.0042, Total iter time: 2.7698 [2021-07-09 04:50:22,181][root][INFO] - ===> Epoch[10341](51705/5): Loss 0.0074, Sem 0.0074 LR: 1.685e-02 Score 99.737 Data time: 0.1604, Forward time: 1.2527, Backward time: 1.2472, DDP time: 0.0042, Total iter time: 2.6143 [2021-07-09 04:50:36,110][root][INFO] - ===> Epoch[10342](51710/5): Loss 0.0074, Sem 0.0074 LR: 1.684e-02 Score 99.736 Data time: 0.2213, Forward time: 1.2527, Backward time: 1.2472, DDP time: 0.0042, Total iter time: 2.7327 [2021-07-09 04:50:49,955][root][INFO] - ===> Epoch[10343](51715/5): Loss 0.0073, Sem 0.0073 LR: 1.683e-02 Score 99.739 Data time: 0.1959, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7282 [2021-07-09 04:51:04,249][root][INFO] - ===> Epoch[10344](51720/5): Loss 0.0067, Sem 0.0067 LR: 1.682e-02 Score 99.764 Data time: 0.2570, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.8169 [2021-07-09 04:51:18,406][root][INFO] - ===> Epoch[10345](51725/5): Loss 0.0071, Sem 0.0071 LR: 1.681e-02 Score 99.746 Data time: 0.2278, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7815 [2021-07-09 04:51:31,359][root][INFO] - ===> Epoch[10346](51730/5): Loss 0.0068, Sem 0.0068 LR: 1.681e-02 Score 99.758 Data time: 0.2390, Forward time: 1.2527, Backward time: 1.2472, DDP time: 0.0042, Total iter time: 2.5521 [2021-07-09 04:51:45,612][root][INFO] - ===> Epoch[10347](51735/5): Loss 0.0072, Sem 0.0072 LR: 1.680e-02 Score 99.740 Data time: 0.2138, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7961 [2021-07-09 04:52:00,525][root][INFO] - ===> Epoch[10348](51740/5): Loss 0.0069, Sem 0.0069 LR: 1.679e-02 Score 99.752 Data time: 0.1854, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.9387 [2021-07-09 04:52:14,470][root][INFO] - ===> Epoch[10349](51745/5): Loss 0.0070, Sem 0.0070 LR: 1.678e-02 Score 99.749 Data time: 0.1858, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7392 [2021-07-09 04:52:28,164][root][INFO] - ===> Epoch[10350](51750/5): Loss 0.0074, Sem 0.0074 LR: 1.677e-02 Score 99.737 Data time: 0.1573, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7078 [2021-07-09 04:52:41,527][root][INFO] - ===> Epoch[10351](51755/5): Loss 0.0067, Sem 0.0067 LR: 1.676e-02 Score 99.758 Data time: 0.2528, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.6327 [2021-07-09 04:52:55,486][root][INFO] - ===> Epoch[10352](51760/5): Loss 0.0072, Sem 0.0072 LR: 1.675e-02 Score 99.740 Data time: 0.2077, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7530 [2021-07-09 04:53:09,648][root][INFO] - ===> Epoch[10353](51765/5): Loss 0.0073, Sem 0.0073 LR: 1.674e-02 Score 99.738 Data time: 0.2549, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7838 [2021-07-09 04:53:23,875][root][INFO] - ===> Epoch[10354](51770/5): Loss 0.0069, Sem 0.0069 LR: 1.673e-02 Score 99.754 Data time: 0.2254, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7977 [2021-07-09 04:53:37,306][root][INFO] - ===> Epoch[10355](51775/5): Loss 0.0072, Sem 0.0072 LR: 1.672e-02 Score 99.744 Data time: 0.2054, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.6472 [2021-07-09 04:53:51,416][root][INFO] - ===> Epoch[10356](51780/5): Loss 0.0066, Sem 0.0066 LR: 1.671e-02 Score 99.764 Data time: 0.2570, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7638 [2021-07-09 04:54:05,654][root][INFO] - ===> Epoch[10357](51785/5): Loss 0.0068, Sem 0.0068 LR: 1.671e-02 Score 99.758 Data time: 0.2321, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.8106 [2021-07-09 04:54:19,088][root][INFO] - ===> Epoch[10358](51790/5): Loss 0.0065, Sem 0.0065 LR: 1.670e-02 Score 99.765 Data time: 0.1991, Forward time: 1.2527, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.6467 [2021-07-09 04:54:34,525][root][INFO] - ===> Epoch[10359](51795/5): Loss 0.0070, Sem 0.0070 LR: 1.669e-02 Score 99.751 Data time: 0.2119, Forward time: 1.2526, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 3.0417 [2021-07-09 04:54:48,370][root][INFO] - ===> Epoch[10360](51800/5): Loss 0.0068, Sem 0.0068 LR: 1.668e-02 Score 99.758 Data time: 0.1971, Forward time: 1.2526, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7311 [2021-07-09 04:54:48,374][root][INFO] - ===> Start testing [2021-07-09 04:55:29,713][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1219 Loss 0.366 (AVG: 0.623) Score 92.039 (AVG: 91.010) mIOU 71.470 mAP 82.502 mAcc 77.852 IOU: 64.515 0.000 80.260 74.939 95.150 91.905 31.189 83.594 98.505 85.118 79.825 86.100 58.015 mAP: 82.585 2.764 88.511 85.403 98.325 98.044 52.435 97.883 99.658 94.668 90.525 92.277 89.444 mAcc: 80.048 0.000 84.872 82.549 96.805 97.274 33.929 94.916 99.382 93.893 89.979 96.371 62.052 [2021-07-09 04:55:29,713][root][INFO] - Finished test. Elapsed time: 41.3391 [2021-07-09 04:55:30,669][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 04:55:43,684][root][INFO] - ===> Epoch[10361](51805/5): Loss 0.0073, Sem 0.0073 LR: 1.667e-02 Score 99.741 Data time: 0.2166, Forward time: 1.2526, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.5607 [2021-07-09 04:55:57,399][root][INFO] - ===> Epoch[10362](51810/5): Loss 0.0069, Sem 0.0069 LR: 1.666e-02 Score 99.753 Data time: 0.1515, Forward time: 1.2526, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7032 [2021-07-09 04:56:11,538][root][INFO] - ===> Epoch[10363](51815/5): Loss 0.0073, Sem 0.0073 LR: 1.665e-02 Score 99.736 Data time: 0.1720, Forward time: 1.2526, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7783 [2021-07-09 04:56:24,994][root][INFO] - ===> Epoch[10364](51820/5): Loss 0.0068, Sem 0.0068 LR: 1.664e-02 Score 99.755 Data time: 0.2118, Forward time: 1.2526, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.6524 [2021-07-09 04:56:39,132][root][INFO] - ===> Epoch[10365](51825/5): Loss 0.0068, Sem 0.0068 LR: 1.663e-02 Score 99.756 Data time: 0.2098, Forward time: 1.2526, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7844 [2021-07-09 04:56:53,790][root][INFO] - ===> Epoch[10366](51830/5): Loss 0.0071, Sem 0.0071 LR: 1.662e-02 Score 99.743 Data time: 0.2253, Forward time: 1.2526, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.8843 [2021-07-09 04:57:07,897][root][INFO] - ===> Epoch[10367](51835/5): Loss 0.0067, Sem 0.0067 LR: 1.661e-02 Score 99.756 Data time: 0.2758, Forward time: 1.2526, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7631 [2021-07-09 04:57:21,900][root][INFO] - ===> Epoch[10368](51840/5): Loss 0.0071, Sem 0.0071 LR: 1.660e-02 Score 99.743 Data time: 0.1867, Forward time: 1.2526, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7606 [2021-07-09 04:57:35,560][root][INFO] - ===> Epoch[10369](51845/5): Loss 0.0071, Sem 0.0071 LR: 1.660e-02 Score 99.745 Data time: 0.2173, Forward time: 1.2526, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.6847 [2021-07-09 04:57:49,683][root][INFO] - ===> Epoch[10370](51850/5): Loss 0.0069, Sem 0.0069 LR: 1.659e-02 Score 99.752 Data time: 0.1677, Forward time: 1.2526, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7746 [2021-07-09 04:58:03,000][root][INFO] - ===> Epoch[10371](51855/5): Loss 0.0070, Sem 0.0070 LR: 1.658e-02 Score 99.749 Data time: 0.2416, Forward time: 1.2526, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.6182 [2021-07-09 04:58:17,062][root][INFO] - ===> Epoch[10372](51860/5): Loss 0.0070, Sem 0.0070 LR: 1.657e-02 Score 99.749 Data time: 0.1515, Forward time: 1.2526, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7703 [2021-07-09 04:58:30,665][root][INFO] - ===> Epoch[10373](51865/5): Loss 0.0072, Sem 0.0072 LR: 1.656e-02 Score 99.742 Data time: 0.2215, Forward time: 1.2526, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.6772 [2021-07-09 04:58:44,737][root][INFO] - ===> Epoch[10374](51870/5): Loss 0.0068, Sem 0.0068 LR: 1.655e-02 Score 99.758 Data time: 0.1982, Forward time: 1.2526, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.7585 [2021-07-09 04:58:58,740][root][INFO] - ===> Epoch[10375](51875/5): Loss 0.0069, Sem 0.0069 LR: 1.654e-02 Score 99.753 Data time: 0.1617, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.7638 [2021-07-09 04:59:12,890][root][INFO] - ===> Epoch[10376](51880/5): Loss 0.0070, Sem 0.0070 LR: 1.653e-02 Score 99.749 Data time: 0.1994, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.7817 [2021-07-09 04:59:25,979][root][INFO] - ===> Epoch[10377](51885/5): Loss 0.0071, Sem 0.0071 LR: 1.652e-02 Score 99.744 Data time: 0.2256, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.5757 [2021-07-09 04:59:39,125][root][INFO] - ===> Epoch[10378](51890/5): Loss 0.0070, Sem 0.0070 LR: 1.651e-02 Score 99.750 Data time: 0.2319, Forward time: 1.2526, Backward time: 1.2473, DDP time: 0.0042, Total iter time: 2.5837 [2021-07-09 04:59:54,641][root][INFO] - ===> Epoch[10379](51895/5): Loss 0.0071, Sem 0.0071 LR: 1.650e-02 Score 99.750 Data time: 0.2014, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 3.0602 [2021-07-09 05:00:08,620][root][INFO] - ===> Epoch[10380](51900/5): Loss 0.0071, Sem 0.0071 LR: 1.649e-02 Score 99.743 Data time: 0.1781, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.7547 [2021-07-09 05:00:21,755][root][INFO] - ===> Epoch[10381](51905/5): Loss 0.0070, Sem 0.0070 LR: 1.649e-02 Score 99.746 Data time: 0.2186, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.5823 [2021-07-09 05:00:36,225][root][INFO] - ===> Epoch[10382](51910/5): Loss 0.0069, Sem 0.0069 LR: 1.648e-02 Score 99.754 Data time: 0.2438, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.8452 [2021-07-09 05:00:50,739][root][INFO] - ===> Epoch[10383](51915/5): Loss 0.0070, Sem 0.0070 LR: 1.647e-02 Score 99.752 Data time: 0.2791, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.8473 [2021-07-09 05:01:05,271][root][INFO] - ===> Epoch[10384](51920/5): Loss 0.0069, Sem 0.0069 LR: 1.646e-02 Score 99.751 Data time: 0.2603, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.8603 [2021-07-09 05:01:19,203][root][INFO] - ===> Epoch[10385](51925/5): Loss 0.0068, Sem 0.0068 LR: 1.645e-02 Score 99.755 Data time: 0.2754, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.7458 [2021-07-09 05:01:33,955][root][INFO] - ===> Epoch[10386](51930/5): Loss 0.0069, Sem 0.0069 LR: 1.644e-02 Score 99.758 Data time: 0.1902, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.9127 [2021-07-09 05:01:48,713][root][INFO] - ===> Epoch[10387](51935/5): Loss 0.0071, Sem 0.0071 LR: 1.643e-02 Score 99.746 Data time: 0.2991, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.9026 [2021-07-09 05:02:02,486][root][INFO] - ===> Epoch[10388](51940/5): Loss 0.0072, Sem 0.0072 LR: 1.642e-02 Score 99.748 Data time: 0.2021, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.7040 [2021-07-09 05:02:16,608][root][INFO] - ===> Epoch[10389](51945/5): Loss 0.0071, Sem 0.0071 LR: 1.641e-02 Score 99.744 Data time: 0.2619, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.7832 [2021-07-09 05:02:31,335][root][INFO] - ===> Epoch[10390](51950/5): Loss 0.0069, Sem 0.0069 LR: 1.640e-02 Score 99.753 Data time: 0.2700, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.8954 [2021-07-09 05:02:45,973][root][INFO] - ===> Epoch[10391](51955/5): Loss 0.0069, Sem 0.0069 LR: 1.639e-02 Score 99.750 Data time: 0.2360, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.8845 [2021-07-09 05:03:00,323][root][INFO] - ===> Epoch[10392](51960/5): Loss 0.0073, Sem 0.0073 LR: 1.638e-02 Score 99.736 Data time: 0.2790, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.8177 [2021-07-09 05:03:14,121][root][INFO] - ===> Epoch[10393](51965/5): Loss 0.0071, Sem 0.0071 LR: 1.638e-02 Score 99.744 Data time: 0.2552, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.7171 [2021-07-09 05:03:28,819][root][INFO] - ===> Epoch[10394](51970/5): Loss 0.0072, Sem 0.0072 LR: 1.637e-02 Score 99.741 Data time: 0.2175, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.8992 [2021-07-09 05:03:43,228][root][INFO] - ===> Epoch[10395](51975/5): Loss 0.0071, Sem 0.0071 LR: 1.636e-02 Score 99.745 Data time: 0.2327, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.8305 [2021-07-09 05:03:57,239][root][INFO] - ===> Epoch[10396](51980/5): Loss 0.0070, Sem 0.0070 LR: 1.635e-02 Score 99.750 Data time: 0.2515, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.7638 [2021-07-09 05:04:10,934][root][INFO] - ===> Epoch[10397](51985/5): Loss 0.0072, Sem 0.0072 LR: 1.634e-02 Score 99.743 Data time: 0.1836, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.6855 [2021-07-09 05:04:25,008][root][INFO] - ===> Epoch[10398](51990/5): Loss 0.0070, Sem 0.0070 LR: 1.633e-02 Score 99.749 Data time: 0.2499, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.7681 [2021-07-09 05:04:38,594][root][INFO] - ===> Epoch[10399](51995/5): Loss 0.0072, Sem 0.0072 LR: 1.632e-02 Score 99.744 Data time: 0.2482, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.6757 [2021-07-09 05:04:52,092][root][INFO] - ===> Epoch[10400](52000/5): Loss 0.0071, Sem 0.0071 LR: 1.631e-02 Score 99.745 Data time: 0.2662, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.6479 [2021-07-09 05:04:52,094][root][INFO] - ===> Start testing [2021-07-09 05:05:32,835][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1292 Loss 0.411 (AVG: 0.622) Score 92.023 (AVG: 91.005) mIOU 71.377 mAP 82.745 mAcc 77.973 IOU: 64.378 0.000 79.983 75.346 94.951 90.983 32.882 83.797 98.625 84.002 79.578 86.124 57.249 mAP: 84.076 3.132 88.247 86.052 98.283 98.008 53.703 97.772 99.625 94.591 90.625 92.137 89.437 mAcc: 78.888 0.000 84.316 83.890 96.692 97.475 35.947 95.066 99.428 94.436 90.192 96.208 61.110 [2021-07-09 05:05:32,836][root][INFO] - Finished test. Elapsed time: 40.7413 [2021-07-09 05:05:33,932][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 05:05:47,158][root][INFO] - ===> Epoch[10401](52005/5): Loss 0.0070, Sem 0.0070 LR: 1.630e-02 Score 99.749 Data time: 0.2690, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.6043 [2021-07-09 05:06:00,806][root][INFO] - ===> Epoch[10402](52010/5): Loss 0.0071, Sem 0.0071 LR: 1.629e-02 Score 99.747 Data time: 0.2030, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.6929 [2021-07-09 05:06:14,517][root][INFO] - ===> Epoch[10403](52015/5): Loss 0.0071, Sem 0.0071 LR: 1.628e-02 Score 99.746 Data time: 0.2598, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.6923 [2021-07-09 05:06:29,134][root][INFO] - ===> Epoch[10404](52020/5): Loss 0.0068, Sem 0.0068 LR: 1.627e-02 Score 99.760 Data time: 0.2455, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.8526 [2021-07-09 05:06:42,951][root][INFO] - ===> Epoch[10405](52025/5): Loss 0.0072, Sem 0.0072 LR: 1.627e-02 Score 99.743 Data time: 0.2285, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.7234 [2021-07-09 05:06:57,225][root][INFO] - ===> Epoch[10406](52030/5): Loss 0.0069, Sem 0.0069 LR: 1.626e-02 Score 99.754 Data time: 0.2215, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.8005 [2021-07-09 05:07:11,851][root][INFO] - ===> Epoch[10407](52035/5): Loss 0.0069, Sem 0.0069 LR: 1.625e-02 Score 99.752 Data time: 0.2062, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.8749 [2021-07-09 05:07:25,708][root][INFO] - ===> Epoch[10408](52040/5): Loss 0.0070, Sem 0.0070 LR: 1.624e-02 Score 99.751 Data time: 0.1967, Forward time: 1.2526, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.7237 [2021-07-09 05:07:40,095][root][INFO] - ===> Epoch[10409](52045/5): Loss 0.0072, Sem 0.0072 LR: 1.623e-02 Score 99.745 Data time: 0.2148, Forward time: 1.2525, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.8341 [2021-07-09 05:07:54,898][root][INFO] - ===> Epoch[10410](52050/5): Loss 0.0072, Sem 0.0072 LR: 1.622e-02 Score 99.743 Data time: 0.1520, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.9189 [2021-07-09 05:08:08,877][root][INFO] - ===> Epoch[10411](52055/5): Loss 0.0068, Sem 0.0068 LR: 1.621e-02 Score 99.757 Data time: 0.2024, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.7358 [2021-07-09 05:08:22,822][root][INFO] - ===> Epoch[10412](52060/5): Loss 0.0073, Sem 0.0073 LR: 1.620e-02 Score 99.738 Data time: 0.2082, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.7422 [2021-07-09 05:08:36,185][root][INFO] - ===> Epoch[10413](52065/5): Loss 0.0070, Sem 0.0070 LR: 1.619e-02 Score 99.748 Data time: 0.1941, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.6288 [2021-07-09 05:08:50,105][root][INFO] - ===> Epoch[10414](52070/5): Loss 0.0072, Sem 0.0072 LR: 1.618e-02 Score 99.743 Data time: 0.2554, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.7360 [2021-07-09 05:09:04,127][root][INFO] - ===> Epoch[10415](52075/5): Loss 0.0065, Sem 0.0065 LR: 1.617e-02 Score 99.768 Data time: 0.1905, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.7582 [2021-07-09 05:09:18,565][root][INFO] - ===> Epoch[10416](52080/5): Loss 0.0069, Sem 0.0069 LR: 1.616e-02 Score 99.754 Data time: 0.2504, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.8488 [2021-07-09 05:09:32,394][root][INFO] - ===> Epoch[10417](52085/5): Loss 0.0070, Sem 0.0070 LR: 1.616e-02 Score 99.749 Data time: 0.1711, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.7236 [2021-07-09 05:09:46,477][root][INFO] - ===> Epoch[10418](52090/5): Loss 0.0070, Sem 0.0070 LR: 1.615e-02 Score 99.750 Data time: 0.1572, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.7759 [2021-07-09 05:09:59,857][root][INFO] - ===> Epoch[10419](52095/5): Loss 0.0068, Sem 0.0068 LR: 1.614e-02 Score 99.757 Data time: 0.1837, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.6334 [2021-07-09 05:10:14,146][root][INFO] - ===> Epoch[10420](52100/5): Loss 0.0067, Sem 0.0067 LR: 1.613e-02 Score 99.761 Data time: 0.1954, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.8123 [2021-07-09 05:10:28,073][root][INFO] - ===> Epoch[10421](52105/5): Loss 0.0069, Sem 0.0069 LR: 1.612e-02 Score 99.755 Data time: 0.2571, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.7407 [2021-07-09 05:10:41,349][root][INFO] - ===> Epoch[10422](52110/5): Loss 0.0068, Sem 0.0068 LR: 1.611e-02 Score 99.755 Data time: 0.2227, Forward time: 1.2525, Backward time: 1.2474, DDP time: 0.0042, Total iter time: 2.6043 [2021-07-09 05:10:55,293][root][INFO] - ===> Epoch[10423](52115/5): Loss 0.0068, Sem 0.0068 LR: 1.610e-02 Score 99.753 Data time: 0.2126, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.7503 [2021-07-09 05:11:09,651][root][INFO] - ===> Epoch[10424](52120/5): Loss 0.0071, Sem 0.0071 LR: 1.609e-02 Score 99.744 Data time: 0.1612, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.8260 [2021-07-09 05:11:24,160][root][INFO] - ===> Epoch[10425](52125/5): Loss 0.0064, Sem 0.0064 LR: 1.608e-02 Score 99.773 Data time: 0.2204, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.8586 [2021-07-09 05:11:38,046][root][INFO] - ===> Epoch[10426](52130/5): Loss 0.0068, Sem 0.0068 LR: 1.607e-02 Score 99.757 Data time: 0.2086, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.7328 [2021-07-09 05:11:52,134][root][INFO] - ===> Epoch[10427](52135/5): Loss 0.0069, Sem 0.0069 LR: 1.606e-02 Score 99.750 Data time: 0.1358, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.7820 [2021-07-09 05:12:06,656][root][INFO] - ===> Epoch[10428](52140/5): Loss 0.0072, Sem 0.0072 LR: 1.605e-02 Score 99.743 Data time: 0.2103, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.8583 [2021-07-09 05:12:21,079][root][INFO] - ===> Epoch[10429](52145/5): Loss 0.0067, Sem 0.0067 LR: 1.604e-02 Score 99.759 Data time: 0.2448, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.8404 [2021-07-09 05:12:34,409][root][INFO] - ===> Epoch[10430](52150/5): Loss 0.0075, Sem 0.0075 LR: 1.604e-02 Score 99.727 Data time: 0.1968, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.6255 [2021-07-09 05:12:48,370][root][INFO] - ===> Epoch[10431](52155/5): Loss 0.0069, Sem 0.0069 LR: 1.603e-02 Score 99.753 Data time: 0.2415, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.7503 [2021-07-09 05:13:02,495][root][INFO] - ===> Epoch[10432](52160/5): Loss 0.0070, Sem 0.0070 LR: 1.602e-02 Score 99.748 Data time: 0.1501, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.7807 [2021-07-09 05:13:17,150][root][INFO] - ===> Epoch[10433](52165/5): Loss 0.0065, Sem 0.0065 LR: 1.601e-02 Score 99.768 Data time: 0.1756, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.8948 [2021-07-09 05:13:31,402][root][INFO] - ===> Epoch[10434](52170/5): Loss 0.0074, Sem 0.0074 LR: 1.600e-02 Score 99.736 Data time: 0.2139, Forward time: 1.2525, Backward time: 1.2476, DDP time: 0.0042, Total iter time: 2.8079 [2021-07-09 05:13:45,493][root][INFO] - ===> Epoch[10435](52175/5): Loss 0.0069, Sem 0.0069 LR: 1.599e-02 Score 99.753 Data time: 0.2242, Forward time: 1.2525, Backward time: 1.2476, DDP time: 0.0042, Total iter time: 2.7760 [2021-07-09 05:13:58,990][root][INFO] - ===> Epoch[10436](52180/5): Loss 0.0072, Sem 0.0072 LR: 1.598e-02 Score 99.742 Data time: 0.2392, Forward time: 1.2525, Backward time: 1.2476, DDP time: 0.0042, Total iter time: 2.6523 [2021-07-09 05:14:12,624][root][INFO] - ===> Epoch[10437](52185/5): Loss 0.0070, Sem 0.0070 LR: 1.597e-02 Score 99.750 Data time: 0.2031, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.6851 [2021-07-09 05:14:27,064][root][INFO] - ===> Epoch[10438](52190/5): Loss 0.0074, Sem 0.0074 LR: 1.596e-02 Score 99.733 Data time: 0.2373, Forward time: 1.2525, Backward time: 1.2476, DDP time: 0.0042, Total iter time: 2.8403 [2021-07-09 05:14:42,011][root][INFO] - ===> Epoch[10439](52195/5): Loss 0.0070, Sem 0.0070 LR: 1.595e-02 Score 99.749 Data time: 0.2764, Forward time: 1.2525, Backward time: 1.2476, DDP time: 0.0042, Total iter time: 2.9403 [2021-07-09 05:14:55,835][root][INFO] - ===> Epoch[10440](52200/5): Loss 0.0068, Sem 0.0068 LR: 1.594e-02 Score 99.758 Data time: 0.2177, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.7177 [2021-07-09 05:14:55,837][root][INFO] - ===> Start testing [2021-07-09 05:15:36,597][root][INFO] - 68/68: Data time: 0.0016, Iter time: 0.1363 Loss 0.411 (AVG: 0.618) Score 91.670 (AVG: 91.089) mIOU 71.548 mAP 82.595 mAcc 78.074 IOU: 64.406 0.000 81.166 75.266 95.214 91.404 32.753 82.745 98.598 84.562 79.657 86.517 57.839 mAP: 82.949 2.459 88.841 85.735 98.219 97.900 54.064 97.844 99.681 93.988 90.316 92.403 89.337 mAcc: 79.313 0.000 84.573 83.398 96.871 97.329 35.978 96.194 99.369 93.814 90.232 96.285 61.602 [2021-07-09 05:15:36,597][root][INFO] - Finished test. Elapsed time: 40.7585 [2021-07-09 05:15:37,646][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 05:15:50,636][root][INFO] - ===> Epoch[10441](52205/5): Loss 0.0070, Sem 0.0070 LR: 1.593e-02 Score 99.750 Data time: 0.2150, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.5588 [2021-07-09 05:16:04,573][root][INFO] - ===> Epoch[10442](52210/5): Loss 0.0070, Sem 0.0070 LR: 1.593e-02 Score 99.753 Data time: 0.2534, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.7476 [2021-07-09 05:16:19,022][root][INFO] - ===> Epoch[10443](52215/5): Loss 0.0072, Sem 0.0072 LR: 1.592e-02 Score 99.741 Data time: 0.2735, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.8436 [2021-07-09 05:16:32,906][root][INFO] - ===> Epoch[10444](52220/5): Loss 0.0072, Sem 0.0072 LR: 1.591e-02 Score 99.741 Data time: 0.1558, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.7285 [2021-07-09 05:16:46,462][root][INFO] - ===> Epoch[10445](52225/5): Loss 0.0070, Sem 0.0070 LR: 1.590e-02 Score 99.752 Data time: 0.2276, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.6669 [2021-07-09 05:17:00,126][root][INFO] - ===> Epoch[10446](52230/5): Loss 0.0074, Sem 0.0074 LR: 1.589e-02 Score 99.737 Data time: 0.1772, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.7003 [2021-07-09 05:17:13,890][root][INFO] - ===> Epoch[10447](52235/5): Loss 0.0068, Sem 0.0068 LR: 1.588e-02 Score 99.756 Data time: 0.1940, Forward time: 1.2525, Backward time: 1.2475, DDP time: 0.0042, Total iter time: 2.7153 [2021-07-09 05:17:28,599][root][INFO] - ===> Epoch[10448](52240/5): Loss 0.0067, Sem 0.0067 LR: 1.587e-02 Score 99.761 Data time: 0.2412, Forward time: 1.2525, Backward time: 1.2476, DDP time: 0.0042, Total iter time: 2.8862 [2021-07-09 05:17:42,765][root][INFO] - ===> Epoch[10449](52245/5): Loss 0.0070, Sem 0.0070 LR: 1.586e-02 Score 99.750 Data time: 0.2088, Forward time: 1.2524, Backward time: 1.2476, DDP time: 0.0042, Total iter time: 2.7920 [2021-07-09 05:17:57,778][root][INFO] - ===> Epoch[10450](52250/5): Loss 0.0068, Sem 0.0068 LR: 1.585e-02 Score 99.756 Data time: 0.2192, Forward time: 1.2524, Backward time: 1.2476, DDP time: 0.0042, Total iter time: 2.9600 [2021-07-09 05:18:11,000][root][INFO] - ===> Epoch[10451](52255/5): Loss 0.0073, Sem 0.0073 LR: 1.584e-02 Score 99.736 Data time: 0.2076, Forward time: 1.2524, Backward time: 1.2476, DDP time: 0.0042, Total iter time: 2.6022 [2021-07-09 05:18:25,547][root][INFO] - ===> Epoch[10452](52260/5): Loss 0.0068, Sem 0.0068 LR: 1.583e-02 Score 99.759 Data time: 0.2211, Forward time: 1.2524, Backward time: 1.2476, DDP time: 0.0042, Total iter time: 2.8603 [2021-07-09 05:18:39,953][root][INFO] - ===> Epoch[10453](52265/5): Loss 0.0071, Sem 0.0071 LR: 1.582e-02 Score 99.746 Data time: 0.1944, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.8406 [2021-07-09 05:18:54,105][root][INFO] - ===> Epoch[10454](52270/5): Loss 0.0069, Sem 0.0069 LR: 1.582e-02 Score 99.754 Data time: 0.1978, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7879 [2021-07-09 05:19:08,134][root][INFO] - ===> Epoch[10455](52275/5): Loss 0.0073, Sem 0.0073 LR: 1.581e-02 Score 99.738 Data time: 0.2479, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7511 [2021-07-09 05:19:22,875][root][INFO] - ===> Epoch[10456](52280/5): Loss 0.0068, Sem 0.0068 LR: 1.580e-02 Score 99.757 Data time: 0.2442, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.9022 [2021-07-09 05:19:35,792][root][INFO] - ===> Epoch[10457](52285/5): Loss 0.0070, Sem 0.0070 LR: 1.579e-02 Score 99.747 Data time: 0.1925, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.5390 [2021-07-09 05:19:50,260][root][INFO] - ===> Epoch[10458](52290/5): Loss 0.0069, Sem 0.0069 LR: 1.578e-02 Score 99.753 Data time: 0.2389, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.8457 [2021-07-09 05:20:04,785][root][INFO] - ===> Epoch[10459](52295/5): Loss 0.0071, Sem 0.0071 LR: 1.577e-02 Score 99.749 Data time: 0.2341, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.8661 [2021-07-09 05:20:19,766][root][INFO] - ===> Epoch[10460](52300/5): Loss 0.0065, Sem 0.0065 LR: 1.576e-02 Score 99.766 Data time: 0.2348, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.9520 [2021-07-09 05:20:33,618][root][INFO] - ===> Epoch[10461](52305/5): Loss 0.0074, Sem 0.0074 LR: 1.575e-02 Score 99.744 Data time: 0.2140, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7250 [2021-07-09 05:20:47,723][root][INFO] - ===> Epoch[10462](52310/5): Loss 0.0071, Sem 0.0071 LR: 1.574e-02 Score 99.747 Data time: 0.2424, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7799 [2021-07-09 05:21:01,757][root][INFO] - ===> Epoch[10463](52315/5): Loss 0.0070, Sem 0.0070 LR: 1.573e-02 Score 99.751 Data time: 0.1770, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7447 [2021-07-09 05:21:16,164][root][INFO] - ===> Epoch[10464](52320/5): Loss 0.0069, Sem 0.0069 LR: 1.572e-02 Score 99.754 Data time: 0.2497, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.8358 [2021-07-09 05:21:30,339][root][INFO] - ===> Epoch[10465](52325/5): Loss 0.0070, Sem 0.0070 LR: 1.571e-02 Score 99.749 Data time: 0.2189, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7931 [2021-07-09 05:21:43,607][root][INFO] - ===> Epoch[10466](52330/5): Loss 0.0069, Sem 0.0069 LR: 1.570e-02 Score 99.752 Data time: 0.2269, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.6040 [2021-07-09 05:21:57,613][root][INFO] - ===> Epoch[10467](52335/5): Loss 0.0068, Sem 0.0068 LR: 1.570e-02 Score 99.759 Data time: 0.2250, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7483 [2021-07-09 05:22:11,957][root][INFO] - ===> Epoch[10468](52340/5): Loss 0.0070, Sem 0.0070 LR: 1.569e-02 Score 99.751 Data time: 0.2343, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.8193 [2021-07-09 05:22:26,644][root][INFO] - ===> Epoch[10469](52345/5): Loss 0.0069, Sem 0.0069 LR: 1.568e-02 Score 99.755 Data time: 0.2073, Forward time: 1.2523, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.9013 [2021-07-09 05:22:40,280][root][INFO] - ===> Epoch[10470](52350/5): Loss 0.0073, Sem 0.0073 LR: 1.567e-02 Score 99.737 Data time: 0.1924, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.6830 [2021-07-09 05:22:53,802][root][INFO] - ===> Epoch[10471](52355/5): Loss 0.0066, Sem 0.0066 LR: 1.566e-02 Score 99.764 Data time: 0.2089, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.6663 [2021-07-09 05:23:06,963][root][INFO] - ===> Epoch[10472](52360/5): Loss 0.0078, Sem 0.0078 LR: 1.565e-02 Score 99.720 Data time: 0.1900, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.5962 [2021-07-09 05:23:21,119][root][INFO] - ===> Epoch[10473](52365/5): Loss 0.0067, Sem 0.0067 LR: 1.564e-02 Score 99.763 Data time: 0.2332, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7874 [2021-07-09 05:23:34,777][root][INFO] - ===> Epoch[10474](52370/5): Loss 0.0073, Sem 0.0073 LR: 1.563e-02 Score 99.739 Data time: 0.1995, Forward time: 1.2523, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.6900 [2021-07-09 05:23:48,790][root][INFO] - ===> Epoch[10475](52375/5): Loss 0.0071, Sem 0.0071 LR: 1.562e-02 Score 99.746 Data time: 0.1691, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7623 [2021-07-09 05:24:02,888][root][INFO] - ===> Epoch[10476](52380/5): Loss 0.0069, Sem 0.0069 LR: 1.561e-02 Score 99.754 Data time: 0.2157, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7793 [2021-07-09 05:24:16,274][root][INFO] - ===> Epoch[10477](52385/5): Loss 0.0068, Sem 0.0068 LR: 1.560e-02 Score 99.758 Data time: 0.2280, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.6348 [2021-07-09 05:24:30,095][root][INFO] - ===> Epoch[10478](52390/5): Loss 0.0073, Sem 0.0073 LR: 1.559e-02 Score 99.736 Data time: 0.1943, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7151 [2021-07-09 05:24:44,363][root][INFO] - ===> Epoch[10479](52395/5): Loss 0.0070, Sem 0.0070 LR: 1.558e-02 Score 99.751 Data time: 0.2417, Forward time: 1.2523, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.8188 [2021-07-09 05:24:58,174][root][INFO] - ===> Epoch[10480](52400/5): Loss 0.0071, Sem 0.0071 LR: 1.558e-02 Score 99.747 Data time: 0.2182, Forward time: 1.2523, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7164 [2021-07-09 05:24:58,176][root][INFO] - ===> Start testing [2021-07-09 05:25:36,938][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1242 Loss 0.360 (AVG: 0.618) Score 93.131 (AVG: 91.096) mIOU 71.675 mAP 82.620 mAcc 77.839 IOU: 64.377 0.000 80.253 75.488 94.949 91.708 32.907 83.056 98.584 85.506 80.283 86.494 58.164 mAP: 82.942 2.696 87.211 85.655 98.344 98.093 54.215 97.562 99.664 95.050 90.651 91.909 90.072 mAcc: 79.183 0.000 83.137 83.594 96.785 97.529 35.642 94.258 99.353 94.110 89.621 96.657 62.044 [2021-07-09 05:25:36,939][root][INFO] - Finished test. Elapsed time: 38.7621 [2021-07-09 05:25:38,044][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 05:25:50,812][root][INFO] - ===> Epoch[10481](52405/5): Loss 0.0069, Sem 0.0069 LR: 1.557e-02 Score 99.754 Data time: 0.2323, Forward time: 1.2523, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.5187 [2021-07-09 05:26:04,825][root][INFO] - ===> Epoch[10482](52410/5): Loss 0.0073, Sem 0.0073 LR: 1.556e-02 Score 99.738 Data time: 0.1718, Forward time: 1.2523, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7661 [2021-07-09 05:26:18,548][root][INFO] - ===> Epoch[10483](52415/5): Loss 0.0070, Sem 0.0070 LR: 1.555e-02 Score 99.749 Data time: 0.2178, Forward time: 1.2523, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.6860 [2021-07-09 05:26:32,612][root][INFO] - ===> Epoch[10484](52420/5): Loss 0.0070, Sem 0.0070 LR: 1.554e-02 Score 99.748 Data time: 0.2093, Forward time: 1.2523, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7702 [2021-07-09 05:26:46,873][root][INFO] - ===> Epoch[10485](52425/5): Loss 0.0069, Sem 0.0069 LR: 1.553e-02 Score 99.753 Data time: 0.2000, Forward time: 1.2523, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.8170 [2021-07-09 05:27:00,617][root][INFO] - ===> Epoch[10486](52430/5): Loss 0.0070, Sem 0.0070 LR: 1.552e-02 Score 99.745 Data time: 0.1594, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7134 [2021-07-09 05:27:14,145][root][INFO] - ===> Epoch[10487](52435/5): Loss 0.0072, Sem 0.0072 LR: 1.551e-02 Score 99.740 Data time: 0.1674, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6578 [2021-07-09 05:27:27,831][root][INFO] - ===> Epoch[10488](52440/5): Loss 0.0070, Sem 0.0070 LR: 1.550e-02 Score 99.747 Data time: 0.1784, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6896 [2021-07-09 05:27:42,252][root][INFO] - ===> Epoch[10489](52445/5): Loss 0.0068, Sem 0.0068 LR: 1.549e-02 Score 99.757 Data time: 0.2036, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8351 [2021-07-09 05:27:56,568][root][INFO] - ===> Epoch[10490](52450/5): Loss 0.0069, Sem 0.0069 LR: 1.548e-02 Score 99.752 Data time: 0.2717, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8053 [2021-07-09 05:28:10,732][root][INFO] - ===> Epoch[10491](52455/5): Loss 0.0070, Sem 0.0070 LR: 1.547e-02 Score 99.748 Data time: 0.3196, Forward time: 1.2523, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7786 [2021-07-09 05:28:25,663][root][INFO] - ===> Epoch[10492](52460/5): Loss 0.0070, Sem 0.0070 LR: 1.546e-02 Score 99.753 Data time: 0.1935, Forward time: 1.2523, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.9447 [2021-07-09 05:28:39,962][root][INFO] - ===> Epoch[10493](52465/5): Loss 0.0072, Sem 0.0072 LR: 1.546e-02 Score 99.747 Data time: 0.2330, Forward time: 1.2523, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.8214 [2021-07-09 05:28:54,604][root][INFO] - ===> Epoch[10494](52470/5): Loss 0.0071, Sem 0.0071 LR: 1.545e-02 Score 99.745 Data time: 0.2023, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8904 [2021-07-09 05:29:09,095][root][INFO] - ===> Epoch[10495](52475/5): Loss 0.0068, Sem 0.0068 LR: 1.544e-02 Score 99.756 Data time: 0.2288, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8480 [2021-07-09 05:29:23,302][root][INFO] - ===> Epoch[10496](52480/5): Loss 0.0070, Sem 0.0070 LR: 1.543e-02 Score 99.750 Data time: 0.2322, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8039 [2021-07-09 05:29:37,789][root][INFO] - ===> Epoch[10497](52485/5): Loss 0.0070, Sem 0.0070 LR: 1.542e-02 Score 99.751 Data time: 0.1919, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8585 [2021-07-09 05:29:51,792][root][INFO] - ===> Epoch[10498](52490/5): Loss 0.0072, Sem 0.0072 LR: 1.541e-02 Score 99.744 Data time: 0.2499, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7515 [2021-07-09 05:30:05,042][root][INFO] - ===> Epoch[10499](52495/5): Loss 0.0072, Sem 0.0072 LR: 1.540e-02 Score 99.745 Data time: 0.2109, Forward time: 1.2523, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.6058 [2021-07-09 05:30:18,911][root][INFO] - ===> Epoch[10500](52500/5): Loss 0.0069, Sem 0.0069 LR: 1.539e-02 Score 99.755 Data time: 0.1920, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7277 [2021-07-09 05:30:32,532][root][INFO] - ===> Epoch[10501](52505/5): Loss 0.0068, Sem 0.0068 LR: 1.538e-02 Score 99.756 Data time: 0.2564, Forward time: 1.2523, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.6785 [2021-07-09 05:30:47,105][root][INFO] - ===> Epoch[10502](52510/5): Loss 0.0069, Sem 0.0069 LR: 1.537e-02 Score 99.752 Data time: 0.2575, Forward time: 1.2523, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.8688 [2021-07-09 05:31:01,242][root][INFO] - ===> Epoch[10503](52515/5): Loss 0.0071, Sem 0.0071 LR: 1.536e-02 Score 99.747 Data time: 0.3054, Forward time: 1.2523, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7671 [2021-07-09 05:31:16,090][root][INFO] - ===> Epoch[10504](52520/5): Loss 0.0068, Sem 0.0068 LR: 1.535e-02 Score 99.755 Data time: 0.2214, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.9258 [2021-07-09 05:31:30,300][root][INFO] - ===> Epoch[10505](52525/5): Loss 0.0068, Sem 0.0068 LR: 1.534e-02 Score 99.756 Data time: 0.2253, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7933 [2021-07-09 05:31:43,567][root][INFO] - ===> Epoch[10506](52530/5): Loss 0.0066, Sem 0.0066 LR: 1.534e-02 Score 99.763 Data time: 0.1981, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.6058 [2021-07-09 05:31:58,328][root][INFO] - ===> Epoch[10507](52535/5): Loss 0.0071, Sem 0.0071 LR: 1.533e-02 Score 99.747 Data time: 0.2177, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.8927 [2021-07-09 05:32:12,662][root][INFO] - ===> Epoch[10508](52540/5): Loss 0.0069, Sem 0.0069 LR: 1.532e-02 Score 99.752 Data time: 0.2363, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.8230 [2021-07-09 05:32:26,977][root][INFO] - ===> Epoch[10509](52545/5): Loss 0.0068, Sem 0.0068 LR: 1.531e-02 Score 99.758 Data time: 0.2219, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.8190 [2021-07-09 05:32:41,748][root][INFO] - ===> Epoch[10510](52550/5): Loss 0.0069, Sem 0.0069 LR: 1.530e-02 Score 99.752 Data time: 0.2081, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.9081 [2021-07-09 05:32:55,772][root][INFO] - ===> Epoch[10511](52555/5): Loss 0.0072, Sem 0.0072 LR: 1.529e-02 Score 99.741 Data time: 0.1582, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7699 [2021-07-09 05:33:09,356][root][INFO] - ===> Epoch[10512](52560/5): Loss 0.0069, Sem 0.0069 LR: 1.528e-02 Score 99.755 Data time: 0.1983, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6712 [2021-07-09 05:33:23,242][root][INFO] - ===> Epoch[10513](52565/5): Loss 0.0071, Sem 0.0071 LR: 1.527e-02 Score 99.743 Data time: 0.1900, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7362 [2021-07-09 05:33:36,434][root][INFO] - ===> Epoch[10514](52570/5): Loss 0.0069, Sem 0.0069 LR: 1.526e-02 Score 99.754 Data time: 0.1685, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.5968 [2021-07-09 05:33:50,924][root][INFO] - ===> Epoch[10515](52575/5): Loss 0.0073, Sem 0.0073 LR: 1.525e-02 Score 99.739 Data time: 0.2184, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8587 [2021-07-09 05:34:05,406][root][INFO] - ===> Epoch[10516](52580/5): Loss 0.0067, Sem 0.0067 LR: 1.524e-02 Score 99.759 Data time: 0.2073, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8506 [2021-07-09 05:34:20,268][root][INFO] - ===> Epoch[10517](52585/5): Loss 0.0068, Sem 0.0068 LR: 1.523e-02 Score 99.757 Data time: 0.2315, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.9230 [2021-07-09 05:34:34,305][root][INFO] - ===> Epoch[10518](52590/5): Loss 0.0069, Sem 0.0069 LR: 1.522e-02 Score 99.756 Data time: 0.2174, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7512 [2021-07-09 05:34:48,967][root][INFO] - ===> Epoch[10519](52595/5): Loss 0.0070, Sem 0.0070 LR: 1.522e-02 Score 99.751 Data time: 0.2484, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8869 [2021-07-09 05:35:02,724][root][INFO] - ===> Epoch[10520](52600/5): Loss 0.0076, Sem 0.0076 LR: 1.521e-02 Score 99.728 Data time: 0.2068, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6759 [2021-07-09 05:35:02,727][root][INFO] - ===> Start testing [2021-07-09 05:35:36,842][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1287 Loss 0.370 (AVG: 0.608) Score 92.906 (AVG: 91.078) mIOU 71.714 mAP 82.623 mAcc 77.851 IOU: 64.588 0.000 80.662 75.364 94.932 91.790 33.001 82.920 98.515 85.288 79.956 86.424 58.841 mAP: 83.365 2.465 87.060 85.811 98.315 98.036 54.123 97.132 99.657 94.668 90.820 92.441 90.201 mAcc: 80.762 0.000 83.231 82.727 96.509 97.546 36.065 93.912 99.259 92.995 89.982 96.536 62.537 [2021-07-09 05:35:36,842][root][INFO] - Finished test. Elapsed time: 34.1147 [2021-07-09 05:35:38,027][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 05:35:51,480][root][INFO] - ===> Epoch[10521](52605/5): Loss 0.0067, Sem 0.0067 LR: 1.520e-02 Score 99.760 Data time: 0.2027, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6517 [2021-07-09 05:36:05,472][root][INFO] - ===> Epoch[10522](52610/5): Loss 0.0072, Sem 0.0072 LR: 1.519e-02 Score 99.745 Data time: 0.1670, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7661 [2021-07-09 05:36:19,243][root][INFO] - ===> Epoch[10523](52615/5): Loss 0.0070, Sem 0.0070 LR: 1.518e-02 Score 99.746 Data time: 0.2287, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6966 [2021-07-09 05:36:32,495][root][INFO] - ===> Epoch[10524](52620/5): Loss 0.0071, Sem 0.0071 LR: 1.517e-02 Score 99.745 Data time: 0.2291, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.5922 [2021-07-09 05:36:47,512][root][INFO] - ===> Epoch[10525](52625/5): Loss 0.0067, Sem 0.0067 LR: 1.516e-02 Score 99.759 Data time: 0.2282, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.9516 [2021-07-09 05:37:01,019][root][INFO] - ===> Epoch[10526](52630/5): Loss 0.0070, Sem 0.0070 LR: 1.515e-02 Score 99.748 Data time: 0.1805, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6620 [2021-07-09 05:37:15,752][root][INFO] - ===> Epoch[10527](52635/5): Loss 0.0069, Sem 0.0069 LR: 1.514e-02 Score 99.754 Data time: 0.2258, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.9026 [2021-07-09 05:37:29,450][root][INFO] - ===> Epoch[10528](52640/5): Loss 0.0070, Sem 0.0070 LR: 1.513e-02 Score 99.750 Data time: 0.1956, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6955 [2021-07-09 05:37:44,356][root][INFO] - ===> Epoch[10529](52645/5): Loss 0.0068, Sem 0.0068 LR: 1.512e-02 Score 99.759 Data time: 0.2817, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.9269 [2021-07-09 05:37:58,223][root][INFO] - ===> Epoch[10530](52650/5): Loss 0.0071, Sem 0.0071 LR: 1.511e-02 Score 99.743 Data time: 0.2077, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7331 [2021-07-09 05:38:12,965][root][INFO] - ===> Epoch[10531](52655/5): Loss 0.0069, Sem 0.0069 LR: 1.510e-02 Score 99.753 Data time: 0.1767, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.9070 [2021-07-09 05:38:26,937][root][INFO] - ===> Epoch[10532](52660/5): Loss 0.0067, Sem 0.0067 LR: 1.510e-02 Score 99.757 Data time: 0.2127, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7514 [2021-07-09 05:38:40,854][root][INFO] - ===> Epoch[10533](52665/5): Loss 0.0070, Sem 0.0070 LR: 1.509e-02 Score 99.751 Data time: 0.2436, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7413 [2021-07-09 05:38:54,983][root][INFO] - ===> Epoch[10534](52670/5): Loss 0.0069, Sem 0.0069 LR: 1.508e-02 Score 99.754 Data time: 0.2286, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7837 [2021-07-09 05:39:08,985][root][INFO] - ===> Epoch[10535](52675/5): Loss 0.0070, Sem 0.0070 LR: 1.507e-02 Score 99.750 Data time: 0.2696, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7514 [2021-07-09 05:39:23,006][root][INFO] - ===> Epoch[10536](52680/5): Loss 0.0072, Sem 0.0072 LR: 1.506e-02 Score 99.739 Data time: 0.2165, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7610 [2021-07-09 05:39:36,685][root][INFO] - ===> Epoch[10537](52685/5): Loss 0.0071, Sem 0.0071 LR: 1.505e-02 Score 99.747 Data time: 0.2201, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6845 [2021-07-09 05:39:50,170][root][INFO] - ===> Epoch[10538](52690/5): Loss 0.0073, Sem 0.0073 LR: 1.504e-02 Score 99.739 Data time: 0.1705, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6533 [2021-07-09 05:40:04,628][root][INFO] - ===> Epoch[10539](52695/5): Loss 0.0068, Sem 0.0068 LR: 1.503e-02 Score 99.759 Data time: 0.1909, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8426 [2021-07-09 05:40:18,572][root][INFO] - ===> Epoch[10540](52700/5): Loss 0.0072, Sem 0.0072 LR: 1.502e-02 Score 99.743 Data time: 0.2232, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7302 [2021-07-09 05:40:33,154][root][INFO] - ===> Epoch[10541](52705/5): Loss 0.0070, Sem 0.0070 LR: 1.501e-02 Score 99.751 Data time: 0.2329, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8679 [2021-07-09 05:40:46,565][root][INFO] - ===> Epoch[10542](52710/5): Loss 0.0067, Sem 0.0067 LR: 1.500e-02 Score 99.760 Data time: 0.2135, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6379 [2021-07-09 05:41:00,734][root][INFO] - ===> Epoch[10543](52715/5): Loss 0.0068, Sem 0.0068 LR: 1.499e-02 Score 99.755 Data time: 0.2172, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7937 [2021-07-09 05:41:15,100][root][INFO] - ===> Epoch[10544](52720/5): Loss 0.0067, Sem 0.0067 LR: 1.498e-02 Score 99.761 Data time: 0.2212, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8218 [2021-07-09 05:41:29,179][root][INFO] - ===> Epoch[10545](52725/5): Loss 0.0068, Sem 0.0068 LR: 1.497e-02 Score 99.759 Data time: 0.2974, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7626 [2021-07-09 05:41:42,906][root][INFO] - ===> Epoch[10546](52730/5): Loss 0.0071, Sem 0.0071 LR: 1.497e-02 Score 99.746 Data time: 0.2594, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7000 [2021-07-09 05:41:57,467][root][INFO] - ===> Epoch[10547](52735/5): Loss 0.0067, Sem 0.0067 LR: 1.496e-02 Score 99.756 Data time: 0.2326, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8703 [2021-07-09 05:42:11,532][root][INFO] - ===> Epoch[10548](52740/5): Loss 0.0070, Sem 0.0070 LR: 1.495e-02 Score 99.750 Data time: 0.2479, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7617 [2021-07-09 05:42:25,483][root][INFO] - ===> Epoch[10549](52745/5): Loss 0.0069, Sem 0.0069 LR: 1.494e-02 Score 99.753 Data time: 0.2670, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7360 [2021-07-09 05:42:39,805][root][INFO] - ===> Epoch[10550](52750/5): Loss 0.0072, Sem 0.0072 LR: 1.493e-02 Score 99.746 Data time: 0.2919, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8160 [2021-07-09 05:42:54,009][root][INFO] - ===> Epoch[10551](52755/5): Loss 0.0069, Sem 0.0069 LR: 1.492e-02 Score 99.754 Data time: 0.2993, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7846 [2021-07-09 05:43:08,571][root][INFO] - ===> Epoch[10552](52760/5): Loss 0.0068, Sem 0.0068 LR: 1.491e-02 Score 99.758 Data time: 0.2292, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8653 [2021-07-09 05:43:23,127][root][INFO] - ===> Epoch[10553](52765/5): Loss 0.0069, Sem 0.0069 LR: 1.490e-02 Score 99.752 Data time: 0.2653, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8686 [2021-07-09 05:43:37,100][root][INFO] - ===> Epoch[10554](52770/5): Loss 0.0072, Sem 0.0072 LR: 1.489e-02 Score 99.742 Data time: 0.2376, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7463 [2021-07-09 05:43:50,399][root][INFO] - ===> Epoch[10555](52775/5): Loss 0.0071, Sem 0.0071 LR: 1.488e-02 Score 99.746 Data time: 0.1731, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6117 [2021-07-09 05:44:06,032][root][INFO] - ===> Epoch[10556](52780/5): Loss 0.0068, Sem 0.0068 LR: 1.487e-02 Score 99.756 Data time: 0.2419, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 3.0791 [2021-07-09 05:44:20,514][root][INFO] - ===> Epoch[10557](52785/5): Loss 0.0072, Sem 0.0072 LR: 1.486e-02 Score 99.744 Data time: 0.1827, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8557 [2021-07-09 05:44:34,676][root][INFO] - ===> Epoch[10558](52790/5): Loss 0.0069, Sem 0.0069 LR: 1.485e-02 Score 99.754 Data time: 0.1996, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7861 [2021-07-09 05:44:48,256][root][INFO] - ===> Epoch[10559](52795/5): Loss 0.0071, Sem 0.0071 LR: 1.485e-02 Score 99.745 Data time: 0.2285, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6706 [2021-07-09 05:45:03,177][root][INFO] - ===> Epoch[10560](52800/5): Loss 0.0070, Sem 0.0070 LR: 1.484e-02 Score 99.749 Data time: 0.1989, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.9427 [2021-07-09 05:45:03,178][root][INFO] - ===> Start testing [2021-07-09 05:45:43,249][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.2340 Loss 0.419 (AVG: 0.636) Score 91.734 (AVG: 91.058) mIOU 71.416 mAP 82.218 mAcc 77.574 IOU: 64.426 0.000 80.492 75.116 95.147 91.550 29.382 85.442 98.561 84.265 79.392 86.246 58.389 mAP: 83.404 2.100 86.772 85.878 98.378 98.097 50.965 97.878 99.657 94.155 90.634 91.760 89.159 mAcc: 79.238 0.000 83.223 83.170 96.773 97.682 31.462 95.831 99.370 93.518 89.524 96.774 61.898 [2021-07-09 05:45:43,249][root][INFO] - Finished test. Elapsed time: 40.0704 [2021-07-09 05:45:44,336][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 05:45:56,898][root][INFO] - ===> Epoch[10561](52805/5): Loss 0.0070, Sem 0.0070 LR: 1.483e-02 Score 99.749 Data time: 0.2291, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.4688 [2021-07-09 05:46:10,658][root][INFO] - ===> Epoch[10562](52810/5): Loss 0.0070, Sem 0.0070 LR: 1.482e-02 Score 99.751 Data time: 0.2661, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7077 [2021-07-09 05:46:25,038][root][INFO] - ===> Epoch[10563](52815/5): Loss 0.0069, Sem 0.0069 LR: 1.481e-02 Score 99.753 Data time: 0.1986, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8375 [2021-07-09 05:46:38,906][root][INFO] - ===> Epoch[10564](52820/5): Loss 0.0070, Sem 0.0070 LR: 1.480e-02 Score 99.751 Data time: 0.1897, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7266 [2021-07-09 05:46:52,600][root][INFO] - ===> Epoch[10565](52825/5): Loss 0.0071, Sem 0.0071 LR: 1.479e-02 Score 99.748 Data time: 0.2704, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6968 [2021-07-09 05:47:06,171][root][INFO] - ===> Epoch[10566](52830/5): Loss 0.0067, Sem 0.0067 LR: 1.478e-02 Score 99.759 Data time: 0.2409, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6629 [2021-07-09 05:47:19,799][root][INFO] - ===> Epoch[10567](52835/5): Loss 0.0073, Sem 0.0073 LR: 1.477e-02 Score 99.739 Data time: 0.1332, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6881 [2021-07-09 05:47:33,605][root][INFO] - ===> Epoch[10568](52840/5): Loss 0.0066, Sem 0.0066 LR: 1.476e-02 Score 99.761 Data time: 0.1759, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7181 [2021-07-09 05:47:47,768][root][INFO] - ===> Epoch[10569](52845/5): Loss 0.0065, Sem 0.0065 LR: 1.475e-02 Score 99.769 Data time: 0.2109, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7763 [2021-07-09 05:48:01,147][root][INFO] - ===> Epoch[10570](52850/5): Loss 0.0068, Sem 0.0068 LR: 1.474e-02 Score 99.758 Data time: 0.2249, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6356 [2021-07-09 05:48:14,660][root][INFO] - ===> Epoch[10571](52855/5): Loss 0.0072, Sem 0.0072 LR: 1.473e-02 Score 99.745 Data time: 0.2627, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6588 [2021-07-09 05:48:28,538][root][INFO] - ===> Epoch[10572](52860/5): Loss 0.0071, Sem 0.0071 LR: 1.472e-02 Score 99.749 Data time: 0.2220, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7242 [2021-07-09 05:48:42,428][root][INFO] - ===> Epoch[10573](52865/5): Loss 0.0071, Sem 0.0071 LR: 1.472e-02 Score 99.748 Data time: 0.1843, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.7383 [2021-07-09 05:48:55,965][root][INFO] - ===> Epoch[10574](52870/5): Loss 0.0067, Sem 0.0067 LR: 1.471e-02 Score 99.761 Data time: 0.1708, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6689 [2021-07-09 05:49:10,166][root][INFO] - ===> Epoch[10575](52875/5): Loss 0.0070, Sem 0.0070 LR: 1.470e-02 Score 99.751 Data time: 0.2426, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7965 [2021-07-09 05:49:23,849][root][INFO] - ===> Epoch[10576](52880/5): Loss 0.0070, Sem 0.0070 LR: 1.469e-02 Score 99.750 Data time: 0.1723, Forward time: 1.2524, Backward time: 1.2477, DDP time: 0.0042, Total iter time: 2.6925 [2021-07-09 05:49:37,956][root][INFO] - ===> Epoch[10577](52885/5): Loss 0.0070, Sem 0.0070 LR: 1.468e-02 Score 99.752 Data time: 0.1908, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7762 [2021-07-09 05:49:52,243][root][INFO] - ===> Epoch[10578](52890/5): Loss 0.0069, Sem 0.0069 LR: 1.467e-02 Score 99.754 Data time: 0.2316, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8076 [2021-07-09 05:50:05,959][root][INFO] - ===> Epoch[10579](52895/5): Loss 0.0068, Sem 0.0068 LR: 1.466e-02 Score 99.761 Data time: 0.2680, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6918 [2021-07-09 05:50:20,946][root][INFO] - ===> Epoch[10580](52900/5): Loss 0.0070, Sem 0.0070 LR: 1.465e-02 Score 99.751 Data time: 0.1984, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.9540 [2021-07-09 05:50:34,743][root][INFO] - ===> Epoch[10581](52905/5): Loss 0.0071, Sem 0.0071 LR: 1.464e-02 Score 99.748 Data time: 0.2259, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7069 [2021-07-09 05:50:48,303][root][INFO] - ===> Epoch[10582](52910/5): Loss 0.0074, Sem 0.0074 LR: 1.463e-02 Score 99.737 Data time: 0.2729, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6635 [2021-07-09 05:51:01,399][root][INFO] - ===> Epoch[10583](52915/5): Loss 0.0072, Sem 0.0072 LR: 1.462e-02 Score 99.743 Data time: 0.1493, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.5833 [2021-07-09 05:51:15,011][root][INFO] - ===> Epoch[10584](52920/5): Loss 0.0071, Sem 0.0071 LR: 1.461e-02 Score 99.743 Data time: 0.1866, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6711 [2021-07-09 05:51:29,317][root][INFO] - ===> Epoch[10585](52925/5): Loss 0.0069, Sem 0.0069 LR: 1.460e-02 Score 99.755 Data time: 0.2173, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8191 [2021-07-09 05:51:43,012][root][INFO] - ===> Epoch[10586](52930/5): Loss 0.0073, Sem 0.0073 LR: 1.459e-02 Score 99.742 Data time: 0.2130, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7021 [2021-07-09 05:51:56,332][root][INFO] - ===> Epoch[10587](52935/5): Loss 0.0072, Sem 0.0072 LR: 1.459e-02 Score 99.742 Data time: 0.1908, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6313 [2021-07-09 05:52:10,096][root][INFO] - ===> Epoch[10588](52940/5): Loss 0.0070, Sem 0.0070 LR: 1.458e-02 Score 99.749 Data time: 0.1964, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7057 [2021-07-09 05:52:25,004][root][INFO] - ===> Epoch[10589](52945/5): Loss 0.0068, Sem 0.0068 LR: 1.457e-02 Score 99.759 Data time: 0.1936, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.9322 [2021-07-09 05:52:38,151][root][INFO] - ===> Epoch[10590](52950/5): Loss 0.0075, Sem 0.0075 LR: 1.456e-02 Score 99.731 Data time: 0.2344, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.5784 [2021-07-09 05:52:52,045][root][INFO] - ===> Epoch[10591](52955/5): Loss 0.0069, Sem 0.0069 LR: 1.455e-02 Score 99.756 Data time: 0.2580, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7242 [2021-07-09 05:53:05,827][root][INFO] - ===> Epoch[10592](52960/5): Loss 0.0071, Sem 0.0071 LR: 1.454e-02 Score 99.744 Data time: 0.2360, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7045 [2021-07-09 05:53:20,166][root][INFO] - ===> Epoch[10593](52965/5): Loss 0.0070, Sem 0.0070 LR: 1.453e-02 Score 99.753 Data time: 0.2683, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8087 [2021-07-09 05:53:33,426][root][INFO] - ===> Epoch[10594](52970/5): Loss 0.0069, Sem 0.0069 LR: 1.452e-02 Score 99.754 Data time: 0.1869, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6010 [2021-07-09 05:53:47,158][root][INFO] - ===> Epoch[10595](52975/5): Loss 0.0071, Sem 0.0071 LR: 1.451e-02 Score 99.747 Data time: 0.2311, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6865 [2021-07-09 05:54:01,188][root][INFO] - ===> Epoch[10596](52980/5): Loss 0.0071, Sem 0.0071 LR: 1.450e-02 Score 99.747 Data time: 0.2133, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7715 [2021-07-09 05:54:16,034][root][INFO] - ===> Epoch[10597](52985/5): Loss 0.0067, Sem 0.0067 LR: 1.449e-02 Score 99.760 Data time: 0.2072, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.9220 [2021-07-09 05:54:30,951][root][INFO] - ===> Epoch[10598](52990/5): Loss 0.0070, Sem 0.0070 LR: 1.448e-02 Score 99.752 Data time: 0.2191, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.9327 [2021-07-09 05:54:45,214][root][INFO] - ===> Epoch[10599](52995/5): Loss 0.0072, Sem 0.0072 LR: 1.447e-02 Score 99.742 Data time: 0.2130, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8117 [2021-07-09 05:54:59,161][root][INFO] - ===> Epoch[10600](53000/5): Loss 0.0067, Sem 0.0067 LR: 1.446e-02 Score 99.758 Data time: 0.2848, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7437 [2021-07-09 05:54:59,163][root][INFO] - ===> Start testing [2021-07-09 05:55:33,614][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1225 Loss 0.405 (AVG: 0.637) Score 92.399 (AVG: 91.006) mIOU 71.258 mAP 82.227 mAcc 77.185 IOU: 64.759 0.000 80.338 75.556 95.051 91.380 25.481 84.640 98.542 86.475 79.962 85.875 58.301 mAP: 82.515 2.765 88.435 85.926 98.423 98.071 48.335 97.747 99.651 94.734 90.794 92.091 89.465 mAcc: 79.816 0.000 83.739 83.689 96.582 97.337 27.018 94.737 99.340 92.788 90.154 96.690 61.517 [2021-07-09 05:55:33,614][root][INFO] - Finished test. Elapsed time: 34.4506 [2021-07-09 05:55:34,767][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 05:55:47,897][root][INFO] - ===> Epoch[10601](53005/5): Loss 0.0072, Sem 0.0072 LR: 1.446e-02 Score 99.741 Data time: 0.2652, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.5991 [2021-07-09 05:56:02,358][root][INFO] - ===> Epoch[10602](53010/5): Loss 0.0071, Sem 0.0071 LR: 1.445e-02 Score 99.747 Data time: 0.2604, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8422 [2021-07-09 05:56:17,299][root][INFO] - ===> Epoch[10603](53015/5): Loss 0.0072, Sem 0.0072 LR: 1.444e-02 Score 99.744 Data time: 0.2867, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.9363 [2021-07-09 05:56:31,756][root][INFO] - ===> Epoch[10604](53020/5): Loss 0.0067, Sem 0.0067 LR: 1.443e-02 Score 99.758 Data time: 0.3552, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8384 [2021-07-09 05:56:46,462][root][INFO] - ===> Epoch[10605](53025/5): Loss 0.0071, Sem 0.0071 LR: 1.442e-02 Score 99.746 Data time: 0.2784, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8966 [2021-07-09 05:57:00,978][root][INFO] - ===> Epoch[10606](53030/5): Loss 0.0073, Sem 0.0073 LR: 1.441e-02 Score 99.741 Data time: 0.2650, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8582 [2021-07-09 05:57:15,491][root][INFO] - ===> Epoch[10607](53035/5): Loss 0.0071, Sem 0.0071 LR: 1.440e-02 Score 99.747 Data time: 0.2578, Forward time: 1.2524, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8595 [2021-07-09 05:57:28,952][root][INFO] - ===> Epoch[10608](53040/5): Loss 0.0072, Sem 0.0072 LR: 1.439e-02 Score 99.744 Data time: 0.1981, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6443 [2021-07-09 05:57:42,580][root][INFO] - ===> Epoch[10609](53045/5): Loss 0.0070, Sem 0.0070 LR: 1.438e-02 Score 99.748 Data time: 0.2249, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.6823 [2021-07-09 05:57:57,195][root][INFO] - ===> Epoch[10610](53050/5): Loss 0.0069, Sem 0.0069 LR: 1.437e-02 Score 99.755 Data time: 0.2550, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8785 [2021-07-09 05:58:10,977][root][INFO] - ===> Epoch[10611](53055/5): Loss 0.0071, Sem 0.0071 LR: 1.436e-02 Score 99.746 Data time: 0.2819, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7031 [2021-07-09 05:58:25,069][root][INFO] - ===> Epoch[10612](53060/5): Loss 0.0071, Sem 0.0071 LR: 1.435e-02 Score 99.748 Data time: 0.1964, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7778 [2021-07-09 05:58:39,468][root][INFO] - ===> Epoch[10613](53065/5): Loss 0.0072, Sem 0.0072 LR: 1.434e-02 Score 99.736 Data time: 0.1970, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8312 [2021-07-09 05:58:53,693][root][INFO] - ===> Epoch[10614](53070/5): Loss 0.0070, Sem 0.0070 LR: 1.433e-02 Score 99.750 Data time: 0.2266, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7898 [2021-07-09 05:59:07,578][root][INFO] - ===> Epoch[10615](53075/5): Loss 0.0070, Sem 0.0070 LR: 1.432e-02 Score 99.749 Data time: 0.1734, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7357 [2021-07-09 05:59:21,904][root][INFO] - ===> Epoch[10616](53080/5): Loss 0.0068, Sem 0.0068 LR: 1.432e-02 Score 99.756 Data time: 0.1835, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.8326 [2021-07-09 05:59:35,727][root][INFO] - ===> Epoch[10617](53085/5): Loss 0.0069, Sem 0.0069 LR: 1.431e-02 Score 99.754 Data time: 0.2313, Forward time: 1.2523, Backward time: 1.2478, DDP time: 0.0042, Total iter time: 2.7277 [2021-07-09 05:59:50,798][root][INFO] - ===> Epoch[10618](53090/5): Loss 0.0069, Sem 0.0069 LR: 1.430e-02 Score 99.752 Data time: 0.1315, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.9702 [2021-07-09 06:00:04,647][root][INFO] - ===> Epoch[10619](53095/5): Loss 0.0066, Sem 0.0066 LR: 1.429e-02 Score 99.764 Data time: 0.2523, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7229 [2021-07-09 06:00:18,601][root][INFO] - ===> Epoch[10620](53100/5): Loss 0.0066, Sem 0.0066 LR: 1.428e-02 Score 99.767 Data time: 0.2343, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7408 [2021-07-09 06:00:32,792][root][INFO] - ===> Epoch[10621](53105/5): Loss 0.0069, Sem 0.0069 LR: 1.427e-02 Score 99.754 Data time: 0.1386, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7861 [2021-07-09 06:00:46,563][root][INFO] - ===> Epoch[10622](53110/5): Loss 0.0071, Sem 0.0071 LR: 1.426e-02 Score 99.744 Data time: 0.2090, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7094 [2021-07-09 06:01:00,901][root][INFO] - ===> Epoch[10623](53115/5): Loss 0.0070, Sem 0.0070 LR: 1.425e-02 Score 99.749 Data time: 0.2887, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8192 [2021-07-09 06:01:15,017][root][INFO] - ===> Epoch[10624](53120/5): Loss 0.0072, Sem 0.0072 LR: 1.424e-02 Score 99.744 Data time: 0.2245, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7770 [2021-07-09 06:01:29,902][root][INFO] - ===> Epoch[10625](53125/5): Loss 0.0068, Sem 0.0068 LR: 1.423e-02 Score 99.757 Data time: 0.2116, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.9348 [2021-07-09 06:01:44,088][root][INFO] - ===> Epoch[10626](53130/5): Loss 0.0069, Sem 0.0069 LR: 1.422e-02 Score 99.754 Data time: 0.1996, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7959 [2021-07-09 06:01:58,000][root][INFO] - ===> Epoch[10627](53135/5): Loss 0.0071, Sem 0.0071 LR: 1.421e-02 Score 99.747 Data time: 0.1543, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7451 [2021-07-09 06:02:11,868][root][INFO] - ===> Epoch[10628](53140/5): Loss 0.0067, Sem 0.0067 LR: 1.420e-02 Score 99.760 Data time: 0.1874, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7292 [2021-07-09 06:02:25,709][root][INFO] - ===> Epoch[10629](53145/5): Loss 0.0070, Sem 0.0070 LR: 1.419e-02 Score 99.746 Data time: 0.2530, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7219 [2021-07-09 06:02:39,447][root][INFO] - ===> Epoch[10630](53150/5): Loss 0.0072, Sem 0.0072 LR: 1.419e-02 Score 99.742 Data time: 0.1999, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7030 [2021-07-09 06:02:53,975][root][INFO] - ===> Epoch[10631](53155/5): Loss 0.0065, Sem 0.0065 LR: 1.418e-02 Score 99.767 Data time: 0.2422, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8576 [2021-07-09 06:03:07,262][root][INFO] - ===> Epoch[10632](53160/5): Loss 0.0076, Sem 0.0076 LR: 1.417e-02 Score 99.731 Data time: 0.2015, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6180 [2021-07-09 06:03:21,939][root][INFO] - ===> Epoch[10633](53165/5): Loss 0.0065, Sem 0.0065 LR: 1.416e-02 Score 99.768 Data time: 0.2018, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8862 [2021-07-09 06:03:35,411][root][INFO] - ===> Epoch[10634](53170/5): Loss 0.0071, Sem 0.0071 LR: 1.415e-02 Score 99.746 Data time: 0.2037, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6563 [2021-07-09 06:03:48,839][root][INFO] - ===> Epoch[10635](53175/5): Loss 0.0072, Sem 0.0072 LR: 1.414e-02 Score 99.743 Data time: 0.2070, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6324 [2021-07-09 06:04:02,256][root][INFO] - ===> Epoch[10636](53180/5): Loss 0.0073, Sem 0.0073 LR: 1.413e-02 Score 99.738 Data time: 0.1967, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6308 [2021-07-09 06:04:16,396][root][INFO] - ===> Epoch[10637](53185/5): Loss 0.0067, Sem 0.0067 LR: 1.412e-02 Score 99.762 Data time: 0.2299, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7801 [2021-07-09 06:04:30,896][root][INFO] - ===> Epoch[10638](53190/5): Loss 0.0069, Sem 0.0069 LR: 1.411e-02 Score 99.754 Data time: 0.2076, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8548 [2021-07-09 06:04:44,670][root][INFO] - ===> Epoch[10639](53195/5): Loss 0.0071, Sem 0.0071 LR: 1.410e-02 Score 99.746 Data time: 0.2003, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7101 [2021-07-09 06:04:58,272][root][INFO] - ===> Epoch[10640](53200/5): Loss 0.0071, Sem 0.0071 LR: 1.409e-02 Score 99.744 Data time: 0.2152, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.6791 [2021-07-09 06:04:58,273][root][INFO] - ===> Start testing [2021-07-09 06:05:37,059][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1236 Loss 0.424 (AVG: 0.627) Score 91.670 (AVG: 91.055) mIOU 71.647 mAP 82.410 mAcc 77.687 IOU: 64.447 0.000 79.816 75.260 94.954 91.612 30.828 85.286 98.578 86.513 79.851 86.108 58.161 mAP: 82.885 2.142 87.806 85.595 98.286 98.157 51.712 97.944 99.635 94.761 90.477 92.175 89.754 mAcc: 78.926 0.000 83.552 83.381 96.771 97.500 33.087 95.009 99.380 93.597 90.401 96.685 61.646 [2021-07-09 06:05:37,059][root][INFO] - Finished test. Elapsed time: 38.7853 [2021-07-09 06:05:38,268][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 06:05:51,004][root][INFO] - ===> Epoch[10641](53205/5): Loss 0.0068, Sem 0.0068 LR: 1.408e-02 Score 99.754 Data time: 0.2057, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.5078 [2021-07-09 06:06:04,376][root][INFO] - ===> Epoch[10642](53210/5): Loss 0.0068, Sem 0.0068 LR: 1.407e-02 Score 99.756 Data time: 0.2686, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6104 [2021-07-09 06:06:19,002][root][INFO] - ===> Epoch[10643](53215/5): Loss 0.0068, Sem 0.0068 LR: 1.406e-02 Score 99.756 Data time: 0.1842, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8727 [2021-07-09 06:06:32,552][root][INFO] - ===> Epoch[10644](53220/5): Loss 0.0074, Sem 0.0074 LR: 1.405e-02 Score 99.738 Data time: 0.1836, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6591 [2021-07-09 06:06:46,302][root][INFO] - ===> Epoch[10645](53225/5): Loss 0.0067, Sem 0.0067 LR: 1.405e-02 Score 99.761 Data time: 0.2284, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6988 [2021-07-09 06:07:00,642][root][INFO] - ===> Epoch[10646](53230/5): Loss 0.0067, Sem 0.0067 LR: 1.404e-02 Score 99.763 Data time: 0.2354, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8259 [2021-07-09 06:07:13,912][root][INFO] - ===> Epoch[10647](53235/5): Loss 0.0071, Sem 0.0071 LR: 1.403e-02 Score 99.749 Data time: 0.2192, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6036 [2021-07-09 06:07:26,653][root][INFO] - ===> Epoch[10648](53240/5): Loss 0.0071, Sem 0.0071 LR: 1.402e-02 Score 99.744 Data time: 0.2309, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.5044 [2021-07-09 06:07:41,020][root][INFO] - ===> Epoch[10649](53245/5): Loss 0.0067, Sem 0.0067 LR: 1.401e-02 Score 99.757 Data time: 0.1942, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8310 [2021-07-09 06:07:54,913][root][INFO] - ===> Epoch[10650](53250/5): Loss 0.0069, Sem 0.0069 LR: 1.400e-02 Score 99.751 Data time: 0.1458, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7477 [2021-07-09 06:08:08,651][root][INFO] - ===> Epoch[10651](53255/5): Loss 0.0072, Sem 0.0072 LR: 1.399e-02 Score 99.744 Data time: 0.1832, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6976 [2021-07-09 06:08:22,956][root][INFO] - ===> Epoch[10652](53260/5): Loss 0.0071, Sem 0.0071 LR: 1.398e-02 Score 99.743 Data time: 0.1279, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8298 [2021-07-09 06:08:37,245][root][INFO] - ===> Epoch[10653](53265/5): Loss 0.0068, Sem 0.0068 LR: 1.397e-02 Score 99.756 Data time: 0.3054, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8134 [2021-07-09 06:08:51,111][root][INFO] - ===> Epoch[10654](53270/5): Loss 0.0072, Sem 0.0072 LR: 1.396e-02 Score 99.742 Data time: 0.2829, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7291 [2021-07-09 06:09:05,374][root][INFO] - ===> Epoch[10655](53275/5): Loss 0.0072, Sem 0.0072 LR: 1.395e-02 Score 99.744 Data time: 0.2577, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8170 [2021-07-09 06:09:19,039][root][INFO] - ===> Epoch[10656](53280/5): Loss 0.0073, Sem 0.0073 LR: 1.394e-02 Score 99.740 Data time: 0.3035, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6924 [2021-07-09 06:09:32,631][root][INFO] - ===> Epoch[10657](53285/5): Loss 0.0067, Sem 0.0067 LR: 1.393e-02 Score 99.761 Data time: 0.2000, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6685 [2021-07-09 06:09:47,118][root][INFO] - ===> Epoch[10658](53290/5): Loss 0.0069, Sem 0.0069 LR: 1.392e-02 Score 99.754 Data time: 0.2400, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8482 [2021-07-09 06:10:02,099][root][INFO] - ===> Epoch[10659](53295/5): Loss 0.0068, Sem 0.0068 LR: 1.391e-02 Score 99.757 Data time: 0.2803, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.9403 [2021-07-09 06:10:16,544][root][INFO] - ===> Epoch[10660](53300/5): Loss 0.0069, Sem 0.0069 LR: 1.391e-02 Score 99.755 Data time: 0.2354, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8461 [2021-07-09 06:10:30,045][root][INFO] - ===> Epoch[10661](53305/5): Loss 0.0072, Sem 0.0072 LR: 1.390e-02 Score 99.739 Data time: 0.2192, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6562 [2021-07-09 06:10:44,062][root][INFO] - ===> Epoch[10662](53310/5): Loss 0.0069, Sem 0.0069 LR: 1.389e-02 Score 99.754 Data time: 0.1689, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7621 [2021-07-09 06:10:58,886][root][INFO] - ===> Epoch[10663](53315/5): Loss 0.0066, Sem 0.0066 LR: 1.388e-02 Score 99.765 Data time: 0.2112, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.9202 [2021-07-09 06:11:11,734][root][INFO] - ===> Epoch[10664](53320/5): Loss 0.0073, Sem 0.0073 LR: 1.387e-02 Score 99.738 Data time: 0.1685, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.5198 [2021-07-09 06:11:27,065][root][INFO] - ===> Epoch[10665](53325/5): Loss 0.0065, Sem 0.0065 LR: 1.386e-02 Score 99.772 Data time: 0.2583, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 3.0175 [2021-07-09 06:11:40,868][root][INFO] - ===> Epoch[10666](53330/5): Loss 0.0070, Sem 0.0070 LR: 1.385e-02 Score 99.747 Data time: 0.1596, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7200 [2021-07-09 06:11:54,536][root][INFO] - ===> Epoch[10667](53335/5): Loss 0.0069, Sem 0.0069 LR: 1.384e-02 Score 99.753 Data time: 0.2410, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6889 [2021-07-09 06:12:08,980][root][INFO] - ===> Epoch[10668](53340/5): Loss 0.0067, Sem 0.0067 LR: 1.383e-02 Score 99.760 Data time: 0.1846, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8440 [2021-07-09 06:12:22,800][root][INFO] - ===> Epoch[10669](53345/5): Loss 0.0070, Sem 0.0070 LR: 1.382e-02 Score 99.752 Data time: 0.2245, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7221 [2021-07-09 06:12:36,344][root][INFO] - ===> Epoch[10670](53350/5): Loss 0.0070, Sem 0.0070 LR: 1.381e-02 Score 99.752 Data time: 0.1550, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6737 [2021-07-09 06:12:51,327][root][INFO] - ===> Epoch[10671](53355/5): Loss 0.0065, Sem 0.0065 LR: 1.380e-02 Score 99.769 Data time: 0.2664, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.9342 [2021-07-09 06:13:05,155][root][INFO] - ===> Epoch[10672](53360/5): Loss 0.0069, Sem 0.0069 LR: 1.379e-02 Score 99.754 Data time: 0.1907, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7173 [2021-07-09 06:13:18,388][root][INFO] - ===> Epoch[10673](53365/5): Loss 0.0068, Sem 0.0068 LR: 1.378e-02 Score 99.756 Data time: 0.1745, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6024 [2021-07-09 06:13:32,093][root][INFO] - ===> Epoch[10674](53370/5): Loss 0.0069, Sem 0.0069 LR: 1.377e-02 Score 99.752 Data time: 0.2120, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6978 [2021-07-09 06:13:46,553][root][INFO] - ===> Epoch[10675](53375/5): Loss 0.0068, Sem 0.0068 LR: 1.377e-02 Score 99.755 Data time: 0.1975, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8533 [2021-07-09 06:14:00,656][root][INFO] - ===> Epoch[10676](53380/5): Loss 0.0071, Sem 0.0071 LR: 1.376e-02 Score 99.744 Data time: 0.2170, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7817 [2021-07-09 06:14:14,604][root][INFO] - ===> Epoch[10677](53385/5): Loss 0.0069, Sem 0.0069 LR: 1.375e-02 Score 99.753 Data time: 0.2307, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7415 [2021-07-09 06:14:29,053][root][INFO] - ===> Epoch[10678](53390/5): Loss 0.0067, Sem 0.0067 LR: 1.374e-02 Score 99.762 Data time: 0.1872, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8368 [2021-07-09 06:14:44,463][root][INFO] - ===> Epoch[10679](53395/5): Loss 0.0069, Sem 0.0069 LR: 1.373e-02 Score 99.755 Data time: 0.2617, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 3.0309 [2021-07-09 06:14:57,978][root][INFO] - ===> Epoch[10680](53400/5): Loss 0.0064, Sem 0.0064 LR: 1.372e-02 Score 99.771 Data time: 0.1847, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6692 [2021-07-09 06:14:57,980][root][INFO] - ===> Start testing [2021-07-09 06:15:37,783][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1225 Loss 0.354 (AVG: 0.632) Score 92.739 (AVG: 91.018) mIOU 71.319 mAP 82.460 mAcc 77.673 IOU: 64.439 0.000 79.786 75.104 95.137 91.490 30.534 84.889 98.591 83.666 79.234 86.117 58.163 mAP: 82.397 2.365 88.471 85.556 98.270 98.099 52.630 97.811 99.664 94.647 90.192 92.178 89.699 mAcc: 79.700 0.000 83.758 82.675 96.793 97.657 32.939 94.525 99.391 93.794 90.225 96.607 61.686 [2021-07-09 06:15:37,784][root][INFO] - Finished test. Elapsed time: 39.8035 [2021-07-09 06:15:38,872][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 06:15:51,585][root][INFO] - ===> Epoch[10681](53405/5): Loss 0.0070, Sem 0.0070 LR: 1.371e-02 Score 99.748 Data time: 0.1598, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.5047 [2021-07-09 06:16:05,929][root][INFO] - ===> Epoch[10682](53410/5): Loss 0.0071, Sem 0.0071 LR: 1.370e-02 Score 99.745 Data time: 0.2527, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8181 [2021-07-09 06:16:19,684][root][INFO] - ===> Epoch[10683](53415/5): Loss 0.0068, Sem 0.0068 LR: 1.369e-02 Score 99.757 Data time: 0.2421, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7069 [2021-07-09 06:16:33,180][root][INFO] - ===> Epoch[10684](53420/5): Loss 0.0067, Sem 0.0067 LR: 1.368e-02 Score 99.760 Data time: 0.2257, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6444 [2021-07-09 06:16:47,297][root][INFO] - ===> Epoch[10685](53425/5): Loss 0.0067, Sem 0.0067 LR: 1.367e-02 Score 99.760 Data time: 0.1601, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7890 [2021-07-09 06:17:01,381][root][INFO] - ===> Epoch[10686](53430/5): Loss 0.0068, Sem 0.0068 LR: 1.366e-02 Score 99.757 Data time: 0.2398, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7440 [2021-07-09 06:17:14,921][root][INFO] - ===> Epoch[10687](53435/5): Loss 0.0068, Sem 0.0068 LR: 1.365e-02 Score 99.755 Data time: 0.1863, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6634 [2021-07-09 06:17:28,410][root][INFO] - ===> Epoch[10688](53440/5): Loss 0.0067, Sem 0.0067 LR: 1.364e-02 Score 99.761 Data time: 0.1889, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6469 [2021-07-09 06:17:42,166][root][INFO] - ===> Epoch[10689](53445/5): Loss 0.0068, Sem 0.0068 LR: 1.363e-02 Score 99.755 Data time: 0.1624, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7120 [2021-07-09 06:17:55,546][root][INFO] - ===> Epoch[10690](53450/5): Loss 0.0070, Sem 0.0070 LR: 1.362e-02 Score 99.750 Data time: 0.2587, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6257 [2021-07-09 06:18:08,812][root][INFO] - ===> Epoch[10691](53455/5): Loss 0.0071, Sem 0.0071 LR: 1.362e-02 Score 99.749 Data time: 0.2440, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.5984 [2021-07-09 06:18:23,096][root][INFO] - ===> Epoch[10692](53460/5): Loss 0.0067, Sem 0.0067 LR: 1.361e-02 Score 99.762 Data time: 0.1982, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8077 [2021-07-09 06:18:37,027][root][INFO] - ===> Epoch[10693](53465/5): Loss 0.0067, Sem 0.0067 LR: 1.360e-02 Score 99.762 Data time: 0.1998, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7456 [2021-07-09 06:18:50,457][root][INFO] - ===> Epoch[10694](53470/5): Loss 0.0068, Sem 0.0068 LR: 1.359e-02 Score 99.759 Data time: 0.2231, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6369 [2021-07-09 06:19:05,776][root][INFO] - ===> Epoch[10695](53475/5): Loss 0.0072, Sem 0.0072 LR: 1.358e-02 Score 99.745 Data time: 0.2131, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 3.0077 [2021-07-09 06:19:19,722][root][INFO] - ===> Epoch[10696](53480/5): Loss 0.0070, Sem 0.0070 LR: 1.357e-02 Score 99.750 Data time: 0.1923, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7479 [2021-07-09 06:19:33,054][root][INFO] - ===> Epoch[10697](53485/5): Loss 0.0069, Sem 0.0069 LR: 1.356e-02 Score 99.751 Data time: 0.2426, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6229 [2021-07-09 06:19:47,271][root][INFO] - ===> Epoch[10698](53490/5): Loss 0.0065, Sem 0.0065 LR: 1.355e-02 Score 99.768 Data time: 0.1752, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7986 [2021-07-09 06:20:01,864][root][INFO] - ===> Epoch[10699](53495/5): Loss 0.0072, Sem 0.0072 LR: 1.354e-02 Score 99.744 Data time: 0.1951, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8839 [2021-07-09 06:20:16,036][root][INFO] - ===> Epoch[10700](53500/5): Loss 0.0067, Sem 0.0067 LR: 1.353e-02 Score 99.759 Data time: 0.2363, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7872 [2021-07-09 06:20:30,581][root][INFO] - ===> Epoch[10701](53505/5): Loss 0.0069, Sem 0.0069 LR: 1.352e-02 Score 99.755 Data time: 0.3166, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8605 [2021-07-09 06:20:44,903][root][INFO] - ===> Epoch[10702](53510/5): Loss 0.0070, Sem 0.0070 LR: 1.351e-02 Score 99.751 Data time: 0.1842, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8185 [2021-07-09 06:20:59,638][root][INFO] - ===> Epoch[10703](53515/5): Loss 0.0066, Sem 0.0066 LR: 1.350e-02 Score 99.765 Data time: 0.2542, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.9048 [2021-07-09 06:21:13,140][root][INFO] - ===> Epoch[10704](53520/5): Loss 0.0071, Sem 0.0071 LR: 1.349e-02 Score 99.746 Data time: 0.2108, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.6540 [2021-07-09 06:21:27,723][root][INFO] - ===> Epoch[10705](53525/5): Loss 0.0071, Sem 0.0071 LR: 1.348e-02 Score 99.749 Data time: 0.1850, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.8742 [2021-07-09 06:21:41,418][root][INFO] - ===> Epoch[10706](53530/5): Loss 0.0069, Sem 0.0069 LR: 1.348e-02 Score 99.755 Data time: 0.2611, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.6871 [2021-07-09 06:21:55,327][root][INFO] - ===> Epoch[10707](53535/5): Loss 0.0067, Sem 0.0067 LR: 1.347e-02 Score 99.762 Data time: 0.2769, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7325 [2021-07-09 06:22:09,438][root][INFO] - ===> Epoch[10708](53540/5): Loss 0.0070, Sem 0.0070 LR: 1.346e-02 Score 99.752 Data time: 0.2351, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7830 [2021-07-09 06:22:23,659][root][INFO] - ===> Epoch[10709](53545/5): Loss 0.0068, Sem 0.0068 LR: 1.345e-02 Score 99.754 Data time: 0.2649, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7905 [2021-07-09 06:22:37,371][root][INFO] - ===> Epoch[10710](53550/5): Loss 0.0066, Sem 0.0066 LR: 1.344e-02 Score 99.764 Data time: 0.1973, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7062 [2021-07-09 06:22:51,760][root][INFO] - ===> Epoch[10711](53555/5): Loss 0.0069, Sem 0.0069 LR: 1.343e-02 Score 99.753 Data time: 0.2423, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8324 [2021-07-09 06:23:05,842][root][INFO] - ===> Epoch[10712](53560/5): Loss 0.0070, Sem 0.0070 LR: 1.342e-02 Score 99.751 Data time: 0.2508, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7680 [2021-07-09 06:23:19,896][root][INFO] - ===> Epoch[10713](53565/5): Loss 0.0067, Sem 0.0067 LR: 1.341e-02 Score 99.762 Data time: 0.2335, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7630 [2021-07-09 06:23:33,717][root][INFO] - ===> Epoch[10714](53570/5): Loss 0.0070, Sem 0.0070 LR: 1.340e-02 Score 99.751 Data time: 0.2206, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7187 [2021-07-09 06:23:47,345][root][INFO] - ===> Epoch[10715](53575/5): Loss 0.0069, Sem 0.0069 LR: 1.339e-02 Score 99.754 Data time: 0.2459, Forward time: 1.2523, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6685 [2021-07-09 06:24:00,893][root][INFO] - ===> Epoch[10716](53580/5): Loss 0.0070, Sem 0.0070 LR: 1.338e-02 Score 99.751 Data time: 0.2506, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6592 [2021-07-09 06:24:14,353][root][INFO] - ===> Epoch[10717](53585/5): Loss 0.0071, Sem 0.0071 LR: 1.337e-02 Score 99.746 Data time: 0.2177, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6411 [2021-07-09 06:24:28,638][root][INFO] - ===> Epoch[10718](53590/5): Loss 0.0067, Sem 0.0067 LR: 1.336e-02 Score 99.762 Data time: 0.2634, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8151 [2021-07-09 06:24:42,620][root][INFO] - ===> Epoch[10719](53595/5): Loss 0.0068, Sem 0.0068 LR: 1.335e-02 Score 99.757 Data time: 0.1743, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7512 [2021-07-09 06:24:57,123][root][INFO] - ===> Epoch[10720](53600/5): Loss 0.0068, Sem 0.0068 LR: 1.334e-02 Score 99.760 Data time: 0.1994, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8500 [2021-07-09 06:24:57,124][root][INFO] - ===> Start testing [2021-07-09 06:25:40,221][root][INFO] - 68/68: Data time: 0.0017, Iter time: 0.1314 Loss 0.423 (AVG: 0.631) Score 91.625 (AVG: 91.000) mIOU 71.365 mAP 82.291 mAcc 77.664 IOU: 64.304 0.000 79.361 75.153 95.032 91.685 30.579 83.624 98.597 85.122 79.860 86.240 58.188 mAP: 82.553 2.103 87.892 85.238 98.245 98.106 51.557 97.832 99.652 94.343 90.272 92.434 89.553 mAcc: 80.246 0.000 84.522 82.749 96.617 97.416 33.170 94.363 99.380 92.739 90.086 96.440 61.904 [2021-07-09 06:25:40,222][root][INFO] - Finished test. Elapsed time: 43.0967 [2021-07-09 06:25:41,402][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 06:25:54,726][root][INFO] - ===> Epoch[10721](53605/5): Loss 0.0068, Sem 0.0068 LR: 1.333e-02 Score 99.758 Data time: 0.1753, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6292 [2021-07-09 06:26:08,744][root][INFO] - ===> Epoch[10722](53610/5): Loss 0.0068, Sem 0.0068 LR: 1.333e-02 Score 99.755 Data time: 0.2569, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7533 [2021-07-09 06:26:22,992][root][INFO] - ===> Epoch[10723](53615/5): Loss 0.0069, Sem 0.0069 LR: 1.332e-02 Score 99.753 Data time: 0.1924, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8059 [2021-07-09 06:26:37,183][root][INFO] - ===> Epoch[10724](53620/5): Loss 0.0067, Sem 0.0067 LR: 1.331e-02 Score 99.760 Data time: 0.2793, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.7936 [2021-07-09 06:26:51,521][root][INFO] - ===> Epoch[10725](53625/5): Loss 0.0068, Sem 0.0068 LR: 1.330e-02 Score 99.759 Data time: 0.2123, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.8266 [2021-07-09 06:27:05,121][root][INFO] - ===> Epoch[10726](53630/5): Loss 0.0070, Sem 0.0070 LR: 1.329e-02 Score 99.747 Data time: 0.2279, Forward time: 1.2522, Backward time: 1.2479, DDP time: 0.0042, Total iter time: 2.6718 [2021-07-09 06:27:18,939][root][INFO] - ===> Epoch[10727](53635/5): Loss 0.0070, Sem 0.0070 LR: 1.328e-02 Score 99.751 Data time: 0.2084, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7142 [2021-07-09 06:27:32,411][root][INFO] - ===> Epoch[10728](53640/5): Loss 0.0071, Sem 0.0071 LR: 1.327e-02 Score 99.748 Data time: 0.1560, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.6596 [2021-07-09 06:27:46,848][root][INFO] - ===> Epoch[10729](53645/5): Loss 0.0072, Sem 0.0072 LR: 1.326e-02 Score 99.742 Data time: 0.2615, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.8416 [2021-07-09 06:28:00,239][root][INFO] - ===> Epoch[10730](53650/5): Loss 0.0066, Sem 0.0066 LR: 1.325e-02 Score 99.761 Data time: 0.1963, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.6282 [2021-07-09 06:28:14,581][root][INFO] - ===> Epoch[10731](53655/5): Loss 0.0068, Sem 0.0068 LR: 1.324e-02 Score 99.756 Data time: 0.2089, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.8219 [2021-07-09 06:28:27,690][root][INFO] - ===> Epoch[10732](53660/5): Loss 0.0065, Sem 0.0065 LR: 1.323e-02 Score 99.765 Data time: 0.1847, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.5776 [2021-07-09 06:28:41,794][root][INFO] - ===> Epoch[10733](53665/5): Loss 0.0070, Sem 0.0070 LR: 1.322e-02 Score 99.748 Data time: 0.2346, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7752 [2021-07-09 06:28:55,878][root][INFO] - ===> Epoch[10734](53670/5): Loss 0.0069, Sem 0.0069 LR: 1.321e-02 Score 99.762 Data time: 0.1974, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7732 [2021-07-09 06:29:09,932][root][INFO] - ===> Epoch[10735](53675/5): Loss 0.0068, Sem 0.0068 LR: 1.320e-02 Score 99.759 Data time: 0.2355, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7707 [2021-07-09 06:29:23,675][root][INFO] - ===> Epoch[10736](53680/5): Loss 0.0065, Sem 0.0065 LR: 1.319e-02 Score 99.767 Data time: 0.2052, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7071 [2021-07-09 06:29:38,295][root][INFO] - ===> Epoch[10737](53685/5): Loss 0.0070, Sem 0.0070 LR: 1.318e-02 Score 99.747 Data time: 0.2027, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.8815 [2021-07-09 06:29:52,813][root][INFO] - ===> Epoch[10738](53690/5): Loss 0.0068, Sem 0.0068 LR: 1.317e-02 Score 99.762 Data time: 0.1977, Forward time: 1.2522, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.8600 [2021-07-09 06:30:06,598][root][INFO] - ===> Epoch[10739](53695/5): Loss 0.0071, Sem 0.0071 LR: 1.317e-02 Score 99.747 Data time: 0.2118, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7129 [2021-07-09 06:30:20,378][root][INFO] - ===> Epoch[10740](53700/5): Loss 0.0072, Sem 0.0072 LR: 1.316e-02 Score 99.744 Data time: 0.2431, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7063 [2021-07-09 06:30:35,034][root][INFO] - ===> Epoch[10741](53705/5): Loss 0.0063, Sem 0.0063 LR: 1.315e-02 Score 99.772 Data time: 0.2081, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.8781 [2021-07-09 06:30:48,757][root][INFO] - ===> Epoch[10742](53710/5): Loss 0.0068, Sem 0.0068 LR: 1.314e-02 Score 99.755 Data time: 0.2276, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.6989 [2021-07-09 06:31:03,147][root][INFO] - ===> Epoch[10743](53715/5): Loss 0.0068, Sem 0.0068 LR: 1.313e-02 Score 99.759 Data time: 0.2568, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.8276 [2021-07-09 06:31:16,793][root][INFO] - ===> Epoch[10744](53720/5): Loss 0.0070, Sem 0.0070 LR: 1.312e-02 Score 99.749 Data time: 0.2397, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.6863 [2021-07-09 06:31:31,629][root][INFO] - ===> Epoch[10745](53725/5): Loss 0.0067, Sem 0.0067 LR: 1.311e-02 Score 99.763 Data time: 0.2261, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.9245 [2021-07-09 06:31:45,433][root][INFO] - ===> Epoch[10746](53730/5): Loss 0.0069, Sem 0.0069 LR: 1.310e-02 Score 99.754 Data time: 0.2364, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7152 [2021-07-09 06:31:59,743][root][INFO] - ===> Epoch[10747](53735/5): Loss 0.0069, Sem 0.0069 LR: 1.309e-02 Score 99.755 Data time: 0.2155, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.8178 [2021-07-09 06:32:14,044][root][INFO] - ===> Epoch[10748](53740/5): Loss 0.0071, Sem 0.0071 LR: 1.308e-02 Score 99.745 Data time: 0.2518, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.8143 [2021-07-09 06:32:27,820][root][INFO] - ===> Epoch[10749](53745/5): Loss 0.0070, Sem 0.0070 LR: 1.307e-02 Score 99.755 Data time: 0.2206, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7069 [2021-07-09 06:32:41,776][root][INFO] - ===> Epoch[10750](53750/5): Loss 0.0068, Sem 0.0068 LR: 1.306e-02 Score 99.755 Data time: 0.1742, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7420 [2021-07-09 06:32:55,023][root][INFO] - ===> Epoch[10751](53755/5): Loss 0.0069, Sem 0.0069 LR: 1.305e-02 Score 99.753 Data time: 0.1654, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.6146 [2021-07-09 06:33:08,771][root][INFO] - ===> Epoch[10752](53760/5): Loss 0.0069, Sem 0.0069 LR: 1.304e-02 Score 99.756 Data time: 0.1816, Forward time: 1.2521, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6987 [2021-07-09 06:33:22,223][root][INFO] - ===> Epoch[10753](53765/5): Loss 0.0068, Sem 0.0068 LR: 1.303e-02 Score 99.757 Data time: 0.2418, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.6417 [2021-07-09 06:33:35,887][root][INFO] - ===> Epoch[10754](53770/5): Loss 0.0070, Sem 0.0070 LR: 1.302e-02 Score 99.749 Data time: 0.1742, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.6908 [2021-07-09 06:33:50,073][root][INFO] - ===> Epoch[10755](53775/5): Loss 0.0071, Sem 0.0071 LR: 1.302e-02 Score 99.744 Data time: 0.2174, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7870 [2021-07-09 06:34:04,253][root][INFO] - ===> Epoch[10756](53780/5): Loss 0.0069, Sem 0.0069 LR: 1.301e-02 Score 99.755 Data time: 0.2086, Forward time: 1.2521, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7842 [2021-07-09 06:34:18,312][root][INFO] - ===> Epoch[10757](53785/5): Loss 0.0071, Sem 0.0071 LR: 1.300e-02 Score 99.749 Data time: 0.2319, Forward time: 1.2521, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7666 [2021-07-09 06:34:32,347][root][INFO] - ===> Epoch[10758](53790/5): Loss 0.0067, Sem 0.0067 LR: 1.299e-02 Score 99.761 Data time: 0.2770, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7512 [2021-07-09 06:34:46,002][root][INFO] - ===> Epoch[10759](53795/5): Loss 0.0071, Sem 0.0071 LR: 1.298e-02 Score 99.747 Data time: 0.2403, Forward time: 1.2521, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6837 [2021-07-09 06:35:00,320][root][INFO] - ===> Epoch[10760](53800/5): Loss 0.0071, Sem 0.0071 LR: 1.297e-02 Score 99.748 Data time: 0.1727, Forward time: 1.2521, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.8255 [2021-07-09 06:35:00,322][root][INFO] - ===> Start testing [2021-07-09 06:35:36,986][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1352 Loss 0.432 (AVG: 0.619) Score 91.008 (AVG: 91.082) mIOU 71.919 mAP 82.688 mAcc 78.286 IOU: 64.067 0.000 79.646 75.184 94.999 91.450 38.261 85.442 98.542 84.500 78.887 86.317 57.652 mAP: 81.787 2.469 86.785 85.718 98.254 97.983 57.626 98.070 99.595 94.915 90.436 92.150 89.158 mAcc: 79.218 0.000 83.258 83.221 96.772 97.531 42.739 94.773 99.351 94.071 89.807 96.408 60.571 [2021-07-09 06:35:36,986][root][INFO] - Finished test. Elapsed time: 36.6631 [2021-07-09 06:35:38,033][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 06:35:51,068][root][INFO] - ===> Epoch[10761](53805/5): Loss 0.0067, Sem 0.0067 LR: 1.296e-02 Score 99.761 Data time: 0.2244, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.5665 [2021-07-09 06:36:05,619][root][INFO] - ===> Epoch[10762](53810/5): Loss 0.0068, Sem 0.0068 LR: 1.295e-02 Score 99.758 Data time: 0.2379, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.8661 [2021-07-09 06:36:20,473][root][INFO] - ===> Epoch[10763](53815/5): Loss 0.0070, Sem 0.0070 LR: 1.294e-02 Score 99.749 Data time: 0.2053, Forward time: 1.2521, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.9197 [2021-07-09 06:36:33,704][root][INFO] - ===> Epoch[10764](53820/5): Loss 0.0070, Sem 0.0070 LR: 1.293e-02 Score 99.752 Data time: 0.2955, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.5949 [2021-07-09 06:36:48,095][root][INFO] - ===> Epoch[10765](53825/5): Loss 0.0069, Sem 0.0069 LR: 1.292e-02 Score 99.757 Data time: 0.2300, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.8354 [2021-07-09 06:37:02,322][root][INFO] - ===> Epoch[10766](53830/5): Loss 0.0069, Sem 0.0069 LR: 1.291e-02 Score 99.750 Data time: 0.2413, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7960 [2021-07-09 06:37:16,828][root][INFO] - ===> Epoch[10767](53835/5): Loss 0.0072, Sem 0.0072 LR: 1.290e-02 Score 99.744 Data time: 0.2116, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.8478 [2021-07-09 06:37:31,789][root][INFO] - ===> Epoch[10768](53840/5): Loss 0.0068, Sem 0.0068 LR: 1.289e-02 Score 99.753 Data time: 0.2686, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.9400 [2021-07-09 06:37:45,665][root][INFO] - ===> Epoch[10769](53845/5): Loss 0.0068, Sem 0.0068 LR: 1.288e-02 Score 99.755 Data time: 0.2061, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7395 [2021-07-09 06:37:59,201][root][INFO] - ===> Epoch[10770](53850/5): Loss 0.0067, Sem 0.0067 LR: 1.287e-02 Score 99.759 Data time: 0.1929, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.6609 [2021-07-09 06:38:12,723][root][INFO] - ===> Epoch[10771](53855/5): Loss 0.0070, Sem 0.0070 LR: 1.286e-02 Score 99.746 Data time: 0.2533, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.6636 [2021-07-09 06:38:26,802][root][INFO] - ===> Epoch[10772](53860/5): Loss 0.0065, Sem 0.0065 LR: 1.286e-02 Score 99.766 Data time: 0.2147, Forward time: 1.2521, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7632 [2021-07-09 06:38:40,885][root][INFO] - ===> Epoch[10773](53865/5): Loss 0.0065, Sem 0.0065 LR: 1.285e-02 Score 99.766 Data time: 0.2026, Forward time: 1.2521, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7750 [2021-07-09 06:38:54,307][root][INFO] - ===> Epoch[10774](53870/5): Loss 0.0073, Sem 0.0073 LR: 1.284e-02 Score 99.740 Data time: 0.2007, Forward time: 1.2521, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6434 [2021-07-09 06:39:08,196][root][INFO] - ===> Epoch[10775](53875/5): Loss 0.0069, Sem 0.0069 LR: 1.283e-02 Score 99.753 Data time: 0.2404, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7310 [2021-07-09 06:39:22,131][root][INFO] - ===> Epoch[10776](53880/5): Loss 0.0070, Sem 0.0070 LR: 1.282e-02 Score 99.752 Data time: 0.2651, Forward time: 1.2521, Backward time: 1.2480, DDP time: 0.0042, Total iter time: 2.7318 [2021-07-09 06:39:36,418][root][INFO] - ===> Epoch[10777](53885/5): Loss 0.0066, Sem 0.0066 LR: 1.281e-02 Score 99.762 Data time: 0.1877, Forward time: 1.2521, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.8228 [2021-07-09 06:39:49,525][root][INFO] - ===> Epoch[10778](53890/5): Loss 0.0072, Sem 0.0072 LR: 1.280e-02 Score 99.743 Data time: 0.1556, Forward time: 1.2521, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.5885 [2021-07-09 06:40:02,848][root][INFO] - ===> Epoch[10779](53895/5): Loss 0.0068, Sem 0.0068 LR: 1.279e-02 Score 99.759 Data time: 0.1486, Forward time: 1.2521, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6285 [2021-07-09 06:40:16,687][root][INFO] - ===> Epoch[10780](53900/5): Loss 0.0072, Sem 0.0072 LR: 1.278e-02 Score 99.747 Data time: 0.2166, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7283 [2021-07-09 06:40:31,047][root][INFO] - ===> Epoch[10781](53905/5): Loss 0.0066, Sem 0.0066 LR: 1.277e-02 Score 99.759 Data time: 0.2109, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.8202 [2021-07-09 06:40:44,791][root][INFO] - ===> Epoch[10782](53910/5): Loss 0.0070, Sem 0.0070 LR: 1.276e-02 Score 99.752 Data time: 0.1873, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7125 [2021-07-09 06:40:58,214][root][INFO] - ===> Epoch[10783](53915/5): Loss 0.0072, Sem 0.0072 LR: 1.275e-02 Score 99.742 Data time: 0.1752, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6409 [2021-07-09 06:41:12,777][root][INFO] - ===> Epoch[10784](53920/5): Loss 0.0068, Sem 0.0068 LR: 1.274e-02 Score 99.758 Data time: 0.2686, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.8671 [2021-07-09 06:41:26,235][root][INFO] - ===> Epoch[10785](53925/5): Loss 0.0067, Sem 0.0067 LR: 1.273e-02 Score 99.761 Data time: 0.1920, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6572 [2021-07-09 06:41:39,700][root][INFO] - ===> Epoch[10786](53930/5): Loss 0.0071, Sem 0.0071 LR: 1.272e-02 Score 99.744 Data time: 0.1890, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6515 [2021-07-09 06:41:53,589][root][INFO] - ===> Epoch[10787](53935/5): Loss 0.0067, Sem 0.0067 LR: 1.271e-02 Score 99.762 Data time: 0.2113, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7289 [2021-07-09 06:42:07,667][root][INFO] - ===> Epoch[10788](53940/5): Loss 0.0072, Sem 0.0072 LR: 1.270e-02 Score 99.745 Data time: 0.1783, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7741 [2021-07-09 06:42:21,323][root][INFO] - ===> Epoch[10789](53945/5): Loss 0.0069, Sem 0.0069 LR: 1.269e-02 Score 99.754 Data time: 0.2313, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6799 [2021-07-09 06:42:35,218][root][INFO] - ===> Epoch[10790](53950/5): Loss 0.0065, Sem 0.0065 LR: 1.269e-02 Score 99.766 Data time: 0.2403, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7340 [2021-07-09 06:42:49,385][root][INFO] - ===> Epoch[10791](53955/5): Loss 0.0069, Sem 0.0069 LR: 1.268e-02 Score 99.753 Data time: 0.1738, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7987 [2021-07-09 06:43:02,841][root][INFO] - ===> Epoch[10792](53960/5): Loss 0.0067, Sem 0.0067 LR: 1.267e-02 Score 99.761 Data time: 0.2321, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6409 [2021-07-09 06:43:17,007][root][INFO] - ===> Epoch[10793](53965/5): Loss 0.0069, Sem 0.0069 LR: 1.266e-02 Score 99.753 Data time: 0.1909, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7916 [2021-07-09 06:43:30,421][root][INFO] - ===> Epoch[10794](53970/5): Loss 0.0072, Sem 0.0072 LR: 1.265e-02 Score 99.743 Data time: 0.2294, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6417 [2021-07-09 06:43:43,874][root][INFO] - ===> Epoch[10795](53975/5): Loss 0.0071, Sem 0.0071 LR: 1.264e-02 Score 99.745 Data time: 0.2319, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6417 [2021-07-09 06:43:58,601][root][INFO] - ===> Epoch[10796](53980/5): Loss 0.0069, Sem 0.0069 LR: 1.263e-02 Score 99.755 Data time: 0.2314, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.8875 [2021-07-09 06:44:12,554][root][INFO] - ===> Epoch[10797](53985/5): Loss 0.0071, Sem 0.0071 LR: 1.262e-02 Score 99.746 Data time: 0.2392, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7441 [2021-07-09 06:44:25,875][root][INFO] - ===> Epoch[10798](53990/5): Loss 0.0069, Sem 0.0069 LR: 1.261e-02 Score 99.756 Data time: 0.2140, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6204 [2021-07-09 06:44:40,133][root][INFO] - ===> Epoch[10799](53995/5): Loss 0.0067, Sem 0.0067 LR: 1.260e-02 Score 99.756 Data time: 0.2089, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.8118 [2021-07-09 06:44:54,909][root][INFO] - ===> Epoch[10800](54000/5): Loss 0.0067, Sem 0.0067 LR: 1.259e-02 Score 99.759 Data time: 0.1798, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.9168 [2021-07-09 06:44:54,910][root][INFO] - ===> Start testing [2021-07-09 06:45:32,001][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1214 Loss 0.478 (AVG: 0.626) Score 91.159 (AVG: 91.115) mIOU 71.758 mAP 82.691 mAcc 78.322 IOU: 64.067 0.000 79.972 75.482 95.004 91.204 36.513 85.417 98.569 82.857 79.695 86.436 57.636 mAP: 83.158 2.708 88.468 85.647 98.309 98.025 54.751 97.949 99.669 94.400 90.651 92.038 89.208 mAcc: 79.375 0.000 84.419 83.400 96.720 97.702 40.118 95.484 99.294 94.367 90.262 96.414 60.626 [2021-07-09 06:45:32,002][root][INFO] - Finished test. Elapsed time: 37.0903 [2021-07-09 06:45:33,068][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 06:45:45,467][root][INFO] - ===> Epoch[10801](54005/5): Loss 0.0065, Sem 0.0065 LR: 1.258e-02 Score 99.771 Data time: 0.2094, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.4409 [2021-07-09 06:45:59,215][root][INFO] - ===> Epoch[10802](54010/5): Loss 0.0068, Sem 0.0068 LR: 1.257e-02 Score 99.759 Data time: 0.2125, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7000 [2021-07-09 06:46:14,085][root][INFO] - ===> Epoch[10803](54015/5): Loss 0.0068, Sem 0.0068 LR: 1.256e-02 Score 99.758 Data time: 0.2170, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.9347 [2021-07-09 06:46:28,136][root][INFO] - ===> Epoch[10804](54020/5): Loss 0.0070, Sem 0.0070 LR: 1.255e-02 Score 99.747 Data time: 0.2086, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7611 [2021-07-09 06:46:41,876][root][INFO] - ===> Epoch[10805](54025/5): Loss 0.0068, Sem 0.0068 LR: 1.254e-02 Score 99.757 Data time: 0.2133, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6997 [2021-07-09 06:46:56,598][root][INFO] - ===> Epoch[10806](54030/5): Loss 0.0070, Sem 0.0070 LR: 1.253e-02 Score 99.748 Data time: 0.2557, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.8875 [2021-07-09 06:47:10,034][root][INFO] - ===> Epoch[10807](54035/5): Loss 0.0069, Sem 0.0069 LR: 1.252e-02 Score 99.755 Data time: 0.1895, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6488 [2021-07-09 06:47:23,526][root][INFO] - ===> Epoch[10808](54040/5): Loss 0.0066, Sem 0.0066 LR: 1.252e-02 Score 99.764 Data time: 0.1871, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6493 [2021-07-09 06:47:37,597][root][INFO] - ===> Epoch[10809](54045/5): Loss 0.0071, Sem 0.0071 LR: 1.251e-02 Score 99.745 Data time: 0.2330, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7729 [2021-07-09 06:47:51,397][root][INFO] - ===> Epoch[10810](54050/5): Loss 0.0067, Sem 0.0067 LR: 1.250e-02 Score 99.760 Data time: 0.2094, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7162 [2021-07-09 06:48:04,707][root][INFO] - ===> Epoch[10811](54055/5): Loss 0.0074, Sem 0.0074 LR: 1.249e-02 Score 99.737 Data time: 0.1700, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6350 [2021-07-09 06:48:19,425][root][INFO] - ===> Epoch[10812](54060/5): Loss 0.0065, Sem 0.0065 LR: 1.248e-02 Score 99.766 Data time: 0.1417, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8955 [2021-07-09 06:48:33,449][root][INFO] - ===> Epoch[10813](54065/5): Loss 0.0070, Sem 0.0070 LR: 1.247e-02 Score 99.750 Data time: 0.2369, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7552 [2021-07-09 06:48:47,663][root][INFO] - ===> Epoch[10814](54070/5): Loss 0.0063, Sem 0.0063 LR: 1.246e-02 Score 99.772 Data time: 0.2808, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7822 [2021-07-09 06:49:02,175][root][INFO] - ===> Epoch[10815](54075/5): Loss 0.0069, Sem 0.0069 LR: 1.245e-02 Score 99.756 Data time: 0.2898, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8448 [2021-07-09 06:49:16,377][root][INFO] - ===> Epoch[10816](54080/5): Loss 0.0069, Sem 0.0069 LR: 1.244e-02 Score 99.753 Data time: 0.2775, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7897 [2021-07-09 06:49:30,993][root][INFO] - ===> Epoch[10817](54085/5): Loss 0.0069, Sem 0.0069 LR: 1.243e-02 Score 99.752 Data time: 0.2283, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.8681 [2021-07-09 06:49:45,600][root][INFO] - ===> Epoch[10818](54090/5): Loss 0.0066, Sem 0.0066 LR: 1.242e-02 Score 99.766 Data time: 0.2504, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8729 [2021-07-09 06:50:00,340][root][INFO] - ===> Epoch[10819](54095/5): Loss 0.0068, Sem 0.0068 LR: 1.241e-02 Score 99.758 Data time: 0.3188, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8989 [2021-07-09 06:50:14,600][root][INFO] - ===> Epoch[10820](54100/5): Loss 0.0066, Sem 0.0066 LR: 1.240e-02 Score 99.766 Data time: 0.2524, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7979 [2021-07-09 06:50:28,952][root][INFO] - ===> Epoch[10821](54105/5): Loss 0.0070, Sem 0.0070 LR: 1.239e-02 Score 99.750 Data time: 0.2343, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.8083 [2021-07-09 06:50:42,316][root][INFO] - ===> Epoch[10822](54110/5): Loss 0.0067, Sem 0.0067 LR: 1.238e-02 Score 99.760 Data time: 0.2284, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6323 [2021-07-09 06:50:56,990][root][INFO] - ===> Epoch[10823](54115/5): Loss 0.0071, Sem 0.0071 LR: 1.237e-02 Score 99.749 Data time: 0.3256, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.8746 [2021-07-09 06:51:09,715][root][INFO] - ===> Epoch[10824](54120/5): Loss 0.0071, Sem 0.0071 LR: 1.236e-02 Score 99.742 Data time: 0.2218, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.5056 [2021-07-09 06:51:24,789][root][INFO] - ===> Epoch[10825](54125/5): Loss 0.0071, Sem 0.0071 LR: 1.235e-02 Score 99.751 Data time: 0.2245, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.9742 [2021-07-09 06:51:38,978][root][INFO] - ===> Epoch[10826](54130/5): Loss 0.0067, Sem 0.0067 LR: 1.235e-02 Score 99.760 Data time: 0.2254, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7935 [2021-07-09 06:51:52,460][root][INFO] - ===> Epoch[10827](54135/5): Loss 0.0069, Sem 0.0069 LR: 1.234e-02 Score 99.755 Data time: 0.1920, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6460 [2021-07-09 06:52:06,641][root][INFO] - ===> Epoch[10828](54140/5): Loss 0.0066, Sem 0.0066 LR: 1.233e-02 Score 99.760 Data time: 0.3112, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7921 [2021-07-09 06:52:21,201][root][INFO] - ===> Epoch[10829](54145/5): Loss 0.0068, Sem 0.0068 LR: 1.232e-02 Score 99.755 Data time: 0.1621, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.8756 [2021-07-09 06:52:34,764][root][INFO] - ===> Epoch[10830](54150/5): Loss 0.0066, Sem 0.0066 LR: 1.231e-02 Score 99.763 Data time: 0.2043, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6758 [2021-07-09 06:52:48,640][root][INFO] - ===> Epoch[10831](54155/5): Loss 0.0069, Sem 0.0069 LR: 1.230e-02 Score 99.755 Data time: 0.2195, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7327 [2021-07-09 06:53:02,600][root][INFO] - ===> Epoch[10832](54160/5): Loss 0.0066, Sem 0.0066 LR: 1.229e-02 Score 99.764 Data time: 0.1898, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7495 [2021-07-09 06:53:15,982][root][INFO] - ===> Epoch[10833](54165/5): Loss 0.0067, Sem 0.0067 LR: 1.228e-02 Score 99.762 Data time: 0.2179, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6295 [2021-07-09 06:53:30,007][root][INFO] - ===> Epoch[10834](54170/5): Loss 0.0071, Sem 0.0071 LR: 1.227e-02 Score 99.748 Data time: 0.1931, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7557 [2021-07-09 06:53:43,797][root][INFO] - ===> Epoch[10835](54175/5): Loss 0.0067, Sem 0.0067 LR: 1.226e-02 Score 99.764 Data time: 0.1675, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7147 [2021-07-09 06:53:57,554][root][INFO] - ===> Epoch[10836](54180/5): Loss 0.0064, Sem 0.0064 LR: 1.225e-02 Score 99.770 Data time: 0.2264, Forward time: 1.2520, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7149 [2021-07-09 06:54:11,469][root][INFO] - ===> Epoch[10837](54185/5): Loss 0.0069, Sem 0.0069 LR: 1.224e-02 Score 99.750 Data time: 0.2069, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7415 [2021-07-09 06:54:24,929][root][INFO] - ===> Epoch[10838](54190/5): Loss 0.0069, Sem 0.0069 LR: 1.223e-02 Score 99.757 Data time: 0.2123, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6501 [2021-07-09 06:54:38,377][root][INFO] - ===> Epoch[10839](54195/5): Loss 0.0069, Sem 0.0069 LR: 1.222e-02 Score 99.754 Data time: 0.2301, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6407 [2021-07-09 06:54:51,669][root][INFO] - ===> Epoch[10840](54200/5): Loss 0.0068, Sem 0.0068 LR: 1.221e-02 Score 99.758 Data time: 0.1948, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6076 [2021-07-09 06:54:51,671][root][INFO] - ===> Start testing [2021-07-09 06:55:25,759][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1409 Loss 0.465 (AVG: 0.633) Score 91.506 (AVG: 91.076) mIOU 71.847 mAP 82.415 mAcc 78.227 IOU: 63.758 0.000 79.907 75.107 95.098 91.148 35.414 85.960 98.550 84.672 79.600 86.380 58.415 mAP: 82.435 2.071 88.557 85.036 98.268 97.896 53.597 98.080 99.598 94.035 90.220 92.006 89.598 mAcc: 79.191 0.000 84.368 83.440 96.707 97.499 38.585 95.470 99.323 93.903 90.149 96.274 62.036 [2021-07-09 06:55:25,759][root][INFO] - Finished test. Elapsed time: 34.0881 [2021-07-09 06:55:26,888][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 06:55:40,958][root][INFO] - ===> Epoch[10841](54205/5): Loss 0.0064, Sem 0.0064 LR: 1.220e-02 Score 99.770 Data time: 0.1782, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7783 [2021-07-09 06:55:55,667][root][INFO] - ===> Epoch[10842](54210/5): Loss 0.0066, Sem 0.0066 LR: 1.219e-02 Score 99.766 Data time: 0.2050, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.8879 [2021-07-09 06:56:09,771][root][INFO] - ===> Epoch[10843](54215/5): Loss 0.0066, Sem 0.0066 LR: 1.218e-02 Score 99.768 Data time: 0.2088, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7658 [2021-07-09 06:56:23,094][root][INFO] - ===> Epoch[10844](54220/5): Loss 0.0067, Sem 0.0067 LR: 1.217e-02 Score 99.760 Data time: 0.1894, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6107 [2021-07-09 06:56:37,002][root][INFO] - ===> Epoch[10845](54225/5): Loss 0.0072, Sem 0.0072 LR: 1.217e-02 Score 99.743 Data time: 0.1937, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.7426 [2021-07-09 06:56:51,362][root][INFO] - ===> Epoch[10846](54230/5): Loss 0.0068, Sem 0.0068 LR: 1.216e-02 Score 99.757 Data time: 0.1930, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.8246 [2021-07-09 06:57:06,662][root][INFO] - ===> Epoch[10847](54235/5): Loss 0.0068, Sem 0.0068 LR: 1.215e-02 Score 99.756 Data time: 0.1986, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 3.0142 [2021-07-09 06:57:20,793][root][INFO] - ===> Epoch[10848](54240/5): Loss 0.0066, Sem 0.0066 LR: 1.214e-02 Score 99.762 Data time: 0.2231, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7740 [2021-07-09 06:57:35,162][root][INFO] - ===> Epoch[10849](54245/5): Loss 0.0067, Sem 0.0067 LR: 1.213e-02 Score 99.761 Data time: 0.2311, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8183 [2021-07-09 06:57:48,659][root][INFO] - ===> Epoch[10850](54250/5): Loss 0.0074, Sem 0.0074 LR: 1.212e-02 Score 99.736 Data time: 0.2543, Forward time: 1.2519, Backward time: 1.2481, DDP time: 0.0042, Total iter time: 2.6540 [2021-07-09 06:58:02,745][root][INFO] - ===> Epoch[10851](54255/5): Loss 0.0067, Sem 0.0067 LR: 1.211e-02 Score 99.762 Data time: 0.2266, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7704 [2021-07-09 06:58:16,536][root][INFO] - ===> Epoch[10852](54260/5): Loss 0.0068, Sem 0.0068 LR: 1.210e-02 Score 99.756 Data time: 0.1817, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7221 [2021-07-09 06:58:30,200][root][INFO] - ===> Epoch[10853](54265/5): Loss 0.0074, Sem 0.0074 LR: 1.209e-02 Score 99.737 Data time: 0.2111, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6892 [2021-07-09 06:58:44,370][root][INFO] - ===> Epoch[10854](54270/5): Loss 0.0069, Sem 0.0069 LR: 1.208e-02 Score 99.753 Data time: 0.1937, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7887 [2021-07-09 06:58:58,612][root][INFO] - ===> Epoch[10855](54275/5): Loss 0.0067, Sem 0.0067 LR: 1.207e-02 Score 99.761 Data time: 0.2041, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8063 [2021-07-09 06:59:13,626][root][INFO] - ===> Epoch[10856](54280/5): Loss 0.0068, Sem 0.0068 LR: 1.206e-02 Score 99.756 Data time: 0.2274, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.9485 [2021-07-09 06:59:27,739][root][INFO] - ===> Epoch[10857](54285/5): Loss 0.0068, Sem 0.0068 LR: 1.205e-02 Score 99.760 Data time: 0.2642, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7721 [2021-07-09 06:59:41,682][root][INFO] - ===> Epoch[10858](54290/5): Loss 0.0067, Sem 0.0067 LR: 1.204e-02 Score 99.759 Data time: 0.1491, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7584 [2021-07-09 06:59:55,864][root][INFO] - ===> Epoch[10859](54295/5): Loss 0.0068, Sem 0.0068 LR: 1.203e-02 Score 99.758 Data time: 0.2160, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7870 [2021-07-09 07:00:11,171][root][INFO] - ===> Epoch[10860](54300/5): Loss 0.0069, Sem 0.0069 LR: 1.202e-02 Score 99.753 Data time: 0.2786, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 3.0160 [2021-07-09 07:00:24,770][root][INFO] - ===> Epoch[10861](54305/5): Loss 0.0065, Sem 0.0065 LR: 1.201e-02 Score 99.766 Data time: 0.1993, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6773 [2021-07-09 07:00:38,525][root][INFO] - ===> Epoch[10862](54310/5): Loss 0.0065, Sem 0.0065 LR: 1.200e-02 Score 99.767 Data time: 0.2479, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7000 [2021-07-09 07:00:51,914][root][INFO] - ===> Epoch[10863](54315/5): Loss 0.0071, Sem 0.0071 LR: 1.199e-02 Score 99.748 Data time: 0.2552, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6330 [2021-07-09 07:01:06,594][root][INFO] - ===> Epoch[10864](54320/5): Loss 0.0068, Sem 0.0068 LR: 1.199e-02 Score 99.755 Data time: 0.1977, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8857 [2021-07-09 07:01:20,329][root][INFO] - ===> Epoch[10865](54325/5): Loss 0.0065, Sem 0.0065 LR: 1.198e-02 Score 99.767 Data time: 0.2061, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7101 [2021-07-09 07:01:34,586][root][INFO] - ===> Epoch[10866](54330/5): Loss 0.0066, Sem 0.0066 LR: 1.197e-02 Score 99.766 Data time: 0.1739, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8158 [2021-07-09 07:01:48,090][root][INFO] - ===> Epoch[10867](54335/5): Loss 0.0069, Sem 0.0069 LR: 1.196e-02 Score 99.752 Data time: 0.1907, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6472 [2021-07-09 07:02:02,227][root][INFO] - ===> Epoch[10868](54340/5): Loss 0.0069, Sem 0.0069 LR: 1.195e-02 Score 99.751 Data time: 0.1773, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7877 [2021-07-09 07:02:16,206][root][INFO] - ===> Epoch[10869](54345/5): Loss 0.0066, Sem 0.0066 LR: 1.194e-02 Score 99.766 Data time: 0.1858, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7495 [2021-07-09 07:02:30,661][root][INFO] - ===> Epoch[10870](54350/5): Loss 0.0065, Sem 0.0065 LR: 1.193e-02 Score 99.767 Data time: 0.2057, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8503 [2021-07-09 07:02:44,727][root][INFO] - ===> Epoch[10871](54355/5): Loss 0.0067, Sem 0.0067 LR: 1.192e-02 Score 99.761 Data time: 0.2464, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7640 [2021-07-09 07:02:58,082][root][INFO] - ===> Epoch[10872](54360/5): Loss 0.0070, Sem 0.0070 LR: 1.191e-02 Score 99.751 Data time: 0.2430, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6220 [2021-07-09 07:03:12,935][root][INFO] - ===> Epoch[10873](54365/5): Loss 0.0068, Sem 0.0068 LR: 1.190e-02 Score 99.755 Data time: 0.1862, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.9318 [2021-07-09 07:03:27,777][root][INFO] - ===> Epoch[10874](54370/5): Loss 0.0065, Sem 0.0065 LR: 1.189e-02 Score 99.766 Data time: 0.2357, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.9229 [2021-07-09 07:03:41,814][root][INFO] - ===> Epoch[10875](54375/5): Loss 0.0066, Sem 0.0066 LR: 1.188e-02 Score 99.765 Data time: 0.2253, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7575 [2021-07-09 07:03:56,084][root][INFO] - ===> Epoch[10876](54380/5): Loss 0.0068, Sem 0.0068 LR: 1.187e-02 Score 99.758 Data time: 0.3144, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8033 [2021-07-09 07:04:10,173][root][INFO] - ===> Epoch[10877](54385/5): Loss 0.0067, Sem 0.0067 LR: 1.186e-02 Score 99.763 Data time: 0.2975, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7744 [2021-07-09 07:04:24,723][root][INFO] - ===> Epoch[10878](54390/5): Loss 0.0068, Sem 0.0068 LR: 1.185e-02 Score 99.758 Data time: 0.2222, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8572 [2021-07-09 07:04:38,991][root][INFO] - ===> Epoch[10879](54395/5): Loss 0.0067, Sem 0.0067 LR: 1.184e-02 Score 99.759 Data time: 0.2108, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8114 [2021-07-09 07:04:53,243][root][INFO] - ===> Epoch[10880](54400/5): Loss 0.0070, Sem 0.0070 LR: 1.183e-02 Score 99.751 Data time: 0.2636, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7951 [2021-07-09 07:04:53,244][root][INFO] - ===> Start testing [2021-07-09 07:05:32,689][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1303 Loss 0.439 (AVG: 0.623) Score 91.297 (AVG: 91.027) mIOU 71.666 mAP 82.349 mAcc 77.868 IOU: 63.983 0.000 79.855 75.281 94.808 91.682 32.882 85.290 98.563 84.204 80.482 86.170 58.458 mAP: 82.500 2.559 87.805 85.860 98.311 97.962 52.565 97.927 99.614 94.027 90.360 91.643 89.405 mAcc: 79.809 0.000 83.517 83.415 96.402 97.625 35.460 95.149 99.370 93.035 90.050 96.337 62.119 [2021-07-09 07:05:32,690][root][INFO] - Finished test. Elapsed time: 39.4446 [2021-07-09 07:05:33,875][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 07:05:46,568][root][INFO] - ===> Epoch[10881](54405/5): Loss 0.0070, Sem 0.0070 LR: 1.182e-02 Score 99.753 Data time: 0.1931, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.4933 [2021-07-09 07:06:00,252][root][INFO] - ===> Epoch[10882](54410/5): Loss 0.0066, Sem 0.0066 LR: 1.181e-02 Score 99.762 Data time: 0.2428, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6884 [2021-07-09 07:06:15,244][root][INFO] - ===> Epoch[10883](54415/5): Loss 0.0069, Sem 0.0069 LR: 1.180e-02 Score 99.755 Data time: 0.2408, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.9403 [2021-07-09 07:06:29,686][root][INFO] - ===> Epoch[10884](54420/5): Loss 0.0065, Sem 0.0065 LR: 1.180e-02 Score 99.769 Data time: 0.1862, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8443 [2021-07-09 07:06:43,304][root][INFO] - ===> Epoch[10885](54425/5): Loss 0.0065, Sem 0.0065 LR: 1.179e-02 Score 99.767 Data time: 0.2266, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6635 [2021-07-09 07:06:57,009][root][INFO] - ===> Epoch[10886](54430/5): Loss 0.0069, Sem 0.0069 LR: 1.178e-02 Score 99.753 Data time: 0.2113, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6943 [2021-07-09 07:07:10,999][root][INFO] - ===> Epoch[10887](54435/5): Loss 0.0067, Sem 0.0067 LR: 1.177e-02 Score 99.760 Data time: 0.1963, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7463 [2021-07-09 07:07:24,850][root][INFO] - ===> Epoch[10888](54440/5): Loss 0.0065, Sem 0.0065 LR: 1.176e-02 Score 99.768 Data time: 0.2334, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7228 [2021-07-09 07:07:38,976][root][INFO] - ===> Epoch[10889](54445/5): Loss 0.0064, Sem 0.0064 LR: 1.175e-02 Score 99.769 Data time: 0.1629, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7882 [2021-07-09 07:07:53,461][root][INFO] - ===> Epoch[10890](54450/5): Loss 0.0066, Sem 0.0066 LR: 1.174e-02 Score 99.765 Data time: 0.1994, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8362 [2021-07-09 07:08:09,192][root][INFO] - ===> Epoch[10891](54455/5): Loss 0.0066, Sem 0.0066 LR: 1.173e-02 Score 99.763 Data time: 0.1826, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 3.1081 [2021-07-09 07:08:22,628][root][INFO] - ===> Epoch[10892](54460/5): Loss 0.0069, Sem 0.0069 LR: 1.172e-02 Score 99.753 Data time: 0.2320, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.6396 [2021-07-09 07:08:36,175][root][INFO] - ===> Epoch[10893](54465/5): Loss 0.0070, Sem 0.0070 LR: 1.171e-02 Score 99.752 Data time: 0.2206, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6633 [2021-07-09 07:08:50,282][root][INFO] - ===> Epoch[10894](54470/5): Loss 0.0068, Sem 0.0068 LR: 1.170e-02 Score 99.755 Data time: 0.2521, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7828 [2021-07-09 07:09:05,363][root][INFO] - ===> Epoch[10895](54475/5): Loss 0.0068, Sem 0.0068 LR: 1.169e-02 Score 99.757 Data time: 0.2481, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.9549 [2021-07-09 07:09:19,001][root][INFO] - ===> Epoch[10896](54480/5): Loss 0.0067, Sem 0.0067 LR: 1.168e-02 Score 99.760 Data time: 0.2109, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6766 [2021-07-09 07:09:33,775][root][INFO] - ===> Epoch[10897](54485/5): Loss 0.0066, Sem 0.0066 LR: 1.167e-02 Score 99.765 Data time: 0.2197, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.8991 [2021-07-09 07:09:48,024][root][INFO] - ===> Epoch[10898](54490/5): Loss 0.0066, Sem 0.0066 LR: 1.166e-02 Score 99.763 Data time: 0.2241, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.8045 [2021-07-09 07:10:01,349][root][INFO] - ===> Epoch[10899](54495/5): Loss 0.0068, Sem 0.0068 LR: 1.165e-02 Score 99.757 Data time: 0.2422, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6103 [2021-07-09 07:10:15,620][root][INFO] - ===> Epoch[10900](54500/5): Loss 0.0067, Sem 0.0067 LR: 1.164e-02 Score 99.764 Data time: 0.2263, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.8131 [2021-07-09 07:10:29,861][root][INFO] - ===> Epoch[10901](54505/5): Loss 0.0067, Sem 0.0067 LR: 1.163e-02 Score 99.762 Data time: 0.1985, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8077 [2021-07-09 07:10:43,872][root][INFO] - ===> Epoch[10902](54510/5): Loss 0.0067, Sem 0.0067 LR: 1.162e-02 Score 99.763 Data time: 0.2639, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7640 [2021-07-09 07:10:57,680][root][INFO] - ===> Epoch[10903](54515/5): Loss 0.0064, Sem 0.0064 LR: 1.161e-02 Score 99.772 Data time: 0.1977, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7176 [2021-07-09 07:11:11,427][root][INFO] - ===> Epoch[10904](54520/5): Loss 0.0067, Sem 0.0067 LR: 1.160e-02 Score 99.763 Data time: 0.1741, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.6981 [2021-07-09 07:11:24,580][root][INFO] - ===> Epoch[10905](54525/5): Loss 0.0068, Sem 0.0068 LR: 1.160e-02 Score 99.756 Data time: 0.1891, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.5798 [2021-07-09 07:11:38,355][root][INFO] - ===> Epoch[10906](54530/5): Loss 0.0066, Sem 0.0066 LR: 1.159e-02 Score 99.763 Data time: 0.1720, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7092 [2021-07-09 07:11:52,720][root][INFO] - ===> Epoch[10907](54535/5): Loss 0.0067, Sem 0.0067 LR: 1.158e-02 Score 99.762 Data time: 0.1941, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.8323 [2021-07-09 07:12:06,238][root][INFO] - ===> Epoch[10908](54540/5): Loss 0.0068, Sem 0.0068 LR: 1.157e-02 Score 99.756 Data time: 0.2470, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.6539 [2021-07-09 07:12:20,230][root][INFO] - ===> Epoch[10909](54545/5): Loss 0.0068, Sem 0.0068 LR: 1.156e-02 Score 99.758 Data time: 0.2350, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7449 [2021-07-09 07:12:34,073][root][INFO] - ===> Epoch[10910](54550/5): Loss 0.0068, Sem 0.0068 LR: 1.155e-02 Score 99.759 Data time: 0.2504, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7141 [2021-07-09 07:12:48,461][root][INFO] - ===> Epoch[10911](54555/5): Loss 0.0068, Sem 0.0068 LR: 1.154e-02 Score 99.758 Data time: 0.1896, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8332 [2021-07-09 07:13:01,650][root][INFO] - ===> Epoch[10912](54560/5): Loss 0.0070, Sem 0.0070 LR: 1.153e-02 Score 99.751 Data time: 0.1588, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.6014 [2021-07-09 07:13:15,350][root][INFO] - ===> Epoch[10913](54565/5): Loss 0.0068, Sem 0.0068 LR: 1.152e-02 Score 99.759 Data time: 0.1981, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.6980 [2021-07-09 07:13:30,007][root][INFO] - ===> Epoch[10914](54570/5): Loss 0.0067, Sem 0.0067 LR: 1.151e-02 Score 99.760 Data time: 0.1900, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.8878 [2021-07-09 07:13:43,555][root][INFO] - ===> Epoch[10915](54575/5): Loss 0.0069, Sem 0.0069 LR: 1.150e-02 Score 99.752 Data time: 0.2562, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.6610 [2021-07-09 07:13:58,091][root][INFO] - ===> Epoch[10916](54580/5): Loss 0.0066, Sem 0.0066 LR: 1.149e-02 Score 99.761 Data time: 0.1450, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.8758 [2021-07-09 07:14:11,960][root][INFO] - ===> Epoch[10917](54585/5): Loss 0.0071, Sem 0.0071 LR: 1.148e-02 Score 99.746 Data time: 0.2280, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7239 [2021-07-09 07:14:25,818][root][INFO] - ===> Epoch[10918](54590/5): Loss 0.0069, Sem 0.0069 LR: 1.147e-02 Score 99.755 Data time: 0.2287, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7169 [2021-07-09 07:14:39,786][root][INFO] - ===> Epoch[10919](54595/5): Loss 0.0065, Sem 0.0065 LR: 1.146e-02 Score 99.766 Data time: 0.2264, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7457 [2021-07-09 07:14:54,267][root][INFO] - ===> Epoch[10920](54600/5): Loss 0.0068, Sem 0.0068 LR: 1.145e-02 Score 99.758 Data time: 0.2443, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.8506 [2021-07-09 07:14:54,268][root][INFO] - ===> Start testing [2021-07-09 07:15:32,360][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1208 Loss 0.484 (AVG: 0.627) Score 90.841 (AVG: 91.065) mIOU 71.773 mAP 82.613 mAcc 78.201 IOU: 64.117 0.000 80.115 75.232 94.947 91.366 35.416 84.879 98.591 84.283 79.659 86.399 58.049 mAP: 82.873 2.390 88.071 85.781 98.199 97.976 54.893 97.870 99.620 94.486 90.156 92.314 89.338 mAcc: 79.436 0.000 83.763 83.171 96.593 97.550 39.034 95.479 99.380 94.011 90.393 96.329 61.476 [2021-07-09 07:15:32,360][root][INFO] - Finished test. Elapsed time: 38.0907 [2021-07-09 07:15:33,467][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 07:15:46,546][root][INFO] - ===> Epoch[10921](54605/5): Loss 0.0067, Sem 0.0067 LR: 1.144e-02 Score 99.764 Data time: 0.1897, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.5704 [2021-07-09 07:16:01,722][root][INFO] - ===> Epoch[10922](54610/5): Loss 0.0068, Sem 0.0068 LR: 1.143e-02 Score 99.760 Data time: 0.2627, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.9928 [2021-07-09 07:16:15,259][root][INFO] - ===> Epoch[10923](54615/5): Loss 0.0067, Sem 0.0067 LR: 1.142e-02 Score 99.761 Data time: 0.2538, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.6633 [2021-07-09 07:16:29,189][root][INFO] - ===> Epoch[10924](54620/5): Loss 0.0067, Sem 0.0067 LR: 1.141e-02 Score 99.759 Data time: 0.2674, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7354 [2021-07-09 07:16:43,303][root][INFO] - ===> Epoch[10925](54625/5): Loss 0.0068, Sem 0.0068 LR: 1.140e-02 Score 99.756 Data time: 0.2827, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7729 [2021-07-09 07:16:56,808][root][INFO] - ===> Epoch[10926](54630/5): Loss 0.0069, Sem 0.0069 LR: 1.139e-02 Score 99.755 Data time: 0.3295, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6610 [2021-07-09 07:17:11,936][root][INFO] - ===> Epoch[10927](54635/5): Loss 0.0066, Sem 0.0066 LR: 1.139e-02 Score 99.765 Data time: 0.2852, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.9825 [2021-07-09 07:17:25,592][root][INFO] - ===> Epoch[10928](54640/5): Loss 0.0069, Sem 0.0069 LR: 1.138e-02 Score 99.754 Data time: 0.2809, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6859 [2021-07-09 07:17:39,442][root][INFO] - ===> Epoch[10929](54645/5): Loss 0.0069, Sem 0.0069 LR: 1.137e-02 Score 99.754 Data time: 0.2769, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7267 [2021-07-09 07:17:53,770][root][INFO] - ===> Epoch[10930](54650/5): Loss 0.0066, Sem 0.0066 LR: 1.136e-02 Score 99.765 Data time: 0.2475, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8250 [2021-07-09 07:18:07,916][root][INFO] - ===> Epoch[10931](54655/5): Loss 0.0066, Sem 0.0066 LR: 1.135e-02 Score 99.764 Data time: 0.2785, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7818 [2021-07-09 07:18:22,889][root][INFO] - ===> Epoch[10932](54660/5): Loss 0.0063, Sem 0.0063 LR: 1.134e-02 Score 99.772 Data time: 0.2762, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.9407 [2021-07-09 07:18:36,749][root][INFO] - ===> Epoch[10933](54665/5): Loss 0.0067, Sem 0.0067 LR: 1.133e-02 Score 99.762 Data time: 0.2438, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7337 [2021-07-09 07:18:51,092][root][INFO] - ===> Epoch[10934](54670/5): Loss 0.0066, Sem 0.0066 LR: 1.132e-02 Score 99.766 Data time: 0.2262, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8191 [2021-07-09 07:19:05,342][root][INFO] - ===> Epoch[10935](54675/5): Loss 0.0069, Sem 0.0069 LR: 1.131e-02 Score 99.755 Data time: 0.2132, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8040 [2021-07-09 07:19:19,333][root][INFO] - ===> Epoch[10936](54680/5): Loss 0.0066, Sem 0.0066 LR: 1.130e-02 Score 99.764 Data time: 0.2612, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7625 [2021-07-09 07:19:33,925][root][INFO] - ===> Epoch[10937](54685/5): Loss 0.0069, Sem 0.0069 LR: 1.129e-02 Score 99.753 Data time: 0.2562, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8697 [2021-07-09 07:19:48,167][root][INFO] - ===> Epoch[10938](54690/5): Loss 0.0065, Sem 0.0065 LR: 1.128e-02 Score 99.767 Data time: 0.2135, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8009 [2021-07-09 07:20:01,796][root][INFO] - ===> Epoch[10939](54695/5): Loss 0.0067, Sem 0.0067 LR: 1.127e-02 Score 99.762 Data time: 0.2423, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6889 [2021-07-09 07:20:15,545][root][INFO] - ===> Epoch[10940](54700/5): Loss 0.0066, Sem 0.0066 LR: 1.126e-02 Score 99.764 Data time: 0.2156, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6969 [2021-07-09 07:20:30,101][root][INFO] - ===> Epoch[10941](54705/5): Loss 0.0066, Sem 0.0066 LR: 1.125e-02 Score 99.762 Data time: 0.2009, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8655 [2021-07-09 07:20:43,408][root][INFO] - ===> Epoch[10942](54710/5): Loss 0.0068, Sem 0.0068 LR: 1.124e-02 Score 99.755 Data time: 0.2303, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6184 [2021-07-09 07:20:57,185][root][INFO] - ===> Epoch[10943](54715/5): Loss 0.0069, Sem 0.0069 LR: 1.123e-02 Score 99.758 Data time: 0.2622, Forward time: 1.2520, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7132 [2021-07-09 07:21:11,074][root][INFO] - ===> Epoch[10944](54720/5): Loss 0.0068, Sem 0.0068 LR: 1.122e-02 Score 99.755 Data time: 0.2436, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7263 [2021-07-09 07:21:24,553][root][INFO] - ===> Epoch[10945](54725/5): Loss 0.0068, Sem 0.0068 LR: 1.121e-02 Score 99.757 Data time: 0.1900, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6509 [2021-07-09 07:21:38,946][root][INFO] - ===> Epoch[10946](54730/5): Loss 0.0066, Sem 0.0066 LR: 1.120e-02 Score 99.765 Data time: 0.2059, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8285 [2021-07-09 07:21:52,957][root][INFO] - ===> Epoch[10947](54735/5): Loss 0.0068, Sem 0.0068 LR: 1.119e-02 Score 99.757 Data time: 0.1456, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7654 [2021-07-09 07:22:06,693][root][INFO] - ===> Epoch[10948](54740/5): Loss 0.0066, Sem 0.0066 LR: 1.118e-02 Score 99.767 Data time: 0.1985, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.6969 [2021-07-09 07:22:20,741][root][INFO] - ===> Epoch[10949](54745/5): Loss 0.0069, Sem 0.0069 LR: 1.117e-02 Score 99.751 Data time: 0.2137, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7591 [2021-07-09 07:22:35,089][root][INFO] - ===> Epoch[10950](54750/5): Loss 0.0064, Sem 0.0064 LR: 1.117e-02 Score 99.770 Data time: 0.1871, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8186 [2021-07-09 07:22:49,247][root][INFO] - ===> Epoch[10951](54755/5): Loss 0.0070, Sem 0.0070 LR: 1.116e-02 Score 99.752 Data time: 0.2650, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7732 [2021-07-09 07:23:03,543][root][INFO] - ===> Epoch[10952](54760/5): Loss 0.0065, Sem 0.0065 LR: 1.115e-02 Score 99.766 Data time: 0.2649, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.8081 [2021-07-09 07:23:18,202][root][INFO] - ===> Epoch[10953](54765/5): Loss 0.0066, Sem 0.0066 LR: 1.114e-02 Score 99.761 Data time: 0.2109, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.8876 [2021-07-09 07:23:33,036][root][INFO] - ===> Epoch[10954](54770/5): Loss 0.0066, Sem 0.0066 LR: 1.113e-02 Score 99.763 Data time: 0.2098, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.9150 [2021-07-09 07:23:46,407][root][INFO] - ===> Epoch[10955](54775/5): Loss 0.0067, Sem 0.0067 LR: 1.112e-02 Score 99.760 Data time: 0.1771, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.6353 [2021-07-09 07:23:59,740][root][INFO] - ===> Epoch[10956](54780/5): Loss 0.0066, Sem 0.0066 LR: 1.111e-02 Score 99.765 Data time: 0.2255, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.6245 [2021-07-09 07:24:13,791][root][INFO] - ===> Epoch[10957](54785/5): Loss 0.0067, Sem 0.0067 LR: 1.110e-02 Score 99.760 Data time: 0.2193, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7668 [2021-07-09 07:24:27,304][root][INFO] - ===> Epoch[10958](54790/5): Loss 0.0066, Sem 0.0066 LR: 1.109e-02 Score 99.766 Data time: 0.1814, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.6592 [2021-07-09 07:24:41,173][root][INFO] - ===> Epoch[10959](54795/5): Loss 0.0066, Sem 0.0066 LR: 1.108e-02 Score 99.764 Data time: 0.2294, Forward time: 1.2519, Backward time: 1.2482, DDP time: 0.0042, Total iter time: 2.7256 [2021-07-09 07:24:55,013][root][INFO] - ===> Epoch[10960](54800/5): Loss 0.0066, Sem 0.0066 LR: 1.107e-02 Score 99.766 Data time: 0.2102, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7307 [2021-07-09 07:24:55,015][root][INFO] - ===> Start testing [2021-07-09 07:25:35,116][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1645 Loss 0.478 (AVG: 0.639) Score 91.650 (AVG: 90.977) mIOU 71.639 mAP 82.210 mAcc 77.792 IOU: 63.948 0.000 81.384 75.084 94.945 91.112 32.299 84.082 98.601 86.187 80.065 86.140 57.464 mAP: 82.641 2.376 87.211 85.745 98.159 97.950 52.086 97.725 99.611 94.303 90.115 91.996 88.817 mAcc: 79.580 0.000 83.822 82.895 96.653 97.833 34.973 95.567 99.341 93.262 90.102 96.402 60.862 [2021-07-09 07:25:35,116][root][INFO] - Finished test. Elapsed time: 40.1002 [2021-07-09 07:25:36,312][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 07:25:50,068][root][INFO] - ===> Epoch[10961](54805/5): Loss 0.0064, Sem 0.0064 LR: 1.106e-02 Score 99.772 Data time: 0.1781, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7109 [2021-07-09 07:26:04,822][root][INFO] - ===> Epoch[10962](54810/5): Loss 0.0068, Sem 0.0068 LR: 1.105e-02 Score 99.757 Data time: 0.2481, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.9003 [2021-07-09 07:26:18,187][root][INFO] - ===> Epoch[10963](54815/5): Loss 0.0066, Sem 0.0066 LR: 1.104e-02 Score 99.763 Data time: 0.2172, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.6322 [2021-07-09 07:26:32,048][root][INFO] - ===> Epoch[10964](54820/5): Loss 0.0066, Sem 0.0066 LR: 1.103e-02 Score 99.763 Data time: 0.1696, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7319 [2021-07-09 07:26:45,696][root][INFO] - ===> Epoch[10965](54825/5): Loss 0.0069, Sem 0.0069 LR: 1.102e-02 Score 99.756 Data time: 0.2366, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.6890 [2021-07-09 07:27:00,309][root][INFO] - ===> Epoch[10966](54830/5): Loss 0.0068, Sem 0.0068 LR: 1.101e-02 Score 99.755 Data time: 0.2069, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.8730 [2021-07-09 07:27:13,988][root][INFO] - ===> Epoch[10967](54835/5): Loss 0.0066, Sem 0.0066 LR: 1.100e-02 Score 99.765 Data time: 0.2899, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.6748 [2021-07-09 07:27:28,357][root][INFO] - ===> Epoch[10968](54840/5): Loss 0.0066, Sem 0.0066 LR: 1.099e-02 Score 99.763 Data time: 0.2358, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.8317 [2021-07-09 07:27:42,936][root][INFO] - ===> Epoch[10969](54845/5): Loss 0.0068, Sem 0.0068 LR: 1.098e-02 Score 99.756 Data time: 0.1630, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.8778 [2021-07-09 07:27:57,459][root][INFO] - ===> Epoch[10970](54850/5): Loss 0.0065, Sem 0.0065 LR: 1.097e-02 Score 99.767 Data time: 0.1822, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.8572 [2021-07-09 07:28:11,854][root][INFO] - ===> Epoch[10971](54855/5): Loss 0.0068, Sem 0.0068 LR: 1.096e-02 Score 99.755 Data time: 0.1941, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.8346 [2021-07-09 07:28:25,991][root][INFO] - ===> Epoch[10972](54860/5): Loss 0.0067, Sem 0.0067 LR: 1.095e-02 Score 99.761 Data time: 0.2510, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7847 [2021-07-09 07:28:39,746][root][INFO] - ===> Epoch[10973](54865/5): Loss 0.0068, Sem 0.0068 LR: 1.095e-02 Score 99.757 Data time: 0.2297, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7010 [2021-07-09 07:28:53,768][root][INFO] - ===> Epoch[10974](54870/5): Loss 0.0068, Sem 0.0068 LR: 1.094e-02 Score 99.759 Data time: 0.2561, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7555 [2021-07-09 07:29:07,265][root][INFO] - ===> Epoch[10975](54875/5): Loss 0.0064, Sem 0.0064 LR: 1.093e-02 Score 99.769 Data time: 0.1888, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.6518 [2021-07-09 07:29:22,343][root][INFO] - ===> Epoch[10976](54880/5): Loss 0.0071, Sem 0.0071 LR: 1.092e-02 Score 99.747 Data time: 0.2220, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.9720 [2021-07-09 07:29:35,798][root][INFO] - ===> Epoch[10977](54885/5): Loss 0.0065, Sem 0.0065 LR: 1.091e-02 Score 99.768 Data time: 0.1654, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.6464 [2021-07-09 07:29:49,913][root][INFO] - ===> Epoch[10978](54890/5): Loss 0.0066, Sem 0.0066 LR: 1.090e-02 Score 99.764 Data time: 0.2493, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7751 [2021-07-09 07:30:04,350][root][INFO] - ===> Epoch[10979](54895/5): Loss 0.0067, Sem 0.0067 LR: 1.089e-02 Score 99.762 Data time: 0.2418, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.8497 [2021-07-09 07:30:19,892][root][INFO] - ===> Epoch[10980](54900/5): Loss 0.0067, Sem 0.0067 LR: 1.088e-02 Score 99.759 Data time: 0.3017, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 3.0612 [2021-07-09 07:30:34,436][root][INFO] - ===> Epoch[10981](54905/5): Loss 0.0069, Sem 0.0069 LR: 1.087e-02 Score 99.756 Data time: 0.2444, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.8763 [2021-07-09 07:30:49,215][root][INFO] - ===> Epoch[10982](54910/5): Loss 0.0064, Sem 0.0064 LR: 1.086e-02 Score 99.772 Data time: 0.3101, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.9107 [2021-07-09 07:31:03,937][root][INFO] - ===> Epoch[10983](54915/5): Loss 0.0063, Sem 0.0063 LR: 1.085e-02 Score 99.772 Data time: 0.2637, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.9027 [2021-07-09 07:31:17,882][root][INFO] - ===> Epoch[10984](54920/5): Loss 0.0070, Sem 0.0070 LR: 1.084e-02 Score 99.750 Data time: 0.3107, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7275 [2021-07-09 07:31:31,974][root][INFO] - ===> Epoch[10985](54925/5): Loss 0.0065, Sem 0.0065 LR: 1.083e-02 Score 99.767 Data time: 0.2557, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7742 [2021-07-09 07:31:46,254][root][INFO] - ===> Epoch[10986](54930/5): Loss 0.0066, Sem 0.0066 LR: 1.082e-02 Score 99.764 Data time: 0.2243, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.8104 [2021-07-09 07:32:00,078][root][INFO] - ===> Epoch[10987](54935/5): Loss 0.0067, Sem 0.0067 LR: 1.081e-02 Score 99.764 Data time: 0.2244, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7199 [2021-07-09 07:32:14,775][root][INFO] - ===> Epoch[10988](54940/5): Loss 0.0068, Sem 0.0068 LR: 1.080e-02 Score 99.758 Data time: 0.2220, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.8961 [2021-07-09 07:32:28,567][root][INFO] - ===> Epoch[10989](54945/5): Loss 0.0068, Sem 0.0068 LR: 1.079e-02 Score 99.756 Data time: 0.2189, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7098 [2021-07-09 07:32:41,674][root][INFO] - ===> Epoch[10990](54950/5): Loss 0.0070, Sem 0.0070 LR: 1.078e-02 Score 99.750 Data time: 0.2420, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.5729 [2021-07-09 07:32:56,742][root][INFO] - ===> Epoch[10991](54955/5): Loss 0.0068, Sem 0.0068 LR: 1.077e-02 Score 99.758 Data time: 0.1791, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.9715 [2021-07-09 07:33:10,441][root][INFO] - ===> Epoch[10992](54960/5): Loss 0.0068, Sem 0.0068 LR: 1.076e-02 Score 99.756 Data time: 0.2171, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.6979 [2021-07-09 07:33:24,579][root][INFO] - ===> Epoch[10993](54965/5): Loss 0.0069, Sem 0.0069 LR: 1.075e-02 Score 99.751 Data time: 0.2556, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7832 [2021-07-09 07:33:38,298][root][INFO] - ===> Epoch[10994](54970/5): Loss 0.0066, Sem 0.0066 LR: 1.074e-02 Score 99.765 Data time: 0.2052, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.6947 [2021-07-09 07:33:52,112][root][INFO] - ===> Epoch[10995](54975/5): Loss 0.0064, Sem 0.0064 LR: 1.073e-02 Score 99.770 Data time: 0.2245, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7144 [2021-07-09 07:34:06,039][root][INFO] - ===> Epoch[10996](54980/5): Loss 0.0068, Sem 0.0068 LR: 1.072e-02 Score 99.760 Data time: 0.2555, Forward time: 1.2519, Backward time: 1.2483, DDP time: 0.0042, Total iter time: 2.7347 [2021-07-09 07:34:20,839][root][INFO] - ===> Epoch[10997](54985/5): Loss 0.0064, Sem 0.0064 LR: 1.071e-02 Score 99.771 Data time: 0.2252, Forward time: 1.2519, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.9065 [2021-07-09 07:34:34,685][root][INFO] - ===> Epoch[10998](54990/5): Loss 0.0070, Sem 0.0070 LR: 1.071e-02 Score 99.750 Data time: 0.2022, Forward time: 1.2519, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7175 [2021-07-09 07:34:48,232][root][INFO] - ===> Epoch[10999](54995/5): Loss 0.0068, Sem 0.0068 LR: 1.070e-02 Score 99.759 Data time: 0.1800, Forward time: 1.2519, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.6663 [2021-07-09 07:35:02,709][root][INFO] - ===> Epoch[11000](55000/5): Loss 0.0065, Sem 0.0065 LR: 1.069e-02 Score 99.766 Data time: 0.2951, Forward time: 1.2519, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.8344 [2021-07-09 07:35:02,711][root][INFO] - ===> Start testing [2021-07-09 07:35:48,083][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1267 Loss 0.462 (AVG: 0.632) Score 91.332 (AVG: 91.063) mIOU 71.687 mAP 82.258 mAcc 78.030 IOU: 64.028 0.000 80.178 75.269 95.090 91.412 34.601 84.077 98.595 84.290 80.101 86.423 57.869 mAP: 82.361 2.417 87.422 85.469 98.363 97.938 53.744 97.862 99.622 93.476 90.137 91.608 88.930 mAcc: 79.819 0.000 84.198 83.435 96.740 97.692 37.563 95.526 99.382 92.423 89.715 96.106 61.794 [2021-07-09 07:35:48,084][root][INFO] - Finished test. Elapsed time: 45.3717 [2021-07-09 07:35:49,115][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 07:36:02,705][root][INFO] - ===> Epoch[11001](55005/5): Loss 0.0071, Sem 0.0071 LR: 1.068e-02 Score 99.744 Data time: 0.2060, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.6711 [2021-07-09 07:36:16,604][root][INFO] - ===> Epoch[11002](55010/5): Loss 0.0067, Sem 0.0067 LR: 1.067e-02 Score 99.765 Data time: 0.1992, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7386 [2021-07-09 07:36:30,496][root][INFO] - ===> Epoch[11003](55015/5): Loss 0.0070, Sem 0.0070 LR: 1.066e-02 Score 99.748 Data time: 0.2308, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7199 [2021-07-09 07:36:44,962][root][INFO] - ===> Epoch[11004](55020/5): Loss 0.0069, Sem 0.0069 LR: 1.065e-02 Score 99.752 Data time: 0.2283, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.8470 [2021-07-09 07:36:59,402][root][INFO] - ===> Epoch[11005](55025/5): Loss 0.0068, Sem 0.0068 LR: 1.064e-02 Score 99.756 Data time: 0.2309, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.8434 [2021-07-09 07:37:13,785][root][INFO] - ===> Epoch[11006](55030/5): Loss 0.0070, Sem 0.0070 LR: 1.063e-02 Score 99.752 Data time: 0.2239, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.8222 [2021-07-09 07:37:27,878][root][INFO] - ===> Epoch[11007](55035/5): Loss 0.0067, Sem 0.0067 LR: 1.062e-02 Score 99.759 Data time: 0.1834, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7872 [2021-07-09 07:37:42,008][root][INFO] - ===> Epoch[11008](55040/5): Loss 0.0065, Sem 0.0065 LR: 1.061e-02 Score 99.766 Data time: 0.2260, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7770 [2021-07-09 07:37:55,204][root][INFO] - ===> Epoch[11009](55045/5): Loss 0.0065, Sem 0.0065 LR: 1.060e-02 Score 99.766 Data time: 0.2715, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.5962 [2021-07-09 07:38:09,241][root][INFO] - ===> Epoch[11010](55050/5): Loss 0.0067, Sem 0.0067 LR: 1.059e-02 Score 99.762 Data time: 0.2386, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7603 [2021-07-09 07:38:23,317][root][INFO] - ===> Epoch[11011](55055/5): Loss 0.0069, Sem 0.0069 LR: 1.058e-02 Score 99.756 Data time: 0.2374, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7733 [2021-07-09 07:38:36,987][root][INFO] - ===> Epoch[11012](55060/5): Loss 0.0071, Sem 0.0071 LR: 1.057e-02 Score 99.749 Data time: 0.1755, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.6982 [2021-07-09 07:38:51,045][root][INFO] - ===> Epoch[11013](55065/5): Loss 0.0064, Sem 0.0064 LR: 1.056e-02 Score 99.770 Data time: 0.2058, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7624 [2021-07-09 07:39:05,451][root][INFO] - ===> Epoch[11014](55070/5): Loss 0.0068, Sem 0.0068 LR: 1.055e-02 Score 99.759 Data time: 0.2183, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.8392 [2021-07-09 07:39:19,785][root][INFO] - ===> Epoch[11015](55075/5): Loss 0.0063, Sem 0.0063 LR: 1.054e-02 Score 99.771 Data time: 0.2112, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.8224 [2021-07-09 07:39:34,197][root][INFO] - ===> Epoch[11016](55080/5): Loss 0.0067, Sem 0.0067 LR: 1.053e-02 Score 99.761 Data time: 0.1589, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8415 [2021-07-09 07:39:48,269][root][INFO] - ===> Epoch[11017](55085/5): Loss 0.0066, Sem 0.0066 LR: 1.052e-02 Score 99.763 Data time: 0.2085, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7659 [2021-07-09 07:40:02,307][root][INFO] - ===> Epoch[11018](55090/5): Loss 0.0066, Sem 0.0066 LR: 1.051e-02 Score 99.766 Data time: 0.2222, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7643 [2021-07-09 07:40:16,495][root][INFO] - ===> Epoch[11019](55095/5): Loss 0.0067, Sem 0.0067 LR: 1.050e-02 Score 99.762 Data time: 0.2526, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7914 [2021-07-09 07:40:30,552][root][INFO] - ===> Epoch[11020](55100/5): Loss 0.0070, Sem 0.0070 LR: 1.049e-02 Score 99.749 Data time: 0.1832, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7667 [2021-07-09 07:40:45,261][root][INFO] - ===> Epoch[11021](55105/5): Loss 0.0067, Sem 0.0067 LR: 1.048e-02 Score 99.765 Data time: 0.2315, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8958 [2021-07-09 07:41:00,191][root][INFO] - ===> Epoch[11022](55110/5): Loss 0.0066, Sem 0.0066 LR: 1.047e-02 Score 99.765 Data time: 0.2436, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9324 [2021-07-09 07:41:14,811][root][INFO] - ===> Epoch[11023](55115/5): Loss 0.0063, Sem 0.0063 LR: 1.046e-02 Score 99.772 Data time: 0.2482, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8789 [2021-07-09 07:41:28,844][root][INFO] - ===> Epoch[11024](55120/5): Loss 0.0068, Sem 0.0068 LR: 1.045e-02 Score 99.756 Data time: 0.2834, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7508 [2021-07-09 07:41:43,021][root][INFO] - ===> Epoch[11025](55125/5): Loss 0.0068, Sem 0.0068 LR: 1.045e-02 Score 99.758 Data time: 0.2049, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7990 [2021-07-09 07:41:56,873][root][INFO] - ===> Epoch[11026](55130/5): Loss 0.0067, Sem 0.0067 LR: 1.044e-02 Score 99.759 Data time: 0.1869, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7175 [2021-07-09 07:42:10,905][root][INFO] - ===> Epoch[11027](55135/5): Loss 0.0066, Sem 0.0066 LR: 1.043e-02 Score 99.766 Data time: 0.1849, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7634 [2021-07-09 07:42:24,937][root][INFO] - ===> Epoch[11028](55140/5): Loss 0.0066, Sem 0.0066 LR: 1.042e-02 Score 99.762 Data time: 0.1846, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7591 [2021-07-09 07:42:38,763][root][INFO] - ===> Epoch[11029](55145/5): Loss 0.0066, Sem 0.0066 LR: 1.041e-02 Score 99.763 Data time: 0.2447, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7090 [2021-07-09 07:42:53,288][root][INFO] - ===> Epoch[11030](55150/5): Loss 0.0066, Sem 0.0066 LR: 1.040e-02 Score 99.767 Data time: 0.2745, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8607 [2021-07-09 07:43:07,555][root][INFO] - ===> Epoch[11031](55155/5): Loss 0.0066, Sem 0.0066 LR: 1.039e-02 Score 99.762 Data time: 0.1879, Forward time: 1.2518, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8065 [2021-07-09 07:43:21,409][root][INFO] - ===> Epoch[11032](55160/5): Loss 0.0069, Sem 0.0069 LR: 1.038e-02 Score 99.756 Data time: 0.2594, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7155 [2021-07-09 07:43:36,751][root][INFO] - ===> Epoch[11033](55165/5): Loss 0.0064, Sem 0.0064 LR: 1.037e-02 Score 99.771 Data time: 0.1702, Forward time: 1.2518, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 3.0226 [2021-07-09 07:43:50,975][root][INFO] - ===> Epoch[11034](55170/5): Loss 0.0068, Sem 0.0068 LR: 1.036e-02 Score 99.756 Data time: 0.3447, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7979 [2021-07-09 07:44:06,145][root][INFO] - ===> Epoch[11035](55175/5): Loss 0.0065, Sem 0.0065 LR: 1.035e-02 Score 99.767 Data time: 0.2483, Forward time: 1.2518, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9948 [2021-07-09 07:44:20,028][root][INFO] - ===> Epoch[11036](55180/5): Loss 0.0068, Sem 0.0068 LR: 1.034e-02 Score 99.755 Data time: 0.2643, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7147 [2021-07-09 07:44:33,820][root][INFO] - ===> Epoch[11037](55185/5): Loss 0.0066, Sem 0.0066 LR: 1.033e-02 Score 99.764 Data time: 0.3253, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7115 [2021-07-09 07:44:47,997][root][INFO] - ===> Epoch[11038](55190/5): Loss 0.0070, Sem 0.0070 LR: 1.032e-02 Score 99.751 Data time: 0.1733, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7844 [2021-07-09 07:45:02,272][root][INFO] - ===> Epoch[11039](55195/5): Loss 0.0067, Sem 0.0067 LR: 1.031e-02 Score 99.761 Data time: 0.1864, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8098 [2021-07-09 07:45:16,269][root][INFO] - ===> Epoch[11040](55200/5): Loss 0.0068, Sem 0.0068 LR: 1.030e-02 Score 99.755 Data time: 0.1731, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7542 [2021-07-09 07:45:16,270][root][INFO] - ===> Start testing [2021-07-09 07:45:54,048][root][INFO] - 68/68: Data time: 0.0016, Iter time: 0.1223 Loss 0.438 (AVG: 0.634) Score 91.724 (AVG: 91.070) mIOU 71.816 mAP 82.325 mAcc 78.021 IOU: 64.022 0.000 80.316 75.064 95.077 91.547 35.564 85.367 98.609 84.241 79.943 86.272 57.582 mAP: 82.197 2.530 87.111 85.166 98.268 97.939 54.311 98.002 99.586 93.879 90.386 91.889 88.963 mAcc: 79.143 0.000 83.520 83.027 96.842 97.583 38.488 95.336 99.405 93.242 90.053 96.483 61.148 [2021-07-09 07:45:54,048][root][INFO] - Finished test. Elapsed time: 37.7766 [2021-07-09 07:45:55,137][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 07:46:08,221][root][INFO] - ===> Epoch[11041](55205/5): Loss 0.0069, Sem 0.0069 LR: 1.029e-02 Score 99.755 Data time: 0.2093, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.5660 [2021-07-09 07:46:22,045][root][INFO] - ===> Epoch[11042](55210/5): Loss 0.0068, Sem 0.0068 LR: 1.028e-02 Score 99.752 Data time: 0.2140, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7216 [2021-07-09 07:46:35,611][root][INFO] - ===> Epoch[11043](55215/5): Loss 0.0064, Sem 0.0064 LR: 1.027e-02 Score 99.768 Data time: 0.1542, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6730 [2021-07-09 07:46:49,636][root][INFO] - ===> Epoch[11044](55220/5): Loss 0.0069, Sem 0.0069 LR: 1.026e-02 Score 99.750 Data time: 0.1932, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7612 [2021-07-09 07:47:03,794][root][INFO] - ===> Epoch[11045](55225/5): Loss 0.0065, Sem 0.0065 LR: 1.025e-02 Score 99.767 Data time: 0.1573, Forward time: 1.2518, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7922 [2021-07-09 07:47:18,171][root][INFO] - ===> Epoch[11046](55230/5): Loss 0.0069, Sem 0.0069 LR: 1.024e-02 Score 99.752 Data time: 0.1946, Forward time: 1.2518, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8283 [2021-07-09 07:47:32,070][root][INFO] - ===> Epoch[11047](55235/5): Loss 0.0069, Sem 0.0069 LR: 1.023e-02 Score 99.755 Data time: 0.1818, Forward time: 1.2518, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7323 [2021-07-09 07:47:45,532][root][INFO] - ===> Epoch[11048](55240/5): Loss 0.0066, Sem 0.0066 LR: 1.022e-02 Score 99.765 Data time: 0.2000, Forward time: 1.2518, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6501 [2021-07-09 07:47:59,039][root][INFO] - ===> Epoch[11049](55245/5): Loss 0.0068, Sem 0.0068 LR: 1.021e-02 Score 99.754 Data time: 0.2711, Forward time: 1.2518, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6524 [2021-07-09 07:48:13,242][root][INFO] - ===> Epoch[11050](55250/5): Loss 0.0063, Sem 0.0063 LR: 1.020e-02 Score 99.774 Data time: 0.1970, Forward time: 1.2518, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7920 [2021-07-09 07:48:27,705][root][INFO] - ===> Epoch[11051](55255/5): Loss 0.0064, Sem 0.0064 LR: 1.019e-02 Score 99.772 Data time: 0.2163, Forward time: 1.2518, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8495 [2021-07-09 07:48:41,822][root][INFO] - ===> Epoch[11052](55260/5): Loss 0.0068, Sem 0.0068 LR: 1.018e-02 Score 99.759 Data time: 0.2420, Forward time: 1.2518, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7681 [2021-07-09 07:48:56,486][root][INFO] - ===> Epoch[11053](55265/5): Loss 0.0065, Sem 0.0065 LR: 1.017e-02 Score 99.767 Data time: 0.1706, Forward time: 1.2518, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8920 [2021-07-09 07:49:09,941][root][INFO] - ===> Epoch[11054](55270/5): Loss 0.0068, Sem 0.0068 LR: 1.017e-02 Score 99.761 Data time: 0.2751, Forward time: 1.2518, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6327 [2021-07-09 07:49:24,849][root][INFO] - ===> Epoch[11055](55275/5): Loss 0.0066, Sem 0.0066 LR: 1.016e-02 Score 99.768 Data time: 0.2323, Forward time: 1.2518, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9297 [2021-07-09 07:49:37,859][root][INFO] - ===> Epoch[11056](55280/5): Loss 0.0068, Sem 0.0068 LR: 1.015e-02 Score 99.756 Data time: 0.2570, Forward time: 1.2518, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.5591 [2021-07-09 07:49:51,072][root][INFO] - ===> Epoch[11057](55285/5): Loss 0.0065, Sem 0.0065 LR: 1.014e-02 Score 99.770 Data time: 0.1880, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6017 [2021-07-09 07:50:05,299][root][INFO] - ===> Epoch[11058](55290/5): Loss 0.0067, Sem 0.0067 LR: 1.013e-02 Score 99.761 Data time: 0.2573, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8022 [2021-07-09 07:50:19,521][root][INFO] - ===> Epoch[11059](55295/5): Loss 0.0064, Sem 0.0064 LR: 1.012e-02 Score 99.770 Data time: 0.2207, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7998 [2021-07-09 07:50:32,965][root][INFO] - ===> Epoch[11060](55300/5): Loss 0.0070, Sem 0.0070 LR: 1.011e-02 Score 99.748 Data time: 0.1766, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6392 [2021-07-09 07:50:46,852][root][INFO] - ===> Epoch[11061](55305/5): Loss 0.0065, Sem 0.0065 LR: 1.010e-02 Score 99.768 Data time: 0.1855, Forward time: 1.2518, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7359 [2021-07-09 07:51:00,523][root][INFO] - ===> Epoch[11062](55310/5): Loss 0.0069, Sem 0.0069 LR: 1.009e-02 Score 99.753 Data time: 0.2031, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6953 [2021-07-09 07:51:14,190][root][INFO] - ===> Epoch[11063](55315/5): Loss 0.0067, Sem 0.0067 LR: 1.008e-02 Score 99.760 Data time: 0.2244, Forward time: 1.2518, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6780 [2021-07-09 07:51:28,188][root][INFO] - ===> Epoch[11064](55320/5): Loss 0.0067, Sem 0.0067 LR: 1.007e-02 Score 99.760 Data time: 0.2037, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7540 [2021-07-09 07:51:42,328][root][INFO] - ===> Epoch[11065](55325/5): Loss 0.0069, Sem 0.0069 LR: 1.006e-02 Score 99.756 Data time: 0.1754, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7811 [2021-07-09 07:51:56,720][root][INFO] - ===> Epoch[11066](55330/5): Loss 0.0065, Sem 0.0065 LR: 1.005e-02 Score 99.769 Data time: 0.2412, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8182 [2021-07-09 07:52:09,987][root][INFO] - ===> Epoch[11067](55335/5): Loss 0.0069, Sem 0.0069 LR: 1.004e-02 Score 99.756 Data time: 0.1869, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6037 [2021-07-09 07:52:24,399][root][INFO] - ===> Epoch[11068](55340/5): Loss 0.0068, Sem 0.0068 LR: 1.003e-02 Score 99.757 Data time: 0.2338, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8323 [2021-07-09 07:52:38,230][root][INFO] - ===> Epoch[11069](55345/5): Loss 0.0069, Sem 0.0069 LR: 1.002e-02 Score 99.753 Data time: 0.2225, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7222 [2021-07-09 07:52:52,038][root][INFO] - ===> Epoch[11070](55350/5): Loss 0.0067, Sem 0.0067 LR: 1.001e-02 Score 99.759 Data time: 0.2032, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7192 [2021-07-09 07:53:06,004][root][INFO] - ===> Epoch[11071](55355/5): Loss 0.0070, Sem 0.0070 LR: 1.000e-02 Score 99.754 Data time: 0.2111, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7454 [2021-07-09 07:53:20,068][root][INFO] - ===> Epoch[11072](55360/5): Loss 0.0067, Sem 0.0067 LR: 9.991e-03 Score 99.760 Data time: 0.2316, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7684 [2021-07-09 07:53:33,789][root][INFO] - ===> Epoch[11073](55365/5): Loss 0.0065, Sem 0.0065 LR: 9.981e-03 Score 99.767 Data time: 0.1876, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6938 [2021-07-09 07:53:47,305][root][INFO] - ===> Epoch[11074](55370/5): Loss 0.0066, Sem 0.0066 LR: 9.972e-03 Score 99.763 Data time: 0.1900, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6565 [2021-07-09 07:54:00,823][root][INFO] - ===> Epoch[11075](55375/5): Loss 0.0071, Sem 0.0071 LR: 9.962e-03 Score 99.748 Data time: 0.1978, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6649 [2021-07-09 07:54:15,468][root][INFO] - ===> Epoch[11076](55380/5): Loss 0.0064, Sem 0.0064 LR: 9.952e-03 Score 99.769 Data time: 0.2736, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8729 [2021-07-09 07:54:29,870][root][INFO] - ===> Epoch[11077](55385/5): Loss 0.0065, Sem 0.0065 LR: 9.943e-03 Score 99.769 Data time: 0.2467, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8343 [2021-07-09 07:54:43,142][root][INFO] - ===> Epoch[11078](55390/5): Loss 0.0069, Sem 0.0069 LR: 9.933e-03 Score 99.750 Data time: 0.2619, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6056 [2021-07-09 07:54:56,649][root][INFO] - ===> Epoch[11079](55395/5): Loss 0.0065, Sem 0.0065 LR: 9.923e-03 Score 99.772 Data time: 0.1826, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6636 [2021-07-09 07:55:10,006][root][INFO] - ===> Epoch[11080](55400/5): Loss 0.0070, Sem 0.0070 LR: 9.913e-03 Score 99.749 Data time: 0.1994, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6376 [2021-07-09 07:55:10,008][root][INFO] - ===> Start testing [2021-07-09 07:55:52,915][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1228 Loss 0.438 (AVG: 0.636) Score 92.045 (AVG: 91.039) mIOU 71.702 mAP 82.114 mAcc 77.929 IOU: 63.990 0.000 80.582 75.141 94.956 91.279 34.287 85.233 98.621 84.423 79.469 86.260 57.879 mAP: 82.515 2.418 86.726 85.252 98.200 97.841 52.995 97.971 99.646 93.203 90.015 91.547 89.154 mAcc: 79.554 0.000 83.345 83.077 96.575 97.456 37.103 95.589 99.395 92.502 90.947 96.343 61.192 [2021-07-09 07:55:52,915][root][INFO] - Finished test. Elapsed time: 42.9065 [2021-07-09 07:55:54,007][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 07:56:07,704][root][INFO] - ===> Epoch[11081](55405/5): Loss 0.0069, Sem 0.0069 LR: 9.904e-03 Score 99.755 Data time: 0.2190, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7022 [2021-07-09 07:56:21,979][root][INFO] - ===> Epoch[11082](55410/5): Loss 0.0068, Sem 0.0068 LR: 9.894e-03 Score 99.760 Data time: 0.2490, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8129 [2021-07-09 07:56:34,663][root][INFO] - ===> Epoch[11083](55415/5): Loss 0.0068, Sem 0.0068 LR: 9.884e-03 Score 99.757 Data time: 0.2562, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.4895 [2021-07-09 07:56:48,030][root][INFO] - ===> Epoch[11084](55420/5): Loss 0.0066, Sem 0.0066 LR: 9.875e-03 Score 99.765 Data time: 0.2476, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6239 [2021-07-09 07:57:01,717][root][INFO] - ===> Epoch[11085](55425/5): Loss 0.0069, Sem 0.0069 LR: 9.865e-03 Score 99.754 Data time: 0.2159, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6887 [2021-07-09 07:57:16,150][root][INFO] - ===> Epoch[11086](55430/5): Loss 0.0063, Sem 0.0063 LR: 9.855e-03 Score 99.777 Data time: 0.2063, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8386 [2021-07-09 07:57:30,898][root][INFO] - ===> Epoch[11087](55435/5): Loss 0.0064, Sem 0.0064 LR: 9.846e-03 Score 99.771 Data time: 0.2599, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9066 [2021-07-09 07:57:45,318][root][INFO] - ===> Epoch[11088](55440/5): Loss 0.0065, Sem 0.0065 LR: 9.836e-03 Score 99.765 Data time: 0.2296, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8344 [2021-07-09 07:57:59,043][root][INFO] - ===> Epoch[11089](55445/5): Loss 0.0067, Sem 0.0067 LR: 9.826e-03 Score 99.759 Data time: 0.2759, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6897 [2021-07-09 07:58:13,324][root][INFO] - ===> Epoch[11090](55450/5): Loss 0.0063, Sem 0.0063 LR: 9.816e-03 Score 99.772 Data time: 0.2242, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8044 [2021-07-09 07:58:28,186][root][INFO] - ===> Epoch[11091](55455/5): Loss 0.0069, Sem 0.0069 LR: 9.807e-03 Score 99.754 Data time: 0.3138, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9204 [2021-07-09 07:58:42,630][root][INFO] - ===> Epoch[11092](55460/5): Loss 0.0067, Sem 0.0067 LR: 9.797e-03 Score 99.758 Data time: 0.2997, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8345 [2021-07-09 07:58:57,875][root][INFO] - ===> Epoch[11093](55465/5): Loss 0.0065, Sem 0.0065 LR: 9.787e-03 Score 99.767 Data time: 0.3294, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9963 [2021-07-09 07:59:11,537][root][INFO] - ===> Epoch[11094](55470/5): Loss 0.0071, Sem 0.0071 LR: 9.778e-03 Score 99.747 Data time: 0.2570, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6879 [2021-07-09 07:59:25,929][root][INFO] - ===> Epoch[11095](55475/5): Loss 0.0066, Sem 0.0066 LR: 9.768e-03 Score 99.762 Data time: 0.3460, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8148 [2021-07-09 07:59:40,229][root][INFO] - ===> Epoch[11096](55480/5): Loss 0.0068, Sem 0.0068 LR: 9.758e-03 Score 99.760 Data time: 0.2187, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8167 [2021-07-09 07:59:54,572][root][INFO] - ===> Epoch[11097](55485/5): Loss 0.0066, Sem 0.0066 LR: 9.748e-03 Score 99.765 Data time: 0.2023, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8126 [2021-07-09 08:00:08,852][root][INFO] - ===> Epoch[11098](55490/5): Loss 0.0065, Sem 0.0065 LR: 9.739e-03 Score 99.769 Data time: 0.1923, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8049 [2021-07-09 08:00:23,064][root][INFO] - ===> Epoch[11099](55495/5): Loss 0.0066, Sem 0.0066 LR: 9.729e-03 Score 99.766 Data time: 0.1851, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7957 [2021-07-09 08:00:37,046][root][INFO] - ===> Epoch[11100](55500/5): Loss 0.0067, Sem 0.0067 LR: 9.719e-03 Score 99.759 Data time: 0.2601, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7477 [2021-07-09 08:00:51,658][root][INFO] - ===> Epoch[11101](55505/5): Loss 0.0066, Sem 0.0066 LR: 9.710e-03 Score 99.767 Data time: 0.2737, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8609 [2021-07-09 08:01:04,937][root][INFO] - ===> Epoch[11102](55510/5): Loss 0.0066, Sem 0.0066 LR: 9.700e-03 Score 99.764 Data time: 0.2513, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6099 [2021-07-09 08:01:19,496][root][INFO] - ===> Epoch[11103](55515/5): Loss 0.0066, Sem 0.0066 LR: 9.690e-03 Score 99.765 Data time: 0.2814, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8686 [2021-07-09 08:01:32,824][root][INFO] - ===> Epoch[11104](55520/5): Loss 0.0069, Sem 0.0069 LR: 9.680e-03 Score 99.755 Data time: 0.2104, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6173 [2021-07-09 08:01:46,693][root][INFO] - ===> Epoch[11105](55525/5): Loss 0.0067, Sem 0.0067 LR: 9.671e-03 Score 99.763 Data time: 0.2508, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7161 [2021-07-09 08:01:59,903][root][INFO] - ===> Epoch[11106](55530/5): Loss 0.0071, Sem 0.0071 LR: 9.661e-03 Score 99.745 Data time: 0.2121, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.5868 [2021-07-09 08:02:14,463][root][INFO] - ===> Epoch[11107](55535/5): Loss 0.0064, Sem 0.0064 LR: 9.651e-03 Score 99.772 Data time: 0.2389, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8617 [2021-07-09 08:02:28,145][root][INFO] - ===> Epoch[11108](55540/5): Loss 0.0068, Sem 0.0068 LR: 9.642e-03 Score 99.757 Data time: 0.2443, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6946 [2021-07-09 08:02:42,166][root][INFO] - ===> Epoch[11109](55545/5): Loss 0.0068, Sem 0.0068 LR: 9.632e-03 Score 99.755 Data time: 0.2204, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7668 [2021-07-09 08:02:56,393][root][INFO] - ===> Epoch[11110](55550/5): Loss 0.0064, Sem 0.0064 LR: 9.622e-03 Score 99.770 Data time: 0.1751, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7906 [2021-07-09 08:03:10,747][root][INFO] - ===> Epoch[11111](55555/5): Loss 0.0066, Sem 0.0066 LR: 9.612e-03 Score 99.765 Data time: 0.2394, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8183 [2021-07-09 08:03:24,958][root][INFO] - ===> Epoch[11112](55560/5): Loss 0.0068, Sem 0.0068 LR: 9.603e-03 Score 99.754 Data time: 0.1832, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7998 [2021-07-09 08:03:39,324][root][INFO] - ===> Epoch[11113](55565/5): Loss 0.0068, Sem 0.0068 LR: 9.593e-03 Score 99.756 Data time: 0.2242, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8255 [2021-07-09 08:03:53,088][root][INFO] - ===> Epoch[11114](55570/5): Loss 0.0067, Sem 0.0067 LR: 9.583e-03 Score 99.761 Data time: 0.2037, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7138 [2021-07-09 08:04:06,548][root][INFO] - ===> Epoch[11115](55575/5): Loss 0.0068, Sem 0.0068 LR: 9.573e-03 Score 99.757 Data time: 0.2329, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6423 [2021-07-09 08:04:20,790][root][INFO] - ===> Epoch[11116](55580/5): Loss 0.0067, Sem 0.0067 LR: 9.564e-03 Score 99.760 Data time: 0.1977, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8082 [2021-07-09 08:04:35,421][root][INFO] - ===> Epoch[11117](55585/5): Loss 0.0070, Sem 0.0070 LR: 9.554e-03 Score 99.751 Data time: 0.1987, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8873 [2021-07-09 08:04:50,088][root][INFO] - ===> Epoch[11118](55590/5): Loss 0.0065, Sem 0.0065 LR: 9.544e-03 Score 99.770 Data time: 0.2770, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8819 [2021-07-09 08:05:02,737][root][INFO] - ===> Epoch[11119](55595/5): Loss 0.0074, Sem 0.0074 LR: 9.534e-03 Score 99.737 Data time: 0.2131, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.4710 [2021-07-09 08:05:16,324][root][INFO] - ===> Epoch[11120](55600/5): Loss 0.0069, Sem 0.0069 LR: 9.525e-03 Score 99.754 Data time: 0.2608, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6721 [2021-07-09 08:05:16,326][root][INFO] - ===> Start testing [2021-07-09 08:05:56,168][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1225 Loss 0.406 (AVG: 0.639) Score 92.004 (AVG: 91.015) mIOU 71.497 mAP 82.039 mAcc 77.501 IOU: 64.342 0.000 79.980 75.213 95.107 91.713 29.628 84.120 98.609 86.651 79.822 86.048 58.230 mAP: 82.200 2.104 86.746 85.096 98.183 97.999 51.626 97.789 99.562 94.379 89.867 91.710 89.253 mAcc: 80.364 0.000 83.037 82.705 96.719 97.411 31.734 94.979 99.441 92.798 90.009 96.462 61.848 [2021-07-09 08:05:56,169][root][INFO] - Finished test. Elapsed time: 39.8422 [2021-07-09 08:05:57,216][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 08:06:10,077][root][INFO] - ===> Epoch[11121](55605/5): Loss 0.0070, Sem 0.0070 LR: 9.515e-03 Score 99.749 Data time: 0.1904, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.5404 [2021-07-09 08:06:24,372][root][INFO] - ===> Epoch[11122](55610/5): Loss 0.0065, Sem 0.0065 LR: 9.505e-03 Score 99.766 Data time: 0.1978, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8021 [2021-07-09 08:06:38,756][root][INFO] - ===> Epoch[11123](55615/5): Loss 0.0065, Sem 0.0065 LR: 9.496e-03 Score 99.770 Data time: 0.1860, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8295 [2021-07-09 08:06:53,578][root][INFO] - ===> Epoch[11124](55620/5): Loss 0.0066, Sem 0.0066 LR: 9.486e-03 Score 99.766 Data time: 0.2220, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9244 [2021-07-09 08:07:07,499][root][INFO] - ===> Epoch[11125](55625/5): Loss 0.0069, Sem 0.0069 LR: 9.476e-03 Score 99.754 Data time: 0.2283, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7371 [2021-07-09 08:07:21,020][root][INFO] - ===> Epoch[11126](55630/5): Loss 0.0069, Sem 0.0069 LR: 9.466e-03 Score 99.751 Data time: 0.2226, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6602 [2021-07-09 08:07:34,509][root][INFO] - ===> Epoch[11127](55635/5): Loss 0.0068, Sem 0.0068 LR: 9.457e-03 Score 99.757 Data time: 0.1938, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6582 [2021-07-09 08:07:48,803][root][INFO] - ===> Epoch[11128](55640/5): Loss 0.0066, Sem 0.0066 LR: 9.447e-03 Score 99.764 Data time: 0.1741, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8095 [2021-07-09 08:08:02,864][root][INFO] - ===> Epoch[11129](55645/5): Loss 0.0064, Sem 0.0064 LR: 9.437e-03 Score 99.771 Data time: 0.2582, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7547 [2021-07-09 08:08:16,349][root][INFO] - ===> Epoch[11130](55650/5): Loss 0.0067, Sem 0.0067 LR: 9.427e-03 Score 99.762 Data time: 0.2090, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6569 [2021-07-09 08:08:29,800][root][INFO] - ===> Epoch[11131](55655/5): Loss 0.0065, Sem 0.0065 LR: 9.418e-03 Score 99.768 Data time: 0.1905, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6467 [2021-07-09 08:08:43,874][root][INFO] - ===> Epoch[11132](55660/5): Loss 0.0066, Sem 0.0066 LR: 9.408e-03 Score 99.766 Data time: 0.2236, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7717 [2021-07-09 08:08:57,342][root][INFO] - ===> Epoch[11133](55665/5): Loss 0.0068, Sem 0.0068 LR: 9.398e-03 Score 99.754 Data time: 0.1827, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6527 [2021-07-09 08:09:11,125][root][INFO] - ===> Epoch[11134](55670/5): Loss 0.0064, Sem 0.0064 LR: 9.388e-03 Score 99.770 Data time: 0.2344, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7081 [2021-07-09 08:09:24,886][root][INFO] - ===> Epoch[11135](55675/5): Loss 0.0066, Sem 0.0066 LR: 9.379e-03 Score 99.762 Data time: 0.1881, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6939 [2021-07-09 08:09:38,325][root][INFO] - ===> Epoch[11136](55680/5): Loss 0.0068, Sem 0.0068 LR: 9.369e-03 Score 99.753 Data time: 0.2689, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6385 [2021-07-09 08:09:52,217][root][INFO] - ===> Epoch[11137](55685/5): Loss 0.0067, Sem 0.0067 LR: 9.359e-03 Score 99.761 Data time: 0.2067, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7393 [2021-07-09 08:10:06,590][root][INFO] - ===> Epoch[11138](55690/5): Loss 0.0065, Sem 0.0065 LR: 9.349e-03 Score 99.767 Data time: 0.1952, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8176 [2021-07-09 08:10:21,081][root][INFO] - ===> Epoch[11139](55695/5): Loss 0.0065, Sem 0.0065 LR: 9.340e-03 Score 99.769 Data time: 0.2637, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8463 [2021-07-09 08:10:35,297][root][INFO] - ===> Epoch[11140](55700/5): Loss 0.0067, Sem 0.0067 LR: 9.330e-03 Score 99.760 Data time: 0.2977, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7973 [2021-07-09 08:10:49,422][root][INFO] - ===> Epoch[11141](55705/5): Loss 0.0067, Sem 0.0067 LR: 9.320e-03 Score 99.762 Data time: 0.1572, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7808 [2021-07-09 08:11:03,498][root][INFO] - ===> Epoch[11142](55710/5): Loss 0.0070, Sem 0.0070 LR: 9.310e-03 Score 99.750 Data time: 0.2380, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7707 [2021-07-09 08:11:18,029][root][INFO] - ===> Epoch[11143](55715/5): Loss 0.0070, Sem 0.0070 LR: 9.300e-03 Score 99.754 Data time: 0.3133, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8559 [2021-07-09 08:11:32,506][root][INFO] - ===> Epoch[11144](55720/5): Loss 0.0068, Sem 0.0068 LR: 9.291e-03 Score 99.756 Data time: 0.2655, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8491 [2021-07-09 08:11:46,504][root][INFO] - ===> Epoch[11145](55725/5): Loss 0.0065, Sem 0.0065 LR: 9.281e-03 Score 99.768 Data time: 0.2180, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7656 [2021-07-09 08:12:00,091][root][INFO] - ===> Epoch[11146](55730/5): Loss 0.0065, Sem 0.0065 LR: 9.271e-03 Score 99.766 Data time: 0.2473, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6696 [2021-07-09 08:12:13,979][root][INFO] - ===> Epoch[11147](55735/5): Loss 0.0064, Sem 0.0064 LR: 9.261e-03 Score 99.772 Data time: 0.2548, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7362 [2021-07-09 08:12:28,383][root][INFO] - ===> Epoch[11148](55740/5): Loss 0.0067, Sem 0.0067 LR: 9.252e-03 Score 99.758 Data time: 0.2454, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8358 [2021-07-09 08:12:42,683][root][INFO] - ===> Epoch[11149](55745/5): Loss 0.0065, Sem 0.0065 LR: 9.242e-03 Score 99.765 Data time: 0.2322, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8152 [2021-07-09 08:12:56,448][root][INFO] - ===> Epoch[11150](55750/5): Loss 0.0066, Sem 0.0066 LR: 9.232e-03 Score 99.765 Data time: 0.2603, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7115 [2021-07-09 08:13:10,649][root][INFO] - ===> Epoch[11151](55755/5): Loss 0.0067, Sem 0.0067 LR: 9.222e-03 Score 99.762 Data time: 0.2574, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8035 [2021-07-09 08:13:24,268][root][INFO] - ===> Epoch[11152](55760/5): Loss 0.0069, Sem 0.0069 LR: 9.213e-03 Score 99.756 Data time: 0.1655, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6636 [2021-07-09 08:13:38,756][root][INFO] - ===> Epoch[11153](55765/5): Loss 0.0061, Sem 0.0061 LR: 9.203e-03 Score 99.781 Data time: 0.2409, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8567 [2021-07-09 08:13:53,434][root][INFO] - ===> Epoch[11154](55770/5): Loss 0.0071, Sem 0.0071 LR: 9.193e-03 Score 99.749 Data time: 0.2883, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8834 [2021-07-09 08:14:06,698][root][INFO] - ===> Epoch[11155](55775/5): Loss 0.0069, Sem 0.0069 LR: 9.183e-03 Score 99.752 Data time: 0.2629, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6127 [2021-07-09 08:14:21,190][root][INFO] - ===> Epoch[11156](55780/5): Loss 0.0066, Sem 0.0066 LR: 9.173e-03 Score 99.765 Data time: 0.1895, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8597 [2021-07-09 08:14:35,311][root][INFO] - ===> Epoch[11157](55785/5): Loss 0.0068, Sem 0.0068 LR: 9.164e-03 Score 99.756 Data time: 0.2501, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7717 [2021-07-09 08:14:49,134][root][INFO] - ===> Epoch[11158](55790/5): Loss 0.0068, Sem 0.0068 LR: 9.154e-03 Score 99.758 Data time: 0.1725, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7320 [2021-07-09 08:15:02,517][root][INFO] - ===> Epoch[11159](55795/5): Loss 0.0066, Sem 0.0066 LR: 9.144e-03 Score 99.762 Data time: 0.2056, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6296 [2021-07-09 08:15:17,657][root][INFO] - ===> Epoch[11160](55800/5): Loss 0.0068, Sem 0.0068 LR: 9.134e-03 Score 99.755 Data time: 0.2574, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9809 [2021-07-09 08:15:17,659][root][INFO] - ===> Start testing [2021-07-09 08:15:57,243][root][INFO] - 68/68: Data time: 0.0016, Iter time: 0.1252 Loss 0.414 (AVG: 0.634) Score 92.807 (AVG: 91.044) mIOU 71.639 mAP 82.202 mAcc 77.998 IOU: 63.974 0.000 80.736 75.018 95.048 90.702 34.579 84.718 98.556 84.074 80.030 86.361 57.511 mAP: 82.430 2.148 88.121 84.806 98.275 97.908 52.591 97.927 99.626 93.925 90.358 92.116 88.392 mAcc: 79.879 0.000 84.600 82.340 96.682 97.467 37.415 96.061 99.381 92.601 90.093 96.509 60.950 [2021-07-09 08:15:57,243][root][INFO] - Finished test. Elapsed time: 39.5834 [2021-07-09 08:15:58,256][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 08:16:12,080][root][INFO] - ===> Epoch[11161](55805/5): Loss 0.0065, Sem 0.0065 LR: 9.125e-03 Score 99.767 Data time: 0.2069, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7247 [2021-07-09 08:16:25,890][root][INFO] - ===> Epoch[11162](55810/5): Loss 0.0067, Sem 0.0067 LR: 9.115e-03 Score 99.761 Data time: 0.2429, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7280 [2021-07-09 08:16:39,543][root][INFO] - ===> Epoch[11163](55815/5): Loss 0.0067, Sem 0.0067 LR: 9.105e-03 Score 99.758 Data time: 0.2202, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6850 [2021-07-09 08:16:53,273][root][INFO] - ===> Epoch[11164](55820/5): Loss 0.0068, Sem 0.0068 LR: 9.095e-03 Score 99.757 Data time: 0.2373, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6862 [2021-07-09 08:17:07,321][root][INFO] - ===> Epoch[11165](55825/5): Loss 0.0068, Sem 0.0068 LR: 9.085e-03 Score 99.756 Data time: 0.2544, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7628 [2021-07-09 08:17:21,277][root][INFO] - ===> Epoch[11166](55830/5): Loss 0.0068, Sem 0.0068 LR: 9.076e-03 Score 99.758 Data time: 0.2123, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7432 [2021-07-09 08:17:35,059][root][INFO] - ===> Epoch[11167](55835/5): Loss 0.0066, Sem 0.0066 LR: 9.066e-03 Score 99.762 Data time: 0.1968, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7152 [2021-07-09 08:17:50,024][root][INFO] - ===> Epoch[11168](55840/5): Loss 0.0065, Sem 0.0065 LR: 9.056e-03 Score 99.770 Data time: 0.3082, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9471 [2021-07-09 08:18:03,514][root][INFO] - ===> Epoch[11169](55845/5): Loss 0.0068, Sem 0.0068 LR: 9.046e-03 Score 99.758 Data time: 0.1701, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6599 [2021-07-09 08:18:17,301][root][INFO] - ===> Epoch[11170](55850/5): Loss 0.0064, Sem 0.0064 LR: 9.036e-03 Score 99.770 Data time: 0.2102, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7140 [2021-07-09 08:18:32,125][root][INFO] - ===> Epoch[11171](55855/5): Loss 0.0068, Sem 0.0068 LR: 9.027e-03 Score 99.756 Data time: 0.1845, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9130 [2021-07-09 08:18:45,698][root][INFO] - ===> Epoch[11172](55860/5): Loss 0.0066, Sem 0.0066 LR: 9.017e-03 Score 99.766 Data time: 0.1968, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6744 [2021-07-09 08:18:59,811][root][INFO] - ===> Epoch[11173](55865/5): Loss 0.0067, Sem 0.0067 LR: 9.007e-03 Score 99.757 Data time: 0.2452, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7794 [2021-07-09 08:19:13,748][root][INFO] - ===> Epoch[11174](55870/5): Loss 0.0066, Sem 0.0066 LR: 8.997e-03 Score 99.765 Data time: 0.2493, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7461 [2021-07-09 08:19:27,868][root][INFO] - ===> Epoch[11175](55875/5): Loss 0.0065, Sem 0.0065 LR: 8.987e-03 Score 99.767 Data time: 0.2269, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7759 [2021-07-09 08:19:41,980][root][INFO] - ===> Epoch[11176](55880/5): Loss 0.0068, Sem 0.0068 LR: 8.978e-03 Score 99.756 Data time: 0.2404, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7797 [2021-07-09 08:19:55,626][root][INFO] - ===> Epoch[11177](55885/5): Loss 0.0069, Sem 0.0069 LR: 8.968e-03 Score 99.756 Data time: 0.1854, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6936 [2021-07-09 08:20:09,752][root][INFO] - ===> Epoch[11178](55890/5): Loss 0.0068, Sem 0.0068 LR: 8.958e-03 Score 99.757 Data time: 0.2024, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7806 [2021-07-09 08:20:24,021][root][INFO] - ===> Epoch[11179](55895/5): Loss 0.0066, Sem 0.0066 LR: 8.948e-03 Score 99.763 Data time: 0.1877, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8124 [2021-07-09 08:20:37,502][root][INFO] - ===> Epoch[11180](55900/5): Loss 0.0071, Sem 0.0071 LR: 8.938e-03 Score 99.754 Data time: 0.1940, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6593 [2021-07-09 08:20:51,630][root][INFO] - ===> Epoch[11181](55905/5): Loss 0.0067, Sem 0.0067 LR: 8.929e-03 Score 99.763 Data time: 0.2765, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7793 [2021-07-09 08:21:05,617][root][INFO] - ===> Epoch[11182](55910/5): Loss 0.0069, Sem 0.0069 LR: 8.919e-03 Score 99.755 Data time: 0.1998, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7549 [2021-07-09 08:21:19,285][root][INFO] - ===> Epoch[11183](55915/5): Loss 0.0063, Sem 0.0063 LR: 8.909e-03 Score 99.775 Data time: 0.2514, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6825 [2021-07-09 08:21:32,984][root][INFO] - ===> Epoch[11184](55920/5): Loss 0.0069, Sem 0.0069 LR: 8.899e-03 Score 99.756 Data time: 0.2498, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6857 [2021-07-09 08:21:46,741][root][INFO] - ===> Epoch[11185](55925/5): Loss 0.0068, Sem 0.0068 LR: 8.889e-03 Score 99.753 Data time: 0.2331, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6897 [2021-07-09 08:22:01,524][root][INFO] - ===> Epoch[11186](55930/5): Loss 0.0066, Sem 0.0066 LR: 8.879e-03 Score 99.764 Data time: 0.1857, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9163 [2021-07-09 08:22:15,909][root][INFO] - ===> Epoch[11187](55935/5): Loss 0.0066, Sem 0.0066 LR: 8.870e-03 Score 99.761 Data time: 0.2502, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8304 [2021-07-09 08:22:31,016][root][INFO] - ===> Epoch[11188](55940/5): Loss 0.0068, Sem 0.0068 LR: 8.860e-03 Score 99.757 Data time: 0.1929, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9719 [2021-07-09 08:22:45,280][root][INFO] - ===> Epoch[11189](55945/5): Loss 0.0065, Sem 0.0065 LR: 8.850e-03 Score 99.767 Data time: 0.2285, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7962 [2021-07-09 08:22:59,482][root][INFO] - ===> Epoch[11190](55950/5): Loss 0.0066, Sem 0.0066 LR: 8.840e-03 Score 99.763 Data time: 0.1786, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8023 [2021-07-09 08:23:13,585][root][INFO] - ===> Epoch[11191](55955/5): Loss 0.0067, Sem 0.0067 LR: 8.830e-03 Score 99.760 Data time: 0.1892, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7684 [2021-07-09 08:23:27,479][root][INFO] - ===> Epoch[11192](55960/5): Loss 0.0067, Sem 0.0067 LR: 8.821e-03 Score 99.763 Data time: 0.2174, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7360 [2021-07-09 08:23:41,975][root][INFO] - ===> Epoch[11193](55965/5): Loss 0.0065, Sem 0.0065 LR: 8.811e-03 Score 99.768 Data time: 0.2642, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8418 [2021-07-09 08:23:55,540][root][INFO] - ===> Epoch[11194](55970/5): Loss 0.0067, Sem 0.0067 LR: 8.801e-03 Score 99.759 Data time: 0.1905, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6686 [2021-07-09 08:24:10,022][root][INFO] - ===> Epoch[11195](55975/5): Loss 0.0066, Sem 0.0066 LR: 8.791e-03 Score 99.762 Data time: 0.2215, Forward time: 1.2516, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.8485 [2021-07-09 08:24:24,150][root][INFO] - ===> Epoch[11196](55980/5): Loss 0.0066, Sem 0.0066 LR: 8.781e-03 Score 99.765 Data time: 0.2182, Forward time: 1.2516, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.7709 [2021-07-09 08:24:37,414][root][INFO] - ===> Epoch[11197](55985/5): Loss 0.0067, Sem 0.0067 LR: 8.771e-03 Score 99.764 Data time: 0.1958, Forward time: 1.2516, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.6034 [2021-07-09 08:24:51,395][root][INFO] - ===> Epoch[11198](55990/5): Loss 0.0068, Sem 0.0068 LR: 8.762e-03 Score 99.755 Data time: 0.1900, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7485 [2021-07-09 08:25:06,130][root][INFO] - ===> Epoch[11199](55995/5): Loss 0.0066, Sem 0.0066 LR: 8.752e-03 Score 99.764 Data time: 0.2505, Forward time: 1.2516, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.9057 [2021-07-09 08:25:20,227][root][INFO] - ===> Epoch[11200](56000/5): Loss 0.0065, Sem 0.0065 LR: 8.742e-03 Score 99.766 Data time: 0.2631, Forward time: 1.2516, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.7692 [2021-07-09 08:25:20,229][root][INFO] - ===> Start testing [2021-07-09 08:26:01,265][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1482 Loss 0.547 (AVG: 0.629) Score 90.275 (AVG: 91.083) mIOU 71.926 mAP 82.445 mAcc 78.331 IOU: 64.208 0.000 80.467 74.924 95.114 91.562 36.014 85.250 98.578 84.389 80.221 86.319 57.996 mAP: 82.331 3.180 88.522 84.985 98.333 97.924 54.850 97.904 99.610 93.172 90.197 91.797 88.984 mAcc: 80.081 0.000 86.217 82.761 96.729 97.284 39.822 95.538 99.375 92.423 90.659 96.116 61.293 [2021-07-09 08:26:01,265][root][INFO] - Finished test. Elapsed time: 41.0352 [2021-07-09 08:26:02,314][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 08:26:15,485][root][INFO] - ===> Epoch[11201](56005/5): Loss 0.0069, Sem 0.0069 LR: 8.732e-03 Score 99.753 Data time: 0.2178, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.5938 [2021-07-09 08:26:29,281][root][INFO] - ===> Epoch[11202](56010/5): Loss 0.0066, Sem 0.0066 LR: 8.722e-03 Score 99.760 Data time: 0.3029, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7085 [2021-07-09 08:26:44,132][root][INFO] - ===> Epoch[11203](56015/5): Loss 0.0068, Sem 0.0068 LR: 8.712e-03 Score 99.758 Data time: 0.3068, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9106 [2021-07-09 08:26:58,620][root][INFO] - ===> Epoch[11204](56020/5): Loss 0.0066, Sem 0.0066 LR: 8.703e-03 Score 99.766 Data time: 0.2192, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8578 [2021-07-09 08:27:12,888][root][INFO] - ===> Epoch[11205](56025/5): Loss 0.0069, Sem 0.0069 LR: 8.693e-03 Score 99.756 Data time: 0.1977, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8168 [2021-07-09 08:27:27,239][root][INFO] - ===> Epoch[11206](56030/5): Loss 0.0063, Sem 0.0063 LR: 8.683e-03 Score 99.777 Data time: 0.2323, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8165 [2021-07-09 08:27:41,958][root][INFO] - ===> Epoch[11207](56035/5): Loss 0.0067, Sem 0.0067 LR: 8.673e-03 Score 99.760 Data time: 0.1874, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8992 [2021-07-09 08:27:55,561][root][INFO] - ===> Epoch[11208](56040/5): Loss 0.0067, Sem 0.0067 LR: 8.663e-03 Score 99.761 Data time: 0.1937, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6846 [2021-07-09 08:28:08,923][root][INFO] - ===> Epoch[11209](56045/5): Loss 0.0067, Sem 0.0067 LR: 8.653e-03 Score 99.761 Data time: 0.2199, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6251 [2021-07-09 08:28:22,507][root][INFO] - ===> Epoch[11210](56050/5): Loss 0.0067, Sem 0.0067 LR: 8.644e-03 Score 99.761 Data time: 0.2506, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6622 [2021-07-09 08:28:36,509][root][INFO] - ===> Epoch[11211](56055/5): Loss 0.0065, Sem 0.0065 LR: 8.634e-03 Score 99.765 Data time: 0.2288, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7605 [2021-07-09 08:28:50,779][root][INFO] - ===> Epoch[11212](56060/5): Loss 0.0067, Sem 0.0067 LR: 8.624e-03 Score 99.755 Data time: 0.2039, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8147 [2021-07-09 08:29:04,548][root][INFO] - ===> Epoch[11213](56065/5): Loss 0.0067, Sem 0.0067 LR: 8.614e-03 Score 99.762 Data time: 0.2535, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7082 [2021-07-09 08:29:17,955][root][INFO] - ===> Epoch[11214](56070/5): Loss 0.0067, Sem 0.0067 LR: 8.604e-03 Score 99.761 Data time: 0.1902, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6337 [2021-07-09 08:29:32,484][root][INFO] - ===> Epoch[11215](56075/5): Loss 0.0068, Sem 0.0068 LR: 8.594e-03 Score 99.757 Data time: 0.1614, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8647 [2021-07-09 08:29:46,660][root][INFO] - ===> Epoch[11216](56080/5): Loss 0.0065, Sem 0.0065 LR: 8.584e-03 Score 99.767 Data time: 0.2411, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7858 [2021-07-09 08:30:01,427][root][INFO] - ===> Epoch[11217](56085/5): Loss 0.0067, Sem 0.0067 LR: 8.575e-03 Score 99.760 Data time: 0.2532, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9079 [2021-07-09 08:30:15,569][root][INFO] - ===> Epoch[11218](56090/5): Loss 0.0068, Sem 0.0068 LR: 8.565e-03 Score 99.756 Data time: 0.2019, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7840 [2021-07-09 08:30:29,576][root][INFO] - ===> Epoch[11219](56095/5): Loss 0.0065, Sem 0.0065 LR: 8.555e-03 Score 99.770 Data time: 0.2811, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7600 [2021-07-09 08:30:43,683][root][INFO] - ===> Epoch[11220](56100/5): Loss 0.0068, Sem 0.0068 LR: 8.545e-03 Score 99.756 Data time: 0.1604, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7858 [2021-07-09 08:30:57,600][root][INFO] - ===> Epoch[11221](56105/5): Loss 0.0067, Sem 0.0067 LR: 8.535e-03 Score 99.765 Data time: 0.2338, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7382 [2021-07-09 08:31:10,734][root][INFO] - ===> Epoch[11222](56110/5): Loss 0.0062, Sem 0.0062 LR: 8.525e-03 Score 99.780 Data time: 0.1362, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.5773 [2021-07-09 08:31:24,845][root][INFO] - ===> Epoch[11223](56115/5): Loss 0.0064, Sem 0.0064 LR: 8.515e-03 Score 99.770 Data time: 0.2090, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7612 [2021-07-09 08:31:38,214][root][INFO] - ===> Epoch[11224](56120/5): Loss 0.0065, Sem 0.0065 LR: 8.506e-03 Score 99.769 Data time: 0.1912, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6357 [2021-07-09 08:31:52,125][root][INFO] - ===> Epoch[11225](56125/5): Loss 0.0065, Sem 0.0065 LR: 8.496e-03 Score 99.766 Data time: 0.2493, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7332 [2021-07-09 08:32:06,707][root][INFO] - ===> Epoch[11226](56130/5): Loss 0.0065, Sem 0.0065 LR: 8.486e-03 Score 99.765 Data time: 0.2095, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8685 [2021-07-09 08:32:20,430][root][INFO] - ===> Epoch[11227](56135/5): Loss 0.0065, Sem 0.0065 LR: 8.476e-03 Score 99.767 Data time: 0.1648, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7013 [2021-07-09 08:32:34,853][root][INFO] - ===> Epoch[11228](56140/5): Loss 0.0067, Sem 0.0067 LR: 8.466e-03 Score 99.760 Data time: 0.2265, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8393 [2021-07-09 08:32:49,063][root][INFO] - ===> Epoch[11229](56145/5): Loss 0.0065, Sem 0.0065 LR: 8.456e-03 Score 99.767 Data time: 0.2014, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8070 [2021-07-09 08:33:03,804][root][INFO] - ===> Epoch[11230](56150/5): Loss 0.0067, Sem 0.0067 LR: 8.446e-03 Score 99.760 Data time: 0.2607, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8980 [2021-07-09 08:33:16,876][root][INFO] - ===> Epoch[11231](56155/5): Loss 0.0065, Sem 0.0065 LR: 8.437e-03 Score 99.766 Data time: 0.2358, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.5771 [2021-07-09 08:33:30,674][root][INFO] - ===> Epoch[11232](56160/5): Loss 0.0068, Sem 0.0068 LR: 8.427e-03 Score 99.759 Data time: 0.2287, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7171 [2021-07-09 08:33:44,421][root][INFO] - ===> Epoch[11233](56165/5): Loss 0.0069, Sem 0.0069 LR: 8.417e-03 Score 99.754 Data time: 0.1863, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7077 [2021-07-09 08:33:58,670][root][INFO] - ===> Epoch[11234](56170/5): Loss 0.0065, Sem 0.0065 LR: 8.407e-03 Score 99.764 Data time: 0.2153, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8098 [2021-07-09 08:34:11,844][root][INFO] - ===> Epoch[11235](56175/5): Loss 0.0066, Sem 0.0066 LR: 8.397e-03 Score 99.765 Data time: 0.2530, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.5950 [2021-07-09 08:34:25,476][root][INFO] - ===> Epoch[11236](56180/5): Loss 0.0065, Sem 0.0065 LR: 8.387e-03 Score 99.769 Data time: 0.2102, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6792 [2021-07-09 08:34:39,581][root][INFO] - ===> Epoch[11237](56185/5): Loss 0.0069, Sem 0.0069 LR: 8.377e-03 Score 99.752 Data time: 0.2554, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7731 [2021-07-09 08:34:53,608][root][INFO] - ===> Epoch[11238](56190/5): Loss 0.0067, Sem 0.0067 LR: 8.367e-03 Score 99.762 Data time: 0.2179, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7554 [2021-07-09 08:35:07,649][root][INFO] - ===> Epoch[11239](56195/5): Loss 0.0067, Sem 0.0067 LR: 8.358e-03 Score 99.761 Data time: 0.1954, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7648 [2021-07-09 08:35:22,438][root][INFO] - ===> Epoch[11240](56200/5): Loss 0.0063, Sem 0.0063 LR: 8.348e-03 Score 99.774 Data time: 0.2050, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9177 [2021-07-09 08:35:22,440][root][INFO] - ===> Start testing [2021-07-09 08:36:05,588][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1364 Loss 0.491 (AVG: 0.630) Score 90.217 (AVG: 91.089) mIOU 71.793 mAP 82.289 mAcc 77.901 IOU: 64.040 0.000 79.842 75.294 95.148 91.006 34.538 85.413 98.540 84.702 80.184 86.132 58.475 mAP: 82.523 1.974 87.471 85.406 98.249 97.794 53.587 97.964 99.605 93.640 90.313 91.852 89.381 mAcc: 79.422 0.000 83.698 83.345 96.874 97.141 37.524 95.403 99.313 91.496 90.427 96.324 61.751 [2021-07-09 08:36:05,588][root][INFO] - Finished test. Elapsed time: 43.1475 [2021-07-09 08:36:06,781][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 08:36:20,035][root][INFO] - ===> Epoch[11241](56205/5): Loss 0.0064, Sem 0.0064 LR: 8.338e-03 Score 99.769 Data time: 0.1937, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6110 [2021-07-09 08:36:33,704][root][INFO] - ===> Epoch[11242](56210/5): Loss 0.0065, Sem 0.0065 LR: 8.328e-03 Score 99.768 Data time: 0.2168, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6882 [2021-07-09 08:36:48,061][root][INFO] - ===> Epoch[11243](56215/5): Loss 0.0073, Sem 0.0073 LR: 8.318e-03 Score 99.738 Data time: 0.1607, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8336 [2021-07-09 08:37:00,802][root][INFO] - ===> Epoch[11244](56220/5): Loss 0.0066, Sem 0.0066 LR: 8.308e-03 Score 99.765 Data time: 0.1789, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.5012 [2021-07-09 08:37:14,330][root][INFO] - ===> Epoch[11245](56225/5): Loss 0.0066, Sem 0.0066 LR: 8.298e-03 Score 99.761 Data time: 0.2359, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6624 [2021-07-09 08:37:28,399][root][INFO] - ===> Epoch[11246](56230/5): Loss 0.0067, Sem 0.0067 LR: 8.288e-03 Score 99.759 Data time: 0.2285, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7607 [2021-07-09 08:37:42,158][root][INFO] - ===> Epoch[11247](56235/5): Loss 0.0065, Sem 0.0065 LR: 8.278e-03 Score 99.768 Data time: 0.2192, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7056 [2021-07-09 08:37:55,994][root][INFO] - ===> Epoch[11248](56240/5): Loss 0.0065, Sem 0.0065 LR: 8.269e-03 Score 99.768 Data time: 0.2185, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7198 [2021-07-09 08:38:09,932][root][INFO] - ===> Epoch[11249](56245/5): Loss 0.0069, Sem 0.0069 LR: 8.259e-03 Score 99.755 Data time: 0.2721, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7351 [2021-07-09 08:38:23,199][root][INFO] - ===> Epoch[11250](56250/5): Loss 0.0064, Sem 0.0064 LR: 8.249e-03 Score 99.771 Data time: 0.1969, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6019 [2021-07-09 08:38:36,713][root][INFO] - ===> Epoch[11251](56255/5): Loss 0.0072, Sem 0.0072 LR: 8.239e-03 Score 99.743 Data time: 0.1680, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6678 [2021-07-09 08:38:52,145][root][INFO] - ===> Epoch[11252](56260/5): Loss 0.0067, Sem 0.0067 LR: 8.229e-03 Score 99.762 Data time: 0.3525, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 3.0355 [2021-07-09 08:39:06,522][root][INFO] - ===> Epoch[11253](56265/5): Loss 0.0067, Sem 0.0067 LR: 8.219e-03 Score 99.760 Data time: 0.3418, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8160 [2021-07-09 08:39:21,070][root][INFO] - ===> Epoch[11254](56270/5): Loss 0.0066, Sem 0.0066 LR: 8.209e-03 Score 99.765 Data time: 0.2382, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8677 [2021-07-09 08:39:35,171][root][INFO] - ===> Epoch[11255](56275/5): Loss 0.0068, Sem 0.0068 LR: 8.199e-03 Score 99.758 Data time: 0.2520, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7851 [2021-07-09 08:39:49,271][root][INFO] - ===> Epoch[11256](56280/5): Loss 0.0066, Sem 0.0066 LR: 8.189e-03 Score 99.766 Data time: 0.2717, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7796 [2021-07-09 08:40:03,106][root][INFO] - ===> Epoch[11257](56285/5): Loss 0.0067, Sem 0.0067 LR: 8.179e-03 Score 99.763 Data time: 0.1824, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7319 [2021-07-09 08:40:17,138][root][INFO] - ===> Epoch[11258](56290/5): Loss 0.0068, Sem 0.0068 LR: 8.170e-03 Score 99.756 Data time: 0.2093, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7570 [2021-07-09 08:40:31,275][root][INFO] - ===> Epoch[11259](56295/5): Loss 0.0069, Sem 0.0069 LR: 8.160e-03 Score 99.754 Data time: 0.2293, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7854 [2021-07-09 08:40:46,591][root][INFO] - ===> Epoch[11260](56300/5): Loss 0.0062, Sem 0.0062 LR: 8.150e-03 Score 99.776 Data time: 0.3297, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 3.0020 [2021-07-09 08:41:00,937][root][INFO] - ===> Epoch[11261](56305/5): Loss 0.0065, Sem 0.0065 LR: 8.140e-03 Score 99.769 Data time: 0.3205, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8314 [2021-07-09 08:41:14,690][root][INFO] - ===> Epoch[11262](56310/5): Loss 0.0066, Sem 0.0066 LR: 8.130e-03 Score 99.765 Data time: 0.2504, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7058 [2021-07-09 08:41:28,975][root][INFO] - ===> Epoch[11263](56315/5): Loss 0.0066, Sem 0.0066 LR: 8.120e-03 Score 99.763 Data time: 0.2160, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8105 [2021-07-09 08:41:42,458][root][INFO] - ===> Epoch[11264](56320/5): Loss 0.0066, Sem 0.0066 LR: 8.110e-03 Score 99.762 Data time: 0.2582, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6572 [2021-07-09 08:41:57,109][root][INFO] - ===> Epoch[11265](56325/5): Loss 0.0062, Sem 0.0062 LR: 8.100e-03 Score 99.778 Data time: 0.1718, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8917 [2021-07-09 08:42:11,210][root][INFO] - ===> Epoch[11266](56330/5): Loss 0.0065, Sem 0.0065 LR: 8.090e-03 Score 99.769 Data time: 0.2593, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7735 [2021-07-09 08:42:26,122][root][INFO] - ===> Epoch[11267](56335/5): Loss 0.0066, Sem 0.0066 LR: 8.080e-03 Score 99.764 Data time: 0.2204, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9270 [2021-07-09 08:42:40,497][root][INFO] - ===> Epoch[11268](56340/5): Loss 0.0066, Sem 0.0066 LR: 8.070e-03 Score 99.767 Data time: 0.1619, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8400 [2021-07-09 08:42:54,673][root][INFO] - ===> Epoch[11269](56345/5): Loss 0.0069, Sem 0.0069 LR: 8.061e-03 Score 99.752 Data time: 0.2508, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7905 [2021-07-09 08:43:08,486][root][INFO] - ===> Epoch[11270](56350/5): Loss 0.0064, Sem 0.0064 LR: 8.051e-03 Score 99.771 Data time: 0.1781, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7183 [2021-07-09 08:43:22,582][root][INFO] - ===> Epoch[11271](56355/5): Loss 0.0066, Sem 0.0066 LR: 8.041e-03 Score 99.766 Data time: 0.2040, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7878 [2021-07-09 08:43:36,128][root][INFO] - ===> Epoch[11272](56360/5): Loss 0.0067, Sem 0.0067 LR: 8.031e-03 Score 99.759 Data time: 0.1952, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6696 [2021-07-09 08:43:49,478][root][INFO] - ===> Epoch[11273](56365/5): Loss 0.0068, Sem 0.0068 LR: 8.021e-03 Score 99.759 Data time: 0.1912, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6290 [2021-07-09 08:44:03,627][root][INFO] - ===> Epoch[11274](56370/5): Loss 0.0069, Sem 0.0069 LR: 8.011e-03 Score 99.755 Data time: 0.1823, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7906 [2021-07-09 08:44:16,889][root][INFO] - ===> Epoch[11275](56375/5): Loss 0.0068, Sem 0.0068 LR: 8.001e-03 Score 99.757 Data time: 0.2160, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6121 [2021-07-09 08:44:31,669][root][INFO] - ===> Epoch[11276](56380/5): Loss 0.0068, Sem 0.0068 LR: 7.991e-03 Score 99.757 Data time: 0.1832, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9078 [2021-07-09 08:44:45,378][root][INFO] - ===> Epoch[11277](56385/5): Loss 0.0063, Sem 0.0063 LR: 7.981e-03 Score 99.775 Data time: 0.2491, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6884 [2021-07-09 08:44:58,882][root][INFO] - ===> Epoch[11278](56390/5): Loss 0.0067, Sem 0.0067 LR: 7.971e-03 Score 99.761 Data time: 0.2381, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6611 [2021-07-09 08:45:12,807][root][INFO] - ===> Epoch[11279](56395/5): Loss 0.0064, Sem 0.0064 LR: 7.961e-03 Score 99.773 Data time: 0.1941, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7355 [2021-07-09 08:45:26,238][root][INFO] - ===> Epoch[11280](56400/5): Loss 0.0066, Sem 0.0066 LR: 7.951e-03 Score 99.768 Data time: 0.1882, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6415 [2021-07-09 08:45:26,239][root][INFO] - ===> Start testing [2021-07-09 08:46:12,414][root][INFO] - 68/68: Data time: 0.0011, Iter time: 0.1298 Loss 0.477 (AVG: 0.643) Score 90.712 (AVG: 91.028) mIOU 71.534 mAP 82.268 mAcc 77.813 IOU: 63.980 0.000 79.210 75.036 95.081 91.289 33.505 85.300 98.615 84.329 79.906 86.140 57.555 mAP: 82.299 2.221 87.924 85.325 98.375 97.922 52.445 97.850 99.587 94.445 90.373 91.960 88.757 mAcc: 79.104 0.000 83.793 83.204 96.717 97.639 36.121 95.396 99.405 92.906 90.351 96.539 60.387 [2021-07-09 08:46:12,414][root][INFO] - Finished test. Elapsed time: 46.1744 [2021-07-09 08:46:13,513][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 08:46:27,225][root][INFO] - ===> Epoch[11281](56405/5): Loss 0.0067, Sem 0.0067 LR: 7.941e-03 Score 99.762 Data time: 0.2112, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7022 [2021-07-09 08:46:40,157][root][INFO] - ===> Epoch[11282](56410/5): Loss 0.0066, Sem 0.0066 LR: 7.931e-03 Score 99.765 Data time: 0.2045, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.5395 [2021-07-09 08:46:54,195][root][INFO] - ===> Epoch[11283](56415/5): Loss 0.0065, Sem 0.0065 LR: 7.921e-03 Score 99.766 Data time: 0.1931, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7668 [2021-07-09 08:47:07,914][root][INFO] - ===> Epoch[11284](56420/5): Loss 0.0068, Sem 0.0068 LR: 7.912e-03 Score 99.758 Data time: 0.1818, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7038 [2021-07-09 08:47:21,429][root][INFO] - ===> Epoch[11285](56425/5): Loss 0.0070, Sem 0.0070 LR: 7.902e-03 Score 99.749 Data time: 0.2131, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6604 [2021-07-09 08:47:36,208][root][INFO] - ===> Epoch[11286](56430/5): Loss 0.0068, Sem 0.0068 LR: 7.892e-03 Score 99.758 Data time: 0.2170, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9130 [2021-07-09 08:47:50,001][root][INFO] - ===> Epoch[11287](56435/5): Loss 0.0068, Sem 0.0068 LR: 7.882e-03 Score 99.759 Data time: 0.2833, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7087 [2021-07-09 08:48:03,194][root][INFO] - ===> Epoch[11288](56440/5): Loss 0.0064, Sem 0.0064 LR: 7.872e-03 Score 99.771 Data time: 0.1723, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.5990 [2021-07-09 08:48:17,526][root][INFO] - ===> Epoch[11289](56445/5): Loss 0.0068, Sem 0.0068 LR: 7.862e-03 Score 99.760 Data time: 0.2336, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8208 [2021-07-09 08:48:30,970][root][INFO] - ===> Epoch[11290](56450/5): Loss 0.0069, Sem 0.0069 LR: 7.852e-03 Score 99.755 Data time: 0.2225, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6476 [2021-07-09 08:48:45,151][root][INFO] - ===> Epoch[11291](56455/5): Loss 0.0067, Sem 0.0067 LR: 7.842e-03 Score 99.763 Data time: 0.2111, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7972 [2021-07-09 08:48:58,529][root][INFO] - ===> Epoch[11292](56460/5): Loss 0.0068, Sem 0.0068 LR: 7.832e-03 Score 99.755 Data time: 0.2625, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6275 [2021-07-09 08:49:13,463][root][INFO] - ===> Epoch[11293](56465/5): Loss 0.0063, Sem 0.0063 LR: 7.822e-03 Score 99.775 Data time: 0.2150, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9388 [2021-07-09 08:49:27,408][root][INFO] - ===> Epoch[11294](56470/5): Loss 0.0068, Sem 0.0068 LR: 7.812e-03 Score 99.757 Data time: 0.2068, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7431 [2021-07-09 08:49:40,698][root][INFO] - ===> Epoch[11295](56475/5): Loss 0.0066, Sem 0.0066 LR: 7.802e-03 Score 99.762 Data time: 0.2186, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6076 [2021-07-09 08:49:55,050][root][INFO] - ===> Epoch[11296](56480/5): Loss 0.0065, Sem 0.0065 LR: 7.792e-03 Score 99.765 Data time: 0.2423, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8258 [2021-07-09 08:50:09,219][root][INFO] - ===> Epoch[11297](56485/5): Loss 0.0069, Sem 0.0069 LR: 7.782e-03 Score 99.754 Data time: 0.2515, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7844 [2021-07-09 08:50:23,679][root][INFO] - ===> Epoch[11298](56490/5): Loss 0.0065, Sem 0.0065 LR: 7.772e-03 Score 99.772 Data time: 0.2129, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8481 [2021-07-09 08:50:38,050][root][INFO] - ===> Epoch[11299](56495/5): Loss 0.0067, Sem 0.0067 LR: 7.762e-03 Score 99.761 Data time: 0.2517, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8197 [2021-07-09 08:50:51,229][root][INFO] - ===> Epoch[11300](56500/5): Loss 0.0067, Sem 0.0067 LR: 7.752e-03 Score 99.763 Data time: 0.2275, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.5839 [2021-07-09 08:51:04,834][root][INFO] - ===> Epoch[11301](56505/5): Loss 0.0066, Sem 0.0066 LR: 7.742e-03 Score 99.765 Data time: 0.1959, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6744 [2021-07-09 08:51:18,405][root][INFO] - ===> Epoch[11302](56510/5): Loss 0.0066, Sem 0.0066 LR: 7.732e-03 Score 99.762 Data time: 0.2016, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6722 [2021-07-09 08:51:32,198][root][INFO] - ===> Epoch[11303](56515/5): Loss 0.0067, Sem 0.0067 LR: 7.722e-03 Score 99.762 Data time: 0.2037, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7229 [2021-07-09 08:51:46,459][root][INFO] - ===> Epoch[11304](56520/5): Loss 0.0067, Sem 0.0067 LR: 7.712e-03 Score 99.760 Data time: 0.2370, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7973 [2021-07-09 08:52:00,152][root][INFO] - ===> Epoch[11305](56525/5): Loss 0.0064, Sem 0.0064 LR: 7.702e-03 Score 99.770 Data time: 0.2242, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6945 [2021-07-09 08:52:14,328][root][INFO] - ===> Epoch[11306](56530/5): Loss 0.0066, Sem 0.0066 LR: 7.692e-03 Score 99.767 Data time: 0.2117, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7963 [2021-07-09 08:52:28,296][root][INFO] - ===> Epoch[11307](56535/5): Loss 0.0063, Sem 0.0063 LR: 7.683e-03 Score 99.776 Data time: 0.1989, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7519 [2021-07-09 08:52:42,382][root][INFO] - ===> Epoch[11308](56540/5): Loss 0.0065, Sem 0.0065 LR: 7.673e-03 Score 99.770 Data time: 0.2702, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7641 [2021-07-09 08:52:56,527][root][INFO] - ===> Epoch[11309](56545/5): Loss 0.0066, Sem 0.0066 LR: 7.663e-03 Score 99.763 Data time: 0.3305, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7805 [2021-07-09 08:53:11,635][root][INFO] - ===> Epoch[11310](56550/5): Loss 0.0066, Sem 0.0066 LR: 7.653e-03 Score 99.764 Data time: 0.2443, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9850 [2021-07-09 08:53:26,412][root][INFO] - ===> Epoch[11311](56555/5): Loss 0.0064, Sem 0.0064 LR: 7.643e-03 Score 99.770 Data time: 0.3017, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8949 [2021-07-09 08:53:40,646][root][INFO] - ===> Epoch[11312](56560/5): Loss 0.0066, Sem 0.0066 LR: 7.633e-03 Score 99.765 Data time: 0.3301, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7994 [2021-07-09 08:53:54,421][root][INFO] - ===> Epoch[11313](56565/5): Loss 0.0070, Sem 0.0070 LR: 7.623e-03 Score 99.749 Data time: 0.2080, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7117 [2021-07-09 08:54:09,383][root][INFO] - ===> Epoch[11314](56570/5): Loss 0.0065, Sem 0.0065 LR: 7.613e-03 Score 99.770 Data time: 0.2093, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9452 [2021-07-09 08:54:23,836][root][INFO] - ===> Epoch[11315](56575/5): Loss 0.0068, Sem 0.0068 LR: 7.603e-03 Score 99.755 Data time: 0.2971, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.8473 [2021-07-09 08:54:38,344][root][INFO] - ===> Epoch[11316](56580/5): Loss 0.0065, Sem 0.0065 LR: 7.593e-03 Score 99.771 Data time: 0.3095, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.8629 [2021-07-09 08:54:53,172][root][INFO] - ===> Epoch[11317](56585/5): Loss 0.0067, Sem 0.0067 LR: 7.583e-03 Score 99.762 Data time: 0.1618, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.9290 [2021-07-09 08:55:07,312][root][INFO] - ===> Epoch[11318](56590/5): Loss 0.0064, Sem 0.0064 LR: 7.573e-03 Score 99.769 Data time: 0.1979, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7955 [2021-07-09 08:55:20,656][root][INFO] - ===> Epoch[11319](56595/5): Loss 0.0066, Sem 0.0066 LR: 7.563e-03 Score 99.765 Data time: 0.2033, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6259 [2021-07-09 08:55:34,649][root][INFO] - ===> Epoch[11320](56600/5): Loss 0.0066, Sem 0.0066 LR: 7.553e-03 Score 99.764 Data time: 0.2044, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7501 [2021-07-09 08:55:34,651][root][INFO] - ===> Start testing [2021-07-09 08:56:13,924][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1582 Loss 0.470 (AVG: 0.637) Score 90.757 (AVG: 91.070) mIOU 71.640 mAP 82.266 mAcc 78.049 IOU: 64.119 0.000 79.610 75.198 95.129 91.258 35.024 84.983 98.585 83.646 79.914 86.273 57.575 mAP: 82.095 2.776 87.391 85.283 98.207 97.908 53.178 97.883 99.595 94.262 90.335 92.128 88.414 mAcc: 79.765 0.000 83.809 82.731 96.807 97.613 38.086 95.857 99.346 92.897 90.554 96.369 60.799 [2021-07-09 08:56:13,924][root][INFO] - Finished test. Elapsed time: 39.2729 [2021-07-09 08:56:14,985][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 08:56:27,730][root][INFO] - ===> Epoch[11321](56605/5): Loss 0.0065, Sem 0.0065 LR: 7.543e-03 Score 99.767 Data time: 0.1893, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.5180 [2021-07-09 08:56:42,545][root][INFO] - ===> Epoch[11322](56610/5): Loss 0.0064, Sem 0.0064 LR: 7.533e-03 Score 99.772 Data time: 0.2445, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.9151 [2021-07-09 08:56:56,517][root][INFO] - ===> Epoch[11323](56615/5): Loss 0.0065, Sem 0.0065 LR: 7.523e-03 Score 99.769 Data time: 0.2156, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7484 [2021-07-09 08:57:10,330][root][INFO] - ===> Epoch[11324](56620/5): Loss 0.0066, Sem 0.0066 LR: 7.513e-03 Score 99.767 Data time: 0.2896, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7194 [2021-07-09 08:57:24,432][root][INFO] - ===> Epoch[11325](56625/5): Loss 0.0066, Sem 0.0066 LR: 7.503e-03 Score 99.763 Data time: 0.2416, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7649 [2021-07-09 08:57:39,031][root][INFO] - ===> Epoch[11326](56630/5): Loss 0.0064, Sem 0.0064 LR: 7.493e-03 Score 99.771 Data time: 0.2421, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.8736 [2021-07-09 08:57:53,029][root][INFO] - ===> Epoch[11327](56635/5): Loss 0.0067, Sem 0.0067 LR: 7.483e-03 Score 99.763 Data time: 0.2692, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7474 [2021-07-09 08:58:06,499][root][INFO] - ===> Epoch[11328](56640/5): Loss 0.0067, Sem 0.0067 LR: 7.473e-03 Score 99.763 Data time: 0.2580, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.6500 [2021-07-09 08:58:21,555][root][INFO] - ===> Epoch[11329](56645/5): Loss 0.0065, Sem 0.0065 LR: 7.463e-03 Score 99.771 Data time: 0.2554, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.9655 [2021-07-09 08:58:34,979][root][INFO] - ===> Epoch[11330](56650/5): Loss 0.0066, Sem 0.0066 LR: 7.453e-03 Score 99.766 Data time: 0.2311, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.6493 [2021-07-09 08:58:49,298][root][INFO] - ===> Epoch[11331](56655/5): Loss 0.0063, Sem 0.0063 LR: 7.443e-03 Score 99.774 Data time: 0.1981, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.8232 [2021-07-09 08:59:03,369][root][INFO] - ===> Epoch[11332](56660/5): Loss 0.0065, Sem 0.0065 LR: 7.433e-03 Score 99.767 Data time: 0.2242, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7642 [2021-07-09 08:59:17,596][root][INFO] - ===> Epoch[11333](56665/5): Loss 0.0066, Sem 0.0066 LR: 7.423e-03 Score 99.765 Data time: 0.1938, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8005 [2021-07-09 08:59:31,578][root][INFO] - ===> Epoch[11334](56670/5): Loss 0.0065, Sem 0.0065 LR: 7.413e-03 Score 99.769 Data time: 0.2487, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7442 [2021-07-09 08:59:46,711][root][INFO] - ===> Epoch[11335](56675/5): Loss 0.0063, Sem 0.0063 LR: 7.403e-03 Score 99.775 Data time: 0.2866, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9814 [2021-07-09 09:00:00,030][root][INFO] - ===> Epoch[11336](56680/5): Loss 0.0069, Sem 0.0069 LR: 7.393e-03 Score 99.754 Data time: 0.2027, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6211 [2021-07-09 09:00:14,436][root][INFO] - ===> Epoch[11337](56685/5): Loss 0.0066, Sem 0.0066 LR: 7.383e-03 Score 99.763 Data time: 0.2326, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8237 [2021-07-09 09:00:28,756][root][INFO] - ===> Epoch[11338](56690/5): Loss 0.0068, Sem 0.0068 LR: 7.373e-03 Score 99.756 Data time: 0.2481, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8062 [2021-07-09 09:00:43,031][root][INFO] - ===> Epoch[11339](56695/5): Loss 0.0066, Sem 0.0066 LR: 7.363e-03 Score 99.766 Data time: 0.2017, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8063 [2021-07-09 09:00:56,778][root][INFO] - ===> Epoch[11340](56700/5): Loss 0.0064, Sem 0.0064 LR: 7.353e-03 Score 99.772 Data time: 0.2073, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7033 [2021-07-09 09:01:11,444][root][INFO] - ===> Epoch[11341](56705/5): Loss 0.0061, Sem 0.0061 LR: 7.343e-03 Score 99.780 Data time: 0.1737, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8882 [2021-07-09 09:01:25,934][root][INFO] - ===> Epoch[11342](56710/5): Loss 0.0067, Sem 0.0067 LR: 7.333e-03 Score 99.759 Data time: 0.2027, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8508 [2021-07-09 09:01:39,679][root][INFO] - ===> Epoch[11343](56715/5): Loss 0.0064, Sem 0.0064 LR: 7.322e-03 Score 99.771 Data time: 0.2364, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7003 [2021-07-09 09:01:53,251][root][INFO] - ===> Epoch[11344](56720/5): Loss 0.0067, Sem 0.0067 LR: 7.312e-03 Score 99.758 Data time: 0.1880, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6712 [2021-07-09 09:02:06,511][root][INFO] - ===> Epoch[11345](56725/5): Loss 0.0069, Sem 0.0069 LR: 7.302e-03 Score 99.751 Data time: 0.2154, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6114 [2021-07-09 09:02:20,992][root][INFO] - ===> Epoch[11346](56730/5): Loss 0.0073, Sem 0.0073 LR: 7.292e-03 Score 99.743 Data time: 0.2343, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8539 [2021-07-09 09:02:34,969][root][INFO] - ===> Epoch[11347](56735/5): Loss 0.0066, Sem 0.0066 LR: 7.282e-03 Score 99.766 Data time: 0.2141, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7508 [2021-07-09 09:02:48,557][root][INFO] - ===> Epoch[11348](56740/5): Loss 0.0065, Sem 0.0065 LR: 7.272e-03 Score 99.770 Data time: 0.1864, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6665 [2021-07-09 09:03:02,599][root][INFO] - ===> Epoch[11349](56745/5): Loss 0.0070, Sem 0.0070 LR: 7.262e-03 Score 99.751 Data time: 0.2463, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7595 [2021-07-09 09:03:16,246][root][INFO] - ===> Epoch[11350](56750/5): Loss 0.0066, Sem 0.0066 LR: 7.252e-03 Score 99.765 Data time: 0.1749, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6937 [2021-07-09 09:03:30,791][root][INFO] - ===> Epoch[11351](56755/5): Loss 0.0067, Sem 0.0067 LR: 7.242e-03 Score 99.763 Data time: 0.1869, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8713 [2021-07-09 09:03:44,281][root][INFO] - ===> Epoch[11352](56760/5): Loss 0.0063, Sem 0.0063 LR: 7.232e-03 Score 99.774 Data time: 0.2246, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6414 [2021-07-09 09:03:58,205][root][INFO] - ===> Epoch[11353](56765/5): Loss 0.0068, Sem 0.0068 LR: 7.222e-03 Score 99.759 Data time: 0.2473, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7364 [2021-07-09 09:04:12,581][root][INFO] - ===> Epoch[11354](56770/5): Loss 0.0065, Sem 0.0065 LR: 7.212e-03 Score 99.767 Data time: 0.2640, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8247 [2021-07-09 09:04:26,190][root][INFO] - ===> Epoch[11355](56775/5): Loss 0.0067, Sem 0.0067 LR: 7.202e-03 Score 99.763 Data time: 0.2680, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6777 [2021-07-09 09:04:40,257][root][INFO] - ===> Epoch[11356](56780/5): Loss 0.0067, Sem 0.0067 LR: 7.192e-03 Score 99.762 Data time: 0.2301, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7696 [2021-07-09 09:04:54,405][root][INFO] - ===> Epoch[11357](56785/5): Loss 0.0067, Sem 0.0067 LR: 7.182e-03 Score 99.763 Data time: 0.2965, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7828 [2021-07-09 09:05:07,477][root][INFO] - ===> Epoch[11358](56790/5): Loss 0.0066, Sem 0.0066 LR: 7.172e-03 Score 99.761 Data time: 0.1701, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.5730 [2021-07-09 09:05:22,002][root][INFO] - ===> Epoch[11359](56795/5): Loss 0.0064, Sem 0.0064 LR: 7.162e-03 Score 99.774 Data time: 0.1914, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8626 [2021-07-09 09:05:36,524][root][INFO] - ===> Epoch[11360](56800/5): Loss 0.0066, Sem 0.0066 LR: 7.152e-03 Score 99.765 Data time: 0.1588, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8567 [2021-07-09 09:05:36,525][root][INFO] - ===> Start testing [2021-07-09 09:06:10,803][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1832 Loss 0.399 (AVG: 0.631) Score 92.216 (AVG: 91.046) mIOU 71.758 mAP 82.263 mAcc 77.959 IOU: 64.194 0.000 80.389 74.965 95.137 91.547 33.819 85.291 98.591 85.125 79.934 86.110 57.748 mAP: 82.123 2.214 87.599 85.345 98.359 97.941 52.448 97.903 99.573 94.671 90.355 91.740 89.143 mAcc: 79.944 0.000 84.044 82.717 96.716 97.490 36.632 95.906 99.404 93.272 89.685 96.404 61.252 [2021-07-09 09:06:10,803][root][INFO] - Finished test. Elapsed time: 34.2775 [2021-07-09 09:06:11,993][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 09:06:25,260][root][INFO] - ===> Epoch[11361](56805/5): Loss 0.0067, Sem 0.0067 LR: 7.142e-03 Score 99.763 Data time: 0.1828, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6171 [2021-07-09 09:06:39,506][root][INFO] - ===> Epoch[11362](56810/5): Loss 0.0066, Sem 0.0066 LR: 7.132e-03 Score 99.762 Data time: 0.2139, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7950 [2021-07-09 09:06:55,561][root][INFO] - ===> Epoch[11363](56815/5): Loss 0.0066, Sem 0.0066 LR: 7.122e-03 Score 99.764 Data time: 0.3107, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 3.1591 [2021-07-09 09:07:09,543][root][INFO] - ===> Epoch[11364](56820/5): Loss 0.0065, Sem 0.0065 LR: 7.112e-03 Score 99.772 Data time: 0.2698, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7532 [2021-07-09 09:07:24,261][root][INFO] - ===> Epoch[11365](56825/5): Loss 0.0065, Sem 0.0065 LR: 7.101e-03 Score 99.769 Data time: 0.2525, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8961 [2021-07-09 09:07:38,641][root][INFO] - ===> Epoch[11366](56830/5): Loss 0.0064, Sem 0.0064 LR: 7.091e-03 Score 99.774 Data time: 0.1833, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8274 [2021-07-09 09:07:52,797][root][INFO] - ===> Epoch[11367](56835/5): Loss 0.0066, Sem 0.0066 LR: 7.081e-03 Score 99.762 Data time: 0.2546, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7813 [2021-07-09 09:08:06,783][root][INFO] - ===> Epoch[11368](56840/5): Loss 0.0066, Sem 0.0066 LR: 7.071e-03 Score 99.766 Data time: 0.2581, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7599 [2021-07-09 09:08:21,192][root][INFO] - ===> Epoch[11369](56845/5): Loss 0.0067, Sem 0.0067 LR: 7.061e-03 Score 99.758 Data time: 0.3287, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8390 [2021-07-09 09:08:35,112][root][INFO] - ===> Epoch[11370](56850/5): Loss 0.0070, Sem 0.0070 LR: 7.051e-03 Score 99.754 Data time: 0.2781, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7458 [2021-07-09 09:08:49,173][root][INFO] - ===> Epoch[11371](56855/5): Loss 0.0065, Sem 0.0065 LR: 7.041e-03 Score 99.771 Data time: 0.2585, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7700 [2021-07-09 09:09:02,812][root][INFO] - ===> Epoch[11372](56860/5): Loss 0.0070, Sem 0.0070 LR: 7.031e-03 Score 99.748 Data time: 0.2336, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.6861 [2021-07-09 09:09:16,792][root][INFO] - ===> Epoch[11373](56865/5): Loss 0.0071, Sem 0.0071 LR: 7.021e-03 Score 99.754 Data time: 0.2542, Forward time: 1.2518, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7494 [2021-07-09 09:09:30,391][root][INFO] - ===> Epoch[11374](56870/5): Loss 0.0066, Sem 0.0066 LR: 7.011e-03 Score 99.766 Data time: 0.1763, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6809 [2021-07-09 09:09:45,111][root][INFO] - ===> Epoch[11375](56875/5): Loss 0.0066, Sem 0.0066 LR: 7.001e-03 Score 99.766 Data time: 0.3056, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9047 [2021-07-09 09:09:58,854][root][INFO] - ===> Epoch[11376](56880/5): Loss 0.0067, Sem 0.0067 LR: 6.991e-03 Score 99.760 Data time: 0.2274, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7004 [2021-07-09 09:10:13,285][root][INFO] - ===> Epoch[11377](56885/5): Loss 0.0063, Sem 0.0063 LR: 6.981e-03 Score 99.776 Data time: 0.2246, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8386 [2021-07-09 09:10:27,561][root][INFO] - ===> Epoch[11378](56890/5): Loss 0.0066, Sem 0.0066 LR: 6.971e-03 Score 99.766 Data time: 0.2032, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8014 [2021-07-09 09:10:40,900][root][INFO] - ===> Epoch[11379](56895/5): Loss 0.0064, Sem 0.0064 LR: 6.960e-03 Score 99.775 Data time: 0.2396, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6192 [2021-07-09 09:10:54,576][root][INFO] - ===> Epoch[11380](56900/5): Loss 0.0061, Sem 0.0061 LR: 6.950e-03 Score 99.780 Data time: 0.2813, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6917 [2021-07-09 09:11:08,405][root][INFO] - ===> Epoch[11381](56905/5): Loss 0.0068, Sem 0.0068 LR: 6.940e-03 Score 99.757 Data time: 0.2041, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7187 [2021-07-09 09:11:22,218][root][INFO] - ===> Epoch[11382](56910/5): Loss 0.0064, Sem 0.0064 LR: 6.930e-03 Score 99.771 Data time: 0.2242, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7185 [2021-07-09 09:11:35,681][root][INFO] - ===> Epoch[11383](56915/5): Loss 0.0064, Sem 0.0064 LR: 6.920e-03 Score 99.771 Data time: 0.2332, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.6461 [2021-07-09 09:11:49,662][root][INFO] - ===> Epoch[11384](56920/5): Loss 0.0067, Sem 0.0067 LR: 6.910e-03 Score 99.758 Data time: 0.2114, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7386 [2021-07-09 09:12:03,825][root][INFO] - ===> Epoch[11385](56925/5): Loss 0.0067, Sem 0.0067 LR: 6.900e-03 Score 99.762 Data time: 0.1754, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7944 [2021-07-09 09:12:17,368][root][INFO] - ===> Epoch[11386](56930/5): Loss 0.0066, Sem 0.0066 LR: 6.890e-03 Score 99.763 Data time: 0.1736, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6706 [2021-07-09 09:12:30,929][root][INFO] - ===> Epoch[11387](56935/5): Loss 0.0068, Sem 0.0068 LR: 6.880e-03 Score 99.755 Data time: 0.2025, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6693 [2021-07-09 09:12:44,639][root][INFO] - ===> Epoch[11388](56940/5): Loss 0.0066, Sem 0.0066 LR: 6.870e-03 Score 99.765 Data time: 0.2066, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.6855 [2021-07-09 09:12:58,629][root][INFO] - ===> Epoch[11389](56945/5): Loss 0.0063, Sem 0.0063 LR: 6.860e-03 Score 99.774 Data time: 0.2464, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7512 [2021-07-09 09:13:12,018][root][INFO] - ===> Epoch[11390](56950/5): Loss 0.0061, Sem 0.0061 LR: 6.849e-03 Score 99.779 Data time: 0.1967, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.6372 [2021-07-09 09:13:26,242][root][INFO] - ===> Epoch[11391](56955/5): Loss 0.0066, Sem 0.0066 LR: 6.839e-03 Score 99.763 Data time: 0.2263, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7981 [2021-07-09 09:13:40,565][root][INFO] - ===> Epoch[11392](56960/5): Loss 0.0066, Sem 0.0066 LR: 6.829e-03 Score 99.766 Data time: 0.2352, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.8089 [2021-07-09 09:13:54,377][root][INFO] - ===> Epoch[11393](56965/5): Loss 0.0065, Sem 0.0065 LR: 6.819e-03 Score 99.767 Data time: 0.1764, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7162 [2021-07-09 09:14:08,531][root][INFO] - ===> Epoch[11394](56970/5): Loss 0.0066, Sem 0.0066 LR: 6.809e-03 Score 99.765 Data time: 0.1797, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7918 [2021-07-09 09:14:22,548][root][INFO] - ===> Epoch[11395](56975/5): Loss 0.0066, Sem 0.0066 LR: 6.799e-03 Score 99.764 Data time: 0.1879, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7565 [2021-07-09 09:14:35,910][root][INFO] - ===> Epoch[11396](56980/5): Loss 0.0067, Sem 0.0067 LR: 6.789e-03 Score 99.758 Data time: 0.2513, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.6238 [2021-07-09 09:14:49,593][root][INFO] - ===> Epoch[11397](56985/5): Loss 0.0065, Sem 0.0065 LR: 6.779e-03 Score 99.769 Data time: 0.2218, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.6965 [2021-07-09 09:15:03,497][root][INFO] - ===> Epoch[11398](56990/5): Loss 0.0067, Sem 0.0067 LR: 6.769e-03 Score 99.762 Data time: 0.2250, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7258 [2021-07-09 09:15:16,657][root][INFO] - ===> Epoch[11399](56995/5): Loss 0.0066, Sem 0.0066 LR: 6.758e-03 Score 99.766 Data time: 0.1939, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.5831 [2021-07-09 09:15:30,712][root][INFO] - ===> Epoch[11400](57000/5): Loss 0.0069, Sem 0.0069 LR: 6.748e-03 Score 99.751 Data time: 0.2222, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.7649 [2021-07-09 09:15:30,714][root][INFO] - ===> Start testing [2021-07-09 09:16:12,295][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1318 Loss 0.440 (AVG: 0.638) Score 91.670 (AVG: 91.073) mIOU 71.729 mAP 82.224 mAcc 77.900 IOU: 64.318 0.000 80.111 75.128 95.213 91.305 33.270 84.630 98.622 86.003 80.034 86.188 57.650 mAP: 82.302 1.947 88.174 85.157 98.267 97.941 52.103 97.848 99.571 94.694 90.207 91.959 88.749 mAcc: 79.617 0.000 84.146 83.023 96.811 97.421 35.856 95.422 99.422 93.538 89.874 96.515 61.048 [2021-07-09 09:16:12,296][root][INFO] - Finished test. Elapsed time: 41.5812 [2021-07-09 09:16:13,404][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 09:16:26,058][root][INFO] - ===> Epoch[11401](57005/5): Loss 0.0066, Sem 0.0066 LR: 6.738e-03 Score 99.766 Data time: 0.1666, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.4956 [2021-07-09 09:16:40,690][root][INFO] - ===> Epoch[11402](57010/5): Loss 0.0064, Sem 0.0064 LR: 6.728e-03 Score 99.770 Data time: 0.2314, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.8773 [2021-07-09 09:16:55,428][root][INFO] - ===> Epoch[11403](57015/5): Loss 0.0064, Sem 0.0064 LR: 6.718e-03 Score 99.772 Data time: 0.2202, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9007 [2021-07-09 09:17:08,574][root][INFO] - ===> Epoch[11404](57020/5): Loss 0.0065, Sem 0.0065 LR: 6.708e-03 Score 99.766 Data time: 0.2643, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.5698 [2021-07-09 09:17:23,573][root][INFO] - ===> Epoch[11405](57025/5): Loss 0.0067, Sem 0.0067 LR: 6.698e-03 Score 99.766 Data time: 0.2329, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9513 [2021-07-09 09:17:37,277][root][INFO] - ===> Epoch[11406](57030/5): Loss 0.0065, Sem 0.0065 LR: 6.688e-03 Score 99.767 Data time: 0.2275, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7008 [2021-07-09 09:17:51,757][root][INFO] - ===> Epoch[11407](57035/5): Loss 0.0065, Sem 0.0065 LR: 6.677e-03 Score 99.767 Data time: 0.2762, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8335 [2021-07-09 09:18:05,460][root][INFO] - ===> Epoch[11408](57040/5): Loss 0.0065, Sem 0.0065 LR: 6.667e-03 Score 99.768 Data time: 0.2253, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6857 [2021-07-09 09:18:19,246][root][INFO] - ===> Epoch[11409](57045/5): Loss 0.0068, Sem 0.0068 LR: 6.657e-03 Score 99.757 Data time: 0.2322, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7108 [2021-07-09 09:18:32,857][root][INFO] - ===> Epoch[11410](57050/5): Loss 0.0065, Sem 0.0065 LR: 6.647e-03 Score 99.769 Data time: 0.1803, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.6849 [2021-07-09 09:18:47,471][root][INFO] - ===> Epoch[11411](57055/5): Loss 0.0065, Sem 0.0065 LR: 6.637e-03 Score 99.772 Data time: 0.1940, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8806 [2021-07-09 09:19:01,587][root][INFO] - ===> Epoch[11412](57060/5): Loss 0.0064, Sem 0.0064 LR: 6.627e-03 Score 99.772 Data time: 0.1697, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7824 [2021-07-09 09:19:15,095][root][INFO] - ===> Epoch[11413](57065/5): Loss 0.0067, Sem 0.0067 LR: 6.617e-03 Score 99.763 Data time: 0.2469, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.6581 [2021-07-09 09:19:29,127][root][INFO] - ===> Epoch[11414](57070/5): Loss 0.0065, Sem 0.0065 LR: 6.607e-03 Score 99.769 Data time: 0.2164, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7682 [2021-07-09 09:19:42,387][root][INFO] - ===> Epoch[11415](57075/5): Loss 0.0065, Sem 0.0065 LR: 6.596e-03 Score 99.769 Data time: 0.1710, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.6107 [2021-07-09 09:19:56,801][root][INFO] - ===> Epoch[11416](57080/5): Loss 0.0067, Sem 0.0067 LR: 6.586e-03 Score 99.757 Data time: 0.2448, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.8268 [2021-07-09 09:20:11,026][root][INFO] - ===> Epoch[11417](57085/5): Loss 0.0065, Sem 0.0065 LR: 6.576e-03 Score 99.770 Data time: 0.1995, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.8015 [2021-07-09 09:20:24,709][root][INFO] - ===> Epoch[11418](57090/5): Loss 0.0068, Sem 0.0068 LR: 6.566e-03 Score 99.759 Data time: 0.2510, Forward time: 1.2517, Backward time: 1.2484, DDP time: 0.0042, Total iter time: 2.6969 [2021-07-09 09:20:39,609][root][INFO] - ===> Epoch[11419](57095/5): Loss 0.0062, Sem 0.0062 LR: 6.556e-03 Score 99.778 Data time: 0.1975, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9340 [2021-07-09 09:20:54,155][root][INFO] - ===> Epoch[11420](57100/5): Loss 0.0067, Sem 0.0067 LR: 6.546e-03 Score 99.762 Data time: 0.2446, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8705 [2021-07-09 09:21:09,675][root][INFO] - ===> Epoch[11421](57105/5): Loss 0.0064, Sem 0.0064 LR: 6.535e-03 Score 99.773 Data time: 0.2672, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 3.0508 [2021-07-09 09:21:24,602][root][INFO] - ===> Epoch[11422](57110/5): Loss 0.0067, Sem 0.0067 LR: 6.525e-03 Score 99.762 Data time: 0.2004, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9474 [2021-07-09 09:21:38,624][root][INFO] - ===> Epoch[11423](57115/5): Loss 0.0063, Sem 0.0063 LR: 6.515e-03 Score 99.776 Data time: 0.2476, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7643 [2021-07-09 09:21:53,922][root][INFO] - ===> Epoch[11424](57120/5): Loss 0.0065, Sem 0.0065 LR: 6.505e-03 Score 99.769 Data time: 0.2104, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 3.0192 [2021-07-09 09:22:07,940][root][INFO] - ===> Epoch[11425](57125/5): Loss 0.0065, Sem 0.0065 LR: 6.495e-03 Score 99.769 Data time: 0.2821, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7500 [2021-07-09 09:22:21,757][root][INFO] - ===> Epoch[11426](57130/5): Loss 0.0068, Sem 0.0068 LR: 6.485e-03 Score 99.757 Data time: 0.1788, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7157 [2021-07-09 09:22:37,155][root][INFO] - ===> Epoch[11427](57135/5): Loss 0.0065, Sem 0.0065 LR: 6.474e-03 Score 99.769 Data time: 0.2480, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 3.0348 [2021-07-09 09:22:51,834][root][INFO] - ===> Epoch[11428](57140/5): Loss 0.0066, Sem 0.0066 LR: 6.464e-03 Score 99.764 Data time: 0.2411, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8836 [2021-07-09 09:23:06,317][root][INFO] - ===> Epoch[11429](57145/5): Loss 0.0067, Sem 0.0067 LR: 6.454e-03 Score 99.760 Data time: 0.2103, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8571 [2021-07-09 09:23:20,357][root][INFO] - ===> Epoch[11430](57150/5): Loss 0.0062, Sem 0.0062 LR: 6.444e-03 Score 99.778 Data time: 0.2158, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7734 [2021-07-09 09:23:34,412][root][INFO] - ===> Epoch[11431](57155/5): Loss 0.0068, Sem 0.0068 LR: 6.434e-03 Score 99.758 Data time: 0.2227, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7550 [2021-07-09 09:23:48,369][root][INFO] - ===> Epoch[11432](57160/5): Loss 0.0066, Sem 0.0066 LR: 6.424e-03 Score 99.764 Data time: 0.2376, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7420 [2021-07-09 09:24:02,600][root][INFO] - ===> Epoch[11433](57165/5): Loss 0.0064, Sem 0.0064 LR: 6.413e-03 Score 99.770 Data time: 0.2573, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8040 [2021-07-09 09:24:17,559][root][INFO] - ===> Epoch[11434](57170/5): Loss 0.0060, Sem 0.0060 LR: 6.403e-03 Score 99.783 Data time: 0.2344, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9376 [2021-07-09 09:24:31,148][root][INFO] - ===> Epoch[11435](57175/5): Loss 0.0064, Sem 0.0064 LR: 6.393e-03 Score 99.769 Data time: 0.2881, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6659 [2021-07-09 09:24:45,284][root][INFO] - ===> Epoch[11436](57180/5): Loss 0.0069, Sem 0.0069 LR: 6.383e-03 Score 99.754 Data time: 0.2746, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7766 [2021-07-09 09:25:00,005][root][INFO] - ===> Epoch[11437](57185/5): Loss 0.0064, Sem 0.0064 LR: 6.373e-03 Score 99.774 Data time: 0.2002, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8979 [2021-07-09 09:25:14,327][root][INFO] - ===> Epoch[11438](57190/5): Loss 0.0067, Sem 0.0067 LR: 6.363e-03 Score 99.761 Data time: 0.2540, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8025 [2021-07-09 09:25:28,621][root][INFO] - ===> Epoch[11439](57195/5): Loss 0.0063, Sem 0.0063 LR: 6.352e-03 Score 99.775 Data time: 0.3095, Forward time: 1.2518, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8099 [2021-07-09 09:25:41,618][root][INFO] - ===> Epoch[11440](57200/5): Loss 0.0067, Sem 0.0067 LR: 6.342e-03 Score 99.762 Data time: 0.2309, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.5484 [2021-07-09 09:25:41,619][root][INFO] - ===> Start testing [2021-07-09 09:26:16,005][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1302 Loss 0.474 (AVG: 0.636) Score 91.217 (AVG: 91.117) mIOU 71.849 mAP 82.417 mAcc 78.170 IOU: 64.469 0.000 80.557 75.211 95.155 91.109 35.826 84.907 98.583 84.091 79.923 86.304 57.903 mAP: 82.439 2.074 87.994 84.955 98.295 97.829 55.156 97.778 99.598 94.415 90.115 91.801 88.970 mAcc: 79.690 0.000 84.316 82.989 96.747 97.484 39.175 96.183 99.366 92.561 90.209 96.389 61.097 [2021-07-09 09:26:16,005][root][INFO] - Finished test. Elapsed time: 34.3851 [2021-07-09 09:26:17,078][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 09:26:29,685][root][INFO] - ===> Epoch[11441](57205/5): Loss 0.0064, Sem 0.0064 LR: 6.332e-03 Score 99.770 Data time: 0.2067, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.4816 [2021-07-09 09:26:43,241][root][INFO] - ===> Epoch[11442](57210/5): Loss 0.0065, Sem 0.0065 LR: 6.322e-03 Score 99.765 Data time: 0.2614, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6621 [2021-07-09 09:26:56,815][root][INFO] - ===> Epoch[11443](57215/5): Loss 0.0066, Sem 0.0066 LR: 6.312e-03 Score 99.767 Data time: 0.1914, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6730 [2021-07-09 09:27:10,469][root][INFO] - ===> Epoch[11444](57220/5): Loss 0.0066, Sem 0.0066 LR: 6.301e-03 Score 99.765 Data time: 0.2121, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6858 [2021-07-09 09:27:24,591][root][INFO] - ===> Epoch[11445](57225/5): Loss 0.0066, Sem 0.0066 LR: 6.291e-03 Score 99.764 Data time: 0.2183, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7765 [2021-07-09 09:27:38,315][root][INFO] - ===> Epoch[11446](57230/5): Loss 0.0065, Sem 0.0065 LR: 6.281e-03 Score 99.766 Data time: 0.2294, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6902 [2021-07-09 09:27:51,778][root][INFO] - ===> Epoch[11447](57235/5): Loss 0.0066, Sem 0.0066 LR: 6.271e-03 Score 99.767 Data time: 0.2408, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6452 [2021-07-09 09:28:06,111][root][INFO] - ===> Epoch[11448](57240/5): Loss 0.0065, Sem 0.0065 LR: 6.261e-03 Score 99.766 Data time: 0.2265, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8147 [2021-07-09 09:28:20,841][root][INFO] - ===> Epoch[11449](57245/5): Loss 0.0066, Sem 0.0066 LR: 6.250e-03 Score 99.763 Data time: 0.2505, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8859 [2021-07-09 09:28:34,960][root][INFO] - ===> Epoch[11450](57250/5): Loss 0.0063, Sem 0.0063 LR: 6.240e-03 Score 99.774 Data time: 0.2523, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7795 [2021-07-09 09:28:49,115][root][INFO] - ===> Epoch[11451](57255/5): Loss 0.0064, Sem 0.0064 LR: 6.230e-03 Score 99.773 Data time: 0.2666, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7847 [2021-07-09 09:29:03,231][root][INFO] - ===> Epoch[11452](57260/5): Loss 0.0066, Sem 0.0066 LR: 6.220e-03 Score 99.766 Data time: 0.2752, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7671 [2021-07-09 09:29:16,975][root][INFO] - ===> Epoch[11453](57265/5): Loss 0.0064, Sem 0.0064 LR: 6.210e-03 Score 99.773 Data time: 0.2401, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7052 [2021-07-09 09:29:32,069][root][INFO] - ===> Epoch[11454](57270/5): Loss 0.0066, Sem 0.0066 LR: 6.199e-03 Score 99.763 Data time: 0.1914, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9687 [2021-07-09 09:29:46,610][root][INFO] - ===> Epoch[11455](57275/5): Loss 0.0065, Sem 0.0065 LR: 6.189e-03 Score 99.770 Data time: 0.1898, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8665 [2021-07-09 09:30:01,302][root][INFO] - ===> Epoch[11456](57280/5): Loss 0.0064, Sem 0.0064 LR: 6.179e-03 Score 99.773 Data time: 0.2019, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8839 [2021-07-09 09:30:15,384][root][INFO] - ===> Epoch[11457](57285/5): Loss 0.0063, Sem 0.0063 LR: 6.169e-03 Score 99.775 Data time: 0.1710, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7745 [2021-07-09 09:30:29,288][root][INFO] - ===> Epoch[11458](57290/5): Loss 0.0064, Sem 0.0064 LR: 6.158e-03 Score 99.768 Data time: 0.2253, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7253 [2021-07-09 09:30:43,276][root][INFO] - ===> Epoch[11459](57295/5): Loss 0.0065, Sem 0.0065 LR: 6.148e-03 Score 99.768 Data time: 0.1749, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7317 [2021-07-09 09:30:57,763][root][INFO] - ===> Epoch[11460](57300/5): Loss 0.0064, Sem 0.0064 LR: 6.138e-03 Score 99.771 Data time: 0.1554, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8553 [2021-07-09 09:31:11,247][root][INFO] - ===> Epoch[11461](57305/5): Loss 0.0069, Sem 0.0069 LR: 6.128e-03 Score 99.753 Data time: 0.1355, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6401 [2021-07-09 09:31:24,384][root][INFO] - ===> Epoch[11462](57310/5): Loss 0.0066, Sem 0.0066 LR: 6.118e-03 Score 99.762 Data time: 0.2042, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.5959 [2021-07-09 09:31:38,081][root][INFO] - ===> Epoch[11463](57315/5): Loss 0.0068, Sem 0.0068 LR: 6.107e-03 Score 99.757 Data time: 0.1562, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7017 [2021-07-09 09:31:51,522][root][INFO] - ===> Epoch[11464](57320/5): Loss 0.0063, Sem 0.0063 LR: 6.097e-03 Score 99.777 Data time: 0.2223, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6321 [2021-07-09 09:32:05,686][root][INFO] - ===> Epoch[11465](57325/5): Loss 0.0067, Sem 0.0067 LR: 6.087e-03 Score 99.762 Data time: 0.1920, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7840 [2021-07-09 09:32:18,857][root][INFO] - ===> Epoch[11466](57330/5): Loss 0.0064, Sem 0.0064 LR: 6.077e-03 Score 99.770 Data time: 0.2148, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.5926 [2021-07-09 09:32:33,041][root][INFO] - ===> Epoch[11467](57335/5): Loss 0.0068, Sem 0.0068 LR: 6.066e-03 Score 99.753 Data time: 0.1966, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7907 [2021-07-09 09:32:46,221][root][INFO] - ===> Epoch[11468](57340/5): Loss 0.0066, Sem 0.0066 LR: 6.056e-03 Score 99.766 Data time: 0.2245, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.5848 [2021-07-09 09:33:00,105][root][INFO] - ===> Epoch[11469](57345/5): Loss 0.0061, Sem 0.0061 LR: 6.046e-03 Score 99.782 Data time: 0.1868, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7346 [2021-07-09 09:33:14,310][root][INFO] - ===> Epoch[11470](57350/5): Loss 0.0065, Sem 0.0065 LR: 6.036e-03 Score 99.768 Data time: 0.2553, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7974 [2021-07-09 09:33:27,413][root][INFO] - ===> Epoch[11471](57355/5): Loss 0.0063, Sem 0.0063 LR: 6.025e-03 Score 99.773 Data time: 0.2332, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.5748 [2021-07-09 09:33:42,135][root][INFO] - ===> Epoch[11472](57360/5): Loss 0.0064, Sem 0.0064 LR: 6.015e-03 Score 99.775 Data time: 0.2080, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9019 [2021-07-09 09:33:56,146][root][INFO] - ===> Epoch[11473](57365/5): Loss 0.0067, Sem 0.0067 LR: 6.005e-03 Score 99.762 Data time: 0.2152, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7529 [2021-07-09 09:34:09,870][root][INFO] - ===> Epoch[11474](57370/5): Loss 0.0063, Sem 0.0063 LR: 5.995e-03 Score 99.774 Data time: 0.2079, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6982 [2021-07-09 09:34:24,905][root][INFO] - ===> Epoch[11475](57375/5): Loss 0.0066, Sem 0.0066 LR: 5.984e-03 Score 99.766 Data time: 0.2539, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9539 [2021-07-09 09:34:38,790][root][INFO] - ===> Epoch[11476](57380/5): Loss 0.0068, Sem 0.0068 LR: 5.974e-03 Score 99.759 Data time: 0.2640, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7250 [2021-07-09 09:34:52,545][root][INFO] - ===> Epoch[11477](57385/5): Loss 0.0065, Sem 0.0065 LR: 5.964e-03 Score 99.769 Data time: 0.2001, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7152 [2021-07-09 09:35:07,125][root][INFO] - ===> Epoch[11478](57390/5): Loss 0.0063, Sem 0.0063 LR: 5.954e-03 Score 99.774 Data time: 0.3501, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8575 [2021-07-09 09:35:21,202][root][INFO] - ===> Epoch[11479](57395/5): Loss 0.0063, Sem 0.0063 LR: 5.943e-03 Score 99.772 Data time: 0.2716, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7595 [2021-07-09 09:35:35,871][root][INFO] - ===> Epoch[11480](57400/5): Loss 0.0063, Sem 0.0063 LR: 5.933e-03 Score 99.778 Data time: 0.3571, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8718 [2021-07-09 09:35:35,872][root][INFO] - ===> Start testing [2021-07-09 09:36:11,080][root][INFO] - 68/68: Data time: 0.0016, Iter time: 0.1253 Loss 0.397 (AVG: 0.633) Score 92.447 (AVG: 91.175) mIOU 72.075 mAP 82.427 mAcc 78.135 IOU: 64.622 0.000 80.717 75.295 95.095 91.742 35.174 85.890 98.616 85.117 80.167 86.312 58.227 mAP: 83.155 2.380 87.312 85.140 98.250 97.900 54.665 97.885 99.539 94.383 90.000 91.759 89.182 mAcc: 79.966 0.000 83.426 83.352 96.707 97.388 38.251 95.472 99.451 93.272 90.117 96.411 61.941 [2021-07-09 09:36:11,081][root][INFO] - Finished test. Elapsed time: 35.2082 [2021-07-09 09:36:12,115][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 09:36:25,408][root][INFO] - ===> Epoch[11481](57405/5): Loss 0.0065, Sem 0.0065 LR: 5.923e-03 Score 99.766 Data time: 0.1954, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6227 [2021-07-09 09:36:40,267][root][INFO] - ===> Epoch[11482](57410/5): Loss 0.0066, Sem 0.0066 LR: 5.913e-03 Score 99.763 Data time: 0.2441, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9234 [2021-07-09 09:36:53,779][root][INFO] - ===> Epoch[11483](57415/5): Loss 0.0065, Sem 0.0065 LR: 5.902e-03 Score 99.767 Data time: 0.2710, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6430 [2021-07-09 09:37:09,167][root][INFO] - ===> Epoch[11484](57420/5): Loss 0.0063, Sem 0.0063 LR: 5.892e-03 Score 99.774 Data time: 0.2982, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 3.0224 [2021-07-09 09:37:23,257][root][INFO] - ===> Epoch[11485](57425/5): Loss 0.0064, Sem 0.0064 LR: 5.882e-03 Score 99.775 Data time: 0.2606, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7651 [2021-07-09 09:37:37,288][root][INFO] - ===> Epoch[11486](57430/5): Loss 0.0065, Sem 0.0065 LR: 5.872e-03 Score 99.769 Data time: 0.2479, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7679 [2021-07-09 09:37:50,616][root][INFO] - ===> Epoch[11487](57435/5): Loss 0.0066, Sem 0.0066 LR: 5.861e-03 Score 99.763 Data time: 0.2261, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6214 [2021-07-09 09:38:04,799][root][INFO] - ===> Epoch[11488](57440/5): Loss 0.0064, Sem 0.0064 LR: 5.851e-03 Score 99.776 Data time: 0.2099, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7843 [2021-07-09 09:38:18,151][root][INFO] - ===> Epoch[11489](57445/5): Loss 0.0066, Sem 0.0066 LR: 5.841e-03 Score 99.766 Data time: 0.1712, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6326 [2021-07-09 09:38:31,965][root][INFO] - ===> Epoch[11490](57450/5): Loss 0.0065, Sem 0.0065 LR: 5.830e-03 Score 99.767 Data time: 0.2714, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7255 [2021-07-09 09:38:46,178][root][INFO] - ===> Epoch[11491](57455/5): Loss 0.0064, Sem 0.0064 LR: 5.820e-03 Score 99.769 Data time: 0.2429, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8029 [2021-07-09 09:39:00,510][root][INFO] - ===> Epoch[11492](57460/5): Loss 0.0062, Sem 0.0062 LR: 5.810e-03 Score 99.779 Data time: 0.2106, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8165 [2021-07-09 09:39:14,341][root][INFO] - ===> Epoch[11493](57465/5): Loss 0.0065, Sem 0.0065 LR: 5.800e-03 Score 99.766 Data time: 0.2222, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7282 [2021-07-09 09:39:28,258][root][INFO] - ===> Epoch[11494](57470/5): Loss 0.0066, Sem 0.0066 LR: 5.789e-03 Score 99.768 Data time: 0.2162, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7382 [2021-07-09 09:39:42,495][root][INFO] - ===> Epoch[11495](57475/5): Loss 0.0061, Sem 0.0061 LR: 5.779e-03 Score 99.780 Data time: 0.2444, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8011 [2021-07-09 09:39:56,910][root][INFO] - ===> Epoch[11496](57480/5): Loss 0.0065, Sem 0.0065 LR: 5.769e-03 Score 99.769 Data time: 0.2455, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8406 [2021-07-09 09:40:12,150][root][INFO] - ===> Epoch[11497](57485/5): Loss 0.0065, Sem 0.0065 LR: 5.758e-03 Score 99.766 Data time: 0.1810, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.9981 [2021-07-09 09:40:25,830][root][INFO] - ===> Epoch[11498](57490/5): Loss 0.0065, Sem 0.0065 LR: 5.748e-03 Score 99.767 Data time: 0.2131, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6819 [2021-07-09 09:40:39,967][root][INFO] - ===> Epoch[11499](57495/5): Loss 0.0067, Sem 0.0067 LR: 5.738e-03 Score 99.761 Data time: 0.1850, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7895 [2021-07-09 09:40:53,945][root][INFO] - ===> Epoch[11500](57500/5): Loss 0.0065, Sem 0.0065 LR: 5.727e-03 Score 99.766 Data time: 0.1802, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7519 [2021-07-09 09:41:07,418][root][INFO] - ===> Epoch[11501](57505/5): Loss 0.0066, Sem 0.0066 LR: 5.717e-03 Score 99.764 Data time: 0.1948, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6488 [2021-07-09 09:41:21,390][root][INFO] - ===> Epoch[11502](57510/5): Loss 0.0067, Sem 0.0067 LR: 5.707e-03 Score 99.762 Data time: 0.1924, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7464 [2021-07-09 09:41:35,471][root][INFO] - ===> Epoch[11503](57515/5): Loss 0.0065, Sem 0.0065 LR: 5.697e-03 Score 99.766 Data time: 0.2255, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7754 [2021-07-09 09:41:49,913][root][INFO] - ===> Epoch[11504](57520/5): Loss 0.0063, Sem 0.0063 LR: 5.686e-03 Score 99.775 Data time: 0.2449, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8438 [2021-07-09 09:42:05,312][root][INFO] - ===> Epoch[11505](57525/5): Loss 0.0065, Sem 0.0065 LR: 5.676e-03 Score 99.769 Data time: 0.2520, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 3.0294 [2021-07-09 09:42:18,747][root][INFO] - ===> Epoch[11506](57530/5): Loss 0.0067, Sem 0.0067 LR: 5.666e-03 Score 99.761 Data time: 0.1979, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6391 [2021-07-09 09:42:32,353][root][INFO] - ===> Epoch[11507](57535/5): Loss 0.0062, Sem 0.0062 LR: 5.655e-03 Score 99.778 Data time: 0.1731, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6761 [2021-07-09 09:42:46,642][root][INFO] - ===> Epoch[11508](57540/5): Loss 0.0064, Sem 0.0064 LR: 5.645e-03 Score 99.773 Data time: 0.2657, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8072 [2021-07-09 09:43:00,492][root][INFO] - ===> Epoch[11509](57545/5): Loss 0.0063, Sem 0.0063 LR: 5.635e-03 Score 99.774 Data time: 0.2220, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7304 [2021-07-09 09:43:14,051][root][INFO] - ===> Epoch[11510](57550/5): Loss 0.0065, Sem 0.0065 LR: 5.624e-03 Score 99.767 Data time: 0.2300, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6670 [2021-07-09 09:43:28,675][root][INFO] - ===> Epoch[11511](57555/5): Loss 0.0064, Sem 0.0064 LR: 5.614e-03 Score 99.769 Data time: 0.2490, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8877 [2021-07-09 09:43:41,873][root][INFO] - ===> Epoch[11512](57560/5): Loss 0.0064, Sem 0.0064 LR: 5.604e-03 Score 99.770 Data time: 0.2268, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.5977 [2021-07-09 09:43:55,844][root][INFO] - ===> Epoch[11513](57565/5): Loss 0.0069, Sem 0.0069 LR: 5.593e-03 Score 99.756 Data time: 0.2071, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7526 [2021-07-09 09:44:08,881][root][INFO] - ===> Epoch[11514](57570/5): Loss 0.0063, Sem 0.0063 LR: 5.583e-03 Score 99.779 Data time: 0.1773, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.5778 [2021-07-09 09:44:23,242][root][INFO] - ===> Epoch[11515](57575/5): Loss 0.0065, Sem 0.0065 LR: 5.573e-03 Score 99.769 Data time: 0.1895, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8314 [2021-07-09 09:44:36,870][root][INFO] - ===> Epoch[11516](57580/5): Loss 0.0065, Sem 0.0065 LR: 5.562e-03 Score 99.768 Data time: 0.2787, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6827 [2021-07-09 09:44:51,375][root][INFO] - ===> Epoch[11517](57585/5): Loss 0.0066, Sem 0.0066 LR: 5.552e-03 Score 99.766 Data time: 0.2225, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8489 [2021-07-09 09:45:04,723][root][INFO] - ===> Epoch[11518](57590/5): Loss 0.0066, Sem 0.0066 LR: 5.542e-03 Score 99.766 Data time: 0.2449, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6180 [2021-07-09 09:45:18,307][root][INFO] - ===> Epoch[11519](57595/5): Loss 0.0066, Sem 0.0066 LR: 5.531e-03 Score 99.761 Data time: 0.2521, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6747 [2021-07-09 09:45:32,596][root][INFO] - ===> Epoch[11520](57600/5): Loss 0.0065, Sem 0.0065 LR: 5.521e-03 Score 99.766 Data time: 0.2148, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8106 [2021-07-09 09:45:32,597][root][INFO] - ===> Start testing [2021-07-09 09:46:05,983][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1258 Loss 0.415 (AVG: 0.630) Score 92.325 (AVG: 91.128) mIOU 71.971 mAP 82.323 mAcc 78.048 IOU: 64.341 0.000 80.360 75.218 95.162 91.439 35.192 85.431 98.604 85.155 80.526 86.231 57.969 mAP: 82.143 1.956 87.187 85.266 98.252 97.915 54.542 97.897 99.608 94.173 90.418 91.965 88.871 mAcc: 79.731 0.000 83.390 83.051 96.821 97.492 38.474 95.608 99.398 92.758 89.971 96.449 61.479 [2021-07-09 09:46:05,983][root][INFO] - Finished test. Elapsed time: 33.3852 [2021-07-09 09:46:07,003][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 09:46:20,660][root][INFO] - ===> Epoch[11521](57605/5): Loss 0.0063, Sem 0.0063 LR: 5.511e-03 Score 99.773 Data time: 0.2098, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6874 [2021-07-09 09:46:34,833][root][INFO] - ===> Epoch[11522](57610/5): Loss 0.0066, Sem 0.0066 LR: 5.500e-03 Score 99.763 Data time: 0.2372, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7834 [2021-07-09 09:46:48,476][root][INFO] - ===> Epoch[11523](57615/5): Loss 0.0061, Sem 0.0061 LR: 5.490e-03 Score 99.782 Data time: 0.2061, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6727 [2021-07-09 09:47:02,121][root][INFO] - ===> Epoch[11524](57620/5): Loss 0.0064, Sem 0.0064 LR: 5.479e-03 Score 99.772 Data time: 0.2049, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6864 [2021-07-09 09:47:16,310][root][INFO] - ===> Epoch[11525](57625/5): Loss 0.0066, Sem 0.0066 LR: 5.469e-03 Score 99.765 Data time: 0.2213, Forward time: 1.2517, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7954 [2021-07-09 09:47:32,070][root][INFO] - ===> Epoch[11526](57630/5): Loss 0.0063, Sem 0.0063 LR: 5.459e-03 Score 99.773 Data time: 0.2130, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 3.1094 [2021-07-09 09:47:44,877][root][INFO] - ===> Epoch[11527](57635/5): Loss 0.0061, Sem 0.0061 LR: 5.448e-03 Score 99.782 Data time: 0.2291, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.5058 [2021-07-09 09:47:59,787][root][INFO] - ===> Epoch[11528](57640/5): Loss 0.0065, Sem 0.0065 LR: 5.438e-03 Score 99.766 Data time: 0.2096, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9356 [2021-07-09 09:48:14,348][root][INFO] - ===> Epoch[11529](57645/5): Loss 0.0062, Sem 0.0062 LR: 5.428e-03 Score 99.777 Data time: 0.2086, Forward time: 1.2517, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8724 [2021-07-09 09:48:28,041][root][INFO] - ===> Epoch[11530](57650/5): Loss 0.0066, Sem 0.0066 LR: 5.417e-03 Score 99.760 Data time: 0.2257, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6897 [2021-07-09 09:48:41,775][root][INFO] - ===> Epoch[11531](57655/5): Loss 0.0068, Sem 0.0068 LR: 5.407e-03 Score 99.759 Data time: 0.1793, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7048 [2021-07-09 09:48:55,859][root][INFO] - ===> Epoch[11532](57660/5): Loss 0.0064, Sem 0.0064 LR: 5.397e-03 Score 99.769 Data time: 0.2506, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7665 [2021-07-09 09:49:09,041][root][INFO] - ===> Epoch[11533](57665/5): Loss 0.0064, Sem 0.0064 LR: 5.386e-03 Score 99.770 Data time: 0.1677, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.5963 [2021-07-09 09:49:23,396][root][INFO] - ===> Epoch[11534](57670/5): Loss 0.0064, Sem 0.0064 LR: 5.376e-03 Score 99.769 Data time: 0.2546, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8254 [2021-07-09 09:49:37,088][root][INFO] - ===> Epoch[11535](57675/5): Loss 0.0062, Sem 0.0062 LR: 5.365e-03 Score 99.776 Data time: 0.2388, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6890 [2021-07-09 09:49:52,533][root][INFO] - ===> Epoch[11536](57680/5): Loss 0.0066, Sem 0.0066 LR: 5.355e-03 Score 99.767 Data time: 0.2877, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 3.0289 [2021-07-09 09:50:06,474][root][INFO] - ===> Epoch[11537](57685/5): Loss 0.0066, Sem 0.0066 LR: 5.345e-03 Score 99.764 Data time: 0.1405, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7508 [2021-07-09 09:50:20,621][root][INFO] - ===> Epoch[11538](57690/5): Loss 0.0066, Sem 0.0066 LR: 5.334e-03 Score 99.761 Data time: 0.2072, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7883 [2021-07-09 09:50:33,970][root][INFO] - ===> Epoch[11539](57695/5): Loss 0.0070, Sem 0.0070 LR: 5.324e-03 Score 99.749 Data time: 0.2671, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6306 [2021-07-09 09:50:47,594][root][INFO] - ===> Epoch[11540](57700/5): Loss 0.0064, Sem 0.0064 LR: 5.314e-03 Score 99.776 Data time: 0.2324, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6710 [2021-07-09 09:51:01,240][root][INFO] - ===> Epoch[11541](57705/5): Loss 0.0067, Sem 0.0067 LR: 5.303e-03 Score 99.761 Data time: 0.3045, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6814 [2021-07-09 09:51:15,730][root][INFO] - ===> Epoch[11542](57710/5): Loss 0.0061, Sem 0.0061 LR: 5.293e-03 Score 99.784 Data time: 0.2802, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8475 [2021-07-09 09:51:30,039][root][INFO] - ===> Epoch[11543](57715/5): Loss 0.0064, Sem 0.0064 LR: 5.282e-03 Score 99.768 Data time: 0.2814, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.8078 [2021-07-09 09:51:44,706][root][INFO] - ===> Epoch[11544](57720/5): Loss 0.0064, Sem 0.0064 LR: 5.272e-03 Score 99.772 Data time: 0.2305, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8859 [2021-07-09 09:51:58,439][root][INFO] - ===> Epoch[11545](57725/5): Loss 0.0066, Sem 0.0066 LR: 5.262e-03 Score 99.763 Data time: 0.2288, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.6987 [2021-07-09 09:52:12,506][root][INFO] - ===> Epoch[11546](57730/5): Loss 0.0066, Sem 0.0066 LR: 5.251e-03 Score 99.763 Data time: 0.2223, Forward time: 1.2516, Backward time: 1.2485, DDP time: 0.0042, Total iter time: 2.7604 [2021-07-09 09:52:26,850][root][INFO] - ===> Epoch[11547](57735/5): Loss 0.0069, Sem 0.0069 LR: 5.241e-03 Score 99.755 Data time: 0.2445, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8288 [2021-07-09 09:52:40,850][root][INFO] - ===> Epoch[11548](57740/5): Loss 0.0064, Sem 0.0064 LR: 5.230e-03 Score 99.772 Data time: 0.1959, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7561 [2021-07-09 09:52:55,188][root][INFO] - ===> Epoch[11549](57745/5): Loss 0.0066, Sem 0.0066 LR: 5.220e-03 Score 99.764 Data time: 0.2687, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8267 [2021-07-09 09:53:08,746][root][INFO] - ===> Epoch[11550](57750/5): Loss 0.0065, Sem 0.0065 LR: 5.209e-03 Score 99.768 Data time: 0.2033, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6594 [2021-07-09 09:53:23,148][root][INFO] - ===> Epoch[11551](57755/5): Loss 0.0064, Sem 0.0064 LR: 5.199e-03 Score 99.769 Data time: 0.1814, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8410 [2021-07-09 09:53:36,590][root][INFO] - ===> Epoch[11552](57760/5): Loss 0.0066, Sem 0.0066 LR: 5.189e-03 Score 99.764 Data time: 0.2265, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6452 [2021-07-09 09:53:50,240][root][INFO] - ===> Epoch[11553](57765/5): Loss 0.0064, Sem 0.0064 LR: 5.178e-03 Score 99.771 Data time: 0.1739, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6898 [2021-07-09 09:54:03,862][root][INFO] - ===> Epoch[11554](57770/5): Loss 0.0066, Sem 0.0066 LR: 5.168e-03 Score 99.766 Data time: 0.2247, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6817 [2021-07-09 09:54:17,797][root][INFO] - ===> Epoch[11555](57775/5): Loss 0.0062, Sem 0.0062 LR: 5.157e-03 Score 99.779 Data time: 0.2022, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7372 [2021-07-09 09:54:32,683][root][INFO] - ===> Epoch[11556](57780/5): Loss 0.0065, Sem 0.0065 LR: 5.147e-03 Score 99.767 Data time: 0.2522, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9292 [2021-07-09 09:54:47,030][root][INFO] - ===> Epoch[11557](57785/5): Loss 0.0071, Sem 0.0071 LR: 5.137e-03 Score 99.748 Data time: 0.1718, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8334 [2021-07-09 09:55:00,632][root][INFO] - ===> Epoch[11558](57790/5): Loss 0.0065, Sem 0.0065 LR: 5.126e-03 Score 99.768 Data time: 0.2261, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6785 [2021-07-09 09:55:15,727][root][INFO] - ===> Epoch[11559](57795/5): Loss 0.0063, Sem 0.0063 LR: 5.116e-03 Score 99.772 Data time: 0.2652, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9703 [2021-07-09 09:55:29,108][root][INFO] - ===> Epoch[11560](57800/5): Loss 0.0070, Sem 0.0070 LR: 5.105e-03 Score 99.749 Data time: 0.2404, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6331 [2021-07-09 09:55:29,109][root][INFO] - ===> Start testing [2021-07-09 09:56:14,371][root][INFO] - 68/68: Data time: 0.1050, Iter time: 0.1257 Loss 0.471 (AVG: 0.638) Score 91.146 (AVG: 91.081) mIOU 71.754 mAP 82.344 mAcc 78.087 IOU: 64.161 0.000 80.156 75.321 95.014 91.605 34.690 85.211 98.602 83.642 80.316 86.248 57.835 mAP: 82.472 2.207 87.798 85.465 98.256 97.894 54.094 97.858 99.579 94.209 90.250 91.549 88.836 mAcc: 79.666 0.000 84.079 83.164 96.685 97.470 37.738 96.009 99.411 93.074 90.085 96.297 61.449 [2021-07-09 09:56:14,371][root][INFO] - Finished test. Elapsed time: 45.2611 [2021-07-09 09:56:15,411][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 09:56:28,150][root][INFO] - ===> Epoch[11561](57805/5): Loss 0.0065, Sem 0.0065 LR: 5.095e-03 Score 99.767 Data time: 0.1875, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.5141 [2021-07-09 09:56:42,389][root][INFO] - ===> Epoch[11562](57810/5): Loss 0.0068, Sem 0.0068 LR: 5.084e-03 Score 99.757 Data time: 0.2394, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.8029 [2021-07-09 09:56:56,476][root][INFO] - ===> Epoch[11563](57815/5): Loss 0.0065, Sem 0.0065 LR: 5.074e-03 Score 99.769 Data time: 0.2337, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7733 [2021-07-09 09:57:11,285][root][INFO] - ===> Epoch[11564](57820/5): Loss 0.0066, Sem 0.0066 LR: 5.063e-03 Score 99.765 Data time: 0.2486, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9277 [2021-07-09 09:57:26,187][root][INFO] - ===> Epoch[11565](57825/5): Loss 0.0064, Sem 0.0064 LR: 5.053e-03 Score 99.769 Data time: 0.2232, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.9379 [2021-07-09 09:57:40,458][root][INFO] - ===> Epoch[11566](57830/5): Loss 0.0066, Sem 0.0066 LR: 5.043e-03 Score 99.762 Data time: 0.2498, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7986 [2021-07-09 09:57:54,048][root][INFO] - ===> Epoch[11567](57835/5): Loss 0.0066, Sem 0.0066 LR: 5.032e-03 Score 99.765 Data time: 0.1734, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.6812 [2021-07-09 09:58:08,032][root][INFO] - ===> Epoch[11568](57840/5): Loss 0.0065, Sem 0.0065 LR: 5.022e-03 Score 99.768 Data time: 0.1878, Forward time: 1.2516, Backward time: 1.2486, DDP time: 0.0042, Total iter time: 2.7545 [2021-07-09 09:58:22,736][root][INFO] - ===> Epoch[11569](57845/5): Loss 0.0065, Sem 0.0065 LR: 5.011e-03 Score 99.771 Data time: 0.2757, Forward time: 1.2516, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.8977 [2021-07-09 09:58:36,804][root][INFO] - ===> Epoch[11570](57850/5): Loss 0.0063, Sem 0.0063 LR: 5.001e-03 Score 99.775 Data time: 0.1114, Forward time: 1.2516, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.7850 [2021-07-09 09:58:50,318][root][INFO] - ===> Epoch[11571](57855/5): Loss 0.0067, Sem 0.0067 LR: 4.990e-03 Score 99.762 Data time: 0.1871, Forward time: 1.2516, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.6587 [2021-07-09 09:59:03,361][root][INFO] - ===> Epoch[11572](57860/5): Loss 0.0067, Sem 0.0067 LR: 4.980e-03 Score 99.758 Data time: 0.1801, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.5687 [2021-07-09 09:59:17,477][root][INFO] - ===> Epoch[11573](57865/5): Loss 0.0067, Sem 0.0067 LR: 4.969e-03 Score 99.759 Data time: 0.1532, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.7842 [2021-07-09 09:59:31,642][root][INFO] - ===> Epoch[11574](57870/5): Loss 0.0064, Sem 0.0064 LR: 4.959e-03 Score 99.771 Data time: 0.2517, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.7780 [2021-07-09 09:59:46,227][root][INFO] - ===> Epoch[11575](57875/5): Loss 0.0065, Sem 0.0065 LR: 4.948e-03 Score 99.767 Data time: 0.2107, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.8688 [2021-07-09 09:59:58,988][root][INFO] - ===> Epoch[11576](57880/5): Loss 0.0065, Sem 0.0065 LR: 4.938e-03 Score 99.769 Data time: 0.2016, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.4944 [2021-07-09 10:00:13,035][root][INFO] - ===> Epoch[11577](57885/5): Loss 0.0065, Sem 0.0065 LR: 4.927e-03 Score 99.769 Data time: 0.2395, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.7584 [2021-07-09 10:00:27,494][root][INFO] - ===> Epoch[11578](57890/5): Loss 0.0065, Sem 0.0065 LR: 4.917e-03 Score 99.770 Data time: 0.2318, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.8444 [2021-07-09 10:00:41,413][root][INFO] - ===> Epoch[11579](57895/5): Loss 0.0064, Sem 0.0064 LR: 4.906e-03 Score 99.772 Data time: 0.2171, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.7428 [2021-07-09 10:00:55,036][root][INFO] - ===> Epoch[11580](57900/5): Loss 0.0063, Sem 0.0063 LR: 4.896e-03 Score 99.772 Data time: 0.2302, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.6677 [2021-07-09 10:01:08,662][root][INFO] - ===> Epoch[11581](57905/5): Loss 0.0065, Sem 0.0065 LR: 4.886e-03 Score 99.769 Data time: 0.2111, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.6791 [2021-07-09 10:01:22,578][root][INFO] - ===> Epoch[11582](57910/5): Loss 0.0067, Sem 0.0067 LR: 4.875e-03 Score 99.762 Data time: 0.1904, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.7365 [2021-07-09 10:01:38,131][root][INFO] - ===> Epoch[11583](57915/5): Loss 0.0064, Sem 0.0064 LR: 4.865e-03 Score 99.770 Data time: 0.1426, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 3.0660 [2021-07-09 10:01:52,321][root][INFO] - ===> Epoch[11584](57920/5): Loss 0.0061, Sem 0.0061 LR: 4.854e-03 Score 99.779 Data time: 0.2327, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.7941 [2021-07-09 10:02:06,926][root][INFO] - ===> Epoch[11585](57925/5): Loss 0.0064, Sem 0.0064 LR: 4.844e-03 Score 99.772 Data time: 0.1777, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8788 [2021-07-09 10:02:21,363][root][INFO] - ===> Epoch[11586](57930/5): Loss 0.0066, Sem 0.0066 LR: 4.833e-03 Score 99.765 Data time: 0.2206, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8377 [2021-07-09 10:02:35,334][root][INFO] - ===> Epoch[11587](57935/5): Loss 0.0069, Sem 0.0069 LR: 4.823e-03 Score 99.754 Data time: 0.1934, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7577 [2021-07-09 10:02:48,834][root][INFO] - ===> Epoch[11588](57940/5): Loss 0.0065, Sem 0.0065 LR: 4.812e-03 Score 99.768 Data time: 0.2150, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.6583 [2021-07-09 10:03:03,502][root][INFO] - ===> Epoch[11589](57945/5): Loss 0.0065, Sem 0.0065 LR: 4.802e-03 Score 99.773 Data time: 0.2109, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8686 [2021-07-09 10:03:17,543][root][INFO] - ===> Epoch[11590](57950/5): Loss 0.0064, Sem 0.0064 LR: 4.791e-03 Score 99.770 Data time: 0.2834, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7572 [2021-07-09 10:03:31,161][root][INFO] - ===> Epoch[11591](57955/5): Loss 0.0063, Sem 0.0063 LR: 4.780e-03 Score 99.775 Data time: 0.2117, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.6659 [2021-07-09 10:03:45,143][root][INFO] - ===> Epoch[11592](57960/5): Loss 0.0065, Sem 0.0065 LR: 4.770e-03 Score 99.769 Data time: 0.2728, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7550 [2021-07-09 10:03:59,893][root][INFO] - ===> Epoch[11593](57965/5): Loss 0.0065, Sem 0.0065 LR: 4.759e-03 Score 99.768 Data time: 0.2196, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8919 [2021-07-09 10:04:14,212][root][INFO] - ===> Epoch[11594](57970/5): Loss 0.0065, Sem 0.0065 LR: 4.749e-03 Score 99.770 Data time: 0.2826, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8085 [2021-07-09 10:04:29,188][root][INFO] - ===> Epoch[11595](57975/5): Loss 0.0063, Sem 0.0063 LR: 4.738e-03 Score 99.775 Data time: 0.2596, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.9473 [2021-07-09 10:04:43,661][root][INFO] - ===> Epoch[11596](57980/5): Loss 0.0065, Sem 0.0065 LR: 4.728e-03 Score 99.767 Data time: 0.2933, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8589 [2021-07-09 10:04:57,558][root][INFO] - ===> Epoch[11597](57985/5): Loss 0.0065, Sem 0.0065 LR: 4.717e-03 Score 99.768 Data time: 0.3441, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7279 [2021-07-09 10:05:13,200][root][INFO] - ===> Epoch[11598](57990/5): Loss 0.0061, Sem 0.0061 LR: 4.707e-03 Score 99.782 Data time: 0.2403, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 3.0719 [2021-07-09 10:05:27,843][root][INFO] - ===> Epoch[11599](57995/5): Loss 0.0063, Sem 0.0063 LR: 4.696e-03 Score 99.775 Data time: 0.2731, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8909 [2021-07-09 10:05:41,853][root][INFO] - ===> Epoch[11600](58000/5): Loss 0.0065, Sem 0.0065 LR: 4.686e-03 Score 99.768 Data time: 0.2372, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7581 [2021-07-09 10:05:41,854][root][INFO] - ===> Start testing [2021-07-09 10:06:28,082][root][INFO] - 68/68: Data time: 0.0014, Iter time: 0.1249 Loss 0.480 (AVG: 0.638) Score 90.963 (AVG: 91.116) mIOU 71.933 mAP 82.340 mAcc 78.119 IOU: 64.346 0.000 79.622 75.310 95.119 91.634 35.410 85.801 98.608 85.026 80.225 86.209 57.812 mAP: 82.662 2.260 87.655 85.492 98.291 97.877 54.025 97.723 99.619 94.444 90.197 91.411 88.769 mAcc: 79.680 0.000 83.976 83.304 96.685 97.440 38.605 95.615 99.392 92.995 90.210 96.376 61.276 [2021-07-09 10:06:28,082][root][INFO] - Finished test. Elapsed time: 46.2268 [2021-07-09 10:06:29,237][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 10:06:42,439][root][INFO] - ===> Epoch[11601](58005/5): Loss 0.0065, Sem 0.0065 LR: 4.675e-03 Score 99.767 Data time: 0.1551, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.6066 [2021-07-09 10:06:56,014][root][INFO] - ===> Epoch[11602](58010/5): Loss 0.0064, Sem 0.0064 LR: 4.665e-03 Score 99.769 Data time: 0.1661, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.6733 [2021-07-09 10:07:10,387][root][INFO] - ===> Epoch[11603](58015/5): Loss 0.0064, Sem 0.0064 LR: 4.654e-03 Score 99.773 Data time: 0.2298, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8287 [2021-07-09 10:07:24,447][root][INFO] - ===> Epoch[11604](58020/5): Loss 0.0064, Sem 0.0064 LR: 4.644e-03 Score 99.769 Data time: 0.2769, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7607 [2021-07-09 10:07:38,696][root][INFO] - ===> Epoch[11605](58025/5): Loss 0.0064, Sem 0.0064 LR: 4.633e-03 Score 99.772 Data time: 0.2446, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8079 [2021-07-09 10:07:53,009][root][INFO] - ===> Epoch[11606](58030/5): Loss 0.0065, Sem 0.0065 LR: 4.622e-03 Score 99.768 Data time: 0.1901, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8109 [2021-07-09 10:08:06,519][root][INFO] - ===> Epoch[11607](58035/5): Loss 0.0066, Sem 0.0066 LR: 4.612e-03 Score 99.761 Data time: 0.2479, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.6561 [2021-07-09 10:08:20,239][root][INFO] - ===> Epoch[11608](58040/5): Loss 0.0064, Sem 0.0064 LR: 4.601e-03 Score 99.770 Data time: 0.2176, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7079 [2021-07-09 10:08:33,261][root][INFO] - ===> Epoch[11609](58045/5): Loss 0.0066, Sem 0.0066 LR: 4.591e-03 Score 99.763 Data time: 0.1707, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.5650 [2021-07-09 10:08:46,933][root][INFO] - ===> Epoch[11610](58050/5): Loss 0.0063, Sem 0.0063 LR: 4.580e-03 Score 99.773 Data time: 0.2106, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.6921 [2021-07-09 10:09:01,470][root][INFO] - ===> Epoch[11611](58055/5): Loss 0.0066, Sem 0.0066 LR: 4.570e-03 Score 99.768 Data time: 0.2533, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8562 [2021-07-09 10:09:16,204][root][INFO] - ===> Epoch[11612](58060/5): Loss 0.0068, Sem 0.0068 LR: 4.559e-03 Score 99.756 Data time: 0.2094, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8867 [2021-07-09 10:09:29,776][root][INFO] - ===> Epoch[11613](58065/5): Loss 0.0062, Sem 0.0062 LR: 4.549e-03 Score 99.780 Data time: 0.2295, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.6650 [2021-07-09 10:09:44,345][root][INFO] - ===> Epoch[11614](58070/5): Loss 0.0065, Sem 0.0065 LR: 4.538e-03 Score 99.768 Data time: 0.2572, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8708 [2021-07-09 10:09:57,767][root][INFO] - ===> Epoch[11615](58075/5): Loss 0.0066, Sem 0.0066 LR: 4.527e-03 Score 99.766 Data time: 0.2691, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.6422 [2021-07-09 10:10:11,679][root][INFO] - ===> Epoch[11616](58080/5): Loss 0.0063, Sem 0.0063 LR: 4.517e-03 Score 99.776 Data time: 0.2404, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7304 [2021-07-09 10:10:25,377][root][INFO] - ===> Epoch[11617](58085/5): Loss 0.0064, Sem 0.0064 LR: 4.506e-03 Score 99.770 Data time: 0.2081, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.6867 [2021-07-09 10:10:39,066][root][INFO] - ===> Epoch[11618](58090/5): Loss 0.0063, Sem 0.0063 LR: 4.496e-03 Score 99.777 Data time: 0.2162, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.6926 [2021-07-09 10:10:52,802][root][INFO] - ===> Epoch[11619](58095/5): Loss 0.0065, Sem 0.0065 LR: 4.485e-03 Score 99.766 Data time: 0.2341, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.6903 [2021-07-09 10:11:07,048][root][INFO] - ===> Epoch[11620](58100/5): Loss 0.0065, Sem 0.0065 LR: 4.474e-03 Score 99.766 Data time: 0.1901, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.8003 [2021-07-09 10:11:20,731][root][INFO] - ===> Epoch[11621](58105/5): Loss 0.0067, Sem 0.0067 LR: 4.464e-03 Score 99.763 Data time: 0.2002, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.6893 [2021-07-09 10:11:34,708][root][INFO] - ===> Epoch[11622](58110/5): Loss 0.0066, Sem 0.0066 LR: 4.453e-03 Score 99.767 Data time: 0.2008, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7568 [2021-07-09 10:11:47,827][root][INFO] - ===> Epoch[11623](58115/5): Loss 0.0066, Sem 0.0066 LR: 4.443e-03 Score 99.764 Data time: 0.1632, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.5736 [2021-07-09 10:12:01,168][root][INFO] - ===> Epoch[11624](58120/5): Loss 0.0065, Sem 0.0065 LR: 4.432e-03 Score 99.767 Data time: 0.2032, Forward time: 1.2515, Backward time: 1.2487, DDP time: 0.0042, Total iter time: 2.6246 [2021-07-09 10:12:15,196][root][INFO] - ===> Epoch[11625](58125/5): Loss 0.0063, Sem 0.0063 LR: 4.421e-03 Score 99.775 Data time: 0.1836, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7609 [2021-07-09 10:12:29,444][root][INFO] - ===> Epoch[11626](58130/5): Loss 0.0065, Sem 0.0065 LR: 4.411e-03 Score 99.767 Data time: 0.1944, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8043 [2021-07-09 10:12:42,928][root][INFO] - ===> Epoch[11627](58135/5): Loss 0.0063, Sem 0.0063 LR: 4.400e-03 Score 99.776 Data time: 0.2202, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.6253 [2021-07-09 10:12:56,743][root][INFO] - ===> Epoch[11628](58140/5): Loss 0.0065, Sem 0.0065 LR: 4.390e-03 Score 99.766 Data time: 0.2288, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7201 [2021-07-09 10:13:10,813][root][INFO] - ===> Epoch[11629](58145/5): Loss 0.0065, Sem 0.0065 LR: 4.379e-03 Score 99.770 Data time: 0.2062, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7706 [2021-07-09 10:13:25,659][root][INFO] - ===> Epoch[11630](58150/5): Loss 0.0068, Sem 0.0068 LR: 4.368e-03 Score 99.756 Data time: 0.2945, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.9249 [2021-07-09 10:13:39,320][root][INFO] - ===> Epoch[11631](58155/5): Loss 0.0064, Sem 0.0064 LR: 4.358e-03 Score 99.773 Data time: 0.1807, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.6921 [2021-07-09 10:13:53,203][root][INFO] - ===> Epoch[11632](58160/5): Loss 0.0064, Sem 0.0064 LR: 4.347e-03 Score 99.773 Data time: 0.1593, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7319 [2021-07-09 10:14:07,255][root][INFO] - ===> Epoch[11633](58165/5): Loss 0.0067, Sem 0.0067 LR: 4.337e-03 Score 99.762 Data time: 0.1876, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7652 [2021-07-09 10:14:21,317][root][INFO] - ===> Epoch[11634](58170/5): Loss 0.0064, Sem 0.0064 LR: 4.326e-03 Score 99.769 Data time: 0.1694, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7714 [2021-07-09 10:14:35,235][root][INFO] - ===> Epoch[11635](58175/5): Loss 0.0062, Sem 0.0062 LR: 4.315e-03 Score 99.779 Data time: 0.2180, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7357 [2021-07-09 10:14:49,294][root][INFO] - ===> Epoch[11636](58180/5): Loss 0.0066, Sem 0.0066 LR: 4.305e-03 Score 99.766 Data time: 0.2123, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7704 [2021-07-09 10:15:03,622][root][INFO] - ===> Epoch[11637](58185/5): Loss 0.0067, Sem 0.0067 LR: 4.294e-03 Score 99.763 Data time: 0.2277, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8116 [2021-07-09 10:15:15,964][root][INFO] - ===> Epoch[11638](58190/5): Loss 0.0065, Sem 0.0065 LR: 4.283e-03 Score 99.769 Data time: 0.2274, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.4224 [2021-07-09 10:15:30,205][root][INFO] - ===> Epoch[11639](58195/5): Loss 0.0066, Sem 0.0066 LR: 4.273e-03 Score 99.766 Data time: 0.2236, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8010 [2021-07-09 10:15:44,511][root][INFO] - ===> Epoch[11640](58200/5): Loss 0.0065, Sem 0.0065 LR: 4.262e-03 Score 99.766 Data time: 0.1753, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8162 [2021-07-09 10:15:44,513][root][INFO] - ===> Start testing [2021-07-09 10:16:20,830][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1300 Loss 0.449 (AVG: 0.636) Score 91.300 (AVG: 91.077) mIOU 71.839 mAP 82.312 mAcc 78.029 IOU: 64.036 0.000 79.720 75.192 95.064 91.368 35.466 85.015 98.589 85.310 79.903 86.323 57.914 mAP: 82.361 2.322 87.607 85.113 98.195 97.803 54.073 97.838 99.624 94.283 90.500 91.448 88.885 mAcc: 79.280 0.000 83.776 83.001 96.838 97.520 38.645 95.706 99.348 92.532 89.976 96.446 61.310 [2021-07-09 10:16:20,830][root][INFO] - Finished test. Elapsed time: 36.3163 [2021-07-09 10:16:21,955][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 10:16:34,806][root][INFO] - ===> Epoch[11641](58205/5): Loss 0.0065, Sem 0.0065 LR: 4.251e-03 Score 99.768 Data time: 0.1957, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.5218 [2021-07-09 10:16:49,498][root][INFO] - ===> Epoch[11642](58210/5): Loss 0.0064, Sem 0.0064 LR: 4.241e-03 Score 99.773 Data time: 0.1723, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.9015 [2021-07-09 10:17:03,506][root][INFO] - ===> Epoch[11643](58215/5): Loss 0.0066, Sem 0.0066 LR: 4.230e-03 Score 99.763 Data time: 0.2560, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7597 [2021-07-09 10:17:17,952][root][INFO] - ===> Epoch[11644](58220/5): Loss 0.0066, Sem 0.0066 LR: 4.219e-03 Score 99.762 Data time: 0.2258, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8416 [2021-07-09 10:17:32,538][root][INFO] - ===> Epoch[11645](58225/5): Loss 0.0064, Sem 0.0064 LR: 4.209e-03 Score 99.774 Data time: 0.1977, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8757 [2021-07-09 10:17:46,165][root][INFO] - ===> Epoch[11646](58230/5): Loss 0.0063, Sem 0.0063 LR: 4.198e-03 Score 99.774 Data time: 0.2244, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.6831 [2021-07-09 10:18:00,770][root][INFO] - ===> Epoch[11647](58235/5): Loss 0.0069, Sem 0.0069 LR: 4.187e-03 Score 99.757 Data time: 0.1927, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8718 [2021-07-09 10:18:14,634][root][INFO] - ===> Epoch[11648](58240/5): Loss 0.0061, Sem 0.0061 LR: 4.177e-03 Score 99.782 Data time: 0.2167, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7278 [2021-07-09 10:18:28,813][root][INFO] - ===> Epoch[11649](58245/5): Loss 0.0065, Sem 0.0065 LR: 4.166e-03 Score 99.769 Data time: 0.1945, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.7897 [2021-07-09 10:18:43,179][root][INFO] - ===> Epoch[11650](58250/5): Loss 0.0064, Sem 0.0064 LR: 4.155e-03 Score 99.769 Data time: 0.2284, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.8211 [2021-07-09 10:18:57,509][root][INFO] - ===> Epoch[11651](58255/5): Loss 0.0065, Sem 0.0065 LR: 4.145e-03 Score 99.769 Data time: 0.2255, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.8117 [2021-07-09 10:19:11,688][root][INFO] - ===> Epoch[11652](58260/5): Loss 0.0063, Sem 0.0063 LR: 4.134e-03 Score 99.774 Data time: 0.2507, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.7874 [2021-07-09 10:19:26,826][root][INFO] - ===> Epoch[11653](58265/5): Loss 0.0063, Sem 0.0063 LR: 4.123e-03 Score 99.774 Data time: 0.2000, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.9828 [2021-07-09 10:19:40,358][root][INFO] - ===> Epoch[11654](58270/5): Loss 0.0062, Sem 0.0062 LR: 4.113e-03 Score 99.778 Data time: 0.2927, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.6588 [2021-07-09 10:19:54,453][root][INFO] - ===> Epoch[11655](58275/5): Loss 0.0065, Sem 0.0065 LR: 4.102e-03 Score 99.768 Data time: 0.2468, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7708 [2021-07-09 10:20:09,416][root][INFO] - ===> Epoch[11656](58280/5): Loss 0.0064, Sem 0.0064 LR: 4.091e-03 Score 99.767 Data time: 0.3016, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.9453 [2021-07-09 10:20:23,753][root][INFO] - ===> Epoch[11657](58285/5): Loss 0.0066, Sem 0.0066 LR: 4.081e-03 Score 99.762 Data time: 0.3020, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8219 [2021-07-09 10:20:38,605][root][INFO] - ===> Epoch[11658](58290/5): Loss 0.0063, Sem 0.0063 LR: 4.070e-03 Score 99.773 Data time: 0.2965, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.9252 [2021-07-09 10:20:53,121][root][INFO] - ===> Epoch[11659](58295/5): Loss 0.0062, Sem 0.0062 LR: 4.059e-03 Score 99.777 Data time: 0.2104, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8610 [2021-07-09 10:21:07,057][root][INFO] - ===> Epoch[11660](58300/5): Loss 0.0061, Sem 0.0061 LR: 4.048e-03 Score 99.780 Data time: 0.2054, Forward time: 1.2515, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7381 [2021-07-09 10:21:21,255][root][INFO] - ===> Epoch[11661](58305/5): Loss 0.0064, Sem 0.0064 LR: 4.038e-03 Score 99.771 Data time: 0.2319, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7817 [2021-07-09 10:21:35,445][root][INFO] - ===> Epoch[11662](58310/5): Loss 0.0065, Sem 0.0065 LR: 4.027e-03 Score 99.766 Data time: 0.1806, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.8024 [2021-07-09 10:21:48,469][root][INFO] - ===> Epoch[11663](58315/5): Loss 0.0062, Sem 0.0062 LR: 4.016e-03 Score 99.776 Data time: 0.1762, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.5685 [2021-07-09 10:22:02,803][root][INFO] - ===> Epoch[11664](58320/5): Loss 0.0066, Sem 0.0066 LR: 4.006e-03 Score 99.764 Data time: 0.2499, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8229 [2021-07-09 10:22:16,301][root][INFO] - ===> Epoch[11665](58325/5): Loss 0.0062, Sem 0.0062 LR: 3.995e-03 Score 99.777 Data time: 0.1926, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.6552 [2021-07-09 10:22:30,078][root][INFO] - ===> Epoch[11666](58330/5): Loss 0.0064, Sem 0.0064 LR: 3.984e-03 Score 99.770 Data time: 0.2278, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7025 [2021-07-09 10:22:43,787][root][INFO] - ===> Epoch[11667](58335/5): Loss 0.0065, Sem 0.0065 LR: 3.973e-03 Score 99.764 Data time: 0.2318, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.6850 [2021-07-09 10:22:58,331][root][INFO] - ===> Epoch[11668](58340/5): Loss 0.0063, Sem 0.0063 LR: 3.963e-03 Score 99.776 Data time: 0.3034, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8647 [2021-07-09 10:23:12,538][root][INFO] - ===> Epoch[11669](58345/5): Loss 0.0066, Sem 0.0066 LR: 3.952e-03 Score 99.763 Data time: 0.2222, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7996 [2021-07-09 10:23:27,029][root][INFO] - ===> Epoch[11670](58350/5): Loss 0.0062, Sem 0.0062 LR: 3.941e-03 Score 99.776 Data time: 0.2174, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.8520 [2021-07-09 10:23:40,327][root][INFO] - ===> Epoch[11671](58355/5): Loss 0.0067, Sem 0.0067 LR: 3.930e-03 Score 99.762 Data time: 0.2181, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.6151 [2021-07-09 10:23:53,482][root][INFO] - ===> Epoch[11672](58360/5): Loss 0.0063, Sem 0.0063 LR: 3.920e-03 Score 99.774 Data time: 0.1699, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.5945 [2021-07-09 10:24:07,608][root][INFO] - ===> Epoch[11673](58365/5): Loss 0.0064, Sem 0.0064 LR: 3.909e-03 Score 99.771 Data time: 0.1893, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7809 [2021-07-09 10:24:21,845][root][INFO] - ===> Epoch[11674](58370/5): Loss 0.0065, Sem 0.0065 LR: 3.898e-03 Score 99.769 Data time: 0.1730, Forward time: 1.2514, Backward time: 1.2488, DDP time: 0.0042, Total iter time: 2.7867 [2021-07-09 10:24:36,366][root][INFO] - ===> Epoch[11675](58375/5): Loss 0.0063, Sem 0.0063 LR: 3.887e-03 Score 99.777 Data time: 0.2033, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.8566 [2021-07-09 10:24:49,884][root][INFO] - ===> Epoch[11676](58380/5): Loss 0.0066, Sem 0.0066 LR: 3.877e-03 Score 99.766 Data time: 0.2155, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.6582 [2021-07-09 10:25:05,188][root][INFO] - ===> Epoch[11677](58385/5): Loss 0.0067, Sem 0.0067 LR: 3.866e-03 Score 99.762 Data time: 0.2298, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 3.0168 [2021-07-09 10:25:19,204][root][INFO] - ===> Epoch[11678](58390/5): Loss 0.0064, Sem 0.0064 LR: 3.855e-03 Score 99.771 Data time: 0.1821, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.7610 [2021-07-09 10:25:32,802][root][INFO] - ===> Epoch[11679](58395/5): Loss 0.0064, Sem 0.0064 LR: 3.844e-03 Score 99.770 Data time: 0.1707, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.6764 [2021-07-09 10:25:46,731][root][INFO] - ===> Epoch[11680](58400/5): Loss 0.0065, Sem 0.0065 LR: 3.834e-03 Score 99.772 Data time: 0.2649, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.7366 [2021-07-09 10:25:46,732][root][INFO] - ===> Start testing [2021-07-09 10:26:22,098][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1359 Loss 0.459 (AVG: 0.634) Score 91.217 (AVG: 91.075) mIOU 71.831 mAP 82.254 mAcc 77.945 IOU: 64.084 0.000 80.141 75.130 95.101 91.412 34.123 84.634 98.573 85.867 80.163 86.334 58.246 mAP: 82.804 2.003 87.506 85.302 98.240 97.885 53.006 97.706 99.569 94.342 90.361 91.578 89.005 mAcc: 79.518 0.000 83.725 83.173 96.776 97.547 37.049 95.636 99.379 92.610 90.114 96.388 61.369 [2021-07-09 10:26:22,098][root][INFO] - Finished test. Elapsed time: 35.3644 [2021-07-09 10:26:23,141][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 10:26:37,111][root][INFO] - ===> Epoch[11681](58405/5): Loss 0.0068, Sem 0.0068 LR: 3.823e-03 Score 99.759 Data time: 0.1871, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.7511 [2021-07-09 10:26:51,125][root][INFO] - ===> Epoch[11682](58410/5): Loss 0.0062, Sem 0.0062 LR: 3.812e-03 Score 99.780 Data time: 0.2138, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.7504 [2021-07-09 10:27:04,958][root][INFO] - ===> Epoch[11683](58415/5): Loss 0.0060, Sem 0.0060 LR: 3.801e-03 Score 99.785 Data time: 0.1851, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.7285 [2021-07-09 10:27:19,777][root][INFO] - ===> Epoch[11684](58420/5): Loss 0.0065, Sem 0.0065 LR: 3.790e-03 Score 99.767 Data time: 0.2007, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.9218 [2021-07-09 10:27:33,308][root][INFO] - ===> Epoch[11685](58425/5): Loss 0.0067, Sem 0.0067 LR: 3.780e-03 Score 99.761 Data time: 0.1811, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.6554 [2021-07-09 10:27:46,760][root][INFO] - ===> Epoch[11686](58430/5): Loss 0.0062, Sem 0.0062 LR: 3.769e-03 Score 99.776 Data time: 0.1952, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.6485 [2021-07-09 10:28:01,270][root][INFO] - ===> Epoch[11687](58435/5): Loss 0.0066, Sem 0.0066 LR: 3.758e-03 Score 99.760 Data time: 0.2158, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.8575 [2021-07-09 10:28:15,501][root][INFO] - ===> Epoch[11688](58440/5): Loss 0.0064, Sem 0.0064 LR: 3.747e-03 Score 99.772 Data time: 0.2236, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.7976 [2021-07-09 10:28:29,438][root][INFO] - ===> Epoch[11689](58445/5): Loss 0.0063, Sem 0.0063 LR: 3.737e-03 Score 99.774 Data time: 0.2132, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.7432 [2021-07-09 10:28:42,824][root][INFO] - ===> Epoch[11690](58450/5): Loss 0.0064, Sem 0.0064 LR: 3.726e-03 Score 99.773 Data time: 0.1770, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.6279 [2021-07-09 10:28:56,742][root][INFO] - ===> Epoch[11691](58455/5): Loss 0.0061, Sem 0.0061 LR: 3.715e-03 Score 99.782 Data time: 0.1980, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.7340 [2021-07-09 10:29:11,525][root][INFO] - ===> Epoch[11692](58460/5): Loss 0.0064, Sem 0.0064 LR: 3.704e-03 Score 99.771 Data time: 0.2352, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.9187 [2021-07-09 10:29:25,455][root][INFO] - ===> Epoch[11693](58465/5): Loss 0.0064, Sem 0.0064 LR: 3.693e-03 Score 99.769 Data time: 0.2351, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.7310 [2021-07-09 10:29:38,755][root][INFO] - ===> Epoch[11694](58470/5): Loss 0.0065, Sem 0.0065 LR: 3.682e-03 Score 99.766 Data time: 0.1821, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.6129 [2021-07-09 10:29:52,566][root][INFO] - ===> Epoch[11695](58475/5): Loss 0.0064, Sem 0.0064 LR: 3.672e-03 Score 99.768 Data time: 0.2427, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.7141 [2021-07-09 10:30:06,118][root][INFO] - ===> Epoch[11696](58480/5): Loss 0.0066, Sem 0.0066 LR: 3.661e-03 Score 99.766 Data time: 0.2350, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.6702 [2021-07-09 10:30:19,279][root][INFO] - ===> Epoch[11697](58485/5): Loss 0.0065, Sem 0.0065 LR: 3.650e-03 Score 99.766 Data time: 0.2466, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.5862 [2021-07-09 10:30:34,055][root][INFO] - ===> Epoch[11698](58490/5): Loss 0.0063, Sem 0.0063 LR: 3.639e-03 Score 99.774 Data time: 0.2210, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.9012 [2021-07-09 10:30:47,599][root][INFO] - ===> Epoch[11699](58495/5): Loss 0.0065, Sem 0.0065 LR: 3.628e-03 Score 99.768 Data time: 0.1986, Forward time: 1.2514, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.6642 [2021-07-09 10:31:01,973][root][INFO] - ===> Epoch[11700](58500/5): Loss 0.0064, Sem 0.0064 LR: 3.617e-03 Score 99.773 Data time: 0.2200, Forward time: 1.2513, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.8280 [2021-07-09 10:31:15,085][root][INFO] - ===> Epoch[11701](58505/5): Loss 0.0066, Sem 0.0066 LR: 3.607e-03 Score 99.764 Data time: 0.1862, Forward time: 1.2513, Backward time: 1.2489, DDP time: 0.0042, Total iter time: 2.5821 [2021-07-09 10:31:30,035][root][INFO] - ===> Epoch[11702](58510/5): Loss 0.0063, Sem 0.0063 LR: 3.596e-03 Score 99.774 Data time: 0.2067, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.9407 [2021-07-09 10:31:44,192][root][INFO] - ===> Epoch[11703](58515/5): Loss 0.0066, Sem 0.0066 LR: 3.585e-03 Score 99.764 Data time: 0.1915, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.7938 [2021-07-09 10:31:59,560][root][INFO] - ===> Epoch[11704](58520/5): Loss 0.0064, Sem 0.0064 LR: 3.574e-03 Score 99.775 Data time: 0.2091, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 3.0263 [2021-07-09 10:32:14,138][root][INFO] - ===> Epoch[11705](58525/5): Loss 0.0063, Sem 0.0063 LR: 3.563e-03 Score 99.777 Data time: 0.2269, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.8647 [2021-07-09 10:32:27,907][root][INFO] - ===> Epoch[11706](58530/5): Loss 0.0069, Sem 0.0069 LR: 3.552e-03 Score 99.755 Data time: 0.2783, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.7094 [2021-07-09 10:32:41,625][root][INFO] - ===> Epoch[11707](58535/5): Loss 0.0063, Sem 0.0063 LR: 3.541e-03 Score 99.776 Data time: 0.1725, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.6998 [2021-07-09 10:32:55,836][root][INFO] - ===> Epoch[11708](58540/5): Loss 0.0069, Sem 0.0069 LR: 3.531e-03 Score 99.753 Data time: 0.2109, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.7852 [2021-07-09 10:33:09,372][root][INFO] - ===> Epoch[11709](58545/5): Loss 0.0065, Sem 0.0065 LR: 3.520e-03 Score 99.766 Data time: 0.2042, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.6652 [2021-07-09 10:33:22,899][root][INFO] - ===> Epoch[11710](58550/5): Loss 0.0065, Sem 0.0065 LR: 3.509e-03 Score 99.767 Data time: 0.2356, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.6559 [2021-07-09 10:33:36,946][root][INFO] - ===> Epoch[11711](58555/5): Loss 0.0069, Sem 0.0069 LR: 3.498e-03 Score 99.755 Data time: 0.1876, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.7658 [2021-07-09 10:33:51,707][root][INFO] - ===> Epoch[11712](58560/5): Loss 0.0061, Sem 0.0061 LR: 3.487e-03 Score 99.782 Data time: 0.2533, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.9028 [2021-07-09 10:34:05,994][root][INFO] - ===> Epoch[11713](58565/5): Loss 0.0066, Sem 0.0066 LR: 3.476e-03 Score 99.767 Data time: 0.3077, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.7872 [2021-07-09 10:34:20,094][root][INFO] - ===> Epoch[11714](58570/5): Loss 0.0065, Sem 0.0065 LR: 3.465e-03 Score 99.768 Data time: 0.2107, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.7703 [2021-07-09 10:34:34,508][root][INFO] - ===> Epoch[11715](58575/5): Loss 0.0064, Sem 0.0064 LR: 3.454e-03 Score 99.767 Data time: 0.2040, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.8276 [2021-07-09 10:34:48,171][root][INFO] - ===> Epoch[11716](58580/5): Loss 0.0063, Sem 0.0063 LR: 3.443e-03 Score 99.774 Data time: 0.2387, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.6822 [2021-07-09 10:35:03,119][root][INFO] - ===> Epoch[11717](58585/5): Loss 0.0065, Sem 0.0065 LR: 3.433e-03 Score 99.767 Data time: 0.2137, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.9523 [2021-07-09 10:35:17,520][root][INFO] - ===> Epoch[11718](58590/5): Loss 0.0066, Sem 0.0066 LR: 3.422e-03 Score 99.766 Data time: 0.2095, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.8355 [2021-07-09 10:35:31,900][root][INFO] - ===> Epoch[11719](58595/5): Loss 0.0065, Sem 0.0065 LR: 3.411e-03 Score 99.767 Data time: 0.2361, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.8264 [2021-07-09 10:35:46,682][root][INFO] - ===> Epoch[11720](58600/5): Loss 0.0063, Sem 0.0063 LR: 3.400e-03 Score 99.775 Data time: 0.2342, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.9210 [2021-07-09 10:35:46,683][root][INFO] - ===> Start testing [2021-07-09 10:36:27,345][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1234 Loss 0.462 (AVG: 0.634) Score 91.445 (AVG: 91.072) mIOU 71.847 mAP 82.359 mAcc 78.097 IOU: 63.981 0.000 79.994 75.075 95.034 91.574 35.282 84.858 98.615 85.046 80.020 86.394 58.144 mAP: 82.667 2.149 87.503 85.176 98.262 97.857 54.127 97.769 99.581 94.207 90.507 91.789 89.070 mAcc: 79.626 0.000 83.844 82.635 96.743 97.298 38.401 95.786 99.440 92.857 90.449 96.354 61.830 [2021-07-09 10:36:27,345][root][INFO] - Finished test. Elapsed time: 40.6616 [2021-07-09 10:36:28,401][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 10:36:41,911][root][INFO] - ===> Epoch[11721](58605/5): Loss 0.0066, Sem 0.0066 LR: 3.389e-03 Score 99.763 Data time: 0.1872, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.6656 [2021-07-09 10:36:56,194][root][INFO] - ===> Epoch[11722](58610/5): Loss 0.0061, Sem 0.0061 LR: 3.378e-03 Score 99.779 Data time: 0.1947, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.8147 [2021-07-09 10:37:09,939][root][INFO] - ===> Epoch[11723](58615/5): Loss 0.0065, Sem 0.0065 LR: 3.367e-03 Score 99.767 Data time: 0.2192, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.7002 [2021-07-09 10:37:24,728][root][INFO] - ===> Epoch[11724](58620/5): Loss 0.0064, Sem 0.0064 LR: 3.356e-03 Score 99.773 Data time: 0.1942, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.9169 [2021-07-09 10:37:38,428][root][INFO] - ===> Epoch[11725](58625/5): Loss 0.0064, Sem 0.0064 LR: 3.345e-03 Score 99.771 Data time: 0.1862, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.6875 [2021-07-09 10:37:52,483][root][INFO] - ===> Epoch[11726](58630/5): Loss 0.0064, Sem 0.0064 LR: 3.334e-03 Score 99.772 Data time: 0.2435, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7592 [2021-07-09 10:38:06,342][root][INFO] - ===> Epoch[11727](58635/5): Loss 0.0063, Sem 0.0063 LR: 3.323e-03 Score 99.775 Data time: 0.1976, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.7231 [2021-07-09 10:38:20,354][root][INFO] - ===> Epoch[11728](58640/5): Loss 0.0063, Sem 0.0063 LR: 3.312e-03 Score 99.774 Data time: 0.1874, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.7676 [2021-07-09 10:38:33,756][root][INFO] - ===> Epoch[11729](58645/5): Loss 0.0067, Sem 0.0067 LR: 3.301e-03 Score 99.763 Data time: 0.2258, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.6348 [2021-07-09 10:38:47,978][root][INFO] - ===> Epoch[11730](58650/5): Loss 0.0065, Sem 0.0065 LR: 3.290e-03 Score 99.769 Data time: 0.1975, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.8013 [2021-07-09 10:39:02,105][root][INFO] - ===> Epoch[11731](58655/5): Loss 0.0062, Sem 0.0062 LR: 3.279e-03 Score 99.780 Data time: 0.2056, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7783 [2021-07-09 10:39:16,975][root][INFO] - ===> Epoch[11732](58660/5): Loss 0.0064, Sem 0.0064 LR: 3.268e-03 Score 99.771 Data time: 0.1710, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.9329 [2021-07-09 10:39:30,616][root][INFO] - ===> Epoch[11733](58665/5): Loss 0.0063, Sem 0.0063 LR: 3.257e-03 Score 99.775 Data time: 0.1646, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.6917 [2021-07-09 10:39:45,219][root][INFO] - ===> Epoch[11734](58670/5): Loss 0.0063, Sem 0.0063 LR: 3.247e-03 Score 99.773 Data time: 0.2024, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8752 [2021-07-09 10:39:59,688][root][INFO] - ===> Epoch[11735](58675/5): Loss 0.0063, Sem 0.0063 LR: 3.236e-03 Score 99.774 Data time: 0.2140, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8311 [2021-07-09 10:40:13,523][root][INFO] - ===> Epoch[11736](58680/5): Loss 0.0063, Sem 0.0063 LR: 3.225e-03 Score 99.772 Data time: 0.2405, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7204 [2021-07-09 10:40:27,599][root][INFO] - ===> Epoch[11737](58685/5): Loss 0.0063, Sem 0.0063 LR: 3.214e-03 Score 99.776 Data time: 0.1980, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7805 [2021-07-09 10:40:41,962][root][INFO] - ===> Epoch[11738](58690/5): Loss 0.0064, Sem 0.0064 LR: 3.203e-03 Score 99.771 Data time: 0.1649, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8326 [2021-07-09 10:40:55,292][root][INFO] - ===> Epoch[11739](58695/5): Loss 0.0064, Sem 0.0064 LR: 3.192e-03 Score 99.772 Data time: 0.2535, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.6103 [2021-07-09 10:41:09,396][root][INFO] - ===> Epoch[11740](58700/5): Loss 0.0065, Sem 0.0065 LR: 3.181e-03 Score 99.766 Data time: 0.2535, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7760 [2021-07-09 10:41:23,026][root][INFO] - ===> Epoch[11741](58705/5): Loss 0.0063, Sem 0.0063 LR: 3.170e-03 Score 99.775 Data time: 0.2016, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.6836 [2021-07-09 10:41:36,924][root][INFO] - ===> Epoch[11742](58710/5): Loss 0.0064, Sem 0.0064 LR: 3.159e-03 Score 99.769 Data time: 0.2953, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7369 [2021-07-09 10:41:52,121][root][INFO] - ===> Epoch[11743](58715/5): Loss 0.0065, Sem 0.0065 LR: 3.148e-03 Score 99.768 Data time: 0.1893, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.9947 [2021-07-09 10:42:06,375][root][INFO] - ===> Epoch[11744](58720/5): Loss 0.0060, Sem 0.0060 LR: 3.137e-03 Score 99.788 Data time: 0.1951, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8071 [2021-07-09 10:42:20,725][root][INFO] - ===> Epoch[11745](58725/5): Loss 0.0063, Sem 0.0063 LR: 3.126e-03 Score 99.776 Data time: 0.1974, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8304 [2021-07-09 10:42:34,552][root][INFO] - ===> Epoch[11746](58730/5): Loss 0.0065, Sem 0.0065 LR: 3.115e-03 Score 99.769 Data time: 0.1820, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7238 [2021-07-09 10:42:48,323][root][INFO] - ===> Epoch[11747](58735/5): Loss 0.0063, Sem 0.0063 LR: 3.103e-03 Score 99.776 Data time: 0.1914, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7069 [2021-07-09 10:43:03,037][root][INFO] - ===> Epoch[11748](58740/5): Loss 0.0064, Sem 0.0064 LR: 3.092e-03 Score 99.770 Data time: 0.2498, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8940 [2021-07-09 10:43:17,283][root][INFO] - ===> Epoch[11749](58745/5): Loss 0.0062, Sem 0.0062 LR: 3.081e-03 Score 99.778 Data time: 0.2641, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7858 [2021-07-09 10:43:31,430][root][INFO] - ===> Epoch[11750](58750/5): Loss 0.0068, Sem 0.0068 LR: 3.070e-03 Score 99.761 Data time: 0.2041, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7799 [2021-07-09 10:43:45,868][root][INFO] - ===> Epoch[11751](58755/5): Loss 0.0064, Sem 0.0064 LR: 3.059e-03 Score 99.773 Data time: 0.2339, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8424 [2021-07-09 10:44:00,037][root][INFO] - ===> Epoch[11752](58760/5): Loss 0.0061, Sem 0.0061 LR: 3.048e-03 Score 99.781 Data time: 0.1759, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7919 [2021-07-09 10:44:13,488][root][INFO] - ===> Epoch[11753](58765/5): Loss 0.0068, Sem 0.0068 LR: 3.037e-03 Score 99.755 Data time: 0.2132, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.6390 [2021-07-09 10:44:27,272][root][INFO] - ===> Epoch[11754](58770/5): Loss 0.0064, Sem 0.0064 LR: 3.026e-03 Score 99.774 Data time: 0.1997, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7155 [2021-07-09 10:44:42,024][root][INFO] - ===> Epoch[11755](58775/5): Loss 0.0061, Sem 0.0061 LR: 3.015e-03 Score 99.780 Data time: 0.1672, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.9000 [2021-07-09 10:44:55,715][root][INFO] - ===> Epoch[11756](58780/5): Loss 0.0064, Sem 0.0064 LR: 3.004e-03 Score 99.770 Data time: 0.2377, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.6888 [2021-07-09 10:45:09,733][root][INFO] - ===> Epoch[11757](58785/5): Loss 0.0065, Sem 0.0065 LR: 2.993e-03 Score 99.767 Data time: 0.2545, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7548 [2021-07-09 10:45:23,187][root][INFO] - ===> Epoch[11758](58790/5): Loss 0.0063, Sem 0.0063 LR: 2.982e-03 Score 99.777 Data time: 0.2128, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.6456 [2021-07-09 10:45:36,653][root][INFO] - ===> Epoch[11759](58795/5): Loss 0.0067, Sem 0.0067 LR: 2.971e-03 Score 99.760 Data time: 0.1980, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.6461 [2021-07-09 10:45:50,885][root][INFO] - ===> Epoch[11760](58800/5): Loss 0.0065, Sem 0.0065 LR: 2.960e-03 Score 99.768 Data time: 0.1750, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8092 [2021-07-09 10:45:50,886][root][INFO] - ===> Start testing [2021-07-09 10:46:27,891][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1252 Loss 0.444 (AVG: 0.640) Score 91.673 (AVG: 91.060) mIOU 71.852 mAP 82.181 mAcc 77.928 IOU: 64.067 0.000 80.492 75.021 95.123 91.156 34.185 85.210 98.565 85.900 79.985 86.216 58.157 mAP: 82.201 2.198 87.562 84.888 98.273 97.757 52.953 97.749 99.606 94.304 90.210 91.588 89.061 mAcc: 79.582 0.000 83.882 82.721 96.739 97.239 37.008 95.705 99.343 92.739 89.886 96.551 61.666 [2021-07-09 10:46:27,891][root][INFO] - Finished test. Elapsed time: 37.0039 [2021-07-09 10:46:29,001][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 10:46:42,642][root][INFO] - ===> Epoch[11761](58805/5): Loss 0.0064, Sem 0.0064 LR: 2.949e-03 Score 99.775 Data time: 0.1940, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.6952 [2021-07-09 10:46:55,402][root][INFO] - ===> Epoch[11762](58810/5): Loss 0.0065, Sem 0.0065 LR: 2.938e-03 Score 99.768 Data time: 0.2087, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.5057 [2021-07-09 10:47:09,179][root][INFO] - ===> Epoch[11763](58815/5): Loss 0.0064, Sem 0.0064 LR: 2.926e-03 Score 99.771 Data time: 0.2314, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7127 [2021-07-09 10:47:24,331][root][INFO] - ===> Epoch[11764](58820/5): Loss 0.0065, Sem 0.0065 LR: 2.915e-03 Score 99.769 Data time: 0.2694, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.9843 [2021-07-09 10:47:38,470][root][INFO] - ===> Epoch[11765](58825/5): Loss 0.0063, Sem 0.0063 LR: 2.904e-03 Score 99.775 Data time: 0.2596, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7783 [2021-07-09 10:47:52,654][root][INFO] - ===> Epoch[11766](58830/5): Loss 0.0064, Sem 0.0064 LR: 2.893e-03 Score 99.771 Data time: 0.2367, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7867 [2021-07-09 10:48:07,052][root][INFO] - ===> Epoch[11767](58835/5): Loss 0.0062, Sem 0.0062 LR: 2.882e-03 Score 99.779 Data time: 0.2649, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8343 [2021-07-09 10:48:21,426][root][INFO] - ===> Epoch[11768](58840/5): Loss 0.0066, Sem 0.0066 LR: 2.871e-03 Score 99.762 Data time: 0.3022, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8323 [2021-07-09 10:48:34,876][root][INFO] - ===> Epoch[11769](58845/5): Loss 0.0064, Sem 0.0064 LR: 2.860e-03 Score 99.771 Data time: 0.2297, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.6434 [2021-07-09 10:48:48,789][root][INFO] - ===> Epoch[11770](58850/5): Loss 0.0064, Sem 0.0064 LR: 2.849e-03 Score 99.773 Data time: 0.2710, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7258 [2021-07-09 10:49:03,338][root][INFO] - ===> Epoch[11771](58855/5): Loss 0.0064, Sem 0.0064 LR: 2.837e-03 Score 99.770 Data time: 0.2416, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8613 [2021-07-09 10:49:18,009][root][INFO] - ===> Epoch[11772](58860/5): Loss 0.0061, Sem 0.0061 LR: 2.826e-03 Score 99.785 Data time: 0.3056, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8909 [2021-07-09 10:49:32,112][root][INFO] - ===> Epoch[11773](58865/5): Loss 0.0064, Sem 0.0064 LR: 2.815e-03 Score 99.772 Data time: 0.2066, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7741 [2021-07-09 10:49:46,499][root][INFO] - ===> Epoch[11774](58870/5): Loss 0.0065, Sem 0.0065 LR: 2.804e-03 Score 99.770 Data time: 0.2051, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8314 [2021-07-09 10:49:59,836][root][INFO] - ===> Epoch[11775](58875/5): Loss 0.0062, Sem 0.0062 LR: 2.793e-03 Score 99.782 Data time: 0.2527, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.6241 [2021-07-09 10:50:13,144][root][INFO] - ===> Epoch[11776](58880/5): Loss 0.0066, Sem 0.0066 LR: 2.782e-03 Score 99.763 Data time: 0.2228, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.6111 [2021-07-09 10:50:26,804][root][INFO] - ===> Epoch[11777](58885/5): Loss 0.0065, Sem 0.0065 LR: 2.770e-03 Score 99.770 Data time: 0.2452, Forward time: 1.2513, Backward time: 1.2490, DDP time: 0.0042, Total iter time: 2.6861 [2021-07-09 10:50:41,260][root][INFO] - ===> Epoch[11778](58890/5): Loss 0.0064, Sem 0.0064 LR: 2.759e-03 Score 99.773 Data time: 0.2160, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8505 [2021-07-09 10:50:56,036][root][INFO] - ===> Epoch[11779](58895/5): Loss 0.0061, Sem 0.0061 LR: 2.748e-03 Score 99.781 Data time: 0.1978, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.9126 [2021-07-09 10:51:09,833][root][INFO] - ===> Epoch[11780](58900/5): Loss 0.0066, Sem 0.0066 LR: 2.737e-03 Score 99.763 Data time: 0.2175, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7051 [2021-07-09 10:51:24,468][root][INFO] - ===> Epoch[11781](58905/5): Loss 0.0063, Sem 0.0063 LR: 2.726e-03 Score 99.777 Data time: 0.2570, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8873 [2021-07-09 10:51:38,241][root][INFO] - ===> Epoch[11782](58910/5): Loss 0.0062, Sem 0.0062 LR: 2.715e-03 Score 99.780 Data time: 0.2306, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7131 [2021-07-09 10:51:52,005][root][INFO] - ===> Epoch[11783](58915/5): Loss 0.0065, Sem 0.0065 LR: 2.703e-03 Score 99.769 Data time: 0.2455, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7116 [2021-07-09 10:52:06,470][root][INFO] - ===> Epoch[11784](58920/5): Loss 0.0064, Sem 0.0064 LR: 2.692e-03 Score 99.773 Data time: 0.2517, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8452 [2021-07-09 10:52:20,622][root][INFO] - ===> Epoch[11785](58925/5): Loss 0.0065, Sem 0.0065 LR: 2.681e-03 Score 99.770 Data time: 0.2451, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7924 [2021-07-09 10:52:36,005][root][INFO] - ===> Epoch[11786](58930/5): Loss 0.0062, Sem 0.0062 LR: 2.670e-03 Score 99.779 Data time: 0.2066, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 3.0410 [2021-07-09 10:52:50,049][root][INFO] - ===> Epoch[11787](58935/5): Loss 0.0066, Sem 0.0066 LR: 2.659e-03 Score 99.764 Data time: 0.1783, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7797 [2021-07-09 10:53:04,148][root][INFO] - ===> Epoch[11788](58940/5): Loss 0.0061, Sem 0.0061 LR: 2.647e-03 Score 99.783 Data time: 0.1997, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7804 [2021-07-09 10:53:18,496][root][INFO] - ===> Epoch[11789](58945/5): Loss 0.0065, Sem 0.0065 LR: 2.636e-03 Score 99.768 Data time: 0.2076, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8292 [2021-07-09 10:53:33,179][root][INFO] - ===> Epoch[11790](58950/5): Loss 0.0065, Sem 0.0065 LR: 2.625e-03 Score 99.768 Data time: 0.2157, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.9019 [2021-07-09 10:53:47,683][root][INFO] - ===> Epoch[11791](58955/5): Loss 0.0062, Sem 0.0062 LR: 2.614e-03 Score 99.774 Data time: 0.2240, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8629 [2021-07-09 10:54:02,213][root][INFO] - ===> Epoch[11792](58960/5): Loss 0.0064, Sem 0.0064 LR: 2.602e-03 Score 99.771 Data time: 0.1517, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8606 [2021-07-09 10:54:16,192][root][INFO] - ===> Epoch[11793](58965/5): Loss 0.0065, Sem 0.0065 LR: 2.591e-03 Score 99.768 Data time: 0.2127, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7620 [2021-07-09 10:54:30,941][root][INFO] - ===> Epoch[11794](58970/5): Loss 0.0062, Sem 0.0062 LR: 2.580e-03 Score 99.777 Data time: 0.2534, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8941 [2021-07-09 10:54:44,310][root][INFO] - ===> Epoch[11795](58975/5): Loss 0.0066, Sem 0.0066 LR: 2.569e-03 Score 99.764 Data time: 0.2150, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.6347 [2021-07-09 10:54:58,782][root][INFO] - ===> Epoch[11796](58980/5): Loss 0.0062, Sem 0.0062 LR: 2.557e-03 Score 99.780 Data time: 0.2427, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8537 [2021-07-09 10:55:13,615][root][INFO] - ===> Epoch[11797](58985/5): Loss 0.0063, Sem 0.0063 LR: 2.546e-03 Score 99.776 Data time: 0.2101, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.9344 [2021-07-09 10:55:27,124][root][INFO] - ===> Epoch[11798](58990/5): Loss 0.0065, Sem 0.0065 LR: 2.535e-03 Score 99.767 Data time: 0.1806, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.6532 [2021-07-09 10:55:42,168][root][INFO] - ===> Epoch[11799](58995/5): Loss 0.0064, Sem 0.0064 LR: 2.523e-03 Score 99.769 Data time: 0.2318, Forward time: 1.2514, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.9521 [2021-07-09 10:55:55,894][root][INFO] - ===> Epoch[11800](59000/5): Loss 0.0063, Sem 0.0063 LR: 2.512e-03 Score 99.774 Data time: 0.2596, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.6974 [2021-07-09 10:55:55,896][root][INFO] - ===> Start testing [2021-07-09 10:56:32,027][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1348 Loss 0.431 (AVG: 0.640) Score 91.708 (AVG: 91.106) mIOU 71.831 mAP 82.299 mAcc 78.087 IOU: 64.150 0.000 80.059 75.286 95.076 91.462 35.650 85.826 98.573 83.341 80.356 86.190 57.830 mAP: 82.298 2.032 87.463 85.468 98.233 97.857 54.290 97.866 99.571 94.232 90.300 91.419 88.853 mAcc: 79.191 0.000 83.372 83.434 96.749 97.612 38.968 95.788 99.370 92.985 90.159 96.429 61.075 [2021-07-09 10:56:32,027][root][INFO] - Finished test. Elapsed time: 36.1308 [2021-07-09 10:56:33,260][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 10:56:46,480][root][INFO] - ===> Epoch[11801](59005/5): Loss 0.0065, Sem 0.0065 LR: 2.501e-03 Score 99.772 Data time: 0.2139, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.6001 [2021-07-09 10:56:59,779][root][INFO] - ===> Epoch[11802](59010/5): Loss 0.0065, Sem 0.0065 LR: 2.490e-03 Score 99.769 Data time: 0.1842, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.6144 [2021-07-09 10:57:14,597][root][INFO] - ===> Epoch[11803](59015/5): Loss 0.0065, Sem 0.0065 LR: 2.478e-03 Score 99.768 Data time: 0.2581, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.9143 [2021-07-09 10:57:28,989][root][INFO] - ===> Epoch[11804](59020/5): Loss 0.0063, Sem 0.0063 LR: 2.467e-03 Score 99.777 Data time: 0.2405, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.8203 [2021-07-09 10:57:42,964][root][INFO] - ===> Epoch[11805](59025/5): Loss 0.0061, Sem 0.0061 LR: 2.456e-03 Score 99.785 Data time: 0.1809, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7463 [2021-07-09 10:57:56,968][root][INFO] - ===> Epoch[11806](59030/5): Loss 0.0067, Sem 0.0067 LR: 2.444e-03 Score 99.761 Data time: 0.1673, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7637 [2021-07-09 10:58:10,021][root][INFO] - ===> Epoch[11807](59035/5): Loss 0.0069, Sem 0.0069 LR: 2.433e-03 Score 99.756 Data time: 0.2077, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.5763 [2021-07-09 10:58:23,843][root][INFO] - ===> Epoch[11808](59040/5): Loss 0.0064, Sem 0.0064 LR: 2.422e-03 Score 99.773 Data time: 0.2382, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.7071 [2021-07-09 10:58:37,641][root][INFO] - ===> Epoch[11809](59045/5): Loss 0.0063, Sem 0.0063 LR: 2.410e-03 Score 99.776 Data time: 0.2040, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.7108 [2021-07-09 10:58:52,373][root][INFO] - ===> Epoch[11810](59050/5): Loss 0.0065, Sem 0.0065 LR: 2.399e-03 Score 99.770 Data time: 0.2079, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.9035 [2021-07-09 10:59:06,380][root][INFO] - ===> Epoch[11811](59055/5): Loss 0.0063, Sem 0.0063 LR: 2.388e-03 Score 99.774 Data time: 0.2236, Forward time: 1.2514, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.7423 [2021-07-09 10:59:19,460][root][INFO] - ===> Epoch[11812](59060/5): Loss 0.0065, Sem 0.0065 LR: 2.376e-03 Score 99.771 Data time: 0.2501, Forward time: 1.2514, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.5564 [2021-07-09 10:59:32,581][root][INFO] - ===> Epoch[11813](59065/5): Loss 0.0069, Sem 0.0069 LR: 2.365e-03 Score 99.753 Data time: 0.2358, Forward time: 1.2513, Backward time: 1.2491, DDP time: 0.0042, Total iter time: 2.5853 [2021-07-09 10:59:47,564][root][INFO] - ===> Epoch[11814](59070/5): Loss 0.0064, Sem 0.0064 LR: 2.353e-03 Score 99.770 Data time: 0.2068, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.9511 [2021-07-09 11:00:01,441][root][INFO] - ===> Epoch[11815](59075/5): Loss 0.0067, Sem 0.0067 LR: 2.342e-03 Score 99.761 Data time: 0.2079, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.7338 [2021-07-09 11:00:16,120][root][INFO] - ===> Epoch[11816](59080/5): Loss 0.0063, Sem 0.0063 LR: 2.331e-03 Score 99.777 Data time: 0.2089, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8831 [2021-07-09 11:00:30,304][root][INFO] - ===> Epoch[11817](59085/5): Loss 0.0065, Sem 0.0065 LR: 2.319e-03 Score 99.768 Data time: 0.1895, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.7963 [2021-07-09 11:00:44,012][root][INFO] - ===> Epoch[11818](59090/5): Loss 0.0064, Sem 0.0064 LR: 2.308e-03 Score 99.773 Data time: 0.2256, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.6840 [2021-07-09 11:00:57,956][root][INFO] - ===> Epoch[11819](59095/5): Loss 0.0064, Sem 0.0064 LR: 2.297e-03 Score 99.771 Data time: 0.2453, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.7477 [2021-07-09 11:01:11,445][root][INFO] - ===> Epoch[11820](59100/5): Loss 0.0064, Sem 0.0064 LR: 2.285e-03 Score 99.768 Data time: 0.2298, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.6515 [2021-07-09 11:01:25,745][root][INFO] - ===> Epoch[11821](59105/5): Loss 0.0064, Sem 0.0064 LR: 2.274e-03 Score 99.773 Data time: 0.1998, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8075 [2021-07-09 11:01:39,956][root][INFO] - ===> Epoch[11822](59110/5): Loss 0.0066, Sem 0.0066 LR: 2.262e-03 Score 99.763 Data time: 0.2131, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8022 [2021-07-09 11:01:53,991][root][INFO] - ===> Epoch[11823](59115/5): Loss 0.0063, Sem 0.0063 LR: 2.251e-03 Score 99.777 Data time: 0.2330, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.7652 [2021-07-09 11:02:07,858][root][INFO] - ===> Epoch[11824](59120/5): Loss 0.0065, Sem 0.0065 LR: 2.239e-03 Score 99.768 Data time: 0.2239, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.7292 [2021-07-09 11:02:21,513][root][INFO] - ===> Epoch[11825](59125/5): Loss 0.0065, Sem 0.0065 LR: 2.228e-03 Score 99.769 Data time: 0.2129, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.6761 [2021-07-09 11:02:35,873][root][INFO] - ===> Epoch[11826](59130/5): Loss 0.0064, Sem 0.0064 LR: 2.217e-03 Score 99.771 Data time: 0.2768, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8282 [2021-07-09 11:02:50,140][root][INFO] - ===> Epoch[11827](59135/5): Loss 0.0062, Sem 0.0062 LR: 2.205e-03 Score 99.781 Data time: 0.3124, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8009 [2021-07-09 11:03:05,136][root][INFO] - ===> Epoch[11828](59140/5): Loss 0.0064, Sem 0.0064 LR: 2.194e-03 Score 99.771 Data time: 0.2551, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.9515 [2021-07-09 11:03:19,607][root][INFO] - ===> Epoch[11829](59145/5): Loss 0.0065, Sem 0.0065 LR: 2.182e-03 Score 99.767 Data time: 0.2570, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8571 [2021-07-09 11:03:33,466][root][INFO] - ===> Epoch[11830](59150/5): Loss 0.0068, Sem 0.0068 LR: 2.171e-03 Score 99.758 Data time: 0.1869, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.7284 [2021-07-09 11:03:47,822][root][INFO] - ===> Epoch[11831](59155/5): Loss 0.0066, Sem 0.0066 LR: 2.159e-03 Score 99.765 Data time: 0.2426, Forward time: 1.2514, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8231 [2021-07-09 11:04:03,493][root][INFO] - ===> Epoch[11832](59160/5): Loss 0.0067, Sem 0.0067 LR: 2.148e-03 Score 99.759 Data time: 0.2794, Forward time: 1.2514, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 3.0918 [2021-07-09 11:04:17,258][root][INFO] - ===> Epoch[11833](59165/5): Loss 0.0065, Sem 0.0065 LR: 2.136e-03 Score 99.768 Data time: 0.2413, Forward time: 1.2514, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.7035 [2021-07-09 11:04:31,640][root][INFO] - ===> Epoch[11834](59170/5): Loss 0.0064, Sem 0.0064 LR: 2.125e-03 Score 99.772 Data time: 0.2271, Forward time: 1.2514, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8198 [2021-07-09 11:04:45,504][root][INFO] - ===> Epoch[11835](59175/5): Loss 0.0062, Sem 0.0062 LR: 2.113e-03 Score 99.776 Data time: 0.2305, Forward time: 1.2514, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.7295 [2021-07-09 11:04:59,770][root][INFO] - ===> Epoch[11836](59180/5): Loss 0.0061, Sem 0.0061 LR: 2.102e-03 Score 99.782 Data time: 0.1898, Forward time: 1.2514, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8095 [2021-07-09 11:05:14,130][root][INFO] - ===> Epoch[11837](59185/5): Loss 0.0066, Sem 0.0066 LR: 2.090e-03 Score 99.766 Data time: 0.2483, Forward time: 1.2514, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8211 [2021-07-09 11:05:28,487][root][INFO] - ===> Epoch[11838](59190/5): Loss 0.0064, Sem 0.0064 LR: 2.079e-03 Score 99.772 Data time: 0.2492, Forward time: 1.2514, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8308 [2021-07-09 11:05:41,834][root][INFO] - ===> Epoch[11839](59195/5): Loss 0.0063, Sem 0.0063 LR: 2.067e-03 Score 99.775 Data time: 0.2367, Forward time: 1.2514, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.6295 [2021-07-09 11:05:56,781][root][INFO] - ===> Epoch[11840](59200/5): Loss 0.0064, Sem 0.0064 LR: 2.056e-03 Score 99.774 Data time: 0.1918, Forward time: 1.2514, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.9421 [2021-07-09 11:05:56,783][root][INFO] - ===> Start testing [2021-07-09 11:06:35,948][root][INFO] - 68/68: Data time: 0.0013, Iter time: 0.1222 Loss 0.411 (AVG: 0.642) Score 91.923 (AVG: 91.064) mIOU 71.732 mAP 82.337 mAcc 78.062 IOU: 63.951 0.000 80.186 75.079 95.050 91.090 35.737 85.309 98.550 83.505 79.950 86.329 57.777 mAP: 82.425 2.444 87.291 85.299 98.301 97.839 54.525 97.801 99.583 94.429 90.286 91.407 88.750 mAcc: 79.241 0.000 83.561 83.338 96.680 97.708 38.937 95.161 99.333 93.498 90.127 96.466 60.750 [2021-07-09 11:06:35,948][root][INFO] - Finished test. Elapsed time: 39.1644 [2021-07-09 11:06:37,029][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 11:06:49,619][root][INFO] - ===> Epoch[11841](59205/5): Loss 0.0064, Sem 0.0064 LR: 2.044e-03 Score 99.770 Data time: 0.1330, Forward time: 1.2514, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.4788 [2021-07-09 11:07:03,161][root][INFO] - ===> Epoch[11842](59210/5): Loss 0.0064, Sem 0.0064 LR: 2.032e-03 Score 99.768 Data time: 0.2427, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.6611 [2021-07-09 11:07:17,590][root][INFO] - ===> Epoch[11843](59215/5): Loss 0.0062, Sem 0.0062 LR: 2.021e-03 Score 99.781 Data time: 0.2329, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8342 [2021-07-09 11:07:31,434][root][INFO] - ===> Epoch[11844](59220/5): Loss 0.0063, Sem 0.0063 LR: 2.009e-03 Score 99.777 Data time: 0.2331, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.7059 [2021-07-09 11:07:46,426][root][INFO] - ===> Epoch[11845](59225/5): Loss 0.0063, Sem 0.0063 LR: 1.998e-03 Score 99.775 Data time: 0.2471, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.9455 [2021-07-09 11:07:59,600][root][INFO] - ===> Epoch[11846](59230/5): Loss 0.0067, Sem 0.0067 LR: 1.986e-03 Score 99.758 Data time: 0.2047, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.5860 [2021-07-09 11:08:13,688][root][INFO] - ===> Epoch[11847](59235/5): Loss 0.0064, Sem 0.0064 LR: 1.975e-03 Score 99.773 Data time: 0.1835, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.7881 [2021-07-09 11:08:28,550][root][INFO] - ===> Epoch[11848](59240/5): Loss 0.0066, Sem 0.0066 LR: 1.963e-03 Score 99.766 Data time: 0.1988, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.9321 [2021-07-09 11:08:42,049][root][INFO] - ===> Epoch[11849](59245/5): Loss 0.0065, Sem 0.0065 LR: 1.951e-03 Score 99.768 Data time: 0.2142, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.6568 [2021-07-09 11:08:56,657][root][INFO] - ===> Epoch[11850](59250/5): Loss 0.0066, Sem 0.0066 LR: 1.940e-03 Score 99.770 Data time: 0.2619, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8686 [2021-07-09 11:09:10,144][root][INFO] - ===> Epoch[11851](59255/5): Loss 0.0062, Sem 0.0062 LR: 1.928e-03 Score 99.781 Data time: 0.2212, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.6627 [2021-07-09 11:09:23,805][root][INFO] - ===> Epoch[11852](59260/5): Loss 0.0065, Sem 0.0065 LR: 1.916e-03 Score 99.768 Data time: 0.2168, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.6900 [2021-07-09 11:09:38,357][root][INFO] - ===> Epoch[11853](59265/5): Loss 0.0065, Sem 0.0065 LR: 1.905e-03 Score 99.770 Data time: 0.1613, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8722 [2021-07-09 11:09:51,986][root][INFO] - ===> Epoch[11854](59270/5): Loss 0.0065, Sem 0.0065 LR: 1.893e-03 Score 99.766 Data time: 0.2129, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.6954 [2021-07-09 11:10:05,105][root][INFO] - ===> Epoch[11855](59275/5): Loss 0.0067, Sem 0.0067 LR: 1.881e-03 Score 99.761 Data time: 0.2124, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.5804 [2021-07-09 11:10:18,785][root][INFO] - ===> Epoch[11856](59280/5): Loss 0.0064, Sem 0.0064 LR: 1.870e-03 Score 99.773 Data time: 0.2172, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.6781 [2021-07-09 11:10:33,271][root][INFO] - ===> Epoch[11857](59285/5): Loss 0.0065, Sem 0.0065 LR: 1.858e-03 Score 99.768 Data time: 0.2322, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8438 [2021-07-09 11:10:47,786][root][INFO] - ===> Epoch[11858](59290/5): Loss 0.0066, Sem 0.0066 LR: 1.846e-03 Score 99.765 Data time: 0.1817, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8521 [2021-07-09 11:11:02,235][root][INFO] - ===> Epoch[11859](59295/5): Loss 0.0061, Sem 0.0061 LR: 1.835e-03 Score 99.780 Data time: 0.2386, Forward time: 1.2513, Backward time: 1.2492, DDP time: 0.0042, Total iter time: 2.8344 [2021-07-09 11:11:16,374][root][INFO] - ===> Epoch[11860](59300/5): Loss 0.0066, Sem 0.0066 LR: 1.823e-03 Score 99.762 Data time: 0.1796, Forward time: 1.2513, Backward time: 1.2493, DDP time: 0.0042, Total iter time: 2.7847 [2021-07-09 11:11:30,640][root][INFO] - ===> Epoch[11861](59305/5): Loss 0.0061, Sem 0.0061 LR: 1.811e-03 Score 99.782 Data time: 0.1798, Forward time: 1.2513, Backward time: 1.2493, DDP time: 0.0042, Total iter time: 2.8137 [2021-07-09 11:11:44,841][root][INFO] - ===> Epoch[11862](59310/5): Loss 0.0064, Sem 0.0064 LR: 1.800e-03 Score 99.770 Data time: 0.2242, Forward time: 1.2513, Backward time: 1.2493, DDP time: 0.0042, Total iter time: 2.7883 [2021-07-09 11:11:59,105][root][INFO] - ===> Epoch[11863](59315/5): Loss 0.0064, Sem 0.0064 LR: 1.788e-03 Score 99.773 Data time: 0.1911, Forward time: 1.2513, Backward time: 1.2493, DDP time: 0.0042, Total iter time: 2.8119 [2021-07-09 11:12:13,503][root][INFO] - ===> Epoch[11864](59320/5): Loss 0.0064, Sem 0.0064 LR: 1.776e-03 Score 99.774 Data time: 0.1470, Forward time: 1.2513, Backward time: 1.2493, DDP time: 0.0042, Total iter time: 2.8379 [2021-07-09 11:12:26,928][root][INFO] - ===> Epoch[11865](59325/5): Loss 0.0062, Sem 0.0062 LR: 1.764e-03 Score 99.778 Data time: 0.2344, Forward time: 1.2513, Backward time: 1.2493, DDP time: 0.0042, Total iter time: 2.6368 [2021-07-09 11:12:41,118][root][INFO] - ===> Epoch[11866](59330/5): Loss 0.0065, Sem 0.0065 LR: 1.753e-03 Score 99.769 Data time: 0.2062, Forward time: 1.2513, Backward time: 1.2493, DDP time: 0.0042, Total iter time: 2.7965 [2021-07-09 11:12:54,529][root][INFO] - ===> Epoch[11867](59335/5): Loss 0.0067, Sem 0.0067 LR: 1.741e-03 Score 99.762 Data time: 0.2445, Forward time: 1.2513, Backward time: 1.2493, DDP time: 0.0042, Total iter time: 2.6318 [2021-07-09 11:13:08,560][root][INFO] - ===> Epoch[11868](59340/5): Loss 0.0065, Sem 0.0065 LR: 1.729e-03 Score 99.770 Data time: 0.2110, Forward time: 1.2512, Backward time: 1.2493, DDP time: 0.0042, Total iter time: 2.7641 [2021-07-09 11:13:23,092][root][INFO] - ===> Epoch[11869](59345/5): Loss 0.0063, Sem 0.0063 LR: 1.717e-03 Score 99.776 Data time: 0.2165, Forward time: 1.2512, Backward time: 1.2493, DDP time: 0.0042, Total iter time: 2.8575 [2021-07-09 11:13:36,655][root][INFO] - ===> Epoch[11870](59350/5): Loss 0.0064, Sem 0.0064 LR: 1.706e-03 Score 99.773 Data time: 0.1906, Forward time: 1.2512, Backward time: 1.2493, DDP time: 0.0042, Total iter time: 2.6686 [2021-07-09 11:13:51,737][root][INFO] - ===> Epoch[11871](59355/5): Loss 0.0065, Sem 0.0065 LR: 1.694e-03 Score 99.768 Data time: 0.2377, Forward time: 1.2512, Backward time: 1.2493, DDP time: 0.0042, Total iter time: 2.9676 [2021-07-09 11:14:05,037][root][INFO] - ===> Epoch[11872](59360/5): Loss 0.0064, Sem 0.0064 LR: 1.682e-03 Score 99.772 Data time: 0.1783, Forward time: 1.2512, Backward time: 1.2493, DDP time: 0.0042, Total iter time: 2.6187 [2021-07-09 11:14:19,311][root][INFO] - ===> Epoch[11873](59365/5): Loss 0.0061, Sem 0.0061 LR: 1.670e-03 Score 99.784 Data time: 0.2469, Forward time: 1.2512, Backward time: 1.2493, DDP time: 0.0042, Total iter time: 2.8048 [2021-07-09 11:14:33,668][root][INFO] - ===> Epoch[11874](59370/5): Loss 0.0062, Sem 0.0062 LR: 1.658e-03 Score 99.778 Data time: 0.2363, Forward time: 1.2512, Backward time: 1.2493, DDP time: 0.0042, Total iter time: 2.8300 [2021-07-09 11:14:47,607][root][INFO] - ===> Epoch[11875](59375/5): Loss 0.0067, Sem 0.0067 LR: 1.647e-03 Score 99.762 Data time: 0.1409, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.7527 [2021-07-09 11:15:01,479][root][INFO] - ===> Epoch[11876](59380/5): Loss 0.0064, Sem 0.0064 LR: 1.635e-03 Score 99.771 Data time: 0.1879, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.7243 [2021-07-09 11:15:15,437][root][INFO] - ===> Epoch[11877](59385/5): Loss 0.0061, Sem 0.0061 LR: 1.623e-03 Score 99.783 Data time: 0.1923, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.7496 [2021-07-09 11:15:29,381][root][INFO] - ===> Epoch[11878](59390/5): Loss 0.0064, Sem 0.0064 LR: 1.611e-03 Score 99.771 Data time: 0.2076, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.7297 [2021-07-09 11:15:43,853][root][INFO] - ===> Epoch[11879](59395/5): Loss 0.0062, Sem 0.0062 LR: 1.599e-03 Score 99.778 Data time: 0.2704, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.8443 [2021-07-09 11:15:58,164][root][INFO] - ===> Epoch[11880](59400/5): Loss 0.0065, Sem 0.0065 LR: 1.587e-03 Score 99.768 Data time: 0.2640, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.8039 [2021-07-09 11:15:58,166][root][INFO] - ===> Start testing [2021-07-09 11:16:41,161][root][INFO] - 68/68: Data time: 0.0026, Iter time: 0.1417 Loss 0.411 (AVG: 0.631) Score 91.811 (AVG: 91.106) mIOU 71.897 mAP 82.350 mAcc 78.037 IOU: 64.281 0.000 80.160 75.169 95.046 91.554 34.929 85.445 98.571 84.797 80.122 86.278 58.305 mAP: 82.652 2.157 87.353 85.305 98.268 97.864 54.230 97.813 99.597 94.329 90.216 91.715 89.057 mAcc: 79.993 0.000 83.544 83.025 96.692 97.413 37.914 95.376 99.363 92.719 90.252 96.373 61.820 [2021-07-09 11:16:41,161][root][INFO] - Finished test. Elapsed time: 42.9936 [2021-07-09 11:16:42,113][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 11:16:54,946][root][INFO] - ===> Epoch[11881](59405/5): Loss 0.0066, Sem 0.0066 LR: 1.575e-03 Score 99.766 Data time: 0.2621, Forward time: 1.2512, Backward time: 1.2493, DDP time: 0.0042, Total iter time: 2.5207 [2021-07-09 11:17:09,696][root][INFO] - ===> Epoch[11882](59410/5): Loss 0.0069, Sem 0.0069 LR: 1.563e-03 Score 99.754 Data time: 0.2911, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.9072 [2021-07-09 11:17:23,269][root][INFO] - ===> Epoch[11883](59415/5): Loss 0.0066, Sem 0.0066 LR: 1.552e-03 Score 99.765 Data time: 0.2409, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.6637 [2021-07-09 11:17:37,554][root][INFO] - ===> Epoch[11884](59420/5): Loss 0.0064, Sem 0.0064 LR: 1.540e-03 Score 99.771 Data time: 0.2145, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.8113 [2021-07-09 11:17:52,034][root][INFO] - ===> Epoch[11885](59425/5): Loss 0.0065, Sem 0.0065 LR: 1.528e-03 Score 99.768 Data time: 0.3331, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.8531 [2021-07-09 11:18:06,611][root][INFO] - ===> Epoch[11886](59430/5): Loss 0.0063, Sem 0.0063 LR: 1.516e-03 Score 99.777 Data time: 0.2021, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.8699 [2021-07-09 11:18:20,947][root][INFO] - ===> Epoch[11887](59435/5): Loss 0.0064, Sem 0.0064 LR: 1.504e-03 Score 99.770 Data time: 0.2706, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.8200 [2021-07-09 11:18:35,950][root][INFO] - ===> Epoch[11888](59440/5): Loss 0.0063, Sem 0.0063 LR: 1.492e-03 Score 99.775 Data time: 0.2256, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.9517 [2021-07-09 11:18:49,907][root][INFO] - ===> Epoch[11889](59445/5): Loss 0.0064, Sem 0.0064 LR: 1.480e-03 Score 99.773 Data time: 0.2508, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.7407 [2021-07-09 11:19:04,466][root][INFO] - ===> Epoch[11890](59450/5): Loss 0.0062, Sem 0.0062 LR: 1.468e-03 Score 99.780 Data time: 0.2165, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.8672 [2021-07-09 11:19:19,788][root][INFO] - ===> Epoch[11891](59455/5): Loss 0.0064, Sem 0.0064 LR: 1.456e-03 Score 99.773 Data time: 0.2361, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 3.0104 [2021-07-09 11:19:34,344][root][INFO] - ===> Epoch[11892](59460/5): Loss 0.0062, Sem 0.0062 LR: 1.444e-03 Score 99.778 Data time: 0.1533, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.8779 [2021-07-09 11:19:47,628][root][INFO] - ===> Epoch[11893](59465/5): Loss 0.0064, Sem 0.0064 LR: 1.432e-03 Score 99.772 Data time: 0.2043, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.6146 [2021-07-09 11:20:01,785][root][INFO] - ===> Epoch[11894](59470/5): Loss 0.0063, Sem 0.0063 LR: 1.420e-03 Score 99.777 Data time: 0.2466, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.7796 [2021-07-09 11:20:15,594][root][INFO] - ===> Epoch[11895](59475/5): Loss 0.0065, Sem 0.0065 LR: 1.408e-03 Score 99.767 Data time: 0.2019, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.7263 [2021-07-09 11:20:29,258][root][INFO] - ===> Epoch[11896](59480/5): Loss 0.0067, Sem 0.0067 LR: 1.396e-03 Score 99.759 Data time: 0.2540, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.6832 [2021-07-09 11:20:43,891][root][INFO] - ===> Epoch[11897](59485/5): Loss 0.0063, Sem 0.0063 LR: 1.384e-03 Score 99.775 Data time: 0.2633, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.8819 [2021-07-09 11:20:57,724][root][INFO] - ===> Epoch[11898](59490/5): Loss 0.0065, Sem 0.0065 LR: 1.372e-03 Score 99.769 Data time: 0.2409, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.7315 [2021-07-09 11:21:11,787][root][INFO] - ===> Epoch[11899](59495/5): Loss 0.0064, Sem 0.0064 LR: 1.360e-03 Score 99.770 Data time: 0.1576, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.7856 [2021-07-09 11:21:24,886][root][INFO] - ===> Epoch[11900](59500/5): Loss 0.0065, Sem 0.0065 LR: 1.347e-03 Score 99.768 Data time: 0.1873, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.5769 [2021-07-09 11:21:38,572][root][INFO] - ===> Epoch[11901](59505/5): Loss 0.0067, Sem 0.0067 LR: 1.335e-03 Score 99.762 Data time: 0.2097, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.6950 [2021-07-09 11:21:52,114][root][INFO] - ===> Epoch[11902](59510/5): Loss 0.0065, Sem 0.0065 LR: 1.323e-03 Score 99.767 Data time: 0.2444, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.6598 [2021-07-09 11:22:05,653][root][INFO] - ===> Epoch[11903](59515/5): Loss 0.0064, Sem 0.0064 LR: 1.311e-03 Score 99.772 Data time: 0.1816, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.6714 [2021-07-09 11:22:20,489][root][INFO] - ===> Epoch[11904](59520/5): Loss 0.0059, Sem 0.0059 LR: 1.299e-03 Score 99.791 Data time: 0.2138, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.9293 [2021-07-09 11:22:34,171][root][INFO] - ===> Epoch[11905](59525/5): Loss 0.0065, Sem 0.0065 LR: 1.287e-03 Score 99.769 Data time: 0.2203, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.6863 [2021-07-09 11:22:47,281][root][INFO] - ===> Epoch[11906](59530/5): Loss 0.0069, Sem 0.0069 LR: 1.275e-03 Score 99.754 Data time: 0.2217, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.5724 [2021-07-09 11:23:02,202][root][INFO] - ===> Epoch[11907](59535/5): Loss 0.0061, Sem 0.0061 LR: 1.262e-03 Score 99.784 Data time: 0.2114, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.9433 [2021-07-09 11:23:15,927][root][INFO] - ===> Epoch[11908](59540/5): Loss 0.0066, Sem 0.0066 LR: 1.250e-03 Score 99.765 Data time: 0.2399, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.6929 [2021-07-09 11:23:29,793][root][INFO] - ===> Epoch[11909](59545/5): Loss 0.0065, Sem 0.0065 LR: 1.238e-03 Score 99.771 Data time: 0.2354, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.7293 [2021-07-09 11:23:44,744][root][INFO] - ===> Epoch[11910](59550/5): Loss 0.0065, Sem 0.0065 LR: 1.226e-03 Score 99.768 Data time: 0.2237, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.9479 [2021-07-09 11:23:58,525][root][INFO] - ===> Epoch[11911](59555/5): Loss 0.0063, Sem 0.0063 LR: 1.214e-03 Score 99.775 Data time: 0.1795, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.7115 [2021-07-09 11:24:13,045][root][INFO] - ===> Epoch[11912](59560/5): Loss 0.0064, Sem 0.0064 LR: 1.201e-03 Score 99.771 Data time: 0.2457, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.8559 [2021-07-09 11:24:26,920][root][INFO] - ===> Epoch[11913](59565/5): Loss 0.0062, Sem 0.0062 LR: 1.189e-03 Score 99.779 Data time: 0.2981, Forward time: 1.2512, Backward time: 1.2494, DDP time: 0.0042, Total iter time: 2.7315 [2021-07-09 11:24:41,374][root][INFO] - ===> Epoch[11914](59570/5): Loss 0.0065, Sem 0.0065 LR: 1.177e-03 Score 99.770 Data time: 0.2262, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.8488 [2021-07-09 11:24:55,591][root][INFO] - ===> Epoch[11915](59575/5): Loss 0.0063, Sem 0.0063 LR: 1.164e-03 Score 99.773 Data time: 0.2519, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.8005 [2021-07-09 11:25:09,679][root][INFO] - ===> Epoch[11916](59580/5): Loss 0.0064, Sem 0.0064 LR: 1.152e-03 Score 99.769 Data time: 0.1806, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7683 [2021-07-09 11:25:23,486][root][INFO] - ===> Epoch[11917](59585/5): Loss 0.0065, Sem 0.0065 LR: 1.140e-03 Score 99.768 Data time: 0.1740, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7190 [2021-07-09 11:25:38,212][root][INFO] - ===> Epoch[11918](59590/5): Loss 0.0063, Sem 0.0063 LR: 1.127e-03 Score 99.776 Data time: 0.2497, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.8939 [2021-07-09 11:25:52,303][root][INFO] - ===> Epoch[11919](59595/5): Loss 0.0064, Sem 0.0064 LR: 1.115e-03 Score 99.773 Data time: 0.1920, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7722 [2021-07-09 11:26:06,204][root][INFO] - ===> Epoch[11920](59600/5): Loss 0.0063, Sem 0.0063 LR: 1.103e-03 Score 99.774 Data time: 0.2300, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7273 [2021-07-09 11:26:06,206][root][INFO] - ===> Start testing [2021-07-09 11:26:45,183][root][INFO] - 68/68: Data time: 0.0015, Iter time: 0.1464 Loss 0.421 (AVG: 0.632) Score 91.496 (AVG: 91.090) mIOU 71.816 mAP 82.373 mAcc 78.026 IOU: 64.068 0.000 79.867 75.209 95.025 91.495 34.162 85.860 98.569 84.482 80.208 86.259 58.407 mAP: 82.720 2.308 87.743 85.498 98.282 97.907 53.562 97.841 99.582 94.400 90.287 91.602 89.121 mAcc: 79.350 0.000 83.582 83.461 96.704 97.646 37.032 95.488 99.386 93.084 90.140 96.338 62.125 [2021-07-09 11:26:45,183][root][INFO] - Finished test. Elapsed time: 38.9771 [2021-07-09 11:26:46,130][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 11:26:59,416][root][INFO] - ===> Epoch[11921](59605/5): Loss 0.0066, Sem 0.0066 LR: 1.090e-03 Score 99.762 Data time: 0.1992, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.6250 [2021-07-09 11:27:13,440][root][INFO] - ===> Epoch[11922](59610/5): Loss 0.0061, Sem 0.0061 LR: 1.078e-03 Score 99.784 Data time: 0.1965, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7593 [2021-07-09 11:27:27,909][root][INFO] - ===> Epoch[11923](59615/5): Loss 0.0063, Sem 0.0063 LR: 1.066e-03 Score 99.774 Data time: 0.2541, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.8522 [2021-07-09 11:27:41,132][root][INFO] - ===> Epoch[11924](59620/5): Loss 0.0065, Sem 0.0065 LR: 1.053e-03 Score 99.767 Data time: 0.2116, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.6003 [2021-07-09 11:27:55,253][root][INFO] - ===> Epoch[11925](59625/5): Loss 0.0062, Sem 0.0062 LR: 1.041e-03 Score 99.781 Data time: 0.1587, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7905 [2021-07-09 11:28:08,075][root][INFO] - ===> Epoch[11926](59630/5): Loss 0.0067, Sem 0.0067 LR: 1.028e-03 Score 99.761 Data time: 0.1785, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.5148 [2021-07-09 11:28:22,579][root][INFO] - ===> Epoch[11927](59635/5): Loss 0.0063, Sem 0.0063 LR: 1.016e-03 Score 99.776 Data time: 0.1814, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.8662 [2021-07-09 11:28:36,157][root][INFO] - ===> Epoch[11928](59640/5): Loss 0.0063, Sem 0.0063 LR: 1.003e-03 Score 99.772 Data time: 0.2149, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.6617 [2021-07-09 11:28:50,189][root][INFO] - ===> Epoch[11929](59645/5): Loss 0.0063, Sem 0.0063 LR: 9.907e-04 Score 99.777 Data time: 0.1783, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7531 [2021-07-09 11:29:04,420][root][INFO] - ===> Epoch[11930](59650/5): Loss 0.0064, Sem 0.0064 LR: 9.782e-04 Score 99.769 Data time: 0.2397, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7974 [2021-07-09 11:29:18,037][root][INFO] - ===> Epoch[11931](59655/5): Loss 0.0064, Sem 0.0064 LR: 9.657e-04 Score 99.772 Data time: 0.1968, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.6811 [2021-07-09 11:29:33,460][root][INFO] - ===> Epoch[11932](59660/5): Loss 0.0061, Sem 0.0061 LR: 9.531e-04 Score 99.782 Data time: 0.2076, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 3.0217 [2021-07-09 11:29:47,700][root][INFO] - ===> Epoch[11933](59665/5): Loss 0.0064, Sem 0.0064 LR: 9.405e-04 Score 99.772 Data time: 0.2042, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.8036 [2021-07-09 11:30:01,853][root][INFO] - ===> Epoch[11934](59670/5): Loss 0.0063, Sem 0.0063 LR: 9.279e-04 Score 99.777 Data time: 0.2304, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7857 [2021-07-09 11:30:16,453][root][INFO] - ===> Epoch[11935](59675/5): Loss 0.0062, Sem 0.0062 LR: 9.153e-04 Score 99.776 Data time: 0.2060, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.8679 [2021-07-09 11:30:30,092][root][INFO] - ===> Epoch[11936](59680/5): Loss 0.0068, Sem 0.0068 LR: 9.026e-04 Score 99.763 Data time: 0.2332, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.6884 [2021-07-09 11:30:45,840][root][INFO] - ===> Epoch[11937](59685/5): Loss 0.0063, Sem 0.0063 LR: 8.900e-04 Score 99.775 Data time: 0.2244, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 3.1074 [2021-07-09 11:30:59,719][root][INFO] - ===> Epoch[11938](59690/5): Loss 0.0061, Sem 0.0061 LR: 8.773e-04 Score 99.784 Data time: 0.1757, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7397 [2021-07-09 11:31:14,134][root][INFO] - ===> Epoch[11939](59695/5): Loss 0.0065, Sem 0.0065 LR: 8.646e-04 Score 99.767 Data time: 0.2518, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.8462 [2021-07-09 11:31:28,273][root][INFO] - ===> Epoch[11940](59700/5): Loss 0.0063, Sem 0.0063 LR: 8.519e-04 Score 99.775 Data time: 0.2670, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7797 [2021-07-09 11:31:42,890][root][INFO] - ===> Epoch[11941](59705/5): Loss 0.0064, Sem 0.0064 LR: 8.391e-04 Score 99.771 Data time: 0.2673, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.8734 [2021-07-09 11:31:56,679][root][INFO] - ===> Epoch[11942](59710/5): Loss 0.0063, Sem 0.0063 LR: 8.263e-04 Score 99.777 Data time: 0.2930, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.6977 [2021-07-09 11:32:10,555][root][INFO] - ===> Epoch[11943](59715/5): Loss 0.0062, Sem 0.0062 LR: 8.136e-04 Score 99.779 Data time: 0.2675, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7275 [2021-07-09 11:32:25,041][root][INFO] - ===> Epoch[11944](59720/5): Loss 0.0067, Sem 0.0067 LR: 8.007e-04 Score 99.762 Data time: 0.2325, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.8460 [2021-07-09 11:32:38,420][root][INFO] - ===> Epoch[11945](59725/5): Loss 0.0068, Sem 0.0068 LR: 7.879e-04 Score 99.757 Data time: 0.2490, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.6350 [2021-07-09 11:32:52,617][root][INFO] - ===> Epoch[11946](59730/5): Loss 0.0065, Sem 0.0065 LR: 7.751e-04 Score 99.768 Data time: 0.2117, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7969 [2021-07-09 11:33:06,859][root][INFO] - ===> Epoch[11947](59735/5): Loss 0.0063, Sem 0.0063 LR: 7.622e-04 Score 99.776 Data time: 0.2077, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.8005 [2021-07-09 11:33:20,901][root][INFO] - ===> Epoch[11948](59740/5): Loss 0.0064, Sem 0.0064 LR: 7.493e-04 Score 99.772 Data time: 0.2633, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7538 [2021-07-09 11:33:35,084][root][INFO] - ===> Epoch[11949](59745/5): Loss 0.0062, Sem 0.0062 LR: 7.363e-04 Score 99.780 Data time: 0.2440, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7857 [2021-07-09 11:33:49,444][root][INFO] - ===> Epoch[11950](59750/5): Loss 0.0065, Sem 0.0065 LR: 7.234e-04 Score 99.769 Data time: 0.2519, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.8302 [2021-07-09 11:34:03,187][root][INFO] - ===> Epoch[11951](59755/5): Loss 0.0063, Sem 0.0063 LR: 7.104e-04 Score 99.774 Data time: 0.2267, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.6995 [2021-07-09 11:34:17,223][root][INFO] - ===> Epoch[11952](59760/5): Loss 0.0064, Sem 0.0064 LR: 6.974e-04 Score 99.770 Data time: 0.2244, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7676 [2021-07-09 11:34:31,821][root][INFO] - ===> Epoch[11953](59765/5): Loss 0.0062, Sem 0.0062 LR: 6.843e-04 Score 99.783 Data time: 0.2216, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.8779 [2021-07-09 11:34:46,117][root][INFO] - ===> Epoch[11954](59770/5): Loss 0.0062, Sem 0.0062 LR: 6.713e-04 Score 99.777 Data time: 0.2695, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7991 [2021-07-09 11:34:59,742][root][INFO] - ===> Epoch[11955](59775/5): Loss 0.0064, Sem 0.0064 LR: 6.582e-04 Score 99.773 Data time: 0.2234, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.6698 [2021-07-09 11:35:13,824][root][INFO] - ===> Epoch[11956](59780/5): Loss 0.0063, Sem 0.0063 LR: 6.451e-04 Score 99.778 Data time: 0.2085, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7744 [2021-07-09 11:35:28,857][root][INFO] - ===> Epoch[11957](59785/5): Loss 0.0064, Sem 0.0064 LR: 6.319e-04 Score 99.773 Data time: 0.2475, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.9634 [2021-07-09 11:35:42,393][root][INFO] - ===> Epoch[11958](59790/5): Loss 0.0065, Sem 0.0065 LR: 6.187e-04 Score 99.770 Data time: 0.2140, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.6703 [2021-07-09 11:35:56,207][root][INFO] - ===> Epoch[11959](59795/5): Loss 0.0063, Sem 0.0063 LR: 6.055e-04 Score 99.775 Data time: 0.1836, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7166 [2021-07-09 11:36:10,304][root][INFO] - ===> Epoch[11960](59800/5): Loss 0.0062, Sem 0.0062 LR: 5.923e-04 Score 99.779 Data time: 0.2792, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7686 [2021-07-09 11:36:10,305][root][INFO] - ===> Start testing [2021-07-09 11:36:47,996][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1279 Loss 0.425 (AVG: 0.636) Score 91.525 (AVG: 91.108) mIOU 71.822 mAP 82.367 mAcc 78.115 IOU: 64.296 0.000 80.099 75.119 95.095 91.320 34.874 85.857 98.565 83.777 80.028 86.279 58.373 mAP: 82.600 2.267 87.453 85.343 98.261 97.898 54.108 97.867 99.589 94.447 90.188 91.628 89.121 mAcc: 80.130 0.000 83.701 82.906 96.702 97.626 37.897 95.704 99.359 93.084 90.174 96.310 61.897 [2021-07-09 11:36:47,996][root][INFO] - Finished test. Elapsed time: 37.6901 [2021-07-09 11:36:48,967][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 11:37:02,030][root][INFO] - ===> Epoch[11961](59805/5): Loss 0.0067, Sem 0.0067 LR: 5.790e-04 Score 99.763 Data time: 0.1670, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.5735 [2021-07-09 11:37:15,352][root][INFO] - ===> Epoch[11962](59810/5): Loss 0.0066, Sem 0.0066 LR: 5.657e-04 Score 99.762 Data time: 0.2230, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.6166 [2021-07-09 11:37:29,726][root][INFO] - ===> Epoch[11963](59815/5): Loss 0.0066, Sem 0.0066 LR: 5.524e-04 Score 99.766 Data time: 0.2933, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.8158 [2021-07-09 11:37:43,616][root][INFO] - ===> Epoch[11964](59820/5): Loss 0.0064, Sem 0.0064 LR: 5.390e-04 Score 99.770 Data time: 0.2102, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7286 [2021-07-09 11:37:57,808][root][INFO] - ===> Epoch[11965](59825/5): Loss 0.0062, Sem 0.0062 LR: 5.256e-04 Score 99.778 Data time: 0.1919, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7979 [2021-07-09 11:38:11,782][root][INFO] - ===> Epoch[11966](59830/5): Loss 0.0064, Sem 0.0064 LR: 5.121e-04 Score 99.770 Data time: 0.2032, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7506 [2021-07-09 11:38:26,885][root][INFO] - ===> Epoch[11967](59835/5): Loss 0.0065, Sem 0.0065 LR: 4.986e-04 Score 99.767 Data time: 0.2844, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.9738 [2021-07-09 11:38:41,567][root][INFO] - ===> Epoch[11968](59840/5): Loss 0.0063, Sem 0.0063 LR: 4.851e-04 Score 99.775 Data time: 0.2597, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.8873 [2021-07-09 11:38:56,658][root][INFO] - ===> Epoch[11969](59845/5): Loss 0.0063, Sem 0.0063 LR: 4.715e-04 Score 99.774 Data time: 0.2111, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.9761 [2021-07-09 11:39:10,758][root][INFO] - ===> Epoch[11970](59850/5): Loss 0.0063, Sem 0.0063 LR: 4.579e-04 Score 99.777 Data time: 0.2951, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7686 [2021-07-09 11:39:25,088][root][INFO] - ===> Epoch[11971](59855/5): Loss 0.0065, Sem 0.0065 LR: 4.442e-04 Score 99.768 Data time: 0.2428, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.8252 [2021-07-09 11:39:38,223][root][INFO] - ===> Epoch[11972](59860/5): Loss 0.0066, Sem 0.0066 LR: 4.305e-04 Score 99.767 Data time: 0.1849, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.5874 [2021-07-09 11:39:52,324][root][INFO] - ===> Epoch[11973](59865/5): Loss 0.0062, Sem 0.0062 LR: 4.167e-04 Score 99.777 Data time: 0.2026, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7749 [2021-07-09 11:40:06,479][root][INFO] - ===> Epoch[11974](59870/5): Loss 0.0062, Sem 0.0062 LR: 4.029e-04 Score 99.777 Data time: 0.2502, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7731 [2021-07-09 11:40:20,471][root][INFO] - ===> Epoch[11975](59875/5): Loss 0.0064, Sem 0.0064 LR: 3.890e-04 Score 99.772 Data time: 0.2086, Forward time: 1.2512, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7590 [2021-07-09 11:40:34,470][root][INFO] - ===> Epoch[11976](59880/5): Loss 0.0064, Sem 0.0064 LR: 3.751e-04 Score 99.773 Data time: 0.1986, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.7507 [2021-07-09 11:40:48,044][root][INFO] - ===> Epoch[11977](59885/5): Loss 0.0064, Sem 0.0064 LR: 3.611e-04 Score 99.774 Data time: 0.2189, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.6684 [2021-07-09 11:41:02,532][root][INFO] - ===> Epoch[11978](59890/5): Loss 0.0064, Sem 0.0064 LR: 3.471e-04 Score 99.774 Data time: 0.2389, Forward time: 1.2511, Backward time: 1.2495, DDP time: 0.0042, Total iter time: 2.8525 [2021-07-09 11:41:17,047][root][INFO] - ===> Epoch[11979](59895/5): Loss 0.0064, Sem 0.0064 LR: 3.330e-04 Score 99.773 Data time: 0.2327, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.8607 [2021-07-09 11:41:31,267][root][INFO] - ===> Epoch[11980](59900/5): Loss 0.0061, Sem 0.0061 LR: 3.188e-04 Score 99.784 Data time: 0.2179, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.8042 [2021-07-09 11:41:45,360][root][INFO] - ===> Epoch[11981](59905/5): Loss 0.0066, Sem 0.0066 LR: 3.046e-04 Score 99.762 Data time: 0.2448, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.7714 [2021-07-09 11:41:59,386][root][INFO] - ===> Epoch[11982](59910/5): Loss 0.0064, Sem 0.0064 LR: 2.903e-04 Score 99.769 Data time: 0.2435, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.7573 [2021-07-09 11:42:13,627][root][INFO] - ===> Epoch[11983](59915/5): Loss 0.0062, Sem 0.0062 LR: 2.759e-04 Score 99.780 Data time: 0.2791, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.7984 [2021-07-09 11:42:27,089][root][INFO] - ===> Epoch[11984](59920/5): Loss 0.0064, Sem 0.0064 LR: 2.614e-04 Score 99.770 Data time: 0.1555, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.6635 [2021-07-09 11:42:40,816][root][INFO] - ===> Epoch[11985](59925/5): Loss 0.0063, Sem 0.0063 LR: 2.468e-04 Score 99.771 Data time: 0.2213, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.6967 [2021-07-09 11:42:54,927][root][INFO] - ===> Epoch[11986](59930/5): Loss 0.0064, Sem 0.0064 LR: 2.322e-04 Score 99.772 Data time: 0.2460, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.7711 [2021-07-09 11:43:08,950][root][INFO] - ===> Epoch[11987](59935/5): Loss 0.0066, Sem 0.0066 LR: 2.174e-04 Score 99.767 Data time: 0.2079, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.7635 [2021-07-09 11:43:22,943][root][INFO] - ===> Epoch[11988](59940/5): Loss 0.0064, Sem 0.0064 LR: 2.025e-04 Score 99.774 Data time: 0.2115, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.7555 [2021-07-09 11:43:36,825][root][INFO] - ===> Epoch[11989](59945/5): Loss 0.0064, Sem 0.0064 LR: 1.875e-04 Score 99.771 Data time: 0.1980, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.7340 [2021-07-09 11:43:50,386][root][INFO] - ===> Epoch[11990](59950/5): Loss 0.0064, Sem 0.0064 LR: 1.724e-04 Score 99.771 Data time: 0.2210, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.6599 [2021-07-09 11:44:05,265][root][INFO] - ===> Epoch[11991](59955/5): Loss 0.0065, Sem 0.0065 LR: 1.571e-04 Score 99.768 Data time: 0.2427, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.9328 [2021-07-09 11:44:18,102][root][INFO] - ===> Epoch[11992](59960/5): Loss 0.0065, Sem 0.0065 LR: 1.416e-04 Score 99.769 Data time: 0.2004, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.5185 [2021-07-09 11:44:31,841][root][INFO] - ===> Epoch[11993](59965/5): Loss 0.0066, Sem 0.0066 LR: 1.260e-04 Score 99.764 Data time: 0.2261, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.6899 [2021-07-09 11:44:46,177][root][INFO] - ===> Epoch[11994](59970/5): Loss 0.0068, Sem 0.0068 LR: 1.101e-04 Score 99.758 Data time: 0.1882, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.8181 [2021-07-09 11:44:59,854][root][INFO] - ===> Epoch[11995](59975/5): Loss 0.0063, Sem 0.0063 LR: 9.400e-05 Score 99.773 Data time: 0.2246, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.6792 [2021-07-09 11:45:14,109][root][INFO] - ===> Epoch[11996](59980/5): Loss 0.0067, Sem 0.0067 LR: 7.756e-05 Score 99.763 Data time: 0.2391, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.8069 [2021-07-09 11:45:28,147][root][INFO] - ===> Epoch[11997](59985/5): Loss 0.0060, Sem 0.0060 LR: 6.072e-05 Score 99.787 Data time: 0.2019, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.7607 [2021-07-09 11:45:41,912][root][INFO] - ===> Epoch[11998](59990/5): Loss 0.0064, Sem 0.0064 LR: 4.334e-05 Score 99.772 Data time: 0.2243, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.7084 [2021-07-09 11:45:56,590][root][INFO] - ===> Epoch[11999](59995/5): Loss 0.0061, Sem 0.0061 LR: 2.512e-05 Score 99.781 Data time: 0.2689, Forward time: 1.2511, Backward time: 1.2496, DDP time: 0.0042, Total iter time: 2.8877 [2021-07-09 11:46:11,018][root][INFO] - ===> Start testing [2021-07-09 11:46:44,886][root][INFO] - 68/68: Data time: 0.0012, Iter time: 0.1269 Loss 0.411 (AVG: 0.636) Score 91.936 (AVG: 91.097) mIOU 71.877 mAP 82.287 mAcc 78.113 IOU: 64.235 0.000 80.204 75.107 95.077 91.503 35.136 85.801 98.576 84.406 80.119 86.227 58.011 mAP: 82.412 2.007 87.212 85.073 98.203 97.885 53.851 97.854 99.565 94.489 90.341 92.054 88.786 mAcc: 79.809 0.000 83.382 82.758 96.811 97.658 38.401 95.717 99.382 93.508 90.164 96.387 61.489 [2021-07-09 11:46:44,886][root][INFO] - Finished test. Elapsed time: 33.8666 [2021-07-09 11:46:45,839][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C.pth [2021-07-09 11:46:46,544][submitit][INFO] - Job completed successfully