[2020-12-31 06:59:01,397][root][INFO] - ===> Configurations [2020-12-31 06:59:01,410][root][INFO] - net: model: Res16UNet34C conv1_kernel_size: 3 weights: /checkpoint/jihou/data_efficient3d/checkpoint/scannet/pretrain/20k/partition8/4096/2/weights/checkpoint.pth weights_for_inner_model: false dilations: - 1 - 1 - 1 - 1 wrapper_type: null wrapper_region_type: 1 wrapper_kernel_size: 3 wrapper_lr: 0.1 meanfield_iterations: 10 crf_spatial_sigma: 1 crf_chromatic_sigma: 12 optimizer: optimizer: SGD lr: 0.1 sgd_momentum: 0.9 sgd_dampening: 0.1 adam_beta1: 0.9 adam_beta2: 0.999 weight_decay: 0.0001 param_histogram_freq: 100 save_param_histogram: false iter_size: 1 bn_momentum: 0.02 scheduler: PolyLR max_iter: 15000 step_size: 20000.0 step_gamma: 0.1 poly_power: 0.9 exp_gamma: 0.95 exp_step_size: 445 data: dataset: StanfordArea5Dataset train_file: null voxel_size: 0.05 data_dir: data sampled_inds: null temporal_dilation: 30 temporal_numseq: 3 point_lim: -1 pre_point_lim: -1 batch_size: 6 val_batch_size: 1 test_batch_size: 1 cache_data: false num_workers: 1 num_val_workers: 1 ignore_label: 255 return_transformation: false ignore_duplicate_class: false partial_crop: 0 train_limit_numpoints: 0 synthia_path: /home/chrischoy/datasets/Synthia/Synthia4D synthia_camera_path: /home/chrischoy/datasets/Synthia/%s/CameraParams/ synthia_camera_intrinsic_file: intrinsics.txt synthia_camera_extrinsics_file: Stereo_Right/Omni_F/%s.txt temporal_rand_dilation: false temporal_rand_numseq: false scannet_path: /home/chrischoy/datasets/scannet/scannet_preprocessed stanford3d_path: /checkpoint/jihou/data_efficient3d/data/stanford3d/pointcloud train: is_train: true stat_freq: 5 val_freq: 200 empty_cache_freq: 1 train_phase: train val_phase: val overwrite_weights: false resume: true resume_optimizer: True, eval_upsample: false lenient_weight_loading: true distributed: slurm augmentation: use_feat_aug: true data_aug_color_trans_ratio: 0.05 data_aug_color_jitter_std: 0.005 normalize_color: true data_aug_scale_min: 0.9 data_aug_scale_max: 1.1 data_aug_hue_max: 0.5 data_aug_saturation_max: 0.2 test: visualize: false save_features: false save_feat_dir: outputs/feat save_prediction: false save_pred_dir: outputs/pred save_groundtruth: false save_gt_dir: outputs/gt test_temporal_average: false visualize_path: outputs/visualize test_phase: test evaluate_original_pointcloud: false test_original_pointcloud: false test_stat_freq: 100 misc: is_cuda: true load_path: null log_step: 50 log_level: INFO num_gpus: 8 seed: 123 log_dir: /checkpoint/jihou/finetune5 load_bn: all_bn resume_config: null instance: true swav: false swav_sup: false detach: false debug: false train_stuff: true [2020-12-31 06:59:01,410][root][INFO] - ===> Initializing dataloader [2020-12-31 06:59:01,416][root][INFO] - voxel size: 0.05 [2020-12-31 06:59:01,416][root][INFO] - Loading StanfordArea5Dataset DatasetPhase.Train: ['area1.txt', 'area2.txt', 'area3.txt', 'area4.txt', 'area6.txt'] [2020-12-31 06:59:01,419][root][INFO] - voxel size: 0.05 [2020-12-31 06:59:01,419][root][INFO] - Loading StanfordArea5Dataset DatasetPhase.Val: area5.txt [2020-12-31 06:59:01,422][root][INFO] - voxel size: 0.05 [2020-12-31 06:59:01,422][root][INFO] - Loading StanfordArea5Dataset DatasetPhase.Test: area5.txt [2020-12-31 06:59:01,422][root][INFO] - ===> Building model [2020-12-31 06:59:01,762][root][INFO] - ===> Number of trainable parameters: Res16UNet34C: 37855760 [2020-12-31 06:59:01,762][root][INFO] - Res16UNet34C( (conv0p1s1): MinkowskiConvolution(in=3, out=32, region_type=RegionType.HYPERCUBE, kernel_size=[3, 3, 3], stride=[1, 1, 1], dilation=[1, 1, 1]) (bn0): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (conv1p1s2): MinkowskiConvolution(in=32, out=32, region_type=RegionType.HYPERCUBE, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bn1): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block1): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=32, out=32, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=32, out=32, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=32, out=32, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=32, out=32, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (conv2p2s2): MinkowskiConvolution(in=32, out=32, region_type=RegionType.HYPERCUBE, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bn2): MinkowskiBatchNorm(32, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block2): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=32, out=64, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=64, out=64, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=32, out=64, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=64, out=64, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=64, out=64, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (2): BasicBlock( (conv1): MinkowskiConvolution(in=64, out=64, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=64, out=64, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (conv3p4s2): MinkowskiConvolution(in=64, out=64, region_type=RegionType.HYPERCUBE, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bn3): MinkowskiBatchNorm(64, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block3): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=64, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=64, out=128, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (2): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (3): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (conv4p8s2): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYPERCUBE, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bn4): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block4): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=128, out=256, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (2): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (3): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (4): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) (5): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (convtr4p16s2): MinkowskiConvolutionTranspose(in=256, out=256, region_type=RegionType.HYPERCUBE, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bntr4): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block5): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=384, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=384, out=256, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=256, out=256, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (convtr5p8s2): MinkowskiConvolutionTranspose(in=256, out=128, region_type=RegionType.HYPERCUBE, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bntr5): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block6): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=192, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=192, out=128, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=128, out=128, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (convtr6p4s2): MinkowskiConvolutionTranspose(in=128, out=96, region_type=RegionType.HYPERCUBE, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bntr6): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block7): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=96, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=96, out=96, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=128, out=96, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=96, out=96, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=96, out=96, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (convtr7p2s2): MinkowskiConvolutionTranspose(in=96, out=96, region_type=RegionType.HYPERCUBE, kernel_size=[2, 2, 2], stride=[2, 2, 2], dilation=[1, 1, 1]) (bntr7): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (block8): Sequential( (0): BasicBlock( (conv1): MinkowskiConvolution(in=128, out=96, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=96, out=96, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() (downsample): Sequential( (0): MinkowskiConvolution(in=128, out=96, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (1): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): MinkowskiConvolution(in=96, out=96, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm1): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): MinkowskiConvolution(in=96, out=96, region_type=RegionType.HYBRID, kernel_volume=27, stride=[1, 1, 1], dilation=[1, 1, 1]) (norm2): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): MinkowskiReLU() ) ) (final): MinkowskiConvolution(in=96, out=13, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (relu): MinkowskiReLU() (offsets_pre): MinkowskiConvolution(in=96, out=96, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) (bntr_offset): MinkowskiBatchNorm(96, eps=1e-05, momentum=0.02, affine=True, track_running_stats=True) (offsets): MinkowskiConvolution(in=96, out=3, region_type=RegionType.HYPERCUBE, kernel_size=[1, 1, 1], stride=[1, 1, 1], dilation=[1, 1, 1]) ) [2020-12-31 06:59:01,767][root][INFO] - ===> Loading weights: /checkpoint/jihou/data_efficient3d/checkpoint/scannet/pretrain/20k/partition8/4096/2/weights/checkpoint.pth [2020-12-31 06:59:02,487][root][INFO] - Loading weights:conv0p1s1.kernel, bn0.bn.weight, bn0.bn.bias, bn0.bn.running_mean, bn0.bn.running_var, bn0.bn.num_batches_tracked, conv1p1s2.kernel, bn1.bn.weight, bn1.bn.bias, bn1.bn.running_mean, bn1.bn.running_var, bn1.bn.num_batches_tracked, block1.0.conv1.kernel, block1.0.norm1.bn.weight, block1.0.norm1.bn.bias, block1.0.norm1.bn.running_mean, block1.0.norm1.bn.running_var, block1.0.norm1.bn.num_batches_tracked, block1.0.conv2.kernel, block1.0.norm2.bn.weight, block1.0.norm2.bn.bias, block1.0.norm2.bn.running_mean, block1.0.norm2.bn.running_var, block1.0.norm2.bn.num_batches_tracked, block1.1.conv1.kernel, block1.1.norm1.bn.weight, block1.1.norm1.bn.bias, block1.1.norm1.bn.running_mean, block1.1.norm1.bn.running_var, block1.1.norm1.bn.num_batches_tracked, block1.1.conv2.kernel, block1.1.norm2.bn.weight, block1.1.norm2.bn.bias, block1.1.norm2.bn.running_mean, block1.1.norm2.bn.running_var, block1.1.norm2.bn.num_batches_tracked, conv2p2s2.kernel, bn2.bn.weight, bn2.bn.bias, bn2.bn.running_mean, bn2.bn.running_var, bn2.bn.num_batches_tracked, block2.0.conv1.kernel, block2.0.norm1.bn.weight, block2.0.norm1.bn.bias, block2.0.norm1.bn.running_mean, block2.0.norm1.bn.running_var, block2.0.norm1.bn.num_batches_tracked, block2.0.conv2.kernel, block2.0.norm2.bn.weight, block2.0.norm2.bn.bias, block2.0.norm2.bn.running_mean, block2.0.norm2.bn.running_var, block2.0.norm2.bn.num_batches_tracked, block2.0.downsample.0.kernel, block2.0.downsample.1.bn.weight, block2.0.downsample.1.bn.bias, block2.0.downsample.1.bn.running_mean, block2.0.downsample.1.bn.running_var, block2.0.downsample.1.bn.num_batches_tracked, block2.1.conv1.kernel, block2.1.norm1.bn.weight, block2.1.norm1.bn.bias, block2.1.norm1.bn.running_mean, block2.1.norm1.bn.running_var, block2.1.norm1.bn.num_batches_tracked, block2.1.conv2.kernel, block2.1.norm2.bn.weight, block2.1.norm2.bn.bias, block2.1.norm2.bn.running_mean, block2.1.norm2.bn.running_var, block2.1.norm2.bn.num_batches_tracked, block2.2.conv1.kernel, block2.2.norm1.bn.weight, block2.2.norm1.bn.bias, block2.2.norm1.bn.running_mean, block2.2.norm1.bn.running_var, block2.2.norm1.bn.num_batches_tracked, block2.2.conv2.kernel, block2.2.norm2.bn.weight, block2.2.norm2.bn.bias, block2.2.norm2.bn.running_mean, block2.2.norm2.bn.running_var, block2.2.norm2.bn.num_batches_tracked, conv3p4s2.kernel, bn3.bn.weight, bn3.bn.bias, bn3.bn.running_mean, bn3.bn.running_var, bn3.bn.num_batches_tracked, block3.0.conv1.kernel, block3.0.norm1.bn.weight, block3.0.norm1.bn.bias, block3.0.norm1.bn.running_mean, block3.0.norm1.bn.running_var, block3.0.norm1.bn.num_batches_tracked, block3.0.conv2.kernel, block3.0.norm2.bn.weight, block3.0.norm2.bn.bias, block3.0.norm2.bn.running_mean, block3.0.norm2.bn.running_var, block3.0.norm2.bn.num_batches_tracked, block3.0.downsample.0.kernel, block3.0.downsample.1.bn.weight, block3.0.downsample.1.bn.bias, block3.0.downsample.1.bn.running_mean, block3.0.downsample.1.bn.running_var, block3.0.downsample.1.bn.num_batches_tracked, block3.1.conv1.kernel, block3.1.norm1.bn.weight, block3.1.norm1.bn.bias, block3.1.norm1.bn.running_mean, block3.1.norm1.bn.running_var, block3.1.norm1.bn.num_batches_tracked, block3.1.conv2.kernel, block3.1.norm2.bn.weight, block3.1.norm2.bn.bias, block3.1.norm2.bn.running_mean, block3.1.norm2.bn.running_var, block3.1.norm2.bn.num_batches_tracked, block3.2.conv1.kernel, block3.2.norm1.bn.weight, block3.2.norm1.bn.bias, block3.2.norm1.bn.running_mean, block3.2.norm1.bn.running_var, block3.2.norm1.bn.num_batches_tracked, block3.2.conv2.kernel, block3.2.norm2.bn.weight, block3.2.norm2.bn.bias, block3.2.norm2.bn.running_mean, block3.2.norm2.bn.running_var, block3.2.norm2.bn.num_batches_tracked, block3.3.conv1.kernel, block3.3.norm1.bn.weight, block3.3.norm1.bn.bias, block3.3.norm1.bn.running_mean, block3.3.norm1.bn.running_var, block3.3.norm1.bn.num_batches_tracked, block3.3.conv2.kernel, block3.3.norm2.bn.weight, block3.3.norm2.bn.bias, block3.3.norm2.bn.running_mean, block3.3.norm2.bn.running_var, block3.3.norm2.bn.num_batches_tracked, conv4p8s2.kernel, bn4.bn.weight, bn4.bn.bias, bn4.bn.running_mean, bn4.bn.running_var, bn4.bn.num_batches_tracked, block4.0.conv1.kernel, block4.0.norm1.bn.weight, block4.0.norm1.bn.bias, block4.0.norm1.bn.running_mean, block4.0.norm1.bn.running_var, block4.0.norm1.bn.num_batches_tracked, block4.0.conv2.kernel, block4.0.norm2.bn.weight, block4.0.norm2.bn.bias, block4.0.norm2.bn.running_mean, block4.0.norm2.bn.running_var, block4.0.norm2.bn.num_batches_tracked, block4.0.downsample.0.kernel, block4.0.downsample.1.bn.weight, block4.0.downsample.1.bn.bias, block4.0.downsample.1.bn.running_mean, block4.0.downsample.1.bn.running_var, block4.0.downsample.1.bn.num_batches_tracked, block4.1.conv1.kernel, block4.1.norm1.bn.weight, block4.1.norm1.bn.bias, block4.1.norm1.bn.running_mean, block4.1.norm1.bn.running_var, block4.1.norm1.bn.num_batches_tracked, block4.1.conv2.kernel, block4.1.norm2.bn.weight, block4.1.norm2.bn.bias, block4.1.norm2.bn.running_mean, block4.1.norm2.bn.running_var, block4.1.norm2.bn.num_batches_tracked, block4.2.conv1.kernel, block4.2.norm1.bn.weight, block4.2.norm1.bn.bias, block4.2.norm1.bn.running_mean, block4.2.norm1.bn.running_var, block4.2.norm1.bn.num_batches_tracked, block4.2.conv2.kernel, block4.2.norm2.bn.weight, block4.2.norm2.bn.bias, block4.2.norm2.bn.running_mean, block4.2.norm2.bn.running_var, block4.2.norm2.bn.num_batches_tracked, block4.3.conv1.kernel, block4.3.norm1.bn.weight, block4.3.norm1.bn.bias, block4.3.norm1.bn.running_mean, block4.3.norm1.bn.running_var, block4.3.norm1.bn.num_batches_tracked, block4.3.conv2.kernel, block4.3.norm2.bn.weight, block4.3.norm2.bn.bias, block4.3.norm2.bn.running_mean, block4.3.norm2.bn.running_var, block4.3.norm2.bn.num_batches_tracked, block4.4.conv1.kernel, block4.4.norm1.bn.weight, block4.4.norm1.bn.bias, block4.4.norm1.bn.running_mean, block4.4.norm1.bn.running_var, block4.4.norm1.bn.num_batches_tracked, block4.4.conv2.kernel, block4.4.norm2.bn.weight, block4.4.norm2.bn.bias, block4.4.norm2.bn.running_mean, block4.4.norm2.bn.running_var, block4.4.norm2.bn.num_batches_tracked, block4.5.conv1.kernel, block4.5.norm1.bn.weight, block4.5.norm1.bn.bias, block4.5.norm1.bn.running_mean, block4.5.norm1.bn.running_var, block4.5.norm1.bn.num_batches_tracked, block4.5.conv2.kernel, block4.5.norm2.bn.weight, block4.5.norm2.bn.bias, block4.5.norm2.bn.running_mean, block4.5.norm2.bn.running_var, block4.5.norm2.bn.num_batches_tracked, convtr4p16s2.kernel, bntr4.bn.weight, bntr4.bn.bias, bntr4.bn.running_mean, bntr4.bn.running_var, bntr4.bn.num_batches_tracked, block5.0.conv1.kernel, block5.0.norm1.bn.weight, block5.0.norm1.bn.bias, block5.0.norm1.bn.running_mean, block5.0.norm1.bn.running_var, block5.0.norm1.bn.num_batches_tracked, block5.0.conv2.kernel, block5.0.norm2.bn.weight, block5.0.norm2.bn.bias, block5.0.norm2.bn.running_mean, block5.0.norm2.bn.running_var, block5.0.norm2.bn.num_batches_tracked, block5.0.downsample.0.kernel, block5.0.downsample.1.bn.weight, block5.0.downsample.1.bn.bias, block5.0.downsample.1.bn.running_mean, block5.0.downsample.1.bn.running_var, block5.0.downsample.1.bn.num_batches_tracked, block5.1.conv1.kernel, block5.1.norm1.bn.weight, block5.1.norm1.bn.bias, block5.1.norm1.bn.running_mean, block5.1.norm1.bn.running_var, block5.1.norm1.bn.num_batches_tracked, block5.1.conv2.kernel, block5.1.norm2.bn.weight, block5.1.norm2.bn.bias, block5.1.norm2.bn.running_mean, block5.1.norm2.bn.running_var, block5.1.norm2.bn.num_batches_tracked, convtr5p8s2.kernel, bntr5.bn.weight, bntr5.bn.bias, bntr5.bn.running_mean, bntr5.bn.running_var, bntr5.bn.num_batches_tracked, block6.0.conv1.kernel, block6.0.norm1.bn.weight, block6.0.norm1.bn.bias, block6.0.norm1.bn.running_mean, block6.0.norm1.bn.running_var, block6.0.norm1.bn.num_batches_tracked, block6.0.conv2.kernel, block6.0.norm2.bn.weight, block6.0.norm2.bn.bias, block6.0.norm2.bn.running_mean, block6.0.norm2.bn.running_var, block6.0.norm2.bn.num_batches_tracked, block6.0.downsample.0.kernel, block6.0.downsample.1.bn.weight, block6.0.downsample.1.bn.bias, block6.0.downsample.1.bn.running_mean, block6.0.downsample.1.bn.running_var, block6.0.downsample.1.bn.num_batches_tracked, block6.1.conv1.kernel, block6.1.norm1.bn.weight, block6.1.norm1.bn.bias, block6.1.norm1.bn.running_mean, block6.1.norm1.bn.running_var, block6.1.norm1.bn.num_batches_tracked, block6.1.conv2.kernel, block6.1.norm2.bn.weight, block6.1.norm2.bn.bias, block6.1.norm2.bn.running_mean, block6.1.norm2.bn.running_var, block6.1.norm2.bn.num_batches_tracked, convtr6p4s2.kernel, bntr6.bn.weight, bntr6.bn.bias, bntr6.bn.running_mean, bntr6.bn.running_var, bntr6.bn.num_batches_tracked, block7.0.conv1.kernel, block7.0.norm1.bn.weight, block7.0.norm1.bn.bias, block7.0.norm1.bn.running_mean, block7.0.norm1.bn.running_var, block7.0.norm1.bn.num_batches_tracked, block7.0.conv2.kernel, block7.0.norm2.bn.weight, block7.0.norm2.bn.bias, block7.0.norm2.bn.running_mean, block7.0.norm2.bn.running_var, block7.0.norm2.bn.num_batches_tracked, block7.0.downsample.0.kernel, block7.0.downsample.1.bn.weight, block7.0.downsample.1.bn.bias, block7.0.downsample.1.bn.running_mean, block7.0.downsample.1.bn.running_var, block7.0.downsample.1.bn.num_batches_tracked, block7.1.conv1.kernel, block7.1.norm1.bn.weight, block7.1.norm1.bn.bias, block7.1.norm1.bn.running_mean, block7.1.norm1.bn.running_var, block7.1.norm1.bn.num_batches_tracked, block7.1.conv2.kernel, block7.1.norm2.bn.weight, block7.1.norm2.bn.bias, block7.1.norm2.bn.running_mean, block7.1.norm2.bn.running_var, block7.1.norm2.bn.num_batches_tracked, convtr7p2s2.kernel, bntr7.bn.weight, bntr7.bn.bias, bntr7.bn.running_mean, bntr7.bn.running_var, bntr7.bn.num_batches_tracked, block8.0.conv1.kernel, block8.0.norm1.bn.weight, block8.0.norm1.bn.bias, block8.0.norm1.bn.running_mean, block8.0.norm1.bn.running_var, block8.0.norm1.bn.num_batches_tracked, block8.0.conv2.kernel, block8.0.norm2.bn.weight, block8.0.norm2.bn.bias, block8.0.norm2.bn.running_mean, block8.0.norm2.bn.running_var, block8.0.norm2.bn.num_batches_tracked, block8.0.downsample.0.kernel, block8.0.downsample.1.bn.weight, block8.0.downsample.1.bn.bias, block8.0.downsample.1.bn.running_mean, block8.0.downsample.1.bn.running_var, block8.0.downsample.1.bn.num_batches_tracked, block8.1.conv1.kernel, block8.1.norm1.bn.weight, block8.1.norm1.bn.bias, block8.1.norm1.bn.running_mean, block8.1.norm1.bn.running_var, block8.1.norm1.bn.num_batches_tracked, block8.1.conv2.kernel, block8.1.norm2.bn.weight, block8.1.norm2.bn.bias, block8.1.norm2.bn.running_mean, block8.1.norm2.bn.running_var, block8.1.norm2.bn.num_batches_tracked [2020-12-31 06:59:10,890][root][INFO] - => no weights.pth [2020-12-31 06:59:10,936][root][INFO] - ===> Start training on 8 GPUs, batch-size=48 [2020-12-31 06:59:22,933][root][INFO] - ===> Epoch[1](1/5): Loss 4.6275, Sem 4.6275, dir 0.0001, norm 2.1684 LR: 9.999e-02 Score 23.372 Data time: 4.2798, Forward time: 1.3578, Backward time: 6.1629, DDP time: 0.0060, Total iter time: 11.9241 [2020-12-31 06:59:37,462][root][INFO] - ===> Epoch[1](5/5): Loss 3.4905, Sem 3.4905, dir -0.1554, norm 1.8336 LR: 9.997e-02 Score 50.591 Data time: 0.1417, Forward time: 0.9776, Backward time: 3.1838, DDP time: 0.0069, Total iter time: 3.5370 [2020-12-31 06:59:51,191][root][INFO] - ===> Epoch[2](10/5): Loss 2.8711, Sem 2.8711, dir -0.2172, norm 1.7621 LR: 9.994e-02 Score 58.810 Data time: 0.2001, Forward time: 0.9412, Backward time: 2.3328, DDP time: 0.0065, Total iter time: 2.6470 [2020-12-31 07:00:04,935][root][INFO] - ===> Epoch[3](15/5): Loss 2.7336, Sem 2.7336, dir -0.2867, norm 1.8072 LR: 9.991e-02 Score 62.504 Data time: 0.1481, Forward time: 0.9331, Backward time: 2.0582, DDP time: 0.0062, Total iter time: 2.6499 [2020-12-31 07:00:18,188][root][INFO] - ===> Epoch[4](20/5): Loss 2.5273, Sem 2.5273, dir -0.3195, norm 1.8039 LR: 9.988e-02 Score 68.549 Data time: 0.1752, Forward time: 0.9505, Backward time: 1.8712, DDP time: 0.0059, Total iter time: 2.5488 [2020-12-31 07:00:30,697][root][INFO] - ===> Epoch[5](25/5): Loss 2.4766, Sem 2.4766, dir -0.3475, norm 1.8298 LR: 9.985e-02 Score 68.938 Data time: 0.4315, Forward time: 0.9064, Backward time: 1.7323, DDP time: 0.0058, Total iter time: 2.4227 [2020-12-31 07:00:45,390][root][INFO] - ===> Epoch[6](30/5): Loss 2.2099, Sem 2.2099, dir -0.4160, norm 1.6754 LR: 9.982e-02 Score 69.890 Data time: 1.1781, Forward time: 0.8839, Backward time: 1.5858, DDP time: 0.0057, Total iter time: 2.8504 [2020-12-31 07:01:01,205][root][INFO] - ===> Epoch[7](35/5): Loss 1.9719, Sem 1.9719, dir -0.4634, norm 1.5505 LR: 9.979e-02 Score 71.819 Data time: 0.8052, Forward time: 0.8940, Backward time: 1.5386, DDP time: 0.0057, Total iter time: 3.0722 [2020-12-31 07:01:13,577][root][INFO] - ===> Epoch[8](40/5): Loss 1.7106, Sem 1.7106, dir -0.5294, norm 1.3827 LR: 9.976e-02 Score 72.402 Data time: 0.1378, Forward time: 0.8978, Backward time: 1.5022, DDP time: 0.0057, Total iter time: 2.3859 [2020-12-31 07:01:25,137][root][INFO] - ===> Epoch[9](45/5): Loss 1.4749, Sem 1.4749, dir -0.5659, norm 1.2389 LR: 9.973e-02 Score 74.007 Data time: 0.1292, Forward time: 0.8921, Backward time: 1.4662, DDP time: 0.0058, Total iter time: 2.2265 [2020-12-31 07:01:38,135][root][INFO] - ===> Epoch[10](50/5): Loss 1.3935, Sem 1.3935, dir -0.5901, norm 1.2020 LR: 9.970e-02 Score 74.687 Data time: 0.1281, Forward time: 0.9057, Backward time: 1.4496, DDP time: 0.0058, Total iter time: 2.5112 [2020-12-31 07:01:49,423][root][INFO] - ===> Epoch[11](55/5): Loss 1.2608, Sem 1.2608, dir -0.6093, norm 1.1664 LR: 9.967e-02 Score 77.345 Data time: 0.1227, Forward time: 0.9056, Backward time: 1.4156, DDP time: 0.0058, Total iter time: 2.1579 [2020-12-31 07:02:02,515][root][INFO] - ===> Epoch[12](60/5): Loss 1.2181, Sem 1.2181, dir -0.6339, norm 1.1351 LR: 9.964e-02 Score 76.668 Data time: 0.1782, Forward time: 0.9110, Backward time: 1.4061, DDP time: 0.0058, Total iter time: 2.5201 [2020-12-31 07:02:16,034][root][INFO] - ===> Epoch[13](65/5): Loss 1.1018, Sem 1.1018, dir -0.6454, norm 1.0599 LR: 9.961e-02 Score 77.701 Data time: 0.0706, Forward time: 0.9012, Backward time: 1.4276, DDP time: 0.0058, Total iter time: 2.6280 [2020-12-31 07:02:28,553][root][INFO] - ===> Epoch[14](70/5): Loss 1.0133, Sem 1.0133, dir -0.6611, norm 1.0268 LR: 9.958e-02 Score 79.039 Data time: 0.0664, Forward time: 0.8895, Backward time: 1.4334, DDP time: 0.0058, Total iter time: 2.4239 [2020-12-31 07:02:40,625][root][INFO] - ===> Epoch[15](75/5): Loss 0.9854, Sem 0.9854, dir -0.6760, norm 1.0206 LR: 9.955e-02 Score 79.204 Data time: 0.1519, Forward time: 0.8960, Backward time: 1.4125, DDP time: 0.0057, Total iter time: 2.3186 [2020-12-31 07:02:52,705][root][INFO] - ===> Epoch[16](80/5): Loss 0.9320, Sem 0.9320, dir -0.6912, norm 0.9903 LR: 9.952e-02 Score 79.396 Data time: 0.1706, Forward time: 0.9043, Backward time: 1.3919, DDP time: 0.0057, Total iter time: 2.3212 [2020-12-31 07:03:04,936][root][INFO] - ===> Epoch[17](85/5): Loss 0.8581, Sem 0.8581, dir -0.7035, norm 0.9616 LR: 9.949e-02 Score 80.556 Data time: 0.1699, Forward time: 0.9059, Backward time: 1.3800, DDP time: 0.0057, Total iter time: 2.3562 [2020-12-31 07:03:16,519][root][INFO] - ===> Epoch[18](90/5): Loss 0.8363, Sem 0.8363, dir -0.7169, norm 0.9522 LR: 9.946e-02 Score 80.387 Data time: 0.0879, Forward time: 0.9081, Backward time: 1.3658, DDP time: 0.0057, Total iter time: 2.2281 [2020-12-31 07:03:30,085][root][INFO] - ===> Epoch[19](95/5): Loss 0.7782, Sem 0.7782, dir -0.7329, norm 0.9375 LR: 9.943e-02 Score 81.390 Data time: 0.1030, Forward time: 0.9051, Backward time: 1.3770, DDP time: 0.0057, Total iter time: 2.6228 [2020-12-31 07:03:42,120][root][INFO] - ===> Epoch[20](100/5): Loss 0.7494, Sem 0.7494, dir -0.7432, norm 0.9218 LR: 9.940e-02 Score 81.331 Data time: 0.0808, Forward time: 0.9042, Backward time: 1.3724, DDP time: 0.0057, Total iter time: 2.3154 [2020-12-31 07:03:54,748][root][INFO] - ===> Epoch[21](105/5): Loss 0.7406, Sem 0.7406, dir -0.7444, norm 0.9114 LR: 9.937e-02 Score 81.349 Data time: 0.0957, Forward time: 0.9060, Backward time: 1.3701, DDP time: 0.0057, Total iter time: 2.4367 [2020-12-31 07:04:06,768][root][INFO] - ===> Epoch[22](110/5): Loss 0.6837, Sem 0.6837, dir -0.7542, norm 0.8684 LR: 9.934e-02 Score 81.336 Data time: 0.1850, Forward time: 0.9098, Backward time: 1.3572, DDP time: 0.0057, Total iter time: 2.3144 [2020-12-31 07:04:19,258][root][INFO] - ===> Epoch[23](115/5): Loss 0.6580, Sem 0.6580, dir -0.7593, norm 0.8664 LR: 9.931e-02 Score 81.851 Data time: 0.0969, Forward time: 0.9062, Backward time: 1.3598, DDP time: 0.0056, Total iter time: 2.4153 [2020-12-31 07:04:31,523][root][INFO] - ===> Epoch[24](120/5): Loss 0.6089, Sem 0.6089, dir -0.7658, norm 0.8449 LR: 9.928e-02 Score 82.604 Data time: 0.0860, Forward time: 0.9048, Backward time: 1.3584, DDP time: 0.0056, Total iter time: 2.3672 [2020-12-31 07:04:43,519][root][INFO] - ===> Epoch[25](125/5): Loss 0.6117, Sem 0.6117, dir -0.7702, norm 0.8527 LR: 9.925e-02 Score 82.627 Data time: 0.0859, Forward time: 0.9029, Backward time: 1.3562, DDP time: 0.0056, Total iter time: 2.3204 [2020-12-31 07:04:55,624][root][INFO] - ===> Epoch[26](130/5): Loss 0.5574, Sem 0.5574, dir -0.7771, norm 0.8270 LR: 9.922e-02 Score 83.211 Data time: 0.1730, Forward time: 0.9027, Backward time: 1.3494, DDP time: 0.0056, Total iter time: 2.3227 [2020-12-31 07:05:08,244][root][INFO] - ===> Epoch[27](135/5): Loss 0.5552, Sem 0.5552, dir -0.7772, norm 0.8403 LR: 9.919e-02 Score 83.624 Data time: 0.2689, Forward time: 0.9093, Backward time: 1.3375, DDP time: 0.0055, Total iter time: 2.4249 [2020-12-31 07:05:21,174][root][INFO] - ===> Epoch[28](140/5): Loss 0.5858, Sem 0.5858, dir -0.7802, norm 0.8645 LR: 9.916e-02 Score 83.506 Data time: 0.2034, Forward time: 0.9165, Backward time: 1.3296, DDP time: 0.0055, Total iter time: 2.4834 [2020-12-31 07:05:32,627][root][INFO] - ===> Epoch[29](145/5): Loss 0.4615, Sem 0.4615, dir -0.7881, norm 0.7749 LR: 9.913e-02 Score 84.252 Data time: 0.0871, Forward time: 0.9162, Backward time: 1.3237, DDP time: 0.0055, Total iter time: 2.2210 [2020-12-31 07:05:44,867][root][INFO] - ===> Epoch[30](150/5): Loss 0.4940, Sem 0.4940, dir -0.7911, norm 0.8117 LR: 9.910e-02 Score 84.283 Data time: 0.1089, Forward time: 0.9213, Backward time: 1.3176, DDP time: 0.0054, Total iter time: 2.3632 [2020-12-31 07:05:56,950][root][INFO] - ===> Epoch[31](155/5): Loss 0.4828, Sem 0.4828, dir -0.7912, norm 0.8013 LR: 9.907e-02 Score 84.203 Data time: 0.1003, Forward time: 0.9238, Backward time: 1.3131, DDP time: 0.0054, Total iter time: 2.3291 [2020-12-31 07:06:09,858][root][INFO] - ===> Epoch[32](160/5): Loss 0.4083, Sem 0.4083, dir -0.7975, norm 0.7639 LR: 9.904e-02 Score 85.374 Data time: 0.2055, Forward time: 0.9314, Backward time: 1.3055, DDP time: 0.0054, Total iter time: 2.4809 [2020-12-31 07:06:21,256][root][INFO] - ===> Epoch[33](165/5): Loss 0.4059, Sem 0.4059, dir -0.8009, norm 0.7735 LR: 9.901e-02 Score 85.635 Data time: 0.0712, Forward time: 0.9290, Backward time: 1.3028, DDP time: 0.0054, Total iter time: 2.2048 [2020-12-31 07:06:32,846][root][INFO] - ===> Epoch[34](170/5): Loss 0.4215, Sem 0.4215, dir -0.7978, norm 0.7815 LR: 9.898e-02 Score 85.423 Data time: 0.1018, Forward time: 0.9291, Backward time: 1.2976, DDP time: 0.0054, Total iter time: 2.2151 [2020-12-31 07:06:45,946][root][INFO] - ===> Epoch[35](175/5): Loss 0.4105, Sem 0.4105, dir -0.7964, norm 0.7809 LR: 9.895e-02 Score 85.808 Data time: 0.1980, Forward time: 0.9336, Backward time: 1.2947, DDP time: 0.0053, Total iter time: 2.5233 [2020-12-31 07:06:58,597][root][INFO] - ===> Epoch[36](180/5): Loss 0.4359, Sem 0.4359, dir -0.7997, norm 0.8064 LR: 9.892e-02 Score 85.636 Data time: 0.1418, Forward time: 0.9346, Backward time: 1.2937, DDP time: 0.0053, Total iter time: 2.4387 [2020-12-31 07:07:10,813][root][INFO] - ===> Epoch[37](185/5): Loss 0.4152, Sem 0.4152, dir -0.8058, norm 0.7843 LR: 9.889e-02 Score 85.555 Data time: 0.1116, Forward time: 0.9347, Backward time: 1.2917, DDP time: 0.0053, Total iter time: 2.3434 [2020-12-31 07:07:23,148][root][INFO] - ===> Epoch[38](190/5): Loss 0.3794, Sem 0.3794, dir -0.8036, norm 0.7522 LR: 9.886e-02 Score 85.542 Data time: 0.1336, Forward time: 0.9357, Backward time: 1.2896, DDP time: 0.0053, Total iter time: 2.3758 [2020-12-31 07:07:35,497][root][INFO] - ===> Epoch[39](195/5): Loss 0.3802, Sem 0.3802, dir -0.8070, norm 0.7736 LR: 9.883e-02 Score 86.273 Data time: 0.1007, Forward time: 0.9331, Backward time: 1.2913, DDP time: 0.0053, Total iter time: 2.3763 [2020-12-31 07:07:47,663][root][INFO] - ===> Epoch[40](200/5): Loss 0.3511, Sem 0.3511, dir -0.8099, norm 0.7641 LR: 9.880e-02 Score 86.652 Data time: 0.0994, Forward time: 0.9329, Backward time: 1.2902, DDP time: 0.0053, Total iter time: 2.3362 [2020-12-31 07:07:47,665][root][INFO] - ===> Start testing [2020-12-31 07:08:31,776][root][INFO] - 68/68: Data time: 0.0024, Iter time: 0.1248 Loss 0.587 (AVG: 0.567) Score 79.798 (AVG: 82.961) mIOU 49.794 mAP 72.223 mAcc 58.503 IOU: 47.166 0.000 41.615 60.306 93.216 78.525 0.270 26.805 97.870 29.007 70.893 74.871 26.779 mAP: 72.765 3.208 78.057 84.069 99.082 95.479 19.418 57.483 99.610 80.505 90.476 87.875 70.874 mAcc: 67.923 0.000 63.256 66.865 96.448 93.599 0.273 33.867 99.437 32.350 82.683 94.466 29.366 [2020-12-31 07:08:31,777][root][INFO] - Finished test. Elapsed time: 44.1098 [2020-12-31 07:08:34,961][root][INFO] - [2020-12-31 07:08:34,961][root][INFO] - ################################################################ [2020-12-31 07:08:34,961][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 07:08:34,961][root][INFO] - ################################################################ [2020-12-31 07:08:34,961][root][INFO] - clutter : nan nan nan [2020-12-31 07:08:34,961][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 07:08:34,961][root][INFO] - board : 0.211 0.494 0.787 [2020-12-31 07:08:34,961][root][INFO] - bookcase : 0.146 0.288 0.488 [2020-12-31 07:08:34,961][root][INFO] - ceiling : 0.803 0.872 0.896 [2020-12-31 07:08:34,962][root][INFO] - chair : 0.587 0.752 0.817 [2020-12-31 07:08:34,962][root][INFO] - column : 0.000 0.000 0.000 [2020-12-31 07:08:34,962][root][INFO] - door : 0.000 0.000 0.199 [2020-12-31 07:08:34,962][root][INFO] - floor : 0.986 0.996 0.996 [2020-12-31 07:08:34,962][root][INFO] - sofa : 0.001 0.011 0.184 [2020-12-31 07:08:34,962][root][INFO] - table : 0.119 0.280 0.626 [2020-12-31 07:08:34,962][root][INFO] - wall : 0.286 0.504 0.644 [2020-12-31 07:08:34,962][root][INFO] - window : 0.040 0.096 0.308 [2020-12-31 07:08:34,962][root][INFO] - ---------------------------------------------------------------- [2020-12-31 07:08:34,962][root][INFO] - average : 0.265 0.358 0.496 [2020-12-31 07:08:34,962][root][INFO] - [2020-12-31 07:08:35,140][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 07:08:35,891][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2020-12-31 07:08:35,892][root][INFO] - Current best mIoU: 49.794 at iter 1 [2020-12-31 07:08:36,472][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_mAP.pth [2020-12-31 07:08:36,473][root][INFO] - Current best mAP@0.5: 0.358 at iter 200 [2020-12-31 07:08:37,016][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter200.pth [2020-12-31 07:08:48,656][root][INFO] - ===> Epoch[41](205/5): Loss 0.3310, Sem 0.3310, dir -0.8149, norm 0.7459 LR: 9.877e-02 Score 86.676 Data time: 0.1119, Forward time: 0.9312, Backward time: 1.2873, DDP time: 0.0053, Total iter time: 2.2198 [2020-12-31 07:09:01,150][root][INFO] - ===> Epoch[42](210/5): Loss 0.3064, Sem 0.3064, dir -0.8102, norm 0.7449 LR: 9.874e-02 Score 87.456 Data time: 0.1792, Forward time: 0.9359, Backward time: 1.2818, DDP time: 0.0053, Total iter time: 2.4065 [2020-12-31 07:09:13,620][root][INFO] - ===> Epoch[43](215/5): Loss 0.2617, Sem 0.2617, dir -0.8202, norm 0.7262 LR: 9.871e-02 Score 88.118 Data time: 0.0928, Forward time: 0.9358, Backward time: 1.2825, DDP time: 0.0053, Total iter time: 2.4231 [2020-12-31 07:09:26,288][root][INFO] - ===> Epoch[44](220/5): Loss 0.2646, Sem 0.2646, dir -0.8191, norm 0.7235 LR: 9.868e-02 Score 87.947 Data time: 0.1473, Forward time: 0.9383, Backward time: 1.2803, DDP time: 0.0053, Total iter time: 2.4323 [2020-12-31 07:09:39,142][root][INFO] - ===> Epoch[45](225/5): Loss 0.2901, Sem 0.2901, dir -0.8188, norm 0.7333 LR: 9.865e-02 Score 87.413 Data time: 0.1021, Forward time: 0.9369, Backward time: 1.2831, DDP time: 0.0053, Total iter time: 2.4834 [2020-12-31 07:09:50,757][root][INFO] - ===> Epoch[46](230/5): Loss 0.2751, Sem 0.2751, dir -0.8167, norm 0.7360 LR: 9.862e-02 Score 88.065 Data time: 0.1004, Forward time: 0.9369, Backward time: 1.2802, DDP time: 0.0053, Total iter time: 2.2357 [2020-12-31 07:10:02,721][root][INFO] - ===> Epoch[47](235/5): Loss 0.2735, Sem 0.2735, dir -0.8193, norm 0.7442 LR: 9.859e-02 Score 88.380 Data time: 0.1115, Forward time: 0.9367, Backward time: 1.2787, DDP time: 0.0053, Total iter time: 2.3049 [2020-12-31 07:10:14,428][root][INFO] - ===> Epoch[48](240/5): Loss 0.2591, Sem 0.2591, dir -0.8185, norm 0.7227 LR: 9.856e-02 Score 88.171 Data time: 0.1438, Forward time: 0.9359, Backward time: 1.2758, DDP time: 0.0053, Total iter time: 2.2461 [2020-12-31 07:10:27,885][root][INFO] - ===> Epoch[49](245/5): Loss 0.2682, Sem 0.2682, dir -0.8223, norm 0.7270 LR: 9.853e-02 Score 87.917 Data time: 0.1159, Forward time: 0.9376, Backward time: 1.2783, DDP time: 0.0053, Total iter time: 2.5961 [2020-12-31 07:10:40,347][root][INFO] - ===> Epoch[50](250/5): Loss 0.2783, Sem 0.2783, dir -0.8213, norm 0.7403 LR: 9.850e-02 Score 88.000 Data time: 0.1497, Forward time: 0.9403, Backward time: 1.2752, DDP time: 0.0052, Total iter time: 2.3917 [2020-12-31 07:10:52,332][root][INFO] - ===> Epoch[51](255/5): Loss 0.2406, Sem 0.2406, dir -0.8278, norm 0.7194 LR: 9.847e-02 Score 88.327 Data time: 0.1503, Forward time: 0.9430, Backward time: 1.2705, DDP time: 0.0052, Total iter time: 2.2975 [2020-12-31 07:11:04,278][root][INFO] - ===> Epoch[52](260/5): Loss 0.2661, Sem 0.2661, dir -0.8240, norm 0.7193 LR: 9.844e-02 Score 87.487 Data time: 0.1135, Forward time: 0.9426, Backward time: 1.2692, DDP time: 0.0052, Total iter time: 2.3088 [2020-12-31 07:11:16,492][root][INFO] - ===> Epoch[53](265/5): Loss 0.1998, Sem 0.1998, dir -0.8315, norm 0.6930 LR: 9.841e-02 Score 88.663 Data time: 0.0960, Forward time: 0.9415, Backward time: 1.2695, DDP time: 0.0052, Total iter time: 2.3501 [2020-12-31 07:11:28,036][root][INFO] - ===> Epoch[54](270/5): Loss 0.2329, Sem 0.2329, dir -0.8263, norm 0.7149 LR: 9.838e-02 Score 88.404 Data time: 0.1807, Forward time: 0.9420, Backward time: 1.2649, DDP time: 0.0052, Total iter time: 2.2195 [2020-12-31 07:11:40,528][root][INFO] - ===> Epoch[55](275/5): Loss 0.2255, Sem 0.2255, dir -0.8248, norm 0.7052 LR: 9.835e-02 Score 88.486 Data time: 0.1197, Forward time: 0.9416, Backward time: 1.2654, DDP time: 0.0052, Total iter time: 2.4115 [2020-12-31 07:11:52,820][root][INFO] - ===> Epoch[56](280/5): Loss 0.1908, Sem 0.1908, dir -0.8303, norm 0.6964 LR: 9.832e-02 Score 89.039 Data time: 0.0932, Forward time: 0.9410, Backward time: 1.2658, DDP time: 0.0052, Total iter time: 2.3599 [2020-12-31 07:12:05,050][root][INFO] - ===> Epoch[57](285/5): Loss 0.1781, Sem 0.1781, dir -0.8330, norm 0.6806 LR: 9.829e-02 Score 88.943 Data time: 0.0810, Forward time: 0.9397, Backward time: 1.2668, DDP time: 0.0052, Total iter time: 2.3634 [2020-12-31 07:12:17,134][root][INFO] - ===> Epoch[58](290/5): Loss 0.1579, Sem 0.1579, dir -0.8362, norm 0.6824 LR: 9.826e-02 Score 89.456 Data time: 0.1275, Forward time: 0.9400, Backward time: 1.2651, DDP time: 0.0052, Total iter time: 2.3109 [2020-12-31 07:12:29,555][root][INFO] - ===> Epoch[59](295/5): Loss 0.1800, Sem 0.1800, dir -0.8318, norm 0.7006 LR: 9.823e-02 Score 89.564 Data time: 0.1559, Forward time: 0.9407, Backward time: 1.2638, DDP time: 0.0053, Total iter time: 2.3881 [2020-12-31 07:12:42,630][root][INFO] - ===> Epoch[60](300/5): Loss 0.1627, Sem 0.1627, dir -0.8334, norm 0.6896 LR: 9.820e-02 Score 89.758 Data time: 0.1097, Forward time: 0.9407, Backward time: 1.2662, DDP time: 0.0053, Total iter time: 2.5340 [2020-12-31 07:12:54,460][root][INFO] - ===> Epoch[61](305/5): Loss 0.1637, Sem 0.1637, dir -0.8369, norm 0.6767 LR: 9.817e-02 Score 89.125 Data time: 0.0916, Forward time: 0.9381, Backward time: 1.2671, DDP time: 0.0053, Total iter time: 2.2846 [2020-12-31 07:13:07,137][root][INFO] - ===> Epoch[62](310/5): Loss 0.1787, Sem 0.1787, dir -0.8369, norm 0.6952 LR: 9.814e-02 Score 89.298 Data time: 0.0974, Forward time: 0.9377, Backward time: 1.2687, DDP time: 0.0053, Total iter time: 2.4587 [2020-12-31 07:13:19,076][root][INFO] - ===> Epoch[63](315/5): Loss 0.1533, Sem 0.1533, dir -0.8370, norm 0.6973 LR: 9.811e-02 Score 90.228 Data time: 0.1178, Forward time: 0.9381, Backward time: 1.2669, DDP time: 0.0053, Total iter time: 2.2958 [2020-12-31 07:13:31,481][root][INFO] - ===> Epoch[64](320/5): Loss 0.1595, Sem 0.1595, dir -0.8362, norm 0.6868 LR: 9.808e-02 Score 89.617 Data time: 0.1070, Forward time: 0.9374, Backward time: 1.2675, DDP time: 0.0053, Total iter time: 2.3865 [2020-12-31 07:13:43,234][root][INFO] - ===> Epoch[65](325/5): Loss 0.1434, Sem 0.1434, dir -0.8376, norm 0.6972 LR: 9.805e-02 Score 90.549 Data time: 0.0986, Forward time: 0.9368, Backward time: 1.2666, DDP time: 0.0053, Total iter time: 2.2573 [2020-12-31 07:13:55,060][root][INFO] - ===> Epoch[66](330/5): Loss 0.1355, Sem 0.1355, dir -0.8402, norm 0.6834 LR: 9.802e-02 Score 90.072 Data time: 0.1245, Forward time: 0.9365, Backward time: 1.2654, DDP time: 0.0053, Total iter time: 2.2968 [2020-12-31 07:14:07,157][root][INFO] - ===> Epoch[67](335/5): Loss 0.1222, Sem 0.1222, dir -0.8449, norm 0.6748 LR: 9.799e-02 Score 90.192 Data time: 0.0984, Forward time: 0.9355, Backward time: 1.2656, DDP time: 0.0053, Total iter time: 2.3311 [2020-12-31 07:14:18,662][root][INFO] - ===> Epoch[68](340/5): Loss 0.0984, Sem 0.0984, dir -0.8445, norm 0.6518 LR: 9.796e-02 Score 90.263 Data time: 0.0953, Forward time: 0.9345, Backward time: 1.2641, DDP time: 0.0053, Total iter time: 2.2043 [2020-12-31 07:14:31,032][root][INFO] - ===> Epoch[69](345/5): Loss 0.1083, Sem 0.1083, dir -0.8457, norm 0.6676 LR: 9.793e-02 Score 90.447 Data time: 0.1204, Forward time: 0.9348, Backward time: 1.2636, DDP time: 0.0053, Total iter time: 2.3808 [2020-12-31 07:14:43,301][root][INFO] - ===> Epoch[70](350/5): Loss 0.1186, Sem 0.1186, dir -0.8455, norm 0.6693 LR: 9.790e-02 Score 90.050 Data time: 0.1101, Forward time: 0.9350, Backward time: 1.2632, DDP time: 0.0053, Total iter time: 2.3601 [2020-12-31 07:14:55,137][root][INFO] - ===> Epoch[71](355/5): Loss 0.0988, Sem 0.0988, dir -0.8457, norm 0.6548 LR: 9.787e-02 Score 90.225 Data time: 0.0775, Forward time: 0.9328, Backward time: 1.2644, DDP time: 0.0053, Total iter time: 2.2942 [2020-12-31 07:15:06,904][root][INFO] - ===> Epoch[72](360/5): Loss 0.0819, Sem 0.0819, dir -0.8477, norm 0.6534 LR: 9.784e-02 Score 90.602 Data time: 0.1151, Forward time: 0.9328, Backward time: 1.2630, DDP time: 0.0053, Total iter time: 2.2577 [2020-12-31 07:15:19,837][root][INFO] - ===> Epoch[73](365/5): Loss 0.1146, Sem 0.1146, dir -0.8493, norm 0.6711 LR: 9.781e-02 Score 89.980 Data time: 0.0694, Forward time: 0.9310, Backward time: 1.2668, DDP time: 0.0053, Total iter time: 2.5050 [2020-12-31 07:15:31,049][root][INFO] - ===> Epoch[74](370/5): Loss 0.1136, Sem 0.1136, dir -0.8443, norm 0.6774 LR: 9.778e-02 Score 90.503 Data time: 0.1347, Forward time: 0.9301, Backward time: 1.2643, DDP time: 0.0053, Total iter time: 2.1629 [2020-12-31 07:15:42,314][root][INFO] - ===> Epoch[75](375/5): Loss 0.0827, Sem 0.0827, dir -0.8454, norm 0.6485 LR: 9.775e-02 Score 90.492 Data time: 0.1291, Forward time: 0.9295, Backward time: 1.2619, DDP time: 0.0053, Total iter time: 2.1607 [2020-12-31 07:15:55,148][root][INFO] - ===> Epoch[76](380/5): Loss 0.0775, Sem 0.0775, dir -0.8493, norm 0.6503 LR: 9.772e-02 Score 90.560 Data time: 0.1068, Forward time: 0.9298, Backward time: 1.2631, DDP time: 0.0053, Total iter time: 2.4841 [2020-12-31 07:16:07,449][root][INFO] - ===> Epoch[77](385/5): Loss 0.0550, Sem 0.0550, dir -0.8507, norm 0.6429 LR: 9.769e-02 Score 91.115 Data time: 0.1107, Forward time: 0.9304, Backward time: 1.2625, DDP time: 0.0053, Total iter time: 2.3775 [2020-12-31 07:16:19,999][root][INFO] - ===> Epoch[78](390/5): Loss 0.0418, Sem 0.0418, dir -0.8525, norm 0.6301 LR: 9.766e-02 Score 91.164 Data time: 0.1852, Forward time: 0.9319, Backward time: 1.2610, DDP time: 0.0053, Total iter time: 2.4159 [2020-12-31 07:16:32,538][root][INFO] - ===> Epoch[79](395/5): Loss 0.0710, Sem 0.0710, dir -0.8541, norm 0.6493 LR: 9.763e-02 Score 90.638 Data time: 0.1228, Forward time: 0.9321, Backward time: 1.2614, DDP time: 0.0053, Total iter time: 2.4260 [2020-12-31 07:16:44,568][root][INFO] - ===> Epoch[80](400/5): Loss 0.0525, Sem 0.0525, dir -0.8511, norm 0.6468 LR: 9.760e-02 Score 91.367 Data time: 0.1046, Forward time: 0.9329, Backward time: 1.2602, DDP time: 0.0053, Total iter time: 2.3199 [2020-12-31 07:16:44,571][root][INFO] - ===> Start testing [2020-12-31 07:17:28,734][root][INFO] - 68/68: Data time: 0.2687, Iter time: 0.1872 Loss 0.860 (AVG: 0.486) Score 73.625 (AVG: 85.940) mIOU 56.724 mAP 79.212 mAcc 63.985 IOU: 53.614 0.000 63.711 67.771 93.961 82.887 4.158 39.414 98.421 41.275 77.085 77.217 37.899 mAP: 77.683 12.422 88.589 87.808 99.247 97.498 34.238 75.105 99.757 88.596 92.567 92.210 84.040 mAcc: 69.097 0.000 75.808 77.515 95.822 94.454 4.220 48.941 99.559 45.363 83.654 96.452 40.926 [2020-12-31 07:17:28,735][root][INFO] - Finished test. Elapsed time: 44.1617 [2020-12-31 07:17:31,754][root][INFO] - [2020-12-31 07:17:31,754][root][INFO] - ################################################################ [2020-12-31 07:17:31,754][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 07:17:31,754][root][INFO] - ################################################################ [2020-12-31 07:17:31,755][root][INFO] - clutter : nan nan nan [2020-12-31 07:17:31,755][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 07:17:31,755][root][INFO] - board : 0.505 0.629 0.826 [2020-12-31 07:17:31,755][root][INFO] - bookcase : 0.252 0.456 0.569 [2020-12-31 07:17:31,755][root][INFO] - ceiling : 0.740 0.765 0.819 [2020-12-31 07:17:31,755][root][INFO] - chair : 0.646 0.793 0.839 [2020-12-31 07:17:31,755][root][INFO] - column : 0.002 0.019 0.037 [2020-12-31 07:17:31,755][root][INFO] - door : 0.014 0.034 0.394 [2020-12-31 07:17:31,755][root][INFO] - floor : 0.988 0.992 0.992 [2020-12-31 07:17:31,755][root][INFO] - sofa : 0.010 0.039 0.382 [2020-12-31 07:17:31,755][root][INFO] - table : 0.139 0.310 0.655 [2020-12-31 07:17:31,755][root][INFO] - wall : 0.357 0.570 0.713 [2020-12-31 07:17:31,755][root][INFO] - window : 0.109 0.356 0.590 [2020-12-31 07:17:31,755][root][INFO] - ---------------------------------------------------------------- [2020-12-31 07:17:31,755][root][INFO] - average : 0.313 0.414 0.568 [2020-12-31 07:17:31,755][root][INFO] - [2020-12-31 07:17:31,944][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 07:17:32,571][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2020-12-31 07:17:32,572][root][INFO] - Current best mIoU: 56.724 at iter 1 [2020-12-31 07:17:33,133][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_mAP.pth [2020-12-31 07:17:33,134][root][INFO] - Current best mAP@0.5: 0.414 at iter 400 [2020-12-31 07:17:33,682][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter400.pth [2020-12-31 07:17:45,091][root][INFO] - ===> Epoch[81](405/5): Loss 0.0521, Sem 0.0521, dir -0.8566, norm 0.6495 LR: 9.757e-02 Score 91.189 Data time: 0.0911, Forward time: 0.9317, Backward time: 1.2591, DDP time: 0.0053, Total iter time: 2.1781 [2020-12-31 07:17:57,135][root][INFO] - ===> Epoch[82](410/5): Loss 0.0418, Sem 0.0418, dir -0.8548, norm 0.6379 LR: 9.754e-02 Score 91.269 Data time: 0.1649, Forward time: 0.9332, Backward time: 1.2567, DDP time: 0.0053, Total iter time: 2.3180 [2020-12-31 07:18:09,084][root][INFO] - ===> Epoch[83](415/5): Loss 0.0483, Sem 0.0483, dir -0.8545, norm 0.6421 LR: 9.751e-02 Score 91.168 Data time: 0.0922, Forward time: 0.9337, Backward time: 1.2558, DDP time: 0.0053, Total iter time: 2.3071 [2020-12-31 07:18:20,968][root][INFO] - ===> Epoch[84](420/5): Loss 0.0550, Sem 0.0550, dir -0.8564, norm 0.6390 LR: 9.748e-02 Score 90.883 Data time: 0.0900, Forward time: 0.9343, Backward time: 1.2546, DDP time: 0.0053, Total iter time: 2.2802 [2020-12-31 07:18:33,801][root][INFO] - ===> Epoch[85](425/5): Loss 0.0251, Sem 0.0251, dir -0.8567, norm 0.6237 LR: 9.745e-02 Score 91.313 Data time: 0.1797, Forward time: 0.9334, Backward time: 1.2560, DDP time: 0.0053, Total iter time: 2.4951 [2020-12-31 07:18:46,117][root][INFO] - ===> Epoch[86](430/5): Loss 0.0172, Sem 0.0172, dir -0.8586, norm 0.6298 LR: 9.742e-02 Score 91.741 Data time: 0.0883, Forward time: 0.9331, Backward time: 1.2566, DDP time: 0.0053, Total iter time: 2.3857 [2020-12-31 07:18:58,375][root][INFO] - ===> Epoch[87](435/5): Loss 0.0260, Sem 0.0260, dir -0.8549, norm 0.6289 LR: 9.739e-02 Score 91.358 Data time: 0.1078, Forward time: 0.9333, Backward time: 1.2564, DDP time: 0.0053, Total iter time: 2.3582 [2020-12-31 07:19:11,022][root][INFO] - ===> Epoch[88](440/5): Loss -0.0129, Sem -0.0129, dir -0.8604, norm 0.6094 LR: 9.736e-02 Score 91.967 Data time: 0.0998, Forward time: 0.9333, Backward time: 1.2574, DDP time: 0.0053, Total iter time: 2.4417 [2020-12-31 07:19:22,674][root][INFO] - ===> Epoch[89](445/5): Loss -0.0170, Sem -0.0170, dir -0.8634, norm 0.6063 LR: 9.733e-02 Score 91.959 Data time: 0.1018, Forward time: 0.9329, Backward time: 1.2563, DDP time: 0.0053, Total iter time: 2.2363 [2020-12-31 07:19:34,929][root][INFO] - ===> Epoch[90](450/5): Loss 0.0238, Sem 0.0238, dir -0.8574, norm 0.6338 LR: 9.730e-02 Score 91.604 Data time: 0.0870, Forward time: 0.9326, Backward time: 1.2568, DDP time: 0.0053, Total iter time: 2.3578 [2020-12-31 07:19:47,387][root][INFO] - ===> Epoch[91](455/5): Loss -0.0211, Sem -0.0211, dir -0.8640, norm 0.6049 LR: 9.727e-02 Score 91.998 Data time: 0.1591, Forward time: 0.9332, Backward time: 1.2560, DDP time: 0.0053, Total iter time: 2.3956 [2020-12-31 07:19:59,955][root][INFO] - ===> Epoch[92](460/5): Loss -0.0112, Sem -0.0112, dir -0.8652, norm 0.6118 LR: 9.724e-02 Score 91.813 Data time: 0.1754, Forward time: 0.9350, Backward time: 1.2543, DDP time: 0.0053, Total iter time: 2.4179 [2020-12-31 07:20:13,315][root][INFO] - ===> Epoch[93](465/5): Loss -0.0133, Sem -0.0133, dir -0.8629, norm 0.6140 LR: 9.721e-02 Score 92.066 Data time: 0.1058, Forward time: 0.9356, Backward time: 1.2558, DDP time: 0.0053, Total iter time: 2.5695 [2020-12-31 07:20:25,565][root][INFO] - ===> Epoch[94](470/5): Loss 0.0028, Sem 0.0028, dir -0.8607, norm 0.6227 LR: 9.718e-02 Score 91.938 Data time: 0.0899, Forward time: 0.9353, Backward time: 1.2561, DDP time: 0.0053, Total iter time: 2.3553 [2020-12-31 07:20:38,604][root][INFO] - ===> Epoch[95](475/5): Loss -0.0022, Sem -0.0022, dir -0.8628, norm 0.6339 LR: 9.715e-02 Score 92.447 Data time: 0.0988, Forward time: 0.9364, Backward time: 1.2568, DDP time: 0.0053, Total iter time: 2.5271 [2020-12-31 07:20:50,012][root][INFO] - ===> Epoch[96](480/5): Loss -0.0013, Sem -0.0013, dir -0.8646, norm 0.6300 LR: 9.712e-02 Score 92.107 Data time: 0.1352, Forward time: 0.9374, Backward time: 1.2539, DDP time: 0.0053, Total iter time: 2.1854 [2020-12-31 07:21:01,881][root][INFO] - ===> Epoch[97](485/5): Loss -0.0151, Sem -0.0151, dir -0.8648, norm 0.6097 LR: 9.709e-02 Score 91.784 Data time: 0.1329, Forward time: 0.9377, Backward time: 1.2527, DDP time: 0.0053, Total iter time: 2.2884 [2020-12-31 07:21:13,818][root][INFO] - ===> Epoch[98](490/5): Loss -0.0182, Sem -0.0182, dir -0.8652, norm 0.6182 LR: 9.706e-02 Score 92.360 Data time: 0.1071, Forward time: 0.9377, Backward time: 1.2519, DDP time: 0.0053, Total iter time: 2.2913 [2020-12-31 07:21:25,411][root][INFO] - ===> Epoch[99](495/5): Loss -0.0177, Sem -0.0177, dir -0.8662, norm 0.6143 LR: 9.703e-02 Score 92.002 Data time: 0.0937, Forward time: 0.9368, Backward time: 1.2516, DDP time: 0.0053, Total iter time: 2.2263 [2020-12-31 07:21:38,017][root][INFO] - ===> Epoch[100](500/5): Loss -0.0361, Sem -0.0361, dir -0.8666, norm 0.5992 LR: 9.700e-02 Score 92.157 Data time: 0.1878, Forward time: 0.9377, Backward time: 1.2505, DDP time: 0.0053, Total iter time: 2.4130 [2020-12-31 07:21:51,407][root][INFO] - ===> Epoch[101](505/5): Loss -0.0342, Sem -0.0342, dir -0.8667, norm 0.6103 LR: 9.697e-02 Score 92.588 Data time: 0.0966, Forward time: 0.9373, Backward time: 1.2529, DDP time: 0.0053, Total iter time: 2.5824 [2020-12-31 07:22:04,056][root][INFO] - ===> Epoch[102](510/5): Loss -0.0638, Sem -0.0638, dir -0.8711, norm 0.5861 LR: 9.693e-02 Score 92.514 Data time: 0.1589, Forward time: 0.9382, Backward time: 1.2523, DDP time: 0.0053, Total iter time: 2.4395 [2020-12-31 07:22:16,071][root][INFO] - ===> Epoch[103](515/5): Loss -0.0442, Sem -0.0442, dir -0.8683, norm 0.5953 LR: 9.690e-02 Score 92.308 Data time: 0.1104, Forward time: 0.9386, Backward time: 1.2515, DDP time: 0.0053, Total iter time: 2.3173 [2020-12-31 07:22:28,765][root][INFO] - ===> Epoch[104](520/5): Loss -0.0543, Sem -0.0543, dir -0.8684, norm 0.5941 LR: 9.687e-02 Score 92.610 Data time: 0.1530, Forward time: 0.9397, Backward time: 1.2509, DDP time: 0.0053, Total iter time: 2.4540 [2020-12-31 07:22:40,265][root][INFO] - ===> Epoch[105](525/5): Loss -0.0721, Sem -0.0721, dir -0.8710, norm 0.5822 LR: 9.684e-02 Score 92.749 Data time: 0.0795, Forward time: 0.9392, Backward time: 1.2504, DDP time: 0.0053, Total iter time: 2.2218 [2020-12-31 07:22:53,092][root][INFO] - ===> Epoch[106](530/5): Loss -0.0533, Sem -0.0533, dir -0.8700, norm 0.6024 LR: 9.681e-02 Score 92.696 Data time: 0.0985, Forward time: 0.9393, Backward time: 1.2514, DDP time: 0.0053, Total iter time: 2.4818 [2020-12-31 07:23:04,842][root][INFO] - ===> Epoch[107](535/5): Loss -0.0266, Sem -0.0266, dir -0.8652, norm 0.6138 LR: 9.678e-02 Score 92.371 Data time: 0.1152, Forward time: 0.9399, Backward time: 1.2498, DDP time: 0.0053, Total iter time: 2.2524 [2020-12-31 07:23:16,964][root][INFO] - ===> Epoch[108](540/5): Loss -0.0585, Sem -0.0585, dir -0.8699, norm 0.6026 LR: 9.675e-02 Score 92.982 Data time: 0.1158, Forward time: 0.9403, Backward time: 1.2490, DDP time: 0.0053, Total iter time: 2.3169 [2020-12-31 07:23:29,136][root][INFO] - ===> Epoch[109](545/5): Loss -0.0852, Sem -0.0852, dir -0.8764, norm 0.5858 LR: 9.672e-02 Score 93.069 Data time: 0.0878, Forward time: 0.9399, Backward time: 1.2495, DDP time: 0.0053, Total iter time: 2.3541 [2020-12-31 07:23:41,789][root][INFO] - ===> Epoch[110](550/5): Loss -0.0778, Sem -0.0778, dir -0.8734, norm 0.5901 LR: 9.669e-02 Score 93.028 Data time: 0.0795, Forward time: 0.9391, Backward time: 1.2510, DDP time: 0.0053, Total iter time: 2.4345 [2020-12-31 07:23:55,613][root][INFO] - ===> Epoch[111](555/5): Loss -0.0661, Sem -0.0661, dir -0.8743, norm 0.5950 LR: 9.666e-02 Score 92.880 Data time: 0.1031, Forward time: 0.9396, Backward time: 1.2531, DDP time: 0.0053, Total iter time: 2.6748 [2020-12-31 07:24:07,655][root][INFO] - ===> Epoch[112](560/5): Loss -0.0916, Sem -0.0916, dir -0.8763, norm 0.5762 LR: 9.663e-02 Score 92.822 Data time: 0.0825, Forward time: 0.9383, Backward time: 1.2540, DDP time: 0.0053, Total iter time: 2.3271 [2020-12-31 07:24:19,816][root][INFO] - ===> Epoch[113](565/5): Loss -0.0463, Sem -0.0463, dir -0.8671, norm 0.5941 LR: 9.660e-02 Score 92.339 Data time: 0.1027, Forward time: 0.9381, Backward time: 1.2538, DDP time: 0.0053, Total iter time: 2.3358 [2020-12-31 07:24:31,575][root][INFO] - ===> Epoch[114](570/5): Loss -0.0589, Sem -0.0589, dir -0.8733, norm 0.5994 LR: 9.657e-02 Score 92.790 Data time: 0.1167, Forward time: 0.9387, Backward time: 1.2524, DDP time: 0.0053, Total iter time: 2.2669 [2020-12-31 07:24:43,985][root][INFO] - ===> Epoch[115](575/5): Loss -0.0744, Sem -0.0744, dir -0.8761, norm 0.5948 LR: 9.654e-02 Score 93.008 Data time: 0.1846, Forward time: 0.9390, Backward time: 1.2518, DDP time: 0.0053, Total iter time: 2.3905 [2020-12-31 07:24:57,109][root][INFO] - ===> Epoch[116](580/5): Loss -0.0754, Sem -0.0754, dir -0.8743, norm 0.5998 LR: 9.651e-02 Score 93.270 Data time: 0.1029, Forward time: 0.9393, Backward time: 1.2528, DDP time: 0.0053, Total iter time: 2.5292 [2020-12-31 07:25:08,809][root][INFO] - ===> Epoch[117](585/5): Loss -0.1082, Sem -0.1082, dir -0.8770, norm 0.5628 LR: 9.648e-02 Score 92.994 Data time: 0.1408, Forward time: 0.9394, Backward time: 1.2516, DDP time: 0.0053, Total iter time: 2.2484 [2020-12-31 07:25:20,397][root][INFO] - ===> Epoch[118](590/5): Loss -0.0908, Sem -0.0908, dir -0.8751, norm 0.5788 LR: 9.645e-02 Score 92.980 Data time: 0.0953, Forward time: 0.9398, Backward time: 1.2504, DDP time: 0.0053, Total iter time: 2.2388 [2020-12-31 07:25:32,324][root][INFO] - ===> Epoch[119](595/5): Loss -0.1136, Sem -0.1136, dir -0.8809, norm 0.5612 LR: 9.642e-02 Score 93.032 Data time: 0.1630, Forward time: 0.9405, Backward time: 1.2487, DDP time: 0.0053, Total iter time: 2.2855 [2020-12-31 07:25:45,355][root][INFO] - ===> Epoch[120](600/5): Loss -0.0994, Sem -0.0994, dir -0.8787, norm 0.5694 LR: 9.639e-02 Score 92.932 Data time: 0.1268, Forward time: 0.9418, Backward time: 1.2487, DDP time: 0.0053, Total iter time: 2.5209 [2020-12-31 07:25:45,357][root][INFO] - ===> Start testing [2020-12-31 07:26:21,267][root][INFO] - 68/68: Data time: 0.0023, Iter time: 0.1085 Loss 0.289 (AVG: 0.447) Score 91.322 (AVG: 87.022) mIOU 62.688 mAP 82.054 mAcc 70.422 IOU: 54.219 0.000 73.619 66.762 92.418 85.274 26.532 55.969 98.149 56.038 79.572 80.912 45.486 mAP: 77.980 11.380 92.086 86.743 99.048 97.017 53.834 84.526 99.770 92.577 93.415 93.375 84.951 mAcc: 75.397 0.000 82.095 74.075 93.724 90.194 31.932 72.084 99.057 62.816 87.770 94.872 51.467 [2020-12-31 07:26:21,268][root][INFO] - Finished test. Elapsed time: 35.9087 [2020-12-31 07:26:24,452][root][INFO] - [2020-12-31 07:26:24,452][root][INFO] - ################################################################ [2020-12-31 07:26:24,452][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 07:26:24,452][root][INFO] - ################################################################ [2020-12-31 07:26:24,452][root][INFO] - clutter : nan nan nan [2020-12-31 07:26:24,452][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 07:26:24,452][root][INFO] - board : 0.606 0.756 0.877 [2020-12-31 07:26:24,452][root][INFO] - bookcase : 0.205 0.358 0.492 [2020-12-31 07:26:24,452][root][INFO] - ceiling : 0.785 0.844 0.856 [2020-12-31 07:26:24,452][root][INFO] - chair : 0.632 0.778 0.820 [2020-12-31 07:26:24,452][root][INFO] - column : 0.048 0.116 0.308 [2020-12-31 07:26:24,452][root][INFO] - door : 0.136 0.308 0.613 [2020-12-31 07:26:24,452][root][INFO] - floor : 0.985 0.991 0.991 [2020-12-31 07:26:24,452][root][INFO] - sofa : 0.214 0.545 0.680 [2020-12-31 07:26:24,452][root][INFO] - table : 0.120 0.272 0.625 [2020-12-31 07:26:24,453][root][INFO] - wall : 0.416 0.643 0.765 [2020-12-31 07:26:24,453][root][INFO] - window : 0.231 0.529 0.687 [2020-12-31 07:26:24,453][root][INFO] - ---------------------------------------------------------------- [2020-12-31 07:26:24,453][root][INFO] - average : 0.365 0.512 0.643 [2020-12-31 07:26:24,453][root][INFO] - [2020-12-31 07:26:24,654][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 07:26:25,279][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2020-12-31 07:26:25,279][root][INFO] - Current best mIoU: 62.688 at iter 1 [2020-12-31 07:26:25,897][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_mAP.pth [2020-12-31 07:26:25,898][root][INFO] - Current best mAP@0.5: 0.512 at iter 600 [2020-12-31 07:26:26,539][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter600.pth [2020-12-31 07:26:38,225][root][INFO] - ===> Epoch[121](605/5): Loss -0.0943, Sem -0.0943, dir -0.8775, norm 0.5827 LR: 9.636e-02 Score 93.279 Data time: 0.1160, Forward time: 0.9428, Backward time: 1.2468, DDP time: 0.0053, Total iter time: 2.2338 [2020-12-31 07:26:51,403][root][INFO] - ===> Epoch[122](610/5): Loss -0.1224, Sem -0.1224, dir -0.8812, norm 0.5633 LR: 9.633e-02 Score 93.393 Data time: 0.0940, Forward time: 0.9440, Backward time: 1.2472, DDP time: 0.0053, Total iter time: 2.5479 [2020-12-31 07:27:03,539][root][INFO] - ===> Epoch[123](615/5): Loss -0.1008, Sem -0.1008, dir -0.8781, norm 0.5754 LR: 9.630e-02 Score 93.213 Data time: 0.1089, Forward time: 0.9448, Backward time: 1.2463, DDP time: 0.0053, Total iter time: 2.3397 [2020-12-31 07:27:15,415][root][INFO] - ===> Epoch[124](620/5): Loss -0.1329, Sem -0.1329, dir -0.8825, norm 0.5595 LR: 9.627e-02 Score 93.541 Data time: 0.1431, Forward time: 0.9446, Backward time: 1.2455, DDP time: 0.0053, Total iter time: 2.2773 [2020-12-31 07:27:27,357][root][INFO] - ===> Epoch[125](625/5): Loss -0.1215, Sem -0.1215, dir -0.8788, norm 0.5620 LR: 9.624e-02 Score 93.443 Data time: 0.1325, Forward time: 0.9450, Backward time: 1.2445, DDP time: 0.0053, Total iter time: 2.2943 [2020-12-31 07:27:39,183][root][INFO] - ===> Epoch[126](630/5): Loss -0.1329, Sem -0.1329, dir -0.8826, norm 0.5613 LR: 9.621e-02 Score 93.649 Data time: 0.1202, Forward time: 0.9449, Backward time: 1.2439, DDP time: 0.0053, Total iter time: 2.2838 [2020-12-31 07:27:52,407][root][INFO] - ===> Epoch[127](635/5): Loss -0.1190, Sem -0.1190, dir -0.8787, norm 0.5577 LR: 9.618e-02 Score 93.266 Data time: 0.1116, Forward time: 0.9448, Backward time: 1.2454, DDP time: 0.0053, Total iter time: 2.5635 [2020-12-31 07:28:04,187][root][INFO] - ===> Epoch[128](640/5): Loss -0.1644, Sem -0.1644, dir -0.8858, norm 0.5366 LR: 9.615e-02 Score 93.769 Data time: 0.0853, Forward time: 0.9450, Backward time: 1.2447, DDP time: 0.0053, Total iter time: 2.2745 [2020-12-31 07:28:16,739][root][INFO] - ===> Epoch[129](645/5): Loss -0.1407, Sem -0.1407, dir -0.8844, norm 0.5488 LR: 9.612e-02 Score 93.413 Data time: 0.0909, Forward time: 0.9449, Backward time: 1.2453, DDP time: 0.0053, Total iter time: 2.4248 [2020-12-31 07:28:28,813][root][INFO] - ===> Epoch[130](650/5): Loss -0.1305, Sem -0.1305, dir -0.8835, norm 0.5661 LR: 9.609e-02 Score 93.719 Data time: 0.1006, Forward time: 0.9453, Backward time: 1.2448, DDP time: 0.0053, Total iter time: 2.3326 [2020-12-31 07:28:40,655][root][INFO] - ===> Epoch[131](655/5): Loss -0.1396, Sem -0.1396, dir -0.8823, norm 0.5578 LR: 9.606e-02 Score 93.765 Data time: 0.0872, Forward time: 0.9448, Backward time: 1.2448, DDP time: 0.0053, Total iter time: 2.2944 [2020-12-31 07:28:52,068][root][INFO] - ===> Epoch[132](660/5): Loss -0.1513, Sem -0.1513, dir -0.8833, norm 0.5446 LR: 9.603e-02 Score 93.712 Data time: 0.0928, Forward time: 0.9450, Backward time: 1.2437, DDP time: 0.0053, Total iter time: 2.2021 [2020-12-31 07:29:04,905][root][INFO] - ===> Epoch[133](665/5): Loss -0.1576, Sem -0.1576, dir -0.8876, norm 0.5554 LR: 9.600e-02 Score 94.051 Data time: 0.1850, Forward time: 0.9458, Backward time: 1.2431, DDP time: 0.0053, Total iter time: 2.4689 [2020-12-31 07:29:16,584][root][INFO] - ===> Epoch[134](670/5): Loss -0.1159, Sem -0.1159, dir -0.8801, norm 0.5658 LR: 9.597e-02 Score 93.308 Data time: 0.1113, Forward time: 0.9462, Backward time: 1.2420, DDP time: 0.0053, Total iter time: 2.2556 [2020-12-31 07:29:28,725][root][INFO] - ===> Epoch[135](675/5): Loss -0.1488, Sem -0.1488, dir -0.8845, norm 0.5570 LR: 9.594e-02 Score 93.992 Data time: 0.1103, Forward time: 0.9464, Backward time: 1.2415, DDP time: 0.0053, Total iter time: 2.3268 [2020-12-31 07:29:40,524][root][INFO] - ===> Epoch[136](680/5): Loss -0.1528, Sem -0.1528, dir -0.8859, norm 0.5482 LR: 9.591e-02 Score 93.727 Data time: 0.0881, Forward time: 0.9467, Backward time: 1.2407, DDP time: 0.0053, Total iter time: 2.2685 [2020-12-31 07:29:52,751][root][INFO] - ===> Epoch[137](685/5): Loss -0.1651, Sem -0.1651, dir -0.8858, norm 0.5371 LR: 9.588e-02 Score 93.793 Data time: 0.1245, Forward time: 0.9472, Backward time: 1.2401, DDP time: 0.0053, Total iter time: 2.3552 [2020-12-31 07:30:05,675][root][INFO] - ===> Epoch[138](690/5): Loss -0.1443, Sem -0.1443, dir -0.8864, norm 0.5622 LR: 9.585e-02 Score 94.004 Data time: 0.1178, Forward time: 0.9476, Backward time: 1.2405, DDP time: 0.0053, Total iter time: 2.5015 [2020-12-31 07:30:17,278][root][INFO] - ===> Epoch[139](695/5): Loss -0.1783, Sem -0.1783, dir -0.8901, norm 0.5288 LR: 9.582e-02 Score 93.819 Data time: 0.0972, Forward time: 0.9478, Backward time: 1.2394, DDP time: 0.0053, Total iter time: 2.2296 [2020-12-31 07:30:29,915][root][INFO] - ===> Epoch[140](700/5): Loss -0.1600, Sem -0.1600, dir -0.8855, norm 0.5447 LR: 9.579e-02 Score 93.805 Data time: 0.1147, Forward time: 0.9484, Backward time: 1.2394, DDP time: 0.0053, Total iter time: 2.4366 [2020-12-31 07:30:41,576][root][INFO] - ===> Epoch[141](705/5): Loss -0.1603, Sem -0.1603, dir -0.8866, norm 0.5481 LR: 9.576e-02 Score 93.994 Data time: 0.0975, Forward time: 0.9484, Backward time: 1.2386, DDP time: 0.0053, Total iter time: 2.2333 [2020-12-31 07:30:54,181][root][INFO] - ===> Epoch[142](710/5): Loss -0.1793, Sem -0.1793, dir -0.8899, norm 0.5318 LR: 9.573e-02 Score 93.981 Data time: 0.1454, Forward time: 0.9481, Backward time: 1.2390, DDP time: 0.0053, Total iter time: 2.4308 [2020-12-31 07:31:06,507][root][INFO] - ===> Epoch[143](715/5): Loss -0.1797, Sem -0.1797, dir -0.8908, norm 0.5349 LR: 9.570e-02 Score 94.023 Data time: 0.1468, Forward time: 0.9489, Backward time: 1.2382, DDP time: 0.0053, Total iter time: 2.3739 [2020-12-31 07:31:17,966][root][INFO] - ===> Epoch[144](720/5): Loss -0.1808, Sem -0.1808, dir -0.8881, norm 0.5352 LR: 9.567e-02 Score 94.187 Data time: 0.1396, Forward time: 0.9493, Backward time: 1.2365, DDP time: 0.0053, Total iter time: 2.1999 [2020-12-31 07:31:29,749][root][INFO] - ===> Epoch[145](725/5): Loss -0.1699, Sem -0.1699, dir -0.8877, norm 0.5433 LR: 9.564e-02 Score 94.169 Data time: 0.1089, Forward time: 0.9494, Backward time: 1.2357, DDP time: 0.0053, Total iter time: 2.2618 [2020-12-31 07:31:42,284][root][INFO] - ===> Epoch[146](730/5): Loss -0.1557, Sem -0.1557, dir -0.8873, norm 0.5510 LR: 9.561e-02 Score 93.899 Data time: 0.1389, Forward time: 0.9502, Backward time: 1.2352, DDP time: 0.0053, Total iter time: 2.4279 [2020-12-31 07:31:54,228][root][INFO] - ===> Epoch[147](735/5): Loss -0.1698, Sem -0.1698, dir -0.8878, norm 0.5370 LR: 9.558e-02 Score 93.936 Data time: 0.1132, Forward time: 0.9505, Backward time: 1.2344, DDP time: 0.0053, Total iter time: 2.3001 [2020-12-31 07:32:06,822][root][INFO] - ===> Epoch[148](740/5): Loss -0.1626, Sem -0.1626, dir -0.8911, norm 0.5484 LR: 9.555e-02 Score 93.960 Data time: 0.0882, Forward time: 0.9504, Backward time: 1.2350, DDP time: 0.0053, Total iter time: 2.4382 [2020-12-31 07:32:19,090][root][INFO] - ===> Epoch[149](745/5): Loss -0.1750, Sem -0.1750, dir -0.8911, norm 0.5365 LR: 9.552e-02 Score 93.836 Data time: 0.0849, Forward time: 0.9506, Backward time: 1.2349, DDP time: 0.0053, Total iter time: 2.3533 [2020-12-31 07:32:30,886][root][INFO] - ===> Epoch[150](750/5): Loss -0.1825, Sem -0.1825, dir -0.8915, norm 0.5420 LR: 9.549e-02 Score 94.447 Data time: 0.1089, Forward time: 0.9511, Backward time: 1.2339, DDP time: 0.0053, Total iter time: 2.2667 [2020-12-31 07:32:42,535][root][INFO] - ===> Epoch[151](755/5): Loss -0.1671, Sem -0.1671, dir -0.8894, norm 0.5474 LR: 9.546e-02 Score 94.082 Data time: 0.0840, Forward time: 0.9506, Backward time: 1.2338, DDP time: 0.0053, Total iter time: 2.2373 [2020-12-31 07:32:54,523][root][INFO] - ===> Epoch[152](760/5): Loss -0.1730, Sem -0.1730, dir -0.8910, norm 0.5509 LR: 9.543e-02 Score 94.304 Data time: 0.1672, Forward time: 0.9512, Backward time: 1.2326, DDP time: 0.0053, Total iter time: 2.3095 [2020-12-31 07:33:06,055][root][INFO] - ===> Epoch[153](765/5): Loss -0.2111, Sem -0.2111, dir -0.8923, norm 0.5197 LR: 9.540e-02 Score 94.535 Data time: 0.1145, Forward time: 0.9515, Backward time: 1.2313, DDP time: 0.0053, Total iter time: 2.2039 [2020-12-31 07:33:17,853][root][INFO] - ===> Epoch[154](770/5): Loss -0.2078, Sem -0.2078, dir -0.8942, norm 0.5223 LR: 9.537e-02 Score 94.480 Data time: 0.1780, Forward time: 0.9512, Backward time: 1.2307, DDP time: 0.0053, Total iter time: 2.2888 [2020-12-31 07:33:30,807][root][INFO] - ===> Epoch[155](775/5): Loss -0.1974, Sem -0.1974, dir -0.8944, norm 0.5330 LR: 9.534e-02 Score 94.495 Data time: 0.1499, Forward time: 0.9526, Backward time: 1.2301, DDP time: 0.0053, Total iter time: 2.4872 [2020-12-31 07:33:43,365][root][INFO] - ===> Epoch[156](780/5): Loss -0.2011, Sem -0.2011, dir -0.8930, norm 0.5232 LR: 9.531e-02 Score 94.290 Data time: 0.1145, Forward time: 0.9527, Backward time: 1.2303, DDP time: 0.0053, Total iter time: 2.4222 [2020-12-31 07:33:54,758][root][INFO] - ===> Epoch[157](785/5): Loss -0.1997, Sem -0.1997, dir -0.8924, norm 0.5345 LR: 9.528e-02 Score 94.614 Data time: 0.0990, Forward time: 0.9534, Backward time: 1.2287, DDP time: 0.0053, Total iter time: 2.1824 [2020-12-31 07:34:06,786][root][INFO] - ===> Epoch[158](790/5): Loss -0.2049, Sem -0.2049, dir -0.8952, norm 0.5320 LR: 9.525e-02 Score 94.699 Data time: 0.1009, Forward time: 0.9532, Backward time: 1.2287, DDP time: 0.0053, Total iter time: 2.3167 [2020-12-31 07:34:20,009][root][INFO] - ===> Epoch[159](795/5): Loss -0.1906, Sem -0.1906, dir -0.8910, norm 0.5388 LR: 9.522e-02 Score 94.504 Data time: 0.1683, Forward time: 0.9541, Backward time: 1.2285, DDP time: 0.0053, Total iter time: 2.5497 [2020-12-31 07:34:32,270][root][INFO] - ===> Epoch[160](800/5): Loss -0.1876, Sem -0.1876, dir -0.8921, norm 0.5439 LR: 9.519e-02 Score 94.563 Data time: 0.0851, Forward time: 0.9541, Backward time: 1.2288, DDP time: 0.0053, Total iter time: 2.3680 [2020-12-31 07:34:32,272][root][INFO] - ===> Start testing [2020-12-31 07:35:09,078][root][INFO] - 68/68: Data time: 0.0025, Iter time: 0.1293 Loss 0.326 (AVG: 0.431) Score 91.795 (AVG: 87.748) mIOU 62.046 mAP 81.482 mAcc 69.700 IOU: 54.506 0.000 59.804 69.700 93.931 86.445 20.355 57.484 98.260 56.508 81.533 81.110 46.963 mAP: 79.092 5.846 90.474 86.944 99.137 97.782 53.054 85.516 99.778 88.762 93.815 92.894 86.178 mAcc: 67.905 0.000 77.233 82.039 97.485 92.024 22.439 67.816 99.331 61.721 90.336 94.709 53.056 [2020-12-31 07:35:09,078][root][INFO] - Finished test. Elapsed time: 36.8041 [2020-12-31 07:35:12,034][root][INFO] - [2020-12-31 07:35:12,034][root][INFO] - ################################################################ [2020-12-31 07:35:12,034][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 07:35:12,034][root][INFO] - ################################################################ [2020-12-31 07:35:12,035][root][INFO] - clutter : nan nan nan [2020-12-31 07:35:12,035][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 07:35:12,035][root][INFO] - board : 0.573 0.743 0.816 [2020-12-31 07:35:12,035][root][INFO] - bookcase : 0.273 0.470 0.565 [2020-12-31 07:35:12,035][root][INFO] - ceiling : 0.772 0.863 0.895 [2020-12-31 07:35:12,035][root][INFO] - chair : 0.659 0.802 0.848 [2020-12-31 07:35:12,035][root][INFO] - column : 0.031 0.075 0.237 [2020-12-31 07:35:12,035][root][INFO] - door : 0.163 0.396 0.664 [2020-12-31 07:35:12,035][root][INFO] - floor : 0.952 0.972 0.990 [2020-12-31 07:35:12,035][root][INFO] - sofa : 0.075 0.279 0.708 [2020-12-31 07:35:12,035][root][INFO] - table : 0.138 0.294 0.625 [2020-12-31 07:35:12,035][root][INFO] - wall : 0.392 0.638 0.763 [2020-12-31 07:35:12,035][root][INFO] - window : 0.263 0.620 0.678 [2020-12-31 07:35:12,035][root][INFO] - ---------------------------------------------------------------- [2020-12-31 07:35:12,035][root][INFO] - average : 0.358 0.513 0.649 [2020-12-31 07:35:12,035][root][INFO] - [2020-12-31 07:35:12,188][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 07:35:12,782][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_mAP.pth [2020-12-31 07:35:12,783][root][INFO] - Current best mAP@0.5: 0.513 at iter 800 [2020-12-31 07:35:13,361][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter800.pth [2020-12-31 07:35:25,201][root][INFO] - ===> Epoch[161](805/5): Loss -0.1714, Sem -0.1714, dir -0.8899, norm 0.5493 LR: 9.516e-02 Score 94.216 Data time: 0.0679, Forward time: 0.9530, Backward time: 1.2295, DDP time: 0.0053, Total iter time: 2.2799 [2020-12-31 07:35:37,398][root][INFO] - ===> Epoch[162](810/5): Loss -0.2116, Sem -0.2116, dir -0.8944, norm 0.5209 LR: 9.513e-02 Score 94.537 Data time: 0.0889, Forward time: 0.9523, Backward time: 1.2301, DDP time: 0.0053, Total iter time: 2.3498 [2020-12-31 07:35:49,797][root][INFO] - ===> Epoch[163](815/5): Loss -0.2152, Sem -0.2152, dir -0.8963, norm 0.5214 LR: 9.510e-02 Score 94.622 Data time: 0.1133, Forward time: 0.9520, Backward time: 1.2305, DDP time: 0.0053, Total iter time: 2.3871 [2020-12-31 07:36:02,383][root][INFO] - ===> Epoch[164](820/5): Loss -0.2127, Sem -0.2127, dir -0.8951, norm 0.5254 LR: 9.507e-02 Score 94.678 Data time: 0.1242, Forward time: 0.9529, Backward time: 1.2301, DDP time: 0.0053, Total iter time: 2.4360 [2020-12-31 07:36:13,738][root][INFO] - ===> Epoch[165](825/5): Loss -0.2370, Sem -0.2370, dir -0.8965, norm 0.4990 LR: 9.504e-02 Score 94.629 Data time: 0.0792, Forward time: 0.9523, Backward time: 1.2299, DDP time: 0.0053, Total iter time: 2.2015 [2020-12-31 07:36:26,035][root][INFO] - ===> Epoch[166](830/5): Loss -0.2034, Sem -0.2034, dir -0.8929, norm 0.5304 LR: 9.501e-02 Score 94.690 Data time: 0.1409, Forward time: 0.9524, Backward time: 1.2298, DDP time: 0.0052, Total iter time: 2.3667 [2020-12-31 07:36:38,418][root][INFO] - ===> Epoch[167](835/5): Loss -0.2297, Sem -0.2297, dir -0.8958, norm 0.5129 LR: 9.498e-02 Score 94.814 Data time: 0.1351, Forward time: 0.9530, Backward time: 1.2293, DDP time: 0.0052, Total iter time: 2.3805 [2020-12-31 07:36:50,895][root][INFO] - ===> Epoch[168](840/5): Loss -0.2103, Sem -0.2103, dir -0.8956, norm 0.5300 LR: 9.495e-02 Score 94.766 Data time: 0.1230, Forward time: 0.9535, Backward time: 1.2291, DDP time: 0.0052, Total iter time: 2.4127 [2020-12-31 07:37:03,005][root][INFO] - ===> Epoch[169](845/5): Loss -0.2308, Sem -0.2308, dir -0.8986, norm 0.5112 LR: 9.492e-02 Score 94.722 Data time: 0.1000, Forward time: 0.9531, Backward time: 1.2294, DDP time: 0.0053, Total iter time: 2.3354 [2020-12-31 07:37:15,205][root][INFO] - ===> Epoch[170](850/5): Loss -0.2175, Sem -0.2175, dir -0.8984, norm 0.5233 LR: 9.489e-02 Score 94.687 Data time: 0.1128, Forward time: 0.9536, Backward time: 1.2289, DDP time: 0.0052, Total iter time: 2.3514 [2020-12-31 07:37:27,502][root][INFO] - ===> Epoch[171](855/5): Loss -0.2370, Sem -0.2370, dir -0.8984, norm 0.5159 LR: 9.486e-02 Score 95.050 Data time: 0.1086, Forward time: 0.9534, Backward time: 1.2292, DDP time: 0.0052, Total iter time: 2.3694 [2020-12-31 07:37:39,144][root][INFO] - ===> Epoch[172](860/5): Loss -0.2126, Sem -0.2126, dir -0.8942, norm 0.5220 LR: 9.483e-02 Score 94.609 Data time: 0.1713, Forward time: 0.9535, Backward time: 1.2281, DDP time: 0.0052, Total iter time: 2.2282 [2020-12-31 07:37:51,652][root][INFO] - ===> Epoch[173](865/5): Loss -0.2196, Sem -0.2196, dir -0.8981, norm 0.5224 LR: 9.480e-02 Score 94.668 Data time: 0.1076, Forward time: 0.9534, Backward time: 1.2286, DDP time: 0.0053, Total iter time: 2.4346 [2020-12-31 07:38:03,896][root][INFO] - ===> Epoch[174](870/5): Loss -0.2539, Sem -0.2539, dir -0.9006, norm 0.5026 LR: 9.476e-02 Score 95.145 Data time: 0.1596, Forward time: 0.9539, Backward time: 1.2280, DDP time: 0.0052, Total iter time: 2.3473 [2020-12-31 07:38:16,192][root][INFO] - ===> Epoch[175](875/5): Loss -0.2162, Sem -0.2162, dir -0.8980, norm 0.5319 LR: 9.473e-02 Score 94.958 Data time: 0.1031, Forward time: 0.9539, Backward time: 1.2280, DDP time: 0.0052, Total iter time: 2.3602 [2020-12-31 07:38:28,143][root][INFO] - ===> Epoch[176](880/5): Loss -0.2367, Sem -0.2367, dir -0.8981, norm 0.5174 LR: 9.470e-02 Score 95.157 Data time: 0.0889, Forward time: 0.9537, Backward time: 1.2279, DDP time: 0.0052, Total iter time: 2.3027 [2020-12-31 07:38:39,707][root][INFO] - ===> Epoch[177](885/5): Loss -0.2382, Sem -0.2382, dir -0.8996, norm 0.5124 LR: 9.467e-02 Score 94.979 Data time: 0.0665, Forward time: 0.9528, Backward time: 1.2284, DDP time: 0.0052, Total iter time: 2.2396 [2020-12-31 07:38:51,992][root][INFO] - ===> Epoch[178](890/5): Loss -0.1981, Sem -0.1981, dir -0.8953, norm 0.5377 LR: 9.464e-02 Score 94.627 Data time: 0.0886, Forward time: 0.9527, Backward time: 1.2286, DDP time: 0.0052, Total iter time: 2.3657 [2020-12-31 07:39:03,263][root][INFO] - ===> Epoch[179](895/5): Loss -0.2542, Sem -0.2542, dir -0.9001, norm 0.5045 LR: 9.461e-02 Score 95.273 Data time: 0.0930, Forward time: 0.9524, Backward time: 1.2279, DDP time: 0.0052, Total iter time: 2.1578 [2020-12-31 07:39:16,512][root][INFO] - ===> Epoch[180](900/5): Loss -0.2354, Sem -0.2354, dir -0.9001, norm 0.5141 LR: 9.458e-02 Score 94.936 Data time: 0.0951, Forward time: 0.9527, Backward time: 1.2288, DDP time: 0.0052, Total iter time: 2.5631 [2020-12-31 07:39:28,775][root][INFO] - ===> Epoch[181](905/5): Loss -0.2628, Sem -0.2628, dir -0.9032, norm 0.4986 LR: 9.455e-02 Score 95.236 Data time: 0.0956, Forward time: 0.9529, Backward time: 1.2289, DDP time: 0.0052, Total iter time: 2.3753 [2020-12-31 07:39:40,817][root][INFO] - ===> Epoch[182](910/5): Loss -0.2462, Sem -0.2462, dir -0.9017, norm 0.5030 LR: 9.452e-02 Score 94.864 Data time: 0.0917, Forward time: 0.9532, Backward time: 1.2286, DDP time: 0.0052, Total iter time: 2.3276 [2020-12-31 07:39:53,243][root][INFO] - ===> Epoch[183](915/5): Loss -0.2416, Sem -0.2416, dir -0.9007, norm 0.5105 LR: 9.449e-02 Score 95.016 Data time: 0.0949, Forward time: 0.9530, Backward time: 1.2289, DDP time: 0.0052, Total iter time: 2.4026 [2020-12-31 07:40:05,370][root][INFO] - ===> Epoch[184](920/5): Loss -0.2780, Sem -0.2780, dir -0.9024, norm 0.4914 LR: 9.446e-02 Score 95.525 Data time: 0.1229, Forward time: 0.9534, Backward time: 1.2285, DDP time: 0.0052, Total iter time: 2.3325 [2020-12-31 07:40:17,582][root][INFO] - ===> Epoch[185](925/5): Loss -0.2617, Sem -0.2617, dir -0.9020, norm 0.4969 LR: 9.443e-02 Score 95.167 Data time: 0.0955, Forward time: 0.9531, Backward time: 1.2289, DDP time: 0.0052, Total iter time: 2.3644 [2020-12-31 07:40:28,940][root][INFO] - ===> Epoch[186](930/5): Loss -0.2487, Sem -0.2487, dir -0.9019, norm 0.5080 LR: 9.440e-02 Score 95.080 Data time: 0.1002, Forward time: 0.9530, Backward time: 1.2281, DDP time: 0.0052, Total iter time: 2.1795 [2020-12-31 07:40:41,649][root][INFO] - ===> Epoch[187](935/5): Loss -0.2568, Sem -0.2568, dir -0.9011, norm 0.5002 LR: 9.437e-02 Score 95.143 Data time: 0.0953, Forward time: 0.9528, Backward time: 1.2288, DDP time: 0.0052, Total iter time: 2.4451 [2020-12-31 07:40:53,482][root][INFO] - ===> Epoch[188](940/5): Loss -0.2607, Sem -0.2607, dir -0.9031, norm 0.4945 LR: 9.434e-02 Score 95.007 Data time: 0.0951, Forward time: 0.9524, Backward time: 1.2289, DDP time: 0.0052, Total iter time: 2.2847 [2020-12-31 07:41:05,297][root][INFO] - ===> Epoch[189](945/5): Loss -0.2693, Sem -0.2693, dir -0.9042, norm 0.4967 LR: 9.431e-02 Score 95.280 Data time: 0.1234, Forward time: 0.9526, Backward time: 1.2283, DDP time: 0.0052, Total iter time: 2.2707 [2020-12-31 07:41:17,456][root][INFO] - ===> Epoch[190](950/5): Loss -0.2709, Sem -0.2709, dir -0.9040, norm 0.4927 LR: 9.428e-02 Score 95.303 Data time: 0.1284, Forward time: 0.9531, Backward time: 1.2277, DDP time: 0.0052, Total iter time: 2.3369 [2020-12-31 07:41:29,627][root][INFO] - ===> Epoch[191](955/5): Loss -0.2674, Sem -0.2674, dir -0.9027, norm 0.4964 LR: 9.425e-02 Score 95.373 Data time: 0.1054, Forward time: 0.9530, Backward time: 1.2277, DDP time: 0.0052, Total iter time: 2.3384 [2020-12-31 07:41:40,666][root][INFO] - ===> Epoch[192](960/5): Loss -0.2804, Sem -0.2804, dir -0.9058, norm 0.4850 LR: 9.422e-02 Score 95.249 Data time: 0.1232, Forward time: 0.9528, Backward time: 1.2267, DDP time: 0.0052, Total iter time: 2.1098 [2020-12-31 07:41:54,155][root][INFO] - ===> Epoch[193](965/5): Loss -0.2880, Sem -0.2880, dir -0.9037, norm 0.4740 LR: 9.419e-02 Score 95.185 Data time: 0.0876, Forward time: 0.9525, Backward time: 1.2283, DDP time: 0.0052, Total iter time: 2.6179 [2020-12-31 07:42:06,006][root][INFO] - ===> Epoch[194](970/5): Loss -0.2789, Sem -0.2789, dir -0.9042, norm 0.4889 LR: 9.416e-02 Score 95.393 Data time: 0.1043, Forward time: 0.9524, Backward time: 1.2281, DDP time: 0.0052, Total iter time: 2.2839 [2020-12-31 07:42:18,064][root][INFO] - ===> Epoch[195](975/5): Loss -0.2935, Sem -0.2935, dir -0.9073, norm 0.4819 LR: 9.413e-02 Score 95.579 Data time: 0.0771, Forward time: 0.9519, Backward time: 1.2286, DDP time: 0.0052, Total iter time: 2.3296 [2020-12-31 07:42:30,416][root][INFO] - ===> Epoch[196](980/5): Loss -0.3120, Sem -0.3120, dir -0.9082, norm 0.4620 LR: 9.410e-02 Score 95.469 Data time: 0.0731, Forward time: 0.9512, Backward time: 1.2295, DDP time: 0.0052, Total iter time: 2.3905 [2020-12-31 07:42:42,337][root][INFO] - ===> Epoch[197](985/5): Loss -0.3004, Sem -0.3004, dir -0.9067, norm 0.4739 LR: 9.407e-02 Score 95.522 Data time: 0.1805, Forward time: 0.9515, Backward time: 1.2285, DDP time: 0.0052, Total iter time: 2.2819 [2020-12-31 07:42:54,805][root][INFO] - ===> Epoch[198](990/5): Loss -0.2993, Sem -0.2993, dir -0.9073, norm 0.4786 LR: 9.404e-02 Score 95.659 Data time: 0.0914, Forward time: 0.9516, Backward time: 1.2287, DDP time: 0.0052, Total iter time: 2.4062 [2020-12-31 07:43:06,435][root][INFO] - ===> Epoch[199](995/5): Loss -0.3169, Sem -0.3169, dir -0.9087, norm 0.4630 LR: 9.401e-02 Score 95.700 Data time: 0.1068, Forward time: 0.9516, Backward time: 1.2282, DDP time: 0.0052, Total iter time: 2.2276 [2020-12-31 07:43:17,860][root][INFO] - ===> Epoch[200](1000/5): Loss -0.2814, Sem -0.2814, dir -0.9062, norm 0.4848 LR: 9.398e-02 Score 95.284 Data time: 0.0813, Forward time: 0.9511, Backward time: 1.2279, DDP time: 0.0052, Total iter time: 2.1910 [2020-12-31 07:43:17,862][root][INFO] - ===> Start testing [2020-12-31 07:43:53,415][root][INFO] - 68/68: Data time: 0.0962, Iter time: 0.1221 Loss 0.360 (AVG: 0.430) Score 90.232 (AVG: 88.399) mIOU 65.564 mAP 82.633 mAcc 74.080 IOU: 58.041 0.000 70.442 70.919 94.077 85.969 29.238 65.176 98.436 66.028 79.979 82.072 51.951 mAP: 81.247 5.432 91.130 86.794 99.121 97.229 56.932 89.603 99.813 91.987 93.650 92.551 88.744 mAcc: 75.202 0.000 84.913 79.352 96.241 92.635 40.234 84.429 99.399 69.376 87.571 92.232 61.451 [2020-12-31 07:43:53,416][root][INFO] - Finished test. Elapsed time: 35.5515 [2020-12-31 07:43:56,837][root][INFO] - [2020-12-31 07:43:56,837][root][INFO] - ################################################################ [2020-12-31 07:43:56,837][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 07:43:56,838][root][INFO] - ################################################################ [2020-12-31 07:43:56,838][root][INFO] - clutter : nan nan nan [2020-12-31 07:43:56,838][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 07:43:56,838][root][INFO] - board : 0.660 0.821 0.920 [2020-12-31 07:43:56,838][root][INFO] - bookcase : 0.244 0.399 0.509 [2020-12-31 07:43:56,838][root][INFO] - ceiling : 0.780 0.819 0.863 [2020-12-31 07:43:56,838][root][INFO] - chair : 0.691 0.828 0.859 [2020-12-31 07:43:56,838][root][INFO] - column : 0.076 0.175 0.304 [2020-12-31 07:43:56,838][root][INFO] - door : 0.382 0.618 0.804 [2020-12-31 07:43:56,838][root][INFO] - floor : 0.922 0.929 0.971 [2020-12-31 07:43:56,838][root][INFO] - sofa : 0.313 0.545 0.708 [2020-12-31 07:43:56,838][root][INFO] - table : 0.143 0.307 0.582 [2020-12-31 07:43:56,838][root][INFO] - wall : 0.446 0.662 0.771 [2020-12-31 07:43:56,838][root][INFO] - window : 0.375 0.680 0.693 [2020-12-31 07:43:56,838][root][INFO] - ---------------------------------------------------------------- [2020-12-31 07:43:56,838][root][INFO] - average : 0.419 0.565 0.666 [2020-12-31 07:43:56,838][root][INFO] - [2020-12-31 07:43:57,024][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 07:43:57,673][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2020-12-31 07:43:57,673][root][INFO] - Current best mIoU: 65.564 at iter 1 [2020-12-31 07:43:58,259][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_mAP.pth [2020-12-31 07:43:58,259][root][INFO] - Current best mAP@0.5: 0.565 at iter 1000 [2020-12-31 07:43:58,891][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter1000.pth [2020-12-31 07:44:10,376][root][INFO] - ===> Epoch[201](1005/5): Loss -0.2888, Sem -0.2888, dir -0.9049, norm 0.4851 LR: 9.395e-02 Score 95.594 Data time: 0.1138, Forward time: 0.9510, Backward time: 1.2272, DDP time: 0.0052, Total iter time: 2.1825 [2020-12-31 07:44:22,418][root][INFO] - ===> Epoch[202](1010/5): Loss -0.2856, Sem -0.2856, dir -0.9068, norm 0.4857 LR: 9.392e-02 Score 95.475 Data time: 0.2342, Forward time: 0.9514, Backward time: 1.2262, DDP time: 0.0052, Total iter time: 2.3205 [2020-12-31 07:44:35,115][root][INFO] - ===> Epoch[203](1015/5): Loss -0.2718, Sem -0.2718, dir -0.9049, norm 0.4886 LR: 9.389e-02 Score 95.157 Data time: 0.1284, Forward time: 0.9513, Backward time: 1.2266, DDP time: 0.0052, Total iter time: 2.4577 [2020-12-31 07:44:46,653][root][INFO] - ===> Epoch[204](1020/5): Loss -0.2954, Sem -0.2954, dir -0.9066, norm 0.4816 LR: 9.386e-02 Score 95.638 Data time: 0.1027, Forward time: 0.9516, Backward time: 1.2259, DDP time: 0.0052, Total iter time: 2.2157 [2020-12-31 07:44:58,725][root][INFO] - ===> Epoch[205](1025/5): Loss -0.3062, Sem -0.3062, dir -0.9090, norm 0.4696 LR: 9.383e-02 Score 95.499 Data time: 0.1127, Forward time: 0.9522, Backward time: 1.2252, DDP time: 0.0052, Total iter time: 2.3297 [2020-12-31 07:45:11,161][root][INFO] - ===> Epoch[206](1030/5): Loss -0.2817, Sem -0.2817, dir -0.9065, norm 0.4899 LR: 9.380e-02 Score 95.482 Data time: 0.0890, Forward time: 0.9520, Backward time: 1.2258, DDP time: 0.0052, Total iter time: 2.4170 [2020-12-31 07:45:22,488][root][INFO] - ===> Epoch[207](1035/5): Loss -0.3005, Sem -0.3005, dir -0.9087, norm 0.4751 LR: 9.377e-02 Score 95.516 Data time: 0.1003, Forward time: 0.9520, Backward time: 1.2250, DDP time: 0.0052, Total iter time: 2.1686 [2020-12-31 07:45:35,914][root][INFO] - ===> Epoch[208](1040/5): Loss -0.2850, Sem -0.2850, dir -0.9067, norm 0.4947 LR: 9.374e-02 Score 95.762 Data time: 0.0939, Forward time: 0.9522, Backward time: 1.2260, DDP time: 0.0052, Total iter time: 2.6093 [2020-12-31 07:45:48,302][root][INFO] - ===> Epoch[209](1045/5): Loss -0.2909, Sem -0.2909, dir -0.9079, norm 0.4885 LR: 9.371e-02 Score 95.666 Data time: 0.1122, Forward time: 0.9524, Backward time: 1.2259, DDP time: 0.0052, Total iter time: 2.3918 [2020-12-31 07:45:59,755][root][INFO] - ===> Epoch[210](1050/5): Loss -0.3175, Sem -0.3175, dir -0.9092, norm 0.4632 LR: 9.368e-02 Score 95.715 Data time: 0.1353, Forward time: 0.9522, Backward time: 1.2253, DDP time: 0.0052, Total iter time: 2.2016 [2020-12-31 07:46:11,476][root][INFO] - ===> Epoch[211](1055/5): Loss -0.3139, Sem -0.3139, dir -0.9104, norm 0.4721 LR: 9.365e-02 Score 95.816 Data time: 0.1229, Forward time: 0.9523, Backward time: 1.2247, DDP time: 0.0052, Total iter time: 2.2449 [2020-12-31 07:46:23,873][root][INFO] - ===> Epoch[212](1060/5): Loss -0.3050, Sem -0.3050, dir -0.9112, norm 0.4805 LR: 9.362e-02 Score 95.796 Data time: 0.0920, Forward time: 0.9524, Backward time: 1.2248, DDP time: 0.0052, Total iter time: 2.3906 [2020-12-31 07:46:36,624][root][INFO] - ===> Epoch[213](1065/5): Loss -0.3093, Sem -0.3093, dir -0.9082, norm 0.4714 LR: 9.359e-02 Score 95.664 Data time: 0.1048, Forward time: 0.9527, Backward time: 1.2250, DDP time: 0.0052, Total iter time: 2.4541 [2020-12-31 07:46:49,344][root][INFO] - ===> Epoch[214](1070/5): Loss -0.3349, Sem -0.3349, dir -0.9120, norm 0.4541 LR: 9.356e-02 Score 95.824 Data time: 0.0828, Forward time: 0.9523, Backward time: 1.2259, DDP time: 0.0052, Total iter time: 2.4532 [2020-12-31 07:47:01,837][root][INFO] - ===> Epoch[215](1075/5): Loss -0.3257, Sem -0.3257, dir -0.9118, norm 0.4660 LR: 9.353e-02 Score 95.977 Data time: 0.0871, Forward time: 0.9518, Backward time: 1.2267, DDP time: 0.0052, Total iter time: 2.4126 [2020-12-31 07:47:14,310][root][INFO] - ===> Epoch[216](1080/5): Loss -0.3210, Sem -0.3210, dir -0.9100, norm 0.4651 LR: 9.350e-02 Score 95.881 Data time: 0.1123, Forward time: 0.9520, Backward time: 1.2267, DDP time: 0.0052, Total iter time: 2.4080 [2020-12-31 07:47:25,827][root][INFO] - ===> Epoch[217](1085/5): Loss -0.3442, Sem -0.3442, dir -0.9160, norm 0.4510 LR: 9.347e-02 Score 95.905 Data time: 0.1066, Forward time: 0.9520, Backward time: 1.2261, DDP time: 0.0052, Total iter time: 2.2121 [2020-12-31 07:47:37,062][root][INFO] - ===> Epoch[218](1090/5): Loss -0.3086, Sem -0.3086, dir -0.9109, norm 0.4711 LR: 9.344e-02 Score 95.627 Data time: 0.1341, Forward time: 0.9521, Backward time: 1.2251, DDP time: 0.0052, Total iter time: 2.1590 [2020-12-31 07:47:49,600][root][INFO] - ===> Epoch[219](1095/5): Loss -0.3221, Sem -0.3221, dir -0.9117, norm 0.4666 LR: 9.341e-02 Score 95.835 Data time: 0.1443, Forward time: 0.9524, Backward time: 1.2248, DDP time: 0.0052, Total iter time: 2.4035 [2020-12-31 07:48:02,096][root][INFO] - ===> Epoch[220](1100/5): Loss -0.3209, Sem -0.3209, dir -0.9111, norm 0.4643 LR: 9.338e-02 Score 95.784 Data time: 0.1321, Forward time: 0.9530, Backward time: 1.2245, DDP time: 0.0052, Total iter time: 2.4040 [2020-12-31 07:48:14,646][root][INFO] - ===> Epoch[221](1105/5): Loss -0.3188, Sem -0.3188, dir -0.9108, norm 0.4623 LR: 9.335e-02 Score 95.645 Data time: 0.1623, Forward time: 0.9530, Backward time: 1.2245, DDP time: 0.0052, Total iter time: 2.4155 [2020-12-31 07:48:26,567][root][INFO] - ===> Epoch[222](1110/5): Loss -0.3261, Sem -0.3261, dir -0.9127, norm 0.4602 LR: 9.332e-02 Score 95.698 Data time: 0.1459, Forward time: 0.9528, Backward time: 1.2242, DDP time: 0.0052, Total iter time: 2.2914 [2020-12-31 07:48:37,875][root][INFO] - ===> Epoch[223](1115/5): Loss -0.3231, Sem -0.3231, dir -0.9115, norm 0.4643 LR: 9.328e-02 Score 95.811 Data time: 0.0937, Forward time: 0.9526, Backward time: 1.2237, DDP time: 0.0052, Total iter time: 2.1732 [2020-12-31 07:48:49,874][root][INFO] - ===> Epoch[224](1120/5): Loss -0.3092, Sem -0.3092, dir -0.9103, norm 0.4696 LR: 9.325e-02 Score 95.627 Data time: 0.1587, Forward time: 0.9526, Backward time: 1.2233, DDP time: 0.0052, Total iter time: 2.3041 [2020-12-31 07:49:00,994][root][INFO] - ===> Epoch[225](1125/5): Loss -0.3119, Sem -0.3119, dir -0.9103, norm 0.4664 LR: 9.322e-02 Score 95.683 Data time: 0.0901, Forward time: 0.9520, Backward time: 1.2230, DDP time: 0.0052, Total iter time: 2.1364 [2020-12-31 07:49:12,893][root][INFO] - ===> Epoch[226](1130/5): Loss -0.3261, Sem -0.3261, dir -0.9104, norm 0.4570 LR: 9.319e-02 Score 95.749 Data time: 0.0926, Forward time: 0.9519, Backward time: 1.2229, DDP time: 0.0052, Total iter time: 2.2924 [2020-12-31 07:49:24,400][root][INFO] - ===> Epoch[227](1135/5): Loss -0.3077, Sem -0.3077, dir -0.9100, norm 0.4728 LR: 9.316e-02 Score 95.688 Data time: 0.0854, Forward time: 0.9515, Backward time: 1.2227, DDP time: 0.0052, Total iter time: 2.2194 [2020-12-31 07:49:36,988][root][INFO] - ===> Epoch[228](1140/5): Loss -0.3090, Sem -0.3090, dir -0.9121, norm 0.4788 LR: 9.313e-02 Score 95.860 Data time: 0.1692, Forward time: 0.9512, Backward time: 1.2231, DDP time: 0.0052, Total iter time: 2.4338 [2020-12-31 07:49:48,973][root][INFO] - ===> Epoch[229](1145/5): Loss -0.3200, Sem -0.3200, dir -0.9128, norm 0.4667 LR: 9.310e-02 Score 95.758 Data time: 0.1000, Forward time: 0.9512, Backward time: 1.2229, DDP time: 0.0052, Total iter time: 2.3189 [2020-12-31 07:50:02,285][root][INFO] - ===> Epoch[230](1150/5): Loss -0.2990, Sem -0.2990, dir -0.9119, norm 0.4803 LR: 9.307e-02 Score 95.553 Data time: 0.1089, Forward time: 0.9512, Backward time: 1.2239, DDP time: 0.0052, Total iter time: 2.5677 [2020-12-31 07:50:14,126][root][INFO] - ===> Epoch[231](1155/5): Loss -0.3582, Sem -0.3582, dir -0.9151, norm 0.4383 LR: 9.304e-02 Score 95.999 Data time: 0.1170, Forward time: 0.9512, Backward time: 1.2235, DDP time: 0.0052, Total iter time: 2.2656 [2020-12-31 07:50:27,197][root][INFO] - ===> Epoch[232](1160/5): Loss -0.3186, Sem -0.3186, dir -0.9143, norm 0.4764 LR: 9.301e-02 Score 96.013 Data time: 0.1581, Forward time: 0.9518, Backward time: 1.2235, DDP time: 0.0052, Total iter time: 2.5189 [2020-12-31 07:50:39,437][root][INFO] - ===> Epoch[233](1165/5): Loss -0.3354, Sem -0.3354, dir -0.9132, norm 0.4549 LR: 9.298e-02 Score 95.842 Data time: 0.1776, Forward time: 0.9519, Backward time: 1.2231, DDP time: 0.0052, Total iter time: 2.3622 [2020-12-31 07:50:50,449][root][INFO] - ===> Epoch[234](1170/5): Loss -0.3020, Sem -0.3020, dir -0.9105, norm 0.4829 LR: 9.295e-02 Score 95.781 Data time: 0.0903, Forward time: 0.9521, Backward time: 1.2222, DDP time: 0.0052, Total iter time: 2.1199 [2020-12-31 07:51:01,912][root][INFO] - ===> Epoch[235](1175/5): Loss -0.3163, Sem -0.3163, dir -0.9128, norm 0.4737 LR: 9.292e-02 Score 95.863 Data time: 0.1223, Forward time: 0.9522, Backward time: 1.2214, DDP time: 0.0052, Total iter time: 2.2025 [2020-12-31 07:51:14,048][root][INFO] - ===> Epoch[236](1180/5): Loss -0.3290, Sem -0.3290, dir -0.9134, norm 0.4661 LR: 9.289e-02 Score 96.104 Data time: 0.1520, Forward time: 0.9525, Backward time: 1.2211, DDP time: 0.0052, Total iter time: 2.3500 [2020-12-31 07:51:26,137][root][INFO] - ===> Epoch[237](1185/5): Loss -0.3400, Sem -0.3400, dir -0.9100, norm 0.4475 LR: 9.286e-02 Score 95.853 Data time: 0.0998, Forward time: 0.9523, Backward time: 1.2212, DDP time: 0.0052, Total iter time: 2.3383 [2020-12-31 07:51:37,607][root][INFO] - ===> Epoch[238](1190/5): Loss -0.3551, Sem -0.3551, dir -0.9145, norm 0.4431 LR: 9.283e-02 Score 96.110 Data time: 0.1556, Forward time: 0.9520, Backward time: 1.2208, DDP time: 0.0052, Total iter time: 2.2168 [2020-12-31 07:51:50,012][root][INFO] - ===> Epoch[239](1195/5): Loss -0.3506, Sem -0.3506, dir -0.9162, norm 0.4510 LR: 9.280e-02 Score 96.160 Data time: 0.1116, Forward time: 0.9518, Backward time: 1.2212, DDP time: 0.0052, Total iter time: 2.4030 [2020-12-31 07:52:02,267][root][INFO] - ===> Epoch[240](1200/5): Loss -0.3390, Sem -0.3390, dir -0.9139, norm 0.4570 LR: 9.277e-02 Score 96.023 Data time: 0.1636, Forward time: 0.9519, Backward time: 1.2209, DDP time: 0.0052, Total iter time: 2.3528 [2020-12-31 07:52:02,269][root][INFO] - ===> Start testing [2020-12-31 07:52:38,934][root][INFO] - 68/68: Data time: 0.0021, Iter time: 0.1097 Loss 0.022 (AVG: 0.406) Score 99.454 (AVG: 89.089) mIOU 66.555 mAP 83.769 mAcc 75.484 IOU: 58.639 0.000 70.827 73.981 92.270 86.032 27.207 69.483 98.146 67.795 81.535 83.954 55.348 mAP: 80.425 4.853 93.004 89.156 98.975 97.148 61.354 91.790 99.796 94.247 93.650 94.093 90.503 mAcc: 74.239 0.000 79.287 85.116 93.358 91.217 32.384 87.644 98.870 87.539 91.428 94.090 66.113 [2020-12-31 07:52:38,935][root][INFO] - Finished test. Elapsed time: 36.6644 [2020-12-31 07:52:41,937][root][INFO] - [2020-12-31 07:52:41,937][root][INFO] - ################################################################ [2020-12-31 07:52:41,937][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 07:52:41,937][root][INFO] - ################################################################ [2020-12-31 07:52:41,937][root][INFO] - clutter : nan nan nan [2020-12-31 07:52:41,937][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 07:52:41,937][root][INFO] - board : 0.669 0.766 0.844 [2020-12-31 07:52:41,937][root][INFO] - bookcase : 0.285 0.458 0.592 [2020-12-31 07:52:41,937][root][INFO] - ceiling : 0.811 0.829 0.863 [2020-12-31 07:52:41,937][root][INFO] - chair : 0.676 0.808 0.836 [2020-12-31 07:52:41,937][root][INFO] - column : 0.046 0.126 0.278 [2020-12-31 07:52:41,937][root][INFO] - door : 0.524 0.788 0.920 [2020-12-31 07:52:41,937][root][INFO] - floor : 0.954 0.987 0.987 [2020-12-31 07:52:41,938][root][INFO] - sofa : 0.480 0.727 0.794 [2020-12-31 07:52:41,938][root][INFO] - table : 0.160 0.347 0.596 [2020-12-31 07:52:41,938][root][INFO] - wall : 0.470 0.709 0.802 [2020-12-31 07:52:41,938][root][INFO] - window : 0.368 0.582 0.694 [2020-12-31 07:52:41,938][root][INFO] - ---------------------------------------------------------------- [2020-12-31 07:52:41,938][root][INFO] - average : 0.454 0.594 0.684 [2020-12-31 07:52:41,938][root][INFO] - [2020-12-31 07:52:42,155][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 07:52:42,795][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2020-12-31 07:52:42,796][root][INFO] - Current best mIoU: 66.555 at iter 1 [2020-12-31 07:52:43,361][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_mAP.pth [2020-12-31 07:52:43,362][root][INFO] - Current best mAP@0.5: 0.594 at iter 1200 [2020-12-31 07:52:43,927][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter1200.pth [2020-12-31 07:52:56,972][root][INFO] - ===> Epoch[241](1205/5): Loss -0.3263, Sem -0.3263, dir -0.9120, norm 0.4676 LR: 9.274e-02 Score 96.024 Data time: 0.1460, Forward time: 0.9528, Backward time: 1.2206, DDP time: 0.0052, Total iter time: 2.4995 [2020-12-31 07:53:09,771][root][INFO] - ===> Epoch[242](1210/5): Loss -0.3472, Sem -0.3472, dir -0.9159, norm 0.4570 LR: 9.271e-02 Score 96.249 Data time: 0.1614, Forward time: 0.9536, Backward time: 1.2202, DDP time: 0.0052, Total iter time: 2.4760 [2020-12-31 07:53:22,138][root][INFO] - ===> Epoch[243](1215/5): Loss -0.3609, Sem -0.3609, dir -0.9161, norm 0.4412 LR: 9.268e-02 Score 96.187 Data time: 0.1681, Forward time: 0.9542, Backward time: 1.2196, DDP time: 0.0052, Total iter time: 2.3811 [2020-12-31 07:53:33,744][root][INFO] - ===> Epoch[244](1220/5): Loss -0.3614, Sem -0.3614, dir -0.9160, norm 0.4433 LR: 9.265e-02 Score 96.312 Data time: 0.0767, Forward time: 0.9541, Backward time: 1.2193, DDP time: 0.0052, Total iter time: 2.2276 [2020-12-31 07:53:45,764][root][INFO] - ===> Epoch[245](1225/5): Loss -0.3521, Sem -0.3521, dir -0.9156, norm 0.4510 LR: 9.262e-02 Score 96.213 Data time: 0.1132, Forward time: 0.9542, Backward time: 1.2190, DDP time: 0.0052, Total iter time: 2.3137 [2020-12-31 07:53:57,131][root][INFO] - ===> Epoch[246](1230/5): Loss -0.3593, Sem -0.3593, dir -0.9168, norm 0.4455 LR: 9.259e-02 Score 96.294 Data time: 0.1446, Forward time: 0.9544, Backward time: 1.2181, DDP time: 0.0052, Total iter time: 2.1891 [2020-12-31 07:54:10,061][root][INFO] - ===> Epoch[247](1235/5): Loss -0.3523, Sem -0.3523, dir -0.9160, norm 0.4492 LR: 9.256e-02 Score 96.182 Data time: 0.1010, Forward time: 0.9545, Backward time: 1.2186, DDP time: 0.0052, Total iter time: 2.5032 [2020-12-31 07:54:22,152][root][INFO] - ===> Epoch[248](1240/5): Loss -0.3570, Sem -0.3570, dir -0.9162, norm 0.4469 LR: 9.253e-02 Score 96.260 Data time: 0.0743, Forward time: 0.9543, Backward time: 1.2188, DDP time: 0.0052, Total iter time: 2.3407 [2020-12-31 07:54:34,516][root][INFO] - ===> Epoch[249](1245/5): Loss -0.3387, Sem -0.3387, dir -0.9158, norm 0.4606 LR: 9.250e-02 Score 96.112 Data time: 0.1082, Forward time: 0.9545, Backward time: 1.2187, DDP time: 0.0052, Total iter time: 2.3803 [2020-12-31 07:54:46,458][root][INFO] - ===> Epoch[250](1250/5): Loss -0.3801, Sem -0.3801, dir -0.9202, norm 0.4271 LR: 9.247e-02 Score 96.292 Data time: 0.1134, Forward time: 0.9543, Backward time: 1.2186, DDP time: 0.0052, Total iter time: 2.3009 [2020-12-31 07:54:57,195][root][INFO] - ===> Epoch[251](1255/5): Loss -0.3864, Sem -0.3864, dir -0.9208, norm 0.4254 LR: 9.244e-02 Score 96.336 Data time: 0.1084, Forward time: 0.9541, Backward time: 1.2177, DDP time: 0.0052, Total iter time: 2.0508 [2020-12-31 07:55:10,273][root][INFO] - ===> Epoch[252](1260/5): Loss -0.3501, Sem -0.3501, dir -0.9156, norm 0.4513 LR: 9.241e-02 Score 96.167 Data time: 0.1164, Forward time: 0.9541, Backward time: 1.2182, DDP time: 0.0052, Total iter time: 2.5173 [2020-12-31 07:55:23,530][root][INFO] - ===> Epoch[253](1265/5): Loss -0.3515, Sem -0.3515, dir -0.9176, norm 0.4523 LR: 9.238e-02 Score 96.190 Data time: 0.1822, Forward time: 0.9547, Backward time: 1.2183, DDP time: 0.0052, Total iter time: 2.5607 [2020-12-31 07:55:36,281][root][INFO] - ===> Epoch[254](1270/5): Loss -0.3687, Sem -0.3687, dir -0.9190, norm 0.4449 LR: 9.235e-02 Score 96.478 Data time: 0.1040, Forward time: 0.9546, Backward time: 1.2187, DDP time: 0.0052, Total iter time: 2.4457 [2020-12-31 07:55:48,751][root][INFO] - ===> Epoch[255](1275/5): Loss -0.3650, Sem -0.3650, dir -0.9168, norm 0.4410 LR: 9.232e-02 Score 96.304 Data time: 0.2041, Forward time: 0.9549, Backward time: 1.2183, DDP time: 0.0052, Total iter time: 2.4028 [2020-12-31 07:56:01,029][root][INFO] - ===> Epoch[256](1280/5): Loss -0.3726, Sem -0.3726, dir -0.9208, norm 0.4377 LR: 9.229e-02 Score 96.324 Data time: 0.1528, Forward time: 0.9556, Backward time: 1.2176, DDP time: 0.0052, Total iter time: 2.3586 [2020-12-31 07:56:13,185][root][INFO] - ===> Epoch[257](1285/5): Loss -0.3622, Sem -0.3622, dir -0.9171, norm 0.4426 LR: 9.226e-02 Score 96.219 Data time: 0.1414, Forward time: 0.9565, Backward time: 1.2167, DDP time: 0.0052, Total iter time: 2.3380 [2020-12-31 07:56:25,738][root][INFO] - ===> Epoch[258](1290/5): Loss -0.3972, Sem -0.3972, dir -0.9223, norm 0.4178 LR: 9.223e-02 Score 96.406 Data time: 0.1475, Forward time: 0.9569, Backward time: 1.2164, DDP time: 0.0052, Total iter time: 2.4224 [2020-12-31 07:56:37,729][root][INFO] - ===> Epoch[259](1295/5): Loss -0.3792, Sem -0.3792, dir -0.9204, norm 0.4334 LR: 9.220e-02 Score 96.427 Data time: 0.1056, Forward time: 0.9571, Backward time: 1.2161, DDP time: 0.0052, Total iter time: 2.3108 [2020-12-31 07:56:49,290][root][INFO] - ===> Epoch[260](1300/5): Loss -0.3672, Sem -0.3672, dir -0.9190, norm 0.4394 LR: 9.217e-02 Score 96.209 Data time: 0.0979, Forward time: 0.9570, Backward time: 1.2158, DDP time: 0.0052, Total iter time: 2.2250 [2020-12-31 07:57:01,743][root][INFO] - ===> Epoch[261](1305/5): Loss -0.3741, Sem -0.3741, dir -0.9204, norm 0.4387 LR: 9.214e-02 Score 96.422 Data time: 0.1067, Forward time: 0.9574, Backward time: 1.2156, DDP time: 0.0052, Total iter time: 2.3908 [2020-12-31 07:57:14,624][root][INFO] - ===> Epoch[262](1310/5): Loss -0.3685, Sem -0.3685, dir -0.9211, norm 0.4334 LR: 9.211e-02 Score 96.008 Data time: 0.1021, Forward time: 0.9574, Backward time: 1.2161, DDP time: 0.0052, Total iter time: 2.4922 [2020-12-31 07:57:26,223][root][INFO] - ===> Epoch[263](1315/5): Loss -0.3553, Sem -0.3553, dir -0.9182, norm 0.4422 LR: 9.207e-02 Score 96.002 Data time: 0.1332, Forward time: 0.9575, Backward time: 1.2155, DDP time: 0.0052, Total iter time: 2.2230 [2020-12-31 07:57:37,817][root][INFO] - ===> Epoch[264](1320/5): Loss -0.3733, Sem -0.3733, dir -0.9215, norm 0.4301 LR: 9.204e-02 Score 96.050 Data time: 0.1000, Forward time: 0.9574, Backward time: 1.2151, DDP time: 0.0052, Total iter time: 2.2328 [2020-12-31 07:57:49,664][root][INFO] - ===> Epoch[265](1325/5): Loss -0.3611, Sem -0.3611, dir -0.9191, norm 0.4439 LR: 9.201e-02 Score 96.166 Data time: 0.0937, Forward time: 0.9574, Backward time: 1.2149, DDP time: 0.0052, Total iter time: 2.2756 [2020-12-31 07:58:01,724][root][INFO] - ===> Epoch[266](1330/5): Loss -0.3684, Sem -0.3684, dir -0.9163, norm 0.4299 LR: 9.198e-02 Score 96.052 Data time: 0.1014, Forward time: 0.9576, Backward time: 1.2147, DDP time: 0.0052, Total iter time: 2.3264 [2020-12-31 07:58:13,757][root][INFO] - ===> Epoch[267](1335/5): Loss -0.3570, Sem -0.3570, dir -0.9191, norm 0.4493 LR: 9.195e-02 Score 96.199 Data time: 0.1412, Forward time: 0.9579, Backward time: 1.2142, DDP time: 0.0052, Total iter time: 2.3175 [2020-12-31 07:58:24,942][root][INFO] - ===> Epoch[268](1340/5): Loss -0.3557, Sem -0.3557, dir -0.9174, norm 0.4467 LR: 9.192e-02 Score 96.164 Data time: 0.1496, Forward time: 0.9580, Backward time: 1.2132, DDP time: 0.0052, Total iter time: 2.1351 [2020-12-31 07:58:36,820][root][INFO] - ===> Epoch[269](1345/5): Loss -0.3919, Sem -0.3919, dir -0.9224, norm 0.4244 LR: 9.189e-02 Score 96.445 Data time: 0.0893, Forward time: 0.9579, Backward time: 1.2132, DDP time: 0.0052, Total iter time: 2.2793 [2020-12-31 07:58:49,294][root][INFO] - ===> Epoch[270](1350/5): Loss -0.3559, Sem -0.3559, dir -0.9190, norm 0.4501 LR: 9.186e-02 Score 96.176 Data time: 0.1443, Forward time: 0.9577, Backward time: 1.2133, DDP time: 0.0052, Total iter time: 2.3986 [2020-12-31 07:59:00,227][root][INFO] - ===> Epoch[271](1355/5): Loss -0.3938, Sem -0.3938, dir -0.9238, norm 0.4201 LR: 9.183e-02 Score 96.317 Data time: 0.0804, Forward time: 0.9575, Backward time: 1.2128, DDP time: 0.0052, Total iter time: 2.1265 [2020-12-31 07:59:11,689][root][INFO] - ===> Epoch[272](1360/5): Loss -0.3682, Sem -0.3682, dir -0.9188, norm 0.4402 LR: 9.180e-02 Score 96.279 Data time: 0.1196, Forward time: 0.9578, Backward time: 1.2120, DDP time: 0.0052, Total iter time: 2.1963 [2020-12-31 07:59:24,184][root][INFO] - ===> Epoch[273](1365/5): Loss -0.3678, Sem -0.3678, dir -0.9184, norm 0.4404 LR: 9.177e-02 Score 96.303 Data time: 0.1475, Forward time: 0.9583, Backward time: 1.2117, DDP time: 0.0052, Total iter time: 2.4080 [2020-12-31 07:59:36,632][root][INFO] - ===> Epoch[274](1370/5): Loss -0.3505, Sem -0.3505, dir -0.9178, norm 0.4618 LR: 9.174e-02 Score 96.473 Data time: 0.0887, Forward time: 0.9580, Backward time: 1.2123, DDP time: 0.0052, Total iter time: 2.4126 [2020-12-31 07:59:49,448][root][INFO] - ===> Epoch[275](1375/5): Loss -0.3825, Sem -0.3825, dir -0.9219, norm 0.4348 LR: 9.171e-02 Score 96.534 Data time: 0.1071, Forward time: 0.9584, Backward time: 1.2125, DDP time: 0.0052, Total iter time: 2.4877 [2020-12-31 08:00:01,862][root][INFO] - ===> Epoch[276](1380/5): Loss -0.3727, Sem -0.3727, dir -0.9198, norm 0.4458 LR: 9.168e-02 Score 96.564 Data time: 0.1414, Forward time: 0.9586, Backward time: 1.2123, DDP time: 0.0052, Total iter time: 2.3892 [2020-12-31 08:00:14,094][root][INFO] - ===> Epoch[277](1385/5): Loss -0.3890, Sem -0.3890, dir -0.9222, norm 0.4270 LR: 9.165e-02 Score 96.456 Data time: 0.0893, Forward time: 0.9586, Backward time: 1.2124, DDP time: 0.0052, Total iter time: 2.3605 [2020-12-31 08:00:25,824][root][INFO] - ===> Epoch[278](1390/5): Loss -0.3733, Sem -0.3733, dir -0.9213, norm 0.4361 LR: 9.162e-02 Score 96.258 Data time: 0.0813, Forward time: 0.9582, Backward time: 1.2125, DDP time: 0.0052, Total iter time: 2.2470 [2020-12-31 08:00:37,575][root][INFO] - ===> Epoch[279](1395/5): Loss -0.3935, Sem -0.3935, dir -0.9238, norm 0.4280 LR: 9.159e-02 Score 96.538 Data time: 0.1358, Forward time: 0.9584, Backward time: 1.2120, DDP time: 0.0052, Total iter time: 2.2583 [2020-12-31 08:00:49,752][root][INFO] - ===> Epoch[280](1400/5): Loss -0.3784, Sem -0.3784, dir -0.9203, norm 0.4315 LR: 9.156e-02 Score 96.338 Data time: 0.1284, Forward time: 0.9585, Backward time: 1.2118, DDP time: 0.0052, Total iter time: 2.3455 [2020-12-31 08:00:49,754][root][INFO] - ===> Start testing [2020-12-31 08:01:27,137][root][INFO] - 68/68: Data time: 0.0025, Iter time: 0.1217 Loss 0.281 (AVG: 0.426) Score 91.789 (AVG: 88.808) mIOU 66.455 mAP 82.968 mAcc 74.519 IOU: 57.971 0.000 70.855 72.072 93.206 86.783 32.258 71.619 98.139 65.711 79.630 83.253 52.418 mAP: 80.908 3.086 93.917 87.681 98.802 97.490 60.651 91.359 99.796 90.422 92.788 93.142 88.540 mAcc: 78.831 0.000 83.917 79.283 94.758 94.125 42.917 85.387 99.353 71.675 87.097 93.629 57.777 [2020-12-31 08:01:27,138][root][INFO] - Finished test. Elapsed time: 37.3819 [2020-12-31 08:01:30,301][root][INFO] - [2020-12-31 08:01:30,301][root][INFO] - ################################################################ [2020-12-31 08:01:30,301][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 08:01:30,302][root][INFO] - ################################################################ [2020-12-31 08:01:30,302][root][INFO] - clutter : nan nan nan [2020-12-31 08:01:30,302][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 08:01:30,302][root][INFO] - board : 0.695 0.856 0.916 [2020-12-31 08:01:30,302][root][INFO] - bookcase : 0.270 0.400 0.585 [2020-12-31 08:01:30,302][root][INFO] - ceiling : 0.740 0.764 0.817 [2020-12-31 08:01:30,302][root][INFO] - chair : 0.708 0.827 0.864 [2020-12-31 08:01:30,302][root][INFO] - column : 0.144 0.304 0.448 [2020-12-31 08:01:30,302][root][INFO] - door : 0.501 0.761 0.898 [2020-12-31 08:01:30,302][root][INFO] - floor : 0.965 0.992 0.992 [2020-12-31 08:01:30,302][root][INFO] - sofa : 0.256 0.349 0.772 [2020-12-31 08:01:30,302][root][INFO] - table : 0.134 0.267 0.583 [2020-12-31 08:01:30,302][root][INFO] - wall : 0.463 0.674 0.776 [2020-12-31 08:01:30,302][root][INFO] - window : 0.367 0.671 0.745 [2020-12-31 08:01:30,302][root][INFO] - ---------------------------------------------------------------- [2020-12-31 08:01:30,302][root][INFO] - average : 0.437 0.572 0.700 [2020-12-31 08:01:30,302][root][INFO] - [2020-12-31 08:01:30,468][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 08:01:31,073][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter1400.pth [2020-12-31 08:01:42,212][root][INFO] - ===> Epoch[281](1405/5): Loss -0.3792, Sem -0.3792, dir -0.9212, norm 0.4339 LR: 9.153e-02 Score 96.385 Data time: 0.0832, Forward time: 0.9579, Backward time: 1.2116, DDP time: 0.0052, Total iter time: 2.1237 [2020-12-31 08:01:54,015][root][INFO] - ===> Epoch[282](1410/5): Loss -0.3858, Sem -0.3858, dir -0.9210, norm 0.4296 LR: 9.150e-02 Score 96.483 Data time: 0.1580, Forward time: 0.9575, Backward time: 1.2116, DDP time: 0.0053, Total iter time: 2.2678 [2020-12-31 08:02:05,556][root][INFO] - ===> Epoch[283](1415/5): Loss -0.3845, Sem -0.3845, dir -0.9222, norm 0.4299 LR: 9.147e-02 Score 96.419 Data time: 0.1147, Forward time: 0.9572, Backward time: 1.2114, DDP time: 0.0053, Total iter time: 2.2094 [2020-12-31 08:02:17,555][root][INFO] - ===> Epoch[284](1420/5): Loss -0.3754, Sem -0.3754, dir -0.9205, norm 0.4425 LR: 9.144e-02 Score 96.573 Data time: 0.1403, Forward time: 0.9573, Backward time: 1.2111, DDP time: 0.0053, Total iter time: 2.3064 [2020-12-31 08:02:28,785][root][INFO] - ===> Epoch[285](1425/5): Loss -0.3901, Sem -0.3901, dir -0.9202, norm 0.4270 LR: 9.141e-02 Score 96.524 Data time: 0.1551, Forward time: 0.9570, Backward time: 1.2107, DDP time: 0.0052, Total iter time: 2.1509 [2020-12-31 08:02:41,035][root][INFO] - ===> Epoch[286](1430/5): Loss -0.4046, Sem -0.4046, dir -0.9235, norm 0.4141 LR: 9.138e-02 Score 96.522 Data time: 0.1174, Forward time: 0.9570, Backward time: 1.2106, DDP time: 0.0052, Total iter time: 2.3556 [2020-12-31 08:02:52,695][root][INFO] - ===> Epoch[287](1435/5): Loss -0.3778, Sem -0.3778, dir -0.9233, norm 0.4445 LR: 9.135e-02 Score 96.642 Data time: 0.1069, Forward time: 0.9572, Backward time: 1.2103, DDP time: 0.0052, Total iter time: 2.2547 [2020-12-31 08:03:04,806][root][INFO] - ===> Epoch[288](1440/5): Loss -0.3797, Sem -0.3797, dir -0.9214, norm 0.4394 LR: 9.132e-02 Score 96.578 Data time: 0.1454, Forward time: 0.9569, Backward time: 1.2104, DDP time: 0.0052, Total iter time: 2.3310 [2020-12-31 08:03:17,124][root][INFO] - ===> Epoch[289](1445/5): Loss -0.3987, Sem -0.3987, dir -0.9246, norm 0.4276 LR: 9.129e-02 Score 96.713 Data time: 0.1482, Forward time: 0.9569, Backward time: 1.2104, DDP time: 0.0052, Total iter time: 2.3722 [2020-12-31 08:03:28,906][root][INFO] - ===> Epoch[290](1450/5): Loss -0.3978, Sem -0.3978, dir -0.9228, norm 0.4229 LR: 9.126e-02 Score 96.550 Data time: 0.1579, Forward time: 0.9571, Backward time: 1.2098, DDP time: 0.0052, Total iter time: 2.2500 [2020-12-31 08:03:40,332][root][INFO] - ===> Epoch[291](1455/5): Loss -0.4026, Sem -0.4026, dir -0.9236, norm 0.4220 LR: 9.123e-02 Score 96.693 Data time: 0.1704, Forward time: 0.9569, Backward time: 1.2094, DDP time: 0.0052, Total iter time: 2.1947 [2020-12-31 08:03:53,304][root][INFO] - ===> Epoch[292](1460/5): Loss -0.4072, Sem -0.4072, dir -0.9235, norm 0.4156 LR: 9.120e-02 Score 96.635 Data time: 0.2250, Forward time: 0.9572, Backward time: 1.2093, DDP time: 0.0052, Total iter time: 2.5030 [2020-12-31 08:04:04,806][root][INFO] - ===> Epoch[293](1465/5): Loss -0.4040, Sem -0.4040, dir -0.9252, norm 0.4173 LR: 9.117e-02 Score 96.540 Data time: 0.1639, Forward time: 0.9570, Backward time: 1.2089, DDP time: 0.0052, Total iter time: 2.2151 [2020-12-31 08:04:16,926][root][INFO] - ===> Epoch[294](1470/5): Loss -0.3982, Sem -0.3982, dir -0.9236, norm 0.4208 LR: 9.114e-02 Score 96.477 Data time: 0.1693, Forward time: 0.9571, Backward time: 1.2087, DDP time: 0.0052, Total iter time: 2.3506 [2020-12-31 08:04:29,274][root][INFO] - ===> Epoch[295](1475/5): Loss -0.4066, Sem -0.4066, dir -0.9233, norm 0.4142 LR: 9.111e-02 Score 96.554 Data time: 0.1420, Forward time: 0.9573, Backward time: 1.2085, DDP time: 0.0052, Total iter time: 2.3724 [2020-12-31 08:04:40,762][root][INFO] - ===> Epoch[296](1480/5): Loss -0.4163, Sem -0.4163, dir -0.9249, norm 0.4089 LR: 9.108e-02 Score 96.639 Data time: 0.0954, Forward time: 0.9571, Backward time: 1.2084, DDP time: 0.0052, Total iter time: 2.2300 [2020-12-31 08:04:53,208][root][INFO] - ===> Epoch[297](1485/5): Loss -0.4009, Sem -0.4009, dir -0.9243, norm 0.4254 LR: 9.104e-02 Score 96.734 Data time: 0.1091, Forward time: 0.9573, Backward time: 1.2085, DDP time: 0.0052, Total iter time: 2.4048 [2020-12-31 08:05:04,856][root][INFO] - ===> Epoch[298](1490/5): Loss -0.4167, Sem -0.4167, dir -0.9243, norm 0.4088 LR: 9.101e-02 Score 96.703 Data time: 0.1307, Forward time: 0.9570, Backward time: 1.2083, DDP time: 0.0052, Total iter time: 2.2439 [2020-12-31 08:05:17,494][root][INFO] - ===> Epoch[299](1495/5): Loss -0.4010, Sem -0.4010, dir -0.9237, norm 0.4187 LR: 9.098e-02 Score 96.535 Data time: 0.0808, Forward time: 0.9565, Backward time: 1.2091, DDP time: 0.0052, Total iter time: 2.4278 [2020-12-31 08:05:30,025][root][INFO] - ===> Epoch[300](1500/5): Loss -0.4212, Sem -0.4212, dir -0.9269, norm 0.4080 LR: 9.095e-02 Score 96.723 Data time: 0.0930, Forward time: 0.9562, Backward time: 1.2096, DDP time: 0.0052, Total iter time: 2.4109 [2020-12-31 08:05:42,686][root][INFO] - ===> Epoch[301](1505/5): Loss -0.4178, Sem -0.4178, dir -0.9263, norm 0.4108 LR: 9.092e-02 Score 96.763 Data time: 0.1107, Forward time: 0.9559, Backward time: 1.2102, DDP time: 0.0052, Total iter time: 2.4380 [2020-12-31 08:05:55,325][root][INFO] - ===> Epoch[302](1510/5): Loss -0.4079, Sem -0.4079, dir -0.9245, norm 0.4144 LR: 9.089e-02 Score 96.551 Data time: 0.1549, Forward time: 0.9559, Backward time: 1.2104, DDP time: 0.0052, Total iter time: 2.4298 [2020-12-31 08:06:07,880][root][INFO] - ===> Epoch[303](1515/5): Loss -0.4135, Sem -0.4135, dir -0.9258, norm 0.4179 LR: 9.086e-02 Score 96.850 Data time: 0.1057, Forward time: 0.9559, Backward time: 1.2106, DDP time: 0.0052, Total iter time: 2.4229 [2020-12-31 08:06:20,022][root][INFO] - ===> Epoch[304](1520/5): Loss -0.4133, Sem -0.4133, dir -0.9267, norm 0.4147 LR: 9.083e-02 Score 96.662 Data time: 0.1301, Forward time: 0.9561, Backward time: 1.2104, DDP time: 0.0052, Total iter time: 2.3371 [2020-12-31 08:06:31,765][root][INFO] - ===> Epoch[305](1525/5): Loss -0.4123, Sem -0.4123, dir -0.9255, norm 0.4134 LR: 9.080e-02 Score 96.698 Data time: 0.0908, Forward time: 0.9557, Backward time: 1.2106, DDP time: 0.0052, Total iter time: 2.2653 [2020-12-31 08:06:43,265][root][INFO] - ===> Epoch[306](1530/5): Loss -0.4178, Sem -0.4178, dir -0.9256, norm 0.4073 LR: 9.077e-02 Score 96.638 Data time: 0.1016, Forward time: 0.9556, Backward time: 1.2103, DDP time: 0.0052, Total iter time: 2.2099 [2020-12-31 08:06:54,702][root][INFO] - ===> Epoch[307](1535/5): Loss -0.4028, Sem -0.4028, dir -0.9241, norm 0.4151 LR: 9.074e-02 Score 96.446 Data time: 0.1163, Forward time: 0.9557, Backward time: 1.2098, DDP time: 0.0052, Total iter time: 2.1962 [2020-12-31 08:07:06,206][root][INFO] - ===> Epoch[308](1540/5): Loss -0.4079, Sem -0.4079, dir -0.9250, norm 0.4226 LR: 9.071e-02 Score 96.861 Data time: 0.0811, Forward time: 0.9553, Backward time: 1.2098, DDP time: 0.0052, Total iter time: 2.2058 [2020-12-31 08:07:19,580][root][INFO] - ===> Epoch[309](1545/5): Loss -0.4009, Sem -0.4009, dir -0.9249, norm 0.4283 LR: 9.068e-02 Score 96.805 Data time: 0.2239, Forward time: 0.9558, Backward time: 1.2098, DDP time: 0.0052, Total iter time: 2.5800 [2020-12-31 08:07:30,775][root][INFO] - ===> Epoch[310](1550/5): Loss -0.4259, Sem -0.4259, dir -0.9270, norm 0.4009 LR: 9.065e-02 Score 96.663 Data time: 0.1050, Forward time: 0.9559, Backward time: 1.2092, DDP time: 0.0052, Total iter time: 2.1725 [2020-12-31 08:07:43,494][root][INFO] - ===> Epoch[311](1555/5): Loss -0.4136, Sem -0.4136, dir -0.9268, norm 0.4126 LR: 9.062e-02 Score 96.673 Data time: 0.0850, Forward time: 0.9557, Backward time: 1.2098, DDP time: 0.0052, Total iter time: 2.4544 [2020-12-31 08:07:54,965][root][INFO] - ===> Epoch[312](1560/5): Loss -0.4134, Sem -0.4134, dir -0.9254, norm 0.4119 LR: 9.059e-02 Score 96.674 Data time: 0.1223, Forward time: 0.9557, Backward time: 1.2095, DDP time: 0.0052, Total iter time: 2.2054 [2020-12-31 08:08:06,978][root][INFO] - ===> Epoch[313](1565/5): Loss -0.4014, Sem -0.4014, dir -0.9262, norm 0.4234 LR: 9.056e-02 Score 96.637 Data time: 0.0954, Forward time: 0.9555, Backward time: 1.2095, DDP time: 0.0052, Total iter time: 2.3086 [2020-12-31 08:08:18,965][root][INFO] - ===> Epoch[314](1570/5): Loss -0.4285, Sem -0.4285, dir -0.9250, norm 0.4004 LR: 9.053e-02 Score 96.747 Data time: 0.1833, Forward time: 0.9554, Backward time: 1.2093, DDP time: 0.0052, Total iter time: 2.3011 [2020-12-31 08:08:30,991][root][INFO] - ===> Epoch[315](1575/5): Loss -0.4182, Sem -0.4182, dir -0.9276, norm 0.4081 LR: 9.050e-02 Score 96.574 Data time: 0.1465, Forward time: 0.9552, Backward time: 1.2093, DDP time: 0.0052, Total iter time: 2.3257 [2020-12-31 08:08:42,244][root][INFO] - ===> Epoch[316](1580/5): Loss -0.4298, Sem -0.4298, dir -0.9284, norm 0.4018 LR: 9.047e-02 Score 96.771 Data time: 0.1134, Forward time: 0.9550, Backward time: 1.2089, DDP time: 0.0052, Total iter time: 2.1588 [2020-12-31 08:08:54,028][root][INFO] - ===> Epoch[317](1585/5): Loss -0.4182, Sem -0.4182, dir -0.9240, norm 0.4086 LR: 9.044e-02 Score 96.728 Data time: 0.0899, Forward time: 0.9548, Backward time: 1.2089, DDP time: 0.0052, Total iter time: 2.2601 [2020-12-31 08:09:06,201][root][INFO] - ===> Epoch[318](1590/5): Loss -0.4275, Sem -0.4275, dir -0.9290, norm 0.4140 LR: 9.041e-02 Score 97.101 Data time: 0.1005, Forward time: 0.9548, Backward time: 1.2090, DDP time: 0.0052, Total iter time: 2.3373 [2020-12-31 08:09:18,421][root][INFO] - ===> Epoch[319](1595/5): Loss -0.4451, Sem -0.4451, dir -0.9291, norm 0.3915 LR: 9.038e-02 Score 96.926 Data time: 0.1785, Forward time: 0.9552, Backward time: 1.2085, DDP time: 0.0052, Total iter time: 2.3494 [2020-12-31 08:09:30,414][root][INFO] - ===> Epoch[320](1600/5): Loss -0.4396, Sem -0.4396, dir -0.9292, norm 0.3996 LR: 9.035e-02 Score 97.018 Data time: 0.1188, Forward time: 0.9553, Backward time: 1.2083, DDP time: 0.0052, Total iter time: 2.3130 [2020-12-31 08:09:30,416][root][INFO] - ===> Start testing [2020-12-31 08:10:06,396][root][INFO] - 68/68: Data time: 0.0023, Iter time: 0.1186 Loss 0.167 (AVG: 0.446) Score 94.208 (AVG: 88.700) mIOU 66.732 mAP 83.701 mAcc 75.423 IOU: 58.213 0.000 72.544 72.071 93.072 86.760 34.292 68.397 98.346 68.946 78.806 82.950 53.116 mAP: 80.740 3.297 93.128 86.970 98.797 97.451 69.537 90.995 99.803 92.567 92.525 93.592 88.709 mAcc: 78.418 0.000 84.376 80.510 94.725 91.837 46.237 80.569 99.062 85.142 86.881 93.249 59.500 [2020-12-31 08:10:06,397][root][INFO] - Finished test. Elapsed time: 35.9787 [2020-12-31 08:10:09,947][root][INFO] - [2020-12-31 08:10:09,947][root][INFO] - ################################################################ [2020-12-31 08:10:09,947][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 08:10:09,947][root][INFO] - ################################################################ [2020-12-31 08:10:09,948][root][INFO] - clutter : nan nan nan [2020-12-31 08:10:09,948][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 08:10:09,948][root][INFO] - board : 0.714 0.827 0.881 [2020-12-31 08:10:09,948][root][INFO] - bookcase : 0.255 0.401 0.541 [2020-12-31 08:10:09,948][root][INFO] - ceiling : 0.729 0.754 0.787 [2020-12-31 08:10:09,948][root][INFO] - chair : 0.725 0.844 0.867 [2020-12-31 08:10:09,948][root][INFO] - column : 0.149 0.270 0.440 [2020-12-31 08:10:09,948][root][INFO] - door : 0.413 0.693 0.857 [2020-12-31 08:10:09,948][root][INFO] - floor : 0.986 0.997 0.997 [2020-12-31 08:10:09,948][root][INFO] - sofa : 0.455 0.658 0.801 [2020-12-31 08:10:09,948][root][INFO] - table : 0.153 0.319 0.618 [2020-12-31 08:10:09,948][root][INFO] - wall : 0.426 0.663 0.757 [2020-12-31 08:10:09,948][root][INFO] - window : 0.369 0.598 0.713 [2020-12-31 08:10:09,948][root][INFO] - ---------------------------------------------------------------- [2020-12-31 08:10:09,948][root][INFO] - average : 0.448 0.585 0.688 [2020-12-31 08:10:09,948][root][INFO] - [2020-12-31 08:10:10,120][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 08:10:10,700][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2020-12-31 08:10:10,701][root][INFO] - Current best mIoU: 66.732 at iter 1 [2020-12-31 08:10:11,238][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter1600.pth [2020-12-31 08:10:22,870][root][INFO] - ===> Epoch[321](1605/5): Loss -0.4237, Sem -0.4237, dir -0.9295, norm 0.4111 LR: 9.032e-02 Score 96.862 Data time: 0.1082, Forward time: 0.9552, Backward time: 1.2080, DDP time: 0.0052, Total iter time: 2.2243 [2020-12-31 08:10:34,580][root][INFO] - ===> Epoch[322](1610/5): Loss -0.4373, Sem -0.4373, dir -0.9302, norm 0.3946 LR: 9.029e-02 Score 96.656 Data time: 0.0979, Forward time: 0.9550, Backward time: 1.2080, DDP time: 0.0052, Total iter time: 2.2516 [2020-12-31 08:10:46,527][root][INFO] - ===> Epoch[323](1615/5): Loss -0.4426, Sem -0.4426, dir -0.9308, norm 0.3953 LR: 9.026e-02 Score 96.877 Data time: 0.0947, Forward time: 0.9550, Backward time: 1.2079, DDP time: 0.0052, Total iter time: 2.3034 [2020-12-31 08:10:59,460][root][INFO] - ===> Epoch[324](1620/5): Loss -0.4207, Sem -0.4207, dir -0.9274, norm 0.4119 LR: 9.023e-02 Score 96.846 Data time: 0.2132, Forward time: 0.9558, Backward time: 1.2075, DDP time: 0.0052, Total iter time: 2.4918 [2020-12-31 08:11:12,019][root][INFO] - ===> Epoch[325](1625/5): Loss -0.4120, Sem -0.4120, dir -0.9271, norm 0.4193 LR: 9.020e-02 Score 96.856 Data time: 0.0828, Forward time: 0.9557, Backward time: 1.2078, DDP time: 0.0052, Total iter time: 2.4159 [2020-12-31 08:11:23,994][root][INFO] - ===> Epoch[326](1630/5): Loss -0.4298, Sem -0.4298, dir -0.9271, norm 0.4067 LR: 9.017e-02 Score 96.995 Data time: 0.1136, Forward time: 0.9554, Backward time: 1.2080, DDP time: 0.0052, Total iter time: 2.3115 [2020-12-31 08:11:36,692][root][INFO] - ===> Epoch[327](1635/5): Loss -0.4171, Sem -0.4171, dir -0.9301, norm 0.4209 LR: 9.013e-02 Score 96.940 Data time: 0.1337, Forward time: 0.9559, Backward time: 1.2078, DDP time: 0.0052, Total iter time: 2.4511 [2020-12-31 08:11:48,391][root][INFO] - ===> Epoch[328](1640/5): Loss -0.4265, Sem -0.4265, dir -0.9283, norm 0.4105 LR: 9.010e-02 Score 96.975 Data time: 0.1372, Forward time: 0.9561, Backward time: 1.2073, DDP time: 0.0052, Total iter time: 2.2522 [2020-12-31 08:12:00,626][root][INFO] - ===> Epoch[329](1645/5): Loss -0.4329, Sem -0.4329, dir -0.9284, norm 0.4051 LR: 9.007e-02 Score 96.987 Data time: 0.1099, Forward time: 0.9561, Backward time: 1.2074, DDP time: 0.0052, Total iter time: 2.3655 [2020-12-31 08:12:12,834][root][INFO] - ===> Epoch[330](1650/5): Loss -0.4435, Sem -0.4435, dir -0.9294, norm 0.3944 LR: 9.004e-02 Score 96.924 Data time: 0.1362, Forward time: 0.9564, Backward time: 1.2070, DDP time: 0.0052, Total iter time: 2.3508 [2020-12-31 08:12:25,438][root][INFO] - ===> Epoch[331](1655/5): Loss -0.4316, Sem -0.4316, dir -0.9301, norm 0.4087 LR: 9.001e-02 Score 97.059 Data time: 0.1169, Forward time: 0.9566, Backward time: 1.2072, DDP time: 0.0052, Total iter time: 2.4341 [2020-12-31 08:12:37,431][root][INFO] - ===> Epoch[332](1660/5): Loss -0.4414, Sem -0.4414, dir -0.9292, norm 0.3971 LR: 8.998e-02 Score 96.970 Data time: 0.0869, Forward time: 0.9565, Backward time: 1.2072, DDP time: 0.0052, Total iter time: 2.3156 [2020-12-31 08:12:49,234][root][INFO] - ===> Epoch[333](1665/5): Loss -0.4564, Sem -0.4564, dir -0.9306, norm 0.3840 LR: 8.995e-02 Score 97.036 Data time: 0.1218, Forward time: 0.9566, Backward time: 1.2069, DDP time: 0.0052, Total iter time: 2.2715 [2020-12-31 08:13:00,976][root][INFO] - ===> Epoch[334](1670/5): Loss -0.4505, Sem -0.4505, dir -0.9303, norm 0.3932 LR: 8.992e-02 Score 97.112 Data time: 0.1167, Forward time: 0.9566, Backward time: 1.2067, DDP time: 0.0052, Total iter time: 2.2507 [2020-12-31 08:13:12,689][root][INFO] - ===> Epoch[335](1675/5): Loss -0.4226, Sem -0.4226, dir -0.9290, norm 0.4135 LR: 8.989e-02 Score 96.934 Data time: 0.1362, Forward time: 0.9567, Backward time: 1.2062, DDP time: 0.0052, Total iter time: 2.2520 [2020-12-31 08:13:24,844][root][INFO] - ===> Epoch[336](1680/5): Loss -0.4317, Sem -0.4317, dir -0.9285, norm 0.4003 LR: 8.986e-02 Score 96.842 Data time: 0.1696, Forward time: 0.9572, Backward time: 1.2056, DDP time: 0.0052, Total iter time: 2.3400 [2020-12-31 08:13:37,131][root][INFO] - ===> Epoch[337](1685/5): Loss -0.4407, Sem -0.4407, dir -0.9298, norm 0.3933 LR: 8.983e-02 Score 96.802 Data time: 0.1012, Forward time: 0.9572, Backward time: 1.2057, DDP time: 0.0052, Total iter time: 2.3569 [2020-12-31 08:13:49,166][root][INFO] - ===> Epoch[338](1690/5): Loss -0.4276, Sem -0.4276, dir -0.9291, norm 0.4095 LR: 8.980e-02 Score 96.954 Data time: 0.1100, Forward time: 0.9571, Backward time: 1.2057, DDP time: 0.0052, Total iter time: 2.3097 [2020-12-31 08:14:00,976][root][INFO] - ===> Epoch[339](1695/5): Loss -0.4454, Sem -0.4454, dir -0.9302, norm 0.3977 LR: 8.977e-02 Score 97.081 Data time: 0.0963, Forward time: 0.9572, Backward time: 1.2055, DDP time: 0.0052, Total iter time: 2.2726 [2020-12-31 08:14:13,716][root][INFO] - ===> Epoch[340](1700/5): Loss -0.4443, Sem -0.4443, dir -0.9301, norm 0.3992 LR: 8.974e-02 Score 97.104 Data time: 0.1251, Forward time: 0.9577, Backward time: 1.2053, DDP time: 0.0052, Total iter time: 2.4493 [2020-12-31 08:14:25,588][root][INFO] - ===> Epoch[341](1705/5): Loss -0.4452, Sem -0.4452, dir -0.9309, norm 0.3931 LR: 8.971e-02 Score 96.901 Data time: 0.1358, Forward time: 0.9575, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.2779 [2020-12-31 08:14:38,119][root][INFO] - ===> Epoch[342](1710/5): Loss -0.4322, Sem -0.4322, dir -0.9300, norm 0.4112 LR: 8.968e-02 Score 97.117 Data time: 0.0903, Forward time: 0.9576, Backward time: 1.2054, DDP time: 0.0052, Total iter time: 2.4129 [2020-12-31 08:14:50,427][root][INFO] - ===> Epoch[343](1715/5): Loss -0.4303, Sem -0.4303, dir -0.9313, norm 0.4103 LR: 8.965e-02 Score 96.981 Data time: 0.1096, Forward time: 0.9578, Backward time: 1.2053, DDP time: 0.0052, Total iter time: 2.3650 [2020-12-31 08:15:01,492][root][INFO] - ===> Epoch[344](1720/5): Loss -0.4534, Sem -0.4534, dir -0.9298, norm 0.3891 LR: 8.962e-02 Score 97.094 Data time: 0.0827, Forward time: 0.9579, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.1201 [2020-12-31 08:15:13,549][root][INFO] - ===> Epoch[345](1725/5): Loss -0.4474, Sem -0.4474, dir -0.9301, norm 0.3950 LR: 8.959e-02 Score 97.079 Data time: 0.1170, Forward time: 0.9582, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.3190 [2020-12-31 08:15:25,240][root][INFO] - ===> Epoch[346](1730/5): Loss -0.4653, Sem -0.4653, dir -0.9327, norm 0.3811 LR: 8.956e-02 Score 97.126 Data time: 0.1945, Forward time: 0.9584, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.2592 [2020-12-31 08:15:38,188][root][INFO] - ===> Epoch[347](1735/5): Loss -0.4468, Sem -0.4468, dir -0.9304, norm 0.3973 LR: 8.953e-02 Score 97.109 Data time: 0.0842, Forward time: 0.9584, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.5023 [2020-12-31 08:15:50,787][root][INFO] - ===> Epoch[348](1740/5): Loss -0.4467, Sem -0.4467, dir -0.9311, norm 0.3922 LR: 8.950e-02 Score 96.941 Data time: 0.1927, Forward time: 0.9587, Backward time: 1.2040, DDP time: 0.0052, Total iter time: 2.4313 [2020-12-31 08:16:02,280][root][INFO] - ===> Epoch[349](1745/5): Loss -0.4683, Sem -0.4683, dir -0.9339, norm 0.3770 LR: 8.947e-02 Score 97.063 Data time: 0.1520, Forward time: 0.9590, Backward time: 1.2034, DDP time: 0.0052, Total iter time: 2.1991 [2020-12-31 08:16:15,489][root][INFO] - ===> Epoch[350](1750/5): Loss -0.4412, Sem -0.4412, dir -0.9301, norm 0.3988 LR: 8.944e-02 Score 96.989 Data time: 0.1361, Forward time: 0.9591, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.5539 [2020-12-31 08:16:26,880][root][INFO] - ===> Epoch[351](1755/5): Loss -0.4562, Sem -0.4562, dir -0.9327, norm 0.3863 LR: 8.941e-02 Score 96.986 Data time: 0.1042, Forward time: 0.9592, Backward time: 1.2033, DDP time: 0.0052, Total iter time: 2.2028 [2020-12-31 08:16:38,965][root][INFO] - ===> Epoch[352](1760/5): Loss -0.4633, Sem -0.4633, dir -0.9329, norm 0.3815 LR: 8.938e-02 Score 97.051 Data time: 0.1184, Forward time: 0.9593, Backward time: 1.2031, DDP time: 0.0052, Total iter time: 2.3196 [2020-12-31 08:16:51,008][root][INFO] - ===> Epoch[353](1765/5): Loss -0.4495, Sem -0.4495, dir -0.9329, norm 0.3950 LR: 8.935e-02 Score 97.048 Data time: 0.1039, Forward time: 0.9594, Backward time: 1.2030, DDP time: 0.0052, Total iter time: 2.3123 [2020-12-31 08:17:03,055][root][INFO] - ===> Epoch[354](1770/5): Loss -0.4435, Sem -0.4435, dir -0.9295, norm 0.3966 LR: 8.932e-02 Score 96.999 Data time: 0.1027, Forward time: 0.9595, Backward time: 1.2029, DDP time: 0.0052, Total iter time: 2.3256 [2020-12-31 08:17:15,478][root][INFO] - ===> Epoch[355](1775/5): Loss -0.4475, Sem -0.4475, dir -0.9313, norm 0.3934 LR: 8.928e-02 Score 96.976 Data time: 0.1254, Forward time: 0.9596, Backward time: 1.2028, DDP time: 0.0052, Total iter time: 2.3855 [2020-12-31 08:17:27,569][root][INFO] - ===> Epoch[356](1780/5): Loss -0.4494, Sem -0.4494, dir -0.9302, norm 0.3920 LR: 8.925e-02 Score 97.005 Data time: 0.1144, Forward time: 0.9597, Backward time: 1.2028, DDP time: 0.0052, Total iter time: 2.3250 [2020-12-31 08:17:39,856][root][INFO] - ===> Epoch[357](1785/5): Loss -0.4500, Sem -0.4500, dir -0.9308, norm 0.3934 LR: 8.922e-02 Score 97.093 Data time: 0.1455, Forward time: 0.9599, Backward time: 1.2025, DDP time: 0.0052, Total iter time: 2.3523 [2020-12-31 08:17:52,070][root][INFO] - ===> Epoch[358](1790/5): Loss -0.4586, Sem -0.4586, dir -0.9327, norm 0.3870 LR: 8.919e-02 Score 97.096 Data time: 0.1451, Forward time: 0.9601, Backward time: 1.2023, DDP time: 0.0052, Total iter time: 2.3503 [2020-12-31 08:18:05,299][root][INFO] - ===> Epoch[359](1795/5): Loss -0.4449, Sem -0.4449, dir -0.9284, norm 0.3901 LR: 8.916e-02 Score 96.892 Data time: 0.0726, Forward time: 0.9599, Backward time: 1.2031, DDP time: 0.0052, Total iter time: 2.5699 [2020-12-31 08:18:18,202][root][INFO] - ===> Epoch[360](1800/5): Loss -0.4593, Sem -0.4593, dir -0.9348, norm 0.3918 LR: 8.913e-02 Score 97.236 Data time: 0.0924, Forward time: 0.9599, Backward time: 1.2036, DDP time: 0.0052, Total iter time: 2.5018 [2020-12-31 08:18:18,205][root][INFO] - ===> Start testing [2020-12-31 08:18:54,545][root][INFO] - 68/68: Data time: 0.3423, Iter time: 0.1468 Loss 0.221 (AVG: 0.447) Score 94.167 (AVG: 88.636) mIOU 64.232 mAP 82.994 mAcc 74.687 IOU: 59.254 0.000 54.991 72.871 93.348 87.472 27.494 67.951 98.032 60.649 80.427 82.269 50.260 mAP: 82.791 2.660 90.553 88.523 98.846 98.165 61.735 91.404 99.810 91.248 93.464 92.314 87.412 mAcc: 76.979 0.000 85.175 83.673 94.998 93.517 31.756 79.975 99.565 89.059 89.839 92.843 53.549 [2020-12-31 08:18:54,546][root][INFO] - Finished test. Elapsed time: 36.3388 [2020-12-31 08:18:57,681][root][INFO] - [2020-12-31 08:18:57,682][root][INFO] - ################################################################ [2020-12-31 08:18:57,682][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 08:18:57,682][root][INFO] - ################################################################ [2020-12-31 08:18:57,682][root][INFO] - clutter : nan nan nan [2020-12-31 08:18:57,682][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 08:18:57,682][root][INFO] - board : 0.573 0.819 0.877 [2020-12-31 08:18:57,682][root][INFO] - bookcase : 0.299 0.436 0.603 [2020-12-31 08:18:57,682][root][INFO] - ceiling : 0.710 0.740 0.781 [2020-12-31 08:18:57,682][root][INFO] - chair : 0.709 0.837 0.861 [2020-12-31 08:18:57,682][root][INFO] - column : 0.088 0.190 0.343 [2020-12-31 08:18:57,682][root][INFO] - door : 0.490 0.754 0.901 [2020-12-31 08:18:57,682][root][INFO] - floor : 0.970 0.994 0.994 [2020-12-31 08:18:57,682][root][INFO] - sofa : 0.530 0.794 0.794 [2020-12-31 08:18:57,682][root][INFO] - table : 0.158 0.360 0.617 [2020-12-31 08:18:57,682][root][INFO] - wall : 0.403 0.627 0.738 [2020-12-31 08:18:57,682][root][INFO] - window : 0.342 0.635 0.714 [2020-12-31 08:18:57,682][root][INFO] - ---------------------------------------------------------------- [2020-12-31 08:18:57,682][root][INFO] - average : 0.439 0.599 0.685 [2020-12-31 08:18:57,682][root][INFO] - [2020-12-31 08:18:57,855][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 08:18:58,479][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_mAP.pth [2020-12-31 08:18:58,479][root][INFO] - Current best mAP@0.5: 0.599 at iter 1800 [2020-12-31 08:18:59,091][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter1800.pth [2020-12-31 08:19:09,882][root][INFO] - ===> Epoch[361](1805/5): Loss -0.4725, Sem -0.4725, dir -0.9320, norm 0.3676 LR: 8.910e-02 Score 96.965 Data time: 0.1508, Forward time: 0.9591, Backward time: 1.2034, DDP time: 0.0052, Total iter time: 2.0641 [2020-12-31 08:19:21,266][root][INFO] - ===> Epoch[362](1810/5): Loss -0.4556, Sem -0.4556, dir -0.9328, norm 0.3854 LR: 8.907e-02 Score 96.976 Data time: 0.1168, Forward time: 0.9593, Backward time: 1.2029, DDP time: 0.0052, Total iter time: 2.1839 [2020-12-31 08:19:34,425][root][INFO] - ===> Epoch[363](1815/5): Loss -0.4551, Sem -0.4551, dir -0.9329, norm 0.3906 LR: 8.904e-02 Score 97.120 Data time: 0.0820, Forward time: 0.9593, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.5366 [2020-12-31 08:19:48,427][root][INFO] - ===> Epoch[364](1820/5): Loss -0.4578, Sem -0.4578, dir -0.9328, norm 0.3866 LR: 8.901e-02 Score 97.060 Data time: 0.1372, Forward time: 0.9596, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.7030 [2020-12-31 08:20:00,783][root][INFO] - ===> Epoch[365](1825/5): Loss -0.4601, Sem -0.4601, dir -0.9327, norm 0.3841 LR: 8.898e-02 Score 97.062 Data time: 0.1896, Forward time: 0.9596, Backward time: 1.2040, DDP time: 0.0052, Total iter time: 2.3760 [2020-12-31 08:20:12,253][root][INFO] - ===> Epoch[366](1830/5): Loss -0.4646, Sem -0.4646, dir -0.9335, norm 0.3812 LR: 8.895e-02 Score 97.077 Data time: 0.1221, Forward time: 0.9596, Backward time: 1.2036, DDP time: 0.0052, Total iter time: 2.2040 [2020-12-31 08:20:23,856][root][INFO] - ===> Epoch[367](1835/5): Loss -0.4539, Sem -0.4539, dir -0.9336, norm 0.3939 LR: 8.892e-02 Score 97.121 Data time: 0.1039, Forward time: 0.9594, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.2314 [2020-12-31 08:20:36,008][root][INFO] - ===> Epoch[368](1840/5): Loss -0.4496, Sem -0.4496, dir -0.9325, norm 0.3954 LR: 8.889e-02 Score 97.135 Data time: 0.1099, Forward time: 0.9596, Backward time: 1.2034, DDP time: 0.0052, Total iter time: 2.3428 [2020-12-31 08:20:48,130][root][INFO] - ===> Epoch[369](1845/5): Loss -0.4529, Sem -0.4529, dir -0.9316, norm 0.3938 LR: 8.886e-02 Score 97.160 Data time: 0.1291, Forward time: 0.9597, Backward time: 1.2032, DDP time: 0.0052, Total iter time: 2.3262 [2020-12-31 08:21:00,529][root][INFO] - ===> Epoch[370](1850/5): Loss -0.4641, Sem -0.4641, dir -0.9327, norm 0.3828 LR: 8.883e-02 Score 97.151 Data time: 0.1756, Forward time: 0.9601, Backward time: 1.2029, DDP time: 0.0052, Total iter time: 2.3847 [2020-12-31 08:21:14,419][root][INFO] - ===> Epoch[371](1855/5): Loss -0.4774, Sem -0.4774, dir -0.9359, norm 0.3767 LR: 8.880e-02 Score 97.244 Data time: 0.0907, Forward time: 0.9601, Backward time: 1.2038, DDP time: 0.0052, Total iter time: 2.6844 [2020-12-31 08:21:26,610][root][INFO] - ===> Epoch[372](1860/5): Loss -0.4810, Sem -0.4810, dir -0.9357, norm 0.3708 LR: 8.877e-02 Score 97.188 Data time: 0.1394, Forward time: 0.9602, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.3536 [2020-12-31 08:21:37,908][root][INFO] - ===> Epoch[373](1865/5): Loss -0.4805, Sem -0.4805, dir -0.9357, norm 0.3714 LR: 8.874e-02 Score 97.207 Data time: 0.0985, Forward time: 0.9601, Backward time: 1.2034, DDP time: 0.0052, Total iter time: 2.1791 [2020-12-31 08:21:50,164][root][INFO] - ===> Epoch[374](1870/5): Loss -0.4661, Sem -0.4661, dir -0.9329, norm 0.3787 LR: 8.871e-02 Score 97.080 Data time: 0.0945, Forward time: 0.9600, Backward time: 1.2036, DDP time: 0.0052, Total iter time: 2.3680 [2020-12-31 08:22:02,423][root][INFO] - ===> Epoch[375](1875/5): Loss -0.4522, Sem -0.4522, dir -0.9323, norm 0.3853 LR: 8.868e-02 Score 96.901 Data time: 0.1059, Forward time: 0.9601, Backward time: 1.2036, DDP time: 0.0052, Total iter time: 2.3664 [2020-12-31 08:22:15,576][root][INFO] - ===> Epoch[376](1880/5): Loss -0.4604, Sem -0.4604, dir -0.9326, norm 0.3836 LR: 8.865e-02 Score 97.083 Data time: 0.0896, Forward time: 0.9600, Backward time: 1.2042, DDP time: 0.0052, Total iter time: 2.5448 [2020-12-31 08:22:28,092][root][INFO] - ===> Epoch[377](1885/5): Loss -0.4581, Sem -0.4581, dir -0.9351, norm 0.3940 LR: 8.862e-02 Score 97.216 Data time: 0.1049, Forward time: 0.9600, Backward time: 1.2044, DDP time: 0.0052, Total iter time: 2.4144 [2020-12-31 08:22:40,273][root][INFO] - ===> Epoch[378](1890/5): Loss -0.4604, Sem -0.4604, dir -0.9338, norm 0.3855 LR: 8.859e-02 Score 97.084 Data time: 0.0942, Forward time: 0.9599, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.3568 [2020-12-31 08:22:51,902][root][INFO] - ===> Epoch[379](1895/5): Loss -0.4544, Sem -0.4544, dir -0.9327, norm 0.3877 LR: 8.856e-02 Score 96.986 Data time: 0.0961, Forward time: 0.9598, Backward time: 1.2044, DDP time: 0.0052, Total iter time: 2.2462 [2020-12-31 08:23:03,420][root][INFO] - ===> Epoch[380](1900/5): Loss -0.4541, Sem -0.4541, dir -0.9330, norm 0.3934 LR: 8.852e-02 Score 97.148 Data time: 0.1315, Forward time: 0.9599, Backward time: 1.2040, DDP time: 0.0052, Total iter time: 2.2073 [2020-12-31 08:23:15,037][root][INFO] - ===> Epoch[381](1905/5): Loss -0.4785, Sem -0.4785, dir -0.9343, norm 0.3706 LR: 8.849e-02 Score 97.176 Data time: 0.1407, Forward time: 0.9599, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.2368 [2020-12-31 08:23:29,060][root][INFO] - ===> Epoch[382](1910/5): Loss -0.4708, Sem -0.4708, dir -0.9356, norm 0.3834 LR: 8.846e-02 Score 97.303 Data time: 0.1016, Forward time: 0.9599, Backward time: 1.2047, DDP time: 0.0052, Total iter time: 2.7181 [2020-12-31 08:23:40,687][root][INFO] - ===> Epoch[383](1915/5): Loss -0.4736, Sem -0.4736, dir -0.9364, norm 0.3790 LR: 8.843e-02 Score 97.246 Data time: 0.1617, Forward time: 0.9598, Backward time: 1.2044, DDP time: 0.0052, Total iter time: 2.2369 [2020-12-31 08:23:52,901][root][INFO] - ===> Epoch[384](1920/5): Loss -0.4656, Sem -0.4656, dir -0.9341, norm 0.3846 LR: 8.840e-02 Score 97.226 Data time: 0.0978, Forward time: 0.9597, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.3657 [2020-12-31 08:24:06,084][root][INFO] - ===> Epoch[385](1925/5): Loss -0.4711, Sem -0.4711, dir -0.9351, norm 0.3833 LR: 8.837e-02 Score 97.297 Data time: 0.1450, Forward time: 0.9601, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.5447 [2020-12-31 08:24:17,686][root][INFO] - ===> Epoch[386](1930/5): Loss -0.4927, Sem -0.4927, dir -0.9365, norm 0.3599 LR: 8.834e-02 Score 97.203 Data time: 0.0972, Forward time: 0.9602, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.2263 [2020-12-31 08:24:29,341][root][INFO] - ===> Epoch[387](1935/5): Loss -0.4868, Sem -0.4868, dir -0.9378, norm 0.3672 LR: 8.831e-02 Score 97.228 Data time: 0.0987, Forward time: 0.9600, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.2411 [2020-12-31 08:24:40,866][root][INFO] - ===> Epoch[388](1940/5): Loss -0.4888, Sem -0.4888, dir -0.9352, norm 0.3614 LR: 8.828e-02 Score 97.230 Data time: 0.1181, Forward time: 0.9599, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.2079 [2020-12-31 08:24:52,965][root][INFO] - ===> Epoch[389](1945/5): Loss -0.4671, Sem -0.4671, dir -0.9342, norm 0.3816 LR: 8.825e-02 Score 97.160 Data time: 0.1092, Forward time: 0.9600, Backward time: 1.2040, DDP time: 0.0052, Total iter time: 2.3245 [2020-12-31 08:25:05,376][root][INFO] - ===> Epoch[390](1950/5): Loss -0.4760, Sem -0.4760, dir -0.9367, norm 0.3812 LR: 8.822e-02 Score 97.338 Data time: 0.0796, Forward time: 0.9598, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.3892 [2020-12-31 08:25:17,051][root][INFO] - ===> Epoch[391](1955/5): Loss -0.4801, Sem -0.4801, dir -0.9363, norm 0.3736 LR: 8.819e-02 Score 97.223 Data time: 0.0664, Forward time: 0.9593, Backward time: 1.2047, DDP time: 0.0052, Total iter time: 2.2537 [2020-12-31 08:25:29,546][root][INFO] - ===> Epoch[392](1960/5): Loss -0.4897, Sem -0.4897, dir -0.9377, norm 0.3691 LR: 8.816e-02 Score 97.371 Data time: 0.1106, Forward time: 0.9594, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.4035 [2020-12-31 08:25:41,689][root][INFO] - ===> Epoch[393](1965/5): Loss -0.4803, Sem -0.4803, dir -0.9359, norm 0.3733 LR: 8.813e-02 Score 97.258 Data time: 0.0877, Forward time: 0.9594, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.3338 [2020-12-31 08:25:53,463][root][INFO] - ===> Epoch[394](1970/5): Loss -0.5173, Sem -0.5173, dir -0.9409, norm 0.3502 LR: 8.810e-02 Score 97.548 Data time: 0.1472, Forward time: 0.9596, Backward time: 1.2044, DDP time: 0.0052, Total iter time: 2.2590 [2020-12-31 08:26:05,848][root][INFO] - ===> Epoch[395](1975/5): Loss -0.4544, Sem -0.4544, dir -0.9355, norm 0.3963 LR: 8.807e-02 Score 97.158 Data time: 0.1307, Forward time: 0.9595, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.3958 [2020-12-31 08:26:18,037][root][INFO] - ===> Epoch[396](1980/5): Loss -0.4896, Sem -0.4896, dir -0.9372, norm 0.3714 LR: 8.804e-02 Score 97.446 Data time: 0.1571, Forward time: 0.9596, Backward time: 1.2044, DDP time: 0.0052, Total iter time: 2.3492 [2020-12-31 08:26:30,831][root][INFO] - ===> Epoch[397](1985/5): Loss -0.4760, Sem -0.4760, dir -0.9368, norm 0.3796 LR: 8.801e-02 Score 97.328 Data time: 0.0979, Forward time: 0.9595, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.4644 [2020-12-31 08:26:42,600][root][INFO] - ===> Epoch[398](1990/5): Loss -0.4782, Sem -0.4782, dir -0.9364, norm 0.3780 LR: 8.798e-02 Score 97.343 Data time: 0.1045, Forward time: 0.9595, Backward time: 1.2047, DDP time: 0.0052, Total iter time: 2.2688 [2020-12-31 08:26:54,605][root][INFO] - ===> Epoch[399](1995/5): Loss -0.4912, Sem -0.4912, dir -0.9368, norm 0.3666 LR: 8.795e-02 Score 97.396 Data time: 0.1589, Forward time: 0.9594, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.3099 [2020-12-31 08:27:06,991][root][INFO] - ===> Epoch[400](2000/5): Loss -0.4819, Sem -0.4819, dir -0.9369, norm 0.3733 LR: 8.792e-02 Score 97.274 Data time: 0.0963, Forward time: 0.9593, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.3902 [2020-12-31 08:27:06,993][root][INFO] - ===> Start testing [2020-12-31 08:27:43,696][root][INFO] - 68/68: Data time: 0.0022, Iter time: 0.1144 Loss 0.291 (AVG: 0.460) Score 92.584 (AVG: 88.692) mIOU 66.129 mAP 83.460 mAcc 74.648 IOU: 59.138 0.000 72.172 72.187 92.394 86.639 29.983 70.010 98.223 65.145 79.883 83.075 50.822 mAP: 82.138 2.698 93.040 87.640 98.729 96.969 65.154 92.786 99.818 90.641 92.531 93.594 89.240 mAcc: 80.047 0.000 84.324 79.877 93.708 90.448 41.430 83.219 99.303 80.821 88.762 94.008 54.471 [2020-12-31 08:27:43,699][root][INFO] - Finished test. Elapsed time: 36.7018 [2020-12-31 08:27:46,747][root][INFO] - [2020-12-31 08:27:46,747][root][INFO] - ################################################################ [2020-12-31 08:27:46,747][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 08:27:46,747][root][INFO] - ################################################################ [2020-12-31 08:27:46,747][root][INFO] - clutter : nan nan nan [2020-12-31 08:27:46,747][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 08:27:46,747][root][INFO] - board : 0.719 0.857 0.904 [2020-12-31 08:27:46,747][root][INFO] - bookcase : 0.250 0.400 0.523 [2020-12-31 08:27:46,747][root][INFO] - ceiling : 0.768 0.810 0.877 [2020-12-31 08:27:46,747][root][INFO] - chair : 0.719 0.845 0.859 [2020-12-31 08:27:46,747][root][INFO] - column : 0.126 0.235 0.345 [2020-12-31 08:27:46,747][root][INFO] - door : 0.472 0.684 0.877 [2020-12-31 08:27:46,747][root][INFO] - floor : 1.000 1.000 1.000 [2020-12-31 08:27:46,747][root][INFO] - sofa : 0.450 0.727 0.801 [2020-12-31 08:27:46,747][root][INFO] - table : 0.145 0.361 0.619 [2020-12-31 08:27:46,748][root][INFO] - wall : 0.481 0.664 0.783 [2020-12-31 08:27:46,748][root][INFO] - window : 0.379 0.658 0.708 [2020-12-31 08:27:46,748][root][INFO] - ---------------------------------------------------------------- [2020-12-31 08:27:46,748][root][INFO] - average : 0.459 0.603 0.691 [2020-12-31 08:27:46,748][root][INFO] - [2020-12-31 08:27:46,907][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 08:27:47,500][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_mAP.pth [2020-12-31 08:27:47,500][root][INFO] - Current best mAP@0.5: 0.603 at iter 2000 [2020-12-31 08:27:48,046][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter2000.pth [2020-12-31 08:27:58,796][root][INFO] - ===> Epoch[401](2005/5): Loss -0.4875, Sem -0.4875, dir -0.9355, norm 0.3675 LR: 8.789e-02 Score 97.273 Data time: 0.1033, Forward time: 0.9591, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.0350 [2020-12-31 08:28:10,231][root][INFO] - ===> Epoch[402](2010/5): Loss -0.4813, Sem -0.4813, dir -0.9382, norm 0.3724 LR: 8.786e-02 Score 97.206 Data time: 0.1017, Forward time: 0.9591, Backward time: 1.2040, DDP time: 0.0052, Total iter time: 2.1850 [2020-12-31 08:28:21,979][root][INFO] - ===> Epoch[403](2015/5): Loss -0.4814, Sem -0.4814, dir -0.9366, norm 0.3733 LR: 8.783e-02 Score 97.289 Data time: 0.1076, Forward time: 0.9590, Backward time: 1.2039, DDP time: 0.0052, Total iter time: 2.2561 [2020-12-31 08:28:33,665][root][INFO] - ===> Epoch[404](2020/5): Loss -0.4921, Sem -0.4921, dir -0.9370, norm 0.3678 LR: 8.779e-02 Score 97.431 Data time: 0.0924, Forward time: 0.9590, Backward time: 1.2038, DDP time: 0.0052, Total iter time: 2.2500 [2020-12-31 08:28:46,089][root][INFO] - ===> Epoch[405](2025/5): Loss -0.4878, Sem -0.4878, dir -0.9363, norm 0.3671 LR: 8.776e-02 Score 97.294 Data time: 0.0888, Forward time: 0.9589, Backward time: 1.2040, DDP time: 0.0052, Total iter time: 2.3887 [2020-12-31 08:28:58,962][root][INFO] - ===> Epoch[406](2030/5): Loss -0.4793, Sem -0.4793, dir -0.9370, norm 0.3747 LR: 8.773e-02 Score 97.240 Data time: 0.0871, Forward time: 0.9587, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.4854 [2020-12-31 08:29:11,830][root][INFO] - ===> Epoch[407](2035/5): Loss -0.4790, Sem -0.4790, dir -0.9362, norm 0.3771 LR: 8.770e-02 Score 97.327 Data time: 0.1101, Forward time: 0.9586, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.4755 [2020-12-31 08:29:23,685][root][INFO] - ===> Epoch[408](2040/5): Loss -0.4650, Sem -0.4650, dir -0.9374, norm 0.3926 LR: 8.767e-02 Score 97.383 Data time: 0.1205, Forward time: 0.9584, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.2835 [2020-12-31 08:29:35,845][root][INFO] - ===> Epoch[409](2045/5): Loss -0.4728, Sem -0.4728, dir -0.9346, norm 0.3808 LR: 8.764e-02 Score 97.269 Data time: 0.1851, Forward time: 0.9583, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.3323 [2020-12-31 08:29:47,585][root][INFO] - ===> Epoch[410](2050/5): Loss -0.4863, Sem -0.4863, dir -0.9375, norm 0.3724 LR: 8.761e-02 Score 97.382 Data time: 0.0866, Forward time: 0.9580, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.2658 [2020-12-31 08:29:59,710][root][INFO] - ===> Epoch[411](2055/5): Loss -0.4885, Sem -0.4885, dir -0.9376, norm 0.3712 LR: 8.758e-02 Score 97.425 Data time: 0.1541, Forward time: 0.9580, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.3239 [2020-12-31 08:30:12,497][root][INFO] - ===> Epoch[412](2060/5): Loss -0.5011, Sem -0.5011, dir -0.9392, norm 0.3605 LR: 8.755e-02 Score 97.411 Data time: 0.1282, Forward time: 0.9580, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.4541 [2020-12-31 08:30:24,793][root][INFO] - ===> Epoch[413](2065/5): Loss -0.4848, Sem -0.4848, dir -0.9371, norm 0.3746 LR: 8.752e-02 Score 97.447 Data time: 0.0931, Forward time: 0.9579, Backward time: 1.2053, DDP time: 0.0052, Total iter time: 2.3697 [2020-12-31 08:30:36,271][root][INFO] - ===> Epoch[414](2070/5): Loss -0.4882, Sem -0.4882, dir -0.9366, norm 0.3694 LR: 8.749e-02 Score 97.373 Data time: 0.1081, Forward time: 0.9578, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.2142 [2020-12-31 08:30:48,401][root][INFO] - ===> Epoch[415](2075/5): Loss -0.5016, Sem -0.5016, dir -0.9396, norm 0.3579 LR: 8.746e-02 Score 97.357 Data time: 0.1337, Forward time: 0.9578, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.3431 [2020-12-31 08:31:01,105][root][INFO] - ===> Epoch[416](2080/5): Loss -0.4845, Sem -0.4845, dir -0.9378, norm 0.3737 LR: 8.743e-02 Score 97.318 Data time: 0.0934, Forward time: 0.9578, Backward time: 1.2054, DDP time: 0.0052, Total iter time: 2.4423 [2020-12-31 08:31:13,243][root][INFO] - ===> Epoch[417](2085/5): Loss -0.4931, Sem -0.4931, dir -0.9382, norm 0.3665 LR: 8.740e-02 Score 97.371 Data time: 0.1142, Forward time: 0.9579, Backward time: 1.2053, DDP time: 0.0052, Total iter time: 2.3430 [2020-12-31 08:31:25,056][root][INFO] - ===> Epoch[418](2090/5): Loss -0.5020, Sem -0.5020, dir -0.9385, norm 0.3580 LR: 8.737e-02 Score 97.411 Data time: 0.0919, Forward time: 0.9576, Backward time: 1.2055, DDP time: 0.0052, Total iter time: 2.2788 [2020-12-31 08:31:37,593][root][INFO] - ===> Epoch[419](2095/5): Loss -0.4912, Sem -0.4912, dir -0.9385, norm 0.3680 LR: 8.734e-02 Score 97.391 Data time: 0.1178, Forward time: 0.9577, Backward time: 1.2055, DDP time: 0.0052, Total iter time: 2.4130 [2020-12-31 08:31:49,858][root][INFO] - ===> Epoch[420](2100/5): Loss -0.5029, Sem -0.5029, dir -0.9394, norm 0.3605 LR: 8.731e-02 Score 97.446 Data time: 0.1225, Forward time: 0.9577, Backward time: 1.2056, DDP time: 0.0052, Total iter time: 2.3724 [2020-12-31 08:32:01,225][root][INFO] - ===> Epoch[421](2105/5): Loss -0.5007, Sem -0.5007, dir -0.9397, norm 0.3620 LR: 8.728e-02 Score 97.442 Data time: 0.0920, Forward time: 0.9578, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.1883 [2020-12-31 08:32:13,813][root][INFO] - ===> Epoch[422](2110/5): Loss -0.4959, Sem -0.4959, dir -0.9391, norm 0.3636 LR: 8.725e-02 Score 97.347 Data time: 0.1179, Forward time: 0.9579, Backward time: 1.2053, DDP time: 0.0052, Total iter time: 2.4261 [2020-12-31 08:32:25,974][root][INFO] - ===> Epoch[423](2115/5): Loss -0.4951, Sem -0.4951, dir -0.9379, norm 0.3637 LR: 8.722e-02 Score 97.373 Data time: 0.1202, Forward time: 0.9582, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.3244 [2020-12-31 08:32:37,543][root][INFO] - ===> Epoch[424](2120/5): Loss -0.5129, Sem -0.5129, dir -0.9395, norm 0.3528 LR: 8.719e-02 Score 97.552 Data time: 0.0887, Forward time: 0.9581, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.2321 [2020-12-31 08:32:51,452][root][INFO] - ===> Epoch[425](2125/5): Loss -0.4927, Sem -0.4927, dir -0.9372, norm 0.3631 LR: 8.716e-02 Score 97.286 Data time: 0.2028, Forward time: 0.9586, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.6840 [2020-12-31 08:33:02,987][root][INFO] - ===> Epoch[426](2130/5): Loss -0.5154, Sem -0.5154, dir -0.9401, norm 0.3492 LR: 8.713e-02 Score 97.483 Data time: 0.1060, Forward time: 0.9587, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.2171 [2020-12-31 08:33:15,475][root][INFO] - ===> Epoch[427](2135/5): Loss -0.5096, Sem -0.5096, dir -0.9416, norm 0.3562 LR: 8.709e-02 Score 97.490 Data time: 0.0827, Forward time: 0.9584, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.4104 [2020-12-31 08:33:27,405][root][INFO] - ===> Epoch[428](2140/5): Loss -0.4998, Sem -0.4998, dir -0.9393, norm 0.3590 LR: 8.706e-02 Score 97.341 Data time: 0.0958, Forward time: 0.9583, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.2875 [2020-12-31 08:33:39,366][root][INFO] - ===> Epoch[429](2145/5): Loss -0.5058, Sem -0.5058, dir -0.9391, norm 0.3584 LR: 8.703e-02 Score 97.494 Data time: 0.1205, Forward time: 0.9583, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.2998 [2020-12-31 08:33:50,662][root][INFO] - ===> Epoch[430](2150/5): Loss -0.4949, Sem -0.4949, dir -0.9387, norm 0.3664 LR: 8.700e-02 Score 97.411 Data time: 0.1039, Forward time: 0.9582, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.1678 [2020-12-31 08:34:02,661][root][INFO] - ===> Epoch[431](2155/5): Loss -0.5066, Sem -0.5066, dir -0.9400, norm 0.3549 LR: 8.697e-02 Score 97.401 Data time: 0.0831, Forward time: 0.9579, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.3148 [2020-12-31 08:34:14,978][root][INFO] - ===> Epoch[432](2160/5): Loss -0.5081, Sem -0.5081, dir -0.9411, norm 0.3542 LR: 8.694e-02 Score 97.397 Data time: 0.1553, Forward time: 0.9581, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.3781 [2020-12-31 08:34:27,382][root][INFO] - ===> Epoch[433](2165/5): Loss -0.4958, Sem -0.4958, dir -0.9394, norm 0.3630 LR: 8.691e-02 Score 97.362 Data time: 0.1409, Forward time: 0.9582, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.3902 [2020-12-31 08:34:40,169][root][INFO] - ===> Epoch[434](2170/5): Loss -0.4954, Sem -0.4954, dir -0.9391, norm 0.3609 LR: 8.688e-02 Score 97.236 Data time: 0.0839, Forward time: 0.9581, Backward time: 1.2054, DDP time: 0.0052, Total iter time: 2.4647 [2020-12-31 08:34:52,218][root][INFO] - ===> Epoch[435](2175/5): Loss -0.5160, Sem -0.5160, dir -0.9413, norm 0.3501 LR: 8.685e-02 Score 97.511 Data time: 0.1550, Forward time: 0.9583, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.3100 [2020-12-31 08:35:04,323][root][INFO] - ===> Epoch[436](2180/5): Loss -0.4982, Sem -0.4982, dir -0.9400, norm 0.3639 LR: 8.682e-02 Score 97.408 Data time: 0.1119, Forward time: 0.9583, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.3315 [2020-12-31 08:35:16,079][root][INFO] - ===> Epoch[437](2185/5): Loss -0.5105, Sem -0.5105, dir -0.9404, norm 0.3576 LR: 8.679e-02 Score 97.597 Data time: 0.1193, Forward time: 0.9583, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.2591 [2020-12-31 08:35:28,143][root][INFO] - ===> Epoch[438](2190/5): Loss -0.5022, Sem -0.5022, dir -0.9393, norm 0.3620 LR: 8.676e-02 Score 97.478 Data time: 0.0807, Forward time: 0.9581, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.3257 [2020-12-31 08:35:39,848][root][INFO] - ===> Epoch[439](2195/5): Loss -0.5086, Sem -0.5086, dir -0.9414, norm 0.3555 LR: 8.673e-02 Score 97.447 Data time: 0.1130, Forward time: 0.9582, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.2506 [2020-12-31 08:35:52,540][root][INFO] - ===> Epoch[440](2200/5): Loss -0.5052, Sem -0.5052, dir -0.9395, norm 0.3617 LR: 8.670e-02 Score 97.595 Data time: 0.1646, Forward time: 0.9581, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.4530 [2020-12-31 08:35:52,543][root][INFO] - ===> Start testing [2020-12-31 08:36:30,240][root][INFO] - 68/68: Data time: 0.0022, Iter time: 0.1205 Loss 0.235 (AVG: 0.465) Score 92.397 (AVG: 89.015) mIOU 67.527 mAP 83.728 mAcc 74.893 IOU: 58.065 0.000 77.052 71.060 92.788 89.296 33.038 74.596 98.507 64.122 80.438 84.135 54.761 mAP: 82.786 3.665 90.448 86.447 98.921 97.430 69.187 94.003 99.819 88.009 93.386 94.271 90.088 mAcc: 81.474 0.000 81.085 77.276 94.258 93.034 43.860 89.064 99.342 68.607 90.461 93.388 61.764 [2020-12-31 08:36:30,241][root][INFO] - Finished test. Elapsed time: 37.6957 [2020-12-31 08:36:33,295][root][INFO] - [2020-12-31 08:36:33,295][root][INFO] - ################################################################ [2020-12-31 08:36:33,296][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 08:36:33,296][root][INFO] - ################################################################ [2020-12-31 08:36:33,296][root][INFO] - clutter : nan nan nan [2020-12-31 08:36:33,296][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 08:36:33,296][root][INFO] - board : 0.731 0.807 0.833 [2020-12-31 08:36:33,296][root][INFO] - bookcase : 0.247 0.392 0.528 [2020-12-31 08:36:33,296][root][INFO] - ceiling : 0.802 0.815 0.867 [2020-12-31 08:36:33,296][root][INFO] - chair : 0.736 0.851 0.874 [2020-12-31 08:36:33,296][root][INFO] - column : 0.154 0.302 0.442 [2020-12-31 08:36:33,296][root][INFO] - door : 0.577 0.793 0.914 [2020-12-31 08:36:33,296][root][INFO] - floor : 1.000 1.000 1.000 [2020-12-31 08:36:33,296][root][INFO] - sofa : 0.311 0.494 0.708 [2020-12-31 08:36:33,296][root][INFO] - table : 0.133 0.332 0.595 [2020-12-31 08:36:33,296][root][INFO] - wall : 0.425 0.672 0.737 [2020-12-31 08:36:33,296][root][INFO] - window : 0.441 0.709 0.748 [2020-12-31 08:36:33,296][root][INFO] - ---------------------------------------------------------------- [2020-12-31 08:36:33,296][root][INFO] - average : 0.463 0.597 0.687 [2020-12-31 08:36:33,296][root][INFO] - [2020-12-31 08:36:33,466][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 08:36:34,081][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2020-12-31 08:36:34,081][root][INFO] - Current best mIoU: 67.527 at iter 1 [2020-12-31 08:36:34,624][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter2200.pth [2020-12-31 08:36:46,159][root][INFO] - ===> Epoch[441](2205/5): Loss -0.5177, Sem -0.5177, dir -0.9408, norm 0.3525 LR: 8.667e-02 Score 97.658 Data time: 0.0812, Forward time: 0.9579, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.2060 [2020-12-31 08:36:58,859][root][INFO] - ===> Epoch[442](2210/5): Loss -0.5128, Sem -0.5128, dir -0.9409, norm 0.3548 LR: 8.664e-02 Score 97.550 Data time: 0.1317, Forward time: 0.9580, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.4401 [2020-12-31 08:37:11,350][root][INFO] - ===> Epoch[443](2215/5): Loss -0.5132, Sem -0.5132, dir -0.9402, norm 0.3556 LR: 8.661e-02 Score 97.604 Data time: 0.0929, Forward time: 0.9580, Backward time: 1.2053, DDP time: 0.0052, Total iter time: 2.4090 [2020-12-31 08:37:24,143][root][INFO] - ===> Epoch[444](2220/5): Loss -0.5156, Sem -0.5156, dir -0.9419, norm 0.3545 LR: 8.658e-02 Score 97.628 Data time: 0.0972, Forward time: 0.9580, Backward time: 1.2055, DDP time: 0.0052, Total iter time: 2.4558 [2020-12-31 08:37:36,664][root][INFO] - ===> Epoch[445](2225/5): Loss -0.5104, Sem -0.5104, dir -0.9399, norm 0.3549 LR: 8.655e-02 Score 97.543 Data time: 0.1042, Forward time: 0.9580, Backward time: 1.2057, DDP time: 0.0052, Total iter time: 2.4208 [2020-12-31 08:37:48,812][root][INFO] - ===> Epoch[446](2230/5): Loss -0.4956, Sem -0.4956, dir -0.9384, norm 0.3636 LR: 8.652e-02 Score 97.372 Data time: 0.0892, Forward time: 0.9577, Backward time: 1.2060, DDP time: 0.0052, Total iter time: 2.3438 [2020-12-31 08:38:01,858][root][INFO] - ===> Epoch[447](2235/5): Loss -0.5196, Sem -0.5196, dir -0.9434, norm 0.3545 LR: 8.649e-02 Score 97.693 Data time: 0.1377, Forward time: 0.9579, Backward time: 1.2062, DDP time: 0.0052, Total iter time: 2.5172 [2020-12-31 08:38:14,931][root][INFO] - ===> Epoch[448](2240/5): Loss -0.5176, Sem -0.5176, dir -0.9404, norm 0.3491 LR: 8.645e-02 Score 97.565 Data time: 0.0731, Forward time: 0.9575, Backward time: 1.2070, DDP time: 0.0052, Total iter time: 2.5302 [2020-12-31 08:38:28,277][root][INFO] - ===> Epoch[449](2245/5): Loss -0.5170, Sem -0.5170, dir -0.9411, norm 0.3511 LR: 8.642e-02 Score 97.588 Data time: 0.1242, Forward time: 0.9578, Backward time: 1.2072, DDP time: 0.0052, Total iter time: 2.5845 [2020-12-31 08:38:40,438][root][INFO] - ===> Epoch[450](2250/5): Loss -0.5155, Sem -0.5155, dir -0.9425, norm 0.3578 LR: 8.639e-02 Score 97.708 Data time: 0.0934, Forward time: 0.9577, Backward time: 1.2073, DDP time: 0.0052, Total iter time: 2.3457 [2020-12-31 08:38:53,307][root][INFO] - ===> Epoch[451](2255/5): Loss -0.5107, Sem -0.5107, dir -0.9416, norm 0.3596 LR: 8.636e-02 Score 97.622 Data time: 0.1153, Forward time: 0.9576, Backward time: 1.2077, DDP time: 0.0052, Total iter time: 2.4805 [2020-12-31 08:39:04,939][root][INFO] - ===> Epoch[452](2260/5): Loss -0.5058, Sem -0.5058, dir -0.9416, norm 0.3593 LR: 8.633e-02 Score 97.513 Data time: 0.1757, Forward time: 0.9578, Backward time: 1.2072, DDP time: 0.0052, Total iter time: 2.2330 [2020-12-31 08:39:16,408][root][INFO] - ===> Epoch[453](2265/5): Loss -0.5348, Sem -0.5348, dir -0.9422, norm 0.3356 LR: 8.630e-02 Score 97.616 Data time: 0.1244, Forward time: 0.9578, Backward time: 1.2069, DDP time: 0.0052, Total iter time: 2.1998 [2020-12-31 08:39:28,704][root][INFO] - ===> Epoch[454](2270/5): Loss -0.4922, Sem -0.4922, dir -0.9381, norm 0.3682 LR: 8.627e-02 Score 97.427 Data time: 0.1008, Forward time: 0.9580, Backward time: 1.2068, DDP time: 0.0052, Total iter time: 2.3596 [2020-12-31 08:39:41,435][root][INFO] - ===> Epoch[455](2275/5): Loss -0.5101, Sem -0.5101, dir -0.9425, norm 0.3612 LR: 8.624e-02 Score 97.634 Data time: 0.1228, Forward time: 0.9583, Backward time: 1.2067, DDP time: 0.0052, Total iter time: 2.4594 [2020-12-31 08:39:53,911][root][INFO] - ===> Epoch[456](2280/5): Loss -0.5135, Sem -0.5135, dir -0.9415, norm 0.3521 LR: 8.621e-02 Score 97.481 Data time: 0.1278, Forward time: 0.9585, Backward time: 1.2067, DDP time: 0.0052, Total iter time: 2.3978 [2020-12-31 08:40:06,380][root][INFO] - ===> Epoch[457](2285/5): Loss -0.5016, Sem -0.5016, dir -0.9394, norm 0.3588 LR: 8.618e-02 Score 97.410 Data time: 0.1134, Forward time: 0.9584, Backward time: 1.2069, DDP time: 0.0052, Total iter time: 2.4002 [2020-12-31 08:40:18,372][root][INFO] - ===> Epoch[458](2290/5): Loss -0.5233, Sem -0.5233, dir -0.9440, norm 0.3492 LR: 8.615e-02 Score 97.635 Data time: 0.1043, Forward time: 0.9584, Backward time: 1.2068, DDP time: 0.0052, Total iter time: 2.3031 [2020-12-31 08:40:29,949][root][INFO] - ===> Epoch[459](2295/5): Loss -0.5193, Sem -0.5193, dir -0.9400, norm 0.3466 LR: 8.612e-02 Score 97.534 Data time: 0.0836, Forward time: 0.9584, Backward time: 1.2067, DDP time: 0.0052, Total iter time: 2.2326 [2020-12-31 08:40:43,354][root][INFO] - ===> Epoch[460](2300/5): Loss -0.5006, Sem -0.5006, dir -0.9404, norm 0.3629 LR: 8.609e-02 Score 97.466 Data time: 0.1500, Forward time: 0.9585, Backward time: 1.2070, DDP time: 0.0052, Total iter time: 2.5885 [2020-12-31 08:40:55,705][root][INFO] - ===> Epoch[461](2305/5): Loss -0.5141, Sem -0.5141, dir -0.9438, norm 0.3531 LR: 8.606e-02 Score 97.460 Data time: 0.1166, Forward time: 0.9589, Backward time: 1.2068, DDP time: 0.0052, Total iter time: 2.3783 [2020-12-31 08:41:07,912][root][INFO] - ===> Epoch[462](2310/5): Loss -0.5228, Sem -0.5228, dir -0.9429, norm 0.3488 LR: 8.603e-02 Score 97.618 Data time: 0.1517, Forward time: 0.9590, Backward time: 1.2066, DDP time: 0.0052, Total iter time: 2.3505 [2020-12-31 08:41:19,806][root][INFO] - ===> Epoch[463](2315/5): Loss -0.5078, Sem -0.5078, dir -0.9410, norm 0.3546 LR: 8.600e-02 Score 97.385 Data time: 0.1290, Forward time: 0.9591, Backward time: 1.2064, DDP time: 0.0052, Total iter time: 2.2962 [2020-12-31 08:41:31,049][root][INFO] - ===> Epoch[464](2320/5): Loss -0.5387, Sem -0.5387, dir -0.9455, norm 0.3383 LR: 8.597e-02 Score 97.716 Data time: 0.1152, Forward time: 0.9590, Backward time: 1.2062, DDP time: 0.0052, Total iter time: 2.1802 [2020-12-31 08:41:44,059][root][INFO] - ===> Epoch[465](2325/5): Loss -0.5279, Sem -0.5279, dir -0.9422, norm 0.3440 LR: 8.594e-02 Score 97.700 Data time: 0.1069, Forward time: 0.9590, Backward time: 1.2064, DDP time: 0.0052, Total iter time: 2.5088 [2020-12-31 08:41:56,120][root][INFO] - ===> Epoch[466](2330/5): Loss -0.5246, Sem -0.5246, dir -0.9436, norm 0.3468 LR: 8.591e-02 Score 97.595 Data time: 0.1114, Forward time: 0.9590, Backward time: 1.2064, DDP time: 0.0052, Total iter time: 2.3152 [2020-12-31 08:42:08,929][root][INFO] - ===> Epoch[467](2335/5): Loss -0.5215, Sem -0.5215, dir -0.9429, norm 0.3520 LR: 8.588e-02 Score 97.660 Data time: 0.1171, Forward time: 0.9593, Backward time: 1.2064, DDP time: 0.0052, Total iter time: 2.4727 [2020-12-31 08:42:21,385][root][INFO] - ===> Epoch[468](2340/5): Loss -0.5252, Sem -0.5252, dir -0.9436, norm 0.3475 LR: 8.584e-02 Score 97.638 Data time: 0.1322, Forward time: 0.9596, Backward time: 1.2063, DDP time: 0.0052, Total iter time: 2.4093 [2020-12-31 08:42:34,133][root][INFO] - ===> Epoch[469](2345/5): Loss -0.5303, Sem -0.5303, dir -0.9439, norm 0.3459 LR: 8.581e-02 Score 97.758 Data time: 0.1896, Forward time: 0.9597, Backward time: 1.2062, DDP time: 0.0052, Total iter time: 2.4595 [2020-12-31 08:42:46,939][root][INFO] - ===> Epoch[470](2350/5): Loss -0.5367, Sem -0.5367, dir -0.9450, norm 0.3373 LR: 8.578e-02 Score 97.639 Data time: 0.1749, Forward time: 0.9599, Backward time: 1.2062, DDP time: 0.0052, Total iter time: 2.4754 [2020-12-31 08:42:58,527][root][INFO] - ===> Epoch[471](2355/5): Loss -0.5165, Sem -0.5165, dir -0.9425, norm 0.3555 LR: 8.575e-02 Score 97.661 Data time: 0.1567, Forward time: 0.9599, Backward time: 1.2059, DDP time: 0.0052, Total iter time: 2.2198 [2020-12-31 08:43:11,855][root][INFO] - ===> Epoch[472](2360/5): Loss -0.5243, Sem -0.5243, dir -0.9425, norm 0.3467 LR: 8.572e-02 Score 97.644 Data time: 0.1166, Forward time: 0.9600, Backward time: 1.2062, DDP time: 0.0052, Total iter time: 2.5616 [2020-12-31 08:43:23,782][root][INFO] - ===> Epoch[473](2365/5): Loss -0.5354, Sem -0.5354, dir -0.9448, norm 0.3418 LR: 8.569e-02 Score 97.745 Data time: 0.0878, Forward time: 0.9598, Backward time: 1.2063, DDP time: 0.0052, Total iter time: 2.3051 [2020-12-31 08:43:35,926][root][INFO] - ===> Epoch[474](2370/5): Loss -0.5293, Sem -0.5293, dir -0.9449, norm 0.3496 LR: 8.566e-02 Score 97.813 Data time: 0.1142, Forward time: 0.9598, Backward time: 1.2063, DDP time: 0.0052, Total iter time: 2.3471 [2020-12-31 08:43:47,798][root][INFO] - ===> Epoch[475](2375/5): Loss -0.5387, Sem -0.5387, dir -0.9462, norm 0.3390 LR: 8.563e-02 Score 97.735 Data time: 0.0794, Forward time: 0.9599, Backward time: 1.2062, DDP time: 0.0052, Total iter time: 2.2906 [2020-12-31 08:43:59,535][root][INFO] - ===> Epoch[476](2380/5): Loss -0.5405, Sem -0.5405, dir -0.9463, norm 0.3349 LR: 8.560e-02 Score 97.646 Data time: 0.0948, Forward time: 0.9597, Backward time: 1.2063, DDP time: 0.0052, Total iter time: 2.2594 [2020-12-31 08:44:13,047][root][INFO] - ===> Epoch[477](2385/5): Loss -0.5069, Sem -0.5069, dir -0.9417, norm 0.3604 LR: 8.557e-02 Score 97.540 Data time: 0.0818, Forward time: 0.9595, Backward time: 1.2071, DDP time: 0.0052, Total iter time: 2.6168 [2020-12-31 08:44:25,099][root][INFO] - ===> Epoch[478](2390/5): Loss -0.5370, Sem -0.5370, dir -0.9466, norm 0.3411 LR: 8.554e-02 Score 97.740 Data time: 0.0883, Forward time: 0.9595, Backward time: 1.2071, DDP time: 0.0052, Total iter time: 2.3152 [2020-12-31 08:44:36,745][root][INFO] - ===> Epoch[479](2395/5): Loss -0.5131, Sem -0.5131, dir -0.9421, norm 0.3506 LR: 8.551e-02 Score 97.367 Data time: 0.1454, Forward time: 0.9596, Backward time: 1.2067, DDP time: 0.0052, Total iter time: 2.2488 [2020-12-31 08:44:48,917][root][INFO] - ===> Epoch[480](2400/5): Loss -0.5253, Sem -0.5253, dir -0.9451, norm 0.3457 LR: 8.548e-02 Score 97.566 Data time: 0.1124, Forward time: 0.9595, Backward time: 1.2069, DDP time: 0.0052, Total iter time: 2.3601 [2020-12-31 08:44:48,920][root][INFO] - ===> Start testing [2020-12-31 08:45:25,318][root][INFO] - 68/68: Data time: 0.2916, Iter time: 0.1937 Loss 0.611 (AVG: 0.476) Score 89.099 (AVG: 88.952) mIOU 65.653 mAP 83.120 mAcc 72.537 IOU: 58.719 0.000 72.607 71.725 93.828 87.639 26.218 71.934 98.522 59.016 78.153 83.453 51.677 mAP: 81.219 2.866 92.872 86.747 98.914 97.023 63.808 92.609 99.791 90.012 93.048 93.143 88.514 mAcc: 79.486 0.000 84.686 79.785 95.316 89.946 30.116 81.772 99.559 61.642 88.254 94.711 57.704 [2020-12-31 08:45:25,319][root][INFO] - Finished test. Elapsed time: 36.3975 [2020-12-31 08:45:28,241][root][INFO] - [2020-12-31 08:45:28,242][root][INFO] - ################################################################ [2020-12-31 08:45:28,242][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 08:45:28,242][root][INFO] - ################################################################ [2020-12-31 08:45:28,242][root][INFO] - clutter : nan nan nan [2020-12-31 08:45:28,242][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 08:45:28,242][root][INFO] - board : 0.722 0.806 0.875 [2020-12-31 08:45:28,242][root][INFO] - bookcase : 0.252 0.386 0.529 [2020-12-31 08:45:28,242][root][INFO] - ceiling : 0.658 0.733 0.769 [2020-12-31 08:45:28,242][root][INFO] - chair : 0.706 0.841 0.854 [2020-12-31 08:45:28,242][root][INFO] - column : 0.100 0.167 0.278 [2020-12-31 08:45:28,242][root][INFO] - door : 0.482 0.754 0.877 [2020-12-31 08:45:28,242][root][INFO] - floor : 0.995 0.998 0.998 [2020-12-31 08:45:28,242][root][INFO] - sofa : 0.213 0.353 0.636 [2020-12-31 08:45:28,242][root][INFO] - table : 0.130 0.288 0.588 [2020-12-31 08:45:28,242][root][INFO] - wall : 0.483 0.686 0.791 [2020-12-31 08:45:28,242][root][INFO] - window : 0.345 0.605 0.716 [2020-12-31 08:45:28,242][root][INFO] - ---------------------------------------------------------------- [2020-12-31 08:45:28,242][root][INFO] - average : 0.424 0.551 0.659 [2020-12-31 08:45:28,242][root][INFO] - [2020-12-31 08:45:28,415][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 08:45:28,963][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter2400.pth [2020-12-31 08:45:40,283][root][INFO] - ===> Epoch[481](2405/5): Loss -0.5332, Sem -0.5332, dir -0.9439, norm 0.3418 LR: 8.545e-02 Score 97.734 Data time: 0.0935, Forward time: 0.9594, Backward time: 1.2066, DDP time: 0.0052, Total iter time: 2.1577 [2020-12-31 08:45:52,547][root][INFO] - ===> Epoch[482](2410/5): Loss -0.5193, Sem -0.5193, dir -0.9434, norm 0.3547 LR: 8.542e-02 Score 97.677 Data time: 0.1024, Forward time: 0.9593, Backward time: 1.2067, DDP time: 0.0052, Total iter time: 2.3563 [2020-12-31 08:46:05,061][root][INFO] - ===> Epoch[483](2415/5): Loss -0.5280, Sem -0.5280, dir -0.9423, norm 0.3420 LR: 8.539e-02 Score 97.624 Data time: 0.1231, Forward time: 0.9593, Backward time: 1.2068, DDP time: 0.0052, Total iter time: 2.4102 [2020-12-31 08:46:17,394][root][INFO] - ===> Epoch[484](2420/5): Loss -0.5034, Sem -0.5034, dir -0.9425, norm 0.3672 LR: 8.536e-02 Score 97.613 Data time: 0.1095, Forward time: 0.9595, Backward time: 1.2068, DDP time: 0.0052, Total iter time: 2.3809 [2020-12-31 08:46:29,052][root][INFO] - ===> Epoch[485](2425/5): Loss -0.5381, Sem -0.5381, dir -0.9455, norm 0.3380 LR: 8.533e-02 Score 97.709 Data time: 0.1313, Forward time: 0.9595, Backward time: 1.2065, DDP time: 0.0052, Total iter time: 2.2383 [2020-12-31 08:46:41,228][root][INFO] - ===> Epoch[486](2430/5): Loss -0.5181, Sem -0.5181, dir -0.9417, norm 0.3502 LR: 8.530e-02 Score 97.538 Data time: 0.0738, Forward time: 0.9592, Backward time: 1.2070, DDP time: 0.0052, Total iter time: 2.3654 [2020-12-31 08:46:53,655][root][INFO] - ===> Epoch[487](2435/5): Loss -0.5225, Sem -0.5225, dir -0.9436, norm 0.3534 LR: 8.526e-02 Score 97.739 Data time: 0.1902, Forward time: 0.9593, Backward time: 1.2068, DDP time: 0.0052, Total iter time: 2.3870 [2020-12-31 08:47:04,847][root][INFO] - ===> Epoch[488](2440/5): Loss -0.5170, Sem -0.5170, dir -0.9415, norm 0.3527 LR: 8.523e-02 Score 97.630 Data time: 0.1020, Forward time: 0.9594, Backward time: 1.2065, DDP time: 0.0052, Total iter time: 2.1594 [2020-12-31 08:47:16,719][root][INFO] - ===> Epoch[489](2445/5): Loss -0.5249, Sem -0.5249, dir -0.9425, norm 0.3477 LR: 8.520e-02 Score 97.674 Data time: 0.1039, Forward time: 0.9595, Backward time: 1.2063, DDP time: 0.0052, Total iter time: 2.2769 [2020-12-31 08:47:29,369][root][INFO] - ===> Epoch[490](2450/5): Loss -0.5318, Sem -0.5318, dir -0.9445, norm 0.3457 LR: 8.517e-02 Score 97.749 Data time: 0.0909, Forward time: 0.9594, Backward time: 1.2066, DDP time: 0.0052, Total iter time: 2.4384 [2020-12-31 08:47:42,221][root][INFO] - ===> Epoch[491](2455/5): Loss -0.5165, Sem -0.5165, dir -0.9436, norm 0.3539 LR: 8.514e-02 Score 97.569 Data time: 0.1589, Forward time: 0.9596, Backward time: 1.2066, DDP time: 0.0052, Total iter time: 2.4774 [2020-12-31 08:47:53,528][root][INFO] - ===> Epoch[492](2460/5): Loss -0.5227, Sem -0.5227, dir -0.9420, norm 0.3473 LR: 8.511e-02 Score 97.629 Data time: 0.0948, Forward time: 0.9595, Backward time: 1.2064, DDP time: 0.0052, Total iter time: 2.1818 [2020-12-31 08:48:05,183][root][INFO] - ===> Epoch[493](2465/5): Loss -0.5284, Sem -0.5284, dir -0.9439, norm 0.3446 LR: 8.508e-02 Score 97.672 Data time: 0.0696, Forward time: 0.9592, Backward time: 1.2066, DDP time: 0.0052, Total iter time: 2.2629 [2020-12-31 08:48:16,854][root][INFO] - ===> Epoch[494](2470/5): Loss -0.5445, Sem -0.5445, dir -0.9469, norm 0.3344 LR: 8.505e-02 Score 97.783 Data time: 0.1572, Forward time: 0.9592, Backward time: 1.2063, DDP time: 0.0052, Total iter time: 2.2411 [2020-12-31 08:48:28,426][root][INFO] - ===> Epoch[495](2475/5): Loss -0.5353, Sem -0.5353, dir -0.9448, norm 0.3403 LR: 8.502e-02 Score 97.697 Data time: 0.1027, Forward time: 0.9593, Backward time: 1.2061, DDP time: 0.0052, Total iter time: 2.2197 [2020-12-31 08:48:40,279][root][INFO] - ===> Epoch[496](2480/5): Loss -0.5263, Sem -0.5263, dir -0.9434, norm 0.3432 LR: 8.499e-02 Score 97.558 Data time: 0.0765, Forward time: 0.9590, Backward time: 1.2063, DDP time: 0.0052, Total iter time: 2.2741 [2020-12-31 08:48:51,334][root][INFO] - ===> Epoch[497](2485/5): Loss -0.5267, Sem -0.5267, dir -0.9444, norm 0.3484 LR: 8.496e-02 Score 97.688 Data time: 0.0940, Forward time: 0.9590, Backward time: 1.2059, DDP time: 0.0052, Total iter time: 2.1353 [2020-12-31 08:49:03,589][root][INFO] - ===> Epoch[498](2490/5): Loss -0.5129, Sem -0.5129, dir -0.9431, norm 0.3549 LR: 8.493e-02 Score 97.519 Data time: 0.1570, Forward time: 0.9593, Backward time: 1.2056, DDP time: 0.0052, Total iter time: 2.3557 [2020-12-31 08:49:15,995][root][INFO] - ===> Epoch[499](2495/5): Loss -0.5262, Sem -0.5262, dir -0.9440, norm 0.3487 LR: 8.490e-02 Score 97.710 Data time: 0.1797, Forward time: 0.9594, Backward time: 1.2055, DDP time: 0.0052, Total iter time: 2.3891 [2020-12-31 08:49:28,304][root][INFO] - ===> Epoch[500](2500/5): Loss -0.5446, Sem -0.5446, dir -0.9458, norm 0.3351 LR: 8.487e-02 Score 97.788 Data time: 0.0797, Forward time: 0.9593, Backward time: 1.2058, DDP time: 0.0052, Total iter time: 2.3746 [2020-12-31 08:49:40,755][root][INFO] - ===> Epoch[501](2505/5): Loss -0.5417, Sem -0.5417, dir -0.9438, norm 0.3375 LR: 8.484e-02 Score 97.814 Data time: 0.1015, Forward time: 0.9591, Backward time: 1.2061, DDP time: 0.0052, Total iter time: 2.4078 [2020-12-31 08:49:52,884][root][INFO] - ===> Epoch[502](2510/5): Loss -0.5296, Sem -0.5296, dir -0.9435, norm 0.3445 LR: 8.481e-02 Score 97.676 Data time: 0.0843, Forward time: 0.9589, Backward time: 1.2063, DDP time: 0.0052, Total iter time: 2.3545 [2020-12-31 08:50:04,404][root][INFO] - ===> Epoch[503](2515/5): Loss -0.5483, Sem -0.5483, dir -0.9448, norm 0.3272 LR: 8.478e-02 Score 97.681 Data time: 0.1036, Forward time: 0.9590, Backward time: 1.2061, DDP time: 0.0052, Total iter time: 2.2146 [2020-12-31 08:50:17,353][root][INFO] - ===> Epoch[504](2520/5): Loss -0.5311, Sem -0.5311, dir -0.9459, norm 0.3475 LR: 8.475e-02 Score 97.780 Data time: 0.1433, Forward time: 0.9591, Backward time: 1.2061, DDP time: 0.0052, Total iter time: 2.4869 [2020-12-31 08:50:29,630][root][INFO] - ===> Epoch[505](2525/5): Loss -0.5482, Sem -0.5482, dir -0.9461, norm 0.3302 LR: 8.471e-02 Score 97.775 Data time: 0.1485, Forward time: 0.9591, Backward time: 1.2062, DDP time: 0.0052, Total iter time: 2.3670 [2020-12-31 08:50:42,125][root][INFO] - ===> Epoch[506](2530/5): Loss -0.5357, Sem -0.5357, dir -0.9452, norm 0.3408 LR: 8.468e-02 Score 97.731 Data time: 0.0899, Forward time: 0.9590, Backward time: 1.2064, DDP time: 0.0052, Total iter time: 2.4024 [2020-12-31 08:50:53,790][root][INFO] - ===> Epoch[507](2535/5): Loss -0.5258, Sem -0.5258, dir -0.9459, norm 0.3518 LR: 8.465e-02 Score 97.740 Data time: 0.0786, Forward time: 0.9589, Backward time: 1.2063, DDP time: 0.0052, Total iter time: 2.2416 [2020-12-31 08:51:06,091][root][INFO] - ===> Epoch[508](2540/5): Loss -0.5341, Sem -0.5341, dir -0.9449, norm 0.3388 LR: 8.462e-02 Score 97.605 Data time: 0.1441, Forward time: 0.9590, Backward time: 1.2062, DDP time: 0.0052, Total iter time: 2.3607 [2020-12-31 08:51:18,489][root][INFO] - ===> Epoch[509](2545/5): Loss -0.5373, Sem -0.5373, dir -0.9440, norm 0.3379 LR: 8.459e-02 Score 97.717 Data time: 0.1090, Forward time: 0.9589, Backward time: 1.2064, DDP time: 0.0052, Total iter time: 2.3843 [2020-12-31 08:51:30,662][root][INFO] - ===> Epoch[510](2550/5): Loss -0.5323, Sem -0.5323, dir -0.9437, norm 0.3435 LR: 8.456e-02 Score 97.729 Data time: 0.0921, Forward time: 0.9588, Backward time: 1.2065, DDP time: 0.0052, Total iter time: 2.3351 [2020-12-31 08:51:43,208][root][INFO] - ===> Epoch[511](2555/5): Loss -0.5328, Sem -0.5328, dir -0.9441, norm 0.3425 LR: 8.453e-02 Score 97.721 Data time: 0.0837, Forward time: 0.9588, Backward time: 1.2068, DDP time: 0.0052, Total iter time: 2.4241 [2020-12-31 08:51:55,746][root][INFO] - ===> Epoch[512](2560/5): Loss -0.5528, Sem -0.5528, dir -0.9487, norm 0.3330 LR: 8.450e-02 Score 97.910 Data time: 0.1311, Forward time: 0.9589, Backward time: 1.2068, DDP time: 0.0052, Total iter time: 2.4225 [2020-12-31 08:52:07,595][root][INFO] - ===> Epoch[513](2565/5): Loss -0.5303, Sem -0.5303, dir -0.9444, norm 0.3473 LR: 8.447e-02 Score 97.786 Data time: 0.2156, Forward time: 0.9590, Backward time: 1.2064, DDP time: 0.0052, Total iter time: 2.2795 [2020-12-31 08:52:19,132][root][INFO] - ===> Epoch[514](2570/5): Loss -0.5388, Sem -0.5388, dir -0.9446, norm 0.3403 LR: 8.444e-02 Score 97.832 Data time: 0.1312, Forward time: 0.9591, Backward time: 1.2061, DDP time: 0.0052, Total iter time: 2.2093 [2020-12-31 08:52:30,836][root][INFO] - ===> Epoch[515](2575/5): Loss -0.5476, Sem -0.5476, dir -0.9473, norm 0.3343 LR: 8.441e-02 Score 97.838 Data time: 0.1398, Forward time: 0.9592, Backward time: 1.2058, DDP time: 0.0052, Total iter time: 2.2475 [2020-12-31 08:52:42,563][root][INFO] - ===> Epoch[516](2580/5): Loss -0.5281, Sem -0.5281, dir -0.9459, norm 0.3498 LR: 8.438e-02 Score 97.790 Data time: 0.1343, Forward time: 0.9592, Backward time: 1.2056, DDP time: 0.0052, Total iter time: 2.2661 [2020-12-31 08:52:55,688][root][INFO] - ===> Epoch[517](2585/5): Loss -0.5538, Sem -0.5538, dir -0.9485, norm 0.3271 LR: 8.435e-02 Score 97.750 Data time: 0.1000, Forward time: 0.9593, Backward time: 1.2059, DDP time: 0.0052, Total iter time: 2.5309 [2020-12-31 08:53:07,801][root][INFO] - ===> Epoch[518](2590/5): Loss -0.5267, Sem -0.5267, dir -0.9428, norm 0.3483 LR: 8.432e-02 Score 97.763 Data time: 0.1408, Forward time: 0.9594, Backward time: 1.2058, DDP time: 0.0052, Total iter time: 2.3228 [2020-12-31 08:53:19,684][root][INFO] - ===> Epoch[519](2595/5): Loss -0.5565, Sem -0.5565, dir -0.9467, norm 0.3228 LR: 8.429e-02 Score 97.774 Data time: 0.1127, Forward time: 0.9594, Backward time: 1.2056, DDP time: 0.0052, Total iter time: 2.2742 [2020-12-31 08:53:31,679][root][INFO] - ===> Epoch[520](2600/5): Loss -0.5341, Sem -0.5341, dir -0.9462, norm 0.3442 LR: 8.426e-02 Score 97.738 Data time: 0.1040, Forward time: 0.9595, Backward time: 1.2055, DDP time: 0.0052, Total iter time: 2.3094 [2020-12-31 08:53:31,682][root][INFO] - ===> Start testing [2020-12-31 08:54:09,464][root][INFO] - 68/68: Data time: 0.0021, Iter time: 0.1260 Loss 0.802 (AVG: 0.472) Score 86.107 (AVG: 89.117) mIOU 65.812 mAP 83.707 mAcc 74.802 IOU: 59.702 0.000 72.741 73.724 91.999 88.243 20.331 65.838 98.371 62.462 81.172 83.910 57.058 mAP: 82.487 2.902 95.723 88.188 98.791 98.351 58.033 91.128 99.807 93.624 93.537 93.867 91.749 mAcc: 78.234 0.000 86.659 82.255 93.438 94.955 22.426 77.438 99.421 89.157 89.883 95.309 63.258 [2020-12-31 08:54:09,465][root][INFO] - Finished test. Elapsed time: 37.7812 [2020-12-31 08:54:12,705][root][INFO] - [2020-12-31 08:54:12,705][root][INFO] - ################################################################ [2020-12-31 08:54:12,705][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 08:54:12,705][root][INFO] - ################################################################ [2020-12-31 08:54:12,705][root][INFO] - clutter : nan nan nan [2020-12-31 08:54:12,705][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 08:54:12,706][root][INFO] - board : 0.721 0.838 0.869 [2020-12-31 08:54:12,706][root][INFO] - bookcase : 0.286 0.457 0.545 [2020-12-31 08:54:12,706][root][INFO] - ceiling : 0.675 0.726 0.754 [2020-12-31 08:54:12,706][root][INFO] - chair : 0.755 0.868 0.892 [2020-12-31 08:54:12,706][root][INFO] - column : 0.086 0.170 0.282 [2020-12-31 08:54:12,706][root][INFO] - door : 0.438 0.663 0.845 [2020-12-31 08:54:12,706][root][INFO] - floor : 0.817 0.845 0.845 [2020-12-31 08:54:12,706][root][INFO] - sofa : 0.382 0.571 0.754 [2020-12-31 08:54:12,706][root][INFO] - table : 0.176 0.332 0.602 [2020-12-31 08:54:12,706][root][INFO] - wall : 0.461 0.677 0.768 [2020-12-31 08:54:12,706][root][INFO] - window : 0.466 0.700 0.747 [2020-12-31 08:54:12,706][root][INFO] - ---------------------------------------------------------------- [2020-12-31 08:54:12,706][root][INFO] - average : 0.439 0.571 0.659 [2020-12-31 08:54:12,706][root][INFO] - [2020-12-31 08:54:12,877][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 08:54:13,540][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter2600.pth [2020-12-31 08:54:25,342][root][INFO] - ===> Epoch[521](2605/5): Loss -0.5284, Sem -0.5284, dir -0.9452, norm 0.3478 LR: 8.423e-02 Score 97.733 Data time: 0.1102, Forward time: 0.9595, Backward time: 1.2054, DDP time: 0.0052, Total iter time: 2.2524 [2020-12-31 08:54:37,707][root][INFO] - ===> Epoch[522](2610/5): Loss -0.5261, Sem -0.5261, dir -0.9449, norm 0.3494 LR: 8.420e-02 Score 97.704 Data time: 0.0749, Forward time: 0.9593, Backward time: 1.2057, DDP time: 0.0052, Total iter time: 2.3923 [2020-12-31 08:54:50,393][root][INFO] - ===> Epoch[523](2615/5): Loss -0.5450, Sem -0.5450, dir -0.9460, norm 0.3335 LR: 8.416e-02 Score 97.772 Data time: 0.1063, Forward time: 0.9592, Backward time: 1.2060, DDP time: 0.0052, Total iter time: 2.4360 [2020-12-31 08:55:02,721][root][INFO] - ===> Epoch[524](2620/5): Loss -0.5138, Sem -0.5138, dir -0.9435, norm 0.3599 LR: 8.413e-02 Score 97.697 Data time: 0.1003, Forward time: 0.9590, Backward time: 1.2062, DDP time: 0.0052, Total iter time: 2.3717 [2020-12-31 08:55:14,870][root][INFO] - ===> Epoch[525](2625/5): Loss -0.5407, Sem -0.5407, dir -0.9456, norm 0.3353 LR: 8.410e-02 Score 97.722 Data time: 0.1149, Forward time: 0.9589, Backward time: 1.2063, DDP time: 0.0052, Total iter time: 2.3371 [2020-12-31 08:55:26,688][root][INFO] - ===> Epoch[526](2630/5): Loss -0.5301, Sem -0.5301, dir -0.9455, norm 0.3470 LR: 8.407e-02 Score 97.719 Data time: 0.0897, Forward time: 0.9589, Backward time: 1.2063, DDP time: 0.0052, Total iter time: 2.2643 [2020-12-31 08:55:38,832][root][INFO] - ===> Epoch[527](2635/5): Loss -0.5275, Sem -0.5275, dir -0.9461, norm 0.3521 LR: 8.404e-02 Score 97.790 Data time: 0.1108, Forward time: 0.9589, Backward time: 1.2062, DDP time: 0.0052, Total iter time: 2.3461 [2020-12-31 08:55:51,139][root][INFO] - ===> Epoch[528](2640/5): Loss -0.5309, Sem -0.5309, dir -0.9450, norm 0.3425 LR: 8.401e-02 Score 97.613 Data time: 0.1085, Forward time: 0.9590, Backward time: 1.2063, DDP time: 0.0052, Total iter time: 2.3650 [2020-12-31 08:56:03,253][root][INFO] - ===> Epoch[529](2645/5): Loss -0.5058, Sem -0.5058, dir -0.9433, norm 0.3584 LR: 8.398e-02 Score 97.450 Data time: 0.0872, Forward time: 0.9588, Backward time: 1.2064, DDP time: 0.0052, Total iter time: 2.3216 [2020-12-31 08:56:15,095][root][INFO] - ===> Epoch[530](2650/5): Loss -0.5146, Sem -0.5146, dir -0.9440, norm 0.3545 LR: 8.395e-02 Score 97.494 Data time: 0.0989, Forward time: 0.9587, Backward time: 1.2064, DDP time: 0.0052, Total iter time: 2.2809 [2020-12-31 08:56:27,600][root][INFO] - ===> Epoch[531](2655/5): Loss -0.5313, Sem -0.5313, dir -0.9453, norm 0.3426 LR: 8.392e-02 Score 97.620 Data time: 0.0840, Forward time: 0.9586, Backward time: 1.2067, DDP time: 0.0052, Total iter time: 2.4137 [2020-12-31 08:56:39,301][root][INFO] - ===> Epoch[532](2660/5): Loss -0.5297, Sem -0.5297, dir -0.9442, norm 0.3427 LR: 8.389e-02 Score 97.593 Data time: 0.1178, Forward time: 0.9586, Backward time: 1.2065, DDP time: 0.0052, Total iter time: 2.2491 [2020-12-31 08:56:51,130][root][INFO] - ===> Epoch[533](2665/5): Loss -0.5203, Sem -0.5203, dir -0.9447, norm 0.3548 LR: 8.386e-02 Score 97.704 Data time: 0.2232, Forward time: 0.9588, Backward time: 1.2060, DDP time: 0.0052, Total iter time: 2.2672 [2020-12-31 08:57:03,059][root][INFO] - ===> Epoch[534](2670/5): Loss -0.5338, Sem -0.5338, dir -0.9459, norm 0.3413 LR: 8.383e-02 Score 97.680 Data time: 0.1146, Forward time: 0.9588, Backward time: 1.2059, DDP time: 0.0052, Total iter time: 2.2856 [2020-12-31 08:57:15,177][root][INFO] - ===> Epoch[535](2675/5): Loss -0.5583, Sem -0.5583, dir -0.9491, norm 0.3243 LR: 8.380e-02 Score 97.802 Data time: 0.0910, Forward time: 0.9586, Backward time: 1.2061, DDP time: 0.0052, Total iter time: 2.3344 [2020-12-31 08:57:26,392][root][INFO] - ===> Epoch[536](2680/5): Loss -0.5384, Sem -0.5384, dir -0.9440, norm 0.3358 LR: 8.377e-02 Score 97.688 Data time: 0.1045, Forward time: 0.9585, Backward time: 1.2059, DDP time: 0.0052, Total iter time: 2.1643 [2020-12-31 08:57:38,265][root][INFO] - ===> Epoch[537](2685/5): Loss -0.5431, Sem -0.5431, dir -0.9465, norm 0.3354 LR: 8.374e-02 Score 97.756 Data time: 0.1884, Forward time: 0.9586, Backward time: 1.2056, DDP time: 0.0052, Total iter time: 2.2789 [2020-12-31 08:57:50,166][root][INFO] - ===> Epoch[538](2690/5): Loss -0.5382, Sem -0.5382, dir -0.9467, norm 0.3401 LR: 8.371e-02 Score 97.740 Data time: 0.1009, Forward time: 0.9586, Backward time: 1.2055, DDP time: 0.0052, Total iter time: 2.2770 [2020-12-31 08:58:02,087][root][INFO] - ===> Epoch[539](2695/5): Loss -0.5380, Sem -0.5380, dir -0.9465, norm 0.3409 LR: 8.368e-02 Score 97.793 Data time: 0.1092, Forward time: 0.9586, Backward time: 1.2055, DDP time: 0.0052, Total iter time: 2.3001 [2020-12-31 08:58:13,569][root][INFO] - ===> Epoch[540](2700/5): Loss -0.5279, Sem -0.5279, dir -0.9448, norm 0.3480 LR: 8.364e-02 Score 97.707 Data time: 0.1159, Forward time: 0.9586, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.2002 [2020-12-31 08:58:24,639][root][INFO] - ===> Epoch[541](2705/5): Loss -0.5486, Sem -0.5486, dir -0.9465, norm 0.3353 LR: 8.361e-02 Score 97.926 Data time: 0.1320, Forward time: 0.9585, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.1241 [2020-12-31 08:58:36,932][root][INFO] - ===> Epoch[542](2710/5): Loss -0.5561, Sem -0.5561, dir -0.9472, norm 0.3270 LR: 8.358e-02 Score 97.849 Data time: 0.0802, Forward time: 0.9582, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.3692 [2020-12-31 08:58:48,562][root][INFO] - ===> Epoch[543](2715/5): Loss -0.5429, Sem -0.5429, dir -0.9467, norm 0.3376 LR: 8.355e-02 Score 97.816 Data time: 0.1220, Forward time: 0.9583, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.2301 [2020-12-31 08:59:00,815][root][INFO] - ===> Epoch[544](2720/5): Loss -0.5486, Sem -0.5486, dir -0.9476, norm 0.3318 LR: 8.352e-02 Score 97.787 Data time: 0.0939, Forward time: 0.9583, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.3620 [2020-12-31 08:59:12,989][root][INFO] - ===> Epoch[545](2725/5): Loss -0.5609, Sem -0.5609, dir -0.9483, norm 0.3203 LR: 8.349e-02 Score 97.776 Data time: 0.1071, Forward time: 0.9582, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.3542 [2020-12-31 08:59:24,612][root][INFO] - ===> Epoch[546](2730/5): Loss -0.5470, Sem -0.5470, dir -0.9470, norm 0.3349 LR: 8.346e-02 Score 97.822 Data time: 0.0751, Forward time: 0.9579, Backward time: 1.2053, DDP time: 0.0052, Total iter time: 2.2463 [2020-12-31 08:59:37,060][root][INFO] - ===> Epoch[547](2735/5): Loss -0.5596, Sem -0.5596, dir -0.9482, norm 0.3229 LR: 8.343e-02 Score 97.854 Data time: 0.1909, Forward time: 0.9581, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.3903 [2020-12-31 08:59:49,003][root][INFO] - ===> Epoch[548](2740/5): Loss -0.5526, Sem -0.5526, dir -0.9468, norm 0.3277 LR: 8.340e-02 Score 97.800 Data time: 0.0946, Forward time: 0.9580, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.2938 [2020-12-31 09:00:01,872][root][INFO] - ===> Epoch[549](2745/5): Loss -0.5583, Sem -0.5583, dir -0.9486, norm 0.3264 LR: 8.337e-02 Score 97.876 Data time: 0.1091, Forward time: 0.9579, Backward time: 1.2055, DDP time: 0.0052, Total iter time: 2.4730 [2020-12-31 09:00:13,726][root][INFO] - ===> Epoch[550](2750/5): Loss -0.5650, Sem -0.5650, dir -0.9492, norm 0.3228 LR: 8.334e-02 Score 97.964 Data time: 0.0927, Forward time: 0.9580, Backward time: 1.2053, DDP time: 0.0052, Total iter time: 2.2803 [2020-12-31 09:00:25,894][root][INFO] - ===> Epoch[551](2755/5): Loss -0.5469, Sem -0.5469, dir -0.9491, norm 0.3357 LR: 8.331e-02 Score 97.815 Data time: 0.0823, Forward time: 0.9579, Backward time: 1.2055, DDP time: 0.0052, Total iter time: 2.3361 [2020-12-31 09:00:37,604][root][INFO] - ===> Epoch[552](2760/5): Loss -0.5502, Sem -0.5502, dir -0.9478, norm 0.3338 LR: 8.328e-02 Score 97.886 Data time: 0.1272, Forward time: 0.9579, Backward time: 1.2053, DDP time: 0.0052, Total iter time: 2.2586 [2020-12-31 09:00:50,023][root][INFO] - ===> Epoch[553](2765/5): Loss -0.5648, Sem -0.5648, dir -0.9508, norm 0.3246 LR: 8.325e-02 Score 97.965 Data time: 0.0884, Forward time: 0.9578, Backward time: 1.2056, DDP time: 0.0052, Total iter time: 2.4057 [2020-12-31 09:01:02,684][root][INFO] - ===> Epoch[554](2770/5): Loss -0.5606, Sem -0.5606, dir -0.9476, norm 0.3240 LR: 8.322e-02 Score 97.903 Data time: 0.1112, Forward time: 0.9577, Backward time: 1.2058, DDP time: 0.0052, Total iter time: 2.4388 [2020-12-31 09:01:14,583][root][INFO] - ===> Epoch[555](2775/5): Loss -0.5646, Sem -0.5646, dir -0.9489, norm 0.3216 LR: 8.319e-02 Score 97.915 Data time: 0.1057, Forward time: 0.9578, Backward time: 1.2057, DDP time: 0.0052, Total iter time: 2.2840 [2020-12-31 09:01:26,252][root][INFO] - ===> Epoch[556](2780/5): Loss -0.5661, Sem -0.5661, dir -0.9501, norm 0.3203 LR: 8.315e-02 Score 97.863 Data time: 0.1638, Forward time: 0.9579, Backward time: 1.2053, DDP time: 0.0052, Total iter time: 2.2418 [2020-12-31 09:01:38,157][root][INFO] - ===> Epoch[557](2785/5): Loss -0.5611, Sem -0.5611, dir -0.9487, norm 0.3253 LR: 8.312e-02 Score 97.952 Data time: 0.0964, Forward time: 0.9580, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.2916 [2020-12-31 09:01:50,612][root][INFO] - ===> Epoch[558](2790/5): Loss -0.5538, Sem -0.5538, dir -0.9473, norm 0.3257 LR: 8.309e-02 Score 97.766 Data time: 0.0731, Forward time: 0.9578, Backward time: 1.2056, DDP time: 0.0052, Total iter time: 2.4296 [2020-12-31 09:02:03,382][root][INFO] - ===> Epoch[559](2795/5): Loss -0.5750, Sem -0.5750, dir -0.9509, norm 0.3148 LR: 8.306e-02 Score 97.963 Data time: 0.0900, Forward time: 0.9577, Backward time: 1.2059, DDP time: 0.0052, Total iter time: 2.4668 [2020-12-31 09:02:15,079][root][INFO] - ===> Epoch[560](2800/5): Loss -0.5658, Sem -0.5658, dir -0.9479, norm 0.3198 LR: 8.303e-02 Score 97.919 Data time: 0.0966, Forward time: 0.9577, Backward time: 1.2058, DDP time: 0.0052, Total iter time: 2.2505 [2020-12-31 09:02:15,081][root][INFO] - ===> Start testing [2020-12-31 09:02:53,246][root][INFO] - 68/68: Data time: 0.7987, Iter time: 0.3231 Loss 1.370 (AVG: 0.476) Score 70.669 (AVG: 88.904) mIOU 67.059 mAP 83.627 mAcc 75.446 IOU: 59.700 0.000 74.953 71.832 93.193 89.293 29.710 70.919 98.421 67.510 79.381 82.828 54.022 mAP: 82.557 3.583 90.636 85.979 98.893 98.118 64.498 94.671 99.827 91.071 93.107 93.502 90.701 mAcc: 81.143 0.000 84.044 78.744 94.661 94.410 39.774 91.673 99.301 78.226 87.382 92.669 58.771 [2020-12-31 09:02:53,247][root][INFO] - Finished test. Elapsed time: 38.1634 [2020-12-31 09:02:56,549][root][INFO] - [2020-12-31 09:02:56,549][root][INFO] - ################################################################ [2020-12-31 09:02:56,549][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 09:02:56,549][root][INFO] - ################################################################ [2020-12-31 09:02:56,549][root][INFO] - clutter : nan nan nan [2020-12-31 09:02:56,549][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 09:02:56,549][root][INFO] - board : 0.674 0.780 0.854 [2020-12-31 09:02:56,549][root][INFO] - bookcase : 0.223 0.347 0.508 [2020-12-31 09:02:56,549][root][INFO] - ceiling : 0.794 0.820 0.854 [2020-12-31 09:02:56,549][root][INFO] - chair : 0.775 0.880 0.896 [2020-12-31 09:02:56,550][root][INFO] - column : 0.140 0.230 0.405 [2020-12-31 09:02:56,550][root][INFO] - door : 0.548 0.727 0.887 [2020-12-31 09:02:56,550][root][INFO] - floor : 0.900 0.902 0.953 [2020-12-31 09:02:56,550][root][INFO] - sofa : 0.367 0.546 0.794 [2020-12-31 09:02:56,550][root][INFO] - table : 0.154 0.316 0.595 [2020-12-31 09:02:56,550][root][INFO] - wall : 0.475 0.676 0.781 [2020-12-31 09:02:56,550][root][INFO] - window : 0.452 0.679 0.752 [2020-12-31 09:02:56,550][root][INFO] - ---------------------------------------------------------------- [2020-12-31 09:02:56,550][root][INFO] - average : 0.458 0.575 0.690 [2020-12-31 09:02:56,550][root][INFO] - [2020-12-31 09:02:56,717][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 09:02:57,289][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter2800.pth [2020-12-31 09:03:08,675][root][INFO] - ===> Epoch[561](2805/5): Loss -0.5495, Sem -0.5495, dir -0.9481, norm 0.3359 LR: 8.300e-02 Score 97.931 Data time: 0.1065, Forward time: 0.9575, Backward time: 1.2057, DDP time: 0.0052, Total iter time: 2.1681 [2020-12-31 09:03:20,325][root][INFO] - ===> Epoch[562](2810/5): Loss -0.5718, Sem -0.5718, dir -0.9508, norm 0.3133 LR: 8.297e-02 Score 97.806 Data time: 0.0951, Forward time: 0.9573, Backward time: 1.2057, DDP time: 0.0052, Total iter time: 2.2481 [2020-12-31 09:03:31,860][root][INFO] - ===> Epoch[563](2815/5): Loss -0.5587, Sem -0.5587, dir -0.9487, norm 0.3248 LR: 8.294e-02 Score 97.816 Data time: 0.0803, Forward time: 0.9572, Backward time: 1.2057, DDP time: 0.0052, Total iter time: 2.2224 [2020-12-31 09:03:44,208][root][INFO] - ===> Epoch[564](2820/5): Loss -0.5577, Sem -0.5577, dir -0.9478, norm 0.3250 LR: 8.291e-02 Score 97.846 Data time: 0.1269, Forward time: 0.9573, Backward time: 1.2056, DDP time: 0.0052, Total iter time: 2.3761 [2020-12-31 09:03:56,651][root][INFO] - ===> Epoch[565](2825/5): Loss -0.5748, Sem -0.5748, dir -0.9496, norm 0.3123 LR: 8.288e-02 Score 97.933 Data time: 0.1722, Forward time: 0.9572, Backward time: 1.2056, DDP time: 0.0052, Total iter time: 2.3955 [2020-12-31 09:04:09,294][root][INFO] - ===> Epoch[566](2830/5): Loss -0.5663, Sem -0.5663, dir -0.9489, norm 0.3193 LR: 8.285e-02 Score 97.905 Data time: 0.1288, Forward time: 0.9574, Backward time: 1.2057, DDP time: 0.0052, Total iter time: 2.4419 [2020-12-31 09:04:21,474][root][INFO] - ===> Epoch[567](2835/5): Loss -0.5698, Sem -0.5698, dir -0.9507, norm 0.3202 LR: 8.282e-02 Score 98.008 Data time: 0.1105, Forward time: 0.9574, Backward time: 1.2057, DDP time: 0.0052, Total iter time: 2.3485 [2020-12-31 09:04:33,477][root][INFO] - ===> Epoch[568](2840/5): Loss -0.5523, Sem -0.5523, dir -0.9469, norm 0.3297 LR: 8.279e-02 Score 97.818 Data time: 0.1263, Forward time: 0.9574, Backward time: 1.2056, DDP time: 0.0052, Total iter time: 2.3017 [2020-12-31 09:04:45,948][root][INFO] - ===> Epoch[569](2845/5): Loss -0.5483, Sem -0.5483, dir -0.9492, norm 0.3346 LR: 8.276e-02 Score 97.832 Data time: 0.1951, Forward time: 0.9576, Backward time: 1.2054, DDP time: 0.0052, Total iter time: 2.4007 [2020-12-31 09:04:57,924][root][INFO] - ===> Epoch[570](2850/5): Loss -0.5534, Sem -0.5534, dir -0.9485, norm 0.3315 LR: 8.273e-02 Score 97.873 Data time: 0.0932, Forward time: 0.9575, Backward time: 1.2054, DDP time: 0.0052, Total iter time: 2.3013 [2020-12-31 09:05:09,887][root][INFO] - ===> Epoch[571](2855/5): Loss -0.5659, Sem -0.5659, dir -0.9471, norm 0.3168 LR: 8.270e-02 Score 97.864 Data time: 0.0623, Forward time: 0.9572, Backward time: 1.2058, DDP time: 0.0052, Total iter time: 2.3134 [2020-12-31 09:05:21,687][root][INFO] - ===> Epoch[572](2860/5): Loss -0.5702, Sem -0.5702, dir -0.9496, norm 0.3168 LR: 8.266e-02 Score 97.915 Data time: 0.1773, Forward time: 0.9571, Backward time: 1.2056, DDP time: 0.0052, Total iter time: 2.2654 [2020-12-31 09:05:33,983][root][INFO] - ===> Epoch[573](2865/5): Loss -0.5618, Sem -0.5618, dir -0.9487, norm 0.3201 LR: 8.263e-02 Score 97.818 Data time: 0.1734, Forward time: 0.9571, Backward time: 1.2055, DDP time: 0.0052, Total iter time: 2.3583 [2020-12-31 09:05:45,442][root][INFO] - ===> Epoch[574](2870/5): Loss -0.5623, Sem -0.5623, dir -0.9508, norm 0.3205 LR: 8.260e-02 Score 97.812 Data time: 0.0995, Forward time: 0.9570, Backward time: 1.2055, DDP time: 0.0052, Total iter time: 2.2023 [2020-12-31 09:05:57,335][root][INFO] - ===> Epoch[575](2875/5): Loss -0.5752, Sem -0.5752, dir -0.9494, norm 0.3101 LR: 8.257e-02 Score 97.871 Data time: 0.0911, Forward time: 0.9568, Backward time: 1.2056, DDP time: 0.0052, Total iter time: 2.2924 [2020-12-31 09:06:09,232][root][INFO] - ===> Epoch[576](2880/5): Loss -0.5620, Sem -0.5620, dir -0.9489, norm 0.3212 LR: 8.254e-02 Score 97.850 Data time: 0.1610, Forward time: 0.9568, Backward time: 1.2054, DDP time: 0.0052, Total iter time: 2.2739 [2020-12-31 09:06:20,662][root][INFO] - ===> Epoch[577](2885/5): Loss -0.5693, Sem -0.5693, dir -0.9502, norm 0.3162 LR: 8.251e-02 Score 97.852 Data time: 0.1507, Forward time: 0.9568, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.1890 [2020-12-31 09:06:32,569][root][INFO] - ===> Epoch[578](2890/5): Loss -0.5764, Sem -0.5764, dir -0.9506, norm 0.3115 LR: 8.248e-02 Score 97.932 Data time: 0.0885, Forward time: 0.9567, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.2958 [2020-12-31 09:06:45,105][root][INFO] - ===> Epoch[579](2895/5): Loss -0.5795, Sem -0.5795, dir -0.9502, norm 0.3090 LR: 8.245e-02 Score 97.947 Data time: 0.1172, Forward time: 0.9568, Backward time: 1.2053, DDP time: 0.0052, Total iter time: 2.4127 [2020-12-31 09:06:57,466][root][INFO] - ===> Epoch[580](2900/5): Loss -0.5676, Sem -0.5676, dir -0.9507, norm 0.3196 LR: 8.242e-02 Score 97.892 Data time: 0.1741, Forward time: 0.9570, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.3696 [2020-12-31 09:07:08,965][root][INFO] - ===> Epoch[581](2905/5): Loss -0.5692, Sem -0.5692, dir -0.9484, norm 0.3132 LR: 8.239e-02 Score 97.807 Data time: 0.1028, Forward time: 0.9568, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.2045 [2020-12-31 09:07:19,658][root][INFO] - ===> Epoch[582](2910/5): Loss -0.5745, Sem -0.5745, dir -0.9509, norm 0.3118 LR: 8.236e-02 Score 97.897 Data time: 0.1673, Forward time: 0.9567, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.0446 [2020-12-31 09:07:31,645][root][INFO] - ===> Epoch[583](2915/5): Loss -0.5626, Sem -0.5626, dir -0.9494, norm 0.3260 LR: 8.233e-02 Score 97.987 Data time: 0.1363, Forward time: 0.9566, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.3098 [2020-12-31 09:07:42,990][root][INFO] - ===> Epoch[584](2920/5): Loss -0.5707, Sem -0.5707, dir -0.9501, norm 0.3148 LR: 8.230e-02 Score 97.846 Data time: 0.0895, Forward time: 0.9565, Backward time: 1.2044, DDP time: 0.0052, Total iter time: 2.1715 [2020-12-31 09:07:55,461][root][INFO] - ===> Epoch[585](2925/5): Loss -0.5666, Sem -0.5666, dir -0.9503, norm 0.3230 LR: 8.227e-02 Score 97.957 Data time: 0.1435, Forward time: 0.9567, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.4021 [2020-12-31 09:08:08,260][root][INFO] - ===> Epoch[586](2930/5): Loss -0.5675, Sem -0.5675, dir -0.9487, norm 0.3194 LR: 8.224e-02 Score 97.945 Data time: 0.0967, Forward time: 0.9566, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.4751 [2020-12-31 09:08:20,781][root][INFO] - ===> Epoch[587](2935/5): Loss -0.5783, Sem -0.5783, dir -0.9507, norm 0.3112 LR: 8.221e-02 Score 97.965 Data time: 0.0993, Forward time: 0.9565, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.4205 [2020-12-31 09:08:32,470][root][INFO] - ===> Epoch[588](2940/5): Loss -0.5663, Sem -0.5663, dir -0.9507, norm 0.3228 LR: 8.217e-02 Score 97.964 Data time: 0.1180, Forward time: 0.9564, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.2462 [2020-12-31 09:08:45,086][root][INFO] - ===> Epoch[589](2945/5): Loss -0.5695, Sem -0.5695, dir -0.9509, norm 0.3200 LR: 8.214e-02 Score 97.973 Data time: 0.1461, Forward time: 0.9565, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.4248 [2020-12-31 09:08:56,404][root][INFO] - ===> Epoch[590](2950/5): Loss -0.5609, Sem -0.5609, dir -0.9492, norm 0.3255 LR: 8.211e-02 Score 97.919 Data time: 0.1336, Forward time: 0.9566, Backward time: 1.2044, DDP time: 0.0052, Total iter time: 2.1711 [2020-12-31 09:09:09,176][root][INFO] - ===> Epoch[591](2955/5): Loss -0.5650, Sem -0.5650, dir -0.9490, norm 0.3219 LR: 8.208e-02 Score 97.938 Data time: 0.0955, Forward time: 0.9566, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.4566 [2020-12-31 09:09:20,507][root][INFO] - ===> Epoch[592](2960/5): Loss -0.5609, Sem -0.5609, dir -0.9499, norm 0.3278 LR: 8.205e-02 Score 97.973 Data time: 0.0987, Forward time: 0.9565, Backward time: 1.2044, DDP time: 0.0052, Total iter time: 2.1711 [2020-12-31 09:09:31,972][root][INFO] - ===> Epoch[593](2965/5): Loss -0.5767, Sem -0.5767, dir -0.9503, norm 0.3124 LR: 8.202e-02 Score 97.973 Data time: 0.1683, Forward time: 0.9566, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.1938 [2020-12-31 09:09:43,935][root][INFO] - ===> Epoch[594](2970/5): Loss -0.5682, Sem -0.5682, dir -0.9506, norm 0.3204 LR: 8.199e-02 Score 97.924 Data time: 0.1409, Forward time: 0.9566, Backward time: 1.2040, DDP time: 0.0052, Total iter time: 2.2964 [2020-12-31 09:09:56,144][root][INFO] - ===> Epoch[595](2975/5): Loss -0.5706, Sem -0.5706, dir -0.9508, norm 0.3178 LR: 8.196e-02 Score 97.914 Data time: 0.1578, Forward time: 0.9566, Backward time: 1.2039, DDP time: 0.0052, Total iter time: 2.3458 [2020-12-31 09:10:07,381][root][INFO] - ===> Epoch[596](2980/5): Loss -0.5806, Sem -0.5806, dir -0.9504, norm 0.3079 LR: 8.193e-02 Score 97.954 Data time: 0.0892, Forward time: 0.9566, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.1556 [2020-12-31 09:10:19,804][root][INFO] - ===> Epoch[597](2985/5): Loss -0.5685, Sem -0.5685, dir -0.9504, norm 0.3219 LR: 8.190e-02 Score 98.015 Data time: 0.1354, Forward time: 0.9567, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.3877 [2020-12-31 09:10:30,411][root][INFO] - ===> Epoch[598](2990/5): Loss -0.5766, Sem -0.5766, dir -0.9511, norm 0.3112 LR: 8.187e-02 Score 97.895 Data time: 0.0816, Forward time: 0.9566, Backward time: 1.2034, DDP time: 0.0052, Total iter time: 2.0258 [2020-12-31 09:10:42,019][root][INFO] - ===> Epoch[599](2995/5): Loss -0.5863, Sem -0.5863, dir -0.9512, norm 0.3059 LR: 8.184e-02 Score 98.038 Data time: 0.0832, Forward time: 0.9564, Backward time: 1.2034, DDP time: 0.0052, Total iter time: 2.2346 [2020-12-31 09:10:54,398][root][INFO] - ===> Epoch[600](3000/5): Loss -0.5716, Sem -0.5716, dir -0.9504, norm 0.3172 LR: 8.181e-02 Score 97.962 Data time: 0.1282, Forward time: 0.9564, Backward time: 1.2034, DDP time: 0.0052, Total iter time: 2.3772 [2020-12-31 09:10:54,402][root][INFO] - ===> Start testing [2020-12-31 09:11:32,188][root][INFO] - 68/68: Data time: 0.0022, Iter time: 0.1089 Loss 0.324 (AVG: 0.476) Score 90.534 (AVG: 89.179) mIOU 67.815 mAP 84.052 mAcc 75.965 IOU: 59.906 0.000 80.297 72.959 92.903 88.879 31.240 72.030 98.322 68.501 80.382 83.712 52.468 mAP: 81.190 2.992 93.210 87.993 98.810 98.199 70.214 93.932 99.819 89.486 93.761 94.451 88.619 mAcc: 78.376 0.000 86.722 80.072 94.302 94.129 41.788 89.180 99.386 80.929 90.641 93.979 58.047 [2020-12-31 09:11:32,196][root][INFO] - Finished test. Elapsed time: 37.7855 [2020-12-31 09:11:35,795][root][INFO] - [2020-12-31 09:11:35,795][root][INFO] - ################################################################ [2020-12-31 09:11:35,795][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 09:11:35,795][root][INFO] - ################################################################ [2020-12-31 09:11:35,795][root][INFO] - clutter : nan nan nan [2020-12-31 09:11:35,795][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 09:11:35,795][root][INFO] - board : 0.693 0.747 0.857 [2020-12-31 09:11:35,795][root][INFO] - bookcase : 0.268 0.432 0.543 [2020-12-31 09:11:35,795][root][INFO] - ceiling : 0.743 0.785 0.837 [2020-12-31 09:11:35,795][root][INFO] - chair : 0.757 0.861 0.879 [2020-12-31 09:11:35,795][root][INFO] - column : 0.097 0.180 0.317 [2020-12-31 09:11:35,795][root][INFO] - door : 0.519 0.733 0.853 [2020-12-31 09:11:35,795][root][INFO] - floor : 1.000 1.000 1.000 [2020-12-31 09:11:35,795][root][INFO] - sofa : 0.443 0.727 0.727 [2020-12-31 09:11:35,795][root][INFO] - table : 0.199 0.377 0.608 [2020-12-31 09:11:35,795][root][INFO] - wall : 0.437 0.633 0.754 [2020-12-31 09:11:35,796][root][INFO] - window : 0.426 0.681 0.753 [2020-12-31 09:11:35,796][root][INFO] - ---------------------------------------------------------------- [2020-12-31 09:11:35,796][root][INFO] - average : 0.465 0.596 0.677 [2020-12-31 09:11:35,796][root][INFO] - [2020-12-31 09:11:35,965][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 09:11:36,620][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2020-12-31 09:11:36,620][root][INFO] - Current best mIoU: 67.815 at iter 1 [2020-12-31 09:11:37,231][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter3000.pth [2020-12-31 09:11:48,548][root][INFO] - ===> Epoch[601](3005/5): Loss -0.5701, Sem -0.5701, dir -0.9505, norm 0.3166 LR: 8.178e-02 Score 97.874 Data time: 0.1107, Forward time: 0.9564, Backward time: 1.2032, DDP time: 0.0052, Total iter time: 2.1516 [2020-12-31 09:12:00,753][root][INFO] - ===> Epoch[602](3010/5): Loss -0.5853, Sem -0.5853, dir -0.9530, norm 0.3096 LR: 8.175e-02 Score 98.050 Data time: 0.1481, Forward time: 0.9563, Backward time: 1.2032, DDP time: 0.0052, Total iter time: 2.3549 [2020-12-31 09:12:12,010][root][INFO] - ===> Epoch[603](3015/5): Loss -0.5817, Sem -0.5817, dir -0.9510, norm 0.3097 LR: 8.171e-02 Score 98.011 Data time: 0.1169, Forward time: 0.9561, Backward time: 1.2031, DDP time: 0.0052, Total iter time: 2.1553 [2020-12-31 09:12:24,590][root][INFO] - ===> Epoch[604](3020/5): Loss -0.5625, Sem -0.5625, dir -0.9490, norm 0.3258 LR: 8.168e-02 Score 97.974 Data time: 0.1159, Forward time: 0.9561, Backward time: 1.2032, DDP time: 0.0052, Total iter time: 2.4112 [2020-12-31 09:12:36,977][root][INFO] - ===> Epoch[605](3025/5): Loss -0.5855, Sem -0.5855, dir -0.9527, norm 0.3098 LR: 8.165e-02 Score 98.106 Data time: 0.2036, Forward time: 0.9562, Backward time: 1.2031, DDP time: 0.0052, Total iter time: 2.3919 [2020-12-31 09:12:49,372][root][INFO] - ===> Epoch[606](3030/5): Loss -0.5787, Sem -0.5787, dir -0.9507, norm 0.3130 LR: 8.162e-02 Score 98.045 Data time: 0.1780, Forward time: 0.9562, Backward time: 1.2031, DDP time: 0.0052, Total iter time: 2.3806 [2020-12-31 09:13:01,880][root][INFO] - ===> Epoch[607](3035/5): Loss -0.5851, Sem -0.5851, dir -0.9515, norm 0.3075 LR: 8.159e-02 Score 98.043 Data time: 0.0932, Forward time: 0.9560, Backward time: 1.2034, DDP time: 0.0052, Total iter time: 2.4134 [2020-12-31 09:13:14,408][root][INFO] - ===> Epoch[608](3040/5): Loss -0.5712, Sem -0.5712, dir -0.9512, norm 0.3190 LR: 8.156e-02 Score 97.986 Data time: 0.0952, Forward time: 0.9560, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.4076 [2020-12-31 09:13:27,038][root][INFO] - ===> Epoch[609](3045/5): Loss -0.5892, Sem -0.5892, dir -0.9516, norm 0.3025 LR: 8.153e-02 Score 98.035 Data time: 0.1355, Forward time: 0.9559, Backward time: 1.2036, DDP time: 0.0052, Total iter time: 2.4322 [2020-12-31 09:13:38,617][root][INFO] - ===> Epoch[610](3050/5): Loss -0.5733, Sem -0.5733, dir -0.9512, norm 0.3161 LR: 8.150e-02 Score 97.967 Data time: 0.0899, Forward time: 0.9558, Backward time: 1.2036, DDP time: 0.0052, Total iter time: 2.2220 [2020-12-31 09:13:50,887][root][INFO] - ===> Epoch[611](3055/5): Loss -0.5839, Sem -0.5839, dir -0.9527, norm 0.3104 LR: 8.147e-02 Score 98.076 Data time: 0.1016, Forward time: 0.9557, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.3588 [2020-12-31 09:14:02,441][root][INFO] - ===> Epoch[612](3060/5): Loss -0.5743, Sem -0.5743, dir -0.9515, norm 0.3164 LR: 8.144e-02 Score 97.983 Data time: 0.0727, Forward time: 0.9554, Backward time: 1.2039, DDP time: 0.0052, Total iter time: 2.2282 [2020-12-31 09:14:14,641][root][INFO] - ===> Epoch[613](3065/5): Loss -0.5815, Sem -0.5815, dir -0.9517, norm 0.3091 LR: 8.141e-02 Score 97.991 Data time: 0.0877, Forward time: 0.9553, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.3559 [2020-12-31 09:14:27,469][root][INFO] - ===> Epoch[614](3070/5): Loss -0.5813, Sem -0.5813, dir -0.9507, norm 0.3079 LR: 8.138e-02 Score 97.964 Data time: 0.1108, Forward time: 0.9553, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.4706 [2020-12-31 09:14:40,011][root][INFO] - ===> Epoch[615](3075/5): Loss -0.5908, Sem -0.5908, dir -0.9525, norm 0.3008 LR: 8.135e-02 Score 97.985 Data time: 0.1097, Forward time: 0.9552, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.4126 [2020-12-31 09:14:52,420][root][INFO] - ===> Epoch[616](3080/5): Loss -0.5754, Sem -0.5754, dir -0.9502, norm 0.3136 LR: 8.132e-02 Score 97.968 Data time: 0.0787, Forward time: 0.9550, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.3925 [2020-12-31 09:15:03,455][root][INFO] - ===> Epoch[617](3085/5): Loss -0.5763, Sem -0.5763, dir -0.9515, norm 0.3129 LR: 8.128e-02 Score 97.963 Data time: 0.1044, Forward time: 0.9549, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.1269 [2020-12-31 09:15:15,222][root][INFO] - ===> Epoch[618](3090/5): Loss -0.5400, Sem -0.5400, dir -0.9482, norm 0.3324 LR: 8.125e-02 Score 97.505 Data time: 0.0732, Forward time: 0.9547, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.2665 [2020-12-31 09:15:26,819][root][INFO] - ===> Epoch[619](3095/5): Loss -0.5479, Sem -0.5479, dir -0.9469, norm 0.3251 LR: 8.122e-02 Score 97.574 Data time: 0.0920, Forward time: 0.9545, Backward time: 1.2047, DDP time: 0.0052, Total iter time: 2.2262 [2020-12-31 09:15:37,992][root][INFO] - ===> Epoch[620](3100/5): Loss -0.5560, Sem -0.5560, dir -0.9489, norm 0.3202 LR: 8.119e-02 Score 97.619 Data time: 0.1398, Forward time: 0.9544, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.1403 [2020-12-31 09:15:49,728][root][INFO] - ===> Epoch[621](3105/5): Loss -0.5509, Sem -0.5509, dir -0.9495, norm 0.3290 LR: 8.116e-02 Score 97.702 Data time: 0.1019, Forward time: 0.9544, Backward time: 1.2044, DDP time: 0.0052, Total iter time: 2.2477 [2020-12-31 09:16:01,806][root][INFO] - ===> Epoch[622](3110/5): Loss -0.5537, Sem -0.5537, dir -0.9496, norm 0.3277 LR: 8.113e-02 Score 97.737 Data time: 0.1763, Forward time: 0.9544, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.3423 [2020-12-31 09:16:14,155][root][INFO] - ===> Epoch[623](3115/5): Loss -0.5606, Sem -0.5606, dir -0.9497, norm 0.3235 LR: 8.110e-02 Score 97.830 Data time: 0.1124, Forward time: 0.9543, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.3939 [2020-12-31 09:16:26,908][root][INFO] - ===> Epoch[624](3120/5): Loss -0.5531, Sem -0.5531, dir -0.9485, norm 0.3309 LR: 8.107e-02 Score 97.876 Data time: 0.1957, Forward time: 0.9544, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.4556 [2020-12-31 09:16:38,622][root][INFO] - ===> Epoch[625](3125/5): Loss -0.5708, Sem -0.5708, dir -0.9495, norm 0.3124 LR: 8.104e-02 Score 97.793 Data time: 0.0947, Forward time: 0.9543, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.2575 [2020-12-31 09:16:50,803][root][INFO] - ===> Epoch[626](3130/5): Loss -0.5533, Sem -0.5533, dir -0.9491, norm 0.3222 LR: 8.101e-02 Score 97.644 Data time: 0.1162, Forward time: 0.9542, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.3488 [2020-12-31 09:17:02,119][root][INFO] - ===> Epoch[627](3135/5): Loss -0.5443, Sem -0.5443, dir -0.9486, norm 0.3265 LR: 8.098e-02 Score 97.434 Data time: 0.0848, Forward time: 0.9541, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.1679 [2020-12-31 09:17:14,494][root][INFO] - ===> Epoch[628](3140/5): Loss -0.5648, Sem -0.5648, dir -0.9497, norm 0.3108 LR: 8.095e-02 Score 97.547 Data time: 0.1142, Forward time: 0.9540, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.3904 [2020-12-31 09:17:26,255][root][INFO] - ===> Epoch[629](3145/5): Loss -0.5575, Sem -0.5575, dir -0.9497, norm 0.3256 LR: 8.092e-02 Score 97.783 Data time: 0.1378, Forward time: 0.9540, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.2638 [2020-12-31 09:17:39,292][root][INFO] - ===> Epoch[630](3150/5): Loss -0.5597, Sem -0.5597, dir -0.9498, norm 0.3257 LR: 8.089e-02 Score 97.847 Data time: 0.1003, Forward time: 0.9540, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.5133 [2020-12-31 09:17:50,816][root][INFO] - ===> Epoch[631](3155/5): Loss -0.5691, Sem -0.5691, dir -0.9493, norm 0.3150 LR: 8.085e-02 Score 97.820 Data time: 0.0814, Forward time: 0.9539, Backward time: 1.2047, DDP time: 0.0052, Total iter time: 2.2113 [2020-12-31 09:18:02,936][root][INFO] - ===> Epoch[632](3160/5): Loss -0.5761, Sem -0.5761, dir -0.9511, norm 0.3157 LR: 8.082e-02 Score 98.024 Data time: 0.0976, Forward time: 0.9539, Backward time: 1.2047, DDP time: 0.0052, Total iter time: 2.3272 [2020-12-31 09:18:14,985][root][INFO] - ===> Epoch[633](3165/5): Loss -0.5706, Sem -0.5706, dir -0.9515, norm 0.3212 LR: 8.079e-02 Score 98.010 Data time: 0.0846, Forward time: 0.9539, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.3212 [2020-12-31 09:18:26,954][root][INFO] - ===> Epoch[634](3170/5): Loss -0.5732, Sem -0.5732, dir -0.9504, norm 0.3145 LR: 8.076e-02 Score 97.901 Data time: 0.0796, Forward time: 0.9537, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.3116 [2020-12-31 09:18:39,303][root][INFO] - ===> Epoch[635](3175/5): Loss -0.5758, Sem -0.5758, dir -0.9503, norm 0.3145 LR: 8.073e-02 Score 97.989 Data time: 0.1795, Forward time: 0.9536, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.3716 [2020-12-31 09:18:50,826][root][INFO] - ===> Epoch[636](3180/5): Loss -0.5832, Sem -0.5832, dir -0.9496, norm 0.3073 LR: 8.070e-02 Score 98.033 Data time: 0.1006, Forward time: 0.9535, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.2273 [2020-12-31 09:19:02,834][root][INFO] - ===> Epoch[637](3185/5): Loss -0.5862, Sem -0.5862, dir -0.9515, norm 0.3032 LR: 8.067e-02 Score 97.930 Data time: 0.0931, Forward time: 0.9534, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.3073 [2020-12-31 09:19:14,739][root][INFO] - ===> Epoch[638](3190/5): Loss -0.5919, Sem -0.5919, dir -0.9532, norm 0.3039 LR: 8.064e-02 Score 98.090 Data time: 0.1131, Forward time: 0.9534, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.2889 [2020-12-31 09:19:27,019][root][INFO] - ===> Epoch[639](3195/5): Loss -0.5908, Sem -0.5908, dir -0.9531, norm 0.3038 LR: 8.061e-02 Score 98.053 Data time: 0.0733, Forward time: 0.9531, Backward time: 1.2053, DDP time: 0.0052, Total iter time: 2.3740 [2020-12-31 09:19:39,244][root][INFO] - ===> Epoch[640](3200/5): Loss -0.5876, Sem -0.5876, dir -0.9525, norm 0.3032 LR: 8.058e-02 Score 97.955 Data time: 0.1931, Forward time: 0.9532, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.3498 [2020-12-31 09:19:39,247][root][INFO] - ===> Start testing [2020-12-31 09:20:16,151][root][INFO] - 68/68: Data time: 0.0022, Iter time: 0.1384 Loss 0.164 (AVG: 0.485) Score 95.333 (AVG: 88.927) mIOU 66.991 mAP 83.333 mAcc 74.457 IOU: 58.196 0.000 76.418 71.916 92.333 87.771 28.751 70.783 98.486 67.428 80.121 83.827 54.850 mAP: 81.484 4.248 93.115 85.708 98.801 97.943 63.139 92.402 99.815 90.489 93.212 93.346 89.620 mAcc: 79.566 0.000 83.285 78.789 93.864 94.149 35.113 86.809 99.256 71.034 88.916 94.078 63.087 [2020-12-31 09:20:16,152][root][INFO] - Finished test. Elapsed time: 36.9025 [2020-12-31 09:20:19,489][root][INFO] - [2020-12-31 09:20:19,490][root][INFO] - ################################################################ [2020-12-31 09:20:19,490][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 09:20:19,490][root][INFO] - ################################################################ [2020-12-31 09:20:19,490][root][INFO] - clutter : nan nan nan [2020-12-31 09:20:19,490][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 09:20:19,490][root][INFO] - board : 0.700 0.749 0.852 [2020-12-31 09:20:19,490][root][INFO] - bookcase : 0.222 0.363 0.478 [2020-12-31 09:20:19,490][root][INFO] - ceiling : 0.732 0.760 0.813 [2020-12-31 09:20:19,490][root][INFO] - chair : 0.751 0.851 0.875 [2020-12-31 09:20:19,490][root][INFO] - column : 0.110 0.174 0.295 [2020-12-31 09:20:19,490][root][INFO] - door : 0.549 0.736 0.894 [2020-12-31 09:20:19,490][root][INFO] - floor : 1.000 1.000 1.000 [2020-12-31 09:20:19,490][root][INFO] - sofa : 0.315 0.542 0.818 [2020-12-31 09:20:19,490][root][INFO] - table : 0.162 0.336 0.615 [2020-12-31 09:20:19,490][root][INFO] - wall : 0.469 0.674 0.771 [2020-12-31 09:20:19,490][root][INFO] - window : 0.430 0.745 0.771 [2020-12-31 09:20:19,490][root][INFO] - ---------------------------------------------------------------- [2020-12-31 09:20:19,490][root][INFO] - average : 0.453 0.578 0.682 [2020-12-31 09:20:19,490][root][INFO] - [2020-12-31 09:20:19,646][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 09:20:20,228][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter3200.pth [2020-12-31 09:20:32,146][root][INFO] - ===> Epoch[641](3205/5): Loss -0.5988, Sem -0.5988, dir -0.9535, norm 0.2993 LR: 8.055e-02 Score 98.147 Data time: 0.1463, Forward time: 0.9531, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.2706 [2020-12-31 09:20:44,133][root][INFO] - ===> Epoch[642](3210/5): Loss -0.5834, Sem -0.5834, dir -0.9522, norm 0.3074 LR: 8.052e-02 Score 97.966 Data time: 0.1210, Forward time: 0.9530, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.3158 [2020-12-31 09:20:56,447][root][INFO] - ===> Epoch[643](3215/5): Loss -0.5963, Sem -0.5963, dir -0.9536, norm 0.3020 LR: 8.049e-02 Score 98.159 Data time: 0.1806, Forward time: 0.9531, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.3778 [2020-12-31 09:21:08,354][root][INFO] - ===> Epoch[644](3220/5): Loss -0.5890, Sem -0.5890, dir -0.9530, norm 0.3040 LR: 8.046e-02 Score 98.010 Data time: 0.0930, Forward time: 0.9530, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.2874 [2020-12-31 09:21:20,560][root][INFO] - ===> Epoch[645](3225/5): Loss -0.5980, Sem -0.5980, dir -0.9544, norm 0.2998 LR: 8.042e-02 Score 98.121 Data time: 0.1068, Forward time: 0.9530, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.3497 [2020-12-31 09:21:32,657][root][INFO] - ===> Epoch[646](3230/5): Loss -0.5881, Sem -0.5881, dir -0.9524, norm 0.3055 LR: 8.039e-02 Score 98.083 Data time: 0.1710, Forward time: 0.9529, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.3226 [2020-12-31 09:21:44,622][root][INFO] - ===> Epoch[647](3235/5): Loss -0.5851, Sem -0.5851, dir -0.9520, norm 0.3063 LR: 8.036e-02 Score 98.000 Data time: 0.1488, Forward time: 0.9529, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.3023 [2020-12-31 09:21:57,174][root][INFO] - ===> Epoch[648](3240/5): Loss -0.6004, Sem -0.6004, dir -0.9557, norm 0.3011 LR: 8.033e-02 Score 98.218 Data time: 0.0774, Forward time: 0.9528, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.4238 [2020-12-31 09:22:09,163][root][INFO] - ===> Epoch[649](3245/5): Loss -0.5856, Sem -0.5856, dir -0.9525, norm 0.3081 LR: 8.030e-02 Score 98.054 Data time: 0.1060, Forward time: 0.9528, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.3144 [2020-12-31 09:22:20,907][root][INFO] - ===> Epoch[650](3250/5): Loss -0.6104, Sem -0.6104, dir -0.9542, norm 0.2880 LR: 8.027e-02 Score 98.128 Data time: 0.1398, Forward time: 0.9528, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.2635 [2020-12-31 09:22:32,510][root][INFO] - ===> Epoch[651](3255/5): Loss -0.5897, Sem -0.5897, dir -0.9519, norm 0.3044 LR: 8.024e-02 Score 98.075 Data time: 0.0900, Forward time: 0.9528, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.2315 [2020-12-31 09:22:44,949][root][INFO] - ===> Epoch[652](3260/5): Loss -0.5938, Sem -0.5938, dir -0.9534, norm 0.2991 LR: 8.021e-02 Score 97.998 Data time: 0.1296, Forward time: 0.9529, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.4007 [2020-12-31 09:22:56,714][root][INFO] - ===> Epoch[653](3265/5): Loss -0.5926, Sem -0.5926, dir -0.9532, norm 0.3038 LR: 8.018e-02 Score 98.149 Data time: 0.1408, Forward time: 0.9530, Backward time: 1.2047, DDP time: 0.0052, Total iter time: 2.2527 [2020-12-31 09:23:08,411][root][INFO] - ===> Epoch[654](3270/5): Loss -0.5972, Sem -0.5972, dir -0.9551, norm 0.3010 LR: 8.015e-02 Score 98.099 Data time: 0.0969, Forward time: 0.9530, Backward time: 1.2047, DDP time: 0.0052, Total iter time: 2.2533 [2020-12-31 09:23:20,344][root][INFO] - ===> Epoch[655](3275/5): Loss -0.5973, Sem -0.5973, dir -0.9529, norm 0.2966 LR: 8.012e-02 Score 98.061 Data time: 0.0856, Forward time: 0.9528, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.2887 [2020-12-31 09:23:31,931][root][INFO] - ===> Epoch[656](3280/5): Loss -0.5912, Sem -0.5912, dir -0.9540, norm 0.3025 LR: 8.009e-02 Score 98.010 Data time: 0.0904, Forward time: 0.9526, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.2316 [2020-12-31 09:23:43,803][root][INFO] - ===> Epoch[657](3285/5): Loss -0.5994, Sem -0.5994, dir -0.9547, norm 0.2987 LR: 8.006e-02 Score 98.115 Data time: 0.1180, Forward time: 0.9526, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.2795 [2020-12-31 09:23:55,470][root][INFO] - ===> Epoch[658](3290/5): Loss -0.5973, Sem -0.5973, dir -0.9543, norm 0.2953 LR: 8.003e-02 Score 97.976 Data time: 0.0825, Forward time: 0.9525, Backward time: 1.2047, DDP time: 0.0052, Total iter time: 2.2570 [2020-12-31 09:24:07,583][root][INFO] - ===> Epoch[659](3295/5): Loss -0.5747, Sem -0.5747, dir -0.9521, norm 0.3191 LR: 7.999e-02 Score 98.054 Data time: 0.1215, Forward time: 0.9525, Backward time: 1.2047, DDP time: 0.0052, Total iter time: 2.3270 [2020-12-31 09:24:19,490][root][INFO] - ===> Epoch[660](3300/5): Loss -0.5984, Sem -0.5984, dir -0.9541, norm 0.2985 LR: 7.996e-02 Score 98.096 Data time: 0.0907, Forward time: 0.9524, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.2912 [2020-12-31 09:24:31,568][root][INFO] - ===> Epoch[661](3305/5): Loss -0.5859, Sem -0.5859, dir -0.9524, norm 0.3054 LR: 7.993e-02 Score 98.026 Data time: 0.0905, Forward time: 0.9523, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.3278 [2020-12-31 09:24:43,472][root][INFO] - ===> Epoch[662](3310/5): Loss -0.5885, Sem -0.5885, dir -0.9541, norm 0.3047 LR: 7.990e-02 Score 97.979 Data time: 0.1654, Forward time: 0.9523, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.2813 [2020-12-31 09:24:56,290][root][INFO] - ===> Epoch[663](3315/5): Loss -0.5894, Sem -0.5894, dir -0.9540, norm 0.3054 LR: 7.987e-02 Score 98.060 Data time: 0.1860, Forward time: 0.9526, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.4629 [2020-12-31 09:25:08,078][root][INFO] - ===> Epoch[664](3320/5): Loss -0.5960, Sem -0.5960, dir -0.9553, norm 0.3028 LR: 7.984e-02 Score 98.115 Data time: 0.1722, Forward time: 0.9525, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.2644 [2020-12-31 09:25:19,980][root][INFO] - ===> Epoch[665](3325/5): Loss -0.5864, Sem -0.5864, dir -0.9524, norm 0.3053 LR: 7.981e-02 Score 98.008 Data time: 0.1037, Forward time: 0.9524, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.2811 [2020-12-31 09:25:31,541][root][INFO] - ===> Epoch[666](3330/5): Loss -0.6060, Sem -0.6060, dir -0.9550, norm 0.2940 LR: 7.978e-02 Score 98.157 Data time: 0.1114, Forward time: 0.9524, Backward time: 1.2044, DDP time: 0.0052, Total iter time: 2.2188 [2020-12-31 09:25:44,905][root][INFO] - ===> Epoch[667](3335/5): Loss -0.5919, Sem -0.5919, dir -0.9538, norm 0.3033 LR: 7.975e-02 Score 98.079 Data time: 0.1609, Forward time: 0.9525, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.5629 [2020-12-31 09:25:56,991][root][INFO] - ===> Epoch[668](3340/5): Loss -0.5953, Sem -0.5953, dir -0.9548, norm 0.2994 LR: 7.972e-02 Score 98.018 Data time: 0.0942, Forward time: 0.9524, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.3227 [2020-12-31 09:26:08,271][root][INFO] - ===> Epoch[669](3345/5): Loss -0.6077, Sem -0.6077, dir -0.9557, norm 0.2941 LR: 7.969e-02 Score 98.213 Data time: 0.1038, Forward time: 0.9523, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.1621 [2020-12-31 09:26:20,250][root][INFO] - ===> Epoch[670](3350/5): Loss -0.5973, Sem -0.5973, dir -0.9535, norm 0.2975 LR: 7.966e-02 Score 98.087 Data time: 0.0725, Forward time: 0.9522, Backward time: 1.2047, DDP time: 0.0052, Total iter time: 2.3215 [2020-12-31 09:26:32,870][root][INFO] - ===> Epoch[671](3355/5): Loss -0.5906, Sem -0.5906, dir -0.9532, norm 0.3038 LR: 7.963e-02 Score 98.025 Data time: 0.1145, Forward time: 0.9522, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.4295 [2020-12-31 09:26:45,014][root][INFO] - ===> Epoch[672](3360/5): Loss -0.5879, Sem -0.5879, dir -0.9530, norm 0.3074 LR: 7.959e-02 Score 98.098 Data time: 0.1067, Forward time: 0.9522, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.3348 [2020-12-31 09:26:57,152][root][INFO] - ===> Epoch[673](3365/5): Loss -0.5973, Sem -0.5973, dir -0.9545, norm 0.3022 LR: 7.956e-02 Score 98.158 Data time: 0.0915, Forward time: 0.9522, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.3340 [2020-12-31 09:27:08,936][root][INFO] - ===> Epoch[674](3370/5): Loss -0.5840, Sem -0.5840, dir -0.9543, norm 0.3158 LR: 7.953e-02 Score 98.185 Data time: 0.1063, Forward time: 0.9521, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.2662 [2020-12-31 09:27:21,168][root][INFO] - ===> Epoch[675](3375/5): Loss -0.5891, Sem -0.5891, dir -0.9540, norm 0.3081 LR: 7.950e-02 Score 98.099 Data time: 0.1007, Forward time: 0.9520, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.3693 [2020-12-31 09:27:33,288][root][INFO] - ===> Epoch[676](3380/5): Loss -0.6009, Sem -0.6009, dir -0.9542, norm 0.2988 LR: 7.947e-02 Score 98.203 Data time: 0.1243, Forward time: 0.9520, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.3216 [2020-12-31 09:27:45,429][root][INFO] - ===> Epoch[677](3385/5): Loss -0.5943, Sem -0.5943, dir -0.9523, norm 0.3006 LR: 7.944e-02 Score 98.078 Data time: 0.0873, Forward time: 0.9519, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.3395 [2020-12-31 09:27:57,305][root][INFO] - ===> Epoch[678](3390/5): Loss -0.5998, Sem -0.5998, dir -0.9541, norm 0.2952 LR: 7.941e-02 Score 98.017 Data time: 0.1469, Forward time: 0.9519, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.2852 [2020-12-31 09:28:08,569][root][INFO] - ===> Epoch[679](3395/5): Loss -0.6026, Sem -0.6026, dir -0.9544, norm 0.2930 LR: 7.938e-02 Score 98.062 Data time: 0.1076, Forward time: 0.9519, Backward time: 1.2047, DDP time: 0.0052, Total iter time: 2.1605 [2020-12-31 09:28:19,534][root][INFO] - ===> Epoch[680](3400/5): Loss -0.5985, Sem -0.5985, dir -0.9533, norm 0.2935 LR: 7.935e-02 Score 97.987 Data time: 0.0933, Forward time: 0.9518, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.1067 [2020-12-31 09:28:19,537][root][INFO] - ===> Start testing [2020-12-31 09:28:57,124][root][INFO] - 68/68: Data time: 0.0020, Iter time: 0.0985 Loss 0.201 (AVG: 0.505) Score 94.259 (AVG: 88.858) mIOU 66.175 mAP 82.430 mAcc 72.943 IOU: 57.691 0.000 76.455 72.322 93.081 89.780 22.853 68.678 98.048 66.645 79.196 83.861 51.664 mAP: 81.497 2.993 93.756 86.214 99.007 97.862 58.534 91.023 99.819 87.490 92.162 93.279 87.961 mAcc: 78.692 0.000 85.763 81.047 94.286 94.319 26.357 82.271 99.161 69.288 86.348 95.290 55.435 [2020-12-31 09:28:57,125][root][INFO] - Finished test. Elapsed time: 37.5853 [2020-12-31 09:29:00,343][root][INFO] - [2020-12-31 09:29:00,343][root][INFO] - ################################################################ [2020-12-31 09:29:00,343][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 09:29:00,343][root][INFO] - ################################################################ [2020-12-31 09:29:00,343][root][INFO] - clutter : nan nan nan [2020-12-31 09:29:00,343][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 09:29:00,343][root][INFO] - board : 0.748 0.831 0.878 [2020-12-31 09:29:00,343][root][INFO] - bookcase : 0.238 0.354 0.515 [2020-12-31 09:29:00,344][root][INFO] - ceiling : 0.681 0.739 0.784 [2020-12-31 09:29:00,344][root][INFO] - chair : 0.765 0.857 0.867 [2020-12-31 09:29:00,344][root][INFO] - column : 0.120 0.167 0.259 [2020-12-31 09:29:00,344][root][INFO] - door : 0.537 0.734 0.881 [2020-12-31 09:29:00,344][root][INFO] - floor : 1.000 1.000 1.000 [2020-12-31 09:29:00,344][root][INFO] - sofa : 0.309 0.509 0.727 [2020-12-31 09:29:00,344][root][INFO] - table : 0.166 0.323 0.624 [2020-12-31 09:29:00,344][root][INFO] - wall : 0.494 0.710 0.810 [2020-12-31 09:29:00,344][root][INFO] - window : 0.366 0.682 0.759 [2020-12-31 09:29:00,344][root][INFO] - ---------------------------------------------------------------- [2020-12-31 09:29:00,344][root][INFO] - average : 0.452 0.575 0.675 [2020-12-31 09:29:00,344][root][INFO] - [2020-12-31 09:29:00,522][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 09:29:01,090][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter3400.pth [2020-12-31 09:29:13,351][root][INFO] - ===> Epoch[681](3405/5): Loss -0.5898, Sem -0.5898, dir -0.9530, norm 0.3048 LR: 7.932e-02 Score 98.071 Data time: 0.0959, Forward time: 0.9518, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.3356 [2020-12-31 09:29:26,175][root][INFO] - ===> Epoch[682](3410/5): Loss -0.6086, Sem -0.6086, dir -0.9561, norm 0.2901 LR: 7.929e-02 Score 98.099 Data time: 0.1506, Forward time: 0.9520, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.4758 [2020-12-31 09:29:38,340][root][INFO] - ===> Epoch[683](3415/5): Loss -0.6000, Sem -0.6000, dir -0.9548, norm 0.2978 LR: 7.926e-02 Score 98.113 Data time: 0.1087, Forward time: 0.9522, Backward time: 1.2044, DDP time: 0.0052, Total iter time: 2.3440 [2020-12-31 09:29:50,501][root][INFO] - ===> Epoch[684](3420/5): Loss -0.5931, Sem -0.5931, dir -0.9538, norm 0.3020 LR: 7.923e-02 Score 98.074 Data time: 0.0966, Forward time: 0.9524, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.3502 [2020-12-31 09:30:02,266][root][INFO] - ===> Epoch[685](3425/5): Loss -0.6121, Sem -0.6121, dir -0.9545, norm 0.2856 LR: 7.919e-02 Score 98.100 Data time: 0.0900, Forward time: 0.9522, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.2720 [2020-12-31 09:30:14,961][root][INFO] - ===> Epoch[686](3430/5): Loss -0.6048, Sem -0.6048, dir -0.9538, norm 0.2919 LR: 7.916e-02 Score 98.086 Data time: 0.0817, Forward time: 0.9522, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.4539 [2020-12-31 09:30:26,440][root][INFO] - ===> Epoch[687](3435/5): Loss -0.6140, Sem -0.6140, dir -0.9564, norm 0.2863 LR: 7.913e-02 Score 98.117 Data time: 0.1321, Forward time: 0.9522, Backward time: 1.2044, DDP time: 0.0052, Total iter time: 2.2069 [2020-12-31 09:30:38,231][root][INFO] - ===> Epoch[688](3440/5): Loss -0.6111, Sem -0.6111, dir -0.9542, norm 0.2872 LR: 7.910e-02 Score 98.126 Data time: 0.0869, Forward time: 0.9520, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.2800 [2020-12-31 09:30:51,004][root][INFO] - ===> Epoch[689](3445/5): Loss -0.6037, Sem -0.6037, dir -0.9549, norm 0.2972 LR: 7.907e-02 Score 98.199 Data time: 0.0948, Forward time: 0.9518, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.4624 [2020-12-31 09:31:02,917][root][INFO] - ===> Epoch[690](3450/5): Loss -0.5943, Sem -0.5943, dir -0.9542, norm 0.3056 LR: 7.904e-02 Score 98.206 Data time: 0.1125, Forward time: 0.9517, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.2954 [2020-12-31 09:31:15,256][root][INFO] - ===> Epoch[691](3455/5): Loss -0.6008, Sem -0.6008, dir -0.9551, norm 0.2984 LR: 7.901e-02 Score 98.147 Data time: 0.1384, Forward time: 0.9517, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.3619 [2020-12-31 09:31:26,985][root][INFO] - ===> Epoch[692](3460/5): Loss -0.5950, Sem -0.5950, dir -0.9546, norm 0.3039 LR: 7.898e-02 Score 98.149 Data time: 0.1029, Forward time: 0.9517, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.2665 [2020-12-31 09:31:39,410][root][INFO] - ===> Epoch[693](3465/5): Loss -0.5994, Sem -0.5994, dir -0.9559, norm 0.3026 LR: 7.895e-02 Score 98.197 Data time: 0.0929, Forward time: 0.9516, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.3899 [2020-12-31 09:31:51,932][root][INFO] - ===> Epoch[694](3470/5): Loss -0.6051, Sem -0.6051, dir -0.9557, norm 0.2965 LR: 7.892e-02 Score 98.210 Data time: 0.0998, Forward time: 0.9517, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.4018 [2020-12-31 09:32:04,057][root][INFO] - ===> Epoch[695](3475/5): Loss -0.5990, Sem -0.5990, dir -0.9541, norm 0.2997 LR: 7.889e-02 Score 98.142 Data time: 0.0924, Forward time: 0.9516, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.3322 [2020-12-31 09:32:15,877][root][INFO] - ===> Epoch[696](3480/5): Loss -0.6092, Sem -0.6092, dir -0.9572, norm 0.2956 LR: 7.886e-02 Score 98.272 Data time: 0.0910, Forward time: 0.9516, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.2807 [2020-12-31 09:32:27,776][root][INFO] - ===> Epoch[697](3485/5): Loss -0.6027, Sem -0.6027, dir -0.9561, norm 0.2999 LR: 7.882e-02 Score 98.223 Data time: 0.1178, Forward time: 0.9516, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.2842 [2020-12-31 09:32:39,834][root][INFO] - ===> Epoch[698](3490/5): Loss -0.6081, Sem -0.6081, dir -0.9549, norm 0.2899 LR: 7.879e-02 Score 98.134 Data time: 0.1645, Forward time: 0.9516, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.3245 [2020-12-31 09:32:51,511][root][INFO] - ===> Epoch[699](3495/5): Loss -0.6076, Sem -0.6076, dir -0.9553, norm 0.2929 LR: 7.876e-02 Score 98.169 Data time: 0.1078, Forward time: 0.9516, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.2493 [2020-12-31 09:33:02,142][root][INFO] - ===> Epoch[700](3500/5): Loss -0.5992, Sem -0.5992, dir -0.9541, norm 0.2982 LR: 7.873e-02 Score 98.147 Data time: 0.1209, Forward time: 0.9514, Backward time: 1.2047, DDP time: 0.0052, Total iter time: 2.0401 [2020-12-31 09:33:14,049][root][INFO] - ===> Epoch[701](3505/5): Loss -0.6137, Sem -0.6137, dir -0.9564, norm 0.2888 LR: 7.870e-02 Score 98.209 Data time: 0.1083, Forward time: 0.9513, Backward time: 1.2047, DDP time: 0.0052, Total iter time: 2.2904 [2020-12-31 09:33:25,569][root][INFO] - ===> Epoch[702](3510/5): Loss -0.6090, Sem -0.6090, dir -0.9555, norm 0.2921 LR: 7.867e-02 Score 98.197 Data time: 0.0815, Forward time: 0.9513, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.2078 [2020-12-31 09:33:37,805][root][INFO] - ===> Epoch[703](3515/5): Loss -0.6153, Sem -0.6153, dir -0.9561, norm 0.2874 LR: 7.864e-02 Score 98.234 Data time: 0.1146, Forward time: 0.9513, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.3488 [2020-12-31 09:33:49,225][root][INFO] - ===> Epoch[704](3520/5): Loss -0.6061, Sem -0.6061, dir -0.9554, norm 0.2939 LR: 7.861e-02 Score 98.187 Data time: 0.1014, Forward time: 0.9513, Backward time: 1.2044, DDP time: 0.0052, Total iter time: 2.1792 [2020-12-31 09:34:02,075][root][INFO] - ===> Epoch[705](3525/5): Loss -0.6091, Sem -0.6091, dir -0.9567, norm 0.2955 LR: 7.858e-02 Score 98.264 Data time: 0.1302, Forward time: 0.9513, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.4719 [2020-12-31 09:34:14,728][root][INFO] - ===> Epoch[706](3530/5): Loss -0.6008, Sem -0.6008, dir -0.9565, norm 0.3026 LR: 7.855e-02 Score 98.257 Data time: 0.0790, Forward time: 0.9511, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.4462 [2020-12-31 09:34:27,117][root][INFO] - ===> Epoch[707](3535/5): Loss -0.6103, Sem -0.6103, dir -0.9556, norm 0.2908 LR: 7.852e-02 Score 98.189 Data time: 0.1058, Forward time: 0.9511, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.3927 [2020-12-31 09:34:39,038][root][INFO] - ===> Epoch[708](3540/5): Loss -0.6029, Sem -0.6029, dir -0.9555, norm 0.2970 LR: 7.849e-02 Score 98.157 Data time: 0.0924, Forward time: 0.9511, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.2886 [2020-12-31 09:34:51,085][root][INFO] - ===> Epoch[709](3545/5): Loss -0.6114, Sem -0.6114, dir -0.9556, norm 0.2916 LR: 7.846e-02 Score 98.271 Data time: 0.1356, Forward time: 0.9512, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.3125 [2020-12-31 09:35:04,172][root][INFO] - ===> Epoch[710](3550/5): Loss -0.6091, Sem -0.6091, dir -0.9556, norm 0.2912 LR: 7.842e-02 Score 98.176 Data time: 0.1246, Forward time: 0.9510, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.5268 [2020-12-31 09:35:15,632][root][INFO] - ===> Epoch[711](3555/5): Loss -0.6169, Sem -0.6169, dir -0.9559, norm 0.2846 LR: 7.839e-02 Score 98.178 Data time: 0.0739, Forward time: 0.9509, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.2025 [2020-12-31 09:35:29,226][root][INFO] - ===> Epoch[712](3560/5): Loss -0.6147, Sem -0.6147, dir -0.9572, norm 0.2906 LR: 7.836e-02 Score 98.292 Data time: 0.1111, Forward time: 0.9510, Backward time: 1.2056, DDP time: 0.0052, Total iter time: 2.6249 [2020-12-31 09:35:40,870][root][INFO] - ===> Epoch[713](3565/5): Loss -0.5990, Sem -0.5990, dir -0.9552, norm 0.3019 LR: 7.833e-02 Score 98.198 Data time: 0.1213, Forward time: 0.9510, Backward time: 1.2054, DDP time: 0.0052, Total iter time: 2.2370 [2020-12-31 09:35:53,128][root][INFO] - ===> Epoch[714](3570/5): Loss -0.5961, Sem -0.5961, dir -0.9546, norm 0.3063 LR: 7.830e-02 Score 98.272 Data time: 0.0913, Forward time: 0.9510, Backward time: 1.2054, DDP time: 0.0052, Total iter time: 2.3369 [2020-12-31 09:36:04,674][root][INFO] - ===> Epoch[715](3575/5): Loss -0.5918, Sem -0.5918, dir -0.9552, norm 0.3080 LR: 7.827e-02 Score 98.159 Data time: 0.1293, Forward time: 0.9512, Backward time: 1.2051, DDP time: 0.0052, Total iter time: 2.2152 [2020-12-31 09:36:16,843][root][INFO] - ===> Epoch[716](3580/5): Loss -0.6023, Sem -0.6023, dir -0.9550, norm 0.2985 LR: 7.824e-02 Score 98.207 Data time: 0.0917, Forward time: 0.9512, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.3417 [2020-12-31 09:36:30,297][root][INFO] - ===> Epoch[717](3585/5): Loss -0.6073, Sem -0.6073, dir -0.9564, norm 0.2968 LR: 7.821e-02 Score 98.253 Data time: 0.1226, Forward time: 0.9512, Backward time: 1.2055, DDP time: 0.0052, Total iter time: 2.5918 [2020-12-31 09:36:41,541][root][INFO] - ===> Epoch[718](3590/5): Loss -0.6130, Sem -0.6130, dir -0.9538, norm 0.2858 LR: 7.818e-02 Score 98.211 Data time: 0.0988, Forward time: 0.9510, Backward time: 1.2054, DDP time: 0.0052, Total iter time: 2.1724 [2020-12-31 09:36:53,949][root][INFO] - ===> Epoch[719](3595/5): Loss -0.6108, Sem -0.6108, dir -0.9570, norm 0.2924 LR: 7.815e-02 Score 98.199 Data time: 0.1395, Forward time: 0.9512, Backward time: 1.2053, DDP time: 0.0052, Total iter time: 2.3844 [2020-12-31 09:37:05,775][root][INFO] - ===> Epoch[720](3600/5): Loss -0.6131, Sem -0.6131, dir -0.9566, norm 0.2892 LR: 7.812e-02 Score 98.201 Data time: 0.0873, Forward time: 0.9512, Backward time: 1.2053, DDP time: 0.0052, Total iter time: 2.2756 [2020-12-31 09:37:05,777][root][INFO] - ===> Start testing [2020-12-31 09:37:43,697][root][INFO] - 68/68: Data time: 0.0019, Iter time: 0.1077 Loss 0.018 (AVG: 0.486) Score 99.508 (AVG: 89.139) mIOU 66.191 mAP 83.149 mAcc 74.848 IOU: 60.907 0.000 75.614 74.510 92.230 87.109 29.358 67.751 98.513 59.321 79.374 82.925 52.872 mAP: 82.082 3.130 93.084 88.034 98.551 97.944 63.748 93.218 99.847 84.110 93.229 93.601 90.356 mAcc: 79.625 0.000 86.749 81.982 93.387 94.124 33.096 87.424 99.345 74.596 91.700 94.048 56.944 [2020-12-31 09:37:43,698][root][INFO] - Finished test. Elapsed time: 37.9182 [2020-12-31 09:37:46,839][root][INFO] - [2020-12-31 09:37:46,839][root][INFO] - ################################################################ [2020-12-31 09:37:46,839][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 09:37:46,839][root][INFO] - ################################################################ [2020-12-31 09:37:46,839][root][INFO] - clutter : nan nan nan [2020-12-31 09:37:46,839][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 09:37:46,839][root][INFO] - board : 0.749 0.828 0.855 [2020-12-31 09:37:46,839][root][INFO] - bookcase : 0.239 0.374 0.538 [2020-12-31 09:37:46,839][root][INFO] - ceiling : 0.707 0.749 0.779 [2020-12-31 09:37:46,839][root][INFO] - chair : 0.747 0.857 0.873 [2020-12-31 09:37:46,840][root][INFO] - column : 0.152 0.240 0.399 [2020-12-31 09:37:46,840][root][INFO] - door : 0.518 0.741 0.830 [2020-12-31 09:37:46,840][root][INFO] - floor : 1.000 1.000 1.000 [2020-12-31 09:37:46,840][root][INFO] - sofa : 0.292 0.365 0.591 [2020-12-31 09:37:46,840][root][INFO] - table : 0.216 0.407 0.636 [2020-12-31 09:37:46,840][root][INFO] - wall : 0.466 0.701 0.799 [2020-12-31 09:37:46,840][root][INFO] - window : 0.401 0.683 0.758 [2020-12-31 09:37:46,840][root][INFO] - ---------------------------------------------------------------- [2020-12-31 09:37:46,840][root][INFO] - average : 0.457 0.579 0.671 [2020-12-31 09:37:46,840][root][INFO] - [2020-12-31 09:37:47,004][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 09:37:47,624][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter3600.pth [2020-12-31 09:37:59,019][root][INFO] - ===> Epoch[721](3605/5): Loss -0.5877, Sem -0.5877, dir -0.9539, norm 0.3059 LR: 7.809e-02 Score 98.047 Data time: 0.0920, Forward time: 0.9511, Backward time: 1.2052, DDP time: 0.0052, Total iter time: 2.1788 [2020-12-31 09:38:11,229][root][INFO] - ===> Epoch[722](3610/5): Loss -0.5941, Sem -0.5941, dir -0.9557, norm 0.3040 LR: 7.805e-02 Score 98.100 Data time: 0.1371, Forward time: 0.9512, Backward time: 1.2050, DDP time: 0.0052, Total iter time: 2.3382 [2020-12-31 09:38:23,331][root][INFO] - ===> Epoch[723](3615/5): Loss -0.6021, Sem -0.6021, dir -0.9554, norm 0.2978 LR: 7.802e-02 Score 98.149 Data time: 0.1291, Forward time: 0.9513, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.3113 [2020-12-31 09:38:35,635][root][INFO] - ===> Epoch[724](3620/5): Loss -0.5977, Sem -0.5977, dir -0.9558, norm 0.3052 LR: 7.799e-02 Score 98.234 Data time: 0.0992, Forward time: 0.9513, Backward time: 1.2049, DDP time: 0.0052, Total iter time: 2.3614 [2020-12-31 09:38:47,190][root][INFO] - ===> Epoch[725](3625/5): Loss -0.6036, Sem -0.6036, dir -0.9556, norm 0.2951 LR: 7.796e-02 Score 98.102 Data time: 0.1046, Forward time: 0.9513, Backward time: 1.2048, DDP time: 0.0052, Total iter time: 2.2225 [2020-12-31 09:38:59,061][root][INFO] - ===> Epoch[726](3630/5): Loss -0.6136, Sem -0.6136, dir -0.9564, norm 0.2909 LR: 7.793e-02 Score 98.251 Data time: 0.1615, Forward time: 0.9514, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.2889 [2020-12-31 09:39:11,695][root][INFO] - ===> Epoch[727](3635/5): Loss -0.6138, Sem -0.6138, dir -0.9568, norm 0.2895 LR: 7.790e-02 Score 98.237 Data time: 0.1625, Forward time: 0.9516, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.4435 [2020-12-31 09:39:22,878][root][INFO] - ===> Epoch[728](3640/5): Loss -0.6129, Sem -0.6129, dir -0.9558, norm 0.2894 LR: 7.787e-02 Score 98.208 Data time: 0.1112, Forward time: 0.9515, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.1591 [2020-12-31 09:39:34,248][root][INFO] - ===> Epoch[729](3645/5): Loss -0.6169, Sem -0.6169, dir -0.9557, norm 0.2846 LR: 7.784e-02 Score 98.196 Data time: 0.0923, Forward time: 0.9515, Backward time: 1.2042, DDP time: 0.0052, Total iter time: 2.1860 [2020-12-31 09:39:46,197][root][INFO] - ===> Epoch[730](3650/5): Loss -0.6146, Sem -0.6146, dir -0.9570, norm 0.2896 LR: 7.781e-02 Score 98.246 Data time: 0.1310, Forward time: 0.9517, Backward time: 1.2040, DDP time: 0.0052, Total iter time: 2.2923 [2020-12-31 09:39:57,623][root][INFO] - ===> Epoch[731](3655/5): Loss -0.6210, Sem -0.6210, dir -0.9563, norm 0.2827 LR: 7.778e-02 Score 98.244 Data time: 0.1511, Forward time: 0.9517, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.1841 [2020-12-31 09:40:09,723][root][INFO] - ===> Epoch[732](3660/5): Loss -0.6241, Sem -0.6241, dir -0.9578, norm 0.2831 LR: 7.775e-02 Score 98.313 Data time: 0.0946, Forward time: 0.9517, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.3366 [2020-12-31 09:40:21,882][root][INFO] - ===> Epoch[733](3665/5): Loss -0.6035, Sem -0.6035, dir -0.9552, norm 0.2987 LR: 7.772e-02 Score 98.243 Data time: 0.0850, Forward time: 0.9518, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.3395 [2020-12-31 09:40:34,379][root][INFO] - ===> Epoch[734](3670/5): Loss -0.6099, Sem -0.6099, dir -0.9556, norm 0.2921 LR: 7.768e-02 Score 98.218 Data time: 0.1116, Forward time: 0.9518, Backward time: 1.2038, DDP time: 0.0052, Total iter time: 2.4024 [2020-12-31 09:40:46,825][root][INFO] - ===> Epoch[735](3675/5): Loss -0.6183, Sem -0.6183, dir -0.9580, norm 0.2859 LR: 7.765e-02 Score 98.212 Data time: 0.1011, Forward time: 0.9519, Backward time: 1.2038, DDP time: 0.0052, Total iter time: 2.4053 [2020-12-31 09:40:58,549][root][INFO] - ===> Epoch[736](3680/5): Loss -0.6289, Sem -0.6289, dir -0.9578, norm 0.2763 LR: 7.762e-02 Score 98.267 Data time: 0.0963, Forward time: 0.9519, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.2615 [2020-12-31 09:41:10,204][root][INFO] - ===> Epoch[737](3685/5): Loss -0.6289, Sem -0.6289, dir -0.9585, norm 0.2769 LR: 7.759e-02 Score 98.247 Data time: 0.0934, Forward time: 0.9519, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.2522 [2020-12-31 09:41:22,426][root][INFO] - ===> Epoch[738](3690/5): Loss -0.6212, Sem -0.6212, dir -0.9560, norm 0.2794 LR: 7.756e-02 Score 98.165 Data time: 0.1172, Forward time: 0.9521, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.3518 [2020-12-31 09:41:34,135][root][INFO] - ===> Epoch[739](3695/5): Loss -0.6284, Sem -0.6284, dir -0.9587, norm 0.2782 LR: 7.753e-02 Score 98.255 Data time: 0.0672, Forward time: 0.9520, Backward time: 1.2036, DDP time: 0.0052, Total iter time: 2.2746 [2020-12-31 09:41:45,768][root][INFO] - ===> Epoch[740](3700/5): Loss -0.6254, Sem -0.6254, dir -0.9582, norm 0.2817 LR: 7.750e-02 Score 98.305 Data time: 0.0823, Forward time: 0.9520, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.2362 [2020-12-31 09:41:58,160][root][INFO] - ===> Epoch[741](3705/5): Loss -0.6150, Sem -0.6150, dir -0.9571, norm 0.2904 LR: 7.747e-02 Score 98.285 Data time: 0.0838, Forward time: 0.9519, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.3861 [2020-12-31 09:42:09,400][root][INFO] - ===> Epoch[742](3710/5): Loss -0.6239, Sem -0.6239, dir -0.9571, norm 0.2798 LR: 7.744e-02 Score 98.222 Data time: 0.0939, Forward time: 0.9518, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.1603 [2020-12-31 09:42:21,233][root][INFO] - ===> Epoch[743](3715/5): Loss -0.6094, Sem -0.6094, dir -0.9561, norm 0.2931 LR: 7.741e-02 Score 98.221 Data time: 0.0976, Forward time: 0.9518, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.2771 [2020-12-31 09:42:34,160][root][INFO] - ===> Epoch[744](3720/5): Loss -0.6091, Sem -0.6091, dir -0.9576, norm 0.2943 LR: 7.738e-02 Score 98.189 Data time: 0.0930, Forward time: 0.9518, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.4905 [2020-12-31 09:42:46,321][root][INFO] - ===> Epoch[745](3725/5): Loss -0.5897, Sem -0.5897, dir -0.9541, norm 0.3082 LR: 7.734e-02 Score 98.143 Data time: 0.1117, Forward time: 0.9519, Backward time: 1.2036, DDP time: 0.0052, Total iter time: 2.3395 [2020-12-31 09:42:58,159][root][INFO] - ===> Epoch[746](3730/5): Loss -0.6192, Sem -0.6192, dir -0.9581, norm 0.2851 LR: 7.731e-02 Score 98.202 Data time: 0.1284, Forward time: 0.9520, Backward time: 1.2034, DDP time: 0.0052, Total iter time: 2.2751 [2020-12-31 09:43:10,481][root][INFO] - ===> Epoch[747](3735/5): Loss -0.6117, Sem -0.6117, dir -0.9575, norm 0.2907 LR: 7.728e-02 Score 98.189 Data time: 0.1032, Forward time: 0.9520, Backward time: 1.2036, DDP time: 0.0052, Total iter time: 2.3746 [2020-12-31 09:43:22,389][root][INFO] - ===> Epoch[748](3740/5): Loss -0.6087, Sem -0.6087, dir -0.9554, norm 0.2921 LR: 7.725e-02 Score 98.168 Data time: 0.0768, Forward time: 0.9519, Backward time: 1.2036, DDP time: 0.0052, Total iter time: 2.2886 [2020-12-31 09:43:34,164][root][INFO] - ===> Epoch[749](3745/5): Loss -0.6236, Sem -0.6236, dir -0.9575, norm 0.2827 LR: 7.722e-02 Score 98.302 Data time: 0.0778, Forward time: 0.9518, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.2599 [2020-12-31 09:43:46,192][root][INFO] - ===> Epoch[750](3750/5): Loss -0.6268, Sem -0.6268, dir -0.9571, norm 0.2796 LR: 7.719e-02 Score 98.292 Data time: 0.0827, Forward time: 0.9517, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.3109 [2020-12-31 09:43:58,303][root][INFO] - ===> Epoch[751](3755/5): Loss -0.6277, Sem -0.6277, dir -0.9586, norm 0.2815 LR: 7.716e-02 Score 98.368 Data time: 0.0960, Forward time: 0.9517, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.3326 [2020-12-31 09:44:10,928][root][INFO] - ===> Epoch[752](3760/5): Loss -0.6082, Sem -0.6082, dir -0.9561, norm 0.2948 LR: 7.713e-02 Score 98.238 Data time: 0.1264, Forward time: 0.9517, Backward time: 1.2038, DDP time: 0.0052, Total iter time: 2.4232 [2020-12-31 09:44:24,617][root][INFO] - ===> Epoch[753](3765/5): Loss -0.6246, Sem -0.6246, dir -0.9578, norm 0.2831 LR: 7.710e-02 Score 98.316 Data time: 0.1156, Forward time: 0.9518, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.6454 [2020-12-31 09:44:35,941][root][INFO] - ===> Epoch[754](3770/5): Loss -0.6207, Sem -0.6207, dir -0.9571, norm 0.2839 LR: 7.707e-02 Score 98.261 Data time: 0.1202, Forward time: 0.9518, Backward time: 1.2039, DDP time: 0.0052, Total iter time: 2.1751 [2020-12-31 09:44:47,346][root][INFO] - ===> Epoch[755](3775/5): Loss -0.6214, Sem -0.6214, dir -0.9562, norm 0.2826 LR: 7.704e-02 Score 98.267 Data time: 0.0993, Forward time: 0.9519, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.1945 [2020-12-31 09:44:59,932][root][INFO] - ===> Epoch[756](3780/5): Loss -0.6081, Sem -0.6081, dir -0.9563, norm 0.2909 LR: 7.701e-02 Score 98.147 Data time: 0.0626, Forward time: 0.9517, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.4356 [2020-12-31 09:45:12,820][root][INFO] - ===> Epoch[757](3785/5): Loss -0.6033, Sem -0.6033, dir -0.9563, norm 0.2883 LR: 7.697e-02 Score 97.866 Data time: 0.0792, Forward time: 0.9517, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.4423 [2020-12-31 09:45:25,504][root][INFO] - ===> Epoch[758](3790/5): Loss -0.6031, Sem -0.6031, dir -0.9545, norm 0.2920 LR: 7.694e-02 Score 98.019 Data time: 0.1117, Forward time: 0.9518, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.4469 [2020-12-31 09:45:38,158][root][INFO] - ===> Epoch[759](3795/5): Loss -0.6055, Sem -0.6055, dir -0.9553, norm 0.2893 LR: 7.691e-02 Score 97.999 Data time: 0.0926, Forward time: 0.9518, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.4342 [2020-12-31 09:45:50,539][root][INFO] - ===> Epoch[760](3800/5): Loss -0.5961, Sem -0.5961, dir -0.9539, norm 0.2986 LR: 7.688e-02 Score 98.030 Data time: 0.1301, Forward time: 0.9518, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.3827 [2020-12-31 09:45:50,543][root][INFO] - ===> Start testing [2020-12-31 09:46:28,256][root][INFO] - 68/68: Data time: 0.0022, Iter time: 0.1039 Loss 0.406 (AVG: 0.471) Score 89.353 (AVG: 89.095) mIOU 66.118 mAP 82.813 mAcc 75.271 IOU: 60.117 0.000 69.141 74.445 93.118 86.589 27.146 69.263 98.345 67.448 79.620 82.886 51.419 mAP: 81.341 2.930 92.678 89.290 98.982 97.638 60.378 90.954 99.836 88.461 93.158 93.760 87.159 mAcc: 75.481 0.000 84.370 82.840 94.479 94.734 33.637 84.496 99.106 84.856 92.644 94.455 57.430 [2020-12-31 09:46:28,258][root][INFO] - Finished test. Elapsed time: 37.7113 [2020-12-31 09:46:31,294][root][INFO] - [2020-12-31 09:46:31,294][root][INFO] - ################################################################ [2020-12-31 09:46:31,294][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 09:46:31,294][root][INFO] - ################################################################ [2020-12-31 09:46:31,294][root][INFO] - clutter : nan nan nan [2020-12-31 09:46:31,294][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 09:46:31,294][root][INFO] - board : 0.713 0.797 0.876 [2020-12-31 09:46:31,294][root][INFO] - bookcase : 0.238 0.380 0.512 [2020-12-31 09:46:31,295][root][INFO] - ceiling : 0.794 0.836 0.856 [2020-12-31 09:46:31,295][root][INFO] - chair : 0.768 0.872 0.894 [2020-12-31 09:46:31,295][root][INFO] - column : 0.145 0.220 0.293 [2020-12-31 09:46:31,295][root][INFO] - door : 0.491 0.659 0.829 [2020-12-31 09:46:31,295][root][INFO] - floor : 0.890 0.921 0.921 [2020-12-31 09:46:31,295][root][INFO] - sofa : 0.411 0.553 0.704 [2020-12-31 09:46:31,295][root][INFO] - table : 0.193 0.366 0.604 [2020-12-31 09:46:31,295][root][INFO] - wall : 0.460 0.678 0.781 [2020-12-31 09:46:31,295][root][INFO] - window : 0.352 0.651 0.703 [2020-12-31 09:46:31,295][root][INFO] - ---------------------------------------------------------------- [2020-12-31 09:46:31,295][root][INFO] - average : 0.455 0.578 0.664 [2020-12-31 09:46:31,295][root][INFO] - [2020-12-31 09:46:31,470][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 09:46:32,053][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter3800.pth [2020-12-31 09:46:42,662][root][INFO] - ===> Epoch[761](3805/5): Loss -0.6101, Sem -0.6101, dir -0.9557, norm 0.2882 LR: 7.685e-02 Score 98.114 Data time: 0.0902, Forward time: 0.9518, Backward time: 1.2042, DDP time: 0.0052, Total iter time: 2.0131 [2020-12-31 09:46:54,508][root][INFO] - ===> Epoch[762](3810/5): Loss -0.6141, Sem -0.6141, dir -0.9566, norm 0.2886 LR: 7.682e-02 Score 98.188 Data time: 0.1333, Forward time: 0.9517, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.2806 [2020-12-31 09:47:05,897][root][INFO] - ===> Epoch[763](3815/5): Loss -0.6250, Sem -0.6250, dir -0.9571, norm 0.2779 LR: 7.679e-02 Score 98.210 Data time: 0.0895, Forward time: 0.9516, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.1945 [2020-12-31 09:47:18,225][root][INFO] - ===> Epoch[764](3820/5): Loss -0.6186, Sem -0.6186, dir -0.9570, norm 0.2835 LR: 7.676e-02 Score 98.180 Data time: 0.0757, Forward time: 0.9514, Backward time: 1.2044, DDP time: 0.0052, Total iter time: 2.3952 [2020-12-31 09:47:29,546][root][INFO] - ===> Epoch[765](3825/5): Loss -0.6273, Sem -0.6273, dir -0.9585, norm 0.2808 LR: 7.673e-02 Score 98.319 Data time: 0.0814, Forward time: 0.9514, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.1680 [2020-12-31 09:47:41,076][root][INFO] - ===> Epoch[766](3830/5): Loss -0.6212, Sem -0.6212, dir -0.9566, norm 0.2812 LR: 7.670e-02 Score 98.209 Data time: 0.1031, Forward time: 0.9514, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.2213 [2020-12-31 09:47:52,840][root][INFO] - ===> Epoch[767](3835/5): Loss -0.6244, Sem -0.6244, dir -0.9574, norm 0.2808 LR: 7.667e-02 Score 98.283 Data time: 0.1802, Forward time: 0.9514, Backward time: 1.2039, DDP time: 0.0052, Total iter time: 2.2561 [2020-12-31 09:48:04,589][root][INFO] - ===> Epoch[768](3840/5): Loss -0.6185, Sem -0.6185, dir -0.9578, norm 0.2888 LR: 7.663e-02 Score 98.303 Data time: 0.1572, Forward time: 0.9513, Backward time: 1.2039, DDP time: 0.0052, Total iter time: 2.2705 [2020-12-31 09:48:16,928][root][INFO] - ===> Epoch[769](3845/5): Loss -0.6112, Sem -0.6112, dir -0.9584, norm 0.2947 LR: 7.660e-02 Score 98.271 Data time: 0.0766, Forward time: 0.9512, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.3979 [2020-12-31 09:48:28,524][root][INFO] - ===> Epoch[770](3850/5): Loss -0.6126, Sem -0.6126, dir -0.9591, norm 0.2985 LR: 7.657e-02 Score 98.401 Data time: 0.1851, Forward time: 0.9512, Backward time: 1.2039, DDP time: 0.0052, Total iter time: 2.2343 [2020-12-31 09:48:41,028][root][INFO] - ===> Epoch[771](3855/5): Loss -0.6072, Sem -0.6072, dir -0.9568, norm 0.2951 LR: 7.654e-02 Score 98.191 Data time: 0.1154, Forward time: 0.9513, Backward time: 1.2039, DDP time: 0.0052, Total iter time: 2.4166 [2020-12-31 09:48:52,824][root][INFO] - ===> Epoch[772](3860/5): Loss -0.6315, Sem -0.6315, dir -0.9596, norm 0.2772 LR: 7.651e-02 Score 98.292 Data time: 0.0849, Forward time: 0.9512, Backward time: 1.2040, DDP time: 0.0052, Total iter time: 2.2749 [2020-12-31 09:49:04,320][root][INFO] - ===> Epoch[773](3865/5): Loss -0.6236, Sem -0.6236, dir -0.9578, norm 0.2829 LR: 7.648e-02 Score 98.300 Data time: 0.1553, Forward time: 0.9512, Backward time: 1.2038, DDP time: 0.0052, Total iter time: 2.2117 [2020-12-31 09:49:16,320][root][INFO] - ===> Epoch[774](3870/5): Loss -0.6222, Sem -0.6222, dir -0.9575, norm 0.2843 LR: 7.645e-02 Score 98.329 Data time: 0.1124, Forward time: 0.9513, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.3120 [2020-12-31 09:49:28,518][root][INFO] - ===> Epoch[775](3875/5): Loss -0.6275, Sem -0.6275, dir -0.9570, norm 0.2778 LR: 7.642e-02 Score 98.276 Data time: 0.1170, Forward time: 0.9513, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.3465 [2020-12-31 09:49:41,011][root][INFO] - ===> Epoch[776](3880/5): Loss -0.6280, Sem -0.6280, dir -0.9588, norm 0.2794 LR: 7.639e-02 Score 98.285 Data time: 0.0998, Forward time: 0.9513, Backward time: 1.2038, DDP time: 0.0052, Total iter time: 2.3998 [2020-12-31 09:49:52,874][root][INFO] - ===> Epoch[777](3885/5): Loss -0.6363, Sem -0.6363, dir -0.9584, norm 0.2687 LR: 7.636e-02 Score 98.218 Data time: 0.1353, Forward time: 0.9515, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.2769 [2020-12-31 09:50:05,343][root][INFO] - ===> Epoch[778](3890/5): Loss -0.6345, Sem -0.6345, dir -0.9591, norm 0.2739 LR: 7.633e-02 Score 98.324 Data time: 0.1338, Forward time: 0.9516, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.3921 [2020-12-31 09:50:18,495][root][INFO] - ===> Epoch[779](3895/5): Loss -0.6220, Sem -0.6220, dir -0.9576, norm 0.2827 LR: 7.629e-02 Score 98.242 Data time: 0.0862, Forward time: 0.9516, Backward time: 1.2038, DDP time: 0.0052, Total iter time: 2.5434 [2020-12-31 09:50:30,872][root][INFO] - ===> Epoch[780](3900/5): Loss -0.6298, Sem -0.6298, dir -0.9595, norm 0.2798 LR: 7.626e-02 Score 98.340 Data time: 0.1004, Forward time: 0.9515, Backward time: 1.2039, DDP time: 0.0052, Total iter time: 2.3857 [2020-12-31 09:50:43,285][root][INFO] - ===> Epoch[781](3905/5): Loss -0.6285, Sem -0.6285, dir -0.9588, norm 0.2820 LR: 7.623e-02 Score 98.398 Data time: 0.0957, Forward time: 0.9515, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.3966 [2020-12-31 09:50:54,817][root][INFO] - ===> Epoch[782](3910/5): Loss -0.6264, Sem -0.6264, dir -0.9575, norm 0.2806 LR: 7.620e-02 Score 98.327 Data time: 0.0923, Forward time: 0.9514, Backward time: 1.2040, DDP time: 0.0052, Total iter time: 2.2151 [2020-12-31 09:51:06,499][root][INFO] - ===> Epoch[783](3915/5): Loss -0.6250, Sem -0.6250, dir -0.9574, norm 0.2790 LR: 7.617e-02 Score 98.240 Data time: 0.0871, Forward time: 0.9512, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.2629 [2020-12-31 09:51:18,203][root][INFO] - ===> Epoch[784](3920/5): Loss -0.6305, Sem -0.6305, dir -0.9592, norm 0.2783 LR: 7.614e-02 Score 98.316 Data time: 0.1003, Forward time: 0.9511, Backward time: 1.2042, DDP time: 0.0052, Total iter time: 2.2562 [2020-12-31 09:51:30,054][root][INFO] - ===> Epoch[785](3925/5): Loss -0.6315, Sem -0.6315, dir -0.9588, norm 0.2776 LR: 7.611e-02 Score 98.357 Data time: 0.1090, Forward time: 0.9511, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.2715 [2020-12-31 09:51:41,623][root][INFO] - ===> Epoch[786](3930/5): Loss -0.6230, Sem -0.6230, dir -0.9568, norm 0.2805 LR: 7.608e-02 Score 98.227 Data time: 0.0977, Forward time: 0.9510, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.2152 [2020-12-31 09:51:53,732][root][INFO] - ===> Epoch[787](3935/5): Loss -0.6364, Sem -0.6364, dir -0.9591, norm 0.2715 LR: 7.605e-02 Score 98.280 Data time: 0.1077, Forward time: 0.9509, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.3235 [2020-12-31 09:52:05,748][root][INFO] - ===> Epoch[788](3940/5): Loss -0.6410, Sem -0.6410, dir -0.9591, norm 0.2691 LR: 7.602e-02 Score 98.376 Data time: 0.2004, Forward time: 0.9509, Backward time: 1.2040, DDP time: 0.0052, Total iter time: 2.3183 [2020-12-31 09:52:17,651][root][INFO] - ===> Epoch[789](3945/5): Loss -0.6444, Sem -0.6444, dir -0.9605, norm 0.2658 LR: 7.599e-02 Score 98.327 Data time: 0.1173, Forward time: 0.9509, Backward time: 1.2040, DDP time: 0.0052, Total iter time: 2.2793 [2020-12-31 09:52:29,806][root][INFO] - ===> Epoch[790](3950/5): Loss -0.6319, Sem -0.6319, dir -0.9588, norm 0.2759 LR: 7.595e-02 Score 98.330 Data time: 0.0795, Forward time: 0.9508, Backward time: 1.2042, DDP time: 0.0052, Total iter time: 2.3472 [2020-12-31 09:52:41,768][root][INFO] - ===> Epoch[791](3955/5): Loss -0.6436, Sem -0.6436, dir -0.9605, norm 0.2668 LR: 7.592e-02 Score 98.364 Data time: 0.1188, Forward time: 0.9508, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.3012 [2020-12-31 09:52:54,291][root][INFO] - ===> Epoch[792](3960/5): Loss -0.6257, Sem -0.6257, dir -0.9590, norm 0.2822 LR: 7.589e-02 Score 98.318 Data time: 0.1602, Forward time: 0.9508, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.4197 [2020-12-31 09:53:05,555][root][INFO] - ===> Epoch[793](3965/5): Loss -0.6235, Sem -0.6235, dir -0.9578, norm 0.2842 LR: 7.586e-02 Score 98.327 Data time: 0.1318, Forward time: 0.9508, Backward time: 1.2039, DDP time: 0.0052, Total iter time: 2.1540 [2020-12-31 09:53:17,829][root][INFO] - ===> Epoch[794](3970/5): Loss -0.6159, Sem -0.6159, dir -0.9578, norm 0.2907 LR: 7.583e-02 Score 98.293 Data time: 0.0733, Forward time: 0.9507, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.3750 [2020-12-31 09:53:30,154][root][INFO] - ===> Epoch[795](3975/5): Loss -0.6299, Sem -0.6299, dir -0.9587, norm 0.2796 LR: 7.580e-02 Score 98.363 Data time: 0.1057, Forward time: 0.9507, Backward time: 1.2042, DDP time: 0.0052, Total iter time: 2.3706 [2020-12-31 09:53:41,628][root][INFO] - ===> Epoch[796](3980/5): Loss -0.6313, Sem -0.6313, dir -0.9593, norm 0.2792 LR: 7.577e-02 Score 98.387 Data time: 0.1377, Forward time: 0.9507, Backward time: 1.2040, DDP time: 0.0052, Total iter time: 2.2140 [2020-12-31 09:53:54,174][root][INFO] - ===> Epoch[797](3985/5): Loss -0.6116, Sem -0.6116, dir -0.9579, norm 0.2944 LR: 7.574e-02 Score 98.276 Data time: 0.2222, Forward time: 0.9509, Backward time: 1.2038, DDP time: 0.0052, Total iter time: 2.4096 [2020-12-31 09:54:06,542][root][INFO] - ===> Epoch[798](3990/5): Loss -0.6231, Sem -0.6231, dir -0.9601, norm 0.2849 LR: 7.571e-02 Score 98.289 Data time: 0.0818, Forward time: 0.9508, Backward time: 1.2039, DDP time: 0.0052, Total iter time: 2.3956 [2020-12-31 09:54:18,627][root][INFO] - ===> Epoch[799](3995/5): Loss -0.6280, Sem -0.6280, dir -0.9594, norm 0.2788 LR: 7.568e-02 Score 98.244 Data time: 0.1078, Forward time: 0.9508, Backward time: 1.2040, DDP time: 0.0052, Total iter time: 2.3403 [2020-12-31 09:54:30,172][root][INFO] - ===> Epoch[800](4000/5): Loss -0.6232, Sem -0.6232, dir -0.9580, norm 0.2813 LR: 7.565e-02 Score 98.233 Data time: 0.1524, Forward time: 0.9507, Backward time: 1.2039, DDP time: 0.0052, Total iter time: 2.2313 [2020-12-31 09:54:30,175][root][INFO] - ===> Start testing [2020-12-31 09:55:07,990][root][INFO] - 68/68: Data time: 0.0021, Iter time: 0.1131 Loss 0.386 (AVG: 0.507) Score 87.305 (AVG: 89.270) mIOU 66.807 mAP 83.282 mAcc 74.045 IOU: 60.351 0.000 76.125 74.673 92.409 88.121 23.627 72.019 98.267 68.295 80.800 83.300 50.504 mAP: 82.278 3.410 93.962 88.552 98.782 98.363 58.141 93.596 99.799 89.487 93.440 93.859 88.992 mAcc: 78.062 0.000 88.058 82.905 93.549 94.833 25.751 83.810 99.061 77.782 89.569 96.291 52.917 [2020-12-31 09:55:07,991][root][INFO] - Finished test. Elapsed time: 37.8136 [2020-12-31 09:55:11,142][root][INFO] - [2020-12-31 09:55:11,142][root][INFO] - ################################################################ [2020-12-31 09:55:11,142][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 09:55:11,142][root][INFO] - ################################################################ [2020-12-31 09:55:11,142][root][INFO] - clutter : nan nan nan [2020-12-31 09:55:11,142][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 09:55:11,143][root][INFO] - board : 0.730 0.831 0.880 [2020-12-31 09:55:11,143][root][INFO] - bookcase : 0.245 0.395 0.532 [2020-12-31 09:55:11,143][root][INFO] - ceiling : 0.729 0.754 0.825 [2020-12-31 09:55:11,143][root][INFO] - chair : 0.749 0.864 0.889 [2020-12-31 09:55:11,143][root][INFO] - column : 0.115 0.215 0.332 [2020-12-31 09:55:11,143][root][INFO] - door : 0.522 0.768 0.888 [2020-12-31 09:55:11,143][root][INFO] - floor : 0.998 1.000 1.000 [2020-12-31 09:55:11,143][root][INFO] - sofa : 0.369 0.537 0.790 [2020-12-31 09:55:11,143][root][INFO] - table : 0.220 0.418 0.671 [2020-12-31 09:55:11,143][root][INFO] - wall : 0.466 0.701 0.813 [2020-12-31 09:55:11,143][root][INFO] - window : 0.374 0.659 0.717 [2020-12-31 09:55:11,143][root][INFO] - ---------------------------------------------------------------- [2020-12-31 09:55:11,143][root][INFO] - average : 0.460 0.595 0.695 [2020-12-31 09:55:11,143][root][INFO] - [2020-12-31 09:55:11,315][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 09:55:11,917][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter4000.pth [2020-12-31 09:55:23,065][root][INFO] - ===> Epoch[801](4005/5): Loss -0.6350, Sem -0.6350, dir -0.9600, norm 0.2739 LR: 7.561e-02 Score 98.291 Data time: 0.1740, Forward time: 0.9506, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.1271 [2020-12-31 09:55:35,535][root][INFO] - ===> Epoch[802](4010/5): Loss -0.6293, Sem -0.6293, dir -0.9601, norm 0.2820 LR: 7.558e-02 Score 98.372 Data time: 0.1414, Forward time: 0.9506, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.4008 [2020-12-31 09:55:48,322][root][INFO] - ===> Epoch[803](4015/5): Loss -0.6465, Sem -0.6465, dir -0.9609, norm 0.2650 LR: 7.555e-02 Score 98.346 Data time: 0.1047, Forward time: 0.9505, Backward time: 1.2040, DDP time: 0.0052, Total iter time: 2.4763 [2020-12-31 09:55:59,585][root][INFO] - ===> Epoch[804](4020/5): Loss -0.6414, Sem -0.6414, dir -0.9584, norm 0.2679 LR: 7.552e-02 Score 98.385 Data time: 0.1177, Forward time: 0.9504, Backward time: 1.2039, DDP time: 0.0052, Total iter time: 2.1499 [2020-12-31 09:56:12,242][root][INFO] - ===> Epoch[805](4025/5): Loss -0.6382, Sem -0.6382, dir -0.9592, norm 0.2692 LR: 7.549e-02 Score 98.260 Data time: 0.0854, Forward time: 0.9502, Backward time: 1.2042, DDP time: 0.0052, Total iter time: 2.4438 [2020-12-31 09:56:24,152][root][INFO] - ===> Epoch[806](4030/5): Loss -0.6376, Sem -0.6376, dir -0.9594, norm 0.2696 LR: 7.546e-02 Score 98.266 Data time: 0.1340, Forward time: 0.9502, Backward time: 1.2042, DDP time: 0.0052, Total iter time: 2.2838 [2020-12-31 09:56:35,392][root][INFO] - ===> Epoch[807](4035/5): Loss -0.6367, Sem -0.6367, dir -0.9595, norm 0.2732 LR: 7.543e-02 Score 98.347 Data time: 0.0897, Forward time: 0.9501, Backward time: 1.2040, DDP time: 0.0052, Total iter time: 2.1545 [2020-12-31 09:56:47,325][root][INFO] - ===> Epoch[808](4040/5): Loss -0.6331, Sem -0.6331, dir -0.9586, norm 0.2749 LR: 7.540e-02 Score 98.330 Data time: 0.0706, Forward time: 0.9500, Backward time: 1.2042, DDP time: 0.0052, Total iter time: 2.3004 [2020-12-31 09:56:58,444][root][INFO] - ===> Epoch[809](4045/5): Loss -0.6397, Sem -0.6397, dir -0.9607, norm 0.2713 LR: 7.537e-02 Score 98.333 Data time: 0.1089, Forward time: 0.9499, Backward time: 1.2040, DDP time: 0.0052, Total iter time: 2.1359 [2020-12-31 09:57:10,779][root][INFO] - ===> Epoch[810](4050/5): Loss -0.6282, Sem -0.6282, dir -0.9585, norm 0.2790 LR: 7.534e-02 Score 98.297 Data time: 0.1060, Forward time: 0.9498, Backward time: 1.2042, DDP time: 0.0052, Total iter time: 2.3796 [2020-12-31 09:57:23,305][root][INFO] - ===> Epoch[811](4055/5): Loss -0.6427, Sem -0.6427, dir -0.9608, norm 0.2690 LR: 7.530e-02 Score 98.376 Data time: 0.1032, Forward time: 0.9499, Backward time: 1.2042, DDP time: 0.0052, Total iter time: 2.4185 [2020-12-31 09:57:36,349][root][INFO] - ===> Epoch[812](4060/5): Loss -0.6214, Sem -0.6214, dir -0.9589, norm 0.2869 LR: 7.527e-02 Score 98.308 Data time: 0.0773, Forward time: 0.9499, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.5310 [2020-12-31 09:57:48,396][root][INFO] - ===> Epoch[813](4065/5): Loss -0.6348, Sem -0.6348, dir -0.9601, norm 0.2752 LR: 7.524e-02 Score 98.338 Data time: 0.2697, Forward time: 0.9499, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.3171 [2020-12-31 09:58:00,868][root][INFO] - ===> Epoch[814](4070/5): Loss -0.6362, Sem -0.6362, dir -0.9609, norm 0.2743 LR: 7.521e-02 Score 98.345 Data time: 0.1188, Forward time: 0.9500, Backward time: 1.2044, DDP time: 0.0052, Total iter time: 2.4030 [2020-12-31 09:58:13,019][root][INFO] - ===> Epoch[815](4075/5): Loss -0.6267, Sem -0.6267, dir -0.9596, norm 0.2831 LR: 7.518e-02 Score 98.344 Data time: 0.1509, Forward time: 0.9500, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.3303 [2020-12-31 09:58:25,398][root][INFO] - ===> Epoch[816](4080/5): Loss -0.6263, Sem -0.6263, dir -0.9598, norm 0.2806 LR: 7.515e-02 Score 98.267 Data time: 0.1255, Forward time: 0.9500, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.3791 [2020-12-31 09:58:37,071][root][INFO] - ===> Epoch[817](4085/5): Loss -0.6112, Sem -0.6112, dir -0.9569, norm 0.2900 LR: 7.512e-02 Score 98.168 Data time: 0.1033, Forward time: 0.9499, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.2436 [2020-12-31 09:58:50,127][root][INFO] - ===> Epoch[818](4090/5): Loss -0.6252, Sem -0.6252, dir -0.9591, norm 0.2817 LR: 7.509e-02 Score 98.274 Data time: 0.1066, Forward time: 0.9499, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.5184 [2020-12-31 09:59:02,060][root][INFO] - ===> Epoch[819](4095/5): Loss -0.6288, Sem -0.6288, dir -0.9590, norm 0.2797 LR: 7.506e-02 Score 98.317 Data time: 0.1116, Forward time: 0.9500, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.3024 [2020-12-31 09:59:14,026][root][INFO] - ===> Epoch[820](4100/5): Loss -0.6246, Sem -0.6246, dir -0.9584, norm 0.2826 LR: 7.503e-02 Score 98.310 Data time: 0.0985, Forward time: 0.9500, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.3075 [2020-12-31 09:59:26,316][root][INFO] - ===> Epoch[821](4105/5): Loss -0.6288, Sem -0.6288, dir -0.9588, norm 0.2801 LR: 7.499e-02 Score 98.333 Data time: 0.0909, Forward time: 0.9499, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.3727 [2020-12-31 09:59:37,792][root][INFO] - ===> Epoch[822](4110/5): Loss -0.6245, Sem -0.6245, dir -0.9588, norm 0.2834 LR: 7.496e-02 Score 98.344 Data time: 0.1480, Forward time: 0.9498, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.2042 [2020-12-31 09:59:49,864][root][INFO] - ===> Epoch[823](4115/5): Loss -0.6300, Sem -0.6300, dir -0.9588, norm 0.2765 LR: 7.493e-02 Score 98.238 Data time: 0.0874, Forward time: 0.9498, Backward time: 1.2046, DDP time: 0.0052, Total iter time: 2.3238 [2020-12-31 10:00:01,776][root][INFO] - ===> Epoch[824](4120/5): Loss -0.6364, Sem -0.6364, dir -0.9605, norm 0.2746 LR: 7.490e-02 Score 98.361 Data time: 0.1473, Forward time: 0.9498, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.2930 [2020-12-31 10:00:13,985][root][INFO] - ===> Epoch[825](4125/5): Loss -0.6109, Sem -0.6109, dir -0.9571, norm 0.2965 LR: 7.487e-02 Score 98.348 Data time: 0.1490, Forward time: 0.9498, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.3545 [2020-12-31 10:00:26,110][root][INFO] - ===> Epoch[826](4130/5): Loss -0.6329, Sem -0.6329, dir -0.9589, norm 0.2792 LR: 7.484e-02 Score 98.437 Data time: 0.1896, Forward time: 0.9499, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.3253 [2020-12-31 10:00:38,470][root][INFO] - ===> Epoch[827](4135/5): Loss -0.6288, Sem -0.6288, dir -0.9585, norm 0.2814 LR: 7.481e-02 Score 98.370 Data time: 0.1284, Forward time: 0.9500, Backward time: 1.2043, DDP time: 0.0052, Total iter time: 2.3829 [2020-12-31 10:00:51,463][root][INFO] - ===> Epoch[828](4140/5): Loss -0.6325, Sem -0.6325, dir -0.9607, norm 0.2793 LR: 7.478e-02 Score 98.402 Data time: 0.1027, Forward time: 0.9500, Backward time: 1.2045, DDP time: 0.0052, Total iter time: 2.5029 [2020-12-31 10:01:03,103][root][INFO] - ===> Epoch[829](4145/5): Loss -0.6332, Sem -0.6332, dir -0.9585, norm 0.2762 LR: 7.475e-02 Score 98.369 Data time: 0.1169, Forward time: 0.9501, Backward time: 1.2042, DDP time: 0.0052, Total iter time: 2.2373 [2020-12-31 10:01:14,182][root][INFO] - ===> Epoch[830](4150/5): Loss -0.6316, Sem -0.6316, dir -0.9592, norm 0.2770 LR: 7.472e-02 Score 98.332 Data time: 0.1051, Forward time: 0.9501, Backward time: 1.2041, DDP time: 0.0052, Total iter time: 2.1258 [2020-12-31 10:01:25,406][root][INFO] - ===> Epoch[831](4155/5): Loss -0.6401, Sem -0.6401, dir -0.9608, norm 0.2704 LR: 7.469e-02 Score 98.313 Data time: 0.1052, Forward time: 0.9500, Backward time: 1.2039, DDP time: 0.0052, Total iter time: 2.1504 [2020-12-31 10:01:38,257][root][INFO] - ===> Epoch[832](4160/5): Loss -0.6392, Sem -0.6392, dir -0.9600, norm 0.2735 LR: 7.465e-02 Score 98.426 Data time: 0.1504, Forward time: 0.9502, Backward time: 1.2039, DDP time: 0.0052, Total iter time: 2.4765 [2020-12-31 10:01:50,183][root][INFO] - ===> Epoch[833](4165/5): Loss -0.6389, Sem -0.6389, dir -0.9592, norm 0.2711 LR: 7.462e-02 Score 98.369 Data time: 0.0997, Forward time: 0.9503, Backward time: 1.2038, DDP time: 0.0052, Total iter time: 2.2975 [2020-12-31 10:02:01,572][root][INFO] - ===> Epoch[834](4170/5): Loss -0.6401, Sem -0.6401, dir -0.9597, norm 0.2695 LR: 7.459e-02 Score 98.336 Data time: 0.1136, Forward time: 0.9503, Backward time: 1.2037, DDP time: 0.0052, Total iter time: 2.1838 [2020-12-31 10:02:13,691][root][INFO] - ===> Epoch[835](4175/5): Loss -0.6496, Sem -0.6496, dir -0.9619, norm 0.2657 LR: 7.456e-02 Score 98.446 Data time: 0.1384, Forward time: 0.9504, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.3295 [2020-12-31 10:02:25,217][root][INFO] - ===> Epoch[836](4180/5): Loss -0.6489, Sem -0.6489, dir -0.9610, norm 0.2650 LR: 7.453e-02 Score 98.424 Data time: 0.1230, Forward time: 0.9502, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.2285 [2020-12-31 10:02:38,310][root][INFO] - ===> Epoch[837](4185/5): Loss -0.6453, Sem -0.6453, dir -0.9616, norm 0.2699 LR: 7.450e-02 Score 98.477 Data time: 0.1339, Forward time: 0.9504, Backward time: 1.2036, DDP time: 0.0052, Total iter time: 2.5255 [2020-12-31 10:02:49,793][root][INFO] - ===> Epoch[838](4190/5): Loss -0.6420, Sem -0.6420, dir -0.9602, norm 0.2698 LR: 7.447e-02 Score 98.388 Data time: 0.1543, Forward time: 0.9503, Backward time: 1.2034, DDP time: 0.0052, Total iter time: 2.2007 [2020-12-31 10:03:01,986][root][INFO] - ===> Epoch[839](4195/5): Loss -0.6452, Sem -0.6452, dir -0.9609, norm 0.2683 LR: 7.444e-02 Score 98.427 Data time: 0.1359, Forward time: 0.9505, Backward time: 1.2033, DDP time: 0.0052, Total iter time: 2.3510 [2020-12-31 10:03:14,075][root][INFO] - ===> Epoch[840](4200/5): Loss -0.6354, Sem -0.6354, dir -0.9607, norm 0.2760 LR: 7.441e-02 Score 98.366 Data time: 0.1037, Forward time: 0.9506, Backward time: 1.2032, DDP time: 0.0052, Total iter time: 2.3361 [2020-12-31 10:03:14,077][root][INFO] - ===> Start testing [2020-12-31 10:03:50,902][root][INFO] - 68/68: Data time: 0.0025, Iter time: 0.1543 Loss 0.831 (AVG: 0.515) Score 85.913 (AVG: 88.581) mIOU 67.331 mAP 83.516 mAcc 76.070 IOU: 59.685 0.000 75.881 73.745 92.654 88.072 23.152 73.190 98.236 72.934 80.098 81.583 56.069 mAP: 82.034 4.297 91.208 86.738 98.809 98.101 60.558 95.327 99.830 91.239 93.087 93.645 90.832 mAcc: 78.841 0.000 83.331 81.339 94.248 95.133 38.631 91.350 98.868 83.238 90.728 90.856 62.344 [2020-12-31 10:03:50,908][root][INFO] - Finished test. Elapsed time: 36.8233 [2020-12-31 10:03:54,016][root][INFO] - [2020-12-31 10:03:54,016][root][INFO] - ################################################################ [2020-12-31 10:03:54,017][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 10:03:54,017][root][INFO] - ################################################################ [2020-12-31 10:03:54,017][root][INFO] - clutter : nan nan nan [2020-12-31 10:03:54,017][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 10:03:54,017][root][INFO] - board : 0.698 0.782 0.851 [2020-12-31 10:03:54,017][root][INFO] - bookcase : 0.242 0.372 0.518 [2020-12-31 10:03:54,017][root][INFO] - ceiling : 0.639 0.674 0.726 [2020-12-31 10:03:54,017][root][INFO] - chair : 0.758 0.864 0.884 [2020-12-31 10:03:54,017][root][INFO] - column : 0.124 0.199 0.305 [2020-12-31 10:03:54,017][root][INFO] - door : 0.647 0.827 0.955 [2020-12-31 10:03:54,017][root][INFO] - floor : 0.941 0.974 0.974 [2020-12-31 10:03:54,017][root][INFO] - sofa : 0.442 0.564 0.842 [2020-12-31 10:03:54,017][root][INFO] - table : 0.191 0.393 0.669 [2020-12-31 10:03:54,017][root][INFO] - wall : 0.404 0.638 0.742 [2020-12-31 10:03:54,017][root][INFO] - window : 0.432 0.740 0.773 [2020-12-31 10:03:54,017][root][INFO] - ---------------------------------------------------------------- [2020-12-31 10:03:54,017][root][INFO] - average : 0.460 0.585 0.687 [2020-12-31 10:03:54,017][root][INFO] - [2020-12-31 10:03:54,194][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 10:03:54,739][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter4200.pth [2020-12-31 10:04:05,342][root][INFO] - ===> Epoch[841](4205/5): Loss -0.6363, Sem -0.6363, dir -0.9605, norm 0.2766 LR: 7.438e-02 Score 98.417 Data time: 0.0799, Forward time: 0.9504, Backward time: 1.2031, DDP time: 0.0052, Total iter time: 2.0309 [2020-12-31 10:04:17,196][root][INFO] - ===> Epoch[842](4210/5): Loss -0.6350, Sem -0.6350, dir -0.9601, norm 0.2748 LR: 7.434e-02 Score 98.336 Data time: 0.1340, Forward time: 0.9503, Backward time: 1.2032, DDP time: 0.0052, Total iter time: 2.2866 [2020-12-31 10:04:29,222][root][INFO] - ===> Epoch[843](4215/5): Loss -0.6381, Sem -0.6381, dir -0.9589, norm 0.2703 LR: 7.431e-02 Score 98.324 Data time: 0.1033, Forward time: 0.9503, Backward time: 1.2032, DDP time: 0.0052, Total iter time: 2.3088 [2020-12-31 10:04:41,661][root][INFO] - ===> Epoch[844](4220/5): Loss -0.6470, Sem -0.6470, dir -0.9613, norm 0.2683 LR: 7.428e-02 Score 98.475 Data time: 0.0875, Forward time: 0.9502, Backward time: 1.2033, DDP time: 0.0052, Total iter time: 2.4000 [2020-12-31 10:04:54,117][root][INFO] - ===> Epoch[845](4225/5): Loss -0.6510, Sem -0.6510, dir -0.9603, norm 0.2614 LR: 7.425e-02 Score 98.399 Data time: 0.0967, Forward time: 0.9502, Backward time: 1.2034, DDP time: 0.0052, Total iter time: 2.3943 [2020-12-31 10:05:05,609][root][INFO] - ===> Epoch[846](4230/5): Loss -0.6431, Sem -0.6431, dir -0.9603, norm 0.2704 LR: 7.422e-02 Score 98.454 Data time: 0.1133, Forward time: 0.9503, Backward time: 1.2032, DDP time: 0.0052, Total iter time: 2.2071 [2020-12-31 10:05:17,464][root][INFO] - ===> Epoch[847](4235/5): Loss -0.6425, Sem -0.6425, dir -0.9612, norm 0.2711 LR: 7.419e-02 Score 98.417 Data time: 0.1483, Forward time: 0.9504, Backward time: 1.2031, DDP time: 0.0052, Total iter time: 2.2786 [2020-12-31 10:05:29,780][root][INFO] - ===> Epoch[848](4240/5): Loss -0.6308, Sem -0.6308, dir -0.9599, norm 0.2810 LR: 7.416e-02 Score 98.425 Data time: 0.0927, Forward time: 0.9503, Backward time: 1.2032, DDP time: 0.0052, Total iter time: 2.3715 [2020-12-31 10:05:41,498][root][INFO] - ===> Epoch[849](4245/5): Loss -0.6398, Sem -0.6398, dir -0.9612, norm 0.2731 LR: 7.413e-02 Score 98.394 Data time: 0.0923, Forward time: 0.9503, Backward time: 1.2031, DDP time: 0.0052, Total iter time: 2.2539 [2020-12-31 10:05:53,868][root][INFO] - ===> Epoch[850](4250/5): Loss -0.6474, Sem -0.6474, dir -0.9604, norm 0.2670 LR: 7.410e-02 Score 98.464 Data time: 0.0777, Forward time: 0.9502, Backward time: 1.2033, DDP time: 0.0052, Total iter time: 2.3949 [2020-12-31 10:06:05,769][root][INFO] - ===> Epoch[851](4255/5): Loss -0.6422, Sem -0.6422, dir -0.9603, norm 0.2696 LR: 7.407e-02 Score 98.386 Data time: 0.1107, Forward time: 0.9503, Backward time: 1.2032, DDP time: 0.0052, Total iter time: 2.2852 [2020-12-31 10:06:17,524][root][INFO] - ===> Epoch[852](4260/5): Loss -0.6483, Sem -0.6483, dir -0.9598, norm 0.2624 LR: 7.403e-02 Score 98.357 Data time: 0.0885, Forward time: 0.9501, Backward time: 1.2032, DDP time: 0.0052, Total iter time: 2.2611 [2020-12-31 10:06:29,638][root][INFO] - ===> Epoch[853](4265/5): Loss -0.6394, Sem -0.6394, dir -0.9601, norm 0.2706 LR: 7.400e-02 Score 98.334 Data time: 0.1367, Forward time: 0.9502, Backward time: 1.2032, DDP time: 0.0052, Total iter time: 2.3249 [2020-12-31 10:06:42,507][root][INFO] - ===> Epoch[854](4270/5): Loss -0.6414, Sem -0.6414, dir -0.9611, norm 0.2698 LR: 7.397e-02 Score 98.353 Data time: 0.0912, Forward time: 0.9501, Backward time: 1.2034, DDP time: 0.0052, Total iter time: 2.4817 [2020-12-31 10:06:54,997][root][INFO] - ===> Epoch[855](4275/5): Loss -0.6341, Sem -0.6341, dir -0.9610, norm 0.2770 LR: 7.394e-02 Score 98.339 Data time: 0.1120, Forward time: 0.9501, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.4120 [2020-12-31 10:07:06,650][root][INFO] - ===> Epoch[856](4280/5): Loss -0.6416, Sem -0.6416, dir -0.9593, norm 0.2692 LR: 7.391e-02 Score 98.393 Data time: 0.0884, Forward time: 0.9500, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.2428 [2020-12-31 10:07:17,700][root][INFO] - ===> Epoch[857](4285/5): Loss -0.6297, Sem -0.6297, dir -0.9606, norm 0.2817 LR: 7.388e-02 Score 98.384 Data time: 0.1046, Forward time: 0.9500, Backward time: 1.2034, DDP time: 0.0052, Total iter time: 2.1206 [2020-12-31 10:07:29,341][root][INFO] - ===> Epoch[858](4290/5): Loss -0.6333, Sem -0.6333, dir -0.9597, norm 0.2746 LR: 7.385e-02 Score 98.282 Data time: 0.0870, Forward time: 0.9500, Backward time: 1.2033, DDP time: 0.0052, Total iter time: 2.2439 [2020-12-31 10:07:41,854][root][INFO] - ===> Epoch[859](4295/5): Loss -0.6384, Sem -0.6384, dir -0.9607, norm 0.2751 LR: 7.382e-02 Score 98.436 Data time: 0.1075, Forward time: 0.9500, Backward time: 1.2034, DDP time: 0.0052, Total iter time: 2.4032 [2020-12-31 10:07:54,112][root][INFO] - ===> Epoch[860](4300/5): Loss -0.6452, Sem -0.6452, dir -0.9609, norm 0.2670 LR: 7.379e-02 Score 98.378 Data time: 0.0944, Forward time: 0.9500, Backward time: 1.2034, DDP time: 0.0052, Total iter time: 2.3495 [2020-12-31 10:08:05,918][root][INFO] - ===> Epoch[861](4305/5): Loss -0.6332, Sem -0.6332, dir -0.9598, norm 0.2774 LR: 7.375e-02 Score 98.336 Data time: 0.1521, Forward time: 0.9500, Backward time: 1.2033, DDP time: 0.0052, Total iter time: 2.2637 [2020-12-31 10:08:17,749][root][INFO] - ===> Epoch[862](4310/5): Loss -0.6380, Sem -0.6380, dir -0.9597, norm 0.2711 LR: 7.372e-02 Score 98.317 Data time: 0.1120, Forward time: 0.9500, Backward time: 1.2032, DDP time: 0.0052, Total iter time: 2.2663 [2020-12-31 10:08:29,503][root][INFO] - ===> Epoch[863](4315/5): Loss -0.6522, Sem -0.6522, dir -0.9618, norm 0.2625 LR: 7.369e-02 Score 98.420 Data time: 0.1147, Forward time: 0.9500, Backward time: 1.2032, DDP time: 0.0052, Total iter time: 2.2577 [2020-12-31 10:08:41,798][root][INFO] - ===> Epoch[864](4320/5): Loss -0.6180, Sem -0.6180, dir -0.9590, norm 0.2893 LR: 7.366e-02 Score 98.315 Data time: 0.1187, Forward time: 0.9500, Backward time: 1.2032, DDP time: 0.0052, Total iter time: 2.3728 [2020-12-31 10:08:53,701][root][INFO] - ===> Epoch[865](4325/5): Loss -0.6172, Sem -0.6172, dir -0.9588, norm 0.2924 LR: 7.363e-02 Score 98.387 Data time: 0.1628, Forward time: 0.9499, Backward time: 1.2032, DDP time: 0.0052, Total iter time: 2.2980 [2020-12-31 10:09:06,662][root][INFO] - ===> Epoch[866](4330/5): Loss -0.6440, Sem -0.6440, dir -0.9618, norm 0.2693 LR: 7.360e-02 Score 98.377 Data time: 0.1203, Forward time: 0.9500, Backward time: 1.2033, DDP time: 0.0052, Total iter time: 2.5066 [2020-12-31 10:09:19,508][root][INFO] - ===> Epoch[867](4335/5): Loss -0.6437, Sem -0.6437, dir -0.9598, norm 0.2655 LR: 7.357e-02 Score 98.312 Data time: 0.0788, Forward time: 0.9499, Backward time: 1.2036, DDP time: 0.0052, Total iter time: 2.4755 [2020-12-31 10:09:30,741][root][INFO] - ===> Epoch[868](4340/5): Loss -0.6525, Sem -0.6525, dir -0.9619, norm 0.2595 LR: 7.354e-02 Score 98.348 Data time: 0.0921, Forward time: 0.9498, Backward time: 1.2034, DDP time: 0.0052, Total iter time: 2.1540 [2020-12-31 10:09:42,971][root][INFO] - ===> Epoch[869](4345/5): Loss -0.6298, Sem -0.6298, dir -0.9600, norm 0.2781 LR: 7.351e-02 Score 98.251 Data time: 0.0901, Forward time: 0.9497, Backward time: 1.2036, DDP time: 0.0052, Total iter time: 2.3671 [2020-12-31 10:09:54,592][root][INFO] - ===> Epoch[870](4350/5): Loss -0.6401, Sem -0.6401, dir -0.9609, norm 0.2717 LR: 7.348e-02 Score 98.365 Data time: 0.1328, Forward time: 0.9498, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.2431 [2020-12-31 10:10:05,951][root][INFO] - ===> Epoch[871](4355/5): Loss -0.6253, Sem -0.6253, dir -0.9587, norm 0.2810 LR: 7.344e-02 Score 98.270 Data time: 0.0691, Forward time: 0.9495, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.1800 [2020-12-31 10:10:18,416][root][INFO] - ===> Epoch[872](4360/5): Loss -0.6405, Sem -0.6405, dir -0.9606, norm 0.2695 LR: 7.341e-02 Score 98.312 Data time: 0.1075, Forward time: 0.9496, Backward time: 1.2036, DDP time: 0.0052, Total iter time: 2.3853 [2020-12-31 10:10:30,266][root][INFO] - ===> Epoch[873](4365/5): Loss -0.6347, Sem -0.6347, dir -0.9601, norm 0.2738 LR: 7.338e-02 Score 98.293 Data time: 0.0983, Forward time: 0.9495, Backward time: 1.2036, DDP time: 0.0052, Total iter time: 2.2715 [2020-12-31 10:10:41,709][root][INFO] - ===> Epoch[874](4370/5): Loss -0.6427, Sem -0.6427, dir -0.9608, norm 0.2676 LR: 7.335e-02 Score 98.346 Data time: 0.0942, Forward time: 0.9494, Backward time: 1.2036, DDP time: 0.0052, Total iter time: 2.1964 [2020-12-31 10:10:53,684][root][INFO] - ===> Epoch[875](4375/5): Loss -0.6312, Sem -0.6312, dir -0.9599, norm 0.2791 LR: 7.332e-02 Score 98.343 Data time: 0.1201, Forward time: 0.9494, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.2977 [2020-12-31 10:11:06,050][root][INFO] - ===> Epoch[876](4380/5): Loss -0.6267, Sem -0.6267, dir -0.9599, norm 0.2821 LR: 7.329e-02 Score 98.299 Data time: 0.1190, Forward time: 0.9495, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.3762 [2020-12-31 10:11:18,232][root][INFO] - ===> Epoch[877](4385/5): Loss -0.6417, Sem -0.6417, dir -0.9612, norm 0.2726 LR: 7.326e-02 Score 98.435 Data time: 0.1426, Forward time: 0.9495, Backward time: 1.2035, DDP time: 0.0052, Total iter time: 2.3502 [2020-12-31 10:11:29,315][root][INFO] - ===> Epoch[878](4390/5): Loss -0.6317, Sem -0.6317, dir -0.9597, norm 0.2776 LR: 7.323e-02 Score 98.359 Data time: 0.1112, Forward time: 0.9494, Backward time: 1.2033, DDP time: 0.0052, Total iter time: 2.1429 [2020-12-31 10:11:41,674][root][INFO] - ===> Epoch[879](4395/5): Loss -0.6280, Sem -0.6280, dir -0.9602, norm 0.2824 LR: 7.320e-02 Score 98.385 Data time: 0.1964, Forward time: 0.9494, Backward time: 1.2033, DDP time: 0.0052, Total iter time: 2.3805 [2020-12-31 10:11:52,979][root][INFO] - ===> Epoch[880](4400/5): Loss -0.6235, Sem -0.6235, dir -0.9599, norm 0.2841 LR: 7.317e-02 Score 98.290 Data time: 0.2242, Forward time: 0.9494, Backward time: 1.2031, DDP time: 0.0052, Total iter time: 2.1721 [2020-12-31 10:11:52,982][root][INFO] - ===> Start testing [2020-12-31 10:12:31,649][root][INFO] - 68/68: Data time: 0.0025, Iter time: 0.1261 Loss 1.425 (AVG: 0.496) Score 65.073 (AVG: 89.322) mIOU 67.504 mAP 82.961 mAcc 74.944 IOU: 59.906 0.000 74.902 72.741 93.464 88.963 34.749 72.431 98.472 65.257 80.413 84.028 52.230 mAP: 82.074 3.315 91.143 86.795 98.982 98.125 64.598 93.556 99.833 83.603 93.005 94.348 89.112 mAcc: 80.774 0.000 80.753 79.153 94.983 94.708 46.371 86.208 99.366 72.326 89.630 94.360 55.644 [2020-12-31 10:12:31,651][root][INFO] - Finished test. Elapsed time: 38.6663 [2020-12-31 10:12:35,019][root][INFO] - [2020-12-31 10:12:35,019][root][INFO] - ################################################################ [2020-12-31 10:12:35,019][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 10:12:35,019][root][INFO] - ################################################################ [2020-12-31 10:12:35,019][root][INFO] - clutter : nan nan nan [2020-12-31 10:12:35,019][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 10:12:35,019][root][INFO] - board : 0.703 0.773 0.826 [2020-12-31 10:12:35,020][root][INFO] - bookcase : 0.185 0.299 0.504 [2020-12-31 10:12:35,020][root][INFO] - ceiling : 0.673 0.728 0.789 [2020-12-31 10:12:35,020][root][INFO] - chair : 0.764 0.876 0.892 [2020-12-31 10:12:35,020][root][INFO] - column : 0.126 0.211 0.310 [2020-12-31 10:12:35,020][root][INFO] - door : 0.561 0.783 0.896 [2020-12-31 10:12:35,020][root][INFO] - floor : 1.000 1.000 1.000 [2020-12-31 10:12:35,020][root][INFO] - sofa : 0.397 0.531 0.636 [2020-12-31 10:12:35,020][root][INFO] - table : 0.183 0.400 0.678 [2020-12-31 10:12:35,020][root][INFO] - wall : 0.451 0.657 0.770 [2020-12-31 10:12:35,020][root][INFO] - window : 0.434 0.684 0.718 [2020-12-31 10:12:35,020][root][INFO] - ---------------------------------------------------------------- [2020-12-31 10:12:35,020][root][INFO] - average : 0.456 0.578 0.668 [2020-12-31 10:12:35,020][root][INFO] - [2020-12-31 10:12:35,183][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 10:12:35,809][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter4400.pth [2020-12-31 10:12:47,424][root][INFO] - ===> Epoch[881](4405/5): Loss -0.6537, Sem -0.6537, dir -0.9610, norm 0.2600 LR: 7.313e-02 Score 98.432 Data time: 0.1022, Forward time: 0.9494, Backward time: 1.2029, DDP time: 0.0052, Total iter time: 2.2269 [2020-12-31 10:12:58,947][root][INFO] - ===> Epoch[882](4410/5): Loss -0.6335, Sem -0.6335, dir -0.9615, norm 0.2788 LR: 7.310e-02 Score 98.347 Data time: 0.0909, Forward time: 0.9494, Backward time: 1.2028, DDP time: 0.0052, Total iter time: 2.2215 [2020-12-31 10:13:10,556][root][INFO] - ===> Epoch[883](4415/5): Loss -0.6384, Sem -0.6384, dir -0.9596, norm 0.2722 LR: 7.307e-02 Score 98.379 Data time: 0.0785, Forward time: 0.9493, Backward time: 1.2029, DDP time: 0.0052, Total iter time: 2.2484 [2020-12-31 10:13:22,405][root][INFO] - ===> Epoch[884](4420/5): Loss -0.6512, Sem -0.6512, dir -0.9616, norm 0.2619 LR: 7.304e-02 Score 98.405 Data time: 0.0693, Forward time: 0.9491, Backward time: 1.2031, DDP time: 0.0052, Total iter time: 2.2959 [2020-12-31 10:13:33,233][root][INFO] - ===> Epoch[885](4425/5): Loss -0.6453, Sem -0.6453, dir -0.9613, norm 0.2671 LR: 7.301e-02 Score 98.366 Data time: 0.1799, Forward time: 0.9490, Backward time: 1.2028, DDP time: 0.0052, Total iter time: 2.0585 [2020-12-31 10:13:45,007][root][INFO] - ===> Epoch[886](4430/5): Loss -0.6311, Sem -0.6311, dir -0.9595, norm 0.2797 LR: 7.298e-02 Score 98.386 Data time: 0.0860, Forward time: 0.9491, Backward time: 1.2027, DDP time: 0.0052, Total iter time: 2.2603 [2020-12-31 10:13:57,056][root][INFO] - ===> Epoch[887](4435/5): Loss -0.6391, Sem -0.6391, dir -0.9597, norm 0.2719 LR: 7.295e-02 Score 98.417 Data time: 0.1851, Forward time: 0.9491, Backward time: 1.2026, DDP time: 0.0052, Total iter time: 2.3200 [2020-12-31 10:14:08,728][root][INFO] - ===> Epoch[888](4440/5): Loss -0.6410, Sem -0.6410, dir -0.9600, norm 0.2715 LR: 7.292e-02 Score 98.420 Data time: 0.0832, Forward time: 0.9490, Backward time: 1.2026, DDP time: 0.0052, Total iter time: 2.2476 [2020-12-31 10:14:20,701][root][INFO] - ===> Epoch[889](4445/5): Loss -0.6601, Sem -0.6601, dir -0.9619, norm 0.2549 LR: 7.289e-02 Score 98.443 Data time: 0.1150, Forward time: 0.9490, Backward time: 1.2027, DDP time: 0.0052, Total iter time: 2.3092 [2020-12-31 10:14:32,942][root][INFO] - ===> Epoch[890](4450/5): Loss -0.6455, Sem -0.6455, dir -0.9610, norm 0.2681 LR: 7.285e-02 Score 98.438 Data time: 0.1270, Forward time: 0.9491, Backward time: 1.2026, DDP time: 0.0052, Total iter time: 2.3516 [2020-12-31 10:14:45,450][root][INFO] - ===> Epoch[891](4455/5): Loss -0.6380, Sem -0.6380, dir -0.9622, norm 0.2771 LR: 7.282e-02 Score 98.434 Data time: 0.2134, Forward time: 0.9491, Backward time: 1.2025, DDP time: 0.0052, Total iter time: 2.4089 [2020-12-31 10:14:56,926][root][INFO] - ===> Epoch[892](4460/5): Loss -0.6473, Sem -0.6473, dir -0.9615, norm 0.2684 LR: 7.279e-02 Score 98.495 Data time: 0.1047, Forward time: 0.9491, Backward time: 1.2024, DDP time: 0.0052, Total iter time: 2.2063 [2020-12-31 10:15:09,717][root][INFO] - ===> Epoch[893](4465/5): Loss -0.6440, Sem -0.6440, dir -0.9600, norm 0.2694 LR: 7.276e-02 Score 98.442 Data time: 0.1453, Forward time: 0.9493, Backward time: 1.2024, DDP time: 0.0052, Total iter time: 2.4665 [2020-12-31 10:15:21,225][root][INFO] - ===> Epoch[894](4470/5): Loss -0.6595, Sem -0.6595, dir -0.9623, norm 0.2574 LR: 7.273e-02 Score 98.486 Data time: 0.0787, Forward time: 0.9491, Backward time: 1.2024, DDP time: 0.0052, Total iter time: 2.2056 [2020-12-31 10:15:34,227][root][INFO] - ===> Epoch[895](4475/5): Loss -0.6380, Sem -0.6380, dir -0.9613, norm 0.2771 LR: 7.270e-02 Score 98.460 Data time: 0.0922, Forward time: 0.9492, Backward time: 1.2026, DDP time: 0.0052, Total iter time: 2.5094 [2020-12-31 10:15:46,374][root][INFO] - ===> Epoch[896](4480/5): Loss -0.6538, Sem -0.6538, dir -0.9622, norm 0.2614 LR: 7.267e-02 Score 98.426 Data time: 0.1082, Forward time: 0.9493, Backward time: 1.2026, DDP time: 0.0052, Total iter time: 2.3407 [2020-12-31 10:15:57,535][root][INFO] - ===> Epoch[897](4485/5): Loss -0.6382, Sem -0.6382, dir -0.9601, norm 0.2737 LR: 7.264e-02 Score 98.402 Data time: 0.0902, Forward time: 0.9493, Backward time: 1.2024, DDP time: 0.0052, Total iter time: 2.1289 [2020-12-31 10:16:09,060][root][INFO] - ===> Epoch[898](4490/5): Loss -0.6476, Sem -0.6476, dir -0.9623, norm 0.2669 LR: 7.261e-02 Score 98.399 Data time: 0.1261, Forward time: 0.9493, Backward time: 1.2022, DDP time: 0.0052, Total iter time: 2.2037 [2020-12-31 10:16:20,424][root][INFO] - ===> Epoch[899](4495/5): Loss -0.6517, Sem -0.6517, dir -0.9620, norm 0.2636 LR: 7.257e-02 Score 98.451 Data time: 0.0943, Forward time: 0.9493, Backward time: 1.2021, DDP time: 0.0052, Total iter time: 2.1699 [2020-12-31 10:16:32,383][root][INFO] - ===> Epoch[900](4500/5): Loss -0.6528, Sem -0.6528, dir -0.9617, norm 0.2606 LR: 7.254e-02 Score 98.379 Data time: 0.1233, Forward time: 0.9493, Backward time: 1.2020, DDP time: 0.0052, Total iter time: 2.2998 [2020-12-31 10:16:44,290][root][INFO] - ===> Epoch[901](4505/5): Loss -0.6571, Sem -0.6571, dir -0.9628, norm 0.2597 LR: 7.251e-02 Score 98.481 Data time: 0.1129, Forward time: 0.9493, Backward time: 1.2019, DDP time: 0.0052, Total iter time: 2.2854 [2020-12-31 10:16:56,451][root][INFO] - ===> Epoch[902](4510/5): Loss -0.6632, Sem -0.6632, dir -0.9622, norm 0.2514 LR: 7.248e-02 Score 98.406 Data time: 0.0933, Forward time: 0.9493, Backward time: 1.2020, DDP time: 0.0052, Total iter time: 2.3387 [2020-12-31 10:17:08,779][root][INFO] - ===> Epoch[903](4515/5): Loss -0.6549, Sem -0.6549, dir -0.9627, norm 0.2614 LR: 7.245e-02 Score 98.461 Data time: 0.1880, Forward time: 0.9493, Backward time: 1.2019, DDP time: 0.0052, Total iter time: 2.3646 [2020-12-31 10:17:20,970][root][INFO] - ===> Epoch[904](4520/5): Loss -0.6540, Sem -0.6540, dir -0.9619, norm 0.2616 LR: 7.242e-02 Score 98.460 Data time: 0.1018, Forward time: 0.9494, Backward time: 1.2020, DDP time: 0.0052, Total iter time: 2.3547 [2020-12-31 10:17:33,295][root][INFO] - ===> Epoch[905](4525/5): Loss -0.6598, Sem -0.6598, dir -0.9632, norm 0.2598 LR: 7.239e-02 Score 98.540 Data time: 0.1038, Forward time: 0.9493, Backward time: 1.2021, DDP time: 0.0052, Total iter time: 2.3715 [2020-12-31 10:17:45,190][root][INFO] - ===> Epoch[906](4530/5): Loss -0.6449, Sem -0.6449, dir -0.9612, norm 0.2698 LR: 7.236e-02 Score 98.457 Data time: 0.1568, Forward time: 0.9494, Backward time: 1.2019, DDP time: 0.0052, Total iter time: 2.2948 [2020-12-31 10:17:57,269][root][INFO] - ===> Epoch[907](4535/5): Loss -0.6466, Sem -0.6466, dir -0.9616, norm 0.2681 LR: 7.233e-02 Score 98.441 Data time: 0.0836, Forward time: 0.9494, Backward time: 1.2020, DDP time: 0.0052, Total iter time: 2.3337 [2020-12-31 10:18:08,571][root][INFO] - ===> Epoch[908](4540/5): Loss -0.6550, Sem -0.6550, dir -0.9626, norm 0.2610 LR: 7.229e-02 Score 98.444 Data time: 0.0871, Forward time: 0.9493, Backward time: 1.2020, DDP time: 0.0052, Total iter time: 2.1778 [2020-12-31 10:18:20,654][root][INFO] - ===> Epoch[909](4545/5): Loss -0.6595, Sem -0.6595, dir -0.9616, norm 0.2545 LR: 7.226e-02 Score 98.406 Data time: 0.0769, Forward time: 0.9492, Backward time: 1.2020, DDP time: 0.0052, Total iter time: 2.3276 [2020-12-31 10:18:32,783][root][INFO] - ===> Epoch[910](4550/5): Loss -0.6643, Sem -0.6643, dir -0.9637, norm 0.2545 LR: 7.223e-02 Score 98.505 Data time: 0.1483, Forward time: 0.9492, Backward time: 1.2020, DDP time: 0.0052, Total iter time: 2.3312 [2020-12-31 10:18:44,641][root][INFO] - ===> Epoch[911](4555/5): Loss -0.6470, Sem -0.6470, dir -0.9611, norm 0.2680 LR: 7.220e-02 Score 98.476 Data time: 0.1046, Forward time: 0.9493, Backward time: 1.2019, DDP time: 0.0052, Total iter time: 2.2694 [2020-12-31 10:18:57,343][root][INFO] - ===> Epoch[912](4560/5): Loss -0.6607, Sem -0.6607, dir -0.9626, norm 0.2565 LR: 7.217e-02 Score 98.484 Data time: 0.0876, Forward time: 0.9492, Backward time: 1.2022, DDP time: 0.0052, Total iter time: 2.4541 [2020-12-31 10:19:08,409][root][INFO] - ===> Epoch[913](4565/5): Loss -0.6528, Sem -0.6528, dir -0.9617, norm 0.2616 LR: 7.214e-02 Score 98.436 Data time: 0.1063, Forward time: 0.9491, Backward time: 1.2020, DDP time: 0.0051, Total iter time: 2.1139 [2020-12-31 10:19:20,382][root][INFO] - ===> Epoch[914](4570/5): Loss -0.6600, Sem -0.6600, dir -0.9626, norm 0.2563 LR: 7.211e-02 Score 98.460 Data time: 0.1031, Forward time: 0.9491, Backward time: 1.2020, DDP time: 0.0051, Total iter time: 2.3022 [2020-12-31 10:19:31,500][root][INFO] - ===> Epoch[915](4575/5): Loss -0.6390, Sem -0.6390, dir -0.9607, norm 0.2729 LR: 7.208e-02 Score 98.367 Data time: 0.0947, Forward time: 0.9490, Backward time: 1.2019, DDP time: 0.0051, Total iter time: 2.1338 [2020-12-31 10:19:43,718][root][INFO] - ===> Epoch[916](4580/5): Loss -0.6591, Sem -0.6591, dir -0.9635, norm 0.2560 LR: 7.205e-02 Score 98.390 Data time: 0.1078, Forward time: 0.9491, Backward time: 1.2019, DDP time: 0.0051, Total iter time: 2.3558 [2020-12-31 10:19:55,704][root][INFO] - ===> Epoch[917](4585/5): Loss -0.6536, Sem -0.6536, dir -0.9633, norm 0.2643 LR: 7.201e-02 Score 98.483 Data time: 0.0904, Forward time: 0.9490, Backward time: 1.2019, DDP time: 0.0051, Total iter time: 2.3078 [2020-12-31 10:20:07,256][root][INFO] - ===> Epoch[918](4590/5): Loss -0.6404, Sem -0.6404, dir -0.9598, norm 0.2701 LR: 7.198e-02 Score 98.376 Data time: 0.0839, Forward time: 0.9489, Backward time: 1.2019, DDP time: 0.0051, Total iter time: 2.2186 [2020-12-31 10:20:19,579][root][INFO] - ===> Epoch[919](4595/5): Loss -0.6470, Sem -0.6470, dir -0.9610, norm 0.2675 LR: 7.195e-02 Score 98.439 Data time: 0.0912, Forward time: 0.9489, Backward time: 1.2020, DDP time: 0.0051, Total iter time: 2.3731 [2020-12-31 10:20:31,918][root][INFO] - ===> Epoch[920](4600/5): Loss -0.6542, Sem -0.6542, dir -0.9624, norm 0.2611 LR: 7.192e-02 Score 98.443 Data time: 0.1282, Forward time: 0.9491, Backward time: 1.2019, DDP time: 0.0052, Total iter time: 2.3781 [2020-12-31 10:20:31,920][root][INFO] - ===> Start testing [2020-12-31 10:21:10,975][root][INFO] - 68/68: Data time: 0.4094, Iter time: 0.2642 Loss 0.447 (AVG: 0.527) Score 89.990 (AVG: 88.961) mIOU 67.379 mAP 82.893 mAcc 74.528 IOU: 57.887 0.000 76.275 71.709 92.301 89.957 30.754 71.707 98.333 67.945 80.758 83.512 54.785 mAP: 80.840 3.442 88.204 86.618 98.871 98.316 60.835 93.250 99.826 90.610 92.965 93.812 90.014 mAcc: 78.657 0.000 79.719 79.155 93.568 94.506 37.519 83.791 99.056 77.417 91.670 95.096 58.712 [2020-12-31 10:21:10,976][root][INFO] - Finished test. Elapsed time: 39.0540 [2020-12-31 10:21:14,474][root][INFO] - [2020-12-31 10:21:14,475][root][INFO] - ################################################################ [2020-12-31 10:21:14,475][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 10:21:14,475][root][INFO] - ################################################################ [2020-12-31 10:21:14,475][root][INFO] - clutter : nan nan nan [2020-12-31 10:21:14,475][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 10:21:14,475][root][INFO] - board : 0.677 0.707 0.760 [2020-12-31 10:21:14,475][root][INFO] - bookcase : 0.208 0.338 0.515 [2020-12-31 10:21:14,475][root][INFO] - ceiling : 0.711 0.749 0.826 [2020-12-31 10:21:14,475][root][INFO] - chair : 0.769 0.865 0.891 [2020-12-31 10:21:14,475][root][INFO] - column : 0.166 0.251 0.329 [2020-12-31 10:21:14,475][root][INFO] - door : 0.576 0.784 0.880 [2020-12-31 10:21:14,475][root][INFO] - floor : 1.000 1.000 1.000 [2020-12-31 10:21:14,475][root][INFO] - sofa : 0.419 0.573 0.727 [2020-12-31 10:21:14,475][root][INFO] - table : 0.181 0.343 0.628 [2020-12-31 10:21:14,475][root][INFO] - wall : 0.466 0.686 0.802 [2020-12-31 10:21:14,475][root][INFO] - window : 0.436 0.701 0.775 [2020-12-31 10:21:14,475][root][INFO] - ---------------------------------------------------------------- [2020-12-31 10:21:14,475][root][INFO] - average : 0.468 0.583 0.678 [2020-12-31 10:21:14,475][root][INFO] - [2020-12-31 10:21:14,665][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 10:21:15,239][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter4600.pth [2020-12-31 10:21:26,931][root][INFO] - ===> Epoch[921](4605/5): Loss -0.6525, Sem -0.6525, dir -0.9626, norm 0.2615 LR: 7.189e-02 Score 98.404 Data time: 0.0842, Forward time: 0.9491, Backward time: 1.2018, DDP time: 0.0052, Total iter time: 2.2361 [2020-12-31 10:21:39,029][root][INFO] - ===> Epoch[922](4610/5): Loss -0.6386, Sem -0.6386, dir -0.9610, norm 0.2752 LR: 7.186e-02 Score 98.427 Data time: 0.1309, Forward time: 0.9490, Backward time: 1.2019, DDP time: 0.0052, Total iter time: 2.3231 [2020-12-31 10:21:50,307][root][INFO] - ===> Epoch[923](4615/5): Loss -0.6464, Sem -0.6464, dir -0.9614, norm 0.2681 LR: 7.183e-02 Score 98.460 Data time: 0.1008, Forward time: 0.9491, Backward time: 1.2016, DDP time: 0.0052, Total iter time: 2.1643 [2020-12-31 10:22:02,379][root][INFO] - ===> Epoch[924](4620/5): Loss -0.6429, Sem -0.6429, dir -0.9624, norm 0.2732 LR: 7.180e-02 Score 98.452 Data time: 0.1473, Forward time: 0.9493, Backward time: 1.2014, DDP time: 0.0051, Total iter time: 2.3178 [2020-12-31 10:22:15,217][root][INFO] - ===> Epoch[925](4625/5): Loss -0.6486, Sem -0.6486, dir -0.9620, norm 0.2686 LR: 7.177e-02 Score 98.499 Data time: 0.1107, Forward time: 0.9494, Backward time: 1.2015, DDP time: 0.0052, Total iter time: 2.4729 [2020-12-31 10:22:26,958][root][INFO] - ===> Epoch[926](4630/5): Loss -0.6393, Sem -0.6393, dir -0.9612, norm 0.2745 LR: 7.173e-02 Score 98.429 Data time: 0.1062, Forward time: 0.9494, Backward time: 1.2014, DDP time: 0.0051, Total iter time: 2.2605 [2020-12-31 10:22:37,883][root][INFO] - ===> Epoch[927](4635/5): Loss -0.6394, Sem -0.6394, dir -0.9607, norm 0.2727 LR: 7.170e-02 Score 98.391 Data time: 0.1027, Forward time: 0.9495, Backward time: 1.2011, DDP time: 0.0051, Total iter time: 2.1009 [2020-12-31 10:22:51,714][root][INFO] - ===> Epoch[928](4640/5): Loss -0.6488, Sem -0.6488, dir -0.9622, norm 0.2692 LR: 7.167e-02 Score 98.516 Data time: 0.1160, Forward time: 0.9497, Backward time: 1.2013, DDP time: 0.0051, Total iter time: 2.6607 [2020-12-31 10:23:04,722][root][INFO] - ===> Epoch[929](4645/5): Loss -0.6404, Sem -0.6404, dir -0.9617, norm 0.2765 LR: 7.164e-02 Score 98.501 Data time: 0.1016, Forward time: 0.9497, Backward time: 1.2015, DDP time: 0.0051, Total iter time: 2.5142 [2020-12-31 10:23:16,695][root][INFO] - ===> Epoch[930](4650/5): Loss -0.6377, Sem -0.6377, dir -0.9611, norm 0.2781 LR: 7.161e-02 Score 98.500 Data time: 0.1266, Forward time: 0.9497, Backward time: 1.2015, DDP time: 0.0051, Total iter time: 2.3245 [2020-12-31 10:23:27,944][root][INFO] - ===> Epoch[931](4655/5): Loss -0.6475, Sem -0.6475, dir -0.9620, norm 0.2678 LR: 7.158e-02 Score 98.454 Data time: 0.0988, Forward time: 0.9497, Backward time: 1.2013, DDP time: 0.0051, Total iter time: 2.1484 [2020-12-31 10:23:39,745][root][INFO] - ===> Epoch[932](4660/5): Loss -0.6344, Sem -0.6344, dir -0.9605, norm 0.2798 LR: 7.155e-02 Score 98.488 Data time: 0.0917, Forward time: 0.9497, Backward time: 1.2013, DDP time: 0.0051, Total iter time: 2.2516 [2020-12-31 10:23:51,706][root][INFO] - ===> Epoch[933](4665/5): Loss -0.6461, Sem -0.6461, dir -0.9628, norm 0.2719 LR: 7.152e-02 Score 98.504 Data time: 0.1145, Forward time: 0.9496, Backward time: 1.2013, DDP time: 0.0051, Total iter time: 2.3002 [2020-12-31 10:24:03,319][root][INFO] - ===> Epoch[934](4670/5): Loss -0.6243, Sem -0.6243, dir -0.9592, norm 0.2843 LR: 7.149e-02 Score 98.338 Data time: 0.0865, Forward time: 0.9495, Backward time: 1.2013, DDP time: 0.0051, Total iter time: 2.2309 [2020-12-31 10:24:14,728][root][INFO] - ===> Epoch[935](4675/5): Loss -0.6344, Sem -0.6344, dir -0.9620, norm 0.2807 LR: 7.145e-02 Score 98.438 Data time: 0.0947, Forward time: 0.9496, Backward time: 1.2011, DDP time: 0.0051, Total iter time: 2.1854 [2020-12-31 10:24:26,586][root][INFO] - ===> Epoch[936](4680/5): Loss -0.6519, Sem -0.6519, dir -0.9628, norm 0.2660 LR: 7.142e-02 Score 98.501 Data time: 0.1433, Forward time: 0.9496, Backward time: 1.2010, DDP time: 0.0051, Total iter time: 2.2702 [2020-12-31 10:24:37,968][root][INFO] - ===> Epoch[937](4685/5): Loss -0.6287, Sem -0.6287, dir -0.9611, norm 0.2816 LR: 7.139e-02 Score 98.342 Data time: 0.1474, Forward time: 0.9496, Backward time: 1.2009, DDP time: 0.0051, Total iter time: 2.1919 [2020-12-31 10:24:49,130][root][INFO] - ===> Epoch[938](4690/5): Loss -0.6324, Sem -0.6324, dir -0.9589, norm 0.2765 LR: 7.136e-02 Score 98.323 Data time: 0.1169, Forward time: 0.9496, Backward time: 1.2007, DDP time: 0.0051, Total iter time: 2.1500 [2020-12-31 10:25:01,699][root][INFO] - ===> Epoch[939](4695/5): Loss -0.6414, Sem -0.6414, dir -0.9615, norm 0.2708 LR: 7.133e-02 Score 98.448 Data time: 0.0842, Forward time: 0.9496, Backward time: 1.2009, DDP time: 0.0051, Total iter time: 2.4328 [2020-12-31 10:25:14,323][root][INFO] - ===> Epoch[940](4700/5): Loss -0.6373, Sem -0.6373, dir -0.9613, norm 0.2765 LR: 7.130e-02 Score 98.400 Data time: 0.0865, Forward time: 0.9496, Backward time: 1.2010, DDP time: 0.0051, Total iter time: 2.4331 [2020-12-31 10:25:26,001][root][INFO] - ===> Epoch[941](4705/5): Loss -0.6520, Sem -0.6520, dir -0.9627, norm 0.2627 LR: 7.127e-02 Score 98.422 Data time: 0.1140, Forward time: 0.9495, Backward time: 1.2010, DDP time: 0.0051, Total iter time: 2.2550 [2020-12-31 10:25:38,003][root][INFO] - ===> Epoch[942](4710/5): Loss -0.6423, Sem -0.6423, dir -0.9611, norm 0.2698 LR: 7.124e-02 Score 98.372 Data time: 0.1367, Forward time: 0.9497, Backward time: 1.2008, DDP time: 0.0051, Total iter time: 2.3008 [2020-12-31 10:25:49,326][root][INFO] - ===> Epoch[943](4715/5): Loss -0.6471, Sem -0.6471, dir -0.9629, norm 0.2668 LR: 7.121e-02 Score 98.397 Data time: 0.1045, Forward time: 0.9498, Backward time: 1.2005, DDP time: 0.0051, Total iter time: 2.1669 [2020-12-31 10:26:01,097][root][INFO] - ===> Epoch[944](4720/5): Loss -0.6441, Sem -0.6441, dir -0.9629, norm 0.2694 LR: 7.117e-02 Score 98.365 Data time: 0.0951, Forward time: 0.9498, Backward time: 1.2005, DDP time: 0.0051, Total iter time: 2.2664 [2020-12-31 10:26:13,314][root][INFO] - ===> Epoch[945](4725/5): Loss -0.6576, Sem -0.6576, dir -0.9620, norm 0.2562 LR: 7.114e-02 Score 98.395 Data time: 0.0983, Forward time: 0.9499, Backward time: 1.2004, DDP time: 0.0051, Total iter time: 2.3550 [2020-12-31 10:26:25,649][root][INFO] - ===> Epoch[946](4730/5): Loss -0.6367, Sem -0.6367, dir -0.9610, norm 0.2737 LR: 7.111e-02 Score 98.352 Data time: 0.1560, Forward time: 0.9501, Backward time: 1.2003, DDP time: 0.0051, Total iter time: 2.3707 [2020-12-31 10:26:37,235][root][INFO] - ===> Epoch[947](4735/5): Loss -0.6475, Sem -0.6475, dir -0.9627, norm 0.2689 LR: 7.108e-02 Score 98.483 Data time: 0.0805, Forward time: 0.9500, Backward time: 1.2003, DDP time: 0.0051, Total iter time: 2.2261 [2020-12-31 10:26:49,095][root][INFO] - ===> Epoch[948](4740/5): Loss -0.6401, Sem -0.6401, dir -0.9605, norm 0.2721 LR: 7.105e-02 Score 98.393 Data time: 0.1268, Forward time: 0.9500, Backward time: 1.2002, DDP time: 0.0051, Total iter time: 2.2892 [2020-12-31 10:27:00,983][root][INFO] - ===> Epoch[949](4745/5): Loss -0.6471, Sem -0.6471, dir -0.9627, norm 0.2694 LR: 7.102e-02 Score 98.478 Data time: 0.0961, Forward time: 0.9500, Backward time: 1.2002, DDP time: 0.0051, Total iter time: 2.2890 [2020-12-31 10:27:13,556][root][INFO] - ===> Epoch[950](4750/5): Loss -0.6463, Sem -0.6463, dir -0.9621, norm 0.2664 LR: 7.099e-02 Score 98.359 Data time: 0.0881, Forward time: 0.9500, Backward time: 1.2004, DDP time: 0.0051, Total iter time: 2.4263 [2020-12-31 10:27:24,647][root][INFO] - ===> Epoch[951](4755/5): Loss -0.6518, Sem -0.6518, dir -0.9625, norm 0.2619 LR: 7.096e-02 Score 98.389 Data time: 0.0950, Forward time: 0.9500, Backward time: 1.2001, DDP time: 0.0051, Total iter time: 2.1209 [2020-12-31 10:27:36,299][root][INFO] - ===> Epoch[952](4760/5): Loss -0.6370, Sem -0.6370, dir -0.9607, norm 0.2702 LR: 7.093e-02 Score 98.280 Data time: 0.0867, Forward time: 0.9500, Backward time: 1.2001, DDP time: 0.0051, Total iter time: 2.2410 [2020-12-31 10:27:48,730][root][INFO] - ===> Epoch[953](4765/5): Loss -0.6394, Sem -0.6394, dir -0.9618, norm 0.2711 LR: 7.089e-02 Score 98.322 Data time: 0.1199, Forward time: 0.9499, Backward time: 1.2002, DDP time: 0.0051, Total iter time: 2.4015 [2020-12-31 10:28:01,013][root][INFO] - ===> Epoch[954](4770/5): Loss -0.6408, Sem -0.6408, dir -0.9622, norm 0.2768 LR: 7.086e-02 Score 98.508 Data time: 0.1602, Forward time: 0.9501, Backward time: 1.2001, DDP time: 0.0051, Total iter time: 2.3496 [2020-12-31 10:28:13,961][root][INFO] - ===> Epoch[955](4775/5): Loss -0.6527, Sem -0.6527, dir -0.9622, norm 0.2610 LR: 7.083e-02 Score 98.388 Data time: 0.1744, Forward time: 0.9501, Backward time: 1.2002, DDP time: 0.0051, Total iter time: 2.5074 [2020-12-31 10:28:26,043][root][INFO] - ===> Epoch[956](4780/5): Loss -0.6595, Sem -0.6595, dir -0.9639, norm 0.2589 LR: 7.080e-02 Score 98.478 Data time: 0.1010, Forward time: 0.9502, Backward time: 1.2001, DDP time: 0.0051, Total iter time: 2.3221 [2020-12-31 10:28:38,001][root][INFO] - ===> Epoch[957](4785/5): Loss -0.6649, Sem -0.6649, dir -0.9636, norm 0.2538 LR: 7.077e-02 Score 98.517 Data time: 0.0934, Forward time: 0.9503, Backward time: 1.2000, DDP time: 0.0051, Total iter time: 2.2997 [2020-12-31 10:28:49,106][root][INFO] - ===> Epoch[958](4790/5): Loss -0.6633, Sem -0.6633, dir -0.9646, norm 0.2563 LR: 7.074e-02 Score 98.513 Data time: 0.1169, Forward time: 0.9503, Backward time: 1.1998, DDP time: 0.0051, Total iter time: 2.1223 [2020-12-31 10:29:01,453][root][INFO] - ===> Epoch[959](4795/5): Loss -0.6573, Sem -0.6573, dir -0.9632, norm 0.2596 LR: 7.071e-02 Score 98.468 Data time: 0.1639, Forward time: 0.9503, Backward time: 1.1998, DDP time: 0.0051, Total iter time: 2.3776 [2020-12-31 10:29:12,810][root][INFO] - ===> Epoch[960](4800/5): Loss -0.6573, Sem -0.6573, dir -0.9631, norm 0.2603 LR: 7.068e-02 Score 98.488 Data time: 0.0720, Forward time: 0.9502, Backward time: 1.1998, DDP time: 0.0051, Total iter time: 2.1952 [2020-12-31 10:29:12,813][root][INFO] - ===> Start testing [2020-12-31 10:29:51,086][root][INFO] - 68/68: Data time: 0.0021, Iter time: 0.1131 Loss 0.160 (AVG: 0.534) Score 95.685 (AVG: 89.235) mIOU 67.347 mAP 83.309 mAcc 74.364 IOU: 59.852 0.000 78.595 72.719 92.960 87.411 26.757 70.532 98.560 70.648 79.819 83.200 54.463 mAP: 81.393 4.776 92.329 86.201 98.960 98.633 62.141 92.167 99.833 88.903 92.566 94.024 91.095 mAcc: 77.131 0.000 83.952 80.707 94.386 95.954 30.158 81.487 99.334 79.884 90.441 96.034 57.257 [2020-12-31 10:29:51,087][root][INFO] - Finished test. Elapsed time: 38.2724 [2020-12-31 10:29:54,463][root][INFO] - [2020-12-31 10:29:54,463][root][INFO] - ################################################################ [2020-12-31 10:29:54,463][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 10:29:54,463][root][INFO] - ################################################################ [2020-12-31 10:29:54,463][root][INFO] - clutter : nan nan nan [2020-12-31 10:29:54,463][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 10:29:54,464][root][INFO] - board : 0.764 0.808 0.876 [2020-12-31 10:29:54,464][root][INFO] - bookcase : 0.230 0.364 0.500 [2020-12-31 10:29:54,464][root][INFO] - ceiling : 0.719 0.779 0.851 [2020-12-31 10:29:54,464][root][INFO] - chair : 0.783 0.877 0.903 [2020-12-31 10:29:54,464][root][INFO] - column : 0.148 0.229 0.307 [2020-12-31 10:29:54,464][root][INFO] - door : 0.445 0.637 0.837 [2020-12-31 10:29:54,464][root][INFO] - floor : 0.857 0.857 0.890 [2020-12-31 10:29:54,464][root][INFO] - sofa : 0.440 0.573 0.794 [2020-12-31 10:29:54,464][root][INFO] - table : 0.153 0.349 0.671 [2020-12-31 10:29:54,464][root][INFO] - wall : 0.432 0.680 0.780 [2020-12-31 10:29:54,464][root][INFO] - window : 0.459 0.748 0.748 [2020-12-31 10:29:54,464][root][INFO] - ---------------------------------------------------------------- [2020-12-31 10:29:54,464][root][INFO] - average : 0.452 0.575 0.680 [2020-12-31 10:29:54,464][root][INFO] - [2020-12-31 10:29:54,627][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 10:29:55,194][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter4800.pth [2020-12-31 10:30:07,132][root][INFO] - ===> Epoch[961](4805/5): Loss -0.6564, Sem -0.6564, dir -0.9623, norm 0.2605 LR: 7.064e-02 Score 98.484 Data time: 0.1018, Forward time: 0.9502, Backward time: 1.1998, DDP time: 0.0051, Total iter time: 2.2855 [2020-12-31 10:30:18,891][root][INFO] - ===> Epoch[962](4810/5): Loss -0.6583, Sem -0.6583, dir -0.9634, norm 0.2606 LR: 7.061e-02 Score 98.528 Data time: 0.1112, Forward time: 0.9502, Backward time: 1.1997, DDP time: 0.0051, Total iter time: 2.2631 [2020-12-31 10:30:31,353][root][INFO] - ===> Epoch[963](4815/5): Loss -0.6582, Sem -0.6582, dir -0.9635, norm 0.2613 LR: 7.058e-02 Score 98.551 Data time: 0.1238, Forward time: 0.9502, Backward time: 1.1997, DDP time: 0.0051, Total iter time: 2.4095 [2020-12-31 10:30:43,156][root][INFO] - ===> Epoch[964](4820/5): Loss -0.6625, Sem -0.6625, dir -0.9634, norm 0.2557 LR: 7.055e-02 Score 98.490 Data time: 0.1117, Forward time: 0.9502, Backward time: 1.1997, DDP time: 0.0051, Total iter time: 2.2612 [2020-12-31 10:30:55,546][root][INFO] - ===> Epoch[965](4825/5): Loss -0.6637, Sem -0.6637, dir -0.9635, norm 0.2557 LR: 7.052e-02 Score 98.531 Data time: 0.1068, Forward time: 0.9502, Backward time: 1.1998, DDP time: 0.0051, Total iter time: 2.3750 [2020-12-31 10:31:07,007][root][INFO] - ===> Epoch[966](4830/5): Loss -0.6549, Sem -0.6549, dir -0.9621, norm 0.2622 LR: 7.049e-02 Score 98.496 Data time: 0.1371, Forward time: 0.9501, Backward time: 1.1997, DDP time: 0.0051, Total iter time: 2.2073 [2020-12-31 10:31:19,453][root][INFO] - ===> Epoch[967](4835/5): Loss -0.6451, Sem -0.6451, dir -0.9627, norm 0.2731 LR: 7.046e-02 Score 98.504 Data time: 0.1780, Forward time: 0.9503, Backward time: 1.1996, DDP time: 0.0051, Total iter time: 2.3967 [2020-12-31 10:31:30,854][root][INFO] - ===> Epoch[968](4840/5): Loss -0.6536, Sem -0.6536, dir -0.9628, norm 0.2638 LR: 7.043e-02 Score 98.479 Data time: 0.1520, Forward time: 0.9504, Backward time: 1.1993, DDP time: 0.0051, Total iter time: 2.1779 [2020-12-31 10:31:42,546][root][INFO] - ===> Epoch[969](4845/5): Loss -0.6586, Sem -0.6586, dir -0.9645, norm 0.2612 LR: 7.040e-02 Score 98.523 Data time: 0.0759, Forward time: 0.9503, Backward time: 1.1994, DDP time: 0.0051, Total iter time: 2.2558 [2020-12-31 10:31:54,517][root][INFO] - ===> Epoch[970](4850/5): Loss -0.6543, Sem -0.6543, dir -0.9627, norm 0.2619 LR: 7.036e-02 Score 98.467 Data time: 0.1000, Forward time: 0.9503, Backward time: 1.1994, DDP time: 0.0051, Total iter time: 2.3004 [2020-12-31 10:32:06,329][root][INFO] - ===> Epoch[971](4855/5): Loss -0.6648, Sem -0.6648, dir -0.9642, norm 0.2550 LR: 7.033e-02 Score 98.520 Data time: 0.0870, Forward time: 0.9503, Backward time: 1.1993, DDP time: 0.0051, Total iter time: 2.2758 [2020-12-31 10:32:17,879][root][INFO] - ===> Epoch[972](4860/5): Loss -0.6692, Sem -0.6692, dir -0.9643, norm 0.2515 LR: 7.030e-02 Score 98.543 Data time: 0.0918, Forward time: 0.9502, Backward time: 1.1993, DDP time: 0.0051, Total iter time: 2.2192 [2020-12-31 10:32:29,608][root][INFO] - ===> Epoch[973](4865/5): Loss -0.6665, Sem -0.6665, dir -0.9636, norm 0.2518 LR: 7.027e-02 Score 98.502 Data time: 0.1297, Forward time: 0.9503, Backward time: 1.1992, DDP time: 0.0051, Total iter time: 2.2466 [2020-12-31 10:32:41,420][root][INFO] - ===> Epoch[974](4870/5): Loss -0.6711, Sem -0.6711, dir -0.9649, norm 0.2494 LR: 7.024e-02 Score 98.531 Data time: 0.1333, Forward time: 0.9503, Backward time: 1.1991, DDP time: 0.0051, Total iter time: 2.2727 [2020-12-31 10:32:53,196][root][INFO] - ===> Epoch[975](4875/5): Loss -0.6661, Sem -0.6661, dir -0.9644, norm 0.2534 LR: 7.021e-02 Score 98.505 Data time: 0.1065, Forward time: 0.9504, Backward time: 1.1990, DDP time: 0.0051, Total iter time: 2.2601 [2020-12-31 10:33:04,490][root][INFO] - ===> Epoch[976](4880/5): Loss -0.6596, Sem -0.6596, dir -0.9628, norm 0.2588 LR: 7.018e-02 Score 98.510 Data time: 0.0900, Forward time: 0.9503, Backward time: 1.1989, DDP time: 0.0051, Total iter time: 2.1617 [2020-12-31 10:33:17,739][root][INFO] - ===> Epoch[977](4885/5): Loss -0.6538, Sem -0.6538, dir -0.9632, norm 0.2618 LR: 7.015e-02 Score 98.428 Data time: 0.1002, Forward time: 0.9503, Backward time: 1.1991, DDP time: 0.0051, Total iter time: 2.5458 [2020-12-31 10:33:30,009][root][INFO] - ===> Epoch[978](4890/5): Loss -0.6610, Sem -0.6610, dir -0.9638, norm 0.2577 LR: 7.011e-02 Score 98.490 Data time: 0.1635, Forward time: 0.9504, Backward time: 1.1990, DDP time: 0.0051, Total iter time: 2.3595 [2020-12-31 10:33:42,112][root][INFO] - ===> Epoch[979](4895/5): Loss -0.6589, Sem -0.6589, dir -0.9639, norm 0.2605 LR: 7.008e-02 Score 98.528 Data time: 0.1047, Forward time: 0.9504, Backward time: 1.1990, DDP time: 0.0051, Total iter time: 2.3272 [2020-12-31 10:33:54,286][root][INFO] - ===> Epoch[980](4900/5): Loss -0.6605, Sem -0.6605, dir -0.9622, norm 0.2558 LR: 7.005e-02 Score 98.451 Data time: 0.1053, Forward time: 0.9503, Backward time: 1.1991, DDP time: 0.0051, Total iter time: 2.3483 [2020-12-31 10:34:06,769][root][INFO] - ===> Epoch[981](4905/5): Loss -0.6576, Sem -0.6576, dir -0.9642, norm 0.2641 LR: 7.002e-02 Score 98.601 Data time: 0.1649, Forward time: 0.9504, Backward time: 1.1991, DDP time: 0.0051, Total iter time: 2.3974 [2020-12-31 10:34:18,644][root][INFO] - ===> Epoch[982](4910/5): Loss -0.6563, Sem -0.6563, dir -0.9632, norm 0.2630 LR: 6.999e-02 Score 98.538 Data time: 0.1000, Forward time: 0.9505, Backward time: 1.1990, DDP time: 0.0051, Total iter time: 2.2822 [2020-12-31 10:34:32,026][root][INFO] - ===> Epoch[983](4915/5): Loss -0.6569, Sem -0.6569, dir -0.9631, norm 0.2608 LR: 6.996e-02 Score 98.477 Data time: 0.0822, Forward time: 0.9504, Backward time: 1.1994, DDP time: 0.0051, Total iter time: 2.5945 [2020-12-31 10:34:44,602][root][INFO] - ===> Epoch[984](4920/5): Loss -0.6578, Sem -0.6578, dir -0.9628, norm 0.2600 LR: 6.993e-02 Score 98.507 Data time: 0.1202, Forward time: 0.9504, Backward time: 1.1994, DDP time: 0.0051, Total iter time: 2.4063 [2020-12-31 10:34:56,897][root][INFO] - ===> Epoch[985](4925/5): Loss -0.6612, Sem -0.6612, dir -0.9629, norm 0.2573 LR: 6.990e-02 Score 98.528 Data time: 0.1032, Forward time: 0.9504, Backward time: 1.1994, DDP time: 0.0051, Total iter time: 2.3659 [2020-12-31 10:35:08,482][root][INFO] - ===> Epoch[986](4930/5): Loss -0.6583, Sem -0.6583, dir -0.9636, norm 0.2589 LR: 6.986e-02 Score 98.456 Data time: 0.0953, Forward time: 0.9504, Backward time: 1.1994, DDP time: 0.0051, Total iter time: 2.2254 [2020-12-31 10:35:22,027][root][INFO] - ===> Epoch[987](4935/5): Loss -0.6659, Sem -0.6659, dir -0.9640, norm 0.2531 LR: 6.983e-02 Score 98.498 Data time: 0.0965, Forward time: 0.9504, Backward time: 1.1997, DDP time: 0.0051, Total iter time: 2.6168 [2020-12-31 10:35:33,982][root][INFO] - ===> Epoch[988](4940/5): Loss -0.6619, Sem -0.6619, dir -0.9637, norm 0.2555 LR: 6.980e-02 Score 98.475 Data time: 0.1527, Forward time: 0.9503, Backward time: 1.1997, DDP time: 0.0051, Total iter time: 2.3019 [2020-12-31 10:35:45,882][root][INFO] - ===> Epoch[989](4945/5): Loss -0.6640, Sem -0.6640, dir -0.9641, norm 0.2556 LR: 6.977e-02 Score 98.531 Data time: 0.0676, Forward time: 0.9502, Backward time: 1.1998, DDP time: 0.0051, Total iter time: 2.3004 [2020-12-31 10:35:58,254][root][INFO] - ===> Epoch[990](4950/5): Loss -0.6549, Sem -0.6549, dir -0.9633, norm 0.2594 LR: 6.974e-02 Score 98.378 Data time: 0.0769, Forward time: 0.9500, Backward time: 1.2001, DDP time: 0.0051, Total iter time: 2.3932 [2020-12-31 10:36:10,647][root][INFO] - ===> Epoch[991](4955/5): Loss -0.6653, Sem -0.6653, dir -0.9629, norm 0.2513 LR: 6.971e-02 Score 98.482 Data time: 0.1433, Forward time: 0.9500, Backward time: 1.2001, DDP time: 0.0051, Total iter time: 2.3947 [2020-12-31 10:36:22,902][root][INFO] - ===> Epoch[992](4960/5): Loss -0.6571, Sem -0.6571, dir -0.9639, norm 0.2613 LR: 6.968e-02 Score 98.499 Data time: 0.0995, Forward time: 0.9501, Backward time: 1.2001, DDP time: 0.0051, Total iter time: 2.3494 [2020-12-31 10:36:34,762][root][INFO] - ===> Epoch[993](4965/5): Loss -0.6627, Sem -0.6627, dir -0.9630, norm 0.2535 LR: 6.965e-02 Score 98.444 Data time: 0.0829, Forward time: 0.9500, Backward time: 1.2001, DDP time: 0.0051, Total iter time: 2.2794 [2020-12-31 10:36:46,953][root][INFO] - ===> Epoch[994](4970/5): Loss -0.6598, Sem -0.6598, dir -0.9638, norm 0.2583 LR: 6.961e-02 Score 98.502 Data time: 0.1705, Forward time: 0.9500, Backward time: 1.2001, DDP time: 0.0051, Total iter time: 2.3687 [2020-12-31 10:36:59,129][root][INFO] - ===> Epoch[995](4975/5): Loss -0.6614, Sem -0.6614, dir -0.9636, norm 0.2574 LR: 6.958e-02 Score 98.515 Data time: 0.1090, Forward time: 0.9501, Backward time: 1.2001, DDP time: 0.0051, Total iter time: 2.3520 [2020-12-31 10:37:11,246][root][INFO] - ===> Epoch[996](4980/5): Loss -0.6633, Sem -0.6633, dir -0.9643, norm 0.2569 LR: 6.955e-02 Score 98.521 Data time: 0.0938, Forward time: 0.9500, Backward time: 1.2002, DDP time: 0.0051, Total iter time: 2.3334 [2020-12-31 10:37:23,622][root][INFO] - ===> Epoch[997](4985/5): Loss -0.6620, Sem -0.6620, dir -0.9639, norm 0.2578 LR: 6.952e-02 Score 98.539 Data time: 0.1177, Forward time: 0.9500, Backward time: 1.2002, DDP time: 0.0051, Total iter time: 2.3828 [2020-12-31 10:37:35,834][root][INFO] - ===> Epoch[998](4990/5): Loss -0.6593, Sem -0.6593, dir -0.9637, norm 0.2603 LR: 6.949e-02 Score 98.537 Data time: 0.1030, Forward time: 0.9501, Backward time: 1.2002, DDP time: 0.0051, Total iter time: 2.3541 [2020-12-31 10:37:47,133][root][INFO] - ===> Epoch[999](4995/5): Loss -0.6703, Sem -0.6703, dir -0.9647, norm 0.2515 LR: 6.946e-02 Score 98.577 Data time: 0.0962, Forward time: 0.9500, Backward time: 1.2001, DDP time: 0.0051, Total iter time: 2.1643 [2020-12-31 10:37:59,133][root][INFO] - ===> Epoch[1000](5000/5): Loss -0.6613, Sem -0.6613, dir -0.9632, norm 0.2563 LR: 6.943e-02 Score 98.489 Data time: 0.1050, Forward time: 0.9500, Backward time: 1.2001, DDP time: 0.0051, Total iter time: 2.3038 [2020-12-31 10:37:59,136][root][INFO] - ===> Start testing [2020-12-31 10:38:37,431][root][INFO] - 68/68: Data time: 0.7322, Iter time: 0.2976 Loss 0.559 (AVG: 0.513) Score 87.402 (AVG: 89.409) mIOU 67.128 mAP 83.059 mAcc 74.350 IOU: 60.422 0.000 77.670 73.921 93.221 89.274 24.699 69.689 98.249 70.709 79.534 84.201 51.072 mAP: 82.058 3.433 93.869 87.528 98.796 98.404 58.211 94.029 99.812 89.292 92.677 93.792 87.866 mAcc: 76.746 0.000 83.426 83.018 94.333 94.272 27.039 90.586 99.446 76.046 92.020 95.349 54.269 [2020-12-31 10:38:37,432][root][INFO] - Finished test. Elapsed time: 38.2944 [2020-12-31 10:38:41,135][root][INFO] - [2020-12-31 10:38:41,135][root][INFO] - ################################################################ [2020-12-31 10:38:41,135][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 10:38:41,135][root][INFO] - ################################################################ [2020-12-31 10:38:41,135][root][INFO] - clutter : nan nan nan [2020-12-31 10:38:41,135][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 10:38:41,135][root][INFO] - board : 0.711 0.801 0.872 [2020-12-31 10:38:41,135][root][INFO] - bookcase : 0.241 0.381 0.498 [2020-12-31 10:38:41,135][root][INFO] - ceiling : 0.702 0.750 0.760 [2020-12-31 10:38:41,135][root][INFO] - chair : 0.747 0.848 0.876 [2020-12-31 10:38:41,135][root][INFO] - column : 0.133 0.212 0.319 [2020-12-31 10:38:41,135][root][INFO] - door : 0.657 0.851 0.927 [2020-12-31 10:38:41,135][root][INFO] - floor : 1.000 1.000 1.000 [2020-12-31 10:38:41,135][root][INFO] - sofa : 0.433 0.584 0.717 [2020-12-31 10:38:41,135][root][INFO] - table : 0.190 0.371 0.644 [2020-12-31 10:38:41,135][root][INFO] - wall : 0.479 0.694 0.796 [2020-12-31 10:38:41,135][root][INFO] - window : 0.411 0.648 0.703 [2020-12-31 10:38:41,135][root][INFO] - ---------------------------------------------------------------- [2020-12-31 10:38:41,136][root][INFO] - average : 0.475 0.595 0.676 [2020-12-31 10:38:41,136][root][INFO] - [2020-12-31 10:38:41,313][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 10:38:41,886][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter5000.pth [2020-12-31 10:38:53,138][root][INFO] - ===> Epoch[1001](5005/5): Loss -0.6692, Sem -0.6692, dir -0.9646, norm 0.2493 LR: 6.940e-02 Score 98.498 Data time: 0.0794, Forward time: 0.9498, Backward time: 1.2002, DDP time: 0.0051, Total iter time: 2.1526 [2020-12-31 10:39:05,292][root][INFO] - ===> Epoch[1002](5010/5): Loss -0.6673, Sem -0.6673, dir -0.9646, norm 0.2536 LR: 6.936e-02 Score 98.540 Data time: 0.1064, Forward time: 0.9499, Backward time: 1.2001, DDP time: 0.0051, Total iter time: 2.3289 [2020-12-31 10:39:18,048][root][INFO] - ===> Epoch[1003](5015/5): Loss -0.6624, Sem -0.6624, dir -0.9649, norm 0.2578 LR: 6.933e-02 Score 98.545 Data time: 0.0894, Forward time: 0.9497, Backward time: 1.2004, DDP time: 0.0051, Total iter time: 2.4659 [2020-12-31 10:39:30,234][root][INFO] - ===> Epoch[1004](5020/5): Loss -0.6760, Sem -0.6760, dir -0.9660, norm 0.2457 LR: 6.930e-02 Score 98.507 Data time: 0.1679, Forward time: 0.9498, Backward time: 1.2003, DDP time: 0.0051, Total iter time: 2.3483 [2020-12-31 10:39:43,113][root][INFO] - ===> Epoch[1005](5025/5): Loss -0.6630, Sem -0.6630, dir -0.9645, norm 0.2585 LR: 6.927e-02 Score 98.562 Data time: 0.0837, Forward time: 0.9498, Backward time: 1.2005, DDP time: 0.0051, Total iter time: 2.4832 [2020-12-31 10:39:54,435][root][INFO] - ===> Epoch[1006](5030/5): Loss -0.6614, Sem -0.6614, dir -0.9636, norm 0.2558 LR: 6.924e-02 Score 98.462 Data time: 0.0790, Forward time: 0.9496, Backward time: 1.2005, DDP time: 0.0051, Total iter time: 2.1653 [2020-12-31 10:40:06,175][root][INFO] - ===> Epoch[1007](5035/5): Loss -0.6676, Sem -0.6676, dir -0.9645, norm 0.2512 LR: 6.921e-02 Score 98.481 Data time: 0.1121, Forward time: 0.9496, Backward time: 1.2004, DDP time: 0.0051, Total iter time: 2.2609 [2020-12-31 10:40:18,254][root][INFO] - ===> Epoch[1008](5040/5): Loss -0.6646, Sem -0.6646, dir -0.9645, norm 0.2549 LR: 6.918e-02 Score 98.491 Data time: 0.1038, Forward time: 0.9496, Backward time: 1.2005, DDP time: 0.0051, Total iter time: 2.3288 [2020-12-31 10:40:31,627][root][INFO] - ===> Epoch[1009](5045/5): Loss -0.6756, Sem -0.6756, dir -0.9650, norm 0.2467 LR: 6.915e-02 Score 98.585 Data time: 0.1204, Forward time: 0.9497, Backward time: 1.2006, DDP time: 0.0051, Total iter time: 2.5678 [2020-12-31 10:40:44,145][root][INFO] - ===> Epoch[1010](5050/5): Loss -0.6665, Sem -0.6665, dir -0.9637, norm 0.2516 LR: 6.911e-02 Score 98.466 Data time: 0.2484, Forward time: 0.9498, Backward time: 1.2005, DDP time: 0.0051, Total iter time: 2.3935 [2020-12-31 10:40:55,677][root][INFO] - ===> Epoch[1011](5055/5): Loss -0.6663, Sem -0.6663, dir -0.9659, norm 0.2569 LR: 6.908e-02 Score 98.596 Data time: 0.1013, Forward time: 0.9498, Backward time: 1.2003, DDP time: 0.0051, Total iter time: 2.2141 [2020-12-31 10:41:07,953][root][INFO] - ===> Epoch[1012](5060/5): Loss -0.6739, Sem -0.6739, dir -0.9648, norm 0.2500 LR: 6.905e-02 Score 98.623 Data time: 0.1192, Forward time: 0.9499, Backward time: 1.2003, DDP time: 0.0051, Total iter time: 2.3601 [2020-12-31 10:41:20,026][root][INFO] - ===> Epoch[1013](5065/5): Loss -0.6610, Sem -0.6610, dir -0.9640, norm 0.2583 LR: 6.902e-02 Score 98.508 Data time: 0.0901, Forward time: 0.9498, Backward time: 1.2004, DDP time: 0.0051, Total iter time: 2.3222 [2020-12-31 10:41:32,015][root][INFO] - ===> Epoch[1014](5070/5): Loss -0.6772, Sem -0.6772, dir -0.9646, norm 0.2433 LR: 6.899e-02 Score 98.555 Data time: 0.1580, Forward time: 0.9499, Backward time: 1.2003, DDP time: 0.0051, Total iter time: 2.3003 [2020-12-31 10:41:43,941][root][INFO] - ===> Epoch[1015](5075/5): Loss -0.6635, Sem -0.6635, dir -0.9635, norm 0.2539 LR: 6.896e-02 Score 98.473 Data time: 0.0785, Forward time: 0.9498, Backward time: 1.2004, DDP time: 0.0051, Total iter time: 2.3174 [2020-12-31 10:41:55,546][root][INFO] - ===> Epoch[1016](5080/5): Loss -0.6673, Sem -0.6673, dir -0.9655, norm 0.2537 LR: 6.893e-02 Score 98.514 Data time: 0.0936, Forward time: 0.9497, Backward time: 1.2004, DDP time: 0.0051, Total iter time: 2.2202 [2020-12-31 10:42:07,459][root][INFO] - ===> Epoch[1017](5085/5): Loss -0.6704, Sem -0.6704, dir -0.9651, norm 0.2511 LR: 6.890e-02 Score 98.564 Data time: 0.1123, Forward time: 0.9499, Backward time: 1.2002, DDP time: 0.0051, Total iter time: 2.2902 [2020-12-31 10:42:19,183][root][INFO] - ===> Epoch[1018](5090/5): Loss -0.6606, Sem -0.6606, dir -0.9641, norm 0.2543 LR: 6.886e-02 Score 98.404 Data time: 0.0925, Forward time: 0.9498, Backward time: 1.2002, DDP time: 0.0051, Total iter time: 2.2541 [2020-12-31 10:42:30,778][root][INFO] - ===> Epoch[1019](5095/5): Loss -0.6702, Sem -0.6702, dir -0.9637, norm 0.2471 LR: 6.883e-02 Score 98.466 Data time: 0.1054, Forward time: 0.9498, Backward time: 1.2001, DDP time: 0.0051, Total iter time: 2.2280 [2020-12-31 10:42:42,523][root][INFO] - ===> Epoch[1020](5100/5): Loss -0.6678, Sem -0.6678, dir -0.9650, norm 0.2522 LR: 6.880e-02 Score 98.529 Data time: 0.0879, Forward time: 0.9498, Backward time: 1.2001, DDP time: 0.0051, Total iter time: 2.2601 [2020-12-31 10:42:54,037][root][INFO] - ===> Epoch[1021](5105/5): Loss -0.6589, Sem -0.6589, dir -0.9637, norm 0.2592 LR: 6.877e-02 Score 98.492 Data time: 0.0958, Forward time: 0.9497, Backward time: 1.2001, DDP time: 0.0051, Total iter time: 2.2247 [2020-12-31 10:43:05,755][root][INFO] - ===> Epoch[1022](5110/5): Loss -0.6620, Sem -0.6620, dir -0.9637, norm 0.2579 LR: 6.874e-02 Score 98.541 Data time: 0.1184, Forward time: 0.9498, Backward time: 1.2000, DDP time: 0.0051, Total iter time: 2.2552 [2020-12-31 10:43:18,097][root][INFO] - ===> Epoch[1023](5115/5): Loss -0.6722, Sem -0.6722, dir -0.9651, norm 0.2494 LR: 6.871e-02 Score 98.551 Data time: 0.1097, Forward time: 0.9498, Backward time: 1.2000, DDP time: 0.0051, Total iter time: 2.3783 [2020-12-31 10:43:29,452][root][INFO] - ===> Epoch[1024](5120/5): Loss -0.6592, Sem -0.6592, dir -0.9635, norm 0.2583 LR: 6.868e-02 Score 98.470 Data time: 0.0890, Forward time: 0.9498, Backward time: 1.1999, DDP time: 0.0051, Total iter time: 2.1933 [2020-12-31 10:43:41,171][root][INFO] - ===> Epoch[1025](5125/5): Loss -0.6578, Sem -0.6578, dir -0.9649, norm 0.2624 LR: 6.865e-02 Score 98.541 Data time: 0.0829, Forward time: 0.9498, Backward time: 1.1999, DDP time: 0.0051, Total iter time: 2.2675 [2020-12-31 10:43:52,908][root][INFO] - ===> Epoch[1026](5130/5): Loss -0.6670, Sem -0.6670, dir -0.9642, norm 0.2534 LR: 6.861e-02 Score 98.526 Data time: 0.0923, Forward time: 0.9498, Backward time: 1.1998, DDP time: 0.0051, Total iter time: 2.2488 [2020-12-31 10:44:04,827][root][INFO] - ===> Epoch[1027](5135/5): Loss -0.6616, Sem -0.6616, dir -0.9641, norm 0.2559 LR: 6.858e-02 Score 98.474 Data time: 0.1063, Forward time: 0.9498, Backward time: 1.1998, DDP time: 0.0051, Total iter time: 2.2912 [2020-12-31 10:44:16,502][root][INFO] - ===> Epoch[1028](5140/5): Loss -0.6600, Sem -0.6600, dir -0.9638, norm 0.2571 LR: 6.855e-02 Score 98.457 Data time: 0.1140, Forward time: 0.9499, Backward time: 1.1997, DDP time: 0.0051, Total iter time: 2.2407 [2020-12-31 10:44:27,943][root][INFO] - ===> Epoch[1029](5145/5): Loss -0.6603, Sem -0.6603, dir -0.9640, norm 0.2575 LR: 6.852e-02 Score 98.462 Data time: 0.1100, Forward time: 0.9498, Backward time: 1.1996, DDP time: 0.0051, Total iter time: 2.1985 [2020-12-31 10:44:39,445][root][INFO] - ===> Epoch[1030](5150/5): Loss -0.6741, Sem -0.6741, dir -0.9646, norm 0.2449 LR: 6.849e-02 Score 98.470 Data time: 0.0902, Forward time: 0.9498, Backward time: 1.1995, DDP time: 0.0051, Total iter time: 2.2049 [2020-12-31 10:44:51,917][root][INFO] - ===> Epoch[1031](5155/5): Loss -0.6658, Sem -0.6658, dir -0.9647, norm 0.2556 LR: 6.846e-02 Score 98.553 Data time: 0.1368, Forward time: 0.9499, Backward time: 1.1996, DDP time: 0.0051, Total iter time: 2.4078 [2020-12-31 10:45:03,584][root][INFO] - ===> Epoch[1032](5160/5): Loss -0.6613, Sem -0.6613, dir -0.9638, norm 0.2565 LR: 6.843e-02 Score 98.473 Data time: 0.1290, Forward time: 0.9499, Backward time: 1.1994, DDP time: 0.0051, Total iter time: 2.2405 [2020-12-31 10:45:16,526][root][INFO] - ===> Epoch[1033](5165/5): Loss -0.6774, Sem -0.6774, dir -0.9662, norm 0.2452 LR: 6.840e-02 Score 98.550 Data time: 0.0974, Forward time: 0.9499, Backward time: 1.1996, DDP time: 0.0051, Total iter time: 2.5002 [2020-12-31 10:45:28,734][root][INFO] - ===> Epoch[1034](5170/5): Loss -0.6777, Sem -0.6777, dir -0.9655, norm 0.2428 LR: 6.836e-02 Score 98.508 Data time: 0.1075, Forward time: 0.9499, Backward time: 1.1996, DDP time: 0.0051, Total iter time: 2.3640 [2020-12-31 10:45:41,123][root][INFO] - ===> Epoch[1035](5175/5): Loss -0.6708, Sem -0.6708, dir -0.9648, norm 0.2488 LR: 6.833e-02 Score 98.493 Data time: 0.1106, Forward time: 0.9500, Backward time: 1.1996, DDP time: 0.0051, Total iter time: 2.3860 [2020-12-31 10:45:52,662][root][INFO] - ===> Epoch[1036](5180/5): Loss -0.6622, Sem -0.6622, dir -0.9630, norm 0.2544 LR: 6.830e-02 Score 98.481 Data time: 0.1058, Forward time: 0.9500, Backward time: 1.1995, DDP time: 0.0051, Total iter time: 2.2208 [2020-12-31 10:46:03,617][root][INFO] - ===> Epoch[1037](5185/5): Loss -0.6739, Sem -0.6739, dir -0.9644, norm 0.2472 LR: 6.827e-02 Score 98.548 Data time: 0.0972, Forward time: 0.9499, Backward time: 1.1994, DDP time: 0.0051, Total iter time: 2.0943 [2020-12-31 10:46:15,688][root][INFO] - ===> Epoch[1038](5190/5): Loss -0.6679, Sem -0.6679, dir -0.9646, norm 0.2519 LR: 6.824e-02 Score 98.496 Data time: 0.0757, Forward time: 0.9498, Backward time: 1.1995, DDP time: 0.0051, Total iter time: 2.3247 [2020-12-31 10:46:27,902][root][INFO] - ===> Epoch[1039](5195/5): Loss -0.6704, Sem -0.6704, dir -0.9650, norm 0.2516 LR: 6.821e-02 Score 98.590 Data time: 0.1191, Forward time: 0.9499, Backward time: 1.1995, DDP time: 0.0051, Total iter time: 2.3390 [2020-12-31 10:46:39,097][root][INFO] - ===> Epoch[1040](5200/5): Loss -0.6807, Sem -0.6807, dir -0.9659, norm 0.2426 LR: 6.818e-02 Score 98.572 Data time: 0.0794, Forward time: 0.9498, Backward time: 1.1994, DDP time: 0.0051, Total iter time: 2.1423 [2020-12-31 10:46:39,099][root][INFO] - ===> Start testing [2020-12-31 10:47:17,063][root][INFO] - 68/68: Data time: 0.9490, Iter time: 0.4143 Loss 0.229 (AVG: 0.493) Score 94.479 (AVG: 89.697) mIOU 67.779 mAP 83.031 mAcc 74.883 IOU: 60.027 0.000 78.606 72.774 93.476 90.023 26.149 77.155 98.439 63.523 80.102 84.978 55.878 mAP: 81.595 4.066 92.100 86.134 98.980 98.271 59.854 95.087 99.826 86.187 92.613 94.166 90.529 mAcc: 80.391 0.000 85.180 80.283 94.814 93.801 29.798 89.485 99.272 75.030 89.921 95.498 60.004 [2020-12-31 10:47:17,064][root][INFO] - Finished test. Elapsed time: 37.9630 [2020-12-31 10:47:20,858][root][INFO] - [2020-12-31 10:47:20,858][root][INFO] - ################################################################ [2020-12-31 10:47:20,858][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 10:47:20,859][root][INFO] - ################################################################ [2020-12-31 10:47:20,859][root][INFO] - clutter : nan nan nan [2020-12-31 10:47:20,859][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 10:47:20,859][root][INFO] - board : 0.767 0.825 0.851 [2020-12-31 10:47:20,859][root][INFO] - bookcase : 0.235 0.389 0.515 [2020-12-31 10:47:20,859][root][INFO] - ceiling : 0.796 0.815 0.900 [2020-12-31 10:47:20,859][root][INFO] - chair : 0.759 0.856 0.879 [2020-12-31 10:47:20,859][root][INFO] - column : 0.157 0.222 0.316 [2020-12-31 10:47:20,859][root][INFO] - door : 0.592 0.784 0.884 [2020-12-31 10:47:20,859][root][INFO] - floor : 0.890 0.905 0.916 [2020-12-31 10:47:20,859][root][INFO] - sofa : 0.396 0.545 0.790 [2020-12-31 10:47:20,859][root][INFO] - table : 0.180 0.345 0.612 [2020-12-31 10:47:20,859][root][INFO] - wall : 0.451 0.679 0.760 [2020-12-31 10:47:20,859][root][INFO] - window : 0.425 0.726 0.767 [2020-12-31 10:47:20,859][root][INFO] - ---------------------------------------------------------------- [2020-12-31 10:47:20,859][root][INFO] - average : 0.471 0.591 0.682 [2020-12-31 10:47:20,859][root][INFO] - [2020-12-31 10:47:21,045][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 10:47:21,644][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter5200.pth [2020-12-31 10:47:33,236][root][INFO] - ===> Epoch[1041](5205/5): Loss -0.6800, Sem -0.6800, dir -0.9653, norm 0.2431 LR: 6.815e-02 Score 98.595 Data time: 0.1024, Forward time: 0.9499, Backward time: 1.1993, DDP time: 0.0051, Total iter time: 2.2171 [2020-12-31 10:47:44,385][root][INFO] - ===> Epoch[1042](5210/5): Loss -0.6808, Sem -0.6808, dir -0.9652, norm 0.2412 LR: 6.811e-02 Score 98.557 Data time: 0.1756, Forward time: 0.9498, Backward time: 1.1991, DDP time: 0.0051, Total iter time: 2.1473 [2020-12-31 10:47:56,708][root][INFO] - ===> Epoch[1043](5215/5): Loss -0.6733, Sem -0.6733, dir -0.9652, norm 0.2484 LR: 6.808e-02 Score 98.558 Data time: 0.1636, Forward time: 0.9499, Backward time: 1.1990, DDP time: 0.0051, Total iter time: 2.3602 [2020-12-31 10:48:09,309][root][INFO] - ===> Epoch[1044](5220/5): Loss -0.6802, Sem -0.6802, dir -0.9654, norm 0.2420 LR: 6.805e-02 Score 98.554 Data time: 0.0802, Forward time: 0.9499, Backward time: 1.1992, DDP time: 0.0051, Total iter time: 2.4313 [2020-12-31 10:48:20,889][root][INFO] - ===> Epoch[1045](5225/5): Loss -0.6595, Sem -0.6595, dir -0.9643, norm 0.2595 LR: 6.802e-02 Score 98.482 Data time: 0.1040, Forward time: 0.9499, Backward time: 1.1990, DDP time: 0.0051, Total iter time: 2.2189 [2020-12-31 10:48:32,552][root][INFO] - ===> Epoch[1046](5230/5): Loss -0.6629, Sem -0.6629, dir -0.9639, norm 0.2554 LR: 6.799e-02 Score 98.478 Data time: 0.1157, Forward time: 0.9499, Backward time: 1.1990, DDP time: 0.0051, Total iter time: 2.2571 [2020-12-31 10:48:46,374][root][INFO] - ===> Epoch[1047](5235/5): Loss -0.6752, Sem -0.6752, dir -0.9658, norm 0.2484 LR: 6.796e-02 Score 98.586 Data time: 0.2284, Forward time: 0.9502, Backward time: 1.1990, DDP time: 0.0051, Total iter time: 2.6626 [2020-12-31 10:48:57,480][root][INFO] - ===> Epoch[1048](5240/5): Loss -0.6657, Sem -0.6657, dir -0.9653, norm 0.2537 LR: 6.793e-02 Score 98.497 Data time: 0.0916, Forward time: 0.9501, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.1226 [2020-12-31 10:49:09,121][root][INFO] - ===> Epoch[1049](5245/5): Loss -0.6657, Sem -0.6657, dir -0.9637, norm 0.2524 LR: 6.789e-02 Score 98.484 Data time: 0.1000, Forward time: 0.9501, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.2378 [2020-12-31 10:49:20,572][root][INFO] - ===> Epoch[1050](5250/5): Loss -0.6705, Sem -0.6705, dir -0.9653, norm 0.2533 LR: 6.786e-02 Score 98.624 Data time: 0.0856, Forward time: 0.9500, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.2080 [2020-12-31 10:49:33,227][root][INFO] - ===> Epoch[1051](5255/5): Loss -0.6709, Sem -0.6709, dir -0.9648, norm 0.2505 LR: 6.783e-02 Score 98.558 Data time: 0.1894, Forward time: 0.9501, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.4342 [2020-12-31 10:49:45,235][root][INFO] - ===> Epoch[1052](5260/5): Loss -0.6634, Sem -0.6634, dir -0.9640, norm 0.2559 LR: 6.780e-02 Score 98.540 Data time: 0.1095, Forward time: 0.9502, Backward time: 1.1987, DDP time: 0.0051, Total iter time: 2.3035 [2020-12-31 10:49:57,467][root][INFO] - ===> Epoch[1053](5265/5): Loss -0.6669, Sem -0.6669, dir -0.9646, norm 0.2543 LR: 6.777e-02 Score 98.555 Data time: 0.0764, Forward time: 0.9501, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.3519 [2020-12-31 10:50:09,343][root][INFO] - ===> Epoch[1054](5270/5): Loss -0.6723, Sem -0.6723, dir -0.9655, norm 0.2503 LR: 6.774e-02 Score 98.571 Data time: 0.1229, Forward time: 0.9501, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.2784 [2020-12-31 10:50:21,370][root][INFO] - ===> Epoch[1055](5275/5): Loss -0.6547, Sem -0.6547, dir -0.9631, norm 0.2638 LR: 6.771e-02 Score 98.545 Data time: 0.1419, Forward time: 0.9501, Backward time: 1.1987, DDP time: 0.0051, Total iter time: 2.3196 [2020-12-31 10:50:33,160][root][INFO] - ===> Epoch[1056](5280/5): Loss -0.6797, Sem -0.6797, dir -0.9666, norm 0.2458 LR: 6.768e-02 Score 98.635 Data time: 0.1259, Forward time: 0.9502, Backward time: 1.1986, DDP time: 0.0051, Total iter time: 2.2713 [2020-12-31 10:50:45,408][root][INFO] - ===> Epoch[1057](5285/5): Loss -0.6645, Sem -0.6645, dir -0.9649, norm 0.2572 LR: 6.764e-02 Score 98.554 Data time: 0.0784, Forward time: 0.9501, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.3725 [2020-12-31 10:50:56,712][root][INFO] - ===> Epoch[1058](5290/5): Loss -0.6667, Sem -0.6667, dir -0.9663, norm 0.2577 LR: 6.761e-02 Score 98.592 Data time: 0.1042, Forward time: 0.9501, Backward time: 1.1987, DDP time: 0.0051, Total iter time: 2.1749 [2020-12-31 10:51:09,383][root][INFO] - ===> Epoch[1059](5295/5): Loss -0.6635, Sem -0.6635, dir -0.9644, norm 0.2594 LR: 6.758e-02 Score 98.618 Data time: 0.2688, Forward time: 0.9502, Backward time: 1.1985, DDP time: 0.0051, Total iter time: 2.4473 [2020-12-31 10:51:21,071][root][INFO] - ===> Epoch[1060](5300/5): Loss -0.6675, Sem -0.6675, dir -0.9642, norm 0.2534 LR: 6.755e-02 Score 98.556 Data time: 0.1441, Forward time: 0.9502, Backward time: 1.1984, DDP time: 0.0051, Total iter time: 2.2386 [2020-12-31 10:51:33,762][root][INFO] - ===> Epoch[1061](5305/5): Loss -0.6605, Sem -0.6605, dir -0.9656, norm 0.2627 LR: 6.752e-02 Score 98.600 Data time: 0.1323, Forward time: 0.9503, Backward time: 1.1985, DDP time: 0.0051, Total iter time: 2.4475 [2020-12-31 10:51:45,678][root][INFO] - ===> Epoch[1062](5310/5): Loss -0.6678, Sem -0.6678, dir -0.9648, norm 0.2539 LR: 6.749e-02 Score 98.568 Data time: 0.1573, Forward time: 0.9502, Backward time: 1.1985, DDP time: 0.0051, Total iter time: 2.2938 [2020-12-31 10:51:57,178][root][INFO] - ===> Epoch[1063](5315/5): Loss -0.6826, Sem -0.6826, dir -0.9669, norm 0.2448 LR: 6.746e-02 Score 98.679 Data time: 0.0824, Forward time: 0.9501, Backward time: 1.1985, DDP time: 0.0051, Total iter time: 2.2048 [2020-12-31 10:52:09,669][root][INFO] - ===> Epoch[1064](5320/5): Loss -0.6608, Sem -0.6608, dir -0.9649, norm 0.2589 LR: 6.742e-02 Score 98.511 Data time: 0.0856, Forward time: 0.9501, Backward time: 1.1986, DDP time: 0.0051, Total iter time: 2.4059 [2020-12-31 10:52:22,050][root][INFO] - ===> Epoch[1065](5325/5): Loss -0.6686, Sem -0.6686, dir -0.9645, norm 0.2533 LR: 6.739e-02 Score 98.585 Data time: 0.1037, Forward time: 0.9501, Backward time: 1.1986, DDP time: 0.0051, Total iter time: 2.3701 [2020-12-31 10:52:34,155][root][INFO] - ===> Epoch[1066](5330/5): Loss -0.6647, Sem -0.6647, dir -0.9661, norm 0.2599 LR: 6.736e-02 Score 98.627 Data time: 0.1117, Forward time: 0.9501, Backward time: 1.1986, DDP time: 0.0051, Total iter time: 2.3239 [2020-12-31 10:52:46,561][root][INFO] - ===> Epoch[1067](5335/5): Loss -0.6761, Sem -0.6761, dir -0.9662, norm 0.2488 LR: 6.733e-02 Score 98.622 Data time: 0.1478, Forward time: 0.9502, Backward time: 1.1986, DDP time: 0.0051, Total iter time: 2.3812 [2020-12-31 10:52:58,181][root][INFO] - ===> Epoch[1068](5340/5): Loss -0.6747, Sem -0.6747, dir -0.9656, norm 0.2485 LR: 6.730e-02 Score 98.590 Data time: 0.1815, Forward time: 0.9501, Backward time: 1.1985, DDP time: 0.0051, Total iter time: 2.2374 [2020-12-31 10:53:10,560][root][INFO] - ===> Epoch[1069](5345/5): Loss -0.6895, Sem -0.6895, dir -0.9671, norm 0.2374 LR: 6.727e-02 Score 98.666 Data time: 0.1609, Forward time: 0.9501, Backward time: 1.1986, DDP time: 0.0051, Total iter time: 2.3963 [2020-12-31 10:53:22,678][root][INFO] - ===> Epoch[1070](5350/5): Loss -0.6624, Sem -0.6624, dir -0.9644, norm 0.2572 LR: 6.724e-02 Score 98.502 Data time: 0.0877, Forward time: 0.9500, Backward time: 1.1987, DDP time: 0.0051, Total iter time: 2.3408 [2020-12-31 10:53:35,856][root][INFO] - ===> Epoch[1071](5355/5): Loss -0.6813, Sem -0.6813, dir -0.9676, norm 0.2454 LR: 6.721e-02 Score 98.635 Data time: 0.1050, Forward time: 0.9500, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.5552 [2020-12-31 10:53:46,967][root][INFO] - ===> Epoch[1072](5360/5): Loss -0.6572, Sem -0.6572, dir -0.9646, norm 0.2635 LR: 6.717e-02 Score 98.550 Data time: 0.1093, Forward time: 0.9501, Backward time: 1.1987, DDP time: 0.0051, Total iter time: 2.1363 [2020-12-31 10:53:58,683][root][INFO] - ===> Epoch[1073](5365/5): Loss -0.6691, Sem -0.6691, dir -0.9658, norm 0.2523 LR: 6.714e-02 Score 98.520 Data time: 0.0883, Forward time: 0.9500, Backward time: 1.1986, DDP time: 0.0051, Total iter time: 2.2494 [2020-12-31 10:54:11,045][root][INFO] - ===> Epoch[1074](5370/5): Loss -0.6759, Sem -0.6759, dir -0.9649, norm 0.2457 LR: 6.711e-02 Score 98.554 Data time: 0.0850, Forward time: 0.9500, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.3854 [2020-12-31 10:54:22,789][root][INFO] - ===> Epoch[1075](5375/5): Loss -0.6744, Sem -0.6744, dir -0.9653, norm 0.2490 LR: 6.708e-02 Score 98.602 Data time: 0.1017, Forward time: 0.9500, Backward time: 1.1987, DDP time: 0.0051, Total iter time: 2.2550 [2020-12-31 10:54:35,600][root][INFO] - ===> Epoch[1076](5380/5): Loss -0.6882, Sem -0.6882, dir -0.9671, norm 0.2388 LR: 6.705e-02 Score 98.659 Data time: 0.1453, Forward time: 0.9500, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.4651 [2020-12-31 10:54:47,518][root][INFO] - ===> Epoch[1077](5385/5): Loss -0.6715, Sem -0.6715, dir -0.9652, norm 0.2521 LR: 6.702e-02 Score 98.596 Data time: 0.0893, Forward time: 0.9500, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.2945 [2020-12-31 10:55:00,342][root][INFO] - ===> Epoch[1078](5390/5): Loss -0.6777, Sem -0.6777, dir -0.9659, norm 0.2454 LR: 6.699e-02 Score 98.607 Data time: 0.0893, Forward time: 0.9499, Backward time: 1.1990, DDP time: 0.0051, Total iter time: 2.4790 [2020-12-31 10:55:12,091][root][INFO] - ===> Epoch[1079](5395/5): Loss -0.6731, Sem -0.6731, dir -0.9654, norm 0.2482 LR: 6.695e-02 Score 98.542 Data time: 0.1190, Forward time: 0.9499, Backward time: 1.1989, DDP time: 0.0051, Total iter time: 2.2592 [2020-12-31 10:55:24,857][root][INFO] - ===> Epoch[1080](5400/5): Loss -0.6683, Sem -0.6683, dir -0.9652, norm 0.2541 LR: 6.692e-02 Score 98.565 Data time: 0.0786, Forward time: 0.9499, Backward time: 1.1991, DDP time: 0.0051, Total iter time: 2.4637 [2020-12-31 10:55:24,860][root][INFO] - ===> Start testing [2020-12-31 10:56:02,767][root][INFO] - 68/68: Data time: 0.0024, Iter time: 0.1294 Loss 0.180 (AVG: 0.541) Score 95.309 (AVG: 89.333) mIOU 67.735 mAP 82.558 mAcc 75.135 IOU: 59.152 0.000 76.438 71.687 92.670 89.690 23.445 74.275 98.357 72.710 80.090 84.750 57.294 mAP: 80.726 3.752 93.132 84.670 98.778 98.461 56.108 93.686 99.848 86.684 92.642 93.464 91.301 mAcc: 82.077 0.000 89.081 77.772 93.817 96.428 24.955 87.039 98.983 78.463 90.071 95.234 62.837 [2020-12-31 10:56:02,768][root][INFO] - Finished test. Elapsed time: 37.9054 [2020-12-31 10:56:06,681][root][INFO] - [2020-12-31 10:56:06,681][root][INFO] - ################################################################ [2020-12-31 10:56:06,681][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 10:56:06,681][root][INFO] - ################################################################ [2020-12-31 10:56:06,681][root][INFO] - clutter : nan nan nan [2020-12-31 10:56:06,681][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 10:56:06,682][root][INFO] - board : 0.690 0.757 0.870 [2020-12-31 10:56:06,682][root][INFO] - bookcase : 0.208 0.341 0.477 [2020-12-31 10:56:06,682][root][INFO] - ceiling : 0.626 0.689 0.741 [2020-12-31 10:56:06,682][root][INFO] - chair : 0.798 0.886 0.904 [2020-12-31 10:56:06,682][root][INFO] - column : 0.150 0.233 0.303 [2020-12-31 10:56:06,682][root][INFO] - door : 0.558 0.758 0.884 [2020-12-31 10:56:06,682][root][INFO] - floor : 0.808 0.842 0.842 [2020-12-31 10:56:06,682][root][INFO] - sofa : 0.378 0.531 0.818 [2020-12-31 10:56:06,682][root][INFO] - table : 0.170 0.340 0.597 [2020-12-31 10:56:06,682][root][INFO] - wall : 0.453 0.664 0.776 [2020-12-31 10:56:06,682][root][INFO] - window : 0.488 0.738 0.774 [2020-12-31 10:56:06,682][root][INFO] - ---------------------------------------------------------------- [2020-12-31 10:56:06,682][root][INFO] - average : 0.444 0.565 0.666 [2020-12-31 10:56:06,682][root][INFO] - [2020-12-31 10:56:06,859][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 10:56:07,421][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter5400.pth [2020-12-31 10:56:18,155][root][INFO] - ===> Epoch[1081](5405/5): Loss -0.6704, Sem -0.6704, dir -0.9662, norm 0.2520 LR: 6.689e-02 Score 98.548 Data time: 0.0722, Forward time: 0.9497, Backward time: 1.1991, DDP time: 0.0051, Total iter time: 2.0433 [2020-12-31 10:56:29,852][root][INFO] - ===> Epoch[1082](5410/5): Loss -0.6698, Sem -0.6698, dir -0.9648, norm 0.2521 LR: 6.686e-02 Score 98.575 Data time: 0.0860, Forward time: 0.9497, Backward time: 1.1991, DDP time: 0.0051, Total iter time: 2.2516 [2020-12-31 10:56:40,977][root][INFO] - ===> Epoch[1083](5415/5): Loss -0.6700, Sem -0.6700, dir -0.9658, norm 0.2536 LR: 6.683e-02 Score 98.594 Data time: 0.0947, Forward time: 0.9496, Backward time: 1.1990, DDP time: 0.0051, Total iter time: 2.1244 [2020-12-31 10:56:52,899][root][INFO] - ===> Epoch[1084](5420/5): Loss -0.6782, Sem -0.6782, dir -0.9662, norm 0.2447 LR: 6.680e-02 Score 98.557 Data time: 0.1014, Forward time: 0.9495, Backward time: 1.1990, DDP time: 0.0051, Total iter time: 2.3063 [2020-12-31 10:57:05,103][root][INFO] - ===> Epoch[1085](5425/5): Loss -0.6696, Sem -0.6696, dir -0.9652, norm 0.2537 LR: 6.677e-02 Score 98.611 Data time: 0.0970, Forward time: 0.9495, Backward time: 1.1991, DDP time: 0.0051, Total iter time: 2.3438 [2020-12-31 10:57:16,220][root][INFO] - ===> Epoch[1086](5430/5): Loss -0.6851, Sem -0.6851, dir -0.9666, norm 0.2393 LR: 6.673e-02 Score 98.585 Data time: 0.0999, Forward time: 0.9495, Backward time: 1.1989, DDP time: 0.0051, Total iter time: 2.1307 [2020-12-31 10:57:28,424][root][INFO] - ===> Epoch[1087](5435/5): Loss -0.6831, Sem -0.6831, dir -0.9664, norm 0.2414 LR: 6.670e-02 Score 98.608 Data time: 0.1000, Forward time: 0.9494, Backward time: 1.1990, DDP time: 0.0051, Total iter time: 2.3515 [2020-12-31 10:57:41,081][root][INFO] - ===> Epoch[1088](5440/5): Loss -0.6932, Sem -0.6932, dir -0.9666, norm 0.2309 LR: 6.667e-02 Score 98.609 Data time: 0.1531, Forward time: 0.9494, Backward time: 1.1991, DDP time: 0.0051, Total iter time: 2.4295 [2020-12-31 10:57:52,697][root][INFO] - ===> Epoch[1089](5445/5): Loss -0.6941, Sem -0.6941, dir -0.9671, norm 0.2328 LR: 6.664e-02 Score 98.643 Data time: 0.1095, Forward time: 0.9494, Backward time: 1.1990, DDP time: 0.0051, Total iter time: 2.2258 [2020-12-31 10:58:03,997][root][INFO] - ===> Epoch[1090](5450/5): Loss -0.6880, Sem -0.6880, dir -0.9671, norm 0.2378 LR: 6.661e-02 Score 98.617 Data time: 0.1258, Forward time: 0.9494, Backward time: 1.1989, DDP time: 0.0051, Total iter time: 2.1588 [2020-12-31 10:58:16,350][root][INFO] - ===> Epoch[1091](5455/5): Loss -0.6918, Sem -0.6918, dir -0.9675, norm 0.2351 LR: 6.658e-02 Score 98.640 Data time: 0.1019, Forward time: 0.9494, Backward time: 1.1990, DDP time: 0.0051, Total iter time: 2.3759 [2020-12-31 10:58:27,775][root][INFO] - ===> Epoch[1092](5460/5): Loss -0.6927, Sem -0.6927, dir -0.9677, norm 0.2326 LR: 6.655e-02 Score 98.576 Data time: 0.0988, Forward time: 0.9493, Backward time: 1.1989, DDP time: 0.0051, Total iter time: 2.1885 [2020-12-31 10:58:39,058][root][INFO] - ===> Epoch[1093](5465/5): Loss -0.6803, Sem -0.6803, dir -0.9660, norm 0.2426 LR: 6.652e-02 Score 98.570 Data time: 0.2218, Forward time: 0.9493, Backward time: 1.1987, DDP time: 0.0051, Total iter time: 2.1537 [2020-12-31 10:58:51,492][root][INFO] - ===> Epoch[1094](5470/5): Loss -0.6821, Sem -0.6821, dir -0.9669, norm 0.2437 LR: 6.648e-02 Score 98.625 Data time: 0.1088, Forward time: 0.9493, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.4153 [2020-12-31 10:59:02,917][root][INFO] - ===> Epoch[1095](5475/5): Loss -0.6844, Sem -0.6844, dir -0.9671, norm 0.2408 LR: 6.645e-02 Score 98.606 Data time: 0.0967, Forward time: 0.9493, Backward time: 1.1987, DDP time: 0.0051, Total iter time: 2.2065 [2020-12-31 10:59:14,671][root][INFO] - ===> Epoch[1096](5480/5): Loss -0.6706, Sem -0.6706, dir -0.9657, norm 0.2512 LR: 6.642e-02 Score 98.549 Data time: 0.0845, Forward time: 0.9492, Backward time: 1.1987, DDP time: 0.0051, Total iter time: 2.2546 [2020-12-31 10:59:26,190][root][INFO] - ===> Epoch[1097](5485/5): Loss -0.6715, Sem -0.6715, dir -0.9665, norm 0.2508 LR: 6.639e-02 Score 98.546 Data time: 0.1148, Forward time: 0.9492, Backward time: 1.1987, DDP time: 0.0051, Total iter time: 2.2083 [2020-12-31 10:59:39,184][root][INFO] - ===> Epoch[1098](5490/5): Loss -0.6736, Sem -0.6736, dir -0.9668, norm 0.2518 LR: 6.636e-02 Score 98.641 Data time: 0.1104, Forward time: 0.9492, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.5081 [2020-12-31 10:59:51,279][root][INFO] - ===> Epoch[1099](5495/5): Loss -0.6741, Sem -0.6741, dir -0.9663, norm 0.2514 LR: 6.633e-02 Score 98.646 Data time: 0.1180, Forward time: 0.9493, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.3268 [2020-12-31 11:00:03,451][root][INFO] - ===> Epoch[1100](5500/5): Loss -0.6811, Sem -0.6811, dir -0.9658, norm 0.2426 LR: 6.630e-02 Score 98.596 Data time: 0.0954, Forward time: 0.9492, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.3405 [2020-12-31 11:00:14,438][root][INFO] - ===> Epoch[1101](5505/5): Loss -0.6761, Sem -0.6761, dir -0.9664, norm 0.2462 LR: 6.626e-02 Score 98.536 Data time: 0.1059, Forward time: 0.9491, Backward time: 1.1987, DDP time: 0.0051, Total iter time: 2.1068 [2020-12-31 11:00:25,624][root][INFO] - ===> Epoch[1102](5510/5): Loss -0.6895, Sem -0.6895, dir -0.9669, norm 0.2356 LR: 6.623e-02 Score 98.601 Data time: 0.1028, Forward time: 0.9491, Backward time: 1.1986, DDP time: 0.0051, Total iter time: 2.1415 [2020-12-31 11:00:38,672][root][INFO] - ===> Epoch[1103](5515/5): Loss -0.6786, Sem -0.6786, dir -0.9670, norm 0.2469 LR: 6.620e-02 Score 98.619 Data time: 0.0842, Forward time: 0.9491, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.5260 [2020-12-31 11:00:50,497][root][INFO] - ===> Epoch[1104](5520/5): Loss -0.6710, Sem -0.6710, dir -0.9655, norm 0.2519 LR: 6.617e-02 Score 98.584 Data time: 0.1116, Forward time: 0.9491, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.2723 [2020-12-31 11:01:02,522][root][INFO] - ===> Epoch[1105](5525/5): Loss -0.6769, Sem -0.6769, dir -0.9668, norm 0.2494 LR: 6.614e-02 Score 98.647 Data time: 0.0953, Forward time: 0.9490, Backward time: 1.1989, DDP time: 0.0051, Total iter time: 2.3155 [2020-12-31 11:01:13,770][root][INFO] - ===> Epoch[1106](5530/5): Loss -0.6714, Sem -0.6714, dir -0.9659, norm 0.2523 LR: 6.611e-02 Score 98.584 Data time: 0.1109, Forward time: 0.9490, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.1645 [2020-12-31 11:01:25,384][root][INFO] - ===> Epoch[1107](5535/5): Loss -0.6826, Sem -0.6826, dir -0.9670, norm 0.2428 LR: 6.608e-02 Score 98.606 Data time: 0.1126, Forward time: 0.9490, Backward time: 1.1987, DDP time: 0.0051, Total iter time: 2.2251 [2020-12-31 11:01:37,828][root][INFO] - ===> Epoch[1108](5540/5): Loss -0.6786, Sem -0.6786, dir -0.9661, norm 0.2454 LR: 6.604e-02 Score 98.598 Data time: 0.0891, Forward time: 0.9489, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.3856 [2020-12-31 11:01:50,046][root][INFO] - ===> Epoch[1109](5545/5): Loss -0.6880, Sem -0.6880, dir -0.9677, norm 0.2403 LR: 6.601e-02 Score 98.692 Data time: 0.1994, Forward time: 0.9490, Backward time: 1.1987, DDP time: 0.0051, Total iter time: 2.3530 [2020-12-31 11:02:01,790][root][INFO] - ===> Epoch[1110](5550/5): Loss -0.6840, Sem -0.6840, dir -0.9659, norm 0.2384 LR: 6.598e-02 Score 98.567 Data time: 0.0872, Forward time: 0.9489, Backward time: 1.1987, DDP time: 0.0051, Total iter time: 2.2682 [2020-12-31 11:02:13,020][root][INFO] - ===> Epoch[1111](5555/5): Loss -0.6929, Sem -0.6929, dir -0.9677, norm 0.2337 LR: 6.595e-02 Score 98.628 Data time: 0.2073, Forward time: 0.9489, Backward time: 1.1985, DDP time: 0.0051, Total iter time: 2.1501 [2020-12-31 11:02:25,388][root][INFO] - ===> Epoch[1112](5560/5): Loss -0.6812, Sem -0.6812, dir -0.9660, norm 0.2436 LR: 6.592e-02 Score 98.630 Data time: 0.1263, Forward time: 0.9489, Backward time: 1.1986, DDP time: 0.0051, Total iter time: 2.3973 [2020-12-31 11:02:37,438][root][INFO] - ===> Epoch[1113](5565/5): Loss -0.6900, Sem -0.6900, dir -0.9678, norm 0.2383 LR: 6.589e-02 Score 98.684 Data time: 0.0779, Forward time: 0.9487, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.3179 [2020-12-31 11:02:49,387][root][INFO] - ===> Epoch[1114](5570/5): Loss -0.6740, Sem -0.6740, dir -0.9656, norm 0.2503 LR: 6.586e-02 Score 98.625 Data time: 0.0967, Forward time: 0.9488, Backward time: 1.1987, DDP time: 0.0051, Total iter time: 2.2968 [2020-12-31 11:03:01,620][root][INFO] - ===> Epoch[1115](5575/5): Loss -0.6831, Sem -0.6831, dir -0.9673, norm 0.2420 LR: 6.582e-02 Score 98.607 Data time: 0.1797, Forward time: 0.9488, Backward time: 1.1986, DDP time: 0.0051, Total iter time: 2.3528 [2020-12-31 11:03:14,200][root][INFO] - ===> Epoch[1116](5580/5): Loss -0.6815, Sem -0.6815, dir -0.9664, norm 0.2447 LR: 6.579e-02 Score 98.668 Data time: 0.0788, Forward time: 0.9488, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.4346 [2020-12-31 11:03:25,812][root][INFO] - ===> Epoch[1117](5585/5): Loss -0.6868, Sem -0.6868, dir -0.9673, norm 0.2391 LR: 6.576e-02 Score 98.621 Data time: 0.1032, Forward time: 0.9488, Backward time: 1.1987, DDP time: 0.0051, Total iter time: 2.2408 [2020-12-31 11:03:38,012][root][INFO] - ===> Epoch[1118](5590/5): Loss -0.6932, Sem -0.6932, dir -0.9680, norm 0.2357 LR: 6.573e-02 Score 98.683 Data time: 0.1207, Forward time: 0.9488, Backward time: 1.1988, DDP time: 0.0051, Total iter time: 2.3663 [2020-12-31 11:03:49,318][root][INFO] - ===> Epoch[1119](5595/5): Loss -0.6889, Sem -0.6889, dir -0.9673, norm 0.2382 LR: 6.570e-02 Score 98.671 Data time: 0.0931, Forward time: 0.9488, Backward time: 1.1987, DDP time: 0.0051, Total iter time: 2.1642 [2020-12-31 11:04:01,013][root][INFO] - ===> Epoch[1120](5600/5): Loss -0.6846, Sem -0.6846, dir -0.9655, norm 0.2387 LR: 6.567e-02 Score 98.582 Data time: 0.1025, Forward time: 0.9487, Backward time: 1.1986, DDP time: 0.0051, Total iter time: 2.2479 [2020-12-31 11:04:01,015][root][INFO] - ===> Start testing [2020-12-31 11:04:39,506][root][INFO] - 68/68: Data time: 1.0045, Iter time: 0.2456 Loss 0.195 (AVG: 0.553) Score 95.849 (AVG: 89.268) mIOU 67.284 mAP 82.054 mAcc 74.379 IOU: 59.015 0.000 78.178 71.288 93.406 88.523 23.454 75.665 98.443 70.725 79.715 83.821 52.453 mAP: 81.044 4.959 93.447 84.709 98.983 98.574 51.685 94.215 99.834 83.481 93.178 93.719 88.876 mAcc: 78.027 0.000 86.052 78.623 95.132 96.710 25.492 86.239 99.327 81.373 88.781 96.327 54.839 [2020-12-31 11:04:39,508][root][INFO] - Finished test. Elapsed time: 38.4901 [2020-12-31 11:04:42,794][root][INFO] - [2020-12-31 11:04:42,794][root][INFO] - ################################################################ [2020-12-31 11:04:42,794][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 11:04:42,794][root][INFO] - ################################################################ [2020-12-31 11:04:42,794][root][INFO] - clutter : nan nan nan [2020-12-31 11:04:42,794][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 11:04:42,794][root][INFO] - board : 0.773 0.830 0.878 [2020-12-31 11:04:42,794][root][INFO] - bookcase : 0.195 0.321 0.477 [2020-12-31 11:04:42,794][root][INFO] - ceiling : 0.679 0.763 0.796 [2020-12-31 11:04:42,794][root][INFO] - chair : 0.779 0.887 0.903 [2020-12-31 11:04:42,794][root][INFO] - column : 0.134 0.198 0.288 [2020-12-31 11:04:42,794][root][INFO] - door : 0.549 0.797 0.855 [2020-12-31 11:04:42,794][root][INFO] - floor : 0.890 0.890 0.966 [2020-12-31 11:04:42,794][root][INFO] - sofa : 0.472 0.610 0.818 [2020-12-31 11:04:42,794][root][INFO] - table : 0.200 0.382 0.645 [2020-12-31 11:04:42,795][root][INFO] - wall : 0.467 0.711 0.783 [2020-12-31 11:04:42,795][root][INFO] - window : 0.394 0.651 0.745 [2020-12-31 11:04:42,795][root][INFO] - ---------------------------------------------------------------- [2020-12-31 11:04:42,795][root][INFO] - average : 0.461 0.587 0.680 [2020-12-31 11:04:42,795][root][INFO] - [2020-12-31 11:04:42,981][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 11:04:43,594][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter5600.pth [2020-12-31 11:04:55,369][root][INFO] - ===> Epoch[1121](5605/5): Loss -0.6876, Sem -0.6876, dir -0.9674, norm 0.2388 LR: 6.564e-02 Score 98.644 Data time: 0.1670, Forward time: 0.9488, Backward time: 1.1985, DDP time: 0.0051, Total iter time: 2.2501 [2020-12-31 11:05:08,209][root][INFO] - ===> Epoch[1122](5610/5): Loss -0.6776, Sem -0.6776, dir -0.9654, norm 0.2408 LR: 6.560e-02 Score 98.445 Data time: 0.1559, Forward time: 0.9488, Backward time: 1.1986, DDP time: 0.0051, Total iter time: 2.4831 [2020-12-31 11:05:20,020][root][INFO] - ===> Epoch[1123](5615/5): Loss -0.6718, Sem -0.6718, dir -0.9660, norm 0.2512 LR: 6.557e-02 Score 98.571 Data time: 0.1343, Forward time: 0.9489, Backward time: 1.1985, DDP time: 0.0051, Total iter time: 2.2653 [2020-12-31 11:05:30,949][root][INFO] - ===> Epoch[1124](5620/5): Loss -0.6667, Sem -0.6667, dir -0.9640, norm 0.2483 LR: 6.554e-02 Score 98.393 Data time: 0.1200, Forward time: 0.9488, Backward time: 1.1984, DDP time: 0.0051, Total iter time: 2.1097 [2020-12-31 11:05:43,285][root][INFO] - ===> Epoch[1125](5625/5): Loss -0.6844, Sem -0.6844, dir -0.9671, norm 0.2393 LR: 6.551e-02 Score 98.577 Data time: 0.2294, Forward time: 0.9488, Backward time: 1.1983, DDP time: 0.0051, Total iter time: 2.3735 [2020-12-31 11:05:55,083][root][INFO] - ===> Epoch[1126](5630/5): Loss -0.6842, Sem -0.6842, dir -0.9666, norm 0.2406 LR: 6.548e-02 Score 98.611 Data time: 0.1262, Forward time: 0.9488, Backward time: 1.1982, DDP time: 0.0051, Total iter time: 2.2836 [2020-12-31 11:06:06,441][root][INFO] - ===> Epoch[1127](5635/5): Loss -0.6756, Sem -0.6756, dir -0.9654, norm 0.2437 LR: 6.545e-02 Score 98.474 Data time: 0.0931, Forward time: 0.9489, Backward time: 1.1981, DDP time: 0.0051, Total iter time: 2.1936 [2020-12-31 11:06:18,685][root][INFO] - ===> Epoch[1128](5640/5): Loss -0.6784, Sem -0.6784, dir -0.9659, norm 0.2394 LR: 6.542e-02 Score 98.400 Data time: 0.1096, Forward time: 0.9489, Backward time: 1.1981, DDP time: 0.0051, Total iter time: 2.3416 [2020-12-31 11:06:29,552][root][INFO] - ===> Epoch[1129](5645/5): Loss -0.6733, Sem -0.6733, dir -0.9653, norm 0.2455 LR: 6.538e-02 Score 98.456 Data time: 0.1801, Forward time: 0.9489, Backward time: 1.1979, DDP time: 0.0051, Total iter time: 2.0730 [2020-12-31 11:06:41,869][root][INFO] - ===> Epoch[1130](5650/5): Loss -0.6837, Sem -0.6837, dir -0.9663, norm 0.2398 LR: 6.535e-02 Score 98.594 Data time: 0.2195, Forward time: 0.9489, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.3680 [2020-12-31 11:06:53,615][root][INFO] - ===> Epoch[1131](5655/5): Loss -0.6810, Sem -0.6810, dir -0.9665, norm 0.2428 LR: 6.532e-02 Score 98.572 Data time: 0.1307, Forward time: 0.9489, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.2602 [2020-12-31 11:07:05,573][root][INFO] - ===> Epoch[1132](5660/5): Loss -0.6848, Sem -0.6848, dir -0.9666, norm 0.2394 LR: 6.529e-02 Score 98.601 Data time: 0.0904, Forward time: 0.9489, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.3032 [2020-12-31 11:07:16,967][root][INFO] - ===> Epoch[1133](5665/5): Loss -0.6851, Sem -0.6851, dir -0.9666, norm 0.2388 LR: 6.526e-02 Score 98.572 Data time: 0.1125, Forward time: 0.9489, Backward time: 1.1976, DDP time: 0.0051, Total iter time: 2.1864 [2020-12-31 11:07:29,325][root][INFO] - ===> Epoch[1134](5670/5): Loss -0.6867, Sem -0.6867, dir -0.9669, norm 0.2384 LR: 6.523e-02 Score 98.607 Data time: 0.0911, Forward time: 0.9488, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.3829 [2020-12-31 11:07:41,511][root][INFO] - ===> Epoch[1135](5675/5): Loss -0.6849, Sem -0.6849, dir -0.9661, norm 0.2389 LR: 6.520e-02 Score 98.584 Data time: 0.1011, Forward time: 0.9488, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.3456 [2020-12-31 11:07:53,974][root][INFO] - ===> Epoch[1136](5680/5): Loss -0.6948, Sem -0.6948, dir -0.9673, norm 0.2300 LR: 6.516e-02 Score 98.582 Data time: 0.2091, Forward time: 0.9490, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.3963 [2020-12-31 11:08:06,138][root][INFO] - ===> Epoch[1137](5685/5): Loss -0.6857, Sem -0.6857, dir -0.9661, norm 0.2373 LR: 6.513e-02 Score 98.562 Data time: 0.2110, Forward time: 0.9490, Backward time: 1.1976, DDP time: 0.0051, Total iter time: 2.3481 [2020-12-31 11:08:17,814][root][INFO] - ===> Epoch[1138](5690/5): Loss -0.6840, Sem -0.6840, dir -0.9669, norm 0.2413 LR: 6.510e-02 Score 98.635 Data time: 0.1629, Forward time: 0.9491, Backward time: 1.1974, DDP time: 0.0051, Total iter time: 2.2397 [2020-12-31 11:08:29,738][root][INFO] - ===> Epoch[1139](5695/5): Loss -0.6838, Sem -0.6838, dir -0.9664, norm 0.2394 LR: 6.507e-02 Score 98.564 Data time: 0.1264, Forward time: 0.9491, Backward time: 1.1973, DDP time: 0.0051, Total iter time: 2.2977 [2020-12-31 11:08:41,489][root][INFO] - ===> Epoch[1140](5700/5): Loss -0.6857, Sem -0.6857, dir -0.9670, norm 0.2402 LR: 6.504e-02 Score 98.628 Data time: 0.0879, Forward time: 0.9491, Backward time: 1.1974, DDP time: 0.0051, Total iter time: 2.2611 [2020-12-31 11:08:52,854][root][INFO] - ===> Epoch[1141](5705/5): Loss -0.6826, Sem -0.6826, dir -0.9673, norm 0.2426 LR: 6.501e-02 Score 98.616 Data time: 0.0924, Forward time: 0.9490, Backward time: 1.1973, DDP time: 0.0051, Total iter time: 2.1868 [2020-12-31 11:09:04,078][root][INFO] - ===> Epoch[1142](5710/5): Loss -0.6841, Sem -0.6841, dir -0.9662, norm 0.2407 LR: 6.498e-02 Score 98.628 Data time: 0.0958, Forward time: 0.9490, Backward time: 1.1972, DDP time: 0.0051, Total iter time: 2.1470 [2020-12-31 11:09:16,223][root][INFO] - ===> Epoch[1143](5715/5): Loss -0.6827, Sem -0.6827, dir -0.9666, norm 0.2416 LR: 6.494e-02 Score 98.579 Data time: 0.1072, Forward time: 0.9491, Backward time: 1.1972, DDP time: 0.0051, Total iter time: 2.3484 [2020-12-31 11:09:28,328][root][INFO] - ===> Epoch[1144](5720/5): Loss -0.6937, Sem -0.6937, dir -0.9684, norm 0.2346 LR: 6.491e-02 Score 98.665 Data time: 0.1628, Forward time: 0.9491, Backward time: 1.1971, DDP time: 0.0051, Total iter time: 2.3305 [2020-12-31 11:09:40,212][root][INFO] - ===> Epoch[1145](5725/5): Loss -0.6942, Sem -0.6942, dir -0.9681, norm 0.2339 LR: 6.488e-02 Score 98.644 Data time: 0.0693, Forward time: 0.9490, Backward time: 1.1972, DDP time: 0.0051, Total iter time: 2.2913 [2020-12-31 11:09:51,928][root][INFO] - ===> Epoch[1146](5730/5): Loss -0.6903, Sem -0.6903, dir -0.9680, norm 0.2375 LR: 6.485e-02 Score 98.661 Data time: 0.0858, Forward time: 0.9489, Backward time: 1.1973, DDP time: 0.0051, Total iter time: 2.2559 [2020-12-31 11:10:03,278][root][INFO] - ===> Epoch[1147](5735/5): Loss -0.6770, Sem -0.6770, dir -0.9659, norm 0.2463 LR: 6.482e-02 Score 98.581 Data time: 0.0939, Forward time: 0.9490, Backward time: 1.1971, DDP time: 0.0051, Total iter time: 2.1789 [2020-12-31 11:10:15,655][root][INFO] - ===> Epoch[1148](5740/5): Loss -0.6741, Sem -0.6741, dir -0.9666, norm 0.2502 LR: 6.479e-02 Score 98.597 Data time: 0.0763, Forward time: 0.9489, Backward time: 1.1973, DDP time: 0.0051, Total iter time: 2.3996 [2020-12-31 11:10:27,590][root][INFO] - ===> Epoch[1149](5745/5): Loss -0.6824, Sem -0.6824, dir -0.9672, norm 0.2443 LR: 6.475e-02 Score 98.645 Data time: 0.1227, Forward time: 0.9489, Backward time: 1.1972, DDP time: 0.0051, Total iter time: 2.2909 [2020-12-31 11:10:39,558][root][INFO] - ===> Epoch[1150](5750/5): Loss -0.6758, Sem -0.6758, dir -0.9663, norm 0.2493 LR: 6.472e-02 Score 98.634 Data time: 0.1130, Forward time: 0.9489, Backward time: 1.1972, DDP time: 0.0051, Total iter time: 2.2997 [2020-12-31 11:10:51,529][root][INFO] - ===> Epoch[1151](5755/5): Loss -0.6814, Sem -0.6814, dir -0.9671, norm 0.2434 LR: 6.469e-02 Score 98.624 Data time: 0.0827, Forward time: 0.9489, Backward time: 1.1973, DDP time: 0.0051, Total iter time: 2.3081 [2020-12-31 11:11:03,155][root][INFO] - ===> Epoch[1152](5760/5): Loss -0.6882, Sem -0.6882, dir -0.9670, norm 0.2364 LR: 6.466e-02 Score 98.583 Data time: 0.0664, Forward time: 0.9487, Backward time: 1.1974, DDP time: 0.0051, Total iter time: 2.2501 [2020-12-31 11:11:15,544][root][INFO] - ===> Epoch[1153](5765/5): Loss -0.6866, Sem -0.6866, dir -0.9664, norm 0.2375 LR: 6.463e-02 Score 98.596 Data time: 0.1179, Forward time: 0.9487, Backward time: 1.1975, DDP time: 0.0051, Total iter time: 2.3963 [2020-12-31 11:11:27,072][root][INFO] - ===> Epoch[1154](5770/5): Loss -0.6943, Sem -0.6943, dir -0.9684, norm 0.2358 LR: 6.460e-02 Score 98.718 Data time: 0.0880, Forward time: 0.9486, Backward time: 1.1974, DDP time: 0.0051, Total iter time: 2.2208 [2020-12-31 11:11:39,766][root][INFO] - ===> Epoch[1155](5775/5): Loss -0.6824, Sem -0.6824, dir -0.9665, norm 0.2420 LR: 6.457e-02 Score 98.599 Data time: 0.0772, Forward time: 0.9485, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.4603 [2020-12-31 11:11:52,363][root][INFO] - ===> Epoch[1156](5780/5): Loss -0.6863, Sem -0.6863, dir -0.9670, norm 0.2402 LR: 6.453e-02 Score 98.659 Data time: 0.0980, Forward time: 0.9485, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.4143 [2020-12-31 11:12:05,062][root][INFO] - ===> Epoch[1157](5785/5): Loss -0.6881, Sem -0.6881, dir -0.9680, norm 0.2396 LR: 6.450e-02 Score 98.657 Data time: 0.0756, Forward time: 0.9484, Backward time: 1.1980, DDP time: 0.0051, Total iter time: 2.4641 [2020-12-31 11:12:16,342][root][INFO] - ===> Epoch[1158](5790/5): Loss -0.6908, Sem -0.6908, dir -0.9686, norm 0.2363 LR: 6.447e-02 Score 98.621 Data time: 0.0951, Forward time: 0.9483, Backward time: 1.1980, DDP time: 0.0051, Total iter time: 2.1654 [2020-12-31 11:12:29,064][root][INFO] - ===> Epoch[1159](5795/5): Loss -0.6919, Sem -0.6919, dir -0.9668, norm 0.2325 LR: 6.444e-02 Score 98.585 Data time: 0.0962, Forward time: 0.9484, Backward time: 1.1981, DDP time: 0.0051, Total iter time: 2.4548 [2020-12-31 11:12:41,212][root][INFO] - ===> Epoch[1160](5800/5): Loss -0.6972, Sem -0.6972, dir -0.9685, norm 0.2309 LR: 6.441e-02 Score 98.669 Data time: 0.0916, Forward time: 0.9483, Backward time: 1.1981, DDP time: 0.0051, Total iter time: 2.3458 [2020-12-31 11:12:41,214][root][INFO] - ===> Start testing [2020-12-31 11:13:19,863][root][INFO] - 68/68: Data time: 0.0025, Iter time: 0.1242 Loss 0.723 (AVG: 0.530) Score 87.791 (AVG: 89.254) mIOU 67.711 mAP 82.203 mAcc 74.622 IOU: 58.475 0.000 79.603 71.929 92.650 90.530 28.791 71.092 98.162 68.867 78.356 84.908 56.876 mAP: 80.079 3.164 93.766 84.532 98.831 98.183 60.340 91.428 99.833 81.795 91.899 94.557 90.234 mAcc: 81.492 0.000 87.153 79.506 93.889 95.492 32.546 82.974 98.755 76.361 86.276 95.587 60.053 [2020-12-31 11:13:19,864][root][INFO] - Finished test. Elapsed time: 38.6477 [2020-12-31 11:13:22,977][root][INFO] - [2020-12-31 11:13:22,977][root][INFO] - ################################################################ [2020-12-31 11:13:22,977][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 11:13:22,977][root][INFO] - ################################################################ [2020-12-31 11:13:22,977][root][INFO] - clutter : nan nan nan [2020-12-31 11:13:22,977][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 11:13:22,977][root][INFO] - board : 0.748 0.846 0.900 [2020-12-31 11:13:22,977][root][INFO] - bookcase : 0.216 0.356 0.483 [2020-12-31 11:13:22,977][root][INFO] - ceiling : 0.603 0.632 0.719 [2020-12-31 11:13:22,977][root][INFO] - chair : 0.759 0.879 0.893 [2020-12-31 11:13:22,977][root][INFO] - column : 0.113 0.177 0.347 [2020-12-31 11:13:22,977][root][INFO] - door : 0.571 0.779 0.865 [2020-12-31 11:13:22,978][root][INFO] - floor : 0.996 1.000 1.000 [2020-12-31 11:13:22,978][root][INFO] - sofa : 0.413 0.610 0.727 [2020-12-31 11:13:22,978][root][INFO] - table : 0.173 0.353 0.666 [2020-12-31 11:13:22,978][root][INFO] - wall : 0.473 0.689 0.791 [2020-12-31 11:13:22,978][root][INFO] - window : 0.475 0.730 0.766 [2020-12-31 11:13:22,978][root][INFO] - ---------------------------------------------------------------- [2020-12-31 11:13:22,978][root][INFO] - average : 0.462 0.588 0.680 [2020-12-31 11:13:22,978][root][INFO] - [2020-12-31 11:13:23,146][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 11:13:23,720][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter5800.pth [2020-12-31 11:13:34,447][root][INFO] - ===> Epoch[1161](5805/5): Loss -0.6897, Sem -0.6897, dir -0.9676, norm 0.2372 LR: 6.438e-02 Score 98.629 Data time: 0.0719, Forward time: 0.9482, Backward time: 1.1980, DDP time: 0.0051, Total iter time: 2.0477 [2020-12-31 11:13:47,171][root][INFO] - ===> Epoch[1162](5810/5): Loss -0.6898, Sem -0.6898, dir -0.9682, norm 0.2395 LR: 6.435e-02 Score 98.698 Data time: 0.1273, Forward time: 0.9483, Backward time: 1.1981, DDP time: 0.0051, Total iter time: 2.4462 [2020-12-31 11:13:58,843][root][INFO] - ===> Epoch[1163](5815/5): Loss -0.6863, Sem -0.6863, dir -0.9672, norm 0.2399 LR: 6.431e-02 Score 98.613 Data time: 0.0719, Forward time: 0.9482, Backward time: 1.1981, DDP time: 0.0051, Total iter time: 2.2516 [2020-12-31 11:14:10,461][root][INFO] - ===> Epoch[1164](5820/5): Loss -0.6890, Sem -0.6890, dir -0.9673, norm 0.2381 LR: 6.428e-02 Score 98.660 Data time: 0.0713, Forward time: 0.9480, Backward time: 1.1982, DDP time: 0.0051, Total iter time: 2.2397 [2020-12-31 11:14:21,536][root][INFO] - ===> Epoch[1165](5825/5): Loss -0.6909, Sem -0.6909, dir -0.9677, norm 0.2350 LR: 6.425e-02 Score 98.593 Data time: 0.1682, Forward time: 0.9480, Backward time: 1.1981, DDP time: 0.0051, Total iter time: 2.1206 [2020-12-31 11:14:34,053][root][INFO] - ===> Epoch[1166](5830/5): Loss -0.6860, Sem -0.6860, dir -0.9669, norm 0.2399 LR: 6.422e-02 Score 98.628 Data time: 0.1176, Forward time: 0.9480, Backward time: 1.1981, DDP time: 0.0051, Total iter time: 2.4124 [2020-12-31 11:14:46,130][root][INFO] - ===> Epoch[1167](5835/5): Loss -0.6860, Sem -0.6860, dir -0.9683, norm 0.2396 LR: 6.419e-02 Score 98.630 Data time: 0.1306, Forward time: 0.9481, Backward time: 1.1981, DDP time: 0.0051, Total iter time: 2.3266 [2020-12-31 11:14:57,115][root][INFO] - ===> Epoch[1168](5840/5): Loss -0.6842, Sem -0.6842, dir -0.9665, norm 0.2388 LR: 6.416e-02 Score 98.547 Data time: 0.0677, Forward time: 0.9479, Backward time: 1.1981, DDP time: 0.0051, Total iter time: 2.1199 [2020-12-31 11:15:08,422][root][INFO] - ===> Epoch[1169](5845/5): Loss -0.6837, Sem -0.6837, dir -0.9672, norm 0.2406 LR: 6.412e-02 Score 98.602 Data time: 0.1813, Forward time: 0.9479, Backward time: 1.1979, DDP time: 0.0051, Total iter time: 2.1600 [2020-12-31 11:15:20,650][root][INFO] - ===> Epoch[1170](5850/5): Loss -0.6839, Sem -0.6839, dir -0.9675, norm 0.2414 LR: 6.409e-02 Score 98.600 Data time: 0.1282, Forward time: 0.9479, Backward time: 1.1980, DDP time: 0.0051, Total iter time: 2.3520 [2020-12-31 11:15:32,277][root][INFO] - ===> Epoch[1171](5855/5): Loss -0.6871, Sem -0.6871, dir -0.9678, norm 0.2394 LR: 6.406e-02 Score 98.613 Data time: 0.1812, Forward time: 0.9479, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.2302 [2020-12-31 11:15:44,603][root][INFO] - ===> Epoch[1172](5860/5): Loss -0.6817, Sem -0.6817, dir -0.9669, norm 0.2433 LR: 6.403e-02 Score 98.622 Data time: 0.1134, Forward time: 0.9479, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.3695 [2020-12-31 11:15:57,510][root][INFO] - ===> Epoch[1173](5865/5): Loss -0.6867, Sem -0.6867, dir -0.9675, norm 0.2401 LR: 6.400e-02 Score 98.635 Data time: 0.1253, Forward time: 0.9480, Backward time: 1.1979, DDP time: 0.0051, Total iter time: 2.4844 [2020-12-31 11:16:08,852][root][INFO] - ===> Epoch[1174](5870/5): Loss -0.6922, Sem -0.6922, dir -0.9671, norm 0.2328 LR: 6.397e-02 Score 98.584 Data time: 0.1183, Forward time: 0.9480, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.1842 [2020-12-31 11:16:20,998][root][INFO] - ===> Epoch[1175](5875/5): Loss -0.6740, Sem -0.6740, dir -0.9672, norm 0.2531 LR: 6.394e-02 Score 98.665 Data time: 0.1163, Forward time: 0.9481, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.3364 [2020-12-31 11:16:32,315][root][INFO] - ===> Epoch[1176](5880/5): Loss -0.6816, Sem -0.6816, dir -0.9671, norm 0.2445 LR: 6.390e-02 Score 98.626 Data time: 0.0979, Forward time: 0.9481, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.1712 [2020-12-31 11:16:44,357][root][INFO] - ===> Epoch[1177](5885/5): Loss -0.6844, Sem -0.6844, dir -0.9673, norm 0.2417 LR: 6.387e-02 Score 98.626 Data time: 0.0979, Forward time: 0.9480, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.3148 [2020-12-31 11:16:56,198][root][INFO] - ===> Epoch[1178](5890/5): Loss -0.6822, Sem -0.6822, dir -0.9670, norm 0.2417 LR: 6.384e-02 Score 98.575 Data time: 0.2523, Forward time: 0.9480, Backward time: 1.1975, DDP time: 0.0051, Total iter time: 2.2774 [2020-12-31 11:17:08,375][root][INFO] - ===> Epoch[1179](5895/5): Loss -0.6925, Sem -0.6925, dir -0.9681, norm 0.2347 LR: 6.381e-02 Score 98.639 Data time: 0.1281, Forward time: 0.9481, Backward time: 1.1975, DDP time: 0.0051, Total iter time: 2.3385 [2020-12-31 11:17:20,925][root][INFO] - ===> Epoch[1180](5900/5): Loss -0.6912, Sem -0.6912, dir -0.9682, norm 0.2380 LR: 6.378e-02 Score 98.693 Data time: 0.1130, Forward time: 0.9481, Backward time: 1.1976, DDP time: 0.0051, Total iter time: 2.4245 [2020-12-31 11:17:33,117][root][INFO] - ===> Epoch[1181](5905/5): Loss -0.6957, Sem -0.6957, dir -0.9681, norm 0.2325 LR: 6.375e-02 Score 98.663 Data time: 0.1169, Forward time: 0.9482, Backward time: 1.1976, DDP time: 0.0051, Total iter time: 2.3491 [2020-12-31 11:17:45,718][root][INFO] - ===> Epoch[1182](5910/5): Loss -0.6824, Sem -0.6824, dir -0.9667, norm 0.2436 LR: 6.372e-02 Score 98.642 Data time: 0.0767, Forward time: 0.9481, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.4386 [2020-12-31 11:17:57,020][root][INFO] - ===> Epoch[1183](5915/5): Loss -0.6882, Sem -0.6882, dir -0.9674, norm 0.2377 LR: 6.368e-02 Score 98.619 Data time: 0.0914, Forward time: 0.9481, Backward time: 1.1976, DDP time: 0.0051, Total iter time: 2.1838 [2020-12-31 11:18:10,098][root][INFO] - ===> Epoch[1184](5920/5): Loss -0.6940, Sem -0.6940, dir -0.9678, norm 0.2326 LR: 6.365e-02 Score 98.637 Data time: 0.0932, Forward time: 0.9481, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.5195 [2020-12-31 11:18:21,753][root][INFO] - ===> Epoch[1185](5925/5): Loss -0.6867, Sem -0.6867, dir -0.9678, norm 0.2409 LR: 6.362e-02 Score 98.660 Data time: 0.0895, Forward time: 0.9481, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.2328 [2020-12-31 11:18:33,495][root][INFO] - ===> Epoch[1186](5930/5): Loss -0.6864, Sem -0.6864, dir -0.9680, norm 0.2408 LR: 6.359e-02 Score 98.652 Data time: 0.0859, Forward time: 0.9480, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.2649 [2020-12-31 11:18:45,783][root][INFO] - ===> Epoch[1187](5935/5): Loss -0.6670, Sem -0.6670, dir -0.9658, norm 0.2566 LR: 6.356e-02 Score 98.598 Data time: 0.1099, Forward time: 0.9480, Backward time: 1.1979, DDP time: 0.0051, Total iter time: 2.3687 [2020-12-31 11:18:57,786][root][INFO] - ===> Epoch[1188](5940/5): Loss -0.6855, Sem -0.6855, dir -0.9672, norm 0.2387 LR: 6.353e-02 Score 98.559 Data time: 0.1049, Forward time: 0.9479, Backward time: 1.1979, DDP time: 0.0051, Total iter time: 2.2996 [2020-12-31 11:19:08,827][root][INFO] - ===> Epoch[1189](5945/5): Loss -0.6779, Sem -0.6779, dir -0.9672, norm 0.2460 LR: 6.349e-02 Score 98.563 Data time: 0.1480, Forward time: 0.9479, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.1107 [2020-12-31 11:19:20,573][root][INFO] - ===> Epoch[1190](5950/5): Loss -0.6851, Sem -0.6851, dir -0.9676, norm 0.2387 LR: 6.346e-02 Score 98.544 Data time: 0.1504, Forward time: 0.9479, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.2514 [2020-12-31 11:19:33,389][root][INFO] - ===> Epoch[1191](5955/5): Loss -0.6847, Sem -0.6847, dir -0.9679, norm 0.2410 LR: 6.343e-02 Score 98.585 Data time: 0.0822, Forward time: 0.9478, Backward time: 1.1979, DDP time: 0.0051, Total iter time: 2.4799 [2020-12-31 11:19:45,173][root][INFO] - ===> Epoch[1192](5960/5): Loss -0.6814, Sem -0.6814, dir -0.9669, norm 0.2414 LR: 6.340e-02 Score 98.548 Data time: 0.1140, Forward time: 0.9479, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.2616 [2020-12-31 11:19:57,739][root][INFO] - ===> Epoch[1193](5965/5): Loss -0.6907, Sem -0.6907, dir -0.9677, norm 0.2350 LR: 6.337e-02 Score 98.605 Data time: 0.1173, Forward time: 0.9479, Backward time: 1.1979, DDP time: 0.0051, Total iter time: 2.4246 [2020-12-31 11:20:08,805][root][INFO] - ===> Epoch[1194](5970/5): Loss -0.6854, Sem -0.6854, dir -0.9673, norm 0.2380 LR: 6.334e-02 Score 98.535 Data time: 0.0947, Forward time: 0.9478, Backward time: 1.1979, DDP time: 0.0051, Total iter time: 2.1234 [2020-12-31 11:20:21,022][root][INFO] - ===> Epoch[1195](5975/5): Loss -0.6862, Sem -0.6862, dir -0.9683, norm 0.2419 LR: 6.330e-02 Score 98.664 Data time: 0.1145, Forward time: 0.9478, Backward time: 1.1979, DDP time: 0.0051, Total iter time: 2.3507 [2020-12-31 11:20:32,121][root][INFO] - ===> Epoch[1196](5980/5): Loss -0.6826, Sem -0.6826, dir -0.9665, norm 0.2397 LR: 6.327e-02 Score 98.554 Data time: 0.1077, Forward time: 0.9478, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.1280 [2020-12-31 11:20:43,794][root][INFO] - ===> Epoch[1197](5985/5): Loss -0.6915, Sem -0.6915, dir -0.9674, norm 0.2359 LR: 6.324e-02 Score 98.662 Data time: 0.0677, Forward time: 0.9476, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.2588 [2020-12-31 11:20:56,498][root][INFO] - ===> Epoch[1198](5990/5): Loss -0.6933, Sem -0.6933, dir -0.9687, norm 0.2364 LR: 6.321e-02 Score 98.705 Data time: 0.1345, Forward time: 0.9476, Backward time: 1.1979, DDP time: 0.0051, Total iter time: 2.4347 [2020-12-31 11:21:09,292][root][INFO] - ===> Epoch[1199](5995/5): Loss -0.6970, Sem -0.6970, dir -0.9685, norm 0.2316 LR: 6.318e-02 Score 98.662 Data time: 0.0814, Forward time: 0.9475, Backward time: 1.1981, DDP time: 0.0051, Total iter time: 2.4685 [2020-12-31 11:21:22,337][root][INFO] - ===> Epoch[1200](6000/5): Loss -0.6840, Sem -0.6840, dir -0.9668, norm 0.2434 LR: 6.315e-02 Score 98.690 Data time: 0.1093, Forward time: 0.9475, Backward time: 1.1983, DDP time: 0.0051, Total iter time: 2.5004 [2020-12-31 11:21:22,340][root][INFO] - ===> Start testing [2020-12-31 11:22:00,192][root][INFO] - 68/68: Data time: 0.0023, Iter time: 0.1271 Loss 0.224 (AVG: 0.522) Score 94.541 (AVG: 89.273) mIOU 67.592 mAP 82.687 mAcc 75.072 IOU: 58.205 0.000 79.036 72.093 93.077 90.293 28.105 74.233 98.357 67.802 80.042 84.284 53.167 mAP: 79.957 4.541 94.533 86.707 98.940 98.132 59.638 93.359 99.833 82.742 93.062 94.157 89.334 mAcc: 76.991 0.000 87.979 77.976 94.426 95.380 31.292 86.882 98.989 78.966 91.542 96.342 59.176 [2020-12-31 11:22:00,195][root][INFO] - Finished test. Elapsed time: 37.8511 [2020-12-31 11:22:03,289][root][INFO] - [2020-12-31 11:22:03,289][root][INFO] - ################################################################ [2020-12-31 11:22:03,289][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 11:22:03,289][root][INFO] - ################################################################ [2020-12-31 11:22:03,289][root][INFO] - clutter : nan nan nan [2020-12-31 11:22:03,289][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 11:22:03,289][root][INFO] - board : 0.793 0.843 0.913 [2020-12-31 11:22:03,290][root][INFO] - bookcase : 0.209 0.331 0.496 [2020-12-31 11:22:03,290][root][INFO] - ceiling : 0.667 0.708 0.767 [2020-12-31 11:22:03,290][root][INFO] - chair : 0.762 0.868 0.886 [2020-12-31 11:22:03,290][root][INFO] - column : 0.162 0.214 0.344 [2020-12-31 11:22:03,290][root][INFO] - door : 0.573 0.797 0.860 [2020-12-31 11:22:03,290][root][INFO] - floor : 0.974 0.993 0.993 [2020-12-31 11:22:03,290][root][INFO] - sofa : 0.472 0.717 0.717 [2020-12-31 11:22:03,290][root][INFO] - table : 0.186 0.376 0.658 [2020-12-31 11:22:03,290][root][INFO] - wall : 0.475 0.712 0.817 [2020-12-31 11:22:03,290][root][INFO] - window : 0.408 0.730 0.763 [2020-12-31 11:22:03,290][root][INFO] - ---------------------------------------------------------------- [2020-12-31 11:22:03,290][root][INFO] - average : 0.473 0.607 0.684 [2020-12-31 11:22:03,290][root][INFO] - [2020-12-31 11:22:03,457][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 11:22:04,114][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_mAP.pth [2020-12-31 11:22:04,114][root][INFO] - Current best mAP@0.5: 0.607 at iter 6000 [2020-12-31 11:22:04,888][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter6000.pth [2020-12-31 11:22:15,274][root][INFO] - ===> Epoch[1201](6005/5): Loss -0.6974, Sem -0.6974, dir -0.9683, norm 0.2309 LR: 6.312e-02 Score 98.672 Data time: 0.0809, Forward time: 0.9474, Backward time: 1.1981, DDP time: 0.0051, Total iter time: 1.9777 [2020-12-31 11:22:26,619][root][INFO] - ===> Epoch[1202](6010/5): Loss -0.6975, Sem -0.6975, dir -0.9673, norm 0.2300 LR: 6.308e-02 Score 98.675 Data time: 0.0991, Forward time: 0.9474, Backward time: 1.1981, DDP time: 0.0051, Total iter time: 2.1806 [2020-12-31 11:22:38,529][root][INFO] - ===> Epoch[1203](6015/5): Loss -0.6991, Sem -0.6991, dir -0.9677, norm 0.2284 LR: 6.305e-02 Score 98.663 Data time: 0.1060, Forward time: 0.9474, Backward time: 1.1980, DDP time: 0.0051, Total iter time: 2.2803 [2020-12-31 11:22:50,291][root][INFO] - ===> Epoch[1204](6020/5): Loss -0.6874, Sem -0.6874, dir -0.9687, norm 0.2410 LR: 6.302e-02 Score 98.647 Data time: 0.1285, Forward time: 0.9475, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.2592 [2020-12-31 11:23:02,325][root][INFO] - ===> Epoch[1205](6025/5): Loss -0.6964, Sem -0.6964, dir -0.9688, norm 0.2344 LR: 6.299e-02 Score 98.733 Data time: 0.1004, Forward time: 0.9475, Backward time: 1.1979, DDP time: 0.0051, Total iter time: 2.3175 [2020-12-31 11:23:14,886][root][INFO] - ===> Epoch[1206](6030/5): Loss -0.6918, Sem -0.6918, dir -0.9684, norm 0.2378 LR: 6.296e-02 Score 98.716 Data time: 0.1811, Forward time: 0.9476, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.4184 [2020-12-31 11:23:25,673][root][INFO] - ===> Epoch[1207](6035/5): Loss -0.7027, Sem -0.7027, dir -0.9685, norm 0.2274 LR: 6.293e-02 Score 98.708 Data time: 0.1058, Forward time: 0.9476, Backward time: 1.1976, DDP time: 0.0051, Total iter time: 2.0662 [2020-12-31 11:23:38,168][root][INFO] - ===> Epoch[1208](6040/5): Loss -0.7010, Sem -0.7010, dir -0.9685, norm 0.2281 LR: 6.289e-02 Score 98.676 Data time: 0.0947, Forward time: 0.9477, Backward time: 1.1976, DDP time: 0.0051, Total iter time: 2.3995 [2020-12-31 11:23:49,724][root][INFO] - ===> Epoch[1209](6045/5): Loss -0.6939, Sem -0.6939, dir -0.9672, norm 0.2317 LR: 6.286e-02 Score 98.612 Data time: 0.1046, Forward time: 0.9477, Backward time: 1.1975, DDP time: 0.0051, Total iter time: 2.2153 [2020-12-31 11:24:01,817][root][INFO] - ===> Epoch[1210](6050/5): Loss -0.7031, Sem -0.7031, dir -0.9690, norm 0.2257 LR: 6.283e-02 Score 98.662 Data time: 0.0695, Forward time: 0.9476, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.3404 [2020-12-31 11:24:13,809][root][INFO] - ===> Epoch[1211](6055/5): Loss -0.7074, Sem -0.7074, dir -0.9691, norm 0.2220 LR: 6.280e-02 Score 98.668 Data time: 0.1061, Forward time: 0.9476, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.3111 [2020-12-31 11:24:25,871][root][INFO] - ===> Epoch[1212](6060/5): Loss -0.7007, Sem -0.7007, dir -0.9676, norm 0.2248 LR: 6.277e-02 Score 98.602 Data time: 0.1573, Forward time: 0.9476, Backward time: 1.1976, DDP time: 0.0051, Total iter time: 2.3239 [2020-12-31 11:24:38,051][root][INFO] - ===> Epoch[1213](6065/5): Loss -0.6919, Sem -0.6919, dir -0.9676, norm 0.2338 LR: 6.274e-02 Score 98.616 Data time: 0.1123, Forward time: 0.9476, Backward time: 1.1976, DDP time: 0.0051, Total iter time: 2.3424 [2020-12-31 11:24:50,441][root][INFO] - ===> Epoch[1214](6070/5): Loss -0.6988, Sem -0.6988, dir -0.9680, norm 0.2275 LR: 6.270e-02 Score 98.607 Data time: 0.1096, Forward time: 0.9477, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.3899 [2020-12-31 11:25:02,878][root][INFO] - ===> Epoch[1215](6075/5): Loss -0.6836, Sem -0.6836, dir -0.9676, norm 0.2414 LR: 6.267e-02 Score 98.588 Data time: 0.0939, Forward time: 0.9477, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.3980 [2020-12-31 11:25:15,256][root][INFO] - ===> Epoch[1216](6080/5): Loss -0.6924, Sem -0.6924, dir -0.9683, norm 0.2344 LR: 6.264e-02 Score 98.623 Data time: 0.1384, Forward time: 0.9477, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.3855 [2020-12-31 11:25:27,286][root][INFO] - ===> Epoch[1217](6085/5): Loss -0.6883, Sem -0.6883, dir -0.9677, norm 0.2362 LR: 6.261e-02 Score 98.576 Data time: 0.1270, Forward time: 0.9477, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.3178 [2020-12-31 11:25:38,804][root][INFO] - ===> Epoch[1218](6090/5): Loss -0.6837, Sem -0.6837, dir -0.9669, norm 0.2412 LR: 6.258e-02 Score 98.592 Data time: 0.0849, Forward time: 0.9476, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.2107 [2020-12-31 11:25:50,825][root][INFO] - ===> Epoch[1219](6095/5): Loss -0.7013, Sem -0.7013, dir -0.9699, norm 0.2270 LR: 6.255e-02 Score 98.611 Data time: 0.1782, Forward time: 0.9476, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.3116 [2020-12-31 11:26:03,590][root][INFO] - ===> Epoch[1220](6100/5): Loss -0.6958, Sem -0.6958, dir -0.9682, norm 0.2323 LR: 6.252e-02 Score 98.667 Data time: 0.1087, Forward time: 0.9477, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.4562 [2020-12-31 11:26:15,889][root][INFO] - ===> Epoch[1221](6105/5): Loss -0.6917, Sem -0.6917, dir -0.9691, norm 0.2358 LR: 6.248e-02 Score 98.613 Data time: 0.0914, Forward time: 0.9476, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.3634 [2020-12-31 11:26:28,507][root][INFO] - ===> Epoch[1222](6110/5): Loss -0.6952, Sem -0.6952, dir -0.9681, norm 0.2322 LR: 6.245e-02 Score 98.657 Data time: 0.0966, Forward time: 0.9476, Backward time: 1.1979, DDP time: 0.0051, Total iter time: 2.4321 [2020-12-31 11:26:40,673][root][INFO] - ===> Epoch[1223](6115/5): Loss -0.6933, Sem -0.6933, dir -0.9682, norm 0.2348 LR: 6.242e-02 Score 98.680 Data time: 0.1331, Forward time: 0.9476, Backward time: 1.1979, DDP time: 0.0051, Total iter time: 2.3412 [2020-12-31 11:26:53,094][root][INFO] - ===> Epoch[1224](6120/5): Loss -0.6911, Sem -0.6911, dir -0.9679, norm 0.2345 LR: 6.239e-02 Score 98.598 Data time: 0.0896, Forward time: 0.9476, Backward time: 1.1980, DDP time: 0.0051, Total iter time: 2.3901 [2020-12-31 11:27:05,867][root][INFO] - ===> Epoch[1225](6125/5): Loss -0.7017, Sem -0.7017, dir -0.9685, norm 0.2280 LR: 6.236e-02 Score 98.704 Data time: 0.1831, Forward time: 0.9478, Backward time: 1.1979, DDP time: 0.0051, Total iter time: 2.4548 [2020-12-31 11:27:17,916][root][INFO] - ===> Epoch[1226](6130/5): Loss -0.6993, Sem -0.6993, dir -0.9687, norm 0.2307 LR: 6.233e-02 Score 98.711 Data time: 0.1369, Forward time: 0.9478, Backward time: 1.1979, DDP time: 0.0051, Total iter time: 2.3155 [2020-12-31 11:27:29,892][root][INFO] - ===> Epoch[1227](6135/5): Loss -0.7052, Sem -0.7052, dir -0.9684, norm 0.2222 LR: 6.229e-02 Score 98.631 Data time: 0.1531, Forward time: 0.9478, Backward time: 1.1978, DDP time: 0.0051, Total iter time: 2.2999 [2020-12-31 11:27:41,665][root][INFO] - ===> Epoch[1228](6140/5): Loss -0.7045, Sem -0.7045, dir -0.9681, norm 0.2252 LR: 6.226e-02 Score 98.712 Data time: 0.1264, Forward time: 0.9479, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.2583 [2020-12-31 11:27:53,563][root][INFO] - ===> Epoch[1229](6145/5): Loss -0.7087, Sem -0.7087, dir -0.9698, norm 0.2240 LR: 6.223e-02 Score 98.760 Data time: 0.0993, Forward time: 0.9479, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.2865 [2020-12-31 11:28:06,000][root][INFO] - ===> Epoch[1230](6150/5): Loss -0.7035, Sem -0.7035, dir -0.9681, norm 0.2241 LR: 6.220e-02 Score 98.658 Data time: 0.0930, Forward time: 0.9478, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.3984 [2020-12-31 11:28:18,611][root][INFO] - ===> Epoch[1231](6155/5): Loss -0.6952, Sem -0.6952, dir -0.9689, norm 0.2331 LR: 6.217e-02 Score 98.646 Data time: 0.1850, Forward time: 0.9479, Backward time: 1.1977, DDP time: 0.0051, Total iter time: 2.4250 [2020-12-31 11:28:29,741][root][INFO] - ===> Epoch[1232](6160/5): Loss -0.6931, Sem -0.6931, dir -0.9693, norm 0.2373 LR: 6.214e-02 Score 98.698 Data time: 0.1066, Forward time: 0.9478, Backward time: 1.1976, DDP time: 0.0051, Total iter time: 2.1312 [2020-12-31 11:28:41,442][root][INFO] - ===> Epoch[1233](6165/5): Loss -0.7015, Sem -0.7015, dir -0.9698, norm 0.2292 LR: 6.210e-02 Score 98.683 Data time: 0.2050, Forward time: 0.9478, Backward time: 1.1975, DDP time: 0.0051, Total iter time: 2.2467 [2020-12-31 11:28:53,546][root][INFO] - ===> Epoch[1234](6170/5): Loss -0.6956, Sem -0.6956, dir -0.9688, norm 0.2345 LR: 6.207e-02 Score 98.703 Data time: 0.1273, Forward time: 0.9479, Backward time: 1.1974, DDP time: 0.0051, Total iter time: 2.3211 [2020-12-31 11:29:04,964][root][INFO] - ===> Epoch[1235](6175/5): Loss -0.7050, Sem -0.7050, dir -0.9694, norm 0.2272 LR: 6.204e-02 Score 98.743 Data time: 0.1154, Forward time: 0.9480, Backward time: 1.1973, DDP time: 0.0051, Total iter time: 2.1901 [2020-12-31 11:29:17,428][root][INFO] - ===> Epoch[1236](6180/5): Loss -0.6980, Sem -0.6980, dir -0.9693, norm 0.2315 LR: 6.201e-02 Score 98.696 Data time: 0.1699, Forward time: 0.9481, Backward time: 1.1972, DDP time: 0.0051, Total iter time: 2.3925 [2020-12-31 11:29:29,258][root][INFO] - ===> Epoch[1237](6185/5): Loss -0.7055, Sem -0.7055, dir -0.9691, norm 0.2255 LR: 6.198e-02 Score 98.717 Data time: 0.1951, Forward time: 0.9481, Backward time: 1.1971, DDP time: 0.0051, Total iter time: 2.2857 [2020-12-31 11:29:40,619][root][INFO] - ===> Epoch[1238](6190/5): Loss -0.6954, Sem -0.6954, dir -0.9691, norm 0.2351 LR: 6.195e-02 Score 98.705 Data time: 0.1009, Forward time: 0.9481, Backward time: 1.1970, DDP time: 0.0051, Total iter time: 2.1804 [2020-12-31 11:29:52,756][root][INFO] - ===> Epoch[1239](6195/5): Loss -0.7079, Sem -0.7079, dir -0.9702, norm 0.2251 LR: 6.191e-02 Score 98.744 Data time: 0.1101, Forward time: 0.9481, Backward time: 1.1970, DDP time: 0.0051, Total iter time: 2.3453 [2020-12-31 11:30:04,847][root][INFO] - ===> Epoch[1240](6200/5): Loss -0.6980, Sem -0.6980, dir -0.9688, norm 0.2324 LR: 6.188e-02 Score 98.715 Data time: 0.0871, Forward time: 0.9482, Backward time: 1.1970, DDP time: 0.0051, Total iter time: 2.3338 [2020-12-31 11:30:04,849][root][INFO] - ===> Start testing [2020-12-31 11:30:43,179][root][INFO] - 68/68: Data time: 0.0022, Iter time: 0.1105 Loss 0.034 (AVG: 0.514) Score 98.719 (AVG: 89.407) mIOU 68.033 mAP 82.720 mAcc 74.982 IOU: 59.597 0.000 80.384 71.435 93.370 88.961 29.114 76.002 98.228 69.380 79.404 84.698 53.852 mAP: 80.570 4.322 93.770 85.620 98.988 98.381 58.924 94.454 99.825 84.376 92.929 93.907 89.299 mAcc: 82.341 0.000 87.145 76.977 94.676 95.799 34.084 88.286 98.844 73.658 88.619 95.411 58.924 [2020-12-31 11:30:43,180][root][INFO] - Finished test. Elapsed time: 38.3295 [2020-12-31 11:30:46,258][root][INFO] - [2020-12-31 11:30:46,259][root][INFO] - ################################################################ [2020-12-31 11:30:46,259][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 11:30:46,259][root][INFO] - ################################################################ [2020-12-31 11:30:46,259][root][INFO] - clutter : nan nan nan [2020-12-31 11:30:46,259][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 11:30:46,259][root][INFO] - board : 0.758 0.825 0.878 [2020-12-31 11:30:46,259][root][INFO] - bookcase : 0.193 0.326 0.481 [2020-12-31 11:30:46,259][root][INFO] - ceiling : 0.614 0.638 0.745 [2020-12-31 11:30:46,259][root][INFO] - chair : 0.767 0.863 0.901 [2020-12-31 11:30:46,259][root][INFO] - column : 0.103 0.137 0.333 [2020-12-31 11:30:46,259][root][INFO] - door : 0.533 0.743 0.881 [2020-12-31 11:30:46,259][root][INFO] - floor : 0.996 1.000 1.000 [2020-12-31 11:30:46,259][root][INFO] - sofa : 0.401 0.593 0.727 [2020-12-31 11:30:46,259][root][INFO] - table : 0.163 0.348 0.687 [2020-12-31 11:30:46,259][root][INFO] - wall : 0.425 0.648 0.743 [2020-12-31 11:30:46,259][root][INFO] - window : 0.454 0.708 0.762 [2020-12-31 11:30:46,259][root][INFO] - ---------------------------------------------------------------- [2020-12-31 11:30:46,259][root][INFO] - average : 0.450 0.569 0.678 [2020-12-31 11:30:46,259][root][INFO] - [2020-12-31 11:30:46,424][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 11:30:46,993][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2020-12-31 11:30:46,993][root][INFO] - Current best mIoU: 68.033 at iter 1 [2020-12-31 11:30:47,608][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter6200.pth [2020-12-31 11:30:58,489][root][INFO] - ===> Epoch[1241](6205/5): Loss -0.7017, Sem -0.7017, dir -0.9686, norm 0.2276 LR: 6.185e-02 Score 98.688 Data time: 0.0935, Forward time: 0.9481, Backward time: 1.1968, DDP time: 0.0051, Total iter time: 2.0725 [2020-12-31 11:31:10,901][root][INFO] - ===> Epoch[1242](6210/5): Loss -0.6973, Sem -0.6973, dir -0.9688, norm 0.2325 LR: 6.182e-02 Score 98.689 Data time: 0.2017, Forward time: 0.9482, Backward time: 1.1967, DDP time: 0.0051, Total iter time: 2.3772 [2020-12-31 11:31:22,295][root][INFO] - ===> Epoch[1243](6215/5): Loss -0.6958, Sem -0.6958, dir -0.9676, norm 0.2303 LR: 6.179e-02 Score 98.630 Data time: 0.0887, Forward time: 0.9483, Backward time: 1.1966, DDP time: 0.0051, Total iter time: 2.1813 [2020-12-31 11:31:34,384][root][INFO] - ===> Epoch[1244](6220/5): Loss -0.6971, Sem -0.6971, dir -0.9681, norm 0.2309 LR: 6.176e-02 Score 98.666 Data time: 0.1119, Forward time: 0.9483, Backward time: 1.1966, DDP time: 0.0051, Total iter time: 2.3138 [2020-12-31 11:31:45,806][root][INFO] - ===> Epoch[1245](6225/5): Loss -0.7025, Sem -0.7025, dir -0.9687, norm 0.2278 LR: 6.172e-02 Score 98.715 Data time: 0.0753, Forward time: 0.9482, Backward time: 1.1966, DDP time: 0.0051, Total iter time: 2.1933 [2020-12-31 11:31:57,518][root][INFO] - ===> Epoch[1246](6230/5): Loss -0.6861, Sem -0.6861, dir -0.9680, norm 0.2386 LR: 6.169e-02 Score 98.610 Data time: 0.1213, Forward time: 0.9482, Backward time: 1.1965, DDP time: 0.0051, Total iter time: 2.2448 [2020-12-31 11:32:09,249][root][INFO] - ===> Epoch[1247](6235/5): Loss -0.7010, Sem -0.7010, dir -0.9686, norm 0.2283 LR: 6.166e-02 Score 98.684 Data time: 0.1283, Forward time: 0.9483, Backward time: 1.1964, DDP time: 0.0051, Total iter time: 2.2445 [2020-12-31 11:32:21,055][root][INFO] - ===> Epoch[1248](6240/5): Loss -0.6924, Sem -0.6924, dir -0.9677, norm 0.2338 LR: 6.163e-02 Score 98.613 Data time: 0.1371, Forward time: 0.9483, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.2740 [2020-12-31 11:32:33,073][root][INFO] - ===> Epoch[1249](6245/5): Loss -0.6836, Sem -0.6836, dir -0.9676, norm 0.2402 LR: 6.160e-02 Score 98.571 Data time: 0.1092, Forward time: 0.9484, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.2992 [2020-12-31 11:32:44,406][root][INFO] - ===> Epoch[1250](6250/5): Loss -0.6912, Sem -0.6912, dir -0.9694, norm 0.2389 LR: 6.157e-02 Score 98.694 Data time: 0.0732, Forward time: 0.9483, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.1684 [2020-12-31 11:32:56,131][root][INFO] - ===> Epoch[1251](6255/5): Loss -0.6988, Sem -0.6988, dir -0.9688, norm 0.2298 LR: 6.153e-02 Score 98.655 Data time: 0.1312, Forward time: 0.9484, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.2489 [2020-12-31 11:33:07,965][root][INFO] - ===> Epoch[1252](6260/5): Loss -0.7020, Sem -0.7020, dir -0.9700, norm 0.2301 LR: 6.150e-02 Score 98.738 Data time: 0.0915, Forward time: 0.9484, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.2752 [2020-12-31 11:33:19,984][root][INFO] - ===> Epoch[1253](6265/5): Loss -0.6941, Sem -0.6941, dir -0.9667, norm 0.2318 LR: 6.147e-02 Score 98.638 Data time: 0.0847, Forward time: 0.9484, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.3086 [2020-12-31 11:33:31,983][root][INFO] - ===> Epoch[1254](6270/5): Loss -0.7009, Sem -0.7009, dir -0.9696, norm 0.2298 LR: 6.144e-02 Score 98.692 Data time: 0.1360, Forward time: 0.9484, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.3057 [2020-12-31 11:33:43,628][root][INFO] - ===> Epoch[1255](6275/5): Loss -0.6949, Sem -0.6949, dir -0.9691, norm 0.2364 LR: 6.141e-02 Score 98.752 Data time: 0.1292, Forward time: 0.9484, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.2452 [2020-12-31 11:33:55,004][root][INFO] - ===> Epoch[1256](6280/5): Loss -0.6998, Sem -0.6998, dir -0.9686, norm 0.2292 LR: 6.138e-02 Score 98.690 Data time: 0.1363, Forward time: 0.9484, Backward time: 1.1959, DDP time: 0.0051, Total iter time: 2.1813 [2020-12-31 11:34:07,113][root][INFO] - ===> Epoch[1257](6285/5): Loss -0.7037, Sem -0.7037, dir -0.9695, norm 0.2282 LR: 6.134e-02 Score 98.727 Data time: 0.0981, Forward time: 0.9484, Backward time: 1.1958, DDP time: 0.0051, Total iter time: 2.3418 [2020-12-31 11:34:19,568][root][INFO] - ===> Epoch[1258](6290/5): Loss -0.6967, Sem -0.6967, dir -0.9687, norm 0.2329 LR: 6.131e-02 Score 98.688 Data time: 0.1106, Forward time: 0.9485, Backward time: 1.1959, DDP time: 0.0051, Total iter time: 2.4032 [2020-12-31 11:34:31,309][root][INFO] - ===> Epoch[1259](6295/5): Loss -0.6927, Sem -0.6927, dir -0.9679, norm 0.2342 LR: 6.128e-02 Score 98.639 Data time: 0.1803, Forward time: 0.9485, Backward time: 1.1958, DDP time: 0.0051, Total iter time: 2.2565 [2020-12-31 11:34:43,038][root][INFO] - ===> Epoch[1260](6300/5): Loss -0.6963, Sem -0.6963, dir -0.9691, norm 0.2320 LR: 6.125e-02 Score 98.661 Data time: 0.1409, Forward time: 0.9486, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.2600 [2020-12-31 11:34:55,352][root][INFO] - ===> Epoch[1261](6305/5): Loss -0.7005, Sem -0.7005, dir -0.9698, norm 0.2323 LR: 6.122e-02 Score 98.763 Data time: 0.1550, Forward time: 0.9486, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.3753 [2020-12-31 11:35:07,520][root][INFO] - ===> Epoch[1262](6310/5): Loss -0.7046, Sem -0.7046, dir -0.9693, norm 0.2257 LR: 6.119e-02 Score 98.699 Data time: 0.1045, Forward time: 0.9486, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.3427 [2020-12-31 11:35:19,534][root][INFO] - ===> Epoch[1263](6315/5): Loss -0.7020, Sem -0.7020, dir -0.9693, norm 0.2289 LR: 6.115e-02 Score 98.716 Data time: 0.0909, Forward time: 0.9485, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.3174 [2020-12-31 11:35:32,049][root][INFO] - ===> Epoch[1264](6320/5): Loss -0.7063, Sem -0.7063, dir -0.9696, norm 0.2241 LR: 6.112e-02 Score 98.674 Data time: 0.0885, Forward time: 0.9486, Backward time: 1.1958, DDP time: 0.0051, Total iter time: 2.4103 [2020-12-31 11:35:43,261][root][INFO] - ===> Epoch[1265](6325/5): Loss -0.7070, Sem -0.7070, dir -0.9698, norm 0.2244 LR: 6.109e-02 Score 98.724 Data time: 0.1212, Forward time: 0.9487, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.1578 [2020-12-31 11:35:55,865][root][INFO] - ===> Epoch[1266](6330/5): Loss -0.6914, Sem -0.6914, dir -0.9685, norm 0.2361 LR: 6.106e-02 Score 98.646 Data time: 0.0787, Forward time: 0.9486, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.4389 [2020-12-31 11:36:07,972][root][INFO] - ===> Epoch[1267](6335/5): Loss -0.6986, Sem -0.6986, dir -0.9694, norm 0.2306 LR: 6.103e-02 Score 98.654 Data time: 0.1652, Forward time: 0.9486, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.3339 [2020-12-31 11:36:20,128][root][INFO] - ===> Epoch[1268](6340/5): Loss -0.7053, Sem -0.7053, dir -0.9689, norm 0.2254 LR: 6.100e-02 Score 98.724 Data time: 0.0926, Forward time: 0.9486, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.3448 [2020-12-31 11:36:31,604][root][INFO] - ===> Epoch[1269](6345/5): Loss -0.6988, Sem -0.6988, dir -0.9692, norm 0.2320 LR: 6.096e-02 Score 98.722 Data time: 0.0914, Forward time: 0.9486, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.1942 [2020-12-31 11:36:43,708][root][INFO] - ===> Epoch[1270](6350/5): Loss -0.6999, Sem -0.6999, dir -0.9690, norm 0.2309 LR: 6.093e-02 Score 98.719 Data time: 0.0734, Forward time: 0.9485, Backward time: 1.1958, DDP time: 0.0051, Total iter time: 2.3290 [2020-12-31 11:36:55,794][root][INFO] - ===> Epoch[1271](6355/5): Loss -0.6935, Sem -0.6935, dir -0.9675, norm 0.2345 LR: 6.090e-02 Score 98.696 Data time: 0.1204, Forward time: 0.9485, Backward time: 1.1958, DDP time: 0.0051, Total iter time: 2.3300 [2020-12-31 11:37:08,157][root][INFO] - ===> Epoch[1272](6360/5): Loss -0.7068, Sem -0.7068, dir -0.9700, norm 0.2252 LR: 6.087e-02 Score 98.724 Data time: 0.0768, Forward time: 0.9484, Backward time: 1.1959, DDP time: 0.0051, Total iter time: 2.3866 [2020-12-31 11:37:20,562][root][INFO] - ===> Epoch[1273](6365/5): Loss -0.7013, Sem -0.7013, dir -0.9695, norm 0.2293 LR: 6.084e-02 Score 98.690 Data time: 0.0969, Forward time: 0.9484, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.3853 [2020-12-31 11:37:32,991][root][INFO] - ===> Epoch[1274](6370/5): Loss -0.6941, Sem -0.6941, dir -0.9685, norm 0.2331 LR: 6.081e-02 Score 98.649 Data time: 0.1397, Forward time: 0.9483, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.3970 [2020-12-31 11:37:45,162][root][INFO] - ===> Epoch[1275](6375/5): Loss -0.6983, Sem -0.6983, dir -0.9689, norm 0.2289 LR: 6.077e-02 Score 98.624 Data time: 0.0972, Forward time: 0.9483, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.3533 [2020-12-31 11:37:57,744][root][INFO] - ===> Epoch[1276](6380/5): Loss -0.7124, Sem -0.7124, dir -0.9704, norm 0.2195 LR: 6.074e-02 Score 98.713 Data time: 0.0807, Forward time: 0.9482, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.4352 [2020-12-31 11:38:10,717][root][INFO] - ===> Epoch[1277](6385/5): Loss -0.7052, Sem -0.7052, dir -0.9703, norm 0.2282 LR: 6.071e-02 Score 98.777 Data time: 0.1037, Forward time: 0.9482, Backward time: 1.1964, DDP time: 0.0051, Total iter time: 2.4953 [2020-12-31 11:38:22,197][root][INFO] - ===> Epoch[1278](6390/5): Loss -0.7091, Sem -0.7091, dir -0.9705, norm 0.2227 LR: 6.068e-02 Score 98.713 Data time: 0.1226, Forward time: 0.9482, Backward time: 1.1964, DDP time: 0.0051, Total iter time: 2.2024 [2020-12-31 11:38:34,792][root][INFO] - ===> Epoch[1279](6395/5): Loss -0.7100, Sem -0.7100, dir -0.9699, norm 0.2219 LR: 6.065e-02 Score 98.730 Data time: 0.1064, Forward time: 0.9482, Backward time: 1.1965, DDP time: 0.0051, Total iter time: 2.4229 [2020-12-31 11:38:46,989][root][INFO] - ===> Epoch[1280](6400/5): Loss -0.7164, Sem -0.7164, dir -0.9708, norm 0.2170 LR: 6.062e-02 Score 98.746 Data time: 0.1256, Forward time: 0.9481, Backward time: 1.1965, DDP time: 0.0051, Total iter time: 2.3523 [2020-12-31 11:38:46,992][root][INFO] - ===> Start testing [2020-12-31 11:39:27,476][root][INFO] - 68/68: Data time: 0.0024, Iter time: 0.1323 Loss 1.465 (AVG: 0.559) Score 65.345 (AVG: 88.545) mIOU 67.035 mAP 82.749 mAcc 74.886 IOU: 58.001 0.000 78.381 71.874 92.745 89.052 25.433 69.983 98.383 70.905 80.296 82.252 54.151 mAP: 80.707 8.150 92.345 84.916 98.772 98.408 58.392 92.521 99.838 86.612 93.279 93.615 88.178 mAcc: 79.137 0.000 85.021 78.583 94.360 96.064 36.448 87.552 99.096 75.710 90.235 92.604 58.711 [2020-12-31 11:39:27,477][root][INFO] - Finished test. Elapsed time: 40.4822 [2020-12-31 11:39:30,732][root][INFO] - [2020-12-31 11:39:30,732][root][INFO] - ################################################################ [2020-12-31 11:39:30,732][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 11:39:30,732][root][INFO] - ################################################################ [2020-12-31 11:39:30,732][root][INFO] - clutter : nan nan nan [2020-12-31 11:39:30,732][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 11:39:30,732][root][INFO] - board : 0.724 0.831 0.881 [2020-12-31 11:39:30,732][root][INFO] - bookcase : 0.214 0.356 0.497 [2020-12-31 11:39:30,732][root][INFO] - ceiling : 0.609 0.654 0.725 [2020-12-31 11:39:30,732][root][INFO] - chair : 0.759 0.861 0.887 [2020-12-31 11:39:30,732][root][INFO] - column : 0.136 0.214 0.310 [2020-12-31 11:39:30,732][root][INFO] - door : 0.545 0.725 0.891 [2020-12-31 11:39:30,732][root][INFO] - floor : 0.961 0.973 0.973 [2020-12-31 11:39:30,732][root][INFO] - sofa : 0.420 0.531 0.727 [2020-12-31 11:39:30,732][root][INFO] - table : 0.199 0.393 0.642 [2020-12-31 11:39:30,732][root][INFO] - wall : 0.444 0.648 0.773 [2020-12-31 11:39:30,732][root][INFO] - window : 0.481 0.706 0.749 [2020-12-31 11:39:30,733][root][INFO] - ---------------------------------------------------------------- [2020-12-31 11:39:30,733][root][INFO] - average : 0.458 0.574 0.671 [2020-12-31 11:39:30,733][root][INFO] - [2020-12-31 11:39:30,911][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 11:39:31,508][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter6400.pth [2020-12-31 11:39:43,165][root][INFO] - ===> Epoch[1281](6405/5): Loss -0.7001, Sem -0.7001, dir -0.9690, norm 0.2297 LR: 6.058e-02 Score 98.692 Data time: 0.1079, Forward time: 0.9481, Backward time: 1.1965, DDP time: 0.0051, Total iter time: 2.2177 [2020-12-31 11:39:55,476][root][INFO] - ===> Epoch[1282](6410/5): Loss -0.6976, Sem -0.6976, dir -0.9681, norm 0.2304 LR: 6.055e-02 Score 98.669 Data time: 0.1607, Forward time: 0.9482, Backward time: 1.1964, DDP time: 0.0051, Total iter time: 2.3604 [2020-12-31 11:40:07,079][root][INFO] - ===> Epoch[1283](6415/5): Loss -0.7035, Sem -0.7035, dir -0.9696, norm 0.2272 LR: 6.052e-02 Score 98.689 Data time: 0.1148, Forward time: 0.9482, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.2317 [2020-12-31 11:40:18,740][root][INFO] - ===> Epoch[1284](6420/5): Loss -0.7020, Sem -0.7020, dir -0.9694, norm 0.2286 LR: 6.049e-02 Score 98.720 Data time: 0.1488, Forward time: 0.9482, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.2332 [2020-12-31 11:40:30,672][root][INFO] - ===> Epoch[1285](6425/5): Loss -0.7031, Sem -0.7031, dir -0.9693, norm 0.2274 LR: 6.046e-02 Score 98.717 Data time: 0.0957, Forward time: 0.9482, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.2964 [2020-12-31 11:40:43,908][root][INFO] - ===> Epoch[1286](6430/5): Loss -0.7058, Sem -0.7058, dir -0.9705, norm 0.2269 LR: 6.043e-02 Score 98.741 Data time: 0.1328, Forward time: 0.9483, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.5519 [2020-12-31 11:40:55,174][root][INFO] - ===> Epoch[1287](6435/5): Loss -0.7007, Sem -0.7007, dir -0.9691, norm 0.2261 LR: 6.039e-02 Score 98.621 Data time: 0.1325, Forward time: 0.9483, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.1673 [2020-12-31 11:41:07,043][root][INFO] - ===> Epoch[1288](6440/5): Loss -0.6933, Sem -0.6933, dir -0.9694, norm 0.2340 LR: 6.036e-02 Score 98.604 Data time: 0.0900, Forward time: 0.9481, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.2766 [2020-12-31 11:41:19,807][root][INFO] - ===> Epoch[1289](6445/5): Loss -0.6978, Sem -0.6978, dir -0.9690, norm 0.2308 LR: 6.033e-02 Score 98.652 Data time: 0.1122, Forward time: 0.9481, Backward time: 1.1964, DDP time: 0.0051, Total iter time: 2.4558 [2020-12-31 11:41:31,000][root][INFO] - ===> Epoch[1290](6450/5): Loss -0.7052, Sem -0.7052, dir -0.9693, norm 0.2231 LR: 6.030e-02 Score 98.634 Data time: 0.0859, Forward time: 0.9481, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.1451 [2020-12-31 11:41:42,140][root][INFO] - ===> Epoch[1291](6455/5): Loss -0.7081, Sem -0.7081, dir -0.9694, norm 0.2223 LR: 6.027e-02 Score 98.685 Data time: 0.1487, Forward time: 0.9481, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.1415 [2020-12-31 11:41:54,430][root][INFO] - ===> Epoch[1292](6460/5): Loss -0.6948, Sem -0.6948, dir -0.9682, norm 0.2340 LR: 6.024e-02 Score 98.694 Data time: 0.0904, Forward time: 0.9481, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.3589 [2020-12-31 11:42:07,199][root][INFO] - ===> Epoch[1293](6465/5): Loss -0.7027, Sem -0.7027, dir -0.9693, norm 0.2265 LR: 6.020e-02 Score 98.653 Data time: 0.0981, Forward time: 0.9481, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.4626 [2020-12-31 11:42:19,109][root][INFO] - ===> Epoch[1294](6470/5): Loss -0.7001, Sem -0.7001, dir -0.9696, norm 0.2298 LR: 6.017e-02 Score 98.697 Data time: 0.0812, Forward time: 0.9480, Backward time: 1.1964, DDP time: 0.0051, Total iter time: 2.2952 [2020-12-31 11:42:30,971][root][INFO] - ===> Epoch[1295](6475/5): Loss -0.7094, Sem -0.7094, dir -0.9690, norm 0.2208 LR: 6.014e-02 Score 98.696 Data time: 0.1163, Forward time: 0.9480, Backward time: 1.1964, DDP time: 0.0051, Total iter time: 2.2752 [2020-12-31 11:42:42,831][root][INFO] - ===> Epoch[1296](6480/5): Loss -0.7045, Sem -0.7045, dir -0.9691, norm 0.2252 LR: 6.011e-02 Score 98.698 Data time: 0.1040, Forward time: 0.9481, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.2760 [2020-12-31 11:42:54,494][root][INFO] - ===> Epoch[1297](6485/5): Loss -0.6915, Sem -0.6915, dir -0.9692, norm 0.2378 LR: 6.008e-02 Score 98.686 Data time: 0.0959, Forward time: 0.9480, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.2375 [2020-12-31 11:43:05,499][root][INFO] - ===> Epoch[1298](6490/5): Loss -0.7026, Sem -0.7026, dir -0.9692, norm 0.2281 LR: 6.004e-02 Score 98.697 Data time: 0.1459, Forward time: 0.9480, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.1091 [2020-12-31 11:43:17,757][root][INFO] - ===> Epoch[1299](6495/5): Loss -0.6979, Sem -0.6979, dir -0.9697, norm 0.2330 LR: 6.001e-02 Score 98.714 Data time: 0.0940, Forward time: 0.9480, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.3588 [2020-12-31 11:43:29,979][root][INFO] - ===> Epoch[1300](6500/5): Loss -0.7021, Sem -0.7021, dir -0.9692, norm 0.2281 LR: 5.998e-02 Score 98.709 Data time: 0.1757, Forward time: 0.9480, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.3489 [2020-12-31 11:43:41,599][root][INFO] - ===> Epoch[1301](6505/5): Loss -0.7086, Sem -0.7086, dir -0.9698, norm 0.2218 LR: 5.995e-02 Score 98.696 Data time: 0.0900, Forward time: 0.9480, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.2313 [2020-12-31 11:43:54,475][root][INFO] - ===> Epoch[1302](6510/5): Loss -0.7098, Sem -0.7098, dir -0.9697, norm 0.2230 LR: 5.992e-02 Score 98.768 Data time: 0.1673, Forward time: 0.9480, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.4910 [2020-12-31 11:44:06,264][root][INFO] - ===> Epoch[1303](6515/5): Loss -0.7040, Sem -0.7040, dir -0.9698, norm 0.2282 LR: 5.989e-02 Score 98.754 Data time: 0.1432, Forward time: 0.9481, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.2605 [2020-12-31 11:44:18,769][root][INFO] - ===> Epoch[1304](6520/5): Loss -0.6952, Sem -0.6952, dir -0.9688, norm 0.2344 LR: 5.985e-02 Score 98.686 Data time: 0.1012, Forward time: 0.9481, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.3955 [2020-12-31 11:44:30,817][root][INFO] - ===> Epoch[1305](6525/5): Loss -0.7141, Sem -0.7141, dir -0.9705, norm 0.2188 LR: 5.982e-02 Score 98.744 Data time: 0.1044, Forward time: 0.9480, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.3162 [2020-12-31 11:44:43,491][root][INFO] - ===> Epoch[1306](6530/5): Loss -0.7036, Sem -0.7036, dir -0.9705, norm 0.2285 LR: 5.979e-02 Score 98.730 Data time: 0.0972, Forward time: 0.9480, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.4356 [2020-12-31 11:44:55,322][root][INFO] - ===> Epoch[1307](6535/5): Loss -0.7112, Sem -0.7112, dir -0.9708, norm 0.2235 LR: 5.976e-02 Score 98.783 Data time: 0.1115, Forward time: 0.9480, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.2608 [2020-12-31 11:45:07,396][root][INFO] - ===> Epoch[1308](6540/5): Loss -0.7101, Sem -0.7101, dir -0.9696, norm 0.2208 LR: 5.973e-02 Score 98.702 Data time: 0.1060, Forward time: 0.9479, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.3189 [2020-12-31 11:45:19,275][root][INFO] - ===> Epoch[1309](6545/5): Loss -0.6996, Sem -0.6996, dir -0.9695, norm 0.2316 LR: 5.970e-02 Score 98.730 Data time: 0.2074, Forward time: 0.9480, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.2500 [2020-12-31 11:45:31,080][root][INFO] - ===> Epoch[1310](6550/5): Loss -0.7057, Sem -0.7057, dir -0.9694, norm 0.2263 LR: 5.966e-02 Score 98.746 Data time: 0.0922, Forward time: 0.9480, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.2722 [2020-12-31 11:45:42,094][root][INFO] - ===> Epoch[1311](6555/5): Loss -0.7079, Sem -0.7079, dir -0.9694, norm 0.2229 LR: 5.963e-02 Score 98.698 Data time: 0.0771, Forward time: 0.9479, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.1167 [2020-12-31 11:45:55,074][root][INFO] - ===> Epoch[1312](6560/5): Loss -0.7070, Sem -0.7070, dir -0.9700, norm 0.2246 LR: 5.960e-02 Score 98.722 Data time: 0.0876, Forward time: 0.9479, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.5133 [2020-12-31 11:46:06,975][root][INFO] - ===> Epoch[1313](6565/5): Loss -0.7065, Sem -0.7065, dir -0.9703, norm 0.2263 LR: 5.957e-02 Score 98.751 Data time: 0.0915, Forward time: 0.9479, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.2970 [2020-12-31 11:46:19,104][root][INFO] - ===> Epoch[1314](6570/5): Loss -0.7099, Sem -0.7099, dir -0.9705, norm 0.2234 LR: 5.954e-02 Score 98.753 Data time: 0.1442, Forward time: 0.9479, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.3409 [2020-12-31 11:46:30,499][root][INFO] - ===> Epoch[1315](6575/5): Loss -0.7052, Sem -0.7052, dir -0.9701, norm 0.2273 LR: 5.950e-02 Score 98.749 Data time: 0.1027, Forward time: 0.9479, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.1913 [2020-12-31 11:46:41,900][root][INFO] - ===> Epoch[1316](6580/5): Loss -0.6949, Sem -0.6949, dir -0.9686, norm 0.2348 LR: 5.947e-02 Score 98.695 Data time: 0.1638, Forward time: 0.9480, Backward time: 1.1959, DDP time: 0.0051, Total iter time: 2.1835 [2020-12-31 11:46:54,425][root][INFO] - ===> Epoch[1317](6585/5): Loss -0.7143, Sem -0.7143, dir -0.9705, norm 0.2195 LR: 5.944e-02 Score 98.774 Data time: 0.0968, Forward time: 0.9479, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.4102 [2020-12-31 11:47:06,288][root][INFO] - ===> Epoch[1318](6590/5): Loss -0.7110, Sem -0.7110, dir -0.9701, norm 0.2195 LR: 5.941e-02 Score 98.680 Data time: 0.0951, Forward time: 0.9478, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.2900 [2020-12-31 11:47:18,403][root][INFO] - ===> Epoch[1319](6595/5): Loss -0.7126, Sem -0.7126, dir -0.9706, norm 0.2205 LR: 5.938e-02 Score 98.748 Data time: 0.1127, Forward time: 0.9480, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.3297 [2020-12-31 11:47:30,244][root][INFO] - ===> Epoch[1320](6600/5): Loss -0.7158, Sem -0.7158, dir -0.9706, norm 0.2182 LR: 5.935e-02 Score 98.771 Data time: 0.0957, Forward time: 0.9480, Backward time: 1.1959, DDP time: 0.0051, Total iter time: 2.2816 [2020-12-31 11:47:30,247][root][INFO] - ===> Start testing [2020-12-31 11:48:08,276][root][INFO] - 68/68: Data time: 0.0024, Iter time: 0.1172 Loss 0.294 (AVG: 0.520) Score 92.642 (AVG: 89.576) mIOU 67.983 mAP 83.060 mAcc 75.399 IOU: 59.937 0.000 81.054 72.496 92.761 89.066 34.343 74.595 98.417 61.113 79.288 84.872 55.839 mAP: 80.679 7.305 93.073 85.974 98.836 98.360 64.208 94.239 99.837 80.063 92.978 94.470 89.758 mAcc: 79.899 0.000 87.780 79.293 94.144 95.928 41.929 88.531 99.269 67.956 90.515 95.184 59.757 [2020-12-31 11:48:08,277][root][INFO] - Finished test. Elapsed time: 38.0280 [2020-12-31 11:48:11,298][root][INFO] - [2020-12-31 11:48:11,298][root][INFO] - ################################################################ [2020-12-31 11:48:11,298][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 11:48:11,298][root][INFO] - ################################################################ [2020-12-31 11:48:11,298][root][INFO] - clutter : nan nan nan [2020-12-31 11:48:11,298][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 11:48:11,298][root][INFO] - board : 0.766 0.826 0.874 [2020-12-31 11:48:11,298][root][INFO] - bookcase : 0.191 0.304 0.493 [2020-12-31 11:48:11,298][root][INFO] - ceiling : 0.594 0.628 0.679 [2020-12-31 11:48:11,298][root][INFO] - chair : 0.803 0.883 0.907 [2020-12-31 11:48:11,298][root][INFO] - column : 0.184 0.288 0.400 [2020-12-31 11:48:11,298][root][INFO] - door : 0.604 0.794 0.892 [2020-12-31 11:48:11,299][root][INFO] - floor : 0.922 0.931 0.961 [2020-12-31 11:48:11,299][root][INFO] - sofa : 0.301 0.545 0.727 [2020-12-31 11:48:11,299][root][INFO] - table : 0.165 0.338 0.634 [2020-12-31 11:48:11,299][root][INFO] - wall : 0.461 0.671 0.775 [2020-12-31 11:48:11,299][root][INFO] - window : 0.458 0.687 0.743 [2020-12-31 11:48:11,299][root][INFO] - ---------------------------------------------------------------- [2020-12-31 11:48:11,299][root][INFO] - average : 0.454 0.575 0.674 [2020-12-31 11:48:11,299][root][INFO] - [2020-12-31 11:48:11,458][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 11:48:12,140][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter6600.pth [2020-12-31 11:48:23,983][root][INFO] - ===> Epoch[1321](6605/5): Loss -0.7111, Sem -0.7111, dir -0.9709, norm 0.2230 LR: 5.931e-02 Score 98.756 Data time: 0.1961, Forward time: 0.9481, Backward time: 1.1958, DDP time: 0.0051, Total iter time: 2.2646 [2020-12-31 11:48:35,577][root][INFO] - ===> Epoch[1322](6610/5): Loss -0.7048, Sem -0.7048, dir -0.9699, norm 0.2282 LR: 5.928e-02 Score 98.760 Data time: 0.1057, Forward time: 0.9481, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.2333 [2020-12-31 11:48:47,897][root][INFO] - ===> Epoch[1323](6615/5): Loss -0.7121, Sem -0.7121, dir -0.9705, norm 0.2214 LR: 5.925e-02 Score 98.771 Data time: 0.0955, Forward time: 0.9481, Backward time: 1.1958, DDP time: 0.0051, Total iter time: 2.3761 [2020-12-31 11:48:59,331][root][INFO] - ===> Epoch[1324](6620/5): Loss -0.6946, Sem -0.6946, dir -0.9691, norm 0.2353 LR: 5.922e-02 Score 98.706 Data time: 0.1151, Forward time: 0.9481, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.1808 [2020-12-31 11:49:11,763][root][INFO] - ===> Epoch[1325](6625/5): Loss -0.7117, Sem -0.7117, dir -0.9706, norm 0.2226 LR: 5.919e-02 Score 98.779 Data time: 0.0988, Forward time: 0.9481, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.3829 [2020-12-31 11:49:23,495][root][INFO] - ===> Epoch[1326](6630/5): Loss -0.7145, Sem -0.7145, dir -0.9706, norm 0.2186 LR: 5.915e-02 Score 98.741 Data time: 0.1266, Forward time: 0.9482, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.2440 [2020-12-31 11:49:35,445][root][INFO] - ===> Epoch[1327](6635/5): Loss -0.7209, Sem -0.7209, dir -0.9715, norm 0.2134 LR: 5.912e-02 Score 98.750 Data time: 0.1147, Forward time: 0.9482, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.3016 [2020-12-31 11:49:48,145][root][INFO] - ===> Epoch[1328](6640/5): Loss -0.7135, Sem -0.7135, dir -0.9700, norm 0.2191 LR: 5.909e-02 Score 98.751 Data time: 0.1338, Forward time: 0.9482, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.4480 [2020-12-31 11:50:00,517][root][INFO] - ===> Epoch[1329](6645/5): Loss -0.7169, Sem -0.7169, dir -0.9714, norm 0.2172 LR: 5.906e-02 Score 98.766 Data time: 0.1038, Forward time: 0.9483, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.3705 [2020-12-31 11:50:12,842][root][INFO] - ===> Epoch[1330](6650/5): Loss -0.7121, Sem -0.7121, dir -0.9705, norm 0.2194 LR: 5.903e-02 Score 98.695 Data time: 0.1361, Forward time: 0.9483, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.3693 [2020-12-31 11:50:24,294][root][INFO] - ===> Epoch[1331](6655/5): Loss -0.7074, Sem -0.7074, dir -0.9697, norm 0.2250 LR: 5.900e-02 Score 98.763 Data time: 0.0690, Forward time: 0.9482, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.2199 [2020-12-31 11:50:36,381][root][INFO] - ===> Epoch[1332](6660/5): Loss -0.7040, Sem -0.7040, dir -0.9696, norm 0.2264 LR: 5.896e-02 Score 98.692 Data time: 0.1385, Forward time: 0.9482, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.3160 [2020-12-31 11:50:48,860][root][INFO] - ===> Epoch[1333](6665/5): Loss -0.7058, Sem -0.7058, dir -0.9696, norm 0.2246 LR: 5.893e-02 Score 98.687 Data time: 0.1814, Forward time: 0.9482, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.4043 [2020-12-31 11:51:01,374][root][INFO] - ===> Epoch[1334](6670/5): Loss -0.7106, Sem -0.7106, dir -0.9702, norm 0.2232 LR: 5.890e-02 Score 98.774 Data time: 0.1213, Forward time: 0.9483, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.4009 [2020-12-31 11:51:13,554][root][INFO] - ===> Epoch[1335](6675/5): Loss -0.7064, Sem -0.7064, dir -0.9692, norm 0.2254 LR: 5.887e-02 Score 98.749 Data time: 0.1170, Forward time: 0.9483, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.3342 [2020-12-31 11:51:25,824][root][INFO] - ===> Epoch[1336](6680/5): Loss -0.7022, Sem -0.7022, dir -0.9708, norm 0.2339 LR: 5.884e-02 Score 98.843 Data time: 0.0863, Forward time: 0.9483, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.3588 [2020-12-31 11:51:37,267][root][INFO] - ===> Epoch[1337](6685/5): Loss -0.7134, Sem -0.7134, dir -0.9704, norm 0.2188 LR: 5.881e-02 Score 98.726 Data time: 0.0948, Forward time: 0.9482, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.1989 [2020-12-31 11:51:48,668][root][INFO] - ===> Epoch[1338](6690/5): Loss -0.7080, Sem -0.7080, dir -0.9710, norm 0.2250 LR: 5.877e-02 Score 98.730 Data time: 0.0945, Forward time: 0.9481, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.1958 [2020-12-31 11:52:00,688][root][INFO] - ===> Epoch[1339](6695/5): Loss -0.7144, Sem -0.7144, dir -0.9710, norm 0.2216 LR: 5.874e-02 Score 98.829 Data time: 0.1481, Forward time: 0.9482, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.3060 [2020-12-31 11:52:13,181][root][INFO] - ===> Epoch[1340](6700/5): Loss -0.7080, Sem -0.7080, dir -0.9704, norm 0.2239 LR: 5.871e-02 Score 98.725 Data time: 0.0793, Forward time: 0.9481, Backward time: 1.1958, DDP time: 0.0051, Total iter time: 2.4151 [2020-12-31 11:52:24,946][root][INFO] - ===> Epoch[1341](6705/5): Loss -0.7122, Sem -0.7122, dir -0.9709, norm 0.2214 LR: 5.868e-02 Score 98.743 Data time: 0.0950, Forward time: 0.9480, Backward time: 1.1958, DDP time: 0.0051, Total iter time: 2.2650 [2020-12-31 11:52:36,651][root][INFO] - ===> Epoch[1342](6710/5): Loss -0.7118, Sem -0.7118, dir -0.9709, norm 0.2218 LR: 5.865e-02 Score 98.750 Data time: 0.1039, Forward time: 0.9480, Backward time: 1.1958, DDP time: 0.0051, Total iter time: 2.2626 [2020-12-31 11:52:49,683][root][INFO] - ===> Epoch[1343](6715/5): Loss -0.7138, Sem -0.7138, dir -0.9708, norm 0.2194 LR: 5.861e-02 Score 98.740 Data time: 0.1164, Forward time: 0.9481, Backward time: 1.1958, DDP time: 0.0051, Total iter time: 2.5118 [2020-12-31 11:53:02,444][root][INFO] - ===> Epoch[1344](6720/5): Loss -0.7160, Sem -0.7160, dir -0.9708, norm 0.2196 LR: 5.858e-02 Score 98.818 Data time: 0.0739, Forward time: 0.9480, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.4695 [2020-12-31 11:53:13,574][root][INFO] - ===> Epoch[1345](6725/5): Loss -0.7177, Sem -0.7177, dir -0.9703, norm 0.2155 LR: 5.855e-02 Score 98.747 Data time: 0.0863, Forward time: 0.9479, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.1291 [2020-12-31 11:53:25,461][root][INFO] - ===> Epoch[1346](6730/5): Loss -0.7160, Sem -0.7160, dir -0.9710, norm 0.2195 LR: 5.852e-02 Score 98.816 Data time: 0.1410, Forward time: 0.9479, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.2837 [2020-12-31 11:53:36,917][root][INFO] - ===> Epoch[1347](6735/5): Loss -0.7073, Sem -0.7073, dir -0.9700, norm 0.2229 LR: 5.849e-02 Score 98.696 Data time: 0.0719, Forward time: 0.9477, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.2072 [2020-12-31 11:53:49,207][root][INFO] - ===> Epoch[1348](6740/5): Loss -0.7006, Sem -0.7006, dir -0.9701, norm 0.2330 LR: 5.845e-02 Score 98.788 Data time: 0.1187, Forward time: 0.9477, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.3517 [2020-12-31 11:54:00,846][root][INFO] - ===> Epoch[1349](6745/5): Loss -0.7027, Sem -0.7027, dir -0.9693, norm 0.2265 LR: 5.842e-02 Score 98.668 Data time: 0.1682, Forward time: 0.9477, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.2199 [2020-12-31 11:54:13,036][root][INFO] - ===> Epoch[1350](6750/5): Loss -0.7193, Sem -0.7193, dir -0.9706, norm 0.2151 LR: 5.839e-02 Score 98.782 Data time: 0.0845, Forward time: 0.9477, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.3493 [2020-12-31 11:54:24,873][root][INFO] - ===> Epoch[1351](6755/5): Loss -0.7036, Sem -0.7036, dir -0.9704, norm 0.2293 LR: 5.836e-02 Score 98.747 Data time: 0.0980, Forward time: 0.9476, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.2709 [2020-12-31 11:54:36,308][root][INFO] - ===> Epoch[1352](6760/5): Loss -0.7106, Sem -0.7106, dir -0.9702, norm 0.2218 LR: 5.833e-02 Score 98.744 Data time: 0.1023, Forward time: 0.9476, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.1997 [2020-12-31 11:54:48,346][root][INFO] - ===> Epoch[1353](6765/5): Loss -0.7085, Sem -0.7085, dir -0.9708, norm 0.2255 LR: 5.830e-02 Score 98.769 Data time: 0.0788, Forward time: 0.9475, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.3202 [2020-12-31 11:55:00,349][root][INFO] - ===> Epoch[1354](6770/5): Loss -0.7124, Sem -0.7124, dir -0.9717, norm 0.2228 LR: 5.826e-02 Score 98.782 Data time: 0.0877, Forward time: 0.9474, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.3027 [2020-12-31 11:55:12,361][root][INFO] - ===> Epoch[1355](6775/5): Loss -0.7108, Sem -0.7108, dir -0.9704, norm 0.2215 LR: 5.823e-02 Score 98.720 Data time: 0.0742, Forward time: 0.9473, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.3157 [2020-12-31 11:55:24,822][root][INFO] - ===> Epoch[1356](6780/5): Loss -0.7106, Sem -0.7106, dir -0.9718, norm 0.2241 LR: 5.820e-02 Score 98.755 Data time: 0.1189, Forward time: 0.9473, Backward time: 1.1964, DDP time: 0.0051, Total iter time: 2.3998 [2020-12-31 11:55:36,768][root][INFO] - ===> Epoch[1357](6785/5): Loss -0.7026, Sem -0.7026, dir -0.9703, norm 0.2302 LR: 5.817e-02 Score 98.761 Data time: 0.1053, Forward time: 0.9473, Backward time: 1.1964, DDP time: 0.0051, Total iter time: 2.2839 [2020-12-31 11:55:48,190][root][INFO] - ===> Epoch[1358](6790/5): Loss -0.7093, Sem -0.7093, dir -0.9703, norm 0.2221 LR: 5.814e-02 Score 98.711 Data time: 0.1074, Forward time: 0.9473, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.1968 [2020-12-31 11:55:59,399][root][INFO] - ===> Epoch[1359](6795/5): Loss -0.7081, Sem -0.7081, dir -0.9717, norm 0.2258 LR: 5.810e-02 Score 98.747 Data time: 0.0731, Forward time: 0.9472, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.1660 [2020-12-31 11:56:12,233][root][INFO] - ===> Epoch[1360](6800/5): Loss -0.7129, Sem -0.7129, dir -0.9699, norm 0.2196 LR: 5.807e-02 Score 98.753 Data time: 0.1266, Forward time: 0.9472, Backward time: 1.1964, DDP time: 0.0051, Total iter time: 2.4701 [2020-12-31 11:56:12,235][root][INFO] - ===> Start testing [2020-12-31 11:56:50,716][root][INFO] - 68/68: Data time: 0.0020, Iter time: 0.1070 Loss 0.385 (AVG: 0.533) Score 90.370 (AVG: 89.451) mIOU 68.482 mAP 83.041 mAcc 75.760 IOU: 59.414 0.000 81.201 71.675 93.220 90.322 30.249 73.972 98.411 70.019 79.995 84.737 57.049 mAP: 80.096 8.366 92.838 85.303 98.652 97.933 59.951 93.423 99.845 85.905 92.932 93.748 90.538 mAcc: 80.907 0.000 86.632 77.929 94.446 94.606 37.093 86.895 98.989 79.400 90.861 95.074 62.042 [2020-12-31 11:56:50,718][root][INFO] - Finished test. Elapsed time: 38.4800 [2020-12-31 11:56:53,857][root][INFO] - [2020-12-31 11:56:53,857][root][INFO] - ################################################################ [2020-12-31 11:56:53,857][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 11:56:53,857][root][INFO] - ################################################################ [2020-12-31 11:56:53,857][root][INFO] - clutter : nan nan nan [2020-12-31 11:56:53,857][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 11:56:53,857][root][INFO] - board : 0.743 0.807 0.903 [2020-12-31 11:56:53,857][root][INFO] - bookcase : 0.188 0.306 0.492 [2020-12-31 11:56:53,857][root][INFO] - ceiling : 0.521 0.527 0.627 [2020-12-31 11:56:53,857][root][INFO] - chair : 0.772 0.865 0.892 [2020-12-31 11:56:53,857][root][INFO] - column : 0.149 0.206 0.350 [2020-12-31 11:56:53,857][root][INFO] - door : 0.504 0.696 0.887 [2020-12-31 11:56:53,857][root][INFO] - floor : 0.939 0.954 0.954 [2020-12-31 11:56:53,857][root][INFO] - sofa : 0.478 0.727 0.727 [2020-12-31 11:56:53,857][root][INFO] - table : 0.168 0.348 0.639 [2020-12-31 11:56:53,857][root][INFO] - wall : 0.426 0.620 0.731 [2020-12-31 11:56:53,857][root][INFO] - window : 0.509 0.748 0.765 [2020-12-31 11:56:53,858][root][INFO] - ---------------------------------------------------------------- [2020-12-31 11:56:53,858][root][INFO] - average : 0.450 0.567 0.664 [2020-12-31 11:56:53,858][root][INFO] - [2020-12-31 11:56:54,008][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 11:56:54,675][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2020-12-31 11:56:54,676][root][INFO] - Current best mIoU: 68.482 at iter 1 [2020-12-31 11:56:55,262][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter6800.pth [2020-12-31 11:57:06,244][root][INFO] - ===> Epoch[1361](6805/5): Loss -0.7180, Sem -0.7180, dir -0.9717, norm 0.2177 LR: 5.804e-02 Score 98.795 Data time: 0.0933, Forward time: 0.9471, Backward time: 1.1964, DDP time: 0.0051, Total iter time: 2.0995 [2020-12-31 11:57:18,020][root][INFO] - ===> Epoch[1362](6810/5): Loss -0.7124, Sem -0.7124, dir -0.9713, norm 0.2227 LR: 5.801e-02 Score 98.783 Data time: 0.1745, Forward time: 0.9472, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.2594 [2020-12-31 11:57:30,176][root][INFO] - ===> Epoch[1363](6815/5): Loss -0.7117, Sem -0.7117, dir -0.9703, norm 0.2219 LR: 5.798e-02 Score 98.781 Data time: 0.1108, Forward time: 0.9471, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.3313 [2020-12-31 11:57:41,738][root][INFO] - ===> Epoch[1364](6820/5): Loss -0.7175, Sem -0.7175, dir -0.9715, norm 0.2190 LR: 5.795e-02 Score 98.831 Data time: 0.1003, Forward time: 0.9471, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.2131 [2020-12-31 11:57:54,091][root][INFO] - ===> Epoch[1365](6825/5): Loss -0.7069, Sem -0.7069, dir -0.9704, norm 0.2248 LR: 5.791e-02 Score 98.701 Data time: 0.0924, Forward time: 0.9471, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.3804 [2020-12-31 11:58:07,282][root][INFO] - ===> Epoch[1366](6830/5): Loss -0.7206, Sem -0.7206, dir -0.9716, norm 0.2144 LR: 5.788e-02 Score 98.786 Data time: 0.1221, Forward time: 0.9472, Backward time: 1.1964, DDP time: 0.0051, Total iter time: 2.5434 [2020-12-31 11:58:19,205][root][INFO] - ===> Epoch[1367](6835/5): Loss -0.7093, Sem -0.7093, dir -0.9708, norm 0.2210 LR: 5.785e-02 Score 98.742 Data time: 0.0922, Forward time: 0.9471, Backward time: 1.1964, DDP time: 0.0051, Total iter time: 2.2940 [2020-12-31 11:58:31,199][root][INFO] - ===> Epoch[1368](6840/5): Loss -0.6976, Sem -0.6976, dir -0.9692, norm 0.2303 LR: 5.782e-02 Score 98.638 Data time: 0.1425, Forward time: 0.9471, Backward time: 1.1964, DDP time: 0.0051, Total iter time: 2.3025 [2020-12-31 11:58:44,042][root][INFO] - ===> Epoch[1369](6845/5): Loss -0.7099, Sem -0.7099, dir -0.9708, norm 0.2243 LR: 5.779e-02 Score 98.792 Data time: 0.0702, Forward time: 0.9470, Backward time: 1.1966, DDP time: 0.0051, Total iter time: 2.4862 [2020-12-31 11:58:55,557][root][INFO] - ===> Epoch[1370](6850/5): Loss -0.7046, Sem -0.7046, dir -0.9704, norm 0.2262 LR: 5.775e-02 Score 98.698 Data time: 0.0862, Forward time: 0.9469, Backward time: 1.1966, DDP time: 0.0051, Total iter time: 2.2099 [2020-12-31 11:59:09,687][root][INFO] - ===> Epoch[1371](6855/5): Loss -0.7140, Sem -0.7140, dir -0.9700, norm 0.2174 LR: 5.772e-02 Score 98.709 Data time: 0.1024, Forward time: 0.9469, Backward time: 1.1969, DDP time: 0.0051, Total iter time: 2.7261 [2020-12-31 11:59:22,102][root][INFO] - ===> Epoch[1372](6860/5): Loss -0.7103, Sem -0.7103, dir -0.9711, norm 0.2221 LR: 5.769e-02 Score 98.709 Data time: 0.1647, Forward time: 0.9470, Backward time: 1.1969, DDP time: 0.0051, Total iter time: 2.3857 [2020-12-31 11:59:34,444][root][INFO] - ===> Epoch[1373](6865/5): Loss -0.7100, Sem -0.7100, dir -0.9701, norm 0.2208 LR: 5.766e-02 Score 98.698 Data time: 0.1188, Forward time: 0.9470, Backward time: 1.1969, DDP time: 0.0051, Total iter time: 2.3755 [2020-12-31 11:59:45,734][root][INFO] - ===> Epoch[1374](6870/5): Loss -0.7224, Sem -0.7224, dir -0.9712, norm 0.2120 LR: 5.763e-02 Score 98.755 Data time: 0.1084, Forward time: 0.9470, Backward time: 1.1968, DDP time: 0.0051, Total iter time: 2.1623 [2020-12-31 11:59:57,564][root][INFO] - ===> Epoch[1375](6875/5): Loss -0.7162, Sem -0.7162, dir -0.9703, norm 0.2160 LR: 5.759e-02 Score 98.730 Data time: 0.1591, Forward time: 0.9470, Backward time: 1.1967, DDP time: 0.0051, Total iter time: 2.2780 [2020-12-31 12:00:09,580][root][INFO] - ===> Epoch[1376](6880/5): Loss -0.7083, Sem -0.7083, dir -0.9699, norm 0.2218 LR: 5.756e-02 Score 98.688 Data time: 0.1047, Forward time: 0.9471, Backward time: 1.1967, DDP time: 0.0051, Total iter time: 2.3101 [2020-12-31 12:00:20,588][root][INFO] - ===> Epoch[1377](6885/5): Loss -0.7099, Sem -0.7099, dir -0.9709, norm 0.2235 LR: 5.753e-02 Score 98.742 Data time: 0.1074, Forward time: 0.9470, Backward time: 1.1965, DDP time: 0.0051, Total iter time: 2.0732 [2020-12-31 12:00:32,632][root][INFO] - ===> Epoch[1378](6890/5): Loss -0.7040, Sem -0.7040, dir -0.9700, norm 0.2289 LR: 5.750e-02 Score 98.768 Data time: 0.1264, Forward time: 0.9470, Backward time: 1.1965, DDP time: 0.0051, Total iter time: 2.2966 [2020-12-31 12:00:44,677][root][INFO] - ===> Epoch[1379](6895/5): Loss -0.7033, Sem -0.7033, dir -0.9706, norm 0.2301 LR: 5.747e-02 Score 98.755 Data time: 0.1269, Forward time: 0.9470, Backward time: 1.1965, DDP time: 0.0051, Total iter time: 2.3093 [2020-12-31 12:00:56,738][root][INFO] - ===> Epoch[1380](6900/5): Loss -0.7153, Sem -0.7153, dir -0.9726, norm 0.2231 LR: 5.743e-02 Score 98.863 Data time: 0.0784, Forward time: 0.9469, Backward time: 1.1966, DDP time: 0.0051, Total iter time: 2.3409 [2020-12-31 12:01:09,271][root][INFO] - ===> Epoch[1381](6905/5): Loss -0.7191, Sem -0.7191, dir -0.9710, norm 0.2135 LR: 5.740e-02 Score 98.716 Data time: 0.0699, Forward time: 0.9468, Backward time: 1.1968, DDP time: 0.0051, Total iter time: 2.4312 [2020-12-31 12:01:21,195][root][INFO] - ===> Epoch[1382](6910/5): Loss -0.7228, Sem -0.7228, dir -0.9728, norm 0.2146 LR: 5.737e-02 Score 98.814 Data time: 0.1014, Forward time: 0.9468, Backward time: 1.1968, DDP time: 0.0051, Total iter time: 2.2874 [2020-12-31 12:01:32,510][root][INFO] - ===> Epoch[1383](6915/5): Loss -0.7149, Sem -0.7149, dir -0.9710, norm 0.2198 LR: 5.734e-02 Score 98.784 Data time: 0.1010, Forward time: 0.9469, Backward time: 1.1967, DDP time: 0.0051, Total iter time: 2.1740 [2020-12-31 12:01:45,381][root][INFO] - ===> Epoch[1384](6920/5): Loss -0.7106, Sem -0.7106, dir -0.9712, norm 0.2235 LR: 5.731e-02 Score 98.775 Data time: 0.1002, Forward time: 0.9469, Backward time: 1.1968, DDP time: 0.0051, Total iter time: 2.4782 [2020-12-31 12:01:56,919][root][INFO] - ===> Epoch[1385](6925/5): Loss -0.7153, Sem -0.7153, dir -0.9712, norm 0.2185 LR: 5.728e-02 Score 98.750 Data time: 0.1210, Forward time: 0.9469, Backward time: 1.1967, DDP time: 0.0051, Total iter time: 2.2270 [2020-12-31 12:02:08,448][root][INFO] - ===> Epoch[1386](6930/5): Loss -0.7137, Sem -0.7137, dir -0.9701, norm 0.2196 LR: 5.724e-02 Score 98.764 Data time: 0.0845, Forward time: 0.9469, Backward time: 1.1967, DDP time: 0.0051, Total iter time: 2.2202 [2020-12-31 12:02:19,869][root][INFO] - ===> Epoch[1387](6935/5): Loss -0.7192, Sem -0.7192, dir -0.9721, norm 0.2175 LR: 5.721e-02 Score 98.814 Data time: 0.0882, Forward time: 0.9468, Backward time: 1.1966, DDP time: 0.0051, Total iter time: 2.1914 [2020-12-31 12:02:31,380][root][INFO] - ===> Epoch[1388](6940/5): Loss -0.7252, Sem -0.7252, dir -0.9725, norm 0.2133 LR: 5.718e-02 Score 98.853 Data time: 0.1144, Forward time: 0.9468, Backward time: 1.1965, DDP time: 0.0051, Total iter time: 2.2049 [2020-12-31 12:02:43,267][root][INFO] - ===> Epoch[1389](6945/5): Loss -0.7156, Sem -0.7156, dir -0.9709, norm 0.2182 LR: 5.715e-02 Score 98.759 Data time: 0.1577, Forward time: 0.9469, Backward time: 1.1964, DDP time: 0.0051, Total iter time: 2.2895 [2020-12-31 12:02:54,952][root][INFO] - ===> Epoch[1390](6950/5): Loss -0.7103, Sem -0.7103, dir -0.9702, norm 0.2206 LR: 5.712e-02 Score 98.674 Data time: 0.0930, Forward time: 0.9469, Backward time: 1.1964, DDP time: 0.0051, Total iter time: 2.2428 [2020-12-31 12:03:07,041][root][INFO] - ===> Epoch[1391](6955/5): Loss -0.7161, Sem -0.7161, dir -0.9712, norm 0.2158 LR: 5.708e-02 Score 98.687 Data time: 0.1999, Forward time: 0.9470, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.3111 [2020-12-31 12:03:18,534][root][INFO] - ===> Epoch[1392](6960/5): Loss -0.7057, Sem -0.7057, dir -0.9702, norm 0.2207 LR: 5.705e-02 Score 98.574 Data time: 0.1314, Forward time: 0.9469, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.2144 [2020-12-31 12:03:29,935][root][INFO] - ===> Epoch[1393](6965/5): Loss -0.7065, Sem -0.7065, dir -0.9701, norm 0.2226 LR: 5.702e-02 Score 98.629 Data time: 0.0736, Forward time: 0.9469, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.1916 [2020-12-31 12:03:41,823][root][INFO] - ===> Epoch[1394](6970/5): Loss -0.7097, Sem -0.7097, dir -0.9699, norm 0.2193 LR: 5.699e-02 Score 98.633 Data time: 0.1245, Forward time: 0.9468, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.2817 [2020-12-31 12:03:53,925][root][INFO] - ===> Epoch[1395](6975/5): Loss -0.7097, Sem -0.7097, dir -0.9692, norm 0.2196 LR: 5.696e-02 Score 98.694 Data time: 0.1607, Forward time: 0.9470, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.3353 [2020-12-31 12:04:06,028][root][INFO] - ===> Epoch[1396](6980/5): Loss -0.7094, Sem -0.7094, dir -0.9699, norm 0.2220 LR: 5.692e-02 Score 98.724 Data time: 0.1400, Forward time: 0.9470, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.3297 [2020-12-31 12:04:17,022][root][INFO] - ===> Epoch[1397](6985/5): Loss -0.7053, Sem -0.7053, dir -0.9693, norm 0.2229 LR: 5.689e-02 Score 98.646 Data time: 0.0979, Forward time: 0.9469, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.1103 [2020-12-31 12:04:29,985][root][INFO] - ===> Epoch[1398](6990/5): Loss -0.7094, Sem -0.7094, dir -0.9698, norm 0.2219 LR: 5.686e-02 Score 98.709 Data time: 0.0883, Forward time: 0.9469, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.5067 [2020-12-31 12:04:42,513][root][INFO] - ===> Epoch[1399](6995/5): Loss -0.7186, Sem -0.7186, dir -0.9718, norm 0.2174 LR: 5.683e-02 Score 98.807 Data time: 0.1052, Forward time: 0.9469, Backward time: 1.1963, DDP time: 0.0051, Total iter time: 2.4135 [2020-12-31 12:04:54,038][root][INFO] - ===> Epoch[1400](7000/5): Loss -0.7184, Sem -0.7184, dir -0.9697, norm 0.2138 LR: 5.680e-02 Score 98.757 Data time: 0.1772, Forward time: 0.9469, Backward time: 1.1962, DDP time: 0.0051, Total iter time: 2.2097 [2020-12-31 12:04:54,040][root][INFO] - ===> Start testing [2020-12-31 12:05:31,232][root][INFO] - 68/68: Data time: 0.0023, Iter time: 0.1110 Loss 0.607 (AVG: 0.509) Score 89.204 (AVG: 89.588) mIOU 68.407 mAP 83.498 mAcc 76.255 IOU: 60.365 0.000 78.886 72.931 93.381 91.313 34.611 72.499 98.479 68.827 80.421 84.541 53.035 mAP: 80.801 4.537 94.284 85.909 98.758 98.573 63.715 93.532 99.843 89.696 93.232 94.076 88.515 mAcc: 80.122 0.000 88.984 79.716 94.793 96.589 42.633 84.958 99.304 81.097 90.156 94.896 58.061 [2020-12-31 12:05:31,234][root][INFO] - Finished test. Elapsed time: 37.1911 [2020-12-31 12:05:34,476][root][INFO] - [2020-12-31 12:05:34,476][root][INFO] - ################################################################ [2020-12-31 12:05:34,476][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 12:05:34,476][root][INFO] - ################################################################ [2020-12-31 12:05:34,477][root][INFO] - clutter : nan nan nan [2020-12-31 12:05:34,477][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 12:05:34,477][root][INFO] - board : 0.735 0.785 0.870 [2020-12-31 12:05:34,477][root][INFO] - bookcase : 0.217 0.352 0.510 [2020-12-31 12:05:34,477][root][INFO] - ceiling : 0.670 0.738 0.789 [2020-12-31 12:05:34,477][root][INFO] - chair : 0.772 0.879 0.898 [2020-12-31 12:05:34,477][root][INFO] - column : 0.147 0.219 0.385 [2020-12-31 12:05:34,477][root][INFO] - door : 0.509 0.682 0.852 [2020-12-31 12:05:34,477][root][INFO] - floor : 0.945 0.978 0.978 [2020-12-31 12:05:34,477][root][INFO] - sofa : 0.503 0.636 0.809 [2020-12-31 12:05:34,477][root][INFO] - table : 0.202 0.396 0.694 [2020-12-31 12:05:34,477][root][INFO] - wall : 0.453 0.680 0.796 [2020-12-31 12:05:34,477][root][INFO] - window : 0.457 0.690 0.724 [2020-12-31 12:05:34,477][root][INFO] - ---------------------------------------------------------------- [2020-12-31 12:05:34,477][root][INFO] - average : 0.467 0.586 0.692 [2020-12-31 12:05:34,477][root][INFO] - [2020-12-31 12:05:34,647][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 12:05:35,230][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter7000.pth [2020-12-31 12:05:45,926][root][INFO] - ===> Epoch[1401](7005/5): Loss -0.7192, Sem -0.7192, dir -0.9725, norm 0.2165 LR: 5.676e-02 Score 98.778 Data time: 0.0890, Forward time: 0.9468, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.0473 [2020-12-31 12:05:57,895][root][INFO] - ===> Epoch[1402](7010/5): Loss -0.7189, Sem -0.7189, dir -0.9717, norm 0.2171 LR: 5.673e-02 Score 98.817 Data time: 0.0986, Forward time: 0.9468, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.3056 [2020-12-31 12:06:09,966][root][INFO] - ===> Epoch[1403](7015/5): Loss -0.7162, Sem -0.7162, dir -0.9704, norm 0.2172 LR: 5.670e-02 Score 98.776 Data time: 0.1315, Forward time: 0.9469, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.3169 [2020-12-31 12:06:21,601][root][INFO] - ===> Epoch[1404](7020/5): Loss -0.7215, Sem -0.7215, dir -0.9714, norm 0.2139 LR: 5.667e-02 Score 98.799 Data time: 0.0953, Forward time: 0.9469, Backward time: 1.1959, DDP time: 0.0051, Total iter time: 2.2432 [2020-12-31 12:06:34,434][root][INFO] - ===> Epoch[1405](7025/5): Loss -0.7190, Sem -0.7190, dir -0.9715, norm 0.2158 LR: 5.664e-02 Score 98.774 Data time: 0.1142, Forward time: 0.9470, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.4729 [2020-12-31 12:06:46,265][root][INFO] - ===> Epoch[1406](7030/5): Loss -0.7144, Sem -0.7144, dir -0.9709, norm 0.2207 LR: 5.660e-02 Score 98.807 Data time: 0.1079, Forward time: 0.9470, Backward time: 1.1959, DDP time: 0.0051, Total iter time: 2.2647 [2020-12-31 12:06:58,478][root][INFO] - ===> Epoch[1407](7035/5): Loss -0.7273, Sem -0.7273, dir -0.9727, norm 0.2109 LR: 5.657e-02 Score 98.845 Data time: 0.0766, Forward time: 0.9470, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.3470 [2020-12-31 12:07:09,927][root][INFO] - ===> Epoch[1408](7040/5): Loss -0.7213, Sem -0.7213, dir -0.9712, norm 0.2139 LR: 5.654e-02 Score 98.788 Data time: 0.1157, Forward time: 0.9469, Backward time: 1.1959, DDP time: 0.0051, Total iter time: 2.1938 [2020-12-31 12:07:22,306][root][INFO] - ===> Epoch[1409](7045/5): Loss -0.7106, Sem -0.7106, dir -0.9702, norm 0.2213 LR: 5.651e-02 Score 98.733 Data time: 0.0964, Forward time: 0.9469, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.3682 [2020-12-31 12:07:33,373][root][INFO] - ===> Epoch[1410](7050/5): Loss -0.7151, Sem -0.7151, dir -0.9717, norm 0.2191 LR: 5.648e-02 Score 98.744 Data time: 0.1011, Forward time: 0.9468, Backward time: 1.1959, DDP time: 0.0051, Total iter time: 2.1233 [2020-12-31 12:07:46,781][root][INFO] - ===> Epoch[1411](7055/5): Loss -0.7119, Sem -0.7119, dir -0.9711, norm 0.2216 LR: 5.645e-02 Score 98.750 Data time: 0.1386, Forward time: 0.9468, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.5948 [2020-12-31 12:07:58,843][root][INFO] - ===> Epoch[1412](7060/5): Loss -0.7066, Sem -0.7066, dir -0.9709, norm 0.2273 LR: 5.641e-02 Score 98.760 Data time: 0.0963, Forward time: 0.9468, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.3153 [2020-12-31 12:08:10,570][root][INFO] - ===> Epoch[1413](7065/5): Loss -0.7071, Sem -0.7071, dir -0.9702, norm 0.2248 LR: 5.638e-02 Score 98.730 Data time: 0.0896, Forward time: 0.9468, Backward time: 1.1961, DDP time: 0.0051, Total iter time: 2.2534 [2020-12-31 12:08:22,259][root][INFO] - ===> Epoch[1414](7070/5): Loss -0.7036, Sem -0.7036, dir -0.9697, norm 0.2248 LR: 5.635e-02 Score 98.635 Data time: 0.1139, Forward time: 0.9469, Backward time: 1.1959, DDP time: 0.0051, Total iter time: 2.2447 [2020-12-31 12:08:34,098][root][INFO] - ===> Epoch[1415](7075/5): Loss -0.7041, Sem -0.7041, dir -0.9700, norm 0.2272 LR: 5.632e-02 Score 98.720 Data time: 0.0913, Forward time: 0.9469, Backward time: 1.1959, DDP time: 0.0051, Total iter time: 2.2731 [2020-12-31 12:08:45,759][root][INFO] - ===> Epoch[1416](7080/5): Loss -0.7073, Sem -0.7073, dir -0.9704, norm 0.2252 LR: 5.629e-02 Score 98.727 Data time: 0.1463, Forward time: 0.9470, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.2391 [2020-12-31 12:08:57,583][root][INFO] - ===> Epoch[1417](7085/5): Loss -0.7129, Sem -0.7129, dir -0.9711, norm 0.2217 LR: 5.625e-02 Score 98.772 Data time: 0.0960, Forward time: 0.9470, Backward time: 1.1958, DDP time: 0.0051, Total iter time: 2.2748 [2020-12-31 12:09:10,791][root][INFO] - ===> Epoch[1418](7090/5): Loss -0.7133, Sem -0.7133, dir -0.9713, norm 0.2213 LR: 5.622e-02 Score 98.774 Data time: 0.0846, Forward time: 0.9470, Backward time: 1.1960, DDP time: 0.0051, Total iter time: 2.5546 [2020-12-31 12:09:22,161][root][INFO] - ===> Epoch[1419](7095/5): Loss -0.7065, Sem -0.7065, dir -0.9706, norm 0.2265 LR: 5.619e-02 Score 98.748 Data time: 0.1039, Forward time: 0.9469, Backward time: 1.1959, DDP time: 0.0051, Total iter time: 2.1767 [2020-12-31 12:09:33,622][root][INFO] - ===> Epoch[1420](7100/5): Loss -0.7209, Sem -0.7209, dir -0.9712, norm 0.2134 LR: 5.616e-02 Score 98.767 Data time: 0.1170, Forward time: 0.9470, Backward time: 1.1958, DDP time: 0.0051, Total iter time: 2.1938 [2020-12-31 12:09:45,712][root][INFO] - ===> Epoch[1421](7105/5): Loss -0.7262, Sem -0.7262, dir -0.9720, norm 0.2097 LR: 5.613e-02 Score 98.785 Data time: 0.1234, Forward time: 0.9470, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.3237 [2020-12-31 12:09:57,974][root][INFO] - ===> Epoch[1422](7110/5): Loss -0.7273, Sem -0.7273, dir -0.9721, norm 0.2102 LR: 5.609e-02 Score 98.831 Data time: 0.0916, Forward time: 0.9470, Backward time: 1.1958, DDP time: 0.0051, Total iter time: 2.3598 [2020-12-31 12:10:09,540][root][INFO] - ===> Epoch[1423](7115/5): Loss -0.7161, Sem -0.7161, dir -0.9709, norm 0.2161 LR: 5.606e-02 Score 98.747 Data time: 0.0892, Forward time: 0.9470, Backward time: 1.1958, DDP time: 0.0051, Total iter time: 2.2288 [2020-12-31 12:10:21,500][root][INFO] - ===> Epoch[1424](7120/5): Loss -0.7234, Sem -0.7234, dir -0.9717, norm 0.2118 LR: 5.603e-02 Score 98.776 Data time: 0.1140, Forward time: 0.9470, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.3015 [2020-12-31 12:10:34,110][root][INFO] - ===> Epoch[1425](7125/5): Loss -0.7138, Sem -0.7138, dir -0.9716, norm 0.2202 LR: 5.600e-02 Score 98.750 Data time: 0.1269, Forward time: 0.9470, Backward time: 1.1958, DDP time: 0.0051, Total iter time: 2.4339 [2020-12-31 12:10:45,995][root][INFO] - ===> Epoch[1426](7130/5): Loss -0.7175, Sem -0.7175, dir -0.9709, norm 0.2160 LR: 5.597e-02 Score 98.764 Data time: 0.1004, Forward time: 0.9471, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.2831 [2020-12-31 12:10:58,332][root][INFO] - ===> Epoch[1427](7135/5): Loss -0.7144, Sem -0.7144, dir -0.9717, norm 0.2184 LR: 5.593e-02 Score 98.739 Data time: 0.1235, Forward time: 0.9471, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.3729 [2020-12-31 12:11:09,545][root][INFO] - ===> Epoch[1428](7140/5): Loss -0.7162, Sem -0.7162, dir -0.9708, norm 0.2165 LR: 5.590e-02 Score 98.738 Data time: 0.1079, Forward time: 0.9471, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.1496 [2020-12-31 12:11:21,613][root][INFO] - ===> Epoch[1429](7145/5): Loss -0.7147, Sem -0.7147, dir -0.9712, norm 0.2187 LR: 5.587e-02 Score 98.737 Data time: 0.1098, Forward time: 0.9470, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.3255 [2020-12-31 12:11:33,336][root][INFO] - ===> Epoch[1430](7150/5): Loss -0.7182, Sem -0.7182, dir -0.9717, norm 0.2168 LR: 5.584e-02 Score 98.778 Data time: 0.0965, Forward time: 0.9470, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.2581 [2020-12-31 12:11:45,807][root][INFO] - ===> Epoch[1431](7155/5): Loss -0.7119, Sem -0.7119, dir -0.9711, norm 0.2232 LR: 5.581e-02 Score 98.795 Data time: 0.1227, Forward time: 0.9471, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.3947 [2020-12-31 12:11:57,763][root][INFO] - ===> Epoch[1432](7160/5): Loss -0.7081, Sem -0.7081, dir -0.9707, norm 0.2249 LR: 5.577e-02 Score 98.753 Data time: 0.0916, Forward time: 0.9471, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.3088 [2020-12-31 12:12:09,303][root][INFO] - ===> Epoch[1433](7165/5): Loss -0.7175, Sem -0.7175, dir -0.9711, norm 0.2184 LR: 5.574e-02 Score 98.833 Data time: 0.1107, Forward time: 0.9471, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.2070 [2020-12-31 12:12:20,610][root][INFO] - ===> Epoch[1434](7170/5): Loss -0.7247, Sem -0.7247, dir -0.9717, norm 0.2101 LR: 5.571e-02 Score 98.762 Data time: 0.1170, Forward time: 0.9471, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.1836 [2020-12-31 12:12:32,419][root][INFO] - ===> Epoch[1435](7175/5): Loss -0.7286, Sem -0.7286, dir -0.9719, norm 0.2074 LR: 5.568e-02 Score 98.810 Data time: 0.1131, Forward time: 0.9472, Backward time: 1.1954, DDP time: 0.0051, Total iter time: 2.2595 [2020-12-31 12:12:45,134][root][INFO] - ===> Epoch[1436](7180/5): Loss -0.7159, Sem -0.7159, dir -0.9715, norm 0.2192 LR: 5.565e-02 Score 98.791 Data time: 0.0835, Forward time: 0.9471, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.4660 [2020-12-31 12:12:58,075][root][INFO] - ===> Epoch[1437](7185/5): Loss -0.7214, Sem -0.7214, dir -0.9716, norm 0.2130 LR: 5.561e-02 Score 98.761 Data time: 0.1044, Forward time: 0.9471, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.5020 [2020-12-31 12:13:09,719][root][INFO] - ===> Epoch[1438](7190/5): Loss -0.7249, Sem -0.7249, dir -0.9715, norm 0.2102 LR: 5.558e-02 Score 98.796 Data time: 0.1045, Forward time: 0.9471, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.2300 [2020-12-31 12:13:21,415][root][INFO] - ===> Epoch[1439](7195/5): Loss -0.7183, Sem -0.7183, dir -0.9713, norm 0.2176 LR: 5.555e-02 Score 98.822 Data time: 0.0920, Forward time: 0.9471, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.2543 [2020-12-31 12:13:32,742][root][INFO] - ===> Epoch[1440](7200/5): Loss -0.7216, Sem -0.7216, dir -0.9715, norm 0.2122 LR: 5.552e-02 Score 98.748 Data time: 0.1327, Forward time: 0.9471, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.1738 [2020-12-31 12:13:32,745][root][INFO] - ===> Start testing [2020-12-31 12:14:11,053][root][INFO] - 68/68: Data time: 0.0024, Iter time: 0.1337 Loss 0.342 (AVG: 0.500) Score 92.815 (AVG: 89.779) mIOU 68.565 mAP 83.613 mAcc 76.175 IOU: 59.846 0.000 77.504 74.195 93.124 90.675 30.888 74.956 98.420 68.103 80.771 84.908 57.950 mAP: 80.450 5.182 92.528 86.882 98.873 98.627 61.408 93.939 99.843 90.661 93.122 94.070 91.381 mAcc: 79.796 0.000 85.950 80.725 94.453 95.902 36.444 87.200 99.206 80.614 91.822 95.003 63.167 [2020-12-31 12:14:11,055][root][INFO] - Finished test. Elapsed time: 38.3075 [2020-12-31 12:14:14,960][root][INFO] - [2020-12-31 12:14:14,960][root][INFO] - ################################################################ [2020-12-31 12:14:14,960][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 12:14:14,961][root][INFO] - ################################################################ [2020-12-31 12:14:14,961][root][INFO] - clutter : nan nan nan [2020-12-31 12:14:14,961][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 12:14:14,961][root][INFO] - board : 0.740 0.828 0.878 [2020-12-31 12:14:14,961][root][INFO] - bookcase : 0.201 0.364 0.514 [2020-12-31 12:14:14,961][root][INFO] - ceiling : 0.530 0.564 0.639 [2020-12-31 12:14:14,961][root][INFO] - chair : 0.801 0.899 0.912 [2020-12-31 12:14:14,961][root][INFO] - column : 0.153 0.238 0.425 [2020-12-31 12:14:14,961][root][INFO] - door : 0.581 0.746 0.890 [2020-12-31 12:14:14,961][root][INFO] - floor : 0.883 0.940 0.940 [2020-12-31 12:14:14,961][root][INFO] - sofa : 0.511 0.690 0.818 [2020-12-31 12:14:14,961][root][INFO] - table : 0.205 0.365 0.605 [2020-12-31 12:14:14,961][root][INFO] - wall : 0.462 0.690 0.779 [2020-12-31 12:14:14,961][root][INFO] - window : 0.528 0.747 0.767 [2020-12-31 12:14:14,961][root][INFO] - ---------------------------------------------------------------- [2020-12-31 12:14:14,961][root][INFO] - average : 0.466 0.589 0.680 [2020-12-31 12:14:14,961][root][INFO] - [2020-12-31 12:14:15,123][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 12:14:15,752][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2020-12-31 12:14:15,753][root][INFO] - Current best mIoU: 68.565 at iter 1 [2020-12-31 12:14:16,376][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter7200.pth [2020-12-31 12:14:27,926][root][INFO] - ===> Epoch[1441](7205/5): Loss -0.7225, Sem -0.7225, dir -0.9715, norm 0.2131 LR: 5.549e-02 Score 98.803 Data time: 0.0912, Forward time: 0.9470, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.2018 [2020-12-31 12:14:40,698][root][INFO] - ===> Epoch[1442](7210/5): Loss -0.7035, Sem -0.7035, dir -0.9708, norm 0.2294 LR: 5.545e-02 Score 98.755 Data time: 0.0905, Forward time: 0.9470, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.4623 [2020-12-31 12:14:52,400][root][INFO] - ===> Epoch[1443](7215/5): Loss -0.7261, Sem -0.7261, dir -0.9716, norm 0.2106 LR: 5.542e-02 Score 98.821 Data time: 0.0799, Forward time: 0.9470, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.2677 [2020-12-31 12:15:05,223][root][INFO] - ===> Epoch[1444](7220/5): Loss -0.7299, Sem -0.7299, dir -0.9717, norm 0.2077 LR: 5.539e-02 Score 98.843 Data time: 0.1134, Forward time: 0.9470, Backward time: 1.1958, DDP time: 0.0051, Total iter time: 2.4760 [2020-12-31 12:15:17,436][root][INFO] - ===> Epoch[1445](7225/5): Loss -0.7185, Sem -0.7185, dir -0.9709, norm 0.2163 LR: 5.536e-02 Score 98.793 Data time: 0.1228, Forward time: 0.9471, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.3418 [2020-12-31 12:15:29,221][root][INFO] - ===> Epoch[1446](7230/5): Loss -0.7187, Sem -0.7187, dir -0.9722, norm 0.2173 LR: 5.532e-02 Score 98.788 Data time: 0.0887, Forward time: 0.9470, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.2633 [2020-12-31 12:15:41,028][root][INFO] - ===> Epoch[1447](7235/5): Loss -0.7206, Sem -0.7206, dir -0.9712, norm 0.2140 LR: 5.529e-02 Score 98.777 Data time: 0.1663, Forward time: 0.9470, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.2744 [2020-12-31 12:15:52,091][root][INFO] - ===> Epoch[1448](7240/5): Loss -0.7197, Sem -0.7197, dir -0.9721, norm 0.2158 LR: 5.526e-02 Score 98.777 Data time: 0.1053, Forward time: 0.9470, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.1301 [2020-12-31 12:16:03,530][root][INFO] - ===> Epoch[1449](7245/5): Loss -0.7029, Sem -0.7029, dir -0.9707, norm 0.2252 LR: 5.523e-02 Score 98.621 Data time: 0.1333, Forward time: 0.9471, Backward time: 1.1954, DDP time: 0.0051, Total iter time: 2.1937 [2020-12-31 12:16:15,170][root][INFO] - ===> Epoch[1450](7250/5): Loss -0.7030, Sem -0.7030, dir -0.9697, norm 0.2250 LR: 5.520e-02 Score 98.609 Data time: 0.0841, Forward time: 0.9471, Backward time: 1.1954, DDP time: 0.0051, Total iter time: 2.2372 [2020-12-31 12:16:27,996][root][INFO] - ===> Epoch[1451](7255/5): Loss -0.6974, Sem -0.6974, dir -0.9695, norm 0.2274 LR: 5.516e-02 Score 98.604 Data time: 0.1005, Forward time: 0.9471, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.4675 [2020-12-31 12:16:40,186][root][INFO] - ===> Epoch[1452](7260/5): Loss -0.6968, Sem -0.6968, dir -0.9702, norm 0.2298 LR: 5.513e-02 Score 98.578 Data time: 0.2291, Forward time: 0.9471, Backward time: 1.1953, DDP time: 0.0051, Total iter time: 2.3365 [2020-12-31 12:16:52,252][root][INFO] - ===> Epoch[1453](7265/5): Loss -0.7051, Sem -0.7051, dir -0.9698, norm 0.2239 LR: 5.510e-02 Score 98.662 Data time: 0.1294, Forward time: 0.9472, Backward time: 1.1953, DDP time: 0.0051, Total iter time: 2.3115 [2020-12-31 12:17:04,623][root][INFO] - ===> Epoch[1454](7270/5): Loss -0.7102, Sem -0.7102, dir -0.9706, norm 0.2213 LR: 5.507e-02 Score 98.702 Data time: 0.1019, Forward time: 0.9472, Backward time: 1.1953, DDP time: 0.0051, Total iter time: 2.3692 [2020-12-31 12:17:16,300][root][INFO] - ===> Epoch[1455](7275/5): Loss -0.7094, Sem -0.7094, dir -0.9707, norm 0.2222 LR: 5.504e-02 Score 98.678 Data time: 0.0778, Forward time: 0.9472, Backward time: 1.1953, DDP time: 0.0051, Total iter time: 2.2566 [2020-12-31 12:17:27,745][root][INFO] - ===> Epoch[1456](7280/5): Loss -0.7106, Sem -0.7106, dir -0.9709, norm 0.2200 LR: 5.500e-02 Score 98.681 Data time: 0.1114, Forward time: 0.9473, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.1980 [2020-12-31 12:17:40,043][root][INFO] - ===> Epoch[1457](7285/5): Loss -0.7131, Sem -0.7131, dir -0.9714, norm 0.2209 LR: 5.497e-02 Score 98.737 Data time: 0.1212, Forward time: 0.9473, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.3685 [2020-12-31 12:17:52,004][root][INFO] - ===> Epoch[1458](7290/5): Loss -0.7180, Sem -0.7180, dir -0.9714, norm 0.2162 LR: 5.494e-02 Score 98.752 Data time: 0.1960, Forward time: 0.9474, Backward time: 1.1951, DDP time: 0.0051, Total iter time: 2.3066 [2020-12-31 12:18:03,002][root][INFO] - ===> Epoch[1459](7295/5): Loss -0.7214, Sem -0.7214, dir -0.9720, norm 0.2125 LR: 5.491e-02 Score 98.750 Data time: 0.0825, Forward time: 0.9474, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.1048 [2020-12-31 12:18:15,187][root][INFO] - ===> Epoch[1460](7300/5): Loss -0.7070, Sem -0.7070, dir -0.9704, norm 0.2236 LR: 5.488e-02 Score 98.676 Data time: 0.1305, Forward time: 0.9474, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.3396 [2020-12-31 12:18:27,097][root][INFO] - ===> Epoch[1461](7305/5): Loss -0.7108, Sem -0.7108, dir -0.9708, norm 0.2229 LR: 5.484e-02 Score 98.760 Data time: 0.0916, Forward time: 0.9474, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.2958 [2020-12-31 12:18:38,521][root][INFO] - ===> Epoch[1462](7310/5): Loss -0.7223, Sem -0.7223, dir -0.9715, norm 0.2130 LR: 5.481e-02 Score 98.793 Data time: 0.0727, Forward time: 0.9473, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.1960 [2020-12-31 12:18:50,592][root][INFO] - ===> Epoch[1463](7315/5): Loss -0.7240, Sem -0.7240, dir -0.9714, norm 0.2115 LR: 5.478e-02 Score 98.795 Data time: 0.1026, Forward time: 0.9473, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.3045 [2020-12-31 12:19:02,614][root][INFO] - ===> Epoch[1464](7320/5): Loss -0.7139, Sem -0.7139, dir -0.9707, norm 0.2200 LR: 5.475e-02 Score 98.791 Data time: 0.1201, Forward time: 0.9473, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.3011 [2020-12-31 12:19:14,762][root][INFO] - ===> Epoch[1465](7325/5): Loss -0.7206, Sem -0.7206, dir -0.9728, norm 0.2153 LR: 5.472e-02 Score 98.761 Data time: 0.1449, Forward time: 0.9473, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.3322 [2020-12-31 12:19:26,894][root][INFO] - ===> Epoch[1466](7330/5): Loss -0.7214, Sem -0.7214, dir -0.9715, norm 0.2141 LR: 5.468e-02 Score 98.793 Data time: 0.0964, Forward time: 0.9473, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.3326 [2020-12-31 12:19:38,862][root][INFO] - ===> Epoch[1467](7335/5): Loss -0.7168, Sem -0.7168, dir -0.9718, norm 0.2197 LR: 5.465e-02 Score 98.811 Data time: 0.2351, Forward time: 0.9473, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.2958 [2020-12-31 12:19:51,582][root][INFO] - ===> Epoch[1468](7340/5): Loss -0.7229, Sem -0.7229, dir -0.9723, norm 0.2130 LR: 5.462e-02 Score 98.782 Data time: 0.1882, Forward time: 0.9473, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.4534 [2020-12-31 12:20:03,745][root][INFO] - ===> Epoch[1469](7345/5): Loss -0.7271, Sem -0.7271, dir -0.9725, norm 0.2093 LR: 5.459e-02 Score 98.798 Data time: 0.0729, Forward time: 0.9472, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.3432 [2020-12-31 12:20:15,564][root][INFO] - ===> Epoch[1470](7350/5): Loss -0.7247, Sem -0.7247, dir -0.9719, norm 0.2114 LR: 5.456e-02 Score 98.803 Data time: 0.1094, Forward time: 0.9473, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.2637 [2020-12-31 12:20:27,233][root][INFO] - ===> Epoch[1471](7355/5): Loss -0.7275, Sem -0.7275, dir -0.9724, norm 0.2094 LR: 5.452e-02 Score 98.802 Data time: 0.0708, Forward time: 0.9472, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.2571 [2020-12-31 12:20:38,334][root][INFO] - ===> Epoch[1472](7360/5): Loss -0.7182, Sem -0.7182, dir -0.9716, norm 0.2162 LR: 5.449e-02 Score 98.764 Data time: 0.1039, Forward time: 0.9472, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.1295 [2020-12-31 12:20:50,390][root][INFO] - ===> Epoch[1473](7365/5): Loss -0.7213, Sem -0.7213, dir -0.9719, norm 0.2112 LR: 5.446e-02 Score 98.744 Data time: 0.1186, Forward time: 0.9472, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.3296 [2020-12-31 12:21:03,217][root][INFO] - ===> Epoch[1474](7370/5): Loss -0.7208, Sem -0.7208, dir -0.9722, norm 0.2162 LR: 5.443e-02 Score 98.834 Data time: 0.1206, Forward time: 0.9472, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.4829 [2020-12-31 12:21:15,274][root][INFO] - ===> Epoch[1475](7375/5): Loss -0.7194, Sem -0.7194, dir -0.9734, norm 0.2203 LR: 5.439e-02 Score 98.872 Data time: 0.0944, Forward time: 0.9472, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.3275 [2020-12-31 12:21:27,049][root][INFO] - ===> Epoch[1476](7380/5): Loss -0.7207, Sem -0.7207, dir -0.9721, norm 0.2160 LR: 5.436e-02 Score 98.806 Data time: 0.1380, Forward time: 0.9473, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.2626 [2020-12-31 12:21:38,667][root][INFO] - ===> Epoch[1477](7385/5): Loss -0.7306, Sem -0.7306, dir -0.9725, norm 0.2067 LR: 5.433e-02 Score 98.819 Data time: 0.1035, Forward time: 0.9472, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.2265 [2020-12-31 12:21:50,422][root][INFO] - ===> Epoch[1478](7390/5): Loss -0.7176, Sem -0.7176, dir -0.9722, norm 0.2190 LR: 5.430e-02 Score 98.821 Data time: 0.1883, Forward time: 0.9473, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.2561 [2020-12-31 12:22:02,678][root][INFO] - ===> Epoch[1479](7395/5): Loss -0.7163, Sem -0.7163, dir -0.9715, norm 0.2186 LR: 5.427e-02 Score 98.771 Data time: 0.0996, Forward time: 0.9473, Backward time: 1.1948, DDP time: 0.0051, Total iter time: 2.3567 [2020-12-31 12:22:14,708][root][INFO] - ===> Epoch[1480](7400/5): Loss -0.7239, Sem -0.7239, dir -0.9717, norm 0.2123 LR: 5.423e-02 Score 98.820 Data time: 0.0964, Forward time: 0.9473, Backward time: 1.1948, DDP time: 0.0051, Total iter time: 2.3231 [2020-12-31 12:22:14,711][root][INFO] - ===> Start testing [2020-12-31 12:22:54,035][root][INFO] - 68/68: Data time: 0.0022, Iter time: 0.1351 Loss 0.562 (AVG: 0.527) Score 88.928 (AVG: 89.343) mIOU 68.255 mAP 83.370 mAcc 75.625 IOU: 58.483 0.000 80.337 73.661 92.167 89.598 33.031 73.251 98.172 68.282 81.257 84.365 54.709 mAP: 80.974 7.524 93.661 86.482 98.764 98.472 61.896 92.558 99.847 88.789 92.927 93.481 88.436 mAcc: 80.064 0.000 88.538 80.804 93.454 95.194 39.467 84.684 99.199 78.009 89.009 94.890 59.813 [2020-12-31 12:22:54,036][root][INFO] - Finished test. Elapsed time: 39.3228 [2020-12-31 12:22:57,204][root][INFO] - [2020-12-31 12:22:57,204][root][INFO] - ################################################################ [2020-12-31 12:22:57,205][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 12:22:57,205][root][INFO] - ################################################################ [2020-12-31 12:22:57,205][root][INFO] - clutter : nan nan nan [2020-12-31 12:22:57,205][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 12:22:57,205][root][INFO] - board : 0.754 0.848 0.893 [2020-12-31 12:22:57,205][root][INFO] - bookcase : 0.195 0.342 0.511 [2020-12-31 12:22:57,205][root][INFO] - ceiling : 0.633 0.653 0.745 [2020-12-31 12:22:57,205][root][INFO] - chair : 0.783 0.872 0.889 [2020-12-31 12:22:57,205][root][INFO] - column : 0.140 0.204 0.283 [2020-12-31 12:22:57,205][root][INFO] - door : 0.615 0.784 0.885 [2020-12-31 12:22:57,205][root][INFO] - floor : 0.780 0.780 0.803 [2020-12-31 12:22:57,205][root][INFO] - sofa : 0.469 0.690 0.818 [2020-12-31 12:22:57,205][root][INFO] - table : 0.170 0.394 0.651 [2020-12-31 12:22:57,205][root][INFO] - wall : 0.434 0.636 0.742 [2020-12-31 12:22:57,205][root][INFO] - window : 0.481 0.720 0.778 [2020-12-31 12:22:57,205][root][INFO] - ---------------------------------------------------------------- [2020-12-31 12:22:57,205][root][INFO] - average : 0.455 0.577 0.667 [2020-12-31 12:22:57,205][root][INFO] - [2020-12-31 12:22:57,365][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 12:22:57,937][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter7400.pth [2020-12-31 12:23:09,981][root][INFO] - ===> Epoch[1481](7405/5): Loss -0.7224, Sem -0.7224, dir -0.9729, norm 0.2143 LR: 5.420e-02 Score 98.790 Data time: 0.0853, Forward time: 0.9472, Backward time: 1.1948, DDP time: 0.0051, Total iter time: 2.3133 [2020-12-31 12:23:22,149][root][INFO] - ===> Epoch[1482](7410/5): Loss -0.7255, Sem -0.7255, dir -0.9720, norm 0.2111 LR: 5.417e-02 Score 98.828 Data time: 0.1158, Forward time: 0.9473, Backward time: 1.1948, DDP time: 0.0051, Total iter time: 2.3417 [2020-12-31 12:23:33,556][root][INFO] - ===> Epoch[1483](7415/5): Loss -0.7226, Sem -0.7226, dir -0.9719, norm 0.2113 LR: 5.414e-02 Score 98.727 Data time: 0.1153, Forward time: 0.9473, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.1929 [2020-12-31 12:23:46,484][root][INFO] - ===> Epoch[1484](7420/5): Loss -0.7277, Sem -0.7277, dir -0.9726, norm 0.2101 LR: 5.411e-02 Score 98.837 Data time: 0.1251, Forward time: 0.9473, Backward time: 1.1948, DDP time: 0.0051, Total iter time: 2.4950 [2020-12-31 12:23:58,047][root][INFO] - ===> Epoch[1485](7425/5): Loss -0.7290, Sem -0.7290, dir -0.9718, norm 0.2069 LR: 5.407e-02 Score 98.790 Data time: 0.0727, Forward time: 0.9472, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.2358 [2020-12-31 12:24:10,762][root][INFO] - ===> Epoch[1486](7430/5): Loss -0.7210, Sem -0.7210, dir -0.9719, norm 0.2143 LR: 5.404e-02 Score 98.784 Data time: 0.0955, Forward time: 0.9472, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.4493 [2020-12-31 12:24:22,973][root][INFO] - ===> Epoch[1487](7435/5): Loss -0.7272, Sem -0.7272, dir -0.9724, norm 0.2081 LR: 5.401e-02 Score 98.787 Data time: 0.1068, Forward time: 0.9472, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.3488 [2020-12-31 12:24:34,576][root][INFO] - ===> Epoch[1488](7440/5): Loss -0.7339, Sem -0.7339, dir -0.9730, norm 0.2048 LR: 5.398e-02 Score 98.861 Data time: 0.0974, Forward time: 0.9472, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.2241 [2020-12-31 12:24:47,154][root][INFO] - ===> Epoch[1489](7445/5): Loss -0.7354, Sem -0.7354, dir -0.9725, norm 0.2026 LR: 5.395e-02 Score 98.847 Data time: 0.0921, Forward time: 0.9472, Backward time: 1.1951, DDP time: 0.0051, Total iter time: 2.4352 [2020-12-31 12:24:58,649][root][INFO] - ===> Epoch[1490](7450/5): Loss -0.7326, Sem -0.7326, dir -0.9730, norm 0.2057 LR: 5.391e-02 Score 98.852 Data time: 0.1002, Forward time: 0.9472, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.2081 [2020-12-31 12:25:10,840][root][INFO] - ===> Epoch[1491](7455/5): Loss -0.7317, Sem -0.7317, dir -0.9731, norm 0.2072 LR: 5.388e-02 Score 98.847 Data time: 0.1126, Forward time: 0.9472, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.3485 [2020-12-31 12:25:22,934][root][INFO] - ===> Epoch[1492](7460/5): Loss -0.7212, Sem -0.7212, dir -0.9715, norm 0.2141 LR: 5.385e-02 Score 98.795 Data time: 0.1233, Forward time: 0.9472, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.3221 [2020-12-31 12:25:35,343][root][INFO] - ===> Epoch[1493](7465/5): Loss -0.7270, Sem -0.7270, dir -0.9725, norm 0.2096 LR: 5.382e-02 Score 98.794 Data time: 0.1068, Forward time: 0.9472, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.3818 [2020-12-31 12:25:46,617][root][INFO] - ===> Epoch[1494](7470/5): Loss -0.7321, Sem -0.7321, dir -0.9731, norm 0.2064 LR: 5.378e-02 Score 98.843 Data time: 0.1035, Forward time: 0.9472, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.1664 [2020-12-31 12:25:58,212][root][INFO] - ===> Epoch[1495](7475/5): Loss -0.7229, Sem -0.7229, dir -0.9725, norm 0.2137 LR: 5.375e-02 Score 98.804 Data time: 0.0762, Forward time: 0.9471, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.2341 [2020-12-31 12:26:11,542][root][INFO] - ===> Epoch[1496](7480/5): Loss -0.7336, Sem -0.7336, dir -0.9730, norm 0.2063 LR: 5.372e-02 Score 98.885 Data time: 0.0906, Forward time: 0.9471, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.5775 [2020-12-31 12:26:23,672][root][INFO] - ===> Epoch[1497](7485/5): Loss -0.7272, Sem -0.7272, dir -0.9718, norm 0.2081 LR: 5.369e-02 Score 98.784 Data time: 0.0847, Forward time: 0.9470, Backward time: 1.1953, DDP time: 0.0051, Total iter time: 2.3396 [2020-12-31 12:26:35,641][root][INFO] - ===> Epoch[1498](7490/5): Loss -0.7311, Sem -0.7311, dir -0.9728, norm 0.2072 LR: 5.366e-02 Score 98.851 Data time: 0.1200, Forward time: 0.9470, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.2993 [2020-12-31 12:26:47,060][root][INFO] - ===> Epoch[1499](7495/5): Loss -0.7209, Sem -0.7209, dir -0.9728, norm 0.2161 LR: 5.362e-02 Score 98.815 Data time: 0.1220, Forward time: 0.9470, Backward time: 1.1951, DDP time: 0.0051, Total iter time: 2.1838 [2020-12-31 12:26:59,340][root][INFO] - ===> Epoch[1500](7500/5): Loss -0.7185, Sem -0.7185, dir -0.9717, norm 0.2169 LR: 5.359e-02 Score 98.783 Data time: 0.1294, Forward time: 0.9470, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.3739 [2020-12-31 12:27:11,148][root][INFO] - ===> Epoch[1501](7505/5): Loss -0.7336, Sem -0.7336, dir -0.9738, norm 0.2054 LR: 5.356e-02 Score 98.835 Data time: 0.1109, Forward time: 0.9471, Backward time: 1.1951, DDP time: 0.0051, Total iter time: 2.2720 [2020-12-31 12:27:23,378][root][INFO] - ===> Epoch[1502](7510/5): Loss -0.7229, Sem -0.7229, dir -0.9714, norm 0.2124 LR: 5.353e-02 Score 98.786 Data time: 0.0862, Forward time: 0.9470, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.3620 [2020-12-31 12:27:35,207][root][INFO] - ===> Epoch[1503](7515/5): Loss -0.7272, Sem -0.7272, dir -0.9726, norm 0.2101 LR: 5.350e-02 Score 98.834 Data time: 0.0943, Forward time: 0.9469, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.2634 [2020-12-31 12:27:48,854][root][INFO] - ===> Epoch[1504](7520/5): Loss -0.7275, Sem -0.7275, dir -0.9732, norm 0.2106 LR: 5.346e-02 Score 98.824 Data time: 0.1288, Forward time: 0.9470, Backward time: 1.1954, DDP time: 0.0051, Total iter time: 2.6384 [2020-12-31 12:28:00,592][root][INFO] - ===> Epoch[1505](7525/5): Loss -0.7266, Sem -0.7266, dir -0.9729, norm 0.2116 LR: 5.343e-02 Score 98.840 Data time: 0.0839, Forward time: 0.9470, Backward time: 1.1954, DDP time: 0.0051, Total iter time: 2.2708 [2020-12-31 12:28:12,730][root][INFO] - ===> Epoch[1506](7530/5): Loss -0.7310, Sem -0.7310, dir -0.9731, norm 0.2082 LR: 5.340e-02 Score 98.858 Data time: 0.1536, Forward time: 0.9470, Backward time: 1.1953, DDP time: 0.0051, Total iter time: 2.3310 [2020-12-31 12:28:25,412][root][INFO] - ===> Epoch[1507](7535/5): Loss -0.7283, Sem -0.7283, dir -0.9729, norm 0.2099 LR: 5.337e-02 Score 98.842 Data time: 0.0992, Forward time: 0.9470, Backward time: 1.1954, DDP time: 0.0051, Total iter time: 2.4520 [2020-12-31 12:28:37,547][root][INFO] - ===> Epoch[1508](7540/5): Loss -0.7202, Sem -0.7202, dir -0.9729, norm 0.2173 LR: 5.333e-02 Score 98.822 Data time: 0.0795, Forward time: 0.9470, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.3376 [2020-12-31 12:28:49,810][root][INFO] - ===> Epoch[1509](7545/5): Loss -0.7212, Sem -0.7212, dir -0.9720, norm 0.2163 LR: 5.330e-02 Score 98.837 Data time: 0.1004, Forward time: 0.9470, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.3507 [2020-12-31 12:29:01,798][root][INFO] - ===> Epoch[1510](7550/5): Loss -0.7282, Sem -0.7282, dir -0.9724, norm 0.2093 LR: 5.327e-02 Score 98.831 Data time: 0.0878, Forward time: 0.9470, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.3126 [2020-12-31 12:29:13,883][root][INFO] - ===> Epoch[1511](7555/5): Loss -0.7303, Sem -0.7303, dir -0.9730, norm 0.2079 LR: 5.324e-02 Score 98.824 Data time: 0.0895, Forward time: 0.9470, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.3251 [2020-12-31 12:29:26,851][root][INFO] - ===> Epoch[1512](7560/5): Loss -0.7269, Sem -0.7269, dir -0.9724, norm 0.2095 LR: 5.321e-02 Score 98.812 Data time: 0.1084, Forward time: 0.9470, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.4936 [2020-12-31 12:29:38,531][root][INFO] - ===> Epoch[1513](7565/5): Loss -0.7274, Sem -0.7274, dir -0.9719, norm 0.2065 LR: 5.317e-02 Score 98.735 Data time: 0.0742, Forward time: 0.9469, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.2657 [2020-12-31 12:29:51,243][root][INFO] - ===> Epoch[1514](7570/5): Loss -0.7247, Sem -0.7247, dir -0.9723, norm 0.2109 LR: 5.314e-02 Score 98.794 Data time: 0.1090, Forward time: 0.9469, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.4507 [2020-12-31 12:30:03,324][root][INFO] - ===> Epoch[1515](7575/5): Loss -0.7300, Sem -0.7300, dir -0.9732, norm 0.2092 LR: 5.311e-02 Score 98.866 Data time: 0.1174, Forward time: 0.9470, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.3200 [2020-12-31 12:30:14,722][root][INFO] - ===> Epoch[1516](7580/5): Loss -0.7387, Sem -0.7387, dir -0.9728, norm 0.1986 LR: 5.308e-02 Score 98.804 Data time: 0.0933, Forward time: 0.9469, Backward time: 1.1957, DDP time: 0.0051, Total iter time: 2.1908 [2020-12-31 12:30:25,755][root][INFO] - ===> Epoch[1517](7585/5): Loss -0.7265, Sem -0.7265, dir -0.9722, norm 0.2093 LR: 5.305e-02 Score 98.794 Data time: 0.1050, Forward time: 0.9469, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.1101 [2020-12-31 12:30:37,034][root][INFO] - ===> Epoch[1518](7590/5): Loss -0.7203, Sem -0.7203, dir -0.9721, norm 0.2157 LR: 5.301e-02 Score 98.800 Data time: 0.0971, Forward time: 0.9469, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.1676 [2020-12-31 12:30:49,505][root][INFO] - ===> Epoch[1519](7595/5): Loss -0.7260, Sem -0.7260, dir -0.9724, norm 0.2109 LR: 5.298e-02 Score 98.826 Data time: 0.0889, Forward time: 0.9469, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.4128 [2020-12-31 12:31:00,824][root][INFO] - ===> Epoch[1520](7600/5): Loss -0.7214, Sem -0.7214, dir -0.9723, norm 0.2142 LR: 5.295e-02 Score 98.770 Data time: 0.0991, Forward time: 0.9469, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.1752 [2020-12-31 12:31:00,827][root][INFO] - ===> Start testing [2020-12-31 12:31:39,614][root][INFO] - 68/68: Data time: 0.0027, Iter time: 0.1300 Loss 0.129 (AVG: 0.517) Score 96.526 (AVG: 89.797) mIOU 68.156 mAP 82.697 mAcc 74.798 IOU: 60.768 0.000 79.855 74.384 93.178 89.131 28.332 75.315 98.344 65.578 80.238 84.479 56.422 mAP: 81.067 5.216 92.965 87.041 98.898 98.520 55.811 94.436 99.844 83.653 92.974 93.738 90.901 mAcc: 80.009 0.000 86.524 81.874 94.591 96.490 31.287 87.094 98.948 70.126 88.318 95.644 61.466 [2020-12-31 12:31:39,615][root][INFO] - Finished test. Elapsed time: 38.7861 [2020-12-31 12:31:42,861][root][INFO] - [2020-12-31 12:31:42,861][root][INFO] - ################################################################ [2020-12-31 12:31:42,861][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 12:31:42,861][root][INFO] - ################################################################ [2020-12-31 12:31:42,861][root][INFO] - clutter : nan nan nan [2020-12-31 12:31:42,861][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 12:31:42,861][root][INFO] - board : 0.779 0.829 0.899 [2020-12-31 12:31:42,861][root][INFO] - bookcase : 0.210 0.335 0.497 [2020-12-31 12:31:42,861][root][INFO] - ceiling : 0.568 0.598 0.629 [2020-12-31 12:31:42,861][root][INFO] - chair : 0.790 0.881 0.900 [2020-12-31 12:31:42,862][root][INFO] - column : 0.139 0.209 0.341 [2020-12-31 12:31:42,862][root][INFO] - door : 0.594 0.767 0.908 [2020-12-31 12:31:42,862][root][INFO] - floor : 0.939 0.953 0.953 [2020-12-31 12:31:42,862][root][INFO] - sofa : 0.396 0.545 0.624 [2020-12-31 12:31:42,862][root][INFO] - table : 0.176 0.379 0.686 [2020-12-31 12:31:42,862][root][INFO] - wall : 0.477 0.717 0.805 [2020-12-31 12:31:42,862][root][INFO] - window : 0.491 0.740 0.774 [2020-12-31 12:31:42,862][root][INFO] - ---------------------------------------------------------------- [2020-12-31 12:31:42,862][root][INFO] - average : 0.463 0.579 0.668 [2020-12-31 12:31:42,862][root][INFO] - [2020-12-31 12:31:43,026][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 12:31:43,686][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter7600.pth [2020-12-31 12:31:55,096][root][INFO] - ===> Epoch[1521](7605/5): Loss -0.7320, Sem -0.7320, dir -0.9729, norm 0.2056 LR: 5.292e-02 Score 98.824 Data time: 0.1035, Forward time: 0.9469, Backward time: 1.1954, DDP time: 0.0051, Total iter time: 2.1710 [2020-12-31 12:32:06,870][root][INFO] - ===> Epoch[1522](7610/5): Loss -0.7300, Sem -0.7300, dir -0.9727, norm 0.2086 LR: 5.288e-02 Score 98.856 Data time: 0.0900, Forward time: 0.9469, Backward time: 1.1954, DDP time: 0.0051, Total iter time: 2.2595 [2020-12-31 12:32:19,479][root][INFO] - ===> Epoch[1523](7615/5): Loss -0.7316, Sem -0.7316, dir -0.9731, norm 0.2079 LR: 5.285e-02 Score 98.885 Data time: 0.0825, Forward time: 0.9468, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.4213 [2020-12-31 12:32:31,336][root][INFO] - ===> Epoch[1524](7620/5): Loss -0.7186, Sem -0.7186, dir -0.9722, norm 0.2163 LR: 5.282e-02 Score 98.775 Data time: 0.1052, Forward time: 0.9468, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.2758 [2020-12-31 12:32:43,135][root][INFO] - ===> Epoch[1525](7625/5): Loss -0.7276, Sem -0.7276, dir -0.9726, norm 0.2104 LR: 5.279e-02 Score 98.847 Data time: 0.0752, Forward time: 0.9468, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.2809 [2020-12-31 12:32:55,410][root][INFO] - ===> Epoch[1526](7630/5): Loss -0.7328, Sem -0.7328, dir -0.9729, norm 0.2039 LR: 5.276e-02 Score 98.787 Data time: 0.1247, Forward time: 0.9469, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.3634 [2020-12-31 12:33:07,739][root][INFO] - ===> Epoch[1527](7635/5): Loss -0.7232, Sem -0.7232, dir -0.9728, norm 0.2131 LR: 5.272e-02 Score 98.787 Data time: 0.0886, Forward time: 0.9469, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.3796 [2020-12-31 12:33:20,607][root][INFO] - ===> Epoch[1528](7640/5): Loss -0.7306, Sem -0.7306, dir -0.9731, norm 0.2077 LR: 5.269e-02 Score 98.834 Data time: 0.1261, Forward time: 0.9469, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.4903 [2020-12-31 12:33:32,388][root][INFO] - ===> Epoch[1529](7645/5): Loss -0.7285, Sem -0.7285, dir -0.9730, norm 0.2083 LR: 5.266e-02 Score 98.803 Data time: 0.1936, Forward time: 0.9469, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.2565 [2020-12-31 12:33:44,601][root][INFO] - ===> Epoch[1530](7650/5): Loss -0.7244, Sem -0.7244, dir -0.9730, norm 0.2142 LR: 5.263e-02 Score 98.864 Data time: 0.0897, Forward time: 0.9469, Backward time: 1.1956, DDP time: 0.0051, Total iter time: 2.3363 [2020-12-31 12:33:56,296][root][INFO] - ===> Epoch[1531](7655/5): Loss -0.7269, Sem -0.7269, dir -0.9728, norm 0.2108 LR: 5.259e-02 Score 98.821 Data time: 0.1440, Forward time: 0.9469, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.2563 [2020-12-31 12:34:07,831][root][INFO] - ===> Epoch[1532](7660/5): Loss -0.7260, Sem -0.7260, dir -0.9728, norm 0.2123 LR: 5.256e-02 Score 98.843 Data time: 0.0903, Forward time: 0.9469, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.2129 [2020-12-31 12:34:19,718][root][INFO] - ===> Epoch[1533](7665/5): Loss -0.7336, Sem -0.7336, dir -0.9730, norm 0.2053 LR: 5.253e-02 Score 98.855 Data time: 0.1377, Forward time: 0.9468, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.2904 [2020-12-31 12:34:31,842][root][INFO] - ===> Epoch[1534](7670/5): Loss -0.7240, Sem -0.7240, dir -0.9730, norm 0.2158 LR: 5.250e-02 Score 98.890 Data time: 0.0912, Forward time: 0.9468, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.3293 [2020-12-31 12:34:43,575][root][INFO] - ===> Epoch[1535](7675/5): Loss -0.7332, Sem -0.7332, dir -0.9734, norm 0.2062 LR: 5.247e-02 Score 98.864 Data time: 0.1406, Forward time: 0.9468, Backward time: 1.1955, DDP time: 0.0051, Total iter time: 2.2605 [2020-12-31 12:34:54,967][root][INFO] - ===> Epoch[1536](7680/5): Loss -0.7305, Sem -0.7305, dir -0.9725, norm 0.2063 LR: 5.243e-02 Score 98.817 Data time: 0.1318, Forward time: 0.9468, Backward time: 1.1954, DDP time: 0.0051, Total iter time: 2.1812 [2020-12-31 12:35:07,922][root][INFO] - ===> Epoch[1537](7685/5): Loss -0.7258, Sem -0.7258, dir -0.9729, norm 0.2102 LR: 5.240e-02 Score 98.787 Data time: 0.1292, Forward time: 0.9469, Backward time: 1.1954, DDP time: 0.0051, Total iter time: 2.4925 [2020-12-31 12:35:19,757][root][INFO] - ===> Epoch[1538](7690/5): Loss -0.7145, Sem -0.7145, dir -0.9713, norm 0.2184 LR: 5.237e-02 Score 98.744 Data time: 0.1225, Forward time: 0.9468, Backward time: 1.1954, DDP time: 0.0051, Total iter time: 2.2710 [2020-12-31 12:35:32,240][root][INFO] - ===> Epoch[1539](7695/5): Loss -0.7110, Sem -0.7110, dir -0.9722, norm 0.2225 LR: 5.234e-02 Score 98.726 Data time: 0.1294, Forward time: 0.9469, Backward time: 1.1954, DDP time: 0.0051, Total iter time: 2.4130 [2020-12-31 12:35:44,659][root][INFO] - ===> Epoch[1540](7700/5): Loss -0.7256, Sem -0.7256, dir -0.9725, norm 0.2123 LR: 5.230e-02 Score 98.852 Data time: 0.1870, Forward time: 0.9470, Backward time: 1.1954, DDP time: 0.0051, Total iter time: 2.3908 [2020-12-31 12:35:56,520][root][INFO] - ===> Epoch[1541](7705/5): Loss -0.7253, Sem -0.7253, dir -0.9726, norm 0.2126 LR: 5.227e-02 Score 98.845 Data time: 0.1827, Forward time: 0.9470, Backward time: 1.1953, DDP time: 0.0051, Total iter time: 2.2809 [2020-12-31 12:36:08,655][root][INFO] - ===> Epoch[1542](7710/5): Loss -0.7230, Sem -0.7230, dir -0.9724, norm 0.2140 LR: 5.224e-02 Score 98.817 Data time: 0.0836, Forward time: 0.9470, Backward time: 1.1954, DDP time: 0.0051, Total iter time: 2.3586 [2020-12-31 12:36:20,228][root][INFO] - ===> Epoch[1543](7715/5): Loss -0.7265, Sem -0.7265, dir -0.9721, norm 0.2102 LR: 5.221e-02 Score 98.806 Data time: 0.1095, Forward time: 0.9470, Backward time: 1.1953, DDP time: 0.0051, Total iter time: 2.2256 [2020-12-31 12:36:31,631][root][INFO] - ===> Epoch[1544](7720/5): Loss -0.7224, Sem -0.7224, dir -0.9732, norm 0.2166 LR: 5.218e-02 Score 98.859 Data time: 0.0960, Forward time: 0.9470, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.1874 [2020-12-31 12:36:44,253][root][INFO] - ===> Epoch[1545](7725/5): Loss -0.7263, Sem -0.7263, dir -0.9733, norm 0.2120 LR: 5.214e-02 Score 98.821 Data time: 0.1327, Forward time: 0.9471, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.4270 [2020-12-31 12:36:56,151][root][INFO] - ===> Epoch[1546](7730/5): Loss -0.7285, Sem -0.7285, dir -0.9721, norm 0.2081 LR: 5.211e-02 Score 98.822 Data time: 0.1936, Forward time: 0.9470, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.2907 [2020-12-31 12:37:08,765][root][INFO] - ===> Epoch[1547](7735/5): Loss -0.7253, Sem -0.7253, dir -0.9732, norm 0.2125 LR: 5.208e-02 Score 98.829 Data time: 0.0914, Forward time: 0.9470, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.4336 [2020-12-31 12:37:20,707][root][INFO] - ===> Epoch[1548](7740/5): Loss -0.7352, Sem -0.7352, dir -0.9740, norm 0.2051 LR: 5.205e-02 Score 98.868 Data time: 0.0852, Forward time: 0.9470, Backward time: 1.1953, DDP time: 0.0051, Total iter time: 2.3008 [2020-12-31 12:37:32,457][root][INFO] - ===> Epoch[1549](7745/5): Loss -0.7283, Sem -0.7283, dir -0.9732, norm 0.2097 LR: 5.201e-02 Score 98.829 Data time: 0.1331, Forward time: 0.9470, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.2536 [2020-12-31 12:37:44,381][root][INFO] - ===> Epoch[1550](7750/5): Loss -0.7328, Sem -0.7328, dir -0.9727, norm 0.2056 LR: 5.198e-02 Score 98.852 Data time: 0.1130, Forward time: 0.9470, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.2872 [2020-12-31 12:37:56,441][root][INFO] - ===> Epoch[1551](7755/5): Loss -0.7311, Sem -0.7311, dir -0.9732, norm 0.2072 LR: 5.195e-02 Score 98.830 Data time: 0.1627, Forward time: 0.9470, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.3212 [2020-12-31 12:38:07,551][root][INFO] - ===> Epoch[1552](7760/5): Loss -0.7309, Sem -0.7309, dir -0.9726, norm 0.2053 LR: 5.192e-02 Score 98.781 Data time: 0.0761, Forward time: 0.9469, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.1376 [2020-12-31 12:38:19,917][root][INFO] - ===> Epoch[1553](7765/5): Loss -0.7246, Sem -0.7246, dir -0.9726, norm 0.2125 LR: 5.188e-02 Score 98.811 Data time: 0.1535, Forward time: 0.9469, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.3716 [2020-12-31 12:38:30,908][root][INFO] - ===> Epoch[1554](7770/5): Loss -0.7240, Sem -0.7240, dir -0.9725, norm 0.2130 LR: 5.185e-02 Score 98.810 Data time: 0.1111, Forward time: 0.9470, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.1207 [2020-12-31 12:38:43,249][root][INFO] - ===> Epoch[1555](7775/5): Loss -0.7278, Sem -0.7278, dir -0.9726, norm 0.2100 LR: 5.182e-02 Score 98.841 Data time: 0.1231, Forward time: 0.9471, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.3776 [2020-12-31 12:38:55,534][root][INFO] - ===> Epoch[1556](7780/5): Loss -0.7291, Sem -0.7291, dir -0.9729, norm 0.2073 LR: 5.179e-02 Score 98.774 Data time: 0.0664, Forward time: 0.9469, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.3829 [2020-12-31 12:39:07,319][root][INFO] - ===> Epoch[1557](7785/5): Loss -0.7365, Sem -0.7365, dir -0.9730, norm 0.2016 LR: 5.176e-02 Score 98.823 Data time: 0.0759, Forward time: 0.9469, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.2688 [2020-12-31 12:39:19,301][root][INFO] - ===> Epoch[1558](7790/5): Loss -0.7355, Sem -0.7355, dir -0.9734, norm 0.2024 LR: 5.172e-02 Score 98.801 Data time: 0.1805, Forward time: 0.9469, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.3090 [2020-12-31 12:39:32,106][root][INFO] - ===> Epoch[1559](7795/5): Loss -0.7382, Sem -0.7382, dir -0.9733, norm 0.2001 LR: 5.169e-02 Score 98.832 Data time: 0.1121, Forward time: 0.9469, Backward time: 1.1953, DDP time: 0.0051, Total iter time: 2.4797 [2020-12-31 12:39:43,791][root][INFO] - ===> Epoch[1560](7800/5): Loss -0.7368, Sem -0.7368, dir -0.9733, norm 0.2020 LR: 5.166e-02 Score 98.850 Data time: 0.1410, Forward time: 0.9469, Backward time: 1.1952, DDP time: 0.0051, Total iter time: 2.2314 [2020-12-31 12:39:43,793][root][INFO] - ===> Start testing [2020-12-31 12:40:22,150][root][INFO] - 68/68: Data time: 0.0025, Iter time: 0.1490 Loss 0.171 (AVG: 0.551) Score 96.041 (AVG: 89.295) mIOU 66.991 mAP 82.665 mAcc 75.061 IOU: 59.244 0.000 76.645 72.528 92.260 89.062 30.441 74.370 98.036 60.132 78.803 84.711 54.654 mAP: 81.443 5.079 92.803 86.318 98.632 98.416 58.499 93.864 99.809 83.583 93.088 93.354 89.756 mAcc: 81.186 0.000 88.438 79.656 93.317 96.219 33.732 87.900 98.944 71.843 90.776 94.873 58.912 [2020-12-31 12:40:22,151][root][INFO] - Finished test. Elapsed time: 38.3555 [2020-12-31 12:40:25,427][root][INFO] - [2020-12-31 12:40:25,427][root][INFO] - ################################################################ [2020-12-31 12:40:25,427][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 12:40:25,427][root][INFO] - ################################################################ [2020-12-31 12:40:25,428][root][INFO] - clutter : nan nan nan [2020-12-31 12:40:25,428][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 12:40:25,428][root][INFO] - board : 0.789 0.826 0.899 [2020-12-31 12:40:25,428][root][INFO] - bookcase : 0.188 0.330 0.485 [2020-12-31 12:40:25,428][root][INFO] - ceiling : 0.504 0.540 0.568 [2020-12-31 12:40:25,428][root][INFO] - chair : 0.771 0.869 0.899 [2020-12-31 12:40:25,428][root][INFO] - column : 0.136 0.182 0.323 [2020-12-31 12:40:25,428][root][INFO] - door : 0.673 0.851 0.923 [2020-12-31 12:40:25,428][root][INFO] - floor : 0.956 0.980 0.986 [2020-12-31 12:40:25,428][root][INFO] - sofa : 0.375 0.531 0.717 [2020-12-31 12:40:25,428][root][INFO] - table : 0.200 0.401 0.638 [2020-12-31 12:40:25,428][root][INFO] - wall : 0.447 0.683 0.760 [2020-12-31 12:40:25,428][root][INFO] - window : 0.460 0.717 0.782 [2020-12-31 12:40:25,428][root][INFO] - ---------------------------------------------------------------- [2020-12-31 12:40:25,428][root][INFO] - average : 0.458 0.576 0.665 [2020-12-31 12:40:25,428][root][INFO] - [2020-12-31 12:40:25,606][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 12:40:26,215][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter7800.pth [2020-12-31 12:40:38,589][root][INFO] - ===> Epoch[1561](7805/5): Loss -0.7390, Sem -0.7390, dir -0.9734, norm 0.1998 LR: 5.163e-02 Score 98.845 Data time: 0.1490, Forward time: 0.9470, Backward time: 1.1951, DDP time: 0.0051, Total iter time: 2.3751 [2020-12-31 12:40:50,366][root][INFO] - ===> Epoch[1562](7810/5): Loss -0.7307, Sem -0.7307, dir -0.9733, norm 0.2084 LR: 5.159e-02 Score 98.856 Data time: 0.1271, Forward time: 0.9471, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.2683 [2020-12-31 12:41:02,975][root][INFO] - ===> Epoch[1563](7815/5): Loss -0.7301, Sem -0.7301, dir -0.9732, norm 0.2087 LR: 5.156e-02 Score 98.856 Data time: 0.1418, Forward time: 0.9472, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.4196 [2020-12-31 12:41:15,312][root][INFO] - ===> Epoch[1564](7820/5): Loss -0.7331, Sem -0.7331, dir -0.9726, norm 0.2048 LR: 5.153e-02 Score 98.838 Data time: 0.1202, Forward time: 0.9473, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.3836 [2020-12-31 12:41:27,565][root][INFO] - ===> Epoch[1565](7825/5): Loss -0.7318, Sem -0.7318, dir -0.9727, norm 0.2067 LR: 5.150e-02 Score 98.864 Data time: 0.0889, Forward time: 0.9472, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.3730 [2020-12-31 12:41:39,867][root][INFO] - ===> Epoch[1566](7830/5): Loss -0.7362, Sem -0.7362, dir -0.9727, norm 0.2016 LR: 5.147e-02 Score 98.842 Data time: 0.1088, Forward time: 0.9472, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.3643 [2020-12-31 12:41:51,831][root][INFO] - ===> Epoch[1567](7835/5): Loss -0.7368, Sem -0.7368, dir -0.9733, norm 0.2028 LR: 5.143e-02 Score 98.872 Data time: 0.1642, Forward time: 0.9473, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.2983 [2020-12-31 12:42:04,427][root][INFO] - ===> Epoch[1568](7840/5): Loss -0.7215, Sem -0.7215, dir -0.9723, norm 0.2168 LR: 5.140e-02 Score 98.871 Data time: 0.1101, Forward time: 0.9473, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.4296 [2020-12-31 12:42:16,306][root][INFO] - ===> Epoch[1569](7845/5): Loss -0.7330, Sem -0.7330, dir -0.9733, norm 0.2054 LR: 5.137e-02 Score 98.830 Data time: 0.1050, Forward time: 0.9473, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.2844 [2020-12-31 12:42:28,742][root][INFO] - ===> Epoch[1570](7850/5): Loss -0.7187, Sem -0.7187, dir -0.9715, norm 0.2177 LR: 5.134e-02 Score 98.827 Data time: 0.1637, Forward time: 0.9474, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.3895 [2020-12-31 12:42:40,007][root][INFO] - ===> Epoch[1571](7855/5): Loss -0.7204, Sem -0.7204, dir -0.9724, norm 0.2166 LR: 5.130e-02 Score 98.830 Data time: 0.1091, Forward time: 0.9474, Backward time: 1.1948, DDP time: 0.0051, Total iter time: 2.1667 [2020-12-31 12:42:52,405][root][INFO] - ===> Epoch[1572](7860/5): Loss -0.7270, Sem -0.7270, dir -0.9723, norm 0.2112 LR: 5.127e-02 Score 98.855 Data time: 0.1058, Forward time: 0.9474, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.3991 [2020-12-31 12:43:05,166][root][INFO] - ===> Epoch[1573](7865/5): Loss -0.7304, Sem -0.7304, dir -0.9731, norm 0.2090 LR: 5.124e-02 Score 98.883 Data time: 0.0999, Forward time: 0.9474, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.4533 [2020-12-31 12:43:16,618][root][INFO] - ===> Epoch[1574](7870/5): Loss -0.7299, Sem -0.7299, dir -0.9723, norm 0.2079 LR: 5.121e-02 Score 98.833 Data time: 0.1039, Forward time: 0.9474, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.1968 [2020-12-31 12:43:28,772][root][INFO] - ===> Epoch[1575](7875/5): Loss -0.7307, Sem -0.7307, dir -0.9727, norm 0.2070 LR: 5.117e-02 Score 98.832 Data time: 0.1241, Forward time: 0.9474, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.3310 [2020-12-31 12:43:41,433][root][INFO] - ===> Epoch[1576](7880/5): Loss -0.7300, Sem -0.7300, dir -0.9727, norm 0.2084 LR: 5.114e-02 Score 98.858 Data time: 0.0870, Forward time: 0.9474, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.4354 [2020-12-31 12:43:54,083][root][INFO] - ===> Epoch[1577](7885/5): Loss -0.7099, Sem -0.7099, dir -0.9714, norm 0.2262 LR: 5.111e-02 Score 98.822 Data time: 0.1017, Forward time: 0.9474, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.4418 [2020-12-31 12:44:05,404][root][INFO] - ===> Epoch[1578](7890/5): Loss -0.7227, Sem -0.7227, dir -0.9716, norm 0.2120 LR: 5.108e-02 Score 98.756 Data time: 0.1117, Forward time: 0.9475, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.1754 [2020-12-31 12:44:17,327][root][INFO] - ===> Epoch[1579](7895/5): Loss -0.7272, Sem -0.7272, dir -0.9734, norm 0.2110 LR: 5.105e-02 Score 98.826 Data time: 0.0815, Forward time: 0.9474, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.2977 [2020-12-31 12:44:29,014][root][INFO] - ===> Epoch[1580](7900/5): Loss -0.7264, Sem -0.7264, dir -0.9724, norm 0.2097 LR: 5.101e-02 Score 98.804 Data time: 0.0953, Forward time: 0.9474, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.2458 [2020-12-31 12:44:40,351][root][INFO] - ===> Epoch[1581](7905/5): Loss -0.7291, Sem -0.7291, dir -0.9726, norm 0.2087 LR: 5.098e-02 Score 98.840 Data time: 0.1326, Forward time: 0.9474, Backward time: 1.1948, DDP time: 0.0051, Total iter time: 2.1683 [2020-12-31 12:44:52,640][root][INFO] - ===> Epoch[1582](7910/5): Loss -0.7267, Sem -0.7267, dir -0.9730, norm 0.2108 LR: 5.095e-02 Score 98.824 Data time: 0.1665, Forward time: 0.9475, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.3569 [2020-12-31 12:45:05,831][root][INFO] - ===> Epoch[1583](7915/5): Loss -0.7312, Sem -0.7312, dir -0.9727, norm 0.2060 LR: 5.092e-02 Score 98.818 Data time: 0.0857, Forward time: 0.9474, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.5546 [2020-12-31 12:45:18,065][root][INFO] - ===> Epoch[1584](7920/5): Loss -0.7345, Sem -0.7345, dir -0.9737, norm 0.2037 LR: 5.088e-02 Score 98.820 Data time: 0.1261, Forward time: 0.9474, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.3522 [2020-12-31 12:45:29,692][root][INFO] - ===> Epoch[1585](7925/5): Loss -0.7277, Sem -0.7277, dir -0.9733, norm 0.2098 LR: 5.085e-02 Score 98.807 Data time: 0.1002, Forward time: 0.9474, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.2347 [2020-12-31 12:45:40,867][root][INFO] - ===> Epoch[1586](7930/5): Loss -0.7201, Sem -0.7201, dir -0.9734, norm 0.2187 LR: 5.082e-02 Score 98.863 Data time: 0.0799, Forward time: 0.9473, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.1469 [2020-12-31 12:45:53,001][root][INFO] - ===> Epoch[1587](7935/5): Loss -0.7237, Sem -0.7237, dir -0.9726, norm 0.2132 LR: 5.079e-02 Score 98.810 Data time: 0.1056, Forward time: 0.9473, Backward time: 1.1950, DDP time: 0.0051, Total iter time: 2.3315 [2020-12-31 12:46:04,690][root][INFO] - ===> Epoch[1588](7940/5): Loss -0.7253, Sem -0.7253, dir -0.9731, norm 0.2126 LR: 5.075e-02 Score 98.815 Data time: 0.1142, Forward time: 0.9474, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.2561 [2020-12-31 12:46:16,369][root][INFO] - ===> Epoch[1589](7945/5): Loss -0.7278, Sem -0.7278, dir -0.9734, norm 0.2114 LR: 5.072e-02 Score 98.864 Data time: 0.0952, Forward time: 0.9474, Backward time: 1.1948, DDP time: 0.0051, Total iter time: 2.2466 [2020-12-31 12:46:28,533][root][INFO] - ===> Epoch[1590](7950/5): Loss -0.7337, Sem -0.7337, dir -0.9732, norm 0.2065 LR: 5.069e-02 Score 98.898 Data time: 0.1308, Forward time: 0.9475, Backward time: 1.1948, DDP time: 0.0051, Total iter time: 2.3341 [2020-12-31 12:46:40,775][root][INFO] - ===> Epoch[1591](7955/5): Loss -0.7304, Sem -0.7304, dir -0.9731, norm 0.2075 LR: 5.066e-02 Score 98.811 Data time: 0.0926, Forward time: 0.9475, Backward time: 1.1948, DDP time: 0.0051, Total iter time: 2.3606 [2020-12-31 12:46:52,918][root][INFO] - ===> Epoch[1592](7960/5): Loss -0.7311, Sem -0.7311, dir -0.9731, norm 0.2070 LR: 5.062e-02 Score 98.830 Data time: 0.1172, Forward time: 0.9475, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.3455 [2020-12-31 12:47:04,519][root][INFO] - ===> Epoch[1593](7965/5): Loss -0.7354, Sem -0.7354, dir -0.9732, norm 0.2043 LR: 5.059e-02 Score 98.880 Data time: 0.1711, Forward time: 0.9476, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.2311 [2020-12-31 12:47:16,560][root][INFO] - ===> Epoch[1594](7970/5): Loss -0.7332, Sem -0.7332, dir -0.9738, norm 0.2067 LR: 5.056e-02 Score 98.872 Data time: 0.1030, Forward time: 0.9476, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.3207 [2020-12-31 12:47:28,384][root][INFO] - ===> Epoch[1595](7975/5): Loss -0.7399, Sem -0.7399, dir -0.9736, norm 0.1993 LR: 5.053e-02 Score 98.845 Data time: 0.1005, Forward time: 0.9476, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.2736 [2020-12-31 12:47:39,477][root][INFO] - ===> Epoch[1596](7980/5): Loss -0.7315, Sem -0.7315, dir -0.9731, norm 0.2070 LR: 5.050e-02 Score 98.845 Data time: 0.1229, Forward time: 0.9477, Backward time: 1.1944, DDP time: 0.0051, Total iter time: 2.1292 [2020-12-31 12:47:52,789][root][INFO] - ===> Epoch[1597](7985/5): Loss -0.7308, Sem -0.7308, dir -0.9735, norm 0.2076 LR: 5.046e-02 Score 98.824 Data time: 0.1110, Forward time: 0.9477, Backward time: 1.1945, DDP time: 0.0051, Total iter time: 2.5764 [2020-12-31 12:48:04,473][root][INFO] - ===> Epoch[1598](7990/5): Loss -0.7392, Sem -0.7392, dir -0.9735, norm 0.2009 LR: 5.043e-02 Score 98.878 Data time: 0.1194, Forward time: 0.9478, Backward time: 1.1944, DDP time: 0.0051, Total iter time: 2.2482 [2020-12-31 12:48:17,572][root][INFO] - ===> Epoch[1599](7995/5): Loss -0.7410, Sem -0.7410, dir -0.9737, norm 0.1994 LR: 5.040e-02 Score 98.896 Data time: 0.0855, Forward time: 0.9477, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.5261 [2020-12-31 12:48:30,563][root][INFO] - ===> Epoch[1600](8000/5): Loss -0.7477, Sem -0.7477, dir -0.9749, norm 0.1949 LR: 5.037e-02 Score 98.906 Data time: 0.1122, Forward time: 0.9478, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.5080 [2020-12-31 12:48:30,565][root][INFO] - ===> Start testing [2020-12-31 12:49:10,224][root][INFO] - 68/68: Data time: 0.0024, Iter time: 0.1440 Loss 0.106 (AVG: 0.520) Score 96.746 (AVG: 89.569) mIOU 67.760 mAP 83.102 mAcc 75.521 IOU: 59.653 0.000 78.728 73.245 92.645 89.741 31.453 76.246 98.576 62.582 79.775 84.577 53.657 mAP: 81.332 4.351 93.066 86.617 98.758 98.638 60.892 94.172 99.848 86.408 92.593 93.874 89.776 mAcc: 80.103 0.000 85.669 80.276 93.882 96.306 34.799 89.043 99.371 80.624 88.690 95.684 57.322 [2020-12-31 12:49:10,225][root][INFO] - Finished test. Elapsed time: 39.6577 [2020-12-31 12:49:13,493][root][INFO] - [2020-12-31 12:49:13,494][root][INFO] - ################################################################ [2020-12-31 12:49:13,494][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 12:49:13,494][root][INFO] - ################################################################ [2020-12-31 12:49:13,494][root][INFO] - clutter : nan nan nan [2020-12-31 12:49:13,494][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 12:49:13,494][root][INFO] - board : 0.762 0.806 0.880 [2020-12-31 12:49:13,494][root][INFO] - bookcase : 0.220 0.376 0.510 [2020-12-31 12:49:13,494][root][INFO] - ceiling : 0.673 0.700 0.778 [2020-12-31 12:49:13,494][root][INFO] - chair : 0.805 0.894 0.903 [2020-12-31 12:49:13,494][root][INFO] - column : 0.141 0.213 0.331 [2020-12-31 12:49:13,494][root][INFO] - door : 0.627 0.814 0.938 [2020-12-31 12:49:13,494][root][INFO] - floor : 0.955 0.967 0.984 [2020-12-31 12:49:13,494][root][INFO] - sofa : 0.541 0.818 0.818 [2020-12-31 12:49:13,494][root][INFO] - table : 0.180 0.353 0.650 [2020-12-31 12:49:13,494][root][INFO] - wall : 0.480 0.707 0.797 [2020-12-31 12:49:13,494][root][INFO] - window : 0.449 0.702 0.737 [2020-12-31 12:49:13,494][root][INFO] - ---------------------------------------------------------------- [2020-12-31 12:49:13,494][root][INFO] - average : 0.486 0.613 0.694 [2020-12-31 12:49:13,494][root][INFO] - [2020-12-31 12:49:13,674][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 12:49:14,278][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_mAP.pth [2020-12-31 12:49:14,278][root][INFO] - Current best mAP@0.5: 0.613 at iter 8000 [2020-12-31 12:49:14,878][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter8000.pth [2020-12-31 12:49:26,265][root][INFO] - ===> Epoch[1601](8005/5): Loss -0.7405, Sem -0.7405, dir -0.9744, norm 0.2010 LR: 5.033e-02 Score 98.903 Data time: 0.0926, Forward time: 0.9477, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.1794 [2020-12-31 12:49:37,921][root][INFO] - ===> Epoch[1602](8010/5): Loss -0.7332, Sem -0.7332, dir -0.9726, norm 0.2023 LR: 5.030e-02 Score 98.761 Data time: 0.1088, Forward time: 0.9477, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.2470 [2020-12-31 12:49:50,322][root][INFO] - ===> Epoch[1603](8015/5): Loss -0.7281, Sem -0.7281, dir -0.9735, norm 0.2103 LR: 5.027e-02 Score 98.854 Data time: 0.1150, Forward time: 0.9477, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.3858 [2020-12-31 12:50:01,554][root][INFO] - ===> Epoch[1604](8020/5): Loss -0.7344, Sem -0.7344, dir -0.9735, norm 0.2042 LR: 5.024e-02 Score 98.835 Data time: 0.0938, Forward time: 0.9477, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.1480 [2020-12-31 12:50:13,047][root][INFO] - ===> Epoch[1605](8025/5): Loss -0.7269, Sem -0.7269, dir -0.9722, norm 0.2099 LR: 5.020e-02 Score 98.829 Data time: 0.1153, Forward time: 0.9477, Backward time: 1.1945, DDP time: 0.0051, Total iter time: 2.2015 [2020-12-31 12:50:24,707][root][INFO] - ===> Epoch[1606](8030/5): Loss -0.7330, Sem -0.7330, dir -0.9739, norm 0.2077 LR: 5.017e-02 Score 98.891 Data time: 0.0905, Forward time: 0.9477, Backward time: 1.1945, DDP time: 0.0051, Total iter time: 2.2370 [2020-12-31 12:50:37,672][root][INFO] - ===> Epoch[1607](8035/5): Loss -0.7240, Sem -0.7240, dir -0.9729, norm 0.2136 LR: 5.014e-02 Score 98.830 Data time: 0.0980, Forward time: 0.9477, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.5017 [2020-12-31 12:50:50,062][root][INFO] - ===> Epoch[1608](8040/5): Loss -0.7338, Sem -0.7338, dir -0.9734, norm 0.2069 LR: 5.011e-02 Score 98.893 Data time: 0.1111, Forward time: 0.9477, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.3889 [2020-12-31 12:51:01,464][root][INFO] - ===> Epoch[1609](8045/5): Loss -0.7302, Sem -0.7302, dir -0.9731, norm 0.2082 LR: 5.007e-02 Score 98.841 Data time: 0.1539, Forward time: 0.9476, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.1776 [2020-12-31 12:51:13,405][root][INFO] - ===> Epoch[1610](8050/5): Loss -0.7322, Sem -0.7322, dir -0.9731, norm 0.2060 LR: 5.004e-02 Score 98.844 Data time: 0.1001, Forward time: 0.9476, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.2959 [2020-12-31 12:51:25,856][root][INFO] - ===> Epoch[1611](8055/5): Loss -0.7420, Sem -0.7420, dir -0.9742, norm 0.2003 LR: 5.001e-02 Score 98.923 Data time: 0.1144, Forward time: 0.9476, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.3946 [2020-12-31 12:51:37,955][root][INFO] - ===> Epoch[1612](8060/5): Loss -0.7323, Sem -0.7323, dir -0.9732, norm 0.2061 LR: 4.998e-02 Score 98.857 Data time: 0.0730, Forward time: 0.9475, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.3414 [2020-12-31 12:51:50,304][root][INFO] - ===> Epoch[1613](8065/5): Loss -0.7298, Sem -0.7298, dir -0.9737, norm 0.2106 LR: 4.994e-02 Score 98.887 Data time: 0.1150, Forward time: 0.9475, Backward time: 1.1948, DDP time: 0.0051, Total iter time: 2.3683 [2020-12-31 12:52:02,460][root][INFO] - ===> Epoch[1614](8070/5): Loss -0.7380, Sem -0.7380, dir -0.9737, norm 0.2015 LR: 4.991e-02 Score 98.854 Data time: 0.1275, Forward time: 0.9475, Backward time: 1.1948, DDP time: 0.0051, Total iter time: 2.3387 [2020-12-31 12:52:13,732][root][INFO] - ===> Epoch[1615](8075/5): Loss -0.7303, Sem -0.7303, dir -0.9737, norm 0.2100 LR: 4.988e-02 Score 98.890 Data time: 0.1008, Forward time: 0.9475, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.1605 [2020-12-31 12:52:24,981][root][INFO] - ===> Epoch[1616](8080/5): Loss -0.7459, Sem -0.7459, dir -0.9742, norm 0.1971 LR: 4.985e-02 Score 98.958 Data time: 0.1622, Forward time: 0.9475, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.1631 [2020-12-31 12:52:36,714][root][INFO] - ===> Epoch[1617](8085/5): Loss -0.7357, Sem -0.7357, dir -0.9727, norm 0.2012 LR: 4.982e-02 Score 98.809 Data time: 0.1687, Forward time: 0.9475, Backward time: 1.1945, DDP time: 0.0051, Total iter time: 2.2443 [2020-12-31 12:52:48,580][root][INFO] - ===> Epoch[1618](8090/5): Loss -0.7401, Sem -0.7401, dir -0.9738, norm 0.2004 LR: 4.978e-02 Score 98.888 Data time: 0.1155, Forward time: 0.9475, Backward time: 1.1945, DDP time: 0.0051, Total iter time: 2.2759 [2020-12-31 12:53:00,451][root][INFO] - ===> Epoch[1619](8095/5): Loss -0.7425, Sem -0.7425, dir -0.9739, norm 0.1981 LR: 4.975e-02 Score 98.885 Data time: 0.1221, Forward time: 0.9475, Backward time: 1.1945, DDP time: 0.0051, Total iter time: 2.2870 [2020-12-31 12:53:11,945][root][INFO] - ===> Epoch[1620](8100/5): Loss -0.7378, Sem -0.7378, dir -0.9739, norm 0.2022 LR: 4.972e-02 Score 98.882 Data time: 0.0956, Forward time: 0.9474, Backward time: 1.1945, DDP time: 0.0051, Total iter time: 2.2031 [2020-12-31 12:53:23,755][root][INFO] - ===> Epoch[1621](8105/5): Loss -0.7334, Sem -0.7334, dir -0.9735, norm 0.2050 LR: 4.969e-02 Score 98.824 Data time: 0.0900, Forward time: 0.9474, Backward time: 1.1945, DDP time: 0.0051, Total iter time: 2.2884 [2020-12-31 12:53:35,477][root][INFO] - ===> Epoch[1622](8110/5): Loss -0.7435, Sem -0.7435, dir -0.9745, norm 0.1970 LR: 4.965e-02 Score 98.863 Data time: 0.0889, Forward time: 0.9473, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.2505 [2020-12-31 12:53:48,473][root][INFO] - ===> Epoch[1623](8115/5): Loss -0.7307, Sem -0.7307, dir -0.9731, norm 0.2089 LR: 4.962e-02 Score 98.870 Data time: 0.1195, Forward time: 0.9473, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.5024 [2020-12-31 12:53:59,889][root][INFO] - ===> Epoch[1624](8120/5): Loss -0.7314, Sem -0.7314, dir -0.9735, norm 0.2066 LR: 4.959e-02 Score 98.820 Data time: 0.1051, Forward time: 0.9473, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.1965 [2020-12-31 12:54:11,902][root][INFO] - ===> Epoch[1625](8125/5): Loss -0.7413, Sem -0.7413, dir -0.9747, norm 0.2007 LR: 4.956e-02 Score 98.901 Data time: 0.0880, Forward time: 0.9473, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.3116 [2020-12-31 12:54:23,556][root][INFO] - ===> Epoch[1626](8130/5): Loss -0.7334, Sem -0.7334, dir -0.9733, norm 0.2039 LR: 4.952e-02 Score 98.788 Data time: 0.0698, Forward time: 0.9472, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.2432 [2020-12-31 12:54:35,073][root][INFO] - ===> Epoch[1627](8135/5): Loss -0.7298, Sem -0.7298, dir -0.9730, norm 0.2081 LR: 4.949e-02 Score 98.830 Data time: 0.1024, Forward time: 0.9471, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.2242 [2020-12-31 12:54:48,676][root][INFO] - ===> Epoch[1628](8140/5): Loss -0.7261, Sem -0.7261, dir -0.9727, norm 0.2140 LR: 4.946e-02 Score 98.917 Data time: 0.0989, Forward time: 0.9472, Backward time: 1.1948, DDP time: 0.0051, Total iter time: 2.6188 [2020-12-31 12:55:00,082][root][INFO] - ===> Epoch[1629](8145/5): Loss -0.7235, Sem -0.7235, dir -0.9728, norm 0.2119 LR: 4.943e-02 Score 98.806 Data time: 0.1430, Forward time: 0.9471, Backward time: 1.1948, DDP time: 0.0051, Total iter time: 2.1813 [2020-12-31 12:55:12,550][root][INFO] - ===> Epoch[1630](8150/5): Loss -0.7278, Sem -0.7278, dir -0.9729, norm 0.2081 LR: 4.939e-02 Score 98.757 Data time: 0.1183, Forward time: 0.9471, Backward time: 1.1949, DDP time: 0.0051, Total iter time: 2.4033 [2020-12-31 12:55:24,806][root][INFO] - ===> Epoch[1631](8155/5): Loss -0.7299, Sem -0.7299, dir -0.9733, norm 0.2085 LR: 4.936e-02 Score 98.835 Data time: 0.2558, Forward time: 0.9471, Backward time: 1.1948, DDP time: 0.0051, Total iter time: 2.3606 [2020-12-31 12:55:35,871][root][INFO] - ===> Epoch[1632](8160/5): Loss -0.7358, Sem -0.7358, dir -0.9737, norm 0.2043 LR: 4.933e-02 Score 98.878 Data time: 0.1360, Forward time: 0.9472, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.1275 [2020-12-31 12:55:48,512][root][INFO] - ===> Epoch[1633](8165/5): Loss -0.7405, Sem -0.7405, dir -0.9731, norm 0.1977 LR: 4.930e-02 Score 98.833 Data time: 0.1045, Forward time: 0.9472, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.4362 [2020-12-31 12:56:00,492][root][INFO] - ===> Epoch[1634](8170/5): Loss -0.7354, Sem -0.7354, dir -0.9734, norm 0.2032 LR: 4.926e-02 Score 98.836 Data time: 0.1133, Forward time: 0.9473, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.3000 [2020-12-31 12:56:12,866][root][INFO] - ===> Epoch[1635](8175/5): Loss -0.7245, Sem -0.7245, dir -0.9734, norm 0.2145 LR: 4.923e-02 Score 98.849 Data time: 0.0837, Forward time: 0.9472, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.3837 [2020-12-31 12:56:24,421][root][INFO] - ===> Epoch[1636](8180/5): Loss -0.7294, Sem -0.7294, dir -0.9736, norm 0.2104 LR: 4.920e-02 Score 98.860 Data time: 0.1107, Forward time: 0.9472, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.2127 [2020-12-31 12:56:37,152][root][INFO] - ===> Epoch[1637](8185/5): Loss -0.7374, Sem -0.7374, dir -0.9741, norm 0.2030 LR: 4.917e-02 Score 98.867 Data time: 0.2392, Forward time: 0.9472, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.4561 [2020-12-31 12:56:49,705][root][INFO] - ===> Epoch[1638](8190/5): Loss -0.7374, Sem -0.7374, dir -0.9741, norm 0.2023 LR: 4.913e-02 Score 98.864 Data time: 0.1370, Forward time: 0.9473, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.4158 [2020-12-31 12:57:01,603][root][INFO] - ===> Epoch[1639](8195/5): Loss -0.7380, Sem -0.7380, dir -0.9737, norm 0.2020 LR: 4.910e-02 Score 98.863 Data time: 0.1098, Forward time: 0.9472, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.2938 [2020-12-31 12:57:13,933][root][INFO] - ===> Epoch[1640](8200/5): Loss -0.7333, Sem -0.7333, dir -0.9735, norm 0.2066 LR: 4.907e-02 Score 98.878 Data time: 0.1112, Forward time: 0.9472, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.3664 [2020-12-31 12:57:13,935][root][INFO] - ===> Start testing [2020-12-31 12:57:52,298][root][INFO] - 68/68: Data time: 0.0026, Iter time: 0.1619 Loss 0.257 (AVG: 0.537) Score 94.696 (AVG: 89.660) mIOU 68.094 mAP 83.375 mAcc 75.639 IOU: 60.417 0.000 81.028 72.360 93.635 90.354 29.369 76.168 98.450 65.792 80.134 84.590 52.926 mAP: 81.190 6.241 93.338 86.110 98.888 98.231 59.647 94.408 99.832 88.598 93.012 94.218 90.167 mAcc: 81.097 0.000 87.205 78.871 94.938 96.092 31.852 89.051 99.232 82.636 91.189 95.584 55.562 [2020-12-31 12:57:52,299][root][INFO] - Finished test. Elapsed time: 38.3618 [2020-12-31 12:57:55,437][root][INFO] - [2020-12-31 12:57:55,437][root][INFO] - ################################################################ [2020-12-31 12:57:55,437][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 12:57:55,437][root][INFO] - ################################################################ [2020-12-31 12:57:55,437][root][INFO] - clutter : nan nan nan [2020-12-31 12:57:55,437][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 12:57:55,437][root][INFO] - board : 0.783 0.848 0.899 [2020-12-31 12:57:55,437][root][INFO] - bookcase : 0.194 0.313 0.498 [2020-12-31 12:57:55,437][root][INFO] - ceiling : 0.579 0.627 0.734 [2020-12-31 12:57:55,437][root][INFO] - chair : 0.779 0.873 0.900 [2020-12-31 12:57:55,437][root][INFO] - column : 0.145 0.224 0.366 [2020-12-31 12:57:55,437][root][INFO] - door : 0.564 0.749 0.870 [2020-12-31 12:57:55,438][root][INFO] - floor : 0.867 0.867 0.885 [2020-12-31 12:57:55,438][root][INFO] - sofa : 0.514 0.809 0.809 [2020-12-31 12:57:55,438][root][INFO] - table : 0.166 0.319 0.642 [2020-12-31 12:57:55,438][root][INFO] - wall : 0.465 0.689 0.772 [2020-12-31 12:57:55,438][root][INFO] - window : 0.425 0.645 0.753 [2020-12-31 12:57:55,438][root][INFO] - ---------------------------------------------------------------- [2020-12-31 12:57:55,438][root][INFO] - average : 0.457 0.580 0.677 [2020-12-31 12:57:55,438][root][INFO] - [2020-12-31 12:57:55,610][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 12:57:56,260][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter8200.pth [2020-12-31 12:58:07,669][root][INFO] - ===> Epoch[1641](8205/5): Loss -0.7300, Sem -0.7300, dir -0.9736, norm 0.2091 LR: 4.904e-02 Score 98.844 Data time: 0.1284, Forward time: 0.9472, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.1786 [2020-12-31 12:58:19,573][root][INFO] - ===> Epoch[1642](8210/5): Loss -0.7325, Sem -0.7325, dir -0.9739, norm 0.2076 LR: 4.900e-02 Score 98.871 Data time: 0.0917, Forward time: 0.9472, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.3226 [2020-12-31 12:58:31,696][root][INFO] - ===> Epoch[1643](8215/5): Loss -0.7332, Sem -0.7332, dir -0.9737, norm 0.2070 LR: 4.897e-02 Score 98.875 Data time: 0.0804, Forward time: 0.9472, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.3348 [2020-12-31 12:58:43,105][root][INFO] - ===> Epoch[1644](8220/5): Loss -0.7412, Sem -0.7412, dir -0.9745, norm 0.1998 LR: 4.894e-02 Score 98.882 Data time: 0.1040, Forward time: 0.9472, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.1941 [2020-12-31 12:58:55,634][root][INFO] - ===> Epoch[1645](8225/5): Loss -0.7363, Sem -0.7363, dir -0.9736, norm 0.2045 LR: 4.891e-02 Score 98.897 Data time: 0.1003, Forward time: 0.9472, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.4110 [2020-12-31 12:59:08,158][root][INFO] - ===> Epoch[1646](8230/5): Loss -0.7447, Sem -0.7447, dir -0.9743, norm 0.1979 LR: 4.887e-02 Score 98.936 Data time: 0.1261, Forward time: 0.9472, Backward time: 1.1947, DDP time: 0.0051, Total iter time: 2.4206 [2020-12-31 12:59:20,217][root][INFO] - ===> Epoch[1647](8235/5): Loss -0.7362, Sem -0.7362, dir -0.9741, norm 0.2037 LR: 4.884e-02 Score 98.883 Data time: 0.1600, Forward time: 0.9472, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.3085 [2020-12-31 12:59:31,985][root][INFO] - ===> Epoch[1648](8240/5): Loss -0.7417, Sem -0.7417, dir -0.9743, norm 0.1997 LR: 4.881e-02 Score 98.900 Data time: 0.0984, Forward time: 0.9472, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.2681 [2020-12-31 12:59:44,245][root][INFO] - ===> Epoch[1649](8245/5): Loss -0.7335, Sem -0.7335, dir -0.9729, norm 0.2053 LR: 4.878e-02 Score 98.867 Data time: 0.1179, Forward time: 0.9473, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.3557 [2020-12-31 12:59:56,153][root][INFO] - ===> Epoch[1650](8250/5): Loss -0.7351, Sem -0.7351, dir -0.9739, norm 0.2041 LR: 4.874e-02 Score 98.844 Data time: 0.1673, Forward time: 0.9472, Backward time: 1.1946, DDP time: 0.0051, Total iter time: 2.2911 [2020-12-31 13:00:07,997][root][INFO] - ===> Epoch[1651](8255/5): Loss -0.7259, Sem -0.7259, dir -0.9731, norm 0.2124 LR: 4.871e-02 Score 98.840 Data time: 0.1178, Forward time: 0.9473, Backward time: 1.1945, DDP time: 0.0051, Total iter time: 2.2656 [2020-12-31 13:00:19,861][root][INFO] - ===> Epoch[1652](8260/5): Loss -0.7371, Sem -0.7371, dir -0.9740, norm 0.2029 LR: 4.868e-02 Score 98.879 Data time: 0.1829, Forward time: 0.9474, Backward time: 1.1944, DDP time: 0.0051, Total iter time: 2.2399 [2020-12-31 13:00:31,304][root][INFO] - ===> Epoch[1653](8265/5): Loss -0.7388, Sem -0.7388, dir -0.9740, norm 0.2022 LR: 4.865e-02 Score 98.907 Data time: 0.0918, Forward time: 0.9474, Backward time: 1.1943, DDP time: 0.0051, Total iter time: 2.1997 [2020-12-31 13:00:42,860][root][INFO] - ===> Epoch[1654](8270/5): Loss -0.7355, Sem -0.7355, dir -0.9739, norm 0.2038 LR: 4.861e-02 Score 98.842 Data time: 0.1157, Forward time: 0.9474, Backward time: 1.1942, DDP time: 0.0051, Total iter time: 2.2131 [2020-12-31 13:00:54,635][root][INFO] - ===> Epoch[1655](8275/5): Loss -0.7456, Sem -0.7456, dir -0.9744, norm 0.1965 LR: 4.858e-02 Score 98.913 Data time: 0.1172, Forward time: 0.9474, Backward time: 1.1942, DDP time: 0.0051, Total iter time: 2.2587 [2020-12-31 13:01:07,063][root][INFO] - ===> Epoch[1656](8280/5): Loss -0.7385, Sem -0.7385, dir -0.9744, norm 0.2009 LR: 4.855e-02 Score 98.854 Data time: 0.1350, Forward time: 0.9475, Backward time: 1.1941, DDP time: 0.0051, Total iter time: 2.3990 [2020-12-31 13:01:19,567][root][INFO] - ===> Epoch[1657](8285/5): Loss -0.7370, Sem -0.7370, dir -0.9735, norm 0.2007 LR: 4.852e-02 Score 98.811 Data time: 0.1104, Forward time: 0.9475, Backward time: 1.1942, DDP time: 0.0051, Total iter time: 2.4127 [2020-12-31 13:01:31,525][root][INFO] - ===> Epoch[1658](8290/5): Loss -0.7460, Sem -0.7460, dir -0.9745, norm 0.1956 LR: 4.848e-02 Score 98.889 Data time: 0.1135, Forward time: 0.9476, Backward time: 1.1941, DDP time: 0.0051, Total iter time: 2.2950 [2020-12-31 13:01:44,164][root][INFO] - ===> Epoch[1659](8295/5): Loss -0.7389, Sem -0.7389, dir -0.9741, norm 0.2018 LR: 4.845e-02 Score 98.891 Data time: 0.1248, Forward time: 0.9477, Backward time: 1.1941, DDP time: 0.0051, Total iter time: 2.4275 [2020-12-31 13:01:55,436][root][INFO] - ===> Epoch[1660](8300/5): Loss -0.7340, Sem -0.7340, dir -0.9741, norm 0.2057 LR: 4.842e-02 Score 98.863 Data time: 0.0982, Forward time: 0.9477, Backward time: 1.1940, DDP time: 0.0051, Total iter time: 2.1676 [2020-12-31 13:02:06,941][root][INFO] - ===> Epoch[1661](8305/5): Loss -0.7395, Sem -0.7395, dir -0.9734, norm 0.2002 LR: 4.839e-02 Score 98.878 Data time: 0.0915, Forward time: 0.9477, Backward time: 1.1939, DDP time: 0.0051, Total iter time: 2.2015 [2020-12-31 13:02:18,156][root][INFO] - ===> Epoch[1662](8310/5): Loss -0.7381, Sem -0.7381, dir -0.9734, norm 0.2011 LR: 4.835e-02 Score 98.855 Data time: 0.0836, Forward time: 0.9476, Backward time: 1.1939, DDP time: 0.0051, Total iter time: 2.1519 [2020-12-31 13:02:30,028][root][INFO] - ===> Epoch[1663](8315/5): Loss -0.7279, Sem -0.7279, dir -0.9733, norm 0.2108 LR: 4.832e-02 Score 98.853 Data time: 0.1262, Forward time: 0.9476, Backward time: 1.1938, DDP time: 0.0051, Total iter time: 2.2708 [2020-12-31 13:02:42,342][root][INFO] - ===> Epoch[1664](8320/5): Loss -0.7406, Sem -0.7406, dir -0.9744, norm 0.2009 LR: 4.829e-02 Score 98.900 Data time: 0.1543, Forward time: 0.9477, Backward time: 1.1938, DDP time: 0.0051, Total iter time: 2.3693 [2020-12-31 13:02:54,903][root][INFO] - ===> Epoch[1665](8325/5): Loss -0.7465, Sem -0.7465, dir -0.9746, norm 0.1960 LR: 4.826e-02 Score 98.917 Data time: 0.1510, Forward time: 0.9478, Backward time: 1.1938, DDP time: 0.0051, Total iter time: 2.4152 [2020-12-31 13:03:06,757][root][INFO] - ===> Epoch[1666](8330/5): Loss -0.7306, Sem -0.7306, dir -0.9745, norm 0.2118 LR: 4.822e-02 Score 98.927 Data time: 0.1085, Forward time: 0.9479, Backward time: 1.1937, DDP time: 0.0051, Total iter time: 2.2847 [2020-12-31 13:03:18,719][root][INFO] - ===> Epoch[1667](8335/5): Loss -0.7310, Sem -0.7310, dir -0.9750, norm 0.2129 LR: 4.819e-02 Score 98.954 Data time: 0.1384, Forward time: 0.9478, Backward time: 1.1937, DDP time: 0.0051, Total iter time: 2.3087 [2020-12-31 13:03:29,974][root][INFO] - ===> Epoch[1668](8340/5): Loss -0.7308, Sem -0.7308, dir -0.9735, norm 0.2085 LR: 4.816e-02 Score 98.852 Data time: 0.1260, Forward time: 0.9478, Backward time: 1.1936, DDP time: 0.0051, Total iter time: 2.1551 [2020-12-31 13:03:42,140][root][INFO] - ===> Epoch[1669](8345/5): Loss -0.7414, Sem -0.7414, dir -0.9743, norm 0.2012 LR: 4.813e-02 Score 98.934 Data time: 0.2047, Forward time: 0.9479, Backward time: 1.1935, DDP time: 0.0051, Total iter time: 2.3484 [2020-12-31 13:03:54,280][root][INFO] - ===> Epoch[1670](8350/5): Loss -0.7489, Sem -0.7489, dir -0.9743, norm 0.1936 LR: 4.809e-02 Score 98.938 Data time: 0.0872, Forward time: 0.9479, Backward time: 1.1935, DDP time: 0.0051, Total iter time: 2.3528 [2020-12-31 13:04:06,814][root][INFO] - ===> Epoch[1671](8355/5): Loss -0.7490, Sem -0.7490, dir -0.9752, norm 0.1937 LR: 4.806e-02 Score 98.908 Data time: 0.1147, Forward time: 0.9480, Backward time: 1.1935, DDP time: 0.0051, Total iter time: 2.4157 [2020-12-31 13:04:18,779][root][INFO] - ===> Epoch[1672](8360/5): Loss -0.7476, Sem -0.7476, dir -0.9744, norm 0.1939 LR: 4.803e-02 Score 98.888 Data time: 0.1131, Forward time: 0.9480, Backward time: 1.1935, DDP time: 0.0051, Total iter time: 2.3038 [2020-12-31 13:04:30,011][root][INFO] - ===> Epoch[1673](8365/5): Loss -0.7427, Sem -0.7427, dir -0.9749, norm 0.1991 LR: 4.800e-02 Score 98.886 Data time: 0.1042, Forward time: 0.9480, Backward time: 1.1934, DDP time: 0.0051, Total iter time: 2.1581 [2020-12-31 13:04:42,160][root][INFO] - ===> Epoch[1674](8370/5): Loss -0.7334, Sem -0.7334, dir -0.9740, norm 0.2076 LR: 4.796e-02 Score 98.909 Data time: 0.0835, Forward time: 0.9479, Backward time: 1.1935, DDP time: 0.0051, Total iter time: 2.3482 [2020-12-31 13:04:53,957][root][INFO] - ===> Epoch[1675](8375/5): Loss -0.7358, Sem -0.7358, dir -0.9746, norm 0.2072 LR: 4.793e-02 Score 98.939 Data time: 0.1253, Forward time: 0.9480, Backward time: 1.1934, DDP time: 0.0051, Total iter time: 2.2630 [2020-12-31 13:05:05,198][root][INFO] - ===> Epoch[1676](8380/5): Loss -0.7325, Sem -0.7325, dir -0.9743, norm 0.2081 LR: 4.790e-02 Score 98.871 Data time: 0.1115, Forward time: 0.9480, Backward time: 1.1933, DDP time: 0.0051, Total iter time: 2.1637 [2020-12-31 13:05:17,092][root][INFO] - ===> Epoch[1677](8385/5): Loss -0.7311, Sem -0.7311, dir -0.9736, norm 0.2091 LR: 4.787e-02 Score 98.885 Data time: 0.1078, Forward time: 0.9481, Backward time: 1.1932, DDP time: 0.0051, Total iter time: 2.2895 [2020-12-31 13:05:29,694][root][INFO] - ===> Epoch[1678](8390/5): Loss -0.7299, Sem -0.7299, dir -0.9741, norm 0.2100 LR: 4.783e-02 Score 98.873 Data time: 0.0903, Forward time: 0.9481, Backward time: 1.1933, DDP time: 0.0051, Total iter time: 2.4220 [2020-12-31 13:05:42,099][root][INFO] - ===> Epoch[1679](8395/5): Loss -0.7380, Sem -0.7380, dir -0.9745, norm 0.2039 LR: 4.780e-02 Score 98.903 Data time: 0.1021, Forward time: 0.9481, Backward time: 1.1933, DDP time: 0.0051, Total iter time: 2.3895 [2020-12-31 13:05:55,226][root][INFO] - ===> Epoch[1680](8400/5): Loss -0.7305, Sem -0.7305, dir -0.9734, norm 0.2092 LR: 4.777e-02 Score 98.888 Data time: 0.1872, Forward time: 0.9483, Backward time: 1.1933, DDP time: 0.0051, Total iter time: 2.5248 [2020-12-31 13:05:55,229][root][INFO] - ===> Start testing [2020-12-31 13:06:33,804][root][INFO] - 68/68: Data time: 0.0023, Iter time: 0.1198 Loss 0.251 (AVG: 0.516) Score 94.547 (AVG: 89.697) mIOU 68.557 mAP 83.510 mAcc 76.023 IOU: 59.987 0.000 76.902 74.476 92.460 89.892 30.978 76.175 98.058 70.934 80.000 84.732 56.650 mAP: 80.718 7.251 92.914 86.421 98.820 98.306 60.460 94.636 99.812 89.381 92.644 93.809 90.455 mAcc: 80.374 0.000 88.973 81.742 93.699 96.348 33.544 87.569 98.611 81.117 89.711 95.470 61.140 [2020-12-31 13:06:33,805][root][INFO] - Finished test. Elapsed time: 38.5739 [2020-12-31 13:06:37,017][root][INFO] - [2020-12-31 13:06:37,017][root][INFO] - ################################################################ [2020-12-31 13:06:37,017][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 13:06:37,017][root][INFO] - ################################################################ [2020-12-31 13:06:37,017][root][INFO] - clutter : nan nan nan [2020-12-31 13:06:37,017][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 13:06:37,017][root][INFO] - board : 0.774 0.840 0.891 [2020-12-31 13:06:37,017][root][INFO] - bookcase : 0.210 0.341 0.510 [2020-12-31 13:06:37,017][root][INFO] - ceiling : 0.527 0.558 0.620 [2020-12-31 13:06:37,018][root][INFO] - chair : 0.767 0.871 0.896 [2020-12-31 13:06:37,018][root][INFO] - column : 0.165 0.263 0.396 [2020-12-31 13:06:37,018][root][INFO] - door : 0.675 0.854 0.950 [2020-12-31 13:06:37,018][root][INFO] - floor : 0.967 0.979 0.979 [2020-12-31 13:06:37,018][root][INFO] - sofa : 0.520 0.801 0.801 [2020-12-31 13:06:37,018][root][INFO] - table : 0.175 0.348 0.640 [2020-12-31 13:06:37,018][root][INFO] - wall : 0.464 0.707 0.779 [2020-12-31 13:06:37,018][root][INFO] - window : 0.470 0.700 0.775 [2020-12-31 13:06:37,018][root][INFO] - ---------------------------------------------------------------- [2020-12-31 13:06:37,018][root][INFO] - average : 0.476 0.605 0.686 [2020-12-31 13:06:37,018][root][INFO] - [2020-12-31 13:06:37,185][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 13:06:37,754][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter8400.pth [2020-12-31 13:06:48,786][root][INFO] - ===> Epoch[1681](8405/5): Loss -0.7387, Sem -0.7387, dir -0.9752, norm 0.2030 LR: 4.774e-02 Score 98.883 Data time: 0.0815, Forward time: 0.9482, Backward time: 1.1933, DDP time: 0.0051, Total iter time: 2.1141 [2020-12-31 13:07:00,594][root][INFO] - ===> Epoch[1682](8410/5): Loss -0.7332, Sem -0.7332, dir -0.9730, norm 0.2057 LR: 4.770e-02 Score 98.866 Data time: 0.0838, Forward time: 0.9481, Backward time: 1.1933, DDP time: 0.0051, Total iter time: 2.2582 [2020-12-31 13:07:11,888][root][INFO] - ===> Epoch[1683](8415/5): Loss -0.7404, Sem -0.7404, dir -0.9741, norm 0.1994 LR: 4.767e-02 Score 98.850 Data time: 0.1234, Forward time: 0.9481, Backward time: 1.1932, DDP time: 0.0051, Total iter time: 2.1648 [2020-12-31 13:07:23,203][root][INFO] - ===> Epoch[1684](8420/5): Loss -0.7398, Sem -0.7398, dir -0.9739, norm 0.2021 LR: 4.764e-02 Score 98.931 Data time: 0.1269, Forward time: 0.9481, Backward time: 1.1932, DDP time: 0.0051, Total iter time: 2.1903 [2020-12-31 13:07:35,471][root][INFO] - ===> Epoch[1685](8425/5): Loss -0.7404, Sem -0.7404, dir -0.9748, norm 0.2014 LR: 4.761e-02 Score 98.894 Data time: 0.2291, Forward time: 0.9482, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.3535 [2020-12-31 13:07:47,630][root][INFO] - ===> Epoch[1686](8430/5): Loss -0.7369, Sem -0.7369, dir -0.9731, norm 0.2026 LR: 4.757e-02 Score 98.872 Data time: 0.0922, Forward time: 0.9482, Backward time: 1.1931, DDP time: 0.0051, Total iter time: 2.3508 [2020-12-31 13:07:59,391][root][INFO] - ===> Epoch[1687](8435/5): Loss -0.7251, Sem -0.7251, dir -0.9729, norm 0.2094 LR: 4.754e-02 Score 98.750 Data time: 0.0757, Forward time: 0.9481, Backward time: 1.1931, DDP time: 0.0051, Total iter time: 2.2740 [2020-12-31 13:08:11,462][root][INFO] - ===> Epoch[1688](8440/5): Loss -0.7196, Sem -0.7196, dir -0.9722, norm 0.2138 LR: 4.751e-02 Score 98.713 Data time: 0.1239, Forward time: 0.9481, Backward time: 1.1931, DDP time: 0.0051, Total iter time: 2.3178 [2020-12-31 13:08:22,709][root][INFO] - ===> Epoch[1689](8445/5): Loss -0.7327, Sem -0.7327, dir -0.9738, norm 0.2035 LR: 4.748e-02 Score 98.757 Data time: 0.0851, Forward time: 0.9481, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.1562 [2020-12-31 13:08:33,817][root][INFO] - ===> Epoch[1690](8450/5): Loss -0.7220, Sem -0.7220, dir -0.9721, norm 0.2116 LR: 4.744e-02 Score 98.720 Data time: 0.0874, Forward time: 0.9481, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.1427 [2020-12-31 13:08:46,215][root][INFO] - ===> Epoch[1691](8455/5): Loss -0.7293, Sem -0.7293, dir -0.9729, norm 0.2067 LR: 4.741e-02 Score 98.773 Data time: 0.1737, Forward time: 0.9482, Backward time: 1.1929, DDP time: 0.0051, Total iter time: 2.3797 [2020-12-31 13:08:58,722][root][INFO] - ===> Epoch[1692](8460/5): Loss -0.7320, Sem -0.7320, dir -0.9736, norm 0.2062 LR: 4.738e-02 Score 98.834 Data time: 0.1166, Forward time: 0.9482, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.4200 [2020-12-31 13:09:10,969][root][INFO] - ===> Epoch[1693](8465/5): Loss -0.7412, Sem -0.7412, dir -0.9736, norm 0.1969 LR: 4.734e-02 Score 98.805 Data time: 0.1518, Forward time: 0.9482, Backward time: 1.1929, DDP time: 0.0051, Total iter time: 2.3536 [2020-12-31 13:09:23,122][root][INFO] - ===> Epoch[1694](8470/5): Loss -0.7375, Sem -0.7375, dir -0.9741, norm 0.2033 LR: 4.731e-02 Score 98.891 Data time: 0.1062, Forward time: 0.9482, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.3445 [2020-12-31 13:09:34,724][root][INFO] - ===> Epoch[1695](8475/5): Loss -0.7386, Sem -0.7386, dir -0.9739, norm 0.1996 LR: 4.728e-02 Score 98.818 Data time: 0.0856, Forward time: 0.9482, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.2422 [2020-12-31 13:09:47,139][root][INFO] - ===> Epoch[1696](8480/5): Loss -0.7396, Sem -0.7396, dir -0.9742, norm 0.2014 LR: 4.725e-02 Score 98.878 Data time: 0.0887, Forward time: 0.9481, Backward time: 1.1931, DDP time: 0.0051, Total iter time: 2.3985 [2020-12-31 13:09:59,318][root][INFO] - ===> Epoch[1697](8485/5): Loss -0.7394, Sem -0.7394, dir -0.9744, norm 0.2027 LR: 4.721e-02 Score 98.930 Data time: 0.1315, Forward time: 0.9481, Backward time: 1.1931, DDP time: 0.0051, Total iter time: 2.3377 [2020-12-31 13:10:11,010][root][INFO] - ===> Epoch[1698](8490/5): Loss -0.7459, Sem -0.7459, dir -0.9750, norm 0.1974 LR: 4.718e-02 Score 98.940 Data time: 0.0992, Forward time: 0.9481, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.2338 [2020-12-31 13:10:22,180][root][INFO] - ===> Epoch[1699](8495/5): Loss -0.7423, Sem -0.7423, dir -0.9740, norm 0.1987 LR: 4.715e-02 Score 98.894 Data time: 0.0844, Forward time: 0.9481, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.1529 [2020-12-31 13:10:34,291][root][INFO] - ===> Epoch[1700](8500/5): Loss -0.7440, Sem -0.7440, dir -0.9742, norm 0.1960 LR: 4.712e-02 Score 98.865 Data time: 0.1100, Forward time: 0.9481, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.3359 [2020-12-31 13:10:45,899][root][INFO] - ===> Epoch[1701](8505/5): Loss -0.7437, Sem -0.7437, dir -0.9749, norm 0.1984 LR: 4.708e-02 Score 98.895 Data time: 0.1642, Forward time: 0.9482, Backward time: 1.1929, DDP time: 0.0051, Total iter time: 2.2365 [2020-12-31 13:10:58,425][root][INFO] - ===> Epoch[1702](8510/5): Loss -0.7412, Sem -0.7412, dir -0.9746, norm 0.1997 LR: 4.705e-02 Score 98.877 Data time: 0.1009, Forward time: 0.9482, Backward time: 1.1929, DDP time: 0.0051, Total iter time: 2.4158 [2020-12-31 13:11:10,606][root][INFO] - ===> Epoch[1703](8515/5): Loss -0.7465, Sem -0.7465, dir -0.9751, norm 0.1967 LR: 4.702e-02 Score 98.934 Data time: 0.0797, Forward time: 0.9482, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.3368 [2020-12-31 13:11:22,464][root][INFO] - ===> Epoch[1704](8520/5): Loss -0.7363, Sem -0.7363, dir -0.9742, norm 0.2043 LR: 4.699e-02 Score 98.883 Data time: 0.1078, Forward time: 0.9481, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.2859 [2020-12-31 13:11:34,889][root][INFO] - ===> Epoch[1705](8525/5): Loss -0.7301, Sem -0.7301, dir -0.9743, norm 0.2100 LR: 4.695e-02 Score 98.866 Data time: 0.0888, Forward time: 0.9481, Backward time: 1.1931, DDP time: 0.0051, Total iter time: 2.3919 [2020-12-31 13:11:46,371][root][INFO] - ===> Epoch[1706](8530/5): Loss -0.7388, Sem -0.7388, dir -0.9738, norm 0.2002 LR: 4.692e-02 Score 98.837 Data time: 0.0832, Forward time: 0.9481, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.2003 [2020-12-31 13:11:59,334][root][INFO] - ===> Epoch[1707](8535/5): Loss -0.7448, Sem -0.7448, dir -0.9748, norm 0.1977 LR: 4.689e-02 Score 98.921 Data time: 0.0822, Forward time: 0.9481, Backward time: 1.1932, DDP time: 0.0051, Total iter time: 2.5002 [2020-12-31 13:12:11,596][root][INFO] - ===> Epoch[1708](8540/5): Loss -0.7472, Sem -0.7472, dir -0.9743, norm 0.1938 LR: 4.686e-02 Score 98.881 Data time: 0.1180, Forward time: 0.9481, Backward time: 1.1932, DDP time: 0.0051, Total iter time: 2.3501 [2020-12-31 13:12:23,263][root][INFO] - ===> Epoch[1709](8545/5): Loss -0.7475, Sem -0.7475, dir -0.9746, norm 0.1946 LR: 4.682e-02 Score 98.915 Data time: 0.0785, Forward time: 0.9480, Backward time: 1.1932, DDP time: 0.0051, Total iter time: 2.2565 [2020-12-31 13:12:35,290][root][INFO] - ===> Epoch[1710](8550/5): Loss -0.7347, Sem -0.7347, dir -0.9741, norm 0.2049 LR: 4.679e-02 Score 98.855 Data time: 0.1059, Forward time: 0.9480, Backward time: 1.1932, DDP time: 0.0051, Total iter time: 2.3142 [2020-12-31 13:12:48,988][root][INFO] - ===> Epoch[1711](8555/5): Loss -0.7417, Sem -0.7417, dir -0.9746, norm 0.1998 LR: 4.676e-02 Score 98.901 Data time: 0.1452, Forward time: 0.9481, Backward time: 1.1933, DDP time: 0.0051, Total iter time: 2.6507 [2020-12-31 13:12:59,857][root][INFO] - ===> Epoch[1712](8560/5): Loss -0.7480, Sem -0.7480, dir -0.9749, norm 0.1931 LR: 4.673e-02 Score 98.877 Data time: 0.0866, Forward time: 0.9480, Backward time: 1.1933, DDP time: 0.0051, Total iter time: 2.0937 [2020-12-31 13:13:11,361][root][INFO] - ===> Epoch[1713](8565/5): Loss -0.7367, Sem -0.7367, dir -0.9738, norm 0.2033 LR: 4.669e-02 Score 98.865 Data time: 0.0981, Forward time: 0.9481, Backward time: 1.1932, DDP time: 0.0051, Total iter time: 2.2089 [2020-12-31 13:13:24,044][root][INFO] - ===> Epoch[1714](8570/5): Loss -0.7405, Sem -0.7405, dir -0.9742, norm 0.2003 LR: 4.666e-02 Score 98.881 Data time: 0.0787, Forward time: 0.9480, Backward time: 1.1933, DDP time: 0.0051, Total iter time: 2.4425 [2020-12-31 13:13:37,317][root][INFO] - ===> Epoch[1715](8575/5): Loss -0.7493, Sem -0.7493, dir -0.9757, norm 0.1941 LR: 4.663e-02 Score 98.927 Data time: 0.1043, Forward time: 0.9480, Backward time: 1.1935, DDP time: 0.0051, Total iter time: 2.5566 [2020-12-31 13:13:49,020][root][INFO] - ===> Epoch[1716](8580/5): Loss -0.7484, Sem -0.7484, dir -0.9745, norm 0.1928 LR: 4.659e-02 Score 98.885 Data time: 0.1393, Forward time: 0.9480, Backward time: 1.1934, DDP time: 0.0051, Total iter time: 2.2429 [2020-12-31 13:14:01,013][root][INFO] - ===> Epoch[1717](8585/5): Loss -0.7494, Sem -0.7494, dir -0.9748, norm 0.1936 LR: 4.656e-02 Score 98.937 Data time: 0.1207, Forward time: 0.9480, Backward time: 1.1934, DDP time: 0.0051, Total iter time: 2.3110 [2020-12-31 13:14:12,731][root][INFO] - ===> Epoch[1718](8590/5): Loss -0.7476, Sem -0.7476, dir -0.9750, norm 0.1944 LR: 4.653e-02 Score 98.889 Data time: 0.1218, Forward time: 0.9481, Backward time: 1.1933, DDP time: 0.0051, Total iter time: 2.2638 [2020-12-31 13:14:24,349][root][INFO] - ===> Epoch[1719](8595/5): Loss -0.7430, Sem -0.7430, dir -0.9738, norm 0.1976 LR: 4.650e-02 Score 98.889 Data time: 0.1172, Forward time: 0.9481, Backward time: 1.1933, DDP time: 0.0051, Total iter time: 2.2430 [2020-12-31 13:14:35,715][root][INFO] - ===> Epoch[1720](8600/5): Loss -0.7516, Sem -0.7516, dir -0.9739, norm 0.1890 LR: 4.646e-02 Score 98.881 Data time: 0.1354, Forward time: 0.9481, Backward time: 1.1932, DDP time: 0.0051, Total iter time: 2.1890 [2020-12-31 13:14:35,717][root][INFO] - ===> Start testing [2020-12-31 13:15:14,237][root][INFO] - 68/68: Data time: 0.3858, Iter time: 0.3057 Loss 0.558 (AVG: 0.516) Score 88.347 (AVG: 89.846) mIOU 68.593 mAP 83.056 mAcc 75.520 IOU: 59.895 0.000 80.369 72.786 93.611 91.155 32.368 73.458 98.479 67.774 79.766 85.439 56.606 mAP: 80.534 4.337 93.803 85.918 98.955 98.614 60.417 92.614 99.823 85.967 93.223 94.335 91.189 mAcc: 80.502 0.000 87.472 80.161 94.736 96.099 36.263 84.583 99.390 76.066 89.977 95.782 60.733 [2020-12-31 13:15:14,238][root][INFO] - Finished test. Elapsed time: 38.5188 [2020-12-31 13:15:18,061][root][INFO] - [2020-12-31 13:15:18,061][root][INFO] - ################################################################ [2020-12-31 13:15:18,061][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 13:15:18,061][root][INFO] - ################################################################ [2020-12-31 13:15:18,061][root][INFO] - clutter : nan nan nan [2020-12-31 13:15:18,062][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 13:15:18,062][root][INFO] - board : 0.764 0.822 0.894 [2020-12-31 13:15:18,062][root][INFO] - bookcase : 0.195 0.335 0.497 [2020-12-31 13:15:18,062][root][INFO] - ceiling : 0.714 0.789 0.811 [2020-12-31 13:15:18,062][root][INFO] - chair : 0.788 0.875 0.902 [2020-12-31 13:15:18,062][root][INFO] - column : 0.162 0.250 0.364 [2020-12-31 13:15:18,062][root][INFO] - door : 0.448 0.650 0.837 [2020-12-31 13:15:18,062][root][INFO] - floor : 0.995 1.000 1.000 [2020-12-31 13:15:18,062][root][INFO] - sofa : 0.439 0.545 0.818 [2020-12-31 13:15:18,062][root][INFO] - table : 0.190 0.331 0.649 [2020-12-31 13:15:18,062][root][INFO] - wall : 0.458 0.701 0.778 [2020-12-31 13:15:18,062][root][INFO] - window : 0.514 0.747 0.766 [2020-12-31 13:15:18,062][root][INFO] - ---------------------------------------------------------------- [2020-12-31 13:15:18,062][root][INFO] - average : 0.472 0.587 0.693 [2020-12-31 13:15:18,062][root][INFO] - [2020-12-31 13:15:18,235][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 13:15:18,867][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2020-12-31 13:15:18,867][root][INFO] - Current best mIoU: 68.593 at iter 1 [2020-12-31 13:15:19,419][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter8600.pth [2020-12-31 13:15:30,954][root][INFO] - ===> Epoch[1721](8605/5): Loss -0.7499, Sem -0.7499, dir -0.9754, norm 0.1924 LR: 4.643e-02 Score 98.913 Data time: 0.0929, Forward time: 0.9481, Backward time: 1.1932, DDP time: 0.0051, Total iter time: 2.2013 [2020-12-31 13:15:43,335][root][INFO] - ===> Epoch[1722](8610/5): Loss -0.7446, Sem -0.7446, dir -0.9741, norm 0.1966 LR: 4.640e-02 Score 98.892 Data time: 0.1300, Forward time: 0.9482, Backward time: 1.1931, DDP time: 0.0051, Total iter time: 2.3907 [2020-12-31 13:15:54,981][root][INFO] - ===> Epoch[1723](8615/5): Loss -0.7415, Sem -0.7415, dir -0.9744, norm 0.1986 LR: 4.637e-02 Score 98.852 Data time: 0.2070, Forward time: 0.9482, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.2321 [2020-12-31 13:16:06,530][root][INFO] - ===> Epoch[1724](8620/5): Loss -0.7449, Sem -0.7449, dir -0.9744, norm 0.1971 LR: 4.633e-02 Score 98.909 Data time: 0.0998, Forward time: 0.9482, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.2318 [2020-12-31 13:16:18,607][root][INFO] - ===> Epoch[1725](8625/5): Loss -0.7464, Sem -0.7464, dir -0.9749, norm 0.1958 LR: 4.630e-02 Score 98.900 Data time: 0.2014, Forward time: 0.9483, Backward time: 1.1929, DDP time: 0.0051, Total iter time: 2.3263 [2020-12-31 13:16:30,586][root][INFO] - ===> Epoch[1726](8630/5): Loss -0.7449, Sem -0.7449, dir -0.9742, norm 0.1971 LR: 4.627e-02 Score 98.915 Data time: 0.0889, Forward time: 0.9483, Backward time: 1.1929, DDP time: 0.0051, Total iter time: 2.3217 [2020-12-31 13:16:42,074][root][INFO] - ===> Epoch[1727](8635/5): Loss -0.7459, Sem -0.7459, dir -0.9747, norm 0.1952 LR: 4.624e-02 Score 98.885 Data time: 0.0927, Forward time: 0.9482, Backward time: 1.1929, DDP time: 0.0051, Total iter time: 2.2080 [2020-12-31 13:16:55,104][root][INFO] - ===> Epoch[1728](8640/5): Loss -0.7404, Sem -0.7404, dir -0.9748, norm 0.2014 LR: 4.620e-02 Score 98.902 Data time: 0.1141, Forward time: 0.9482, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.5171 [2020-12-31 13:17:06,917][root][INFO] - ===> Epoch[1729](8645/5): Loss -0.7441, Sem -0.7441, dir -0.9751, norm 0.1993 LR: 4.617e-02 Score 98.933 Data time: 0.1118, Forward time: 0.9482, Backward time: 1.1929, DDP time: 0.0051, Total iter time: 2.2670 [2020-12-31 13:17:18,220][root][INFO] - ===> Epoch[1730](8650/5): Loss -0.7381, Sem -0.7381, dir -0.9740, norm 0.2020 LR: 4.614e-02 Score 98.864 Data time: 0.1172, Forward time: 0.9483, Backward time: 1.1928, DDP time: 0.0051, Total iter time: 2.1707 [2020-12-31 13:17:30,037][root][INFO] - ===> Epoch[1731](8655/5): Loss -0.7555, Sem -0.7555, dir -0.9761, norm 0.1891 LR: 4.610e-02 Score 98.933 Data time: 0.2018, Forward time: 0.9483, Backward time: 1.1927, DDP time: 0.0051, Total iter time: 2.2735 [2020-12-31 13:17:43,836][root][INFO] - ===> Epoch[1732](8660/5): Loss -0.7436, Sem -0.7436, dir -0.9739, norm 0.1969 LR: 4.607e-02 Score 98.874 Data time: 0.1021, Forward time: 0.9483, Backward time: 1.1929, DDP time: 0.0051, Total iter time: 2.6763 [2020-12-31 13:17:55,033][root][INFO] - ===> Epoch[1733](8665/5): Loss -0.7497, Sem -0.7497, dir -0.9754, norm 0.1931 LR: 4.604e-02 Score 98.898 Data time: 0.0763, Forward time: 0.9482, Backward time: 1.1929, DDP time: 0.0051, Total iter time: 2.1600 [2020-12-31 13:18:06,875][root][INFO] - ===> Epoch[1734](8670/5): Loss -0.7411, Sem -0.7411, dir -0.9745, norm 0.1993 LR: 4.601e-02 Score 98.857 Data time: 0.0722, Forward time: 0.9481, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.2790 [2020-12-31 13:18:19,317][root][INFO] - ===> Epoch[1735](8675/5): Loss -0.7372, Sem -0.7372, dir -0.9737, norm 0.2027 LR: 4.597e-02 Score 98.872 Data time: 0.1000, Forward time: 0.9482, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.4187 [2020-12-31 13:18:31,965][root][INFO] - ===> Epoch[1736](8680/5): Loss -0.7475, Sem -0.7475, dir -0.9752, norm 0.1959 LR: 4.594e-02 Score 98.933 Data time: 0.2026, Forward time: 0.9482, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.4316 [2020-12-31 13:18:44,077][root][INFO] - ===> Epoch[1737](8685/5): Loss -0.7340, Sem -0.7340, dir -0.9746, norm 0.2068 LR: 4.591e-02 Score 98.866 Data time: 0.1186, Forward time: 0.9482, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.3174 [2020-12-31 13:18:57,033][root][INFO] - ===> Epoch[1738](8690/5): Loss -0.7364, Sem -0.7364, dir -0.9748, norm 0.2063 LR: 4.588e-02 Score 98.927 Data time: 0.1546, Forward time: 0.9482, Backward time: 1.1931, DDP time: 0.0051, Total iter time: 2.4902 [2020-12-31 13:19:08,504][root][INFO] - ===> Epoch[1739](8695/5): Loss -0.7415, Sem -0.7415, dir -0.9751, norm 0.2013 LR: 4.584e-02 Score 98.913 Data time: 0.0987, Forward time: 0.9482, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.2056 [2020-12-31 13:19:20,502][root][INFO] - ===> Epoch[1740](8700/5): Loss -0.7385, Sem -0.7385, dir -0.9751, norm 0.2028 LR: 4.581e-02 Score 98.868 Data time: 0.1054, Forward time: 0.9482, Backward time: 1.1931, DDP time: 0.0051, Total iter time: 2.3061 [2020-12-31 13:19:31,720][root][INFO] - ===> Epoch[1741](8705/5): Loss -0.7464, Sem -0.7464, dir -0.9744, norm 0.1952 LR: 4.578e-02 Score 98.905 Data time: 0.0863, Forward time: 0.9482, Backward time: 1.1930, DDP time: 0.0051, Total iter time: 2.1536 [2020-12-31 13:19:43,132][root][INFO] - ===> Epoch[1742](8710/5): Loss -0.7504, Sem -0.7504, dir -0.9757, norm 0.1939 LR: 4.574e-02 Score 98.946 Data time: 0.1318, Forward time: 0.9482, Backward time: 1.1929, DDP time: 0.0051, Total iter time: 2.1924 [2020-12-31 13:19:54,715][root][INFO] - ===> Epoch[1743](8715/5): Loss -0.7433, Sem -0.7433, dir -0.9745, norm 0.1976 LR: 4.571e-02 Score 98.887 Data time: 0.1511, Forward time: 0.9482, Backward time: 1.1928, DDP time: 0.0051, Total iter time: 2.2202 [2020-12-31 13:20:07,009][root][INFO] - ===> Epoch[1744](8720/5): Loss -0.7415, Sem -0.7415, dir -0.9751, norm 0.2025 LR: 4.568e-02 Score 98.960 Data time: 0.0938, Forward time: 0.9482, Backward time: 1.1928, DDP time: 0.0051, Total iter time: 2.3760 [2020-12-31 13:20:19,281][root][INFO] - ===> Epoch[1745](8725/5): Loss -0.7403, Sem -0.7403, dir -0.9748, norm 0.2011 LR: 4.565e-02 Score 98.881 Data time: 0.1604, Forward time: 0.9482, Backward time: 1.1928, DDP time: 0.0051, Total iter time: 2.3598 [2020-12-31 13:20:30,999][root][INFO] - ===> Epoch[1746](8730/5): Loss -0.7448, Sem -0.7448, dir -0.9752, norm 0.1971 LR: 4.561e-02 Score 98.882 Data time: 0.0894, Forward time: 0.9482, Backward time: 1.1928, DDP time: 0.0051, Total iter time: 2.2572 [2020-12-31 13:20:43,025][root][INFO] - ===> Epoch[1747](8735/5): Loss -0.7388, Sem -0.7388, dir -0.9743, norm 0.2022 LR: 4.558e-02 Score 98.880 Data time: 0.0839, Forward time: 0.9482, Backward time: 1.1929, DDP time: 0.0051, Total iter time: 2.3032 [2020-12-31 13:20:55,138][root][INFO] - ===> Epoch[1748](8740/5): Loss -0.7487, Sem -0.7487, dir -0.9751, norm 0.1935 LR: 4.555e-02 Score 98.905 Data time: 0.1018, Forward time: 0.9482, Backward time: 1.1928, DDP time: 0.0051, Total iter time: 2.3287 [2020-12-31 13:21:06,827][root][INFO] - ===> Epoch[1749](8745/5): Loss -0.7370, Sem -0.7370, dir -0.9744, norm 0.2037 LR: 4.552e-02 Score 98.869 Data time: 0.1320, Forward time: 0.9483, Backward time: 1.1927, DDP time: 0.0051, Total iter time: 2.2407 [2020-12-31 13:21:18,462][root][INFO] - ===> Epoch[1750](8750/5): Loss -0.7384, Sem -0.7384, dir -0.9748, norm 0.2023 LR: 4.548e-02 Score 98.881 Data time: 0.1064, Forward time: 0.9484, Backward time: 1.1926, DDP time: 0.0051, Total iter time: 2.2346 [2020-12-31 13:21:30,604][root][INFO] - ===> Epoch[1751](8755/5): Loss -0.7455, Sem -0.7455, dir -0.9748, norm 0.1962 LR: 4.545e-02 Score 98.887 Data time: 0.0838, Forward time: 0.9483, Backward time: 1.1927, DDP time: 0.0051, Total iter time: 2.3365 [2020-12-31 13:21:42,667][root][INFO] - ===> Epoch[1752](8760/5): Loss -0.7428, Sem -0.7428, dir -0.9752, norm 0.1998 LR: 4.542e-02 Score 98.916 Data time: 0.1100, Forward time: 0.9484, Backward time: 1.1927, DDP time: 0.0051, Total iter time: 2.3324 [2020-12-31 13:21:54,219][root][INFO] - ===> Epoch[1753](8765/5): Loss -0.7481, Sem -0.7481, dir -0.9747, norm 0.1944 LR: 4.538e-02 Score 98.921 Data time: 0.1819, Forward time: 0.9483, Backward time: 1.1926, DDP time: 0.0051, Total iter time: 2.2230 [2020-12-31 13:22:05,322][root][INFO] - ===> Epoch[1754](8770/5): Loss -0.7364, Sem -0.7364, dir -0.9736, norm 0.2031 LR: 4.535e-02 Score 98.857 Data time: 0.0894, Forward time: 0.9483, Backward time: 1.1925, DDP time: 0.0051, Total iter time: 2.1247 [2020-12-31 13:22:17,665][root][INFO] - ===> Epoch[1755](8775/5): Loss -0.7475, Sem -0.7475, dir -0.9744, norm 0.1932 LR: 4.532e-02 Score 98.869 Data time: 0.1847, Forward time: 0.9484, Backward time: 1.1925, DDP time: 0.0051, Total iter time: 2.3767 [2020-12-31 13:22:29,105][root][INFO] - ===> Epoch[1756](8780/5): Loss -0.7386, Sem -0.7386, dir -0.9745, norm 0.2014 LR: 4.529e-02 Score 98.849 Data time: 0.0938, Forward time: 0.9484, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.2055 [2020-12-31 13:22:41,558][root][INFO] - ===> Epoch[1757](8785/5): Loss -0.7352, Sem -0.7352, dir -0.9740, norm 0.2023 LR: 4.525e-02 Score 98.810 Data time: 0.0787, Forward time: 0.9483, Backward time: 1.1925, DDP time: 0.0051, Total iter time: 2.4068 [2020-12-31 13:22:53,953][root][INFO] - ===> Epoch[1758](8790/5): Loss -0.7366, Sem -0.7366, dir -0.9746, norm 0.2032 LR: 4.522e-02 Score 98.842 Data time: 0.0897, Forward time: 0.9483, Backward time: 1.1926, DDP time: 0.0051, Total iter time: 2.3882 [2020-12-31 13:23:06,316][root][INFO] - ===> Epoch[1759](8795/5): Loss -0.7388, Sem -0.7388, dir -0.9741, norm 0.2012 LR: 4.519e-02 Score 98.870 Data time: 0.1845, Forward time: 0.9483, Backward time: 1.1926, DDP time: 0.0051, Total iter time: 2.3784 [2020-12-31 13:23:18,457][root][INFO] - ===> Epoch[1760](8800/5): Loss -0.7435, Sem -0.7435, dir -0.9747, norm 0.1978 LR: 4.516e-02 Score 98.884 Data time: 0.0939, Forward time: 0.9484, Backward time: 1.1926, DDP time: 0.0051, Total iter time: 2.3304 [2020-12-31 13:23:18,459][root][INFO] - ===> Start testing [2020-12-31 13:23:57,377][root][INFO] - 68/68: Data time: 0.0024, Iter time: 0.1319 Loss 0.202 (AVG: 0.500) Score 95.730 (AVG: 89.824) mIOU 68.257 mAP 83.316 mAcc 76.335 IOU: 59.972 0.000 81.835 73.141 94.309 91.249 31.748 76.557 98.237 58.437 79.769 84.924 57.160 mAP: 80.596 6.476 92.934 86.252 99.042 98.634 61.573 94.198 99.799 85.375 93.027 94.208 90.995 mAcc: 78.703 0.000 87.550 82.802 95.761 96.966 40.076 86.504 98.952 81.215 87.381 94.404 62.046 [2020-12-31 13:23:57,378][root][INFO] - Finished test. Elapsed time: 38.9168 [2020-12-31 13:24:01,243][root][INFO] - [2020-12-31 13:24:01,243][root][INFO] - ################################################################ [2020-12-31 13:24:01,244][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 13:24:01,244][root][INFO] - ################################################################ [2020-12-31 13:24:01,244][root][INFO] - clutter : nan nan nan [2020-12-31 13:24:01,244][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 13:24:01,244][root][INFO] - board : 0.746 0.829 0.904 [2020-12-31 13:24:01,244][root][INFO] - bookcase : 0.209 0.345 0.516 [2020-12-31 13:24:01,244][root][INFO] - ceiling : 0.572 0.619 0.667 [2020-12-31 13:24:01,244][root][INFO] - chair : 0.778 0.890 0.908 [2020-12-31 13:24:01,244][root][INFO] - column : 0.132 0.209 0.404 [2020-12-31 13:24:01,244][root][INFO] - door : 0.625 0.783 0.908 [2020-12-31 13:24:01,244][root][INFO] - floor : 0.871 0.899 0.910 [2020-12-31 13:24:01,244][root][INFO] - sofa : 0.487 0.704 0.772 [2020-12-31 13:24:01,244][root][INFO] - table : 0.209 0.434 0.659 [2020-12-31 13:24:01,244][root][INFO] - wall : 0.477 0.707 0.782 [2020-12-31 13:24:01,244][root][INFO] - window : 0.506 0.741 0.760 [2020-12-31 13:24:01,244][root][INFO] - ---------------------------------------------------------------- [2020-12-31 13:24:01,244][root][INFO] - average : 0.468 0.597 0.682 [2020-12-31 13:24:01,244][root][INFO] - [2020-12-31 13:24:01,399][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 13:24:01,966][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter8800.pth [2020-12-31 13:24:13,534][root][INFO] - ===> Epoch[1761](8805/5): Loss -0.7440, Sem -0.7440, dir -0.9746, norm 0.1977 LR: 4.512e-02 Score 98.894 Data time: 0.1251, Forward time: 0.9484, Backward time: 1.1925, DDP time: 0.0051, Total iter time: 2.2038 [2020-12-31 13:24:25,183][root][INFO] - ===> Epoch[1762](8810/5): Loss -0.7385, Sem -0.7385, dir -0.9740, norm 0.2006 LR: 4.509e-02 Score 98.837 Data time: 0.1054, Forward time: 0.9484, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.2449 [2020-12-31 13:24:37,504][root][INFO] - ===> Epoch[1763](8815/5): Loss -0.7458, Sem -0.7458, dir -0.9753, norm 0.1982 LR: 4.506e-02 Score 98.954 Data time: 0.0984, Forward time: 0.9484, Backward time: 1.1925, DDP time: 0.0051, Total iter time: 2.3703 [2020-12-31 13:24:48,992][root][INFO] - ===> Epoch[1764](8820/5): Loss -0.7425, Sem -0.7425, dir -0.9750, norm 0.2004 LR: 4.502e-02 Score 98.930 Data time: 0.1077, Forward time: 0.9484, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.1953 [2020-12-31 13:25:00,408][root][INFO] - ===> Epoch[1765](8825/5): Loss -0.7477, Sem -0.7477, dir -0.9744, norm 0.1932 LR: 4.499e-02 Score 98.882 Data time: 0.1134, Forward time: 0.9483, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.1928 [2020-12-31 13:25:12,657][root][INFO] - ===> Epoch[1766](8830/5): Loss -0.7376, Sem -0.7376, dir -0.9755, norm 0.2055 LR: 4.496e-02 Score 98.930 Data time: 0.1292, Forward time: 0.9484, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.3524 [2020-12-31 13:25:24,081][root][INFO] - ===> Epoch[1767](8835/5): Loss -0.7364, Sem -0.7364, dir -0.9739, norm 0.2018 LR: 4.493e-02 Score 98.850 Data time: 0.1013, Forward time: 0.9483, Backward time: 1.1923, DDP time: 0.0051, Total iter time: 2.1838 [2020-12-31 13:25:36,070][root][INFO] - ===> Epoch[1768](8840/5): Loss -0.7375, Sem -0.7375, dir -0.9742, norm 0.2028 LR: 4.489e-02 Score 98.890 Data time: 0.0904, Forward time: 0.9483, Backward time: 1.1923, DDP time: 0.0051, Total iter time: 2.2958 [2020-12-31 13:25:49,279][root][INFO] - ===> Epoch[1769](8845/5): Loss -0.7439, Sem -0.7439, dir -0.9746, norm 0.1967 LR: 4.486e-02 Score 98.871 Data time: 0.1185, Forward time: 0.9483, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.5480 [2020-12-31 13:26:00,649][root][INFO] - ===> Epoch[1770](8850/5): Loss -0.7498, Sem -0.7498, dir -0.9748, norm 0.1913 LR: 4.483e-02 Score 98.869 Data time: 0.1201, Forward time: 0.9483, Backward time: 1.1923, DDP time: 0.0051, Total iter time: 2.1635 [2020-12-31 13:26:11,707][root][INFO] - ===> Epoch[1771](8855/5): Loss -0.7427, Sem -0.7427, dir -0.9747, norm 0.1982 LR: 4.479e-02 Score 98.876 Data time: 0.1350, Forward time: 0.9483, Backward time: 1.1923, DDP time: 0.0051, Total iter time: 2.1146 [2020-12-31 13:26:23,259][root][INFO] - ===> Epoch[1772](8860/5): Loss -0.7468, Sem -0.7468, dir -0.9754, norm 0.1973 LR: 4.476e-02 Score 98.959 Data time: 0.0893, Forward time: 0.9483, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.2108 [2020-12-31 13:26:36,063][root][INFO] - ===> Epoch[1773](8865/5): Loss -0.7381, Sem -0.7381, dir -0.9750, norm 0.2035 LR: 4.473e-02 Score 98.895 Data time: 0.1335, Forward time: 0.9484, Backward time: 1.1923, DDP time: 0.0051, Total iter time: 2.4679 [2020-12-31 13:26:47,984][root][INFO] - ===> Epoch[1774](8870/5): Loss -0.7478, Sem -0.7478, dir -0.9750, norm 0.1938 LR: 4.470e-02 Score 98.877 Data time: 0.0911, Forward time: 0.9483, Backward time: 1.1923, DDP time: 0.0051, Total iter time: 2.3012 [2020-12-31 13:26:59,599][root][INFO] - ===> Epoch[1775](8875/5): Loss -0.7441, Sem -0.7441, dir -0.9749, norm 0.1992 LR: 4.466e-02 Score 98.942 Data time: 0.1914, Forward time: 0.9483, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.2257 [2020-12-31 13:27:11,431][root][INFO] - ===> Epoch[1776](8880/5): Loss -0.7412, Sem -0.7412, dir -0.9741, norm 0.1997 LR: 4.463e-02 Score 98.874 Data time: 0.1397, Forward time: 0.9483, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.2659 [2020-12-31 13:27:22,544][root][INFO] - ===> Epoch[1777](8885/5): Loss -0.7423, Sem -0.7423, dir -0.9756, norm 0.2017 LR: 4.460e-02 Score 98.944 Data time: 0.1133, Forward time: 0.9483, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.1306 [2020-12-31 13:27:34,671][root][INFO] - ===> Epoch[1778](8890/5): Loss -0.7316, Sem -0.7316, dir -0.9738, norm 0.2066 LR: 4.456e-02 Score 98.830 Data time: 0.1186, Forward time: 0.9483, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.3320 [2020-12-31 13:27:46,171][root][INFO] - ===> Epoch[1779](8895/5): Loss -0.7441, Sem -0.7441, dir -0.9745, norm 0.1990 LR: 4.453e-02 Score 98.948 Data time: 0.1124, Forward time: 0.9483, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.2198 [2020-12-31 13:27:57,910][root][INFO] - ===> Epoch[1780](8900/5): Loss -0.7439, Sem -0.7439, dir -0.9746, norm 0.1967 LR: 4.450e-02 Score 98.857 Data time: 0.0769, Forward time: 0.9482, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.2687 [2020-12-31 13:28:09,384][root][INFO] - ===> Epoch[1781](8905/5): Loss -0.7507, Sem -0.7507, dir -0.9759, norm 0.1936 LR: 4.447e-02 Score 98.939 Data time: 0.1020, Forward time: 0.9483, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.2024 [2020-12-31 13:28:20,910][root][INFO] - ===> Epoch[1782](8910/5): Loss -0.7450, Sem -0.7450, dir -0.9750, norm 0.1979 LR: 4.443e-02 Score 98.927 Data time: 0.1166, Forward time: 0.9483, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.2018 [2020-12-31 13:28:32,611][root][INFO] - ===> Epoch[1783](8915/5): Loss -0.7486, Sem -0.7486, dir -0.9751, norm 0.1934 LR: 4.440e-02 Score 98.884 Data time: 0.0934, Forward time: 0.9483, Backward time: 1.1919, DDP time: 0.0051, Total iter time: 2.2534 [2020-12-31 13:28:44,677][root][INFO] - ===> Epoch[1784](8920/5): Loss -0.7413, Sem -0.7413, dir -0.9747, norm 0.2009 LR: 4.437e-02 Score 98.909 Data time: 0.0869, Forward time: 0.9482, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.3395 [2020-12-31 13:28:56,844][root][INFO] - ===> Epoch[1785](8925/5): Loss -0.7325, Sem -0.7325, dir -0.9739, norm 0.2081 LR: 4.434e-02 Score 98.888 Data time: 0.1047, Forward time: 0.9482, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.3402 [2020-12-31 13:29:09,026][root][INFO] - ===> Epoch[1786](8930/5): Loss -0.7456, Sem -0.7456, dir -0.9755, norm 0.1971 LR: 4.430e-02 Score 98.905 Data time: 0.1281, Forward time: 0.9482, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.3348 [2020-12-31 13:29:21,135][root][INFO] - ===> Epoch[1787](8935/5): Loss -0.7396, Sem -0.7396, dir -0.9759, norm 0.2052 LR: 4.427e-02 Score 98.964 Data time: 0.1488, Forward time: 0.9482, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.3286 [2020-12-31 13:29:34,333][root][INFO] - ===> Epoch[1788](8940/5): Loss -0.7449, Sem -0.7449, dir -0.9753, norm 0.1990 LR: 4.424e-02 Score 98.944 Data time: 0.1090, Forward time: 0.9482, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.5580 [2020-12-31 13:29:46,558][root][INFO] - ===> Epoch[1789](8945/5): Loss -0.7522, Sem -0.7522, dir -0.9759, norm 0.1917 LR: 4.420e-02 Score 98.936 Data time: 0.1016, Forward time: 0.9483, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.3458 [2020-12-31 13:29:58,357][root][INFO] - ===> Epoch[1790](8950/5): Loss -0.7444, Sem -0.7444, dir -0.9748, norm 0.1978 LR: 4.417e-02 Score 98.931 Data time: 0.0860, Forward time: 0.9482, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.2716 [2020-12-31 13:30:11,084][root][INFO] - ===> Epoch[1791](8955/5): Loss -0.7595, Sem -0.7595, dir -0.9769, norm 0.1864 LR: 4.414e-02 Score 98.957 Data time: 0.0928, Forward time: 0.9482, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.4551 [2020-12-31 13:30:22,135][root][INFO] - ===> Epoch[1792](8960/5): Loss -0.7490, Sem -0.7490, dir -0.9747, norm 0.1936 LR: 4.411e-02 Score 98.936 Data time: 0.0794, Forward time: 0.9481, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.1122 [2020-12-31 13:30:34,243][root][INFO] - ===> Epoch[1793](8965/5): Loss -0.7429, Sem -0.7429, dir -0.9748, norm 0.1986 LR: 4.407e-02 Score 98.885 Data time: 0.1326, Forward time: 0.9481, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.3296 [2020-12-31 13:30:46,030][root][INFO] - ===> Epoch[1794](8970/5): Loss -0.7523, Sem -0.7523, dir -0.9757, norm 0.1914 LR: 4.404e-02 Score 98.932 Data time: 0.1030, Forward time: 0.9481, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.2645 [2020-12-31 13:30:57,733][root][INFO] - ===> Epoch[1795](8975/5): Loss -0.7523, Sem -0.7523, dir -0.9751, norm 0.1904 LR: 4.401e-02 Score 98.914 Data time: 0.0787, Forward time: 0.9481, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.2530 [2020-12-31 13:31:10,023][root][INFO] - ===> Epoch[1796](8980/5): Loss -0.7515, Sem -0.7515, dir -0.9760, norm 0.1931 LR: 4.397e-02 Score 98.946 Data time: 0.1308, Forward time: 0.9482, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.3636 [2020-12-31 13:31:21,952][root][INFO] - ===> Epoch[1797](8985/5): Loss -0.7493, Sem -0.7493, dir -0.9754, norm 0.1939 LR: 4.394e-02 Score 98.932 Data time: 0.0980, Forward time: 0.9482, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.2946 [2020-12-31 13:31:34,026][root][INFO] - ===> Epoch[1798](8990/5): Loss -0.7506, Sem -0.7506, dir -0.9751, norm 0.1923 LR: 4.391e-02 Score 98.929 Data time: 0.0786, Forward time: 0.9482, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.3405 [2020-12-31 13:31:46,472][root][INFO] - ===> Epoch[1799](8995/5): Loss -0.7443, Sem -0.7443, dir -0.9744, norm 0.1968 LR: 4.388e-02 Score 98.904 Data time: 0.1341, Forward time: 0.9482, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.4030 [2020-12-31 13:31:57,666][root][INFO] - ===> Epoch[1800](9000/5): Loss -0.7500, Sem -0.7500, dir -0.9754, norm 0.1924 LR: 4.384e-02 Score 98.903 Data time: 0.1442, Forward time: 0.9482, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.1466 [2020-12-31 13:31:57,669][root][INFO] - ===> Start testing [2020-12-31 13:32:36,602][root][INFO] - 68/68: Data time: 0.0023, Iter time: 0.1348 Loss 0.126 (AVG: 0.542) Score 96.595 (AVG: 89.460) mIOU 67.060 mAP 82.991 mAcc 74.852 IOU: 59.674 0.000 78.448 71.955 92.225 89.904 30.169 76.761 97.833 54.162 79.600 85.225 55.823 mAP: 81.470 7.086 93.141 85.756 98.828 98.250 59.784 94.696 99.622 82.839 92.978 94.027 90.407 mAcc: 81.828 0.000 88.649 78.484 93.448 94.623 33.179 88.168 99.316 70.610 89.217 95.665 59.883 [2020-12-31 13:32:36,603][root][INFO] - Finished test. Elapsed time: 38.9327 [2020-12-31 13:32:39,732][root][INFO] - [2020-12-31 13:32:39,732][root][INFO] - ################################################################ [2020-12-31 13:32:39,732][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 13:32:39,732][root][INFO] - ################################################################ [2020-12-31 13:32:39,733][root][INFO] - clutter : nan nan nan [2020-12-31 13:32:39,733][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 13:32:39,733][root][INFO] - board : 0.771 0.865 0.893 [2020-12-31 13:32:39,733][root][INFO] - bookcase : 0.195 0.347 0.492 [2020-12-31 13:32:39,733][root][INFO] - ceiling : 0.518 0.541 0.629 [2020-12-31 13:32:39,733][root][INFO] - chair : 0.771 0.862 0.884 [2020-12-31 13:32:39,733][root][INFO] - column : 0.153 0.253 0.348 [2020-12-31 13:32:39,733][root][INFO] - door : 0.672 0.837 0.949 [2020-12-31 13:32:39,733][root][INFO] - floor : 0.970 0.979 0.979 [2020-12-31 13:32:39,733][root][INFO] - sofa : 0.345 0.624 0.818 [2020-12-31 13:32:39,733][root][INFO] - table : 0.157 0.309 0.577 [2020-12-31 13:32:39,733][root][INFO] - wall : 0.491 0.716 0.808 [2020-12-31 13:32:39,733][root][INFO] - window : 0.488 0.745 0.779 [2020-12-31 13:32:39,733][root][INFO] - ---------------------------------------------------------------- [2020-12-31 13:32:39,733][root][INFO] - average : 0.461 0.590 0.680 [2020-12-31 13:32:39,733][root][INFO] - [2020-12-31 13:32:39,910][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 13:32:40,507][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter9000.pth [2020-12-31 13:32:52,621][root][INFO] - ===> Epoch[1801](9005/5): Loss -0.7403, Sem -0.7403, dir -0.9753, norm 0.2025 LR: 4.381e-02 Score 98.906 Data time: 0.1307, Forward time: 0.9483, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.3234 [2020-12-31 13:33:05,004][root][INFO] - ===> Epoch[1802](9010/5): Loss -0.7498, Sem -0.7498, dir -0.9756, norm 0.1953 LR: 4.378e-02 Score 98.979 Data time: 0.1251, Forward time: 0.9483, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.3952 [2020-12-31 13:33:17,538][root][INFO] - ===> Epoch[1803](9015/5): Loss -0.7397, Sem -0.7397, dir -0.9746, norm 0.2012 LR: 4.374e-02 Score 98.877 Data time: 0.1095, Forward time: 0.9483, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.4195 [2020-12-31 13:33:28,905][root][INFO] - ===> Epoch[1804](9020/5): Loss -0.7455, Sem -0.7455, dir -0.9751, norm 0.1963 LR: 4.371e-02 Score 98.888 Data time: 0.1231, Forward time: 0.9483, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.1871 [2020-12-31 13:33:40,631][root][INFO] - ===> Epoch[1805](9025/5): Loss -0.7483, Sem -0.7483, dir -0.9752, norm 0.1948 LR: 4.368e-02 Score 98.925 Data time: 0.0900, Forward time: 0.9482, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.2629 [2020-12-31 13:33:52,501][root][INFO] - ===> Epoch[1806](9030/5): Loss -0.7555, Sem -0.7555, dir -0.9758, norm 0.1885 LR: 4.364e-02 Score 98.930 Data time: 0.0968, Forward time: 0.9482, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.2904 [2020-12-31 13:34:04,266][root][INFO] - ===> Epoch[1807](9035/5): Loss -0.7528, Sem -0.7528, dir -0.9752, norm 0.1901 LR: 4.361e-02 Score 98.932 Data time: 0.1070, Forward time: 0.9482, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.2559 [2020-12-31 13:34:16,994][root][INFO] - ===> Epoch[1808](9040/5): Loss -0.7539, Sem -0.7539, dir -0.9752, norm 0.1888 LR: 4.358e-02 Score 98.908 Data time: 0.0778, Forward time: 0.9481, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.4570 [2020-12-31 13:34:28,506][root][INFO] - ===> Epoch[1809](9045/5): Loss -0.7512, Sem -0.7512, dir -0.9757, norm 0.1921 LR: 4.355e-02 Score 98.918 Data time: 0.1137, Forward time: 0.9481, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.2118 [2020-12-31 13:34:40,251][root][INFO] - ===> Epoch[1810](9050/5): Loss -0.7600, Sem -0.7600, dir -0.9760, norm 0.1845 LR: 4.351e-02 Score 98.949 Data time: 0.1171, Forward time: 0.9481, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.2534 [2020-12-31 13:34:52,004][root][INFO] - ===> Epoch[1811](9055/5): Loss -0.7535, Sem -0.7535, dir -0.9762, norm 0.1909 LR: 4.348e-02 Score 98.930 Data time: 0.1519, Forward time: 0.9482, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.2606 [2020-12-31 13:35:04,316][root][INFO] - ===> Epoch[1812](9060/5): Loss -0.7518, Sem -0.7518, dir -0.9756, norm 0.1914 LR: 4.345e-02 Score 98.924 Data time: 0.0915, Forward time: 0.9482, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.3801 [2020-12-31 13:35:16,409][root][INFO] - ===> Epoch[1813](9065/5): Loss -0.7510, Sem -0.7510, dir -0.9755, norm 0.1923 LR: 4.341e-02 Score 98.915 Data time: 0.1375, Forward time: 0.9482, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.3255 [2020-12-31 13:35:27,618][root][INFO] - ===> Epoch[1814](9070/5): Loss -0.7519, Sem -0.7519, dir -0.9755, norm 0.1902 LR: 4.338e-02 Score 98.880 Data time: 0.0830, Forward time: 0.9482, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.1578 [2020-12-31 13:35:40,055][root][INFO] - ===> Epoch[1815](9075/5): Loss -0.7536, Sem -0.7536, dir -0.9754, norm 0.1900 LR: 4.335e-02 Score 98.940 Data time: 0.0933, Forward time: 0.9482, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.3917 [2020-12-31 13:35:52,245][root][INFO] - ===> Epoch[1816](9080/5): Loss -0.7539, Sem -0.7539, dir -0.9758, norm 0.1886 LR: 4.332e-02 Score 98.892 Data time: 0.0931, Forward time: 0.9481, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.3505 [2020-12-31 13:36:04,772][root][INFO] - ===> Epoch[1817](9085/5): Loss -0.7478, Sem -0.7478, dir -0.9750, norm 0.1952 LR: 4.328e-02 Score 98.930 Data time: 0.2243, Forward time: 0.9482, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.4072 [2020-12-31 13:36:16,369][root][INFO] - ===> Epoch[1818](9090/5): Loss -0.7482, Sem -0.7482, dir -0.9749, norm 0.1939 LR: 4.325e-02 Score 98.910 Data time: 0.1103, Forward time: 0.9482, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.2214 [2020-12-31 13:36:28,582][root][INFO] - ===> Epoch[1819](9095/5): Loss -0.7563, Sem -0.7563, dir -0.9762, norm 0.1891 LR: 4.322e-02 Score 98.968 Data time: 0.1212, Forward time: 0.9482, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.3422 [2020-12-31 13:36:41,389][root][INFO] - ===> Epoch[1820](9100/5): Loss -0.7466, Sem -0.7466, dir -0.9749, norm 0.1956 LR: 4.318e-02 Score 98.909 Data time: 0.0848, Forward time: 0.9482, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.4765 [2020-12-31 13:36:53,135][root][INFO] - ===> Epoch[1821](9105/5): Loss -0.7518, Sem -0.7518, dir -0.9754, norm 0.1915 LR: 4.315e-02 Score 98.932 Data time: 0.1507, Forward time: 0.9482, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.2596 [2020-12-31 13:37:04,557][root][INFO] - ===> Epoch[1822](9110/5): Loss -0.7451, Sem -0.7451, dir -0.9751, norm 0.1978 LR: 4.312e-02 Score 98.924 Data time: 0.1357, Forward time: 0.9482, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.1860 [2020-12-31 13:37:17,209][root][INFO] - ===> Epoch[1823](9115/5): Loss -0.7376, Sem -0.7376, dir -0.9752, norm 0.2069 LR: 4.309e-02 Score 98.969 Data time: 0.0930, Forward time: 0.9482, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.4540 [2020-12-31 13:37:29,290][root][INFO] - ===> Epoch[1824](9120/5): Loss -0.7398, Sem -0.7398, dir -0.9758, norm 0.2042 LR: 4.305e-02 Score 98.931 Data time: 0.0942, Forward time: 0.9482, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.3231 [2020-12-31 13:37:41,323][root][INFO] - ===> Epoch[1825](9125/5): Loss -0.7423, Sem -0.7423, dir -0.9754, norm 0.2014 LR: 4.302e-02 Score 98.934 Data time: 0.0933, Forward time: 0.9482, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.3103 [2020-12-31 13:37:54,473][root][INFO] - ===> Epoch[1826](9130/5): Loss -0.7360, Sem -0.7360, dir -0.9749, norm 0.2074 LR: 4.299e-02 Score 98.944 Data time: 0.1553, Forward time: 0.9482, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.5299 [2020-12-31 13:38:06,895][root][INFO] - ===> Epoch[1827](9135/5): Loss -0.7366, Sem -0.7366, dir -0.9753, norm 0.2064 LR: 4.295e-02 Score 98.910 Data time: 0.1085, Forward time: 0.9482, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.3994 [2020-12-31 13:38:18,518][root][INFO] - ===> Epoch[1828](9140/5): Loss -0.7400, Sem -0.7400, dir -0.9728, norm 0.2006 LR: 4.292e-02 Score 98.920 Data time: 0.1070, Forward time: 0.9482, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.2387 [2020-12-31 13:38:31,172][root][INFO] - ===> Epoch[1829](9145/5): Loss -0.7473, Sem -0.7473, dir -0.9755, norm 0.1970 LR: 4.289e-02 Score 98.955 Data time: 0.1122, Forward time: 0.9482, Backward time: 1.1923, DDP time: 0.0051, Total iter time: 2.4304 [2020-12-31 13:38:43,371][root][INFO] - ===> Epoch[1830](9150/5): Loss -0.7435, Sem -0.7435, dir -0.9748, norm 0.1977 LR: 4.285e-02 Score 98.878 Data time: 0.0996, Forward time: 0.9482, Backward time: 1.1923, DDP time: 0.0051, Total iter time: 2.3475 [2020-12-31 13:38:55,617][root][INFO] - ===> Epoch[1831](9155/5): Loss -0.7356, Sem -0.7356, dir -0.9736, norm 0.2060 LR: 4.282e-02 Score 98.935 Data time: 0.1091, Forward time: 0.9481, Backward time: 1.1923, DDP time: 0.0051, Total iter time: 2.3570 [2020-12-31 13:39:07,477][root][INFO] - ===> Epoch[1832](9160/5): Loss -0.7390, Sem -0.7390, dir -0.9746, norm 0.2029 LR: 4.279e-02 Score 98.897 Data time: 0.1757, Forward time: 0.9481, Backward time: 1.1923, DDP time: 0.0051, Total iter time: 2.2772 [2020-12-31 13:39:19,958][root][INFO] - ===> Epoch[1833](9165/5): Loss -0.7450, Sem -0.7450, dir -0.9752, norm 0.1973 LR: 4.276e-02 Score 98.898 Data time: 0.1038, Forward time: 0.9481, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.4206 [2020-12-31 13:39:31,899][root][INFO] - ===> Epoch[1834](9170/5): Loss -0.7469, Sem -0.7469, dir -0.9750, norm 0.1956 LR: 4.272e-02 Score 98.906 Data time: 0.1033, Forward time: 0.9481, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.2998 [2020-12-31 13:39:44,067][root][INFO] - ===> Epoch[1835](9175/5): Loss -0.7471, Sem -0.7471, dir -0.9758, norm 0.1976 LR: 4.269e-02 Score 98.963 Data time: 0.0861, Forward time: 0.9481, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.3355 [2020-12-31 13:39:55,738][root][INFO] - ===> Epoch[1836](9180/5): Loss -0.7451, Sem -0.7451, dir -0.9751, norm 0.1966 LR: 4.266e-02 Score 98.893 Data time: 0.1092, Forward time: 0.9480, Backward time: 1.1925, DDP time: 0.0051, Total iter time: 2.2552 [2020-12-31 13:40:07,203][root][INFO] - ===> Epoch[1837](9185/5): Loss -0.7504, Sem -0.7504, dir -0.9751, norm 0.1919 LR: 4.262e-02 Score 98.902 Data time: 0.1326, Forward time: 0.9479, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.2113 [2020-12-31 13:40:19,616][root][INFO] - ===> Epoch[1838](9190/5): Loss -0.7439, Sem -0.7439, dir -0.9750, norm 0.1988 LR: 4.259e-02 Score 98.926 Data time: 0.0928, Forward time: 0.9480, Backward time: 1.1925, DDP time: 0.0051, Total iter time: 2.3804 [2020-12-31 13:40:31,699][root][INFO] - ===> Epoch[1839](9195/5): Loss -0.7536, Sem -0.7536, dir -0.9762, norm 0.1899 LR: 4.256e-02 Score 98.938 Data time: 0.1626, Forward time: 0.9480, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.3391 [2020-12-31 13:40:44,511][root][INFO] - ===> Epoch[1840](9200/5): Loss -0.7479, Sem -0.7479, dir -0.9749, norm 0.1956 LR: 4.253e-02 Score 98.957 Data time: 0.1538, Forward time: 0.9481, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.4648 [2020-12-31 13:40:44,513][root][INFO] - ===> Start testing [2020-12-31 13:41:24,194][root][INFO] - 68/68: Data time: 0.0670, Iter time: 0.1974 Loss 0.546 (AVG: 0.511) Score 89.654 (AVG: 89.696) mIOU 68.353 mAP 83.147 mAcc 75.324 IOU: 59.679 0.000 78.681 72.950 93.062 90.723 32.364 77.489 98.391 65.514 79.688 84.722 55.330 mAP: 80.616 5.646 94.593 86.355 98.820 98.011 58.658 94.614 99.821 85.869 93.358 94.204 90.350 mAcc: 79.283 0.000 88.679 80.893 94.134 94.080 36.020 87.455 99.090 74.181 90.087 95.951 59.362 [2020-12-31 13:41:24,195][root][INFO] - Finished test. Elapsed time: 39.6791 [2020-12-31 13:41:27,279][root][INFO] - [2020-12-31 13:41:27,279][root][INFO] - ################################################################ [2020-12-31 13:41:27,279][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 13:41:27,280][root][INFO] - ################################################################ [2020-12-31 13:41:27,280][root][INFO] - clutter : nan nan nan [2020-12-31 13:41:27,280][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 13:41:27,280][root][INFO] - board : 0.800 0.901 0.924 [2020-12-31 13:41:27,280][root][INFO] - bookcase : 0.189 0.326 0.480 [2020-12-31 13:41:27,280][root][INFO] - ceiling : 0.604 0.670 0.718 [2020-12-31 13:41:27,280][root][INFO] - chair : 0.759 0.847 0.864 [2020-12-31 13:41:27,280][root][INFO] - column : 0.161 0.244 0.387 [2020-12-31 13:41:27,280][root][INFO] - door : 0.526 0.738 0.889 [2020-12-31 13:41:27,280][root][INFO] - floor : 0.993 1.000 1.000 [2020-12-31 13:41:27,280][root][INFO] - sofa : 0.386 0.545 0.809 [2020-12-31 13:41:27,280][root][INFO] - table : 0.186 0.411 0.642 [2020-12-31 13:41:27,280][root][INFO] - wall : 0.475 0.713 0.795 [2020-12-31 13:41:27,280][root][INFO] - window : 0.459 0.725 0.783 [2020-12-31 13:41:27,280][root][INFO] - ---------------------------------------------------------------- [2020-12-31 13:41:27,280][root][INFO] - average : 0.461 0.593 0.691 [2020-12-31 13:41:27,280][root][INFO] - [2020-12-31 13:41:27,464][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 13:41:28,033][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter9200.pth [2020-12-31 13:41:39,280][root][INFO] - ===> Epoch[1841](9205/5): Loss -0.7482, Sem -0.7482, dir -0.9751, norm 0.1965 LR: 4.249e-02 Score 98.977 Data time: 0.0822, Forward time: 0.9480, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.1652 [2020-12-31 13:41:51,652][root][INFO] - ===> Epoch[1842](9210/5): Loss -0.7457, Sem -0.7457, dir -0.9753, norm 0.1978 LR: 4.246e-02 Score 98.938 Data time: 0.0999, Forward time: 0.9481, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.3932 [2020-12-31 13:42:03,120][root][INFO] - ===> Epoch[1843](9215/5): Loss -0.7546, Sem -0.7546, dir -0.9752, norm 0.1892 LR: 4.243e-02 Score 98.951 Data time: 0.1581, Forward time: 0.9481, Backward time: 1.1923, DDP time: 0.0051, Total iter time: 2.1989 [2020-12-31 13:42:15,746][root][INFO] - ===> Epoch[1844](9220/5): Loss -0.7618, Sem -0.7618, dir -0.9765, norm 0.1851 LR: 4.239e-02 Score 99.019 Data time: 0.1275, Forward time: 0.9481, Backward time: 1.1923, DDP time: 0.0051, Total iter time: 2.4323 [2020-12-31 13:42:28,365][root][INFO] - ===> Epoch[1845](9225/5): Loss -0.7407, Sem -0.7407, dir -0.9739, norm 0.1984 LR: 4.236e-02 Score 98.824 Data time: 0.0968, Forward time: 0.9481, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.4283 [2020-12-31 13:42:40,733][root][INFO] - ===> Epoch[1846](9230/5): Loss -0.7596, Sem -0.7596, dir -0.9760, norm 0.1847 LR: 4.233e-02 Score 98.949 Data time: 0.1069, Forward time: 0.9481, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.3768 [2020-12-31 13:42:52,388][root][INFO] - ===> Epoch[1847](9235/5): Loss -0.7521, Sem -0.7521, dir -0.9755, norm 0.1912 LR: 4.229e-02 Score 98.922 Data time: 0.0810, Forward time: 0.9481, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.2382 [2020-12-31 13:43:04,756][root][INFO] - ===> Epoch[1848](9240/5): Loss -0.7486, Sem -0.7486, dir -0.9754, norm 0.1946 LR: 4.226e-02 Score 98.925 Data time: 0.0967, Forward time: 0.9481, Backward time: 1.1925, DDP time: 0.0051, Total iter time: 2.3775 [2020-12-31 13:43:16,353][root][INFO] - ===> Epoch[1849](9245/5): Loss -0.7495, Sem -0.7495, dir -0.9753, norm 0.1943 LR: 4.223e-02 Score 98.941 Data time: 0.1369, Forward time: 0.9481, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.2159 [2020-12-31 13:43:28,995][root][INFO] - ===> Epoch[1850](9250/5): Loss -0.7507, Sem -0.7507, dir -0.9756, norm 0.1928 LR: 4.219e-02 Score 98.927 Data time: 0.1151, Forward time: 0.9480, Backward time: 1.1925, DDP time: 0.0051, Total iter time: 2.4371 [2020-12-31 13:43:40,300][root][INFO] - ===> Epoch[1851](9255/5): Loss -0.7464, Sem -0.7464, dir -0.9748, norm 0.1966 LR: 4.216e-02 Score 98.935 Data time: 0.1034, Forward time: 0.9481, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.1665 [2020-12-31 13:43:52,902][root][INFO] - ===> Epoch[1852](9260/5): Loss -0.7570, Sem -0.7570, dir -0.9754, norm 0.1872 LR: 4.213e-02 Score 98.947 Data time: 0.1112, Forward time: 0.9481, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.4215 [2020-12-31 13:44:05,079][root][INFO] - ===> Epoch[1853](9265/5): Loss -0.7528, Sem -0.7528, dir -0.9758, norm 0.1912 LR: 4.210e-02 Score 98.936 Data time: 0.1288, Forward time: 0.9481, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.3406 [2020-12-31 13:44:16,708][root][INFO] - ===> Epoch[1854](9270/5): Loss -0.7501, Sem -0.7501, dir -0.9757, norm 0.1934 LR: 4.206e-02 Score 98.910 Data time: 0.0855, Forward time: 0.9482, Backward time: 1.1924, DDP time: 0.0051, Total iter time: 2.2327 [2020-12-31 13:44:28,150][root][INFO] - ===> Epoch[1855](9275/5): Loss -0.7528, Sem -0.7528, dir -0.9750, norm 0.1906 LR: 4.203e-02 Score 98.936 Data time: 0.0911, Forward time: 0.9482, Backward time: 1.1923, DDP time: 0.0051, Total iter time: 2.1944 [2020-12-31 13:44:39,490][root][INFO] - ===> Epoch[1856](9280/5): Loss -0.7594, Sem -0.7594, dir -0.9770, norm 0.1878 LR: 4.200e-02 Score 99.001 Data time: 0.1786, Forward time: 0.9482, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.1739 [2020-12-31 13:44:52,334][root][INFO] - ===> Epoch[1857](9285/5): Loss -0.7567, Sem -0.7567, dir -0.9762, norm 0.1897 LR: 4.196e-02 Score 99.001 Data time: 0.1163, Forward time: 0.9482, Backward time: 1.1923, DDP time: 0.0051, Total iter time: 2.4774 [2020-12-31 13:45:03,904][root][INFO] - ===> Epoch[1858](9290/5): Loss -0.7545, Sem -0.7545, dir -0.9754, norm 0.1906 LR: 4.193e-02 Score 98.980 Data time: 0.1336, Forward time: 0.9482, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.2137 [2020-12-31 13:45:16,236][root][INFO] - ===> Epoch[1859](9295/5): Loss -0.7441, Sem -0.7441, dir -0.9748, norm 0.1984 LR: 4.190e-02 Score 98.916 Data time: 0.0927, Forward time: 0.9482, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.3717 [2020-12-31 13:45:28,248][root][INFO] - ===> Epoch[1860](9300/5): Loss -0.7500, Sem -0.7500, dir -0.9763, norm 0.1960 LR: 4.186e-02 Score 98.982 Data time: 0.1048, Forward time: 0.9482, Backward time: 1.1922, DDP time: 0.0051, Total iter time: 2.3045 [2020-12-31 13:45:39,385][root][INFO] - ===> Epoch[1861](9305/5): Loss -0.7559, Sem -0.7559, dir -0.9754, norm 0.1875 LR: 4.183e-02 Score 98.926 Data time: 0.1676, Forward time: 0.9482, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.1287 [2020-12-31 13:45:51,343][root][INFO] - ===> Epoch[1862](9310/5): Loss -0.7531, Sem -0.7531, dir -0.9755, norm 0.1910 LR: 4.180e-02 Score 98.950 Data time: 0.1566, Forward time: 0.9482, Backward time: 1.1921, DDP time: 0.0051, Total iter time: 2.2977 [2020-12-31 13:46:03,750][root][INFO] - ===> Epoch[1863](9315/5): Loss -0.7464, Sem -0.7464, dir -0.9756, norm 0.1976 LR: 4.177e-02 Score 98.950 Data time: 0.1389, Forward time: 0.9483, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.3847 [2020-12-31 13:46:15,183][root][INFO] - ===> Epoch[1864](9320/5): Loss -0.7568, Sem -0.7568, dir -0.9761, norm 0.1873 LR: 4.173e-02 Score 98.924 Data time: 0.1013, Forward time: 0.9483, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.1904 [2020-12-31 13:46:28,178][root][INFO] - ===> Epoch[1865](9325/5): Loss -0.7483, Sem -0.7483, dir -0.9759, norm 0.1958 LR: 4.170e-02 Score 98.953 Data time: 0.1562, Forward time: 0.9484, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.4833 [2020-12-31 13:46:39,659][root][INFO] - ===> Epoch[1866](9330/5): Loss -0.7439, Sem -0.7439, dir -0.9753, norm 0.1987 LR: 4.167e-02 Score 98.910 Data time: 0.1298, Forward time: 0.9483, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.2173 [2020-12-31 13:46:51,609][root][INFO] - ===> Epoch[1867](9335/5): Loss -0.7490, Sem -0.7490, dir -0.9749, norm 0.1946 LR: 4.163e-02 Score 98.957 Data time: 0.0839, Forward time: 0.9483, Backward time: 1.1920, DDP time: 0.0051, Total iter time: 2.2994 [2020-12-31 13:47:02,972][root][INFO] - ===> Epoch[1868](9340/5): Loss -0.7468, Sem -0.7468, dir -0.9751, norm 0.1965 LR: 4.160e-02 Score 98.940 Data time: 0.1456, Forward time: 0.9483, Backward time: 1.1919, DDP time: 0.0051, Total iter time: 2.1777 [2020-12-31 13:47:14,496][root][INFO] - ===> Epoch[1869](9345/5): Loss -0.7564, Sem -0.7564, dir -0.9761, norm 0.1876 LR: 4.157e-02 Score 98.919 Data time: 0.1673, Forward time: 0.9483, Backward time: 1.1918, DDP time: 0.0051, Total iter time: 2.2118 [2020-12-31 13:47:26,473][root][INFO] - ===> Epoch[1870](9350/5): Loss -0.7560, Sem -0.7560, dir -0.9760, norm 0.1892 LR: 4.153e-02 Score 98.962 Data time: 0.0853, Forward time: 0.9483, Backward time: 1.1918, DDP time: 0.0051, Total iter time: 2.3048 [2020-12-31 13:47:38,578][root][INFO] - ===> Epoch[1871](9355/5): Loss -0.7589, Sem -0.7589, dir -0.9760, norm 0.1855 LR: 4.150e-02 Score 98.947 Data time: 0.0934, Forward time: 0.9483, Backward time: 1.1918, DDP time: 0.0051, Total iter time: 2.3420 [2020-12-31 13:47:50,060][root][INFO] - ===> Epoch[1872](9360/5): Loss -0.7587, Sem -0.7587, dir -0.9760, norm 0.1869 LR: 4.147e-02 Score 98.981 Data time: 0.0842, Forward time: 0.9483, Backward time: 1.1918, DDP time: 0.0051, Total iter time: 2.2038 [2020-12-31 13:48:01,477][root][INFO] - ===> Epoch[1873](9365/5): Loss -0.7564, Sem -0.7564, dir -0.9761, norm 0.1875 LR: 4.143e-02 Score 98.929 Data time: 0.1363, Forward time: 0.9483, Backward time: 1.1917, DDP time: 0.0051, Total iter time: 2.1953 [2020-12-31 13:48:13,353][root][INFO] - ===> Epoch[1874](9370/5): Loss -0.7532, Sem -0.7532, dir -0.9753, norm 0.1899 LR: 4.140e-02 Score 98.924 Data time: 0.1385, Forward time: 0.9483, Backward time: 1.1917, DDP time: 0.0051, Total iter time: 2.2766 [2020-12-31 13:48:25,241][root][INFO] - ===> Epoch[1875](9375/5): Loss -0.7541, Sem -0.7541, dir -0.9760, norm 0.1902 LR: 4.137e-02 Score 98.935 Data time: 0.1095, Forward time: 0.9483, Backward time: 1.1917, DDP time: 0.0051, Total iter time: 2.2858 [2020-12-31 13:48:38,441][root][INFO] - ===> Epoch[1876](9380/5): Loss -0.7554, Sem -0.7554, dir -0.9762, norm 0.1893 LR: 4.134e-02 Score 98.946 Data time: 0.2017, Forward time: 0.9483, Backward time: 1.1917, DDP time: 0.0051, Total iter time: 2.5563 [2020-12-31 13:48:50,304][root][INFO] - ===> Epoch[1877](9385/5): Loss -0.7640, Sem -0.7640, dir -0.9763, norm 0.1815 LR: 4.130e-02 Score 98.963 Data time: 0.0980, Forward time: 0.9484, Backward time: 1.1917, DDP time: 0.0051, Total iter time: 2.2704 [2020-12-31 13:49:01,203][root][INFO] - ===> Epoch[1878](9390/5): Loss -0.7471, Sem -0.7471, dir -0.9758, norm 0.1962 LR: 4.127e-02 Score 98.912 Data time: 0.0767, Forward time: 0.9484, Backward time: 1.1916, DDP time: 0.0051, Total iter time: 2.1187 [2020-12-31 13:49:13,990][root][INFO] - ===> Epoch[1879](9395/5): Loss -0.7531, Sem -0.7531, dir -0.9757, norm 0.1890 LR: 4.124e-02 Score 98.865 Data time: 0.0977, Forward time: 0.9483, Backward time: 1.1917, DDP time: 0.0051, Total iter time: 2.4582 [2020-12-31 13:49:25,609][root][INFO] - ===> Epoch[1880](9400/5): Loss -0.7533, Sem -0.7533, dir -0.9755, norm 0.1906 LR: 4.120e-02 Score 98.943 Data time: 0.1075, Forward time: 0.9484, Backward time: 1.1917, DDP time: 0.0051, Total iter time: 2.2356 [2020-12-31 13:49:25,612][root][INFO] - ===> Start testing [2020-12-31 13:50:03,425][root][INFO] - 68/68: Data time: 0.0020, Iter time: 0.0974 Loss 0.323 (AVG: 0.540) Score 93.072 (AVG: 89.380) mIOU 67.341 mAP 82.268 mAcc 74.477 IOU: 59.396 0.000 81.079 72.586 92.158 89.336 28.561 77.831 98.366 56.741 78.211 84.749 56.413 mAP: 81.033 7.804 92.683 85.371 98.797 98.111 56.160 94.671 99.837 77.690 92.872 93.884 90.569 mAcc: 82.022 0.000 87.807 79.946 93.161 95.247 32.054 87.247 99.110 67.887 88.441 95.304 59.970 [2020-12-31 13:50:03,426][root][INFO] - Finished test. Elapsed time: 37.8120 [2020-12-31 13:50:06,508][root][INFO] - [2020-12-31 13:50:06,508][root][INFO] - ################################################################ [2020-12-31 13:50:06,508][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 13:50:06,508][root][INFO] - ################################################################ [2020-12-31 13:50:06,508][root][INFO] - clutter : nan nan nan [2020-12-31 13:50:06,508][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 13:50:06,508][root][INFO] - board : 0.729 0.845 0.892 [2020-12-31 13:50:06,508][root][INFO] - bookcase : 0.203 0.337 0.482 [2020-12-31 13:50:06,508][root][INFO] - ceiling : 0.640 0.725 0.779 [2020-12-31 13:50:06,508][root][INFO] - chair : 0.777 0.865 0.888 [2020-12-31 13:50:06,508][root][INFO] - column : 0.149 0.194 0.317 [2020-12-31 13:50:06,508][root][INFO] - door : 0.546 0.752 0.884 [2020-12-31 13:50:06,508][root][INFO] - floor : 0.979 0.993 0.995 [2020-12-31 13:50:06,508][root][INFO] - sofa : 0.346 0.636 0.636 [2020-12-31 13:50:06,509][root][INFO] - table : 0.172 0.352 0.629 [2020-12-31 13:50:06,509][root][INFO] - wall : 0.469 0.700 0.810 [2020-12-31 13:50:06,509][root][INFO] - window : 0.462 0.720 0.779 [2020-12-31 13:50:06,509][root][INFO] - ---------------------------------------------------------------- [2020-12-31 13:50:06,509][root][INFO] - average : 0.456 0.593 0.674 [2020-12-31 13:50:06,509][root][INFO] - [2020-12-31 13:50:06,669][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 13:50:07,262][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter9400.pth [2020-12-31 13:50:19,519][root][INFO] - ===> Epoch[1881](9405/5): Loss -0.7510, Sem -0.7510, dir -0.9767, norm 0.1942 LR: 4.117e-02 Score 98.955 Data time: 0.1317, Forward time: 0.9484, Backward time: 1.1916, DDP time: 0.0051, Total iter time: 2.3413 [2020-12-31 13:50:31,056][root][INFO] - ===> Epoch[1882](9410/5): Loss -0.7410, Sem -0.7410, dir -0.9752, norm 0.2015 LR: 4.114e-02 Score 98.903 Data time: 0.1126, Forward time: 0.9485, Backward time: 1.1915, DDP time: 0.0051, Total iter time: 2.1969 [2020-12-31 13:50:43,359][root][INFO] - ===> Epoch[1883](9415/5): Loss -0.7510, Sem -0.7510, dir -0.9754, norm 0.1919 LR: 4.110e-02 Score 98.909 Data time: 0.1134, Forward time: 0.9485, Backward time: 1.1915, DDP time: 0.0051, Total iter time: 2.3702 [2020-12-31 13:50:55,361][root][INFO] - ===> Epoch[1884](9420/5): Loss -0.7540, Sem -0.7540, dir -0.9764, norm 0.1908 LR: 4.107e-02 Score 98.956 Data time: 0.1152, Forward time: 0.9485, Backward time: 1.1915, DDP time: 0.0051, Total iter time: 2.3040 [2020-12-31 13:51:07,738][root][INFO] - ===> Epoch[1885](9425/5): Loss -0.7577, Sem -0.7577, dir -0.9769, norm 0.1881 LR: 4.104e-02 Score 98.956 Data time: 0.0880, Forward time: 0.9485, Backward time: 1.1916, DDP time: 0.0051, Total iter time: 2.3863 [2020-12-31 13:51:18,896][root][INFO] - ===> Epoch[1886](9430/5): Loss -0.7459, Sem -0.7459, dir -0.9745, norm 0.1949 LR: 4.100e-02 Score 98.884 Data time: 0.1069, Forward time: 0.9485, Backward time: 1.1915, DDP time: 0.0051, Total iter time: 2.1368 [2020-12-31 13:51:31,156][root][INFO] - ===> Epoch[1887](9435/5): Loss -0.7563, Sem -0.7563, dir -0.9760, norm 0.1880 LR: 4.097e-02 Score 98.936 Data time: 0.1265, Forward time: 0.9485, Backward time: 1.1915, DDP time: 0.0051, Total iter time: 2.3587 [2020-12-31 13:51:44,033][root][INFO] - ===> Epoch[1888](9440/5): Loss -0.7565, Sem -0.7565, dir -0.9752, norm 0.1858 LR: 4.094e-02 Score 98.902 Data time: 0.1039, Forward time: 0.9485, Backward time: 1.1916, DDP time: 0.0051, Total iter time: 2.4849 [2020-12-31 13:51:55,862][root][INFO] - ===> Epoch[1889](9445/5): Loss -0.7595, Sem -0.7595, dir -0.9765, norm 0.1855 LR: 4.091e-02 Score 98.948 Data time: 0.1537, Forward time: 0.9485, Backward time: 1.1915, DDP time: 0.0051, Total iter time: 2.2750 [2020-12-31 13:52:08,378][root][INFO] - ===> Epoch[1890](9450/5): Loss -0.7639, Sem -0.7639, dir -0.9772, norm 0.1825 LR: 4.087e-02 Score 98.979 Data time: 0.1510, Forward time: 0.9486, Backward time: 1.1915, DDP time: 0.0051, Total iter time: 2.4031 [2020-12-31 13:52:19,853][root][INFO] - ===> Epoch[1891](9455/5): Loss -0.7581, Sem -0.7581, dir -0.9751, norm 0.1846 LR: 4.084e-02 Score 98.916 Data time: 0.1455, Forward time: 0.9486, Backward time: 1.1914, DDP time: 0.0051, Total iter time: 2.2053 [2020-12-31 13:52:32,213][root][INFO] - ===> Epoch[1892](9460/5): Loss -0.7592, Sem -0.7592, dir -0.9763, norm 0.1856 LR: 4.081e-02 Score 98.959 Data time: 0.1401, Forward time: 0.9487, Backward time: 1.1914, DDP time: 0.0051, Total iter time: 2.3919 [2020-12-31 13:52:43,304][root][INFO] - ===> Epoch[1893](9465/5): Loss -0.7532, Sem -0.7532, dir -0.9758, norm 0.1905 LR: 4.077e-02 Score 98.924 Data time: 0.0996, Forward time: 0.9486, Backward time: 1.1913, DDP time: 0.0051, Total iter time: 2.1345 [2020-12-31 13:52:56,181][root][INFO] - ===> Epoch[1894](9470/5): Loss -0.7631, Sem -0.7631, dir -0.9762, norm 0.1823 LR: 4.074e-02 Score 98.965 Data time: 0.0847, Forward time: 0.9486, Backward time: 1.1915, DDP time: 0.0051, Total iter time: 2.4866 [2020-12-31 13:53:09,200][root][INFO] - ===> Epoch[1895](9475/5): Loss -0.7534, Sem -0.7534, dir -0.9756, norm 0.1909 LR: 4.071e-02 Score 98.945 Data time: 0.1525, Forward time: 0.9487, Backward time: 1.1915, DDP time: 0.0051, Total iter time: 2.5109 [2020-12-31 13:53:20,413][root][INFO] - ===> Epoch[1896](9480/5): Loss -0.7555, Sem -0.7555, dir -0.9768, norm 0.1909 LR: 4.067e-02 Score 98.998 Data time: 0.1048, Forward time: 0.9487, Backward time: 1.1914, DDP time: 0.0051, Total iter time: 2.1619 [2020-12-31 13:53:32,347][root][INFO] - ===> Epoch[1897](9485/5): Loss -0.7481, Sem -0.7481, dir -0.9757, norm 0.1957 LR: 4.064e-02 Score 98.932 Data time: 0.1425, Forward time: 0.9487, Backward time: 1.1913, DDP time: 0.0051, Total iter time: 2.2983 [2020-12-31 13:53:43,809][root][INFO] - ===> Epoch[1898](9490/5): Loss -0.7540, Sem -0.7540, dir -0.9763, norm 0.1909 LR: 4.061e-02 Score 98.951 Data time: 0.1507, Forward time: 0.9487, Backward time: 1.1912, DDP time: 0.0051, Total iter time: 2.2014 [2020-12-31 13:53:56,083][root][INFO] - ===> Epoch[1899](9495/5): Loss -0.7624, Sem -0.7624, dir -0.9761, norm 0.1829 LR: 4.057e-02 Score 98.963 Data time: 0.1179, Forward time: 0.9488, Backward time: 1.1912, DDP time: 0.0051, Total iter time: 2.3625 [2020-12-31 13:54:07,517][root][INFO] - ===> Epoch[1900](9500/5): Loss -0.7531, Sem -0.7531, dir -0.9756, norm 0.1901 LR: 4.054e-02 Score 98.919 Data time: 0.0781, Forward time: 0.9487, Backward time: 1.1912, DDP time: 0.0051, Total iter time: 2.2067 [2020-12-31 13:54:19,494][root][INFO] - ===> Epoch[1901](9505/5): Loss -0.7623, Sem -0.7623, dir -0.9766, norm 0.1831 LR: 4.051e-02 Score 98.951 Data time: 0.1158, Forward time: 0.9487, Backward time: 1.1912, DDP time: 0.0051, Total iter time: 2.3048 [2020-12-31 13:54:31,054][root][INFO] - ===> Epoch[1902](9510/5): Loss -0.7486, Sem -0.7486, dir -0.9755, norm 0.1953 LR: 4.047e-02 Score 98.930 Data time: 0.1027, Forward time: 0.9487, Backward time: 1.1912, DDP time: 0.0051, Total iter time: 2.2221 [2020-12-31 13:54:42,200][root][INFO] - ===> Epoch[1903](9515/5): Loss -0.7499, Sem -0.7499, dir -0.9765, norm 0.1957 LR: 4.044e-02 Score 98.961 Data time: 0.1164, Forward time: 0.9487, Backward time: 1.1911, DDP time: 0.0051, Total iter time: 2.1375 [2020-12-31 13:54:55,160][root][INFO] - ===> Epoch[1904](9520/5): Loss -0.7536, Sem -0.7536, dir -0.9767, norm 0.1927 LR: 4.041e-02 Score 98.983 Data time: 0.0765, Forward time: 0.9486, Backward time: 1.1913, DDP time: 0.0051, Total iter time: 2.5082 [2020-12-31 13:55:07,373][root][INFO] - ===> Epoch[1905](9525/5): Loss -0.7525, Sem -0.7525, dir -0.9761, norm 0.1924 LR: 4.037e-02 Score 98.963 Data time: 0.1369, Forward time: 0.9487, Backward time: 1.1913, DDP time: 0.0051, Total iter time: 2.3434 [2020-12-31 13:55:19,040][root][INFO] - ===> Epoch[1906](9530/5): Loss -0.7541, Sem -0.7541, dir -0.9757, norm 0.1901 LR: 4.034e-02 Score 98.945 Data time: 0.1169, Forward time: 0.9487, Backward time: 1.1912, DDP time: 0.0051, Total iter time: 2.2410 [2020-12-31 13:55:30,691][root][INFO] - ===> Epoch[1907](9535/5): Loss -0.7637, Sem -0.7637, dir -0.9768, norm 0.1827 LR: 4.031e-02 Score 98.984 Data time: 0.0873, Forward time: 0.9486, Backward time: 1.1912, DDP time: 0.0051, Total iter time: 2.2436 [2020-12-31 13:55:42,593][root][INFO] - ===> Epoch[1908](9540/5): Loss -0.7592, Sem -0.7592, dir -0.9759, norm 0.1848 LR: 4.028e-02 Score 98.924 Data time: 0.1189, Forward time: 0.9487, Backward time: 1.1912, DDP time: 0.0051, Total iter time: 2.2885 [2020-12-31 13:55:54,505][root][INFO] - ===> Epoch[1909](9545/5): Loss -0.7542, Sem -0.7542, dir -0.9763, norm 0.1909 LR: 4.024e-02 Score 98.960 Data time: 0.1303, Forward time: 0.9486, Backward time: 1.1912, DDP time: 0.0051, Total iter time: 2.2958 [2020-12-31 13:56:06,832][root][INFO] - ===> Epoch[1910](9550/5): Loss -0.7544, Sem -0.7544, dir -0.9755, norm 0.1885 LR: 4.021e-02 Score 98.916 Data time: 0.1433, Forward time: 0.9487, Backward time: 1.1912, DDP time: 0.0051, Total iter time: 2.3660 [2020-12-31 13:56:18,251][root][INFO] - ===> Epoch[1911](9555/5): Loss -0.7566, Sem -0.7566, dir -0.9758, norm 0.1878 LR: 4.018e-02 Score 98.946 Data time: 0.1190, Forward time: 0.9487, Backward time: 1.1911, DDP time: 0.0051, Total iter time: 2.1981 [2020-12-31 13:56:30,047][root][INFO] - ===> Epoch[1912](9560/5): Loss -0.7620, Sem -0.7620, dir -0.9769, norm 0.1837 LR: 4.014e-02 Score 98.981 Data time: 0.0876, Forward time: 0.9487, Backward time: 1.1911, DDP time: 0.0051, Total iter time: 2.2775 [2020-12-31 13:56:41,166][root][INFO] - ===> Epoch[1913](9565/5): Loss -0.7549, Sem -0.7549, dir -0.9762, norm 0.1905 LR: 4.011e-02 Score 98.971 Data time: 0.0903, Forward time: 0.9487, Backward time: 1.1910, DDP time: 0.0051, Total iter time: 2.1316 [2020-12-31 13:56:52,673][root][INFO] - ===> Epoch[1914](9570/5): Loss -0.7596, Sem -0.7596, dir -0.9753, norm 0.1850 LR: 4.008e-02 Score 98.971 Data time: 0.0848, Forward time: 0.9486, Backward time: 1.1911, DDP time: 0.0051, Total iter time: 2.2119 [2020-12-31 13:57:04,986][root][INFO] - ===> Epoch[1915](9575/5): Loss -0.7639, Sem -0.7639, dir -0.9766, norm 0.1816 LR: 4.004e-02 Score 98.957 Data time: 0.1193, Forward time: 0.9486, Backward time: 1.1911, DDP time: 0.0051, Total iter time: 2.3743 [2020-12-31 13:57:17,202][root][INFO] - ===> Epoch[1916](9580/5): Loss -0.7626, Sem -0.7626, dir -0.9765, norm 0.1836 LR: 4.001e-02 Score 98.978 Data time: 0.1522, Forward time: 0.9487, Backward time: 1.1910, DDP time: 0.0051, Total iter time: 2.3650 [2020-12-31 13:57:29,582][root][INFO] - ===> Epoch[1917](9585/5): Loss -0.7525, Sem -0.7525, dir -0.9757, norm 0.1916 LR: 3.998e-02 Score 98.948 Data time: 0.0998, Forward time: 0.9487, Backward time: 1.1911, DDP time: 0.0051, Total iter time: 2.3733 [2020-12-31 13:57:41,725][root][INFO] - ===> Epoch[1918](9590/5): Loss -0.7578, Sem -0.7578, dir -0.9761, norm 0.1857 LR: 3.994e-02 Score 98.910 Data time: 0.1013, Forward time: 0.9486, Backward time: 1.1912, DDP time: 0.0051, Total iter time: 2.3529 [2020-12-31 13:57:53,068][root][INFO] - ===> Epoch[1919](9595/5): Loss -0.7621, Sem -0.7621, dir -0.9761, norm 0.1822 LR: 3.991e-02 Score 98.934 Data time: 0.1158, Forward time: 0.9486, Backward time: 1.1911, DDP time: 0.0051, Total iter time: 2.1769 [2020-12-31 13:58:05,471][root][INFO] - ===> Epoch[1920](9600/5): Loss -0.7598, Sem -0.7598, dir -0.9763, norm 0.1860 LR: 3.988e-02 Score 98.977 Data time: 0.1368, Forward time: 0.9486, Backward time: 1.1911, DDP time: 0.0051, Total iter time: 2.4017 [2020-12-31 13:58:05,473][root][INFO] - ===> Start testing [2020-12-31 13:58:44,614][root][INFO] - 68/68: Data time: 0.0024, Iter time: 0.1712 Loss 1.194 (AVG: 0.541) Score 74.966 (AVG: 89.613) mIOU 68.403 mAP 83.252 mAcc 75.593 IOU: 59.569 0.000 81.879 71.643 93.730 91.168 25.968 76.398 98.420 69.675 81.430 84.707 54.656 mAP: 80.457 6.221 92.785 85.351 98.958 98.371 57.235 93.775 99.822 92.415 92.995 94.046 89.846 mAcc: 80.674 0.000 88.881 78.830 94.995 96.040 28.117 85.555 99.156 85.142 91.317 95.778 58.224 [2020-12-31 13:58:44,616][root][INFO] - Finished test. Elapsed time: 39.1401 [2020-12-31 13:58:48,264][root][INFO] - [2020-12-31 13:58:48,264][root][INFO] - ################################################################ [2020-12-31 13:58:48,264][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 13:58:48,264][root][INFO] - ################################################################ [2020-12-31 13:58:48,264][root][INFO] - clutter : nan nan nan [2020-12-31 13:58:48,264][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 13:58:48,264][root][INFO] - board : 0.787 0.848 0.899 [2020-12-31 13:58:48,264][root][INFO] - bookcase : 0.202 0.341 0.493 [2020-12-31 13:58:48,264][root][INFO] - ceiling : 0.575 0.667 0.695 [2020-12-31 13:58:48,264][root][INFO] - chair : 0.789 0.873 0.902 [2020-12-31 13:58:48,264][root][INFO] - column : 0.131 0.208 0.316 [2020-12-31 13:58:48,264][root][INFO] - door : 0.576 0.769 0.900 [2020-12-31 13:58:48,264][root][INFO] - floor : 0.796 0.835 0.851 [2020-12-31 13:58:48,264][root][INFO] - sofa : 0.541 0.818 0.818 [2020-12-31 13:58:48,264][root][INFO] - table : 0.224 0.430 0.660 [2020-12-31 13:58:48,264][root][INFO] - wall : 0.469 0.678 0.801 [2020-12-31 13:58:48,264][root][INFO] - window : 0.431 0.654 0.734 [2020-12-31 13:58:48,264][root][INFO] - ---------------------------------------------------------------- [2020-12-31 13:58:48,265][root][INFO] - average : 0.460 0.593 0.672 [2020-12-31 13:58:48,265][root][INFO] - [2020-12-31 13:58:48,432][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 13:58:49,005][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter9600.pth [2020-12-31 13:59:01,318][root][INFO] - ===> Epoch[1921](9605/5): Loss -0.7611, Sem -0.7611, dir -0.9764, norm 0.1840 LR: 3.984e-02 Score 98.951 Data time: 0.1298, Forward time: 0.9486, Backward time: 1.1911, DDP time: 0.0051, Total iter time: 2.3633 [2020-12-31 13:59:12,809][root][INFO] - ===> Epoch[1922](9610/5): Loss -0.7611, Sem -0.7611, dir -0.9770, norm 0.1858 LR: 3.981e-02 Score 98.995 Data time: 0.0965, Forward time: 0.9487, Backward time: 1.1910, DDP time: 0.0051, Total iter time: 2.2198 [2020-12-31 13:59:24,644][root][INFO] - ===> Epoch[1923](9615/5): Loss -0.7552, Sem -0.7552, dir -0.9761, norm 0.1888 LR: 3.978e-02 Score 98.928 Data time: 0.1069, Forward time: 0.9487, Backward time: 1.1910, DDP time: 0.0051, Total iter time: 2.2850 [2020-12-31 13:59:35,662][root][INFO] - ===> Epoch[1924](9620/5): Loss -0.7552, Sem -0.7552, dir -0.9762, norm 0.1904 LR: 3.974e-02 Score 98.979 Data time: 0.0910, Forward time: 0.9487, Backward time: 1.1909, DDP time: 0.0051, Total iter time: 2.1131 [2020-12-31 13:59:47,827][root][INFO] - ===> Epoch[1925](9625/5): Loss -0.7552, Sem -0.7552, dir -0.9760, norm 0.1884 LR: 3.971e-02 Score 98.915 Data time: 0.1039, Forward time: 0.9487, Backward time: 1.1909, DDP time: 0.0051, Total iter time: 2.3439 [2020-12-31 13:59:59,256][root][INFO] - ===> Epoch[1926](9630/5): Loss -0.7563, Sem -0.7563, dir -0.9765, norm 0.1886 LR: 3.968e-02 Score 98.947 Data time: 0.0866, Forward time: 0.9487, Backward time: 1.1909, DDP time: 0.0051, Total iter time: 2.1906 [2020-12-31 14:00:12,480][root][INFO] - ===> Epoch[1927](9635/5): Loss -0.7579, Sem -0.7579, dir -0.9763, norm 0.1873 LR: 3.964e-02 Score 98.967 Data time: 0.0871, Forward time: 0.9487, Backward time: 1.1910, DDP time: 0.0051, Total iter time: 2.4998 [2020-12-31 14:00:24,970][root][INFO] - ===> Epoch[1928](9640/5): Loss -0.7631, Sem -0.7631, dir -0.9767, norm 0.1852 LR: 3.961e-02 Score 99.055 Data time: 0.1011, Forward time: 0.9488, Backward time: 1.1910, DDP time: 0.0051, Total iter time: 2.4163 [2020-12-31 14:00:36,582][root][INFO] - ===> Epoch[1929](9645/5): Loss -0.7652, Sem -0.7652, dir -0.9771, norm 0.1811 LR: 3.958e-02 Score 98.971 Data time: 0.0907, Forward time: 0.9487, Backward time: 1.1910, DDP time: 0.0051, Total iter time: 2.2432 [2020-12-31 14:00:48,387][root][INFO] - ===> Epoch[1930](9650/5): Loss -0.7624, Sem -0.7624, dir -0.9766, norm 0.1837 LR: 3.954e-02 Score 98.972 Data time: 0.0871, Forward time: 0.9487, Backward time: 1.1910, DDP time: 0.0051, Total iter time: 2.2781 [2020-12-31 14:01:00,095][root][INFO] - ===> Epoch[1931](9655/5): Loss -0.7526, Sem -0.7526, dir -0.9756, norm 0.1919 LR: 3.951e-02 Score 98.965 Data time: 0.1116, Forward time: 0.9488, Backward time: 1.1909, DDP time: 0.0051, Total iter time: 2.2447 [2020-12-31 14:01:12,490][root][INFO] - ===> Epoch[1932](9660/5): Loss -0.7564, Sem -0.7564, dir -0.9759, norm 0.1883 LR: 3.948e-02 Score 98.957 Data time: 0.1294, Forward time: 0.9489, Backward time: 1.1909, DDP time: 0.0051, Total iter time: 2.3899 [2020-12-31 14:01:24,123][root][INFO] - ===> Epoch[1933](9665/5): Loss -0.7566, Sem -0.7566, dir -0.9761, norm 0.1877 LR: 3.944e-02 Score 98.927 Data time: 0.1494, Forward time: 0.9489, Backward time: 1.1907, DDP time: 0.0051, Total iter time: 2.2273 [2020-12-31 14:01:36,403][root][INFO] - ===> Epoch[1934](9670/5): Loss -0.7576, Sem -0.7576, dir -0.9760, norm 0.1886 LR: 3.941e-02 Score 99.003 Data time: 0.1009, Forward time: 0.9489, Backward time: 1.1908, DDP time: 0.0051, Total iter time: 2.3626 [2020-12-31 14:01:48,508][root][INFO] - ===> Epoch[1935](9675/5): Loss -0.7569, Sem -0.7569, dir -0.9765, norm 0.1881 LR: 3.938e-02 Score 98.956 Data time: 0.1414, Forward time: 0.9490, Backward time: 1.1907, DDP time: 0.0051, Total iter time: 2.3360 [2020-12-31 14:01:59,795][root][INFO] - ===> Epoch[1936](9680/5): Loss -0.7556, Sem -0.7556, dir -0.9757, norm 0.1878 LR: 3.934e-02 Score 98.923 Data time: 0.1110, Forward time: 0.9490, Backward time: 1.1907, DDP time: 0.0051, Total iter time: 2.1655 [2020-12-31 14:02:11,755][root][INFO] - ===> Epoch[1937](9685/5): Loss -0.7516, Sem -0.7516, dir -0.9758, norm 0.1913 LR: 3.931e-02 Score 98.909 Data time: 0.1117, Forward time: 0.9490, Backward time: 1.1906, DDP time: 0.0051, Total iter time: 2.2972 [2020-12-31 14:02:23,364][root][INFO] - ===> Epoch[1938](9690/5): Loss -0.7610, Sem -0.7610, dir -0.9767, norm 0.1854 LR: 3.928e-02 Score 98.985 Data time: 0.0892, Forward time: 0.9490, Backward time: 1.1906, DDP time: 0.0051, Total iter time: 2.2299 [2020-12-31 14:02:34,730][root][INFO] - ===> Epoch[1939](9695/5): Loss -0.7492, Sem -0.7492, dir -0.9761, norm 0.1952 LR: 3.924e-02 Score 98.940 Data time: 0.1248, Forward time: 0.9490, Backward time: 1.1905, DDP time: 0.0051, Total iter time: 2.1786 [2020-12-31 14:02:46,833][root][INFO] - ===> Epoch[1940](9700/5): Loss -0.7582, Sem -0.7582, dir -0.9765, norm 0.1871 LR: 3.921e-02 Score 98.965 Data time: 0.1010, Forward time: 0.9490, Backward time: 1.1905, DDP time: 0.0051, Total iter time: 2.3277 [2020-12-31 14:02:59,737][root][INFO] - ===> Epoch[1941](9705/5): Loss -0.7539, Sem -0.7539, dir -0.9756, norm 0.1911 LR: 3.918e-02 Score 98.970 Data time: 0.1070, Forward time: 0.9490, Backward time: 1.1906, DDP time: 0.0051, Total iter time: 2.4899 [2020-12-31 14:03:11,208][root][INFO] - ===> Epoch[1942](9710/5): Loss -0.7569, Sem -0.7569, dir -0.9762, norm 0.1884 LR: 3.914e-02 Score 98.952 Data time: 0.1236, Forward time: 0.9491, Backward time: 1.1905, DDP time: 0.0051, Total iter time: 2.2073 [2020-12-31 14:03:23,736][root][INFO] - ===> Epoch[1943](9715/5): Loss -0.7494, Sem -0.7494, dir -0.9760, norm 0.1947 LR: 3.911e-02 Score 98.929 Data time: 0.2483, Forward time: 0.9491, Backward time: 1.1904, DDP time: 0.0051, Total iter time: 2.4113 [2020-12-31 14:03:35,606][root][INFO] - ===> Epoch[1944](9720/5): Loss -0.7541, Sem -0.7541, dir -0.9762, norm 0.1910 LR: 3.908e-02 Score 98.965 Data time: 0.1197, Forward time: 0.9492, Backward time: 1.1904, DDP time: 0.0051, Total iter time: 2.2785 [2020-12-31 14:03:46,856][root][INFO] - ===> Epoch[1945](9725/5): Loss -0.7511, Sem -0.7511, dir -0.9766, norm 0.1935 LR: 3.905e-02 Score 98.948 Data time: 0.1533, Forward time: 0.9492, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.1574 [2020-12-31 14:03:59,065][root][INFO] - ===> Epoch[1946](9730/5): Loss -0.7550, Sem -0.7550, dir -0.9764, norm 0.1892 LR: 3.901e-02 Score 98.926 Data time: 0.1250, Forward time: 0.9492, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.3421 [2020-12-31 14:04:11,124][root][INFO] - ===> Epoch[1947](9735/5): Loss -0.7668, Sem -0.7668, dir -0.9773, norm 0.1798 LR: 3.898e-02 Score 98.958 Data time: 0.1029, Forward time: 0.9493, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.3221 [2020-12-31 14:04:23,974][root][INFO] - ===> Epoch[1948](9740/5): Loss -0.7595, Sem -0.7595, dir -0.9764, norm 0.1857 LR: 3.895e-02 Score 98.950 Data time: 0.0818, Forward time: 0.9493, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.4846 [2020-12-31 14:04:36,531][root][INFO] - ===> Epoch[1949](9745/5): Loss -0.7638, Sem -0.7638, dir -0.9763, norm 0.1817 LR: 3.891e-02 Score 98.955 Data time: 0.1037, Forward time: 0.9493, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.4211 [2020-12-31 14:04:48,794][root][INFO] - ===> Epoch[1950](9750/5): Loss -0.7655, Sem -0.7655, dir -0.9765, norm 0.1797 LR: 3.888e-02 Score 98.955 Data time: 0.0947, Forward time: 0.9493, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.3453 [2020-12-31 14:05:00,620][root][INFO] - ===> Epoch[1951](9755/5): Loss -0.7576, Sem -0.7576, dir -0.9761, norm 0.1871 LR: 3.885e-02 Score 98.944 Data time: 0.1286, Forward time: 0.9493, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.2783 [2020-12-31 14:05:12,544][root][INFO] - ===> Epoch[1952](9760/5): Loss -0.7596, Sem -0.7596, dir -0.9764, norm 0.1857 LR: 3.881e-02 Score 98.953 Data time: 0.1037, Forward time: 0.9493, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.2967 [2020-12-31 14:05:24,512][root][INFO] - ===> Epoch[1953](9765/5): Loss -0.7632, Sem -0.7632, dir -0.9766, norm 0.1833 LR: 3.878e-02 Score 98.990 Data time: 0.1672, Forward time: 0.9494, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.2933 [2020-12-31 14:05:36,850][root][INFO] - ===> Epoch[1954](9770/5): Loss -0.7542, Sem -0.7542, dir -0.9763, norm 0.1910 LR: 3.875e-02 Score 98.961 Data time: 0.1284, Forward time: 0.9494, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.3777 [2020-12-31 14:05:48,768][root][INFO] - ===> Epoch[1955](9775/5): Loss -0.7661, Sem -0.7661, dir -0.9765, norm 0.1808 LR: 3.871e-02 Score 99.011 Data time: 0.1208, Forward time: 0.9494, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.2895 [2020-12-31 14:06:00,386][root][INFO] - ===> Epoch[1956](9780/5): Loss -0.7663, Sem -0.7663, dir -0.9769, norm 0.1803 LR: 3.868e-02 Score 98.998 Data time: 0.0984, Forward time: 0.9494, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.2402 [2020-12-31 14:06:12,355][root][INFO] - ===> Epoch[1957](9785/5): Loss -0.7576, Sem -0.7576, dir -0.9765, norm 0.1881 LR: 3.865e-02 Score 98.970 Data time: 0.0982, Forward time: 0.9495, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.3116 [2020-12-31 14:06:24,451][root][INFO] - ===> Epoch[1958](9790/5): Loss -0.7606, Sem -0.7606, dir -0.9773, norm 0.1864 LR: 3.861e-02 Score 98.988 Data time: 0.1054, Forward time: 0.9495, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.3237 [2020-12-31 14:06:37,014][root][INFO] - ===> Epoch[1959](9795/5): Loss -0.7641, Sem -0.7641, dir -0.9766, norm 0.1824 LR: 3.858e-02 Score 98.995 Data time: 0.1034, Forward time: 0.9495, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.4202 [2020-12-31 14:06:49,389][root][INFO] - ===> Epoch[1960](9800/5): Loss -0.7589, Sem -0.7589, dir -0.9766, norm 0.1848 LR: 3.855e-02 Score 98.898 Data time: 0.1205, Forward time: 0.9495, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.3908 [2020-12-31 14:06:49,391][root][INFO] - ===> Start testing [2020-12-31 14:07:28,519][root][INFO] - 68/68: Data time: 0.0033, Iter time: 0.2072 Loss 1.051 (AVG: 0.515) Score 80.690 (AVG: 89.977) mIOU 68.256 mAP 83.212 mAcc 76.018 IOU: 60.416 0.000 79.702 73.891 93.723 91.018 28.354 76.851 98.013 61.013 80.437 85.438 58.470 mAP: 80.966 10.554 92.622 86.402 99.041 97.907 55.615 94.293 99.791 86.948 92.650 94.035 90.925 mAcc: 80.053 0.000 86.049 81.790 95.082 96.994 31.821 86.309 98.894 81.768 90.500 95.335 63.645 [2020-12-31 14:07:28,520][root][INFO] - Finished test. Elapsed time: 39.1268 [2020-12-31 14:07:31,770][root][INFO] - [2020-12-31 14:07:31,770][root][INFO] - ################################################################ [2020-12-31 14:07:31,770][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 14:07:31,770][root][INFO] - ################################################################ [2020-12-31 14:07:31,771][root][INFO] - clutter : nan nan nan [2020-12-31 14:07:31,771][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 14:07:31,771][root][INFO] - board : 0.766 0.814 0.900 [2020-12-31 14:07:31,771][root][INFO] - bookcase : 0.199 0.345 0.522 [2020-12-31 14:07:31,771][root][INFO] - ceiling : 0.587 0.637 0.726 [2020-12-31 14:07:31,771][root][INFO] - chair : 0.799 0.891 0.914 [2020-12-31 14:07:31,771][root][INFO] - column : 0.137 0.209 0.331 [2020-12-31 14:07:31,771][root][INFO] - door : 0.427 0.605 0.825 [2020-12-31 14:07:31,771][root][INFO] - floor : 0.936 0.950 0.977 [2020-12-31 14:07:31,771][root][INFO] - sofa : 0.526 0.809 0.809 [2020-12-31 14:07:31,771][root][INFO] - table : 0.177 0.348 0.610 [2020-12-31 14:07:31,771][root][INFO] - wall : 0.459 0.679 0.762 [2020-12-31 14:07:31,771][root][INFO] - window : 0.496 0.763 0.783 [2020-12-31 14:07:31,771][root][INFO] - ---------------------------------------------------------------- [2020-12-31 14:07:31,771][root][INFO] - average : 0.459 0.587 0.680 [2020-12-31 14:07:31,771][root][INFO] - [2020-12-31 14:07:31,959][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 14:07:32,574][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter9800.pth [2020-12-31 14:07:43,502][root][INFO] - ===> Epoch[1961](9805/5): Loss -0.7657, Sem -0.7657, dir -0.9762, norm 0.1793 LR: 3.851e-02 Score 98.956 Data time: 0.0981, Forward time: 0.9495, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.0844 [2020-12-31 14:07:55,415][root][INFO] - ===> Epoch[1962](9810/5): Loss -0.7599, Sem -0.7599, dir -0.9774, norm 0.1873 LR: 3.848e-02 Score 98.980 Data time: 0.0936, Forward time: 0.9495, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.2849 [2020-12-31 14:08:07,930][root][INFO] - ===> Epoch[1963](9815/5): Loss -0.7557, Sem -0.7557, dir -0.9767, norm 0.1886 LR: 3.845e-02 Score 98.925 Data time: 0.1094, Forward time: 0.9494, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.4165 [2020-12-31 14:08:19,955][root][INFO] - ===> Epoch[1964](9820/5): Loss -0.7557, Sem -0.7557, dir -0.9764, norm 0.1886 LR: 3.841e-02 Score 98.930 Data time: 0.1133, Forward time: 0.9494, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.3078 [2020-12-31 14:08:32,182][root][INFO] - ===> Epoch[1965](9825/5): Loss -0.7557, Sem -0.7557, dir -0.9762, norm 0.1876 LR: 3.838e-02 Score 98.923 Data time: 0.1016, Forward time: 0.9494, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.3620 [2020-12-31 14:08:44,028][root][INFO] - ===> Epoch[1966](9830/5): Loss -0.7555, Sem -0.7555, dir -0.9763, norm 0.1899 LR: 3.835e-02 Score 98.970 Data time: 0.1275, Forward time: 0.9494, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.2758 [2020-12-31 14:08:56,660][root][INFO] - ===> Epoch[1967](9835/5): Loss -0.7627, Sem -0.7627, dir -0.9764, norm 0.1826 LR: 3.831e-02 Score 98.961 Data time: 0.1253, Forward time: 0.9494, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.4322 [2020-12-31 14:09:08,179][root][INFO] - ===> Epoch[1968](9840/5): Loss -0.7564, Sem -0.7564, dir -0.9760, norm 0.1880 LR: 3.828e-02 Score 98.949 Data time: 0.1095, Forward time: 0.9493, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.2152 [2020-12-31 14:09:20,058][root][INFO] - ===> Epoch[1969](9845/5): Loss -0.7548, Sem -0.7548, dir -0.9759, norm 0.1892 LR: 3.824e-02 Score 98.940 Data time: 0.1284, Forward time: 0.9494, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.2818 [2020-12-31 14:09:32,346][root][INFO] - ===> Epoch[1970](9850/5): Loss -0.7613, Sem -0.7613, dir -0.9762, norm 0.1848 LR: 3.821e-02 Score 98.992 Data time: 0.0974, Forward time: 0.9493, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.3669 [2020-12-31 14:09:45,038][root][INFO] - ===> Epoch[1971](9855/5): Loss -0.7568, Sem -0.7568, dir -0.9767, norm 0.1896 LR: 3.818e-02 Score 98.978 Data time: 0.0806, Forward time: 0.9493, Backward time: 1.1904, DDP time: 0.0051, Total iter time: 2.4480 [2020-12-31 14:09:56,665][root][INFO] - ===> Epoch[1972](9860/5): Loss -0.7654, Sem -0.7654, dir -0.9769, norm 0.1811 LR: 3.814e-02 Score 98.980 Data time: 0.1398, Forward time: 0.9493, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.2366 [2020-12-31 14:10:09,054][root][INFO] - ===> Epoch[1973](9865/5): Loss -0.7580, Sem -0.7580, dir -0.9767, norm 0.1887 LR: 3.811e-02 Score 98.994 Data time: 0.1356, Forward time: 0.9494, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.3882 [2020-12-31 14:10:21,582][root][INFO] - ===> Epoch[1974](9870/5): Loss -0.7561, Sem -0.7561, dir -0.9767, norm 0.1897 LR: 3.808e-02 Score 98.976 Data time: 0.2074, Forward time: 0.9495, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.4051 [2020-12-31 14:10:33,870][root][INFO] - ===> Epoch[1975](9875/5): Loss -0.7578, Sem -0.7578, dir -0.9765, norm 0.1884 LR: 3.804e-02 Score 98.995 Data time: 0.2014, Forward time: 0.9494, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.3755 [2020-12-31 14:10:46,302][root][INFO] - ===> Epoch[1976](9880/5): Loss -0.7571, Sem -0.7571, dir -0.9762, norm 0.1883 LR: 3.801e-02 Score 98.966 Data time: 0.0718, Forward time: 0.9494, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.4009 [2020-12-31 14:10:57,653][root][INFO] - ===> Epoch[1977](9885/5): Loss -0.7633, Sem -0.7633, dir -0.9768, norm 0.1838 LR: 3.798e-02 Score 99.010 Data time: 0.1162, Forward time: 0.9494, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.1819 [2020-12-31 14:11:08,940][root][INFO] - ===> Epoch[1978](9890/5): Loss -0.7577, Sem -0.7577, dir -0.9767, norm 0.1873 LR: 3.794e-02 Score 98.955 Data time: 0.1161, Forward time: 0.9494, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.1441 [2020-12-31 14:11:20,746][root][INFO] - ===> Epoch[1979](9895/5): Loss -0.7680, Sem -0.7680, dir -0.9771, norm 0.1783 LR: 3.791e-02 Score 98.975 Data time: 0.0820, Forward time: 0.9493, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.2789 [2020-12-31 14:11:31,796][root][INFO] - ===> Epoch[1980](9900/5): Loss -0.7597, Sem -0.7597, dir -0.9756, norm 0.1832 LR: 3.788e-02 Score 98.934 Data time: 0.1014, Forward time: 0.9493, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.1167 [2020-12-31 14:11:42,846][root][INFO] - ===> Epoch[1981](9905/5): Loss -0.7626, Sem -0.7626, dir -0.9768, norm 0.1829 LR: 3.784e-02 Score 98.959 Data time: 0.1598, Forward time: 0.9493, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.1189 [2020-12-31 14:11:54,540][root][INFO] - ===> Epoch[1982](9910/5): Loss -0.7618, Sem -0.7618, dir -0.9765, norm 0.1824 LR: 3.781e-02 Score 98.943 Data time: 0.0947, Forward time: 0.9492, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.2446 [2020-12-31 14:12:05,984][root][INFO] - ===> Epoch[1983](9915/5): Loss -0.7636, Sem -0.7636, dir -0.9773, norm 0.1831 LR: 3.778e-02 Score 98.973 Data time: 0.1461, Forward time: 0.9493, Backward time: 1.1900, DDP time: 0.0051, Total iter time: 2.2068 [2020-12-31 14:12:18,019][root][INFO] - ===> Epoch[1984](9920/5): Loss -0.7649, Sem -0.7649, dir -0.9765, norm 0.1810 LR: 3.774e-02 Score 98.970 Data time: 0.1002, Forward time: 0.9493, Backward time: 1.1900, DDP time: 0.0051, Total iter time: 2.3099 [2020-12-31 14:12:30,423][root][INFO] - ===> Epoch[1985](9925/5): Loss -0.7519, Sem -0.7519, dir -0.9757, norm 0.1925 LR: 3.771e-02 Score 98.967 Data time: 0.1152, Forward time: 0.9492, Backward time: 1.1900, DDP time: 0.0051, Total iter time: 2.3897 [2020-12-31 14:12:42,534][root][INFO] - ===> Epoch[1986](9930/5): Loss -0.7683, Sem -0.7683, dir -0.9770, norm 0.1786 LR: 3.768e-02 Score 98.989 Data time: 0.2195, Forward time: 0.9492, Backward time: 1.1900, DDP time: 0.0051, Total iter time: 2.3361 [2020-12-31 14:12:54,904][root][INFO] - ===> Epoch[1987](9935/5): Loss -0.7579, Sem -0.7579, dir -0.9762, norm 0.1870 LR: 3.764e-02 Score 98.964 Data time: 0.1530, Forward time: 0.9492, Backward time: 1.1900, DDP time: 0.0051, Total iter time: 2.3866 [2020-12-31 14:13:06,763][root][INFO] - ===> Epoch[1988](9940/5): Loss -0.7598, Sem -0.7598, dir -0.9770, norm 0.1866 LR: 3.761e-02 Score 98.979 Data time: 0.0863, Forward time: 0.9492, Backward time: 1.1900, DDP time: 0.0051, Total iter time: 2.2731 [2020-12-31 14:13:19,228][root][INFO] - ===> Epoch[1989](9945/5): Loss -0.7683, Sem -0.7683, dir -0.9771, norm 0.1784 LR: 3.758e-02 Score 98.989 Data time: 0.0788, Forward time: 0.9492, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.3867 [2020-12-31 14:13:31,778][root][INFO] - ===> Epoch[1990](9950/5): Loss -0.7555, Sem -0.7555, dir -0.9761, norm 0.1891 LR: 3.754e-02 Score 98.940 Data time: 0.1356, Forward time: 0.9491, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.4251 [2020-12-31 14:13:43,333][root][INFO] - ===> Epoch[1991](9955/5): Loss -0.7600, Sem -0.7600, dir -0.9769, norm 0.1851 LR: 3.751e-02 Score 98.938 Data time: 0.1665, Forward time: 0.9491, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.2262 [2020-12-31 14:13:56,704][root][INFO] - ===> Epoch[1992](9960/5): Loss -0.7566, Sem -0.7566, dir -0.9760, norm 0.1870 LR: 3.748e-02 Score 98.913 Data time: 0.1168, Forward time: 0.9492, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.5765 [2020-12-31 14:14:08,904][root][INFO] - ===> Epoch[1993](9965/5): Loss -0.7511, Sem -0.7511, dir -0.9756, norm 0.1920 LR: 3.744e-02 Score 98.915 Data time: 0.0824, Forward time: 0.9491, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.3521 [2020-12-31 14:14:20,283][root][INFO] - ===> Epoch[1994](9970/5): Loss -0.7637, Sem -0.7637, dir -0.9774, norm 0.1844 LR: 3.741e-02 Score 99.028 Data time: 0.0998, Forward time: 0.9492, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.1784 [2020-12-31 14:14:32,751][root][INFO] - ===> Epoch[1995](9975/5): Loss -0.7631, Sem -0.7631, dir -0.9764, norm 0.1826 LR: 3.738e-02 Score 98.965 Data time: 0.0798, Forward time: 0.9491, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.4111 [2020-12-31 14:14:44,111][root][INFO] - ===> Epoch[1996](9980/5): Loss -0.7620, Sem -0.7620, dir -0.9767, norm 0.1827 LR: 3.734e-02 Score 98.944 Data time: 0.1602, Forward time: 0.9492, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.1806 [2020-12-31 14:14:56,556][root][INFO] - ===> Epoch[1997](9985/5): Loss -0.7632, Sem -0.7632, dir -0.9760, norm 0.1817 LR: 3.731e-02 Score 98.948 Data time: 0.1069, Forward time: 0.9491, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.3976 [2020-12-31 14:15:07,910][root][INFO] - ===> Epoch[1998](9990/5): Loss -0.7534, Sem -0.7534, dir -0.9767, norm 0.1921 LR: 3.728e-02 Score 98.969 Data time: 0.0965, Forward time: 0.9491, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.1953 [2020-12-31 14:15:19,445][root][INFO] - ===> Epoch[1999](9995/5): Loss -0.7571, Sem -0.7571, dir -0.9768, norm 0.1883 LR: 3.724e-02 Score 98.947 Data time: 0.0835, Forward time: 0.9491, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.2292 [2020-12-31 14:15:30,977][root][INFO] - ===> Epoch[2000](10000/5): Loss -0.7684, Sem -0.7684, dir -0.9776, norm 0.1791 LR: 3.721e-02 Score 98.993 Data time: 0.1042, Forward time: 0.9491, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.2112 [2020-12-31 14:15:30,979][root][INFO] - ===> Start testing [2020-12-31 14:16:09,815][root][INFO] - 68/68: Data time: 0.0025, Iter time: 0.1462 Loss 0.176 (AVG: 0.541) Score 95.642 (AVG: 89.649) mIOU 67.743 mAP 83.094 mAcc 75.333 IOU: 58.957 0.000 76.794 72.174 93.505 90.778 27.344 76.377 98.458 62.985 80.493 85.212 57.579 mAP: 80.502 5.440 94.198 85.339 98.939 98.342 57.321 94.429 99.831 87.696 93.277 94.473 90.437 mAcc: 80.154 0.000 89.748 79.199 95.012 95.468 29.662 87.355 99.308 76.786 89.584 95.446 61.607 [2020-12-31 14:16:09,816][root][INFO] - Finished test. Elapsed time: 38.8349 [2020-12-31 14:16:13,495][root][INFO] - [2020-12-31 14:16:13,495][root][INFO] - ################################################################ [2020-12-31 14:16:13,495][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 14:16:13,495][root][INFO] - ################################################################ [2020-12-31 14:16:13,495][root][INFO] - clutter : nan nan nan [2020-12-31 14:16:13,495][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 14:16:13,495][root][INFO] - board : 0.783 0.868 0.896 [2020-12-31 14:16:13,495][root][INFO] - bookcase : 0.198 0.337 0.507 [2020-12-31 14:16:13,495][root][INFO] - ceiling : 0.630 0.678 0.750 [2020-12-31 14:16:13,495][root][INFO] - chair : 0.783 0.880 0.884 [2020-12-31 14:16:13,495][root][INFO] - column : 0.169 0.231 0.285 [2020-12-31 14:16:13,495][root][INFO] - door : 0.496 0.664 0.892 [2020-12-31 14:16:13,495][root][INFO] - floor : 0.970 0.977 0.977 [2020-12-31 14:16:13,495][root][INFO] - sofa : 0.435 0.601 0.818 [2020-12-31 14:16:13,495][root][INFO] - table : 0.188 0.366 0.628 [2020-12-31 14:16:13,495][root][INFO] - wall : 0.483 0.702 0.804 [2020-12-31 14:16:13,496][root][INFO] - window : 0.497 0.765 0.784 [2020-12-31 14:16:13,496][root][INFO] - ---------------------------------------------------------------- [2020-12-31 14:16:13,496][root][INFO] - average : 0.469 0.589 0.685 [2020-12-31 14:16:13,496][root][INFO] - [2020-12-31 14:16:13,665][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 14:16:14,294][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter10000.pth [2020-12-31 14:16:24,819][root][INFO] - ===> Epoch[2001](10005/5): Loss -0.7545, Sem -0.7545, dir -0.9759, norm 0.1902 LR: 3.718e-02 Score 98.961 Data time: 0.0725, Forward time: 0.9490, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.0200 [2020-12-31 14:16:37,121][root][INFO] - ===> Epoch[2002](10010/5): Loss -0.7660, Sem -0.7660, dir -0.9766, norm 0.1801 LR: 3.714e-02 Score 98.975 Data time: 0.1028, Forward time: 0.9490, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.3614 [2020-12-31 14:16:49,410][root][INFO] - ===> Epoch[2003](10015/5): Loss -0.7560, Sem -0.7560, dir -0.9762, norm 0.1883 LR: 3.711e-02 Score 98.930 Data time: 0.1089, Forward time: 0.9490, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.3664 [2020-12-31 14:17:02,058][root][INFO] - ===> Epoch[2004](10020/5): Loss -0.7647, Sem -0.7647, dir -0.9774, norm 0.1815 LR: 3.707e-02 Score 98.962 Data time: 0.1123, Forward time: 0.9490, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.4285 [2020-12-31 14:17:13,927][root][INFO] - ===> Epoch[2005](10025/5): Loss -0.7601, Sem -0.7601, dir -0.9764, norm 0.1850 LR: 3.704e-02 Score 98.956 Data time: 0.0945, Forward time: 0.9490, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.2846 [2020-12-31 14:17:25,096][root][INFO] - ===> Epoch[2006](10030/5): Loss -0.7616, Sem -0.7616, dir -0.9765, norm 0.1843 LR: 3.701e-02 Score 98.978 Data time: 0.0724, Forward time: 0.9489, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.1458 [2020-12-31 14:17:36,893][root][INFO] - ===> Epoch[2007](10035/5): Loss -0.7633, Sem -0.7633, dir -0.9772, norm 0.1838 LR: 3.697e-02 Score 98.999 Data time: 0.1068, Forward time: 0.9490, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.2748 [2020-12-31 14:17:48,509][root][INFO] - ===> Epoch[2008](10040/5): Loss -0.7608, Sem -0.7608, dir -0.9770, norm 0.1868 LR: 3.694e-02 Score 99.009 Data time: 0.1586, Forward time: 0.9489, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.2288 [2020-12-31 14:18:01,035][root][INFO] - ===> Epoch[2009](10045/5): Loss -0.7635, Sem -0.7635, dir -0.9769, norm 0.1838 LR: 3.691e-02 Score 99.001 Data time: 0.1507, Forward time: 0.9489, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.4206 [2020-12-31 14:18:14,085][root][INFO] - ===> Epoch[2010](10050/5): Loss -0.7584, Sem -0.7584, dir -0.9769, norm 0.1876 LR: 3.687e-02 Score 98.975 Data time: 0.1858, Forward time: 0.9490, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.5192 [2020-12-31 14:18:25,407][root][INFO] - ===> Epoch[2011](10055/5): Loss -0.7617, Sem -0.7617, dir -0.9777, norm 0.1854 LR: 3.684e-02 Score 98.993 Data time: 0.1197, Forward time: 0.9490, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.1706 [2020-12-31 14:18:37,834][root][INFO] - ===> Epoch[2012](10060/5): Loss -0.7675, Sem -0.7675, dir -0.9767, norm 0.1784 LR: 3.681e-02 Score 98.966 Data time: 0.0884, Forward time: 0.9490, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.3852 [2020-12-31 14:18:49,113][root][INFO] - ===> Epoch[2013](10065/5): Loss -0.7596, Sem -0.7596, dir -0.9766, norm 0.1855 LR: 3.677e-02 Score 98.953 Data time: 0.2129, Forward time: 0.9489, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.1666 [2020-12-31 14:19:01,603][root][INFO] - ===> Epoch[2014](10070/5): Loss -0.7658, Sem -0.7658, dir -0.9768, norm 0.1791 LR: 3.674e-02 Score 98.943 Data time: 0.1142, Forward time: 0.9489, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.3986 [2020-12-31 14:19:13,680][root][INFO] - ===> Epoch[2015](10075/5): Loss -0.7657, Sem -0.7657, dir -0.9771, norm 0.1804 LR: 3.671e-02 Score 98.956 Data time: 0.0901, Forward time: 0.9489, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.3303 [2020-12-31 14:19:26,711][root][INFO] - ===> Epoch[2016](10080/5): Loss -0.7722, Sem -0.7722, dir -0.9780, norm 0.1767 LR: 3.667e-02 Score 99.031 Data time: 0.1410, Forward time: 0.9489, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.5137 [2020-12-31 14:19:38,513][root][INFO] - ===> Epoch[2017](10085/5): Loss -0.7637, Sem -0.7637, dir -0.9768, norm 0.1828 LR: 3.664e-02 Score 98.984 Data time: 0.0919, Forward time: 0.9489, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.2697 [2020-12-31 14:19:50,552][root][INFO] - ===> Epoch[2018](10090/5): Loss -0.7655, Sem -0.7655, dir -0.9772, norm 0.1814 LR: 3.661e-02 Score 98.992 Data time: 0.1637, Forward time: 0.9489, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.3092 [2020-12-31 14:20:01,866][root][INFO] - ===> Epoch[2019](10095/5): Loss -0.7606, Sem -0.7606, dir -0.9765, norm 0.1847 LR: 3.657e-02 Score 98.952 Data time: 0.1754, Forward time: 0.9489, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.1763 [2020-12-31 14:20:13,745][root][INFO] - ===> Epoch[2020](10100/5): Loss -0.7525, Sem -0.7525, dir -0.9764, norm 0.1921 LR: 3.654e-02 Score 98.947 Data time: 0.0806, Forward time: 0.9489, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.2936 [2020-12-31 14:20:25,507][root][INFO] - ===> Epoch[2021](10105/5): Loss -0.7651, Sem -0.7651, dir -0.9772, norm 0.1823 LR: 3.650e-02 Score 99.000 Data time: 0.1307, Forward time: 0.9489, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.2685 [2020-12-31 14:20:37,638][root][INFO] - ===> Epoch[2022](10110/5): Loss -0.7594, Sem -0.7594, dir -0.9775, norm 0.1875 LR: 3.647e-02 Score 98.981 Data time: 0.1009, Forward time: 0.9489, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.3344 [2020-12-31 14:20:50,148][root][INFO] - ===> Epoch[2023](10115/5): Loss -0.7690, Sem -0.7690, dir -0.9767, norm 0.1774 LR: 3.644e-02 Score 98.980 Data time: 0.1263, Forward time: 0.9489, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.3997 [2020-12-31 14:21:01,054][root][INFO] - ===> Epoch[2024](10120/5): Loss -0.7613, Sem -0.7613, dir -0.9770, norm 0.1827 LR: 3.640e-02 Score 98.912 Data time: 0.0928, Forward time: 0.9489, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.0864 [2020-12-31 14:21:12,592][root][INFO] - ===> Epoch[2025](10125/5): Loss -0.7679, Sem -0.7679, dir -0.9771, norm 0.1787 LR: 3.637e-02 Score 98.977 Data time: 0.0636, Forward time: 0.9488, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.2304 [2020-12-31 14:21:25,086][root][INFO] - ===> Epoch[2026](10130/5): Loss -0.7661, Sem -0.7661, dir -0.9772, norm 0.1800 LR: 3.634e-02 Score 98.978 Data time: 0.0801, Forward time: 0.9488, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.4260 [2020-12-31 14:21:36,494][root][INFO] - ===> Epoch[2027](10135/5): Loss -0.7677, Sem -0.7677, dir -0.9775, norm 0.1795 LR: 3.630e-02 Score 98.983 Data time: 0.1436, Forward time: 0.9488, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.1921 [2020-12-31 14:21:48,171][root][INFO] - ===> Epoch[2028](10140/5): Loss -0.7626, Sem -0.7626, dir -0.9769, norm 0.1833 LR: 3.627e-02 Score 98.968 Data time: 0.1025, Forward time: 0.9488, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.2467 [2020-12-31 14:22:00,051][root][INFO] - ===> Epoch[2029](10145/5): Loss -0.7736, Sem -0.7736, dir -0.9775, norm 0.1738 LR: 3.624e-02 Score 98.981 Data time: 0.1224, Forward time: 0.9488, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.2898 [2020-12-31 14:22:12,355][root][INFO] - ===> Epoch[2030](10150/5): Loss -0.7651, Sem -0.7651, dir -0.9771, norm 0.1809 LR: 3.620e-02 Score 98.969 Data time: 0.1154, Forward time: 0.9488, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.3720 [2020-12-31 14:22:24,025][root][INFO] - ===> Epoch[2031](10155/5): Loss -0.7737, Sem -0.7737, dir -0.9774, norm 0.1743 LR: 3.617e-02 Score 99.009 Data time: 0.1217, Forward time: 0.9488, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.2462 [2020-12-31 14:22:35,947][root][INFO] - ===> Epoch[2032](10160/5): Loss -0.7670, Sem -0.7670, dir -0.9772, norm 0.1803 LR: 3.614e-02 Score 99.009 Data time: 0.0975, Forward time: 0.9488, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.2959 [2020-12-31 14:22:48,503][root][INFO] - ===> Epoch[2033](10165/5): Loss -0.7651, Sem -0.7651, dir -0.9776, norm 0.1828 LR: 3.610e-02 Score 98.998 Data time: 0.1004, Forward time: 0.9488, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.4147 [2020-12-31 14:23:01,287][root][INFO] - ===> Epoch[2034](10170/5): Loss -0.7633, Sem -0.7633, dir -0.9780, norm 0.1847 LR: 3.607e-02 Score 98.995 Data time: 0.1936, Forward time: 0.9488, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.4658 [2020-12-31 14:23:13,085][root][INFO] - ===> Epoch[2035](10175/5): Loss -0.7660, Sem -0.7660, dir -0.9771, norm 0.1814 LR: 3.603e-02 Score 99.005 Data time: 0.1489, Forward time: 0.9488, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.2689 [2020-12-31 14:23:24,760][root][INFO] - ===> Epoch[2036](10180/5): Loss -0.7672, Sem -0.7672, dir -0.9773, norm 0.1805 LR: 3.600e-02 Score 99.006 Data time: 0.0827, Forward time: 0.9488, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.2528 [2020-12-31 14:23:36,454][root][INFO] - ===> Epoch[2037](10185/5): Loss -0.7680, Sem -0.7680, dir -0.9772, norm 0.1786 LR: 3.597e-02 Score 98.975 Data time: 0.1288, Forward time: 0.9487, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.2424 [2020-12-31 14:23:48,230][root][INFO] - ===> Epoch[2038](10190/5): Loss -0.7609, Sem -0.7609, dir -0.9767, norm 0.1859 LR: 3.593e-02 Score 98.998 Data time: 0.0977, Forward time: 0.9487, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.2532 [2020-12-31 14:23:59,878][root][INFO] - ===> Epoch[2039](10195/5): Loss -0.7698, Sem -0.7698, dir -0.9774, norm 0.1773 LR: 3.590e-02 Score 98.993 Data time: 0.1309, Forward time: 0.9487, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.2344 [2020-12-31 14:24:12,148][root][INFO] - ===> Epoch[2040](10200/5): Loss -0.7688, Sem -0.7688, dir -0.9775, norm 0.1789 LR: 3.587e-02 Score 99.003 Data time: 0.1225, Forward time: 0.9487, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.3560 [2020-12-31 14:24:12,151][root][INFO] - ===> Start testing [2020-12-31 14:24:51,550][root][INFO] - 68/68: Data time: 0.0025, Iter time: 0.1420 Loss 0.232 (AVG: 0.515) Score 95.367 (AVG: 89.828) mIOU 68.619 mAP 83.316 mAcc 75.959 IOU: 60.125 0.000 82.666 72.745 93.661 91.270 31.863 77.812 98.227 61.017 80.988 84.847 56.828 mAP: 81.855 8.801 93.504 85.370 98.918 98.429 57.318 94.797 99.801 85.873 93.692 93.899 90.856 mAcc: 80.080 0.000 89.367 80.780 95.029 96.457 36.362 88.726 98.855 75.434 90.415 95.060 60.895 [2020-12-31 14:24:51,551][root][INFO] - Finished test. Elapsed time: 39.3990 [2020-12-31 14:24:54,713][root][INFO] - [2020-12-31 14:24:54,713][root][INFO] - ################################################################ [2020-12-31 14:24:54,713][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 14:24:54,713][root][INFO] - ################################################################ [2020-12-31 14:24:54,713][root][INFO] - clutter : nan nan nan [2020-12-31 14:24:54,713][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 14:24:54,713][root][INFO] - board : 0.777 0.866 0.892 [2020-12-31 14:24:54,713][root][INFO] - bookcase : 0.210 0.340 0.501 [2020-12-31 14:24:54,713][root][INFO] - ceiling : 0.615 0.693 0.737 [2020-12-31 14:24:54,713][root][INFO] - chair : 0.802 0.897 0.917 [2020-12-31 14:24:54,713][root][INFO] - column : 0.147 0.226 0.391 [2020-12-31 14:24:54,713][root][INFO] - door : 0.496 0.697 0.871 [2020-12-31 14:24:54,713][root][INFO] - floor : 0.953 0.965 0.965 [2020-12-31 14:24:54,713][root][INFO] - sofa : 0.406 0.591 0.818 [2020-12-31 14:24:54,713][root][INFO] - table : 0.215 0.413 0.675 [2020-12-31 14:24:54,713][root][INFO] - wall : 0.471 0.698 0.778 [2020-12-31 14:24:54,713][root][INFO] - window : 0.477 0.709 0.776 [2020-12-31 14:24:54,714][root][INFO] - ---------------------------------------------------------------- [2020-12-31 14:24:54,714][root][INFO] - average : 0.464 0.591 0.693 [2020-12-31 14:24:54,714][root][INFO] - [2020-12-31 14:24:54,891][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 14:24:55,482][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2020-12-31 14:24:55,483][root][INFO] - Current best mIoU: 68.619 at iter 1 [2020-12-31 14:24:56,077][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter10200.pth [2020-12-31 14:25:07,492][root][INFO] - ===> Epoch[2041](10205/5): Loss -0.7664, Sem -0.7664, dir -0.9766, norm 0.1797 LR: 3.583e-02 Score 98.985 Data time: 0.1042, Forward time: 0.9487, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.1754 [2020-12-31 14:25:19,636][root][INFO] - ===> Epoch[2042](10210/5): Loss -0.7705, Sem -0.7705, dir -0.9778, norm 0.1783 LR: 3.580e-02 Score 99.035 Data time: 0.1174, Forward time: 0.9487, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.3249 [2020-12-31 14:25:31,907][root][INFO] - ===> Epoch[2043](10215/5): Loss -0.7722, Sem -0.7722, dir -0.9776, norm 0.1762 LR: 3.577e-02 Score 99.029 Data time: 0.1055, Forward time: 0.9487, Backward time: 1.1901, DDP time: 0.0051, Total iter time: 2.3603 [2020-12-31 14:25:43,621][root][INFO] - ===> Epoch[2044](10220/5): Loss -0.7708, Sem -0.7708, dir -0.9770, norm 0.1770 LR: 3.573e-02 Score 99.027 Data time: 0.0906, Forward time: 0.9486, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.2694 [2020-12-31 14:25:55,525][root][INFO] - ===> Epoch[2045](10225/5): Loss -0.7650, Sem -0.7650, dir -0.9768, norm 0.1814 LR: 3.570e-02 Score 98.977 Data time: 0.1675, Forward time: 0.9486, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.3064 [2020-12-31 14:26:06,982][root][INFO] - ===> Epoch[2046](10230/5): Loss -0.7703, Sem -0.7703, dir -0.9775, norm 0.1780 LR: 3.566e-02 Score 99.018 Data time: 0.0788, Forward time: 0.9485, Backward time: 1.1902, DDP time: 0.0051, Total iter time: 2.1968 [2020-12-31 14:26:19,589][root][INFO] - ===> Epoch[2047](10235/5): Loss -0.7687, Sem -0.7687, dir -0.9772, norm 0.1790 LR: 3.563e-02 Score 99.017 Data time: 0.0897, Forward time: 0.9485, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.4247 [2020-12-31 14:26:31,624][root][INFO] - ===> Epoch[2048](10240/5): Loss -0.7619, Sem -0.7619, dir -0.9771, norm 0.1853 LR: 3.560e-02 Score 99.000 Data time: 0.1429, Forward time: 0.9485, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.3136 [2020-12-31 14:26:44,435][root][INFO] - ===> Epoch[2049](10245/5): Loss -0.7718, Sem -0.7718, dir -0.9774, norm 0.1756 LR: 3.556e-02 Score 99.004 Data time: 0.0956, Forward time: 0.9485, Backward time: 1.1904, DDP time: 0.0051, Total iter time: 2.4913 [2020-12-31 14:26:56,346][root][INFO] - ===> Epoch[2050](10250/5): Loss -0.7644, Sem -0.7644, dir -0.9768, norm 0.1814 LR: 3.553e-02 Score 98.966 Data time: 0.1498, Forward time: 0.9484, Backward time: 1.1904, DDP time: 0.0051, Total iter time: 2.2991 [2020-12-31 14:27:07,308][root][INFO] - ===> Epoch[2051](10255/5): Loss -0.7734, Sem -0.7734, dir -0.9778, norm 0.1745 LR: 3.550e-02 Score 99.002 Data time: 0.1273, Forward time: 0.9484, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.0932 [2020-12-31 14:27:19,548][root][INFO] - ===> Epoch[2052](10260/5): Loss -0.7626, Sem -0.7626, dir -0.9779, norm 0.1860 LR: 3.546e-02 Score 99.017 Data time: 0.1773, Forward time: 0.9484, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.3751 [2020-12-31 14:27:31,705][root][INFO] - ===> Epoch[2053](10265/5): Loss -0.7639, Sem -0.7639, dir -0.9777, norm 0.1841 LR: 3.543e-02 Score 98.992 Data time: 0.1798, Forward time: 0.9484, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.3559 [2020-12-31 14:27:43,323][root][INFO] - ===> Epoch[2054](10270/5): Loss -0.7680, Sem -0.7680, dir -0.9774, norm 0.1789 LR: 3.540e-02 Score 98.987 Data time: 0.1390, Forward time: 0.9484, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.2258 [2020-12-31 14:27:55,982][root][INFO] - ===> Epoch[2055](10275/5): Loss -0.7652, Sem -0.7652, dir -0.9768, norm 0.1814 LR: 3.536e-02 Score 98.980 Data time: 0.1035, Forward time: 0.9484, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.4345 [2020-12-31 14:28:07,886][root][INFO] - ===> Epoch[2056](10280/5): Loss -0.7725, Sem -0.7725, dir -0.9782, norm 0.1774 LR: 3.533e-02 Score 99.049 Data time: 0.1091, Forward time: 0.9483, Backward time: 1.1904, DDP time: 0.0051, Total iter time: 2.2965 [2020-12-31 14:28:19,377][root][INFO] - ===> Epoch[2057](10285/5): Loss -0.7688, Sem -0.7688, dir -0.9771, norm 0.1773 LR: 3.529e-02 Score 98.968 Data time: 0.1224, Forward time: 0.9483, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.2101 [2020-12-31 14:28:30,571][root][INFO] - ===> Epoch[2058](10290/5): Loss -0.7691, Sem -0.7691, dir -0.9774, norm 0.1786 LR: 3.526e-02 Score 98.998 Data time: 0.0774, Forward time: 0.9483, Backward time: 1.1903, DDP time: 0.0051, Total iter time: 2.1679 [2020-12-31 14:28:43,184][root][INFO] - ===> Epoch[2059](10295/5): Loss -0.7716, Sem -0.7716, dir -0.9779, norm 0.1763 LR: 3.523e-02 Score 98.986 Data time: 0.0734, Forward time: 0.9482, Backward time: 1.1905, DDP time: 0.0051, Total iter time: 2.4394 [2020-12-31 14:28:55,055][root][INFO] - ===> Epoch[2060](10300/5): Loss -0.7623, Sem -0.7623, dir -0.9767, norm 0.1845 LR: 3.519e-02 Score 99.008 Data time: 0.1209, Forward time: 0.9482, Backward time: 1.1904, DDP time: 0.0051, Total iter time: 2.2873 [2020-12-31 14:29:08,114][root][INFO] - ===> Epoch[2061](10305/5): Loss -0.7627, Sem -0.7627, dir -0.9770, norm 0.1853 LR: 3.516e-02 Score 99.029 Data time: 0.1259, Forward time: 0.9483, Backward time: 1.1905, DDP time: 0.0051, Total iter time: 2.5236 [2020-12-31 14:29:21,260][root][INFO] - ===> Epoch[2062](10310/5): Loss -0.7662, Sem -0.7662, dir -0.9770, norm 0.1820 LR: 3.513e-02 Score 99.033 Data time: 0.0826, Forward time: 0.9482, Backward time: 1.1906, DDP time: 0.0051, Total iter time: 2.5342 [2020-12-31 14:29:33,050][root][INFO] - ===> Epoch[2063](10315/5): Loss -0.7639, Sem -0.7639, dir -0.9772, norm 0.1841 LR: 3.509e-02 Score 99.021 Data time: 0.0743, Forward time: 0.9482, Backward time: 1.1906, DDP time: 0.0051, Total iter time: 2.2956 [2020-12-31 14:29:44,727][root][INFO] - ===> Epoch[2064](10320/5): Loss -0.7639, Sem -0.7639, dir -0.9772, norm 0.1835 LR: 3.506e-02 Score 98.996 Data time: 0.1786, Forward time: 0.9482, Backward time: 1.1906, DDP time: 0.0051, Total iter time: 2.2544 [2020-12-31 14:29:56,863][root][INFO] - ===> Epoch[2065](10325/5): Loss -0.7663, Sem -0.7663, dir -0.9774, norm 0.1809 LR: 3.503e-02 Score 98.994 Data time: 0.0862, Forward time: 0.9482, Backward time: 1.1906, DDP time: 0.0051, Total iter time: 2.3376 [2020-12-31 14:30:08,669][root][INFO] - ===> Epoch[2066](10330/5): Loss -0.7698, Sem -0.7698, dir -0.9769, norm 0.1773 LR: 3.499e-02 Score 98.991 Data time: 0.1587, Forward time: 0.9482, Backward time: 1.1906, DDP time: 0.0051, Total iter time: 2.2728 [2020-12-31 14:30:20,194][root][INFO] - ===> Epoch[2067](10335/5): Loss -0.7763, Sem -0.7763, dir -0.9773, norm 0.1712 LR: 3.496e-02 Score 99.015 Data time: 0.0886, Forward time: 0.9482, Backward time: 1.1906, DDP time: 0.0051, Total iter time: 2.2156 [2020-12-31 14:30:31,844][root][INFO] - ===> Epoch[2068](10340/5): Loss -0.7631, Sem -0.7631, dir -0.9773, norm 0.1840 LR: 3.492e-02 Score 98.986 Data time: 0.1048, Forward time: 0.9482, Backward time: 1.1906, DDP time: 0.0051, Total iter time: 2.2361 [2020-12-31 14:30:44,173][root][INFO] - ===> Epoch[2069](10345/5): Loss -0.7713, Sem -0.7713, dir -0.9771, norm 0.1765 LR: 3.489e-02 Score 99.019 Data time: 0.1347, Forward time: 0.9482, Backward time: 1.1905, DDP time: 0.0050, Total iter time: 2.3697 [2020-12-31 14:30:56,509][root][INFO] - ===> Epoch[2070](10350/5): Loss -0.7709, Sem -0.7709, dir -0.9777, norm 0.1777 LR: 3.486e-02 Score 99.030 Data time: 0.1014, Forward time: 0.9482, Backward time: 1.1905, DDP time: 0.0050, Total iter time: 2.3745 [2020-12-31 14:31:08,013][root][INFO] - ===> Epoch[2071](10355/5): Loss -0.7714, Sem -0.7714, dir -0.9772, norm 0.1774 LR: 3.482e-02 Score 99.046 Data time: 0.0860, Forward time: 0.9482, Backward time: 1.1905, DDP time: 0.0050, Total iter time: 2.2050 [2020-12-31 14:31:19,939][root][INFO] - ===> Epoch[2072](10360/5): Loss -0.7700, Sem -0.7700, dir -0.9776, norm 0.1786 LR: 3.479e-02 Score 99.021 Data time: 0.0977, Forward time: 0.9481, Backward time: 1.1906, DDP time: 0.0050, Total iter time: 2.3016 [2020-12-31 14:31:32,253][root][INFO] - ===> Epoch[2073](10365/5): Loss -0.7726, Sem -0.7726, dir -0.9776, norm 0.1754 LR: 3.476e-02 Score 99.000 Data time: 0.1026, Forward time: 0.9481, Backward time: 1.1906, DDP time: 0.0050, Total iter time: 2.3692 [2020-12-31 14:31:44,181][root][INFO] - ===> Epoch[2074](10370/5): Loss -0.7664, Sem -0.7664, dir -0.9769, norm 0.1803 LR: 3.472e-02 Score 98.994 Data time: 0.0969, Forward time: 0.9481, Backward time: 1.1907, DDP time: 0.0050, Total iter time: 2.2924 [2020-12-31 14:31:55,887][root][INFO] - ===> Epoch[2075](10375/5): Loss -0.7696, Sem -0.7696, dir -0.9776, norm 0.1780 LR: 3.469e-02 Score 99.012 Data time: 0.1627, Forward time: 0.9481, Backward time: 1.1906, DDP time: 0.0050, Total iter time: 2.2619 [2020-12-31 14:32:07,694][root][INFO] - ===> Epoch[2076](10380/5): Loss -0.7675, Sem -0.7675, dir -0.9773, norm 0.1803 LR: 3.465e-02 Score 99.022 Data time: 0.1273, Forward time: 0.9481, Backward time: 1.1906, DDP time: 0.0050, Total iter time: 2.2699 [2020-12-31 14:32:18,666][root][INFO] - ===> Epoch[2077](10385/5): Loss -0.7657, Sem -0.7657, dir -0.9770, norm 0.1803 LR: 3.462e-02 Score 98.967 Data time: 0.0796, Forward time: 0.9480, Backward time: 1.1906, DDP time: 0.0050, Total iter time: 2.1110 [2020-12-31 14:32:31,305][root][INFO] - ===> Epoch[2078](10390/5): Loss -0.7667, Sem -0.7667, dir -0.9777, norm 0.1805 LR: 3.459e-02 Score 98.992 Data time: 0.1038, Forward time: 0.9481, Backward time: 1.1906, DDP time: 0.0051, Total iter time: 2.4430 [2020-12-31 14:32:43,218][root][INFO] - ===> Epoch[2079](10395/5): Loss -0.7653, Sem -0.7653, dir -0.9769, norm 0.1811 LR: 3.455e-02 Score 98.994 Data time: 0.1224, Forward time: 0.9480, Backward time: 1.1906, DDP time: 0.0051, Total iter time: 2.2964 [2020-12-31 14:32:55,766][root][INFO] - ===> Epoch[2080](10400/5): Loss -0.7652, Sem -0.7652, dir -0.9774, norm 0.1825 LR: 3.452e-02 Score 99.014 Data time: 0.2138, Forward time: 0.9481, Backward time: 1.1905, DDP time: 0.0050, Total iter time: 2.4100 [2020-12-31 14:32:55,769][root][INFO] - ===> Start testing [2020-12-31 14:33:35,693][root][INFO] - 68/68: Data time: 0.0024, Iter time: 0.1423 Loss 0.522 (AVG: 0.519) Score 90.157 (AVG: 89.749) mIOU 67.689 mAP 83.501 mAcc 76.487 IOU: 60.068 0.000 77.832 72.846 92.906 90.251 33.508 77.319 98.412 56.173 77.669 85.579 57.394 mAP: 80.890 10.900 93.097 85.721 98.854 98.387 60.107 94.487 99.829 85.312 92.982 94.078 90.868 mAcc: 80.532 0.000 90.093 80.332 94.281 96.116 37.352 86.895 99.243 83.129 89.345 95.193 61.818 [2020-12-31 14:33:35,694][root][INFO] - Finished test. Elapsed time: 39.9236 [2020-12-31 14:33:39,474][root][INFO] - [2020-12-31 14:33:39,474][root][INFO] - ################################################################ [2020-12-31 14:33:39,474][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 14:33:39,474][root][INFO] - ################################################################ [2020-12-31 14:33:39,474][root][INFO] - clutter : nan nan nan [2020-12-31 14:33:39,474][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 14:33:39,474][root][INFO] - board : 0.790 0.870 0.899 [2020-12-31 14:33:39,474][root][INFO] - bookcase : 0.195 0.333 0.499 [2020-12-31 14:33:39,474][root][INFO] - ceiling : 0.586 0.612 0.709 [2020-12-31 14:33:39,474][root][INFO] - chair : 0.783 0.885 0.894 [2020-12-31 14:33:39,474][root][INFO] - column : 0.165 0.246 0.408 [2020-12-31 14:33:39,474][root][INFO] - door : 0.532 0.671 0.885 [2020-12-31 14:33:39,474][root][INFO] - floor : 0.946 0.962 0.977 [2020-12-31 14:33:39,474][root][INFO] - sofa : 0.548 0.801 0.801 [2020-12-31 14:33:39,474][root][INFO] - table : 0.191 0.366 0.650 [2020-12-31 14:33:39,475][root][INFO] - wall : 0.487 0.692 0.785 [2020-12-31 14:33:39,475][root][INFO] - window : 0.464 0.696 0.773 [2020-12-31 14:33:39,475][root][INFO] - ---------------------------------------------------------------- [2020-12-31 14:33:39,475][root][INFO] - average : 0.474 0.595 0.690 [2020-12-31 14:33:39,475][root][INFO] - [2020-12-31 14:33:39,634][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 14:33:40,227][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter10400.pth [2020-12-31 14:33:52,037][root][INFO] - ===> Epoch[2081](10405/5): Loss -0.7660, Sem -0.7660, dir -0.9775, norm 0.1817 LR: 3.449e-02 Score 98.999 Data time: 0.0952, Forward time: 0.9481, Backward time: 1.1905, DDP time: 0.0050, Total iter time: 2.2710 [2020-12-31 14:34:03,741][root][INFO] - ===> Epoch[2082](10410/5): Loss -0.7687, Sem -0.7687, dir -0.9776, norm 0.1791 LR: 3.445e-02 Score 99.006 Data time: 0.0934, Forward time: 0.9481, Backward time: 1.1905, DDP time: 0.0050, Total iter time: 2.2543 [2020-12-31 14:34:17,087][root][INFO] - ===> Epoch[2083](10415/5): Loss -0.7602, Sem -0.7602, dir -0.9771, norm 0.1870 LR: 3.442e-02 Score 99.007 Data time: 0.1102, Forward time: 0.9482, Backward time: 1.1906, DDP time: 0.0050, Total iter time: 2.5794 [2020-12-31 14:34:28,053][root][INFO] - ===> Epoch[2084](10420/5): Loss -0.7737, Sem -0.7737, dir -0.9772, norm 0.1728 LR: 3.438e-02 Score 98.978 Data time: 0.1202, Forward time: 0.9482, Backward time: 1.1905, DDP time: 0.0050, Total iter time: 2.1095 [2020-12-31 14:34:39,669][root][INFO] - ===> Epoch[2085](10425/5): Loss -0.7641, Sem -0.7641, dir -0.9776, norm 0.1832 LR: 3.435e-02 Score 98.983 Data time: 0.0999, Forward time: 0.9482, Backward time: 1.1904, DDP time: 0.0050, Total iter time: 2.2362 [2020-12-31 14:34:51,500][root][INFO] - ===> Epoch[2086](10430/5): Loss -0.7667, Sem -0.7667, dir -0.9772, norm 0.1805 LR: 3.432e-02 Score 99.010 Data time: 0.1253, Forward time: 0.9483, Backward time: 1.1904, DDP time: 0.0050, Total iter time: 2.2811 [2020-12-31 14:35:03,029][root][INFO] - ===> Epoch[2087](10435/5): Loss -0.7729, Sem -0.7729, dir -0.9777, norm 0.1763 LR: 3.428e-02 Score 99.049 Data time: 0.1262, Forward time: 0.9483, Backward time: 1.1903, DDP time: 0.0050, Total iter time: 2.2122 [2020-12-31 14:35:14,166][root][INFO] - ===> Epoch[2088](10440/5): Loss -0.7679, Sem -0.7679, dir -0.9772, norm 0.1786 LR: 3.425e-02 Score 98.976 Data time: 0.1066, Forward time: 0.9483, Backward time: 1.1902, DDP time: 0.0050, Total iter time: 2.1279 [2020-12-31 14:35:26,690][root][INFO] - ===> Epoch[2089](10445/5): Loss -0.7693, Sem -0.7693, dir -0.9776, norm 0.1787 LR: 3.422e-02 Score 99.010 Data time: 0.0922, Forward time: 0.9483, Backward time: 1.1903, DDP time: 0.0050, Total iter time: 2.4172 [2020-12-31 14:35:38,161][root][INFO] - ===> Epoch[2090](10450/5): Loss -0.7691, Sem -0.7691, dir -0.9781, norm 0.1786 LR: 3.418e-02 Score 98.994 Data time: 0.1562, Forward time: 0.9483, Backward time: 1.1902, DDP time: 0.0050, Total iter time: 2.1992 [2020-12-31 14:35:50,280][root][INFO] - ===> Epoch[2091](10455/5): Loss -0.7642, Sem -0.7642, dir -0.9771, norm 0.1820 LR: 3.415e-02 Score 98.959 Data time: 0.0833, Forward time: 0.9483, Backward time: 1.1902, DDP time: 0.0050, Total iter time: 2.3421 [2020-12-31 14:36:01,628][root][INFO] - ===> Epoch[2092](10460/5): Loss -0.7783, Sem -0.7783, dir -0.9781, norm 0.1709 LR: 3.411e-02 Score 99.027 Data time: 0.1037, Forward time: 0.9483, Backward time: 1.1901, DDP time: 0.0050, Total iter time: 2.1925 [2020-12-31 14:36:13,342][root][INFO] - ===> Epoch[2093](10465/5): Loss -0.7708, Sem -0.7708, dir -0.9771, norm 0.1752 LR: 3.408e-02 Score 98.967 Data time: 0.1042, Forward time: 0.9483, Backward time: 1.1901, DDP time: 0.0050, Total iter time: 2.2623 [2020-12-31 14:36:26,632][root][INFO] - ===> Epoch[2094](10470/5): Loss -0.7772, Sem -0.7772, dir -0.9780, norm 0.1708 LR: 3.405e-02 Score 99.002 Data time: 0.1223, Forward time: 0.9484, Backward time: 1.1902, DDP time: 0.0050, Total iter time: 2.5696 [2020-12-31 14:36:38,720][root][INFO] - ===> Epoch[2095](10475/5): Loss -0.7718, Sem -0.7718, dir -0.9770, norm 0.1756 LR: 3.401e-02 Score 99.005 Data time: 0.1056, Forward time: 0.9484, Backward time: 1.1902, DDP time: 0.0050, Total iter time: 2.3185 [2020-12-31 14:36:51,068][root][INFO] - ===> Epoch[2096](10480/5): Loss -0.7769, Sem -0.7769, dir -0.9779, norm 0.1710 LR: 3.398e-02 Score 98.994 Data time: 0.1309, Forward time: 0.9484, Backward time: 1.1902, DDP time: 0.0050, Total iter time: 2.3854 [2020-12-31 14:37:02,687][root][INFO] - ===> Epoch[2097](10485/5): Loss -0.7713, Sem -0.7713, dir -0.9780, norm 0.1770 LR: 3.394e-02 Score 99.001 Data time: 0.1148, Forward time: 0.9484, Backward time: 1.1902, DDP time: 0.0050, Total iter time: 2.2361 [2020-12-31 14:37:14,926][root][INFO] - ===> Epoch[2098](10490/5): Loss -0.7728, Sem -0.7728, dir -0.9774, norm 0.1738 LR: 3.391e-02 Score 98.972 Data time: 0.0813, Forward time: 0.9483, Backward time: 1.1903, DDP time: 0.0050, Total iter time: 2.3666 [2020-12-31 14:37:26,466][root][INFO] - ===> Epoch[2099](10495/5): Loss -0.7680, Sem -0.7680, dir -0.9785, norm 0.1814 LR: 3.388e-02 Score 99.033 Data time: 0.0993, Forward time: 0.9484, Backward time: 1.1902, DDP time: 0.0050, Total iter time: 2.2101 [2020-12-31 14:37:38,866][root][INFO] - ===> Epoch[2100](10500/5): Loss -0.7739, Sem -0.7739, dir -0.9778, norm 0.1749 LR: 3.384e-02 Score 99.021 Data time: 0.0834, Forward time: 0.9483, Backward time: 1.1903, DDP time: 0.0050, Total iter time: 2.3966 [2020-12-31 14:37:51,445][root][INFO] - ===> Epoch[2101](10505/5): Loss -0.7704, Sem -0.7704, dir -0.9783, norm 0.1790 LR: 3.381e-02 Score 99.031 Data time: 0.1142, Forward time: 0.9484, Backward time: 1.1903, DDP time: 0.0050, Total iter time: 2.4213 [2020-12-31 14:38:03,937][root][INFO] - ===> Epoch[2102](10510/5): Loss -0.7698, Sem -0.7698, dir -0.9779, norm 0.1785 LR: 3.378e-02 Score 99.009 Data time: 0.1051, Forward time: 0.9484, Backward time: 1.1903, DDP time: 0.0050, Total iter time: 2.4085 [2020-12-31 14:38:16,282][root][INFO] - ===> Epoch[2103](10515/5): Loss -0.7707, Sem -0.7707, dir -0.9778, norm 0.1779 LR: 3.374e-02 Score 99.017 Data time: 0.1026, Forward time: 0.9484, Backward time: 1.1903, DDP time: 0.0050, Total iter time: 2.3783 [2020-12-31 14:38:27,057][root][INFO] - ===> Epoch[2104](10520/5): Loss -0.7658, Sem -0.7658, dir -0.9772, norm 0.1815 LR: 3.371e-02 Score 99.006 Data time: 0.0863, Forward time: 0.9484, Backward time: 1.1902, DDP time: 0.0050, Total iter time: 2.0659 [2020-12-31 14:38:39,451][root][INFO] - ===> Epoch[2105](10525/5): Loss -0.7750, Sem -0.7750, dir -0.9772, norm 0.1723 LR: 3.367e-02 Score 98.982 Data time: 0.0985, Forward time: 0.9484, Backward time: 1.1902, DDP time: 0.0050, Total iter time: 2.3839 [2020-12-31 14:38:52,264][root][INFO] - ===> Epoch[2106](10530/5): Loss -0.7641, Sem -0.7641, dir -0.9768, norm 0.1817 LR: 3.364e-02 Score 98.979 Data time: 0.0912, Forward time: 0.9484, Backward time: 1.1903, DDP time: 0.0050, Total iter time: 2.4707 [2020-12-31 14:39:04,093][root][INFO] - ===> Epoch[2107](10535/5): Loss -0.7718, Sem -0.7718, dir -0.9772, norm 0.1757 LR: 3.361e-02 Score 99.025 Data time: 0.1057, Forward time: 0.9484, Backward time: 1.1903, DDP time: 0.0050, Total iter time: 2.2688 [2020-12-31 14:39:16,244][root][INFO] - ===> Epoch[2108](10540/5): Loss -0.7745, Sem -0.7745, dir -0.9777, norm 0.1728 LR: 3.357e-02 Score 98.983 Data time: 0.0987, Forward time: 0.9484, Backward time: 1.1903, DDP time: 0.0050, Total iter time: 2.3402 [2020-12-31 14:39:28,069][root][INFO] - ===> Epoch[2109](10545/5): Loss -0.7623, Sem -0.7623, dir -0.9778, norm 0.1849 LR: 3.354e-02 Score 98.989 Data time: 0.1239, Forward time: 0.9484, Backward time: 1.1903, DDP time: 0.0050, Total iter time: 2.2789 [2020-12-31 14:39:40,186][root][INFO] - ===> Epoch[2110](10550/5): Loss -0.7653, Sem -0.7653, dir -0.9777, norm 0.1830 LR: 3.350e-02 Score 99.016 Data time: 0.1150, Forward time: 0.9484, Backward time: 1.1903, DDP time: 0.0050, Total iter time: 2.3249 [2020-12-31 14:39:51,858][root][INFO] - ===> Epoch[2111](10555/5): Loss -0.7679, Sem -0.7679, dir -0.9772, norm 0.1792 LR: 3.347e-02 Score 98.988 Data time: 0.1128, Forward time: 0.9484, Backward time: 1.1903, DDP time: 0.0050, Total iter time: 2.2410 [2020-12-31 14:40:04,765][root][INFO] - ===> Epoch[2112](10560/5): Loss -0.7678, Sem -0.7678, dir -0.9776, norm 0.1801 LR: 3.344e-02 Score 99.032 Data time: 0.0978, Forward time: 0.9484, Backward time: 1.1904, DDP time: 0.0050, Total iter time: 2.4887 [2020-12-31 14:40:16,295][root][INFO] - ===> Epoch[2113](10565/5): Loss -0.7691, Sem -0.7691, dir -0.9779, norm 0.1778 LR: 3.340e-02 Score 98.965 Data time: 0.0997, Forward time: 0.9484, Backward time: 1.1904, DDP time: 0.0050, Total iter time: 2.2165 [2020-12-31 14:40:28,367][root][INFO] - ===> Epoch[2114](10570/5): Loss -0.7757, Sem -0.7757, dir -0.9782, norm 0.1736 LR: 3.337e-02 Score 99.030 Data time: 0.1041, Forward time: 0.9484, Backward time: 1.1904, DDP time: 0.0050, Total iter time: 2.3107 [2020-12-31 14:40:40,652][root][INFO] - ===> Epoch[2115](10575/5): Loss -0.7668, Sem -0.7668, dir -0.9769, norm 0.1786 LR: 3.334e-02 Score 98.983 Data time: 0.0807, Forward time: 0.9484, Backward time: 1.1904, DDP time: 0.0050, Total iter time: 2.3737 [2020-12-31 14:40:52,475][root][INFO] - ===> Epoch[2116](10580/5): Loss -0.7687, Sem -0.7687, dir -0.9774, norm 0.1791 LR: 3.330e-02 Score 99.019 Data time: 0.1050, Forward time: 0.9484, Backward time: 1.1904, DDP time: 0.0050, Total iter time: 2.2674 [2020-12-31 14:41:03,636][root][INFO] - ===> Epoch[2117](10585/5): Loss -0.7798, Sem -0.7798, dir -0.9776, norm 0.1681 LR: 3.327e-02 Score 98.996 Data time: 0.1039, Forward time: 0.9484, Backward time: 1.1903, DDP time: 0.0050, Total iter time: 2.1434 [2020-12-31 14:41:15,427][root][INFO] - ===> Epoch[2118](10590/5): Loss -0.7768, Sem -0.7768, dir -0.9778, norm 0.1719 LR: 3.323e-02 Score 99.020 Data time: 0.1744, Forward time: 0.9483, Backward time: 1.1903, DDP time: 0.0050, Total iter time: 2.2619 [2020-12-31 14:41:27,115][root][INFO] - ===> Epoch[2119](10595/5): Loss -0.7760, Sem -0.7760, dir -0.9781, norm 0.1745 LR: 3.320e-02 Score 99.081 Data time: 0.0907, Forward time: 0.9484, Backward time: 1.1902, DDP time: 0.0050, Total iter time: 2.2473 [2020-12-31 14:41:38,710][root][INFO] - ===> Epoch[2120](10600/5): Loss -0.7789, Sem -0.7789, dir -0.9775, norm 0.1696 LR: 3.317e-02 Score 99.020 Data time: 0.1543, Forward time: 0.9484, Backward time: 1.1901, DDP time: 0.0050, Total iter time: 2.2240 [2020-12-31 14:41:38,713][root][INFO] - ===> Start testing [2020-12-31 14:42:18,069][root][INFO] - 68/68: Data time: 0.5296, Iter time: 0.3044 Loss 0.234 (AVG: 0.547) Score 95.332 (AVG: 89.725) mIOU 67.275 mAP 82.590 mAcc 74.554 IOU: 60.529 0.000 82.098 72.972 93.755 90.969 21.860 74.827 98.473 59.358 80.925 84.776 54.034 mAP: 81.511 9.995 93.391 86.035 98.933 98.327 52.164 94.467 99.820 82.045 92.959 94.167 89.854 mAcc: 81.578 0.000 89.380 80.193 94.996 95.855 23.723 86.920 99.215 74.092 90.588 95.751 56.917 [2020-12-31 14:42:18,070][root][INFO] - Finished test. Elapsed time: 39.3555 [2020-12-31 14:42:21,246][root][INFO] - [2020-12-31 14:42:21,246][root][INFO] - ################################################################ [2020-12-31 14:42:21,246][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 14:42:21,246][root][INFO] - ################################################################ [2020-12-31 14:42:21,246][root][INFO] - clutter : nan nan nan [2020-12-31 14:42:21,246][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 14:42:21,246][root][INFO] - board : 0.790 0.873 0.902 [2020-12-31 14:42:21,246][root][INFO] - bookcase : 0.190 0.321 0.487 [2020-12-31 14:42:21,247][root][INFO] - ceiling : 0.557 0.653 0.736 [2020-12-31 14:42:21,247][root][INFO] - chair : 0.775 0.870 0.889 [2020-12-31 14:42:21,247][root][INFO] - column : 0.135 0.192 0.293 [2020-12-31 14:42:21,247][root][INFO] - door : 0.654 0.848 0.933 [2020-12-31 14:42:21,247][root][INFO] - floor : 0.982 0.988 0.988 [2020-12-31 14:42:21,247][root][INFO] - sofa : 0.402 0.545 0.809 [2020-12-31 14:42:21,247][root][INFO] - table : 0.191 0.366 0.641 [2020-12-31 14:42:21,247][root][INFO] - wall : 0.498 0.714 0.814 [2020-12-31 14:42:21,247][root][INFO] - window : 0.439 0.702 0.745 [2020-12-31 14:42:21,247][root][INFO] - ---------------------------------------------------------------- [2020-12-31 14:42:21,247][root][INFO] - average : 0.468 0.590 0.686 [2020-12-31 14:42:21,247][root][INFO] - [2020-12-31 14:42:21,439][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 14:42:21,983][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter10600.pth [2020-12-31 14:42:33,035][root][INFO] - ===> Epoch[2121](10605/5): Loss -0.7704, Sem -0.7704, dir -0.9773, norm 0.1772 LR: 3.313e-02 Score 98.996 Data time: 0.1003, Forward time: 0.9484, Backward time: 1.1901, DDP time: 0.0050, Total iter time: 2.1173 [2020-12-31 14:42:44,452][root][INFO] - ===> Epoch[2122](10610/5): Loss -0.7718, Sem -0.7718, dir -0.9773, norm 0.1749 LR: 3.310e-02 Score 98.968 Data time: 0.1695, Forward time: 0.9484, Backward time: 1.1900, DDP time: 0.0050, Total iter time: 2.1842 [2020-12-31 14:42:57,131][root][INFO] - ===> Epoch[2123](10615/5): Loss -0.7652, Sem -0.7652, dir -0.9773, norm 0.1817 LR: 3.306e-02 Score 98.990 Data time: 0.1968, Forward time: 0.9484, Backward time: 1.1900, DDP time: 0.0050, Total iter time: 2.4346 [2020-12-31 14:43:08,364][root][INFO] - ===> Epoch[2124](10620/5): Loss -0.7725, Sem -0.7725, dir -0.9773, norm 0.1754 LR: 3.303e-02 Score 99.026 Data time: 0.1130, Forward time: 0.9484, Backward time: 1.1899, DDP time: 0.0050, Total iter time: 2.1632 [2020-12-31 14:43:20,613][root][INFO] - ===> Epoch[2125](10625/5): Loss -0.7687, Sem -0.7687, dir -0.9774, norm 0.1791 LR: 3.300e-02 Score 99.012 Data time: 0.1279, Forward time: 0.9484, Backward time: 1.1899, DDP time: 0.0050, Total iter time: 2.3631 [2020-12-31 14:43:32,517][root][INFO] - ===> Epoch[2126](10630/5): Loss -0.7665, Sem -0.7665, dir -0.9778, norm 0.1828 LR: 3.296e-02 Score 99.042 Data time: 0.1344, Forward time: 0.9484, Backward time: 1.1899, DDP time: 0.0050, Total iter time: 2.2947 [2020-12-31 14:43:44,773][root][INFO] - ===> Epoch[2127](10635/5): Loss -0.7754, Sem -0.7754, dir -0.9785, norm 0.1743 LR: 3.293e-02 Score 99.024 Data time: 0.1923, Forward time: 0.9485, Backward time: 1.1898, DDP time: 0.0050, Total iter time: 2.3506 [2020-12-31 14:43:56,783][root][INFO] - ===> Epoch[2128](10640/5): Loss -0.7707, Sem -0.7707, dir -0.9778, norm 0.1785 LR: 3.289e-02 Score 99.049 Data time: 0.0883, Forward time: 0.9485, Backward time: 1.1898, DDP time: 0.0050, Total iter time: 2.3097 [2020-12-31 14:44:09,780][root][INFO] - ===> Epoch[2129](10645/5): Loss -0.7743, Sem -0.7743, dir -0.9787, norm 0.1765 LR: 3.286e-02 Score 99.061 Data time: 0.1259, Forward time: 0.9486, Backward time: 1.1898, DDP time: 0.0050, Total iter time: 2.5102 [2020-12-31 14:44:21,198][root][INFO] - ===> Epoch[2130](10650/5): Loss -0.7694, Sem -0.7694, dir -0.9778, norm 0.1791 LR: 3.283e-02 Score 99.023 Data time: 0.1040, Forward time: 0.9486, Backward time: 1.1898, DDP time: 0.0050, Total iter time: 2.1912 [2020-12-31 14:44:34,160][root][INFO] - ===> Epoch[2131](10655/5): Loss -0.7653, Sem -0.7653, dir -0.9772, norm 0.1812 LR: 3.279e-02 Score 98.966 Data time: 0.0893, Forward time: 0.9486, Backward time: 1.1899, DDP time: 0.0050, Total iter time: 2.5199 [2020-12-31 14:44:44,985][root][INFO] - ===> Epoch[2132](10660/5): Loss -0.7670, Sem -0.7670, dir -0.9771, norm 0.1787 LR: 3.276e-02 Score 98.971 Data time: 0.1175, Forward time: 0.9486, Backward time: 1.1898, DDP time: 0.0050, Total iter time: 2.0740 [2020-12-31 14:44:56,241][root][INFO] - ===> Epoch[2133](10665/5): Loss -0.7676, Sem -0.7676, dir -0.9772, norm 0.1790 LR: 3.272e-02 Score 98.966 Data time: 0.0780, Forward time: 0.9485, Backward time: 1.1898, DDP time: 0.0051, Total iter time: 2.1564 [2020-12-31 14:45:07,734][root][INFO] - ===> Epoch[2134](10670/5): Loss -0.7677, Sem -0.7677, dir -0.9772, norm 0.1796 LR: 3.269e-02 Score 99.006 Data time: 0.1182, Forward time: 0.9486, Backward time: 1.1897, DDP time: 0.0051, Total iter time: 2.1978 [2020-12-31 14:45:20,298][root][INFO] - ===> Epoch[2135](10675/5): Loss -0.7659, Sem -0.7659, dir -0.9770, norm 0.1814 LR: 3.266e-02 Score 99.015 Data time: 0.1097, Forward time: 0.9486, Backward time: 1.1897, DDP time: 0.0050, Total iter time: 2.4289 [2020-12-31 14:45:31,214][root][INFO] - ===> Epoch[2136](10680/5): Loss -0.7700, Sem -0.7700, dir -0.9780, norm 0.1795 LR: 3.262e-02 Score 99.038 Data time: 0.1048, Forward time: 0.9486, Backward time: 1.1896, DDP time: 0.0050, Total iter time: 2.0920 [2020-12-31 14:45:42,925][root][INFO] - ===> Epoch[2137](10685/5): Loss -0.7644, Sem -0.7644, dir -0.9766, norm 0.1808 LR: 3.259e-02 Score 98.951 Data time: 0.0847, Forward time: 0.9486, Backward time: 1.1896, DDP time: 0.0050, Total iter time: 2.2581 [2020-12-31 14:45:54,551][root][INFO] - ===> Epoch[2138](10690/5): Loss -0.7743, Sem -0.7743, dir -0.9779, norm 0.1744 LR: 3.255e-02 Score 99.022 Data time: 0.0967, Forward time: 0.9485, Backward time: 1.1896, DDP time: 0.0050, Total iter time: 2.2371 [2020-12-31 14:46:07,267][root][INFO] - ===> Epoch[2139](10695/5): Loss -0.7775, Sem -0.7775, dir -0.9784, norm 0.1716 LR: 3.252e-02 Score 99.017 Data time: 0.0993, Forward time: 0.9486, Backward time: 1.1897, DDP time: 0.0051, Total iter time: 2.4585 [2020-12-31 14:46:18,345][root][INFO] - ===> Epoch[2140](10700/5): Loss -0.7793, Sem -0.7793, dir -0.9783, norm 0.1693 LR: 3.249e-02 Score 99.009 Data time: 0.1422, Forward time: 0.9486, Backward time: 1.1896, DDP time: 0.0051, Total iter time: 2.1234 [2020-12-31 14:46:31,180][root][INFO] - ===> Epoch[2141](10705/5): Loss -0.7690, Sem -0.7690, dir -0.9780, norm 0.1793 LR: 3.245e-02 Score 99.001 Data time: 0.1862, Forward time: 0.9486, Backward time: 1.1896, DDP time: 0.0051, Total iter time: 2.4708 [2020-12-31 14:46:43,736][root][INFO] - ===> Epoch[2142](10710/5): Loss -0.7757, Sem -0.7757, dir -0.9775, norm 0.1708 LR: 3.242e-02 Score 98.973 Data time: 0.1303, Forward time: 0.9486, Backward time: 1.1896, DDP time: 0.0050, Total iter time: 2.4208 [2020-12-31 14:46:55,974][root][INFO] - ===> Epoch[2143](10715/5): Loss -0.7760, Sem -0.7760, dir -0.9783, norm 0.1721 LR: 3.238e-02 Score 98.984 Data time: 0.1492, Forward time: 0.9487, Backward time: 1.1896, DDP time: 0.0050, Total iter time: 2.3493 [2020-12-31 14:47:08,241][root][INFO] - ===> Epoch[2144](10720/5): Loss -0.7714, Sem -0.7714, dir -0.9780, norm 0.1763 LR: 3.235e-02 Score 98.990 Data time: 0.1232, Forward time: 0.9487, Backward time: 1.1896, DDP time: 0.0050, Total iter time: 2.3613 [2020-12-31 14:47:20,778][root][INFO] - ===> Epoch[2145](10725/5): Loss -0.7718, Sem -0.7718, dir -0.9781, norm 0.1769 LR: 3.232e-02 Score 99.025 Data time: 0.1065, Forward time: 0.9488, Backward time: 1.1896, DDP time: 0.0051, Total iter time: 2.4211 [2020-12-31 14:47:33,194][root][INFO] - ===> Epoch[2146](10730/5): Loss -0.7765, Sem -0.7765, dir -0.9784, norm 0.1735 LR: 3.228e-02 Score 99.047 Data time: 0.0924, Forward time: 0.9487, Backward time: 1.1896, DDP time: 0.0051, Total iter time: 2.4010 [2020-12-31 14:47:46,280][root][INFO] - ===> Epoch[2147](10735/5): Loss -0.7788, Sem -0.7788, dir -0.9780, norm 0.1706 LR: 3.225e-02 Score 99.034 Data time: 0.0686, Forward time: 0.9487, Backward time: 1.1898, DDP time: 0.0051, Total iter time: 2.5342 [2020-12-31 14:47:58,598][root][INFO] - ===> Epoch[2148](10740/5): Loss -0.7774, Sem -0.7774, dir -0.9786, norm 0.1724 LR: 3.221e-02 Score 99.027 Data time: 0.0712, Forward time: 0.9486, Backward time: 1.1899, DDP time: 0.0051, Total iter time: 2.3758 [2020-12-31 14:48:10,066][root][INFO] - ===> Epoch[2149](10745/5): Loss -0.7739, Sem -0.7739, dir -0.9781, norm 0.1750 LR: 3.218e-02 Score 99.020 Data time: 0.1042, Forward time: 0.9486, Backward time: 1.1899, DDP time: 0.0051, Total iter time: 2.1987 [2020-12-31 14:48:21,125][root][INFO] - ===> Epoch[2150](10750/5): Loss -0.7632, Sem -0.7632, dir -0.9774, norm 0.1830 LR: 3.215e-02 Score 98.961 Data time: 0.1113, Forward time: 0.9485, Backward time: 1.1898, DDP time: 0.0051, Total iter time: 2.1258 [2020-12-31 14:48:33,390][root][INFO] - ===> Epoch[2151](10755/5): Loss -0.7708, Sem -0.7708, dir -0.9785, norm 0.1787 LR: 3.211e-02 Score 99.028 Data time: 0.1596, Forward time: 0.9486, Backward time: 1.1898, DDP time: 0.0051, Total iter time: 2.3528 [2020-12-31 14:48:46,038][root][INFO] - ===> Epoch[2152](10760/5): Loss -0.7804, Sem -0.7804, dir -0.9789, norm 0.1709 LR: 3.208e-02 Score 99.073 Data time: 0.1145, Forward time: 0.9486, Backward time: 1.1899, DDP time: 0.0051, Total iter time: 2.4386 [2020-12-31 14:48:58,595][root][INFO] - ===> Epoch[2153](10765/5): Loss -0.7801, Sem -0.7801, dir -0.9788, norm 0.1702 LR: 3.204e-02 Score 99.050 Data time: 0.1092, Forward time: 0.9486, Backward time: 1.1899, DDP time: 0.0051, Total iter time: 2.4160 [2020-12-31 14:49:10,389][root][INFO] - ===> Epoch[2154](10770/5): Loss -0.7712, Sem -0.7712, dir -0.9772, norm 0.1766 LR: 3.201e-02 Score 99.005 Data time: 0.0895, Forward time: 0.9485, Backward time: 1.1899, DDP time: 0.0051, Total iter time: 2.2766 [2020-12-31 14:49:21,509][root][INFO] - ===> Epoch[2155](10775/5): Loss -0.7746, Sem -0.7746, dir -0.9785, norm 0.1743 LR: 3.198e-02 Score 99.014 Data time: 0.1012, Forward time: 0.9485, Backward time: 1.1899, DDP time: 0.0051, Total iter time: 2.1307 [2020-12-31 14:49:33,548][root][INFO] - ===> Epoch[2156](10780/5): Loss -0.7757, Sem -0.7757, dir -0.9782, norm 0.1743 LR: 3.194e-02 Score 99.057 Data time: 0.1185, Forward time: 0.9485, Backward time: 1.1898, DDP time: 0.0051, Total iter time: 2.3037 [2020-12-31 14:49:45,984][root][INFO] - ===> Epoch[2157](10785/5): Loss -0.7709, Sem -0.7709, dir -0.9782, norm 0.1784 LR: 3.191e-02 Score 99.030 Data time: 0.1286, Forward time: 0.9486, Backward time: 1.1898, DDP time: 0.0051, Total iter time: 2.3925 [2020-12-31 14:49:56,971][root][INFO] - ===> Epoch[2158](10790/5): Loss -0.7731, Sem -0.7731, dir -0.9774, norm 0.1748 LR: 3.187e-02 Score 99.003 Data time: 0.1549, Forward time: 0.9486, Backward time: 1.1897, DDP time: 0.0051, Total iter time: 2.1087 [2020-12-31 14:50:08,596][root][INFO] - ===> Epoch[2159](10795/5): Loss -0.7758, Sem -0.7758, dir -0.9778, norm 0.1727 LR: 3.184e-02 Score 99.018 Data time: 0.1370, Forward time: 0.9485, Backward time: 1.1897, DDP time: 0.0051, Total iter time: 2.2362 [2020-12-31 14:50:21,246][root][INFO] - ===> Epoch[2160](10800/5): Loss -0.7778, Sem -0.7778, dir -0.9786, norm 0.1721 LR: 3.181e-02 Score 99.047 Data time: 0.0922, Forward time: 0.9485, Backward time: 1.1898, DDP time: 0.0051, Total iter time: 2.4477 [2020-12-31 14:50:21,249][root][INFO] - ===> Start testing [2020-12-31 14:51:00,393][root][INFO] - 68/68: Data time: 0.0018, Iter time: 0.0721 Loss 0.667 (AVG: 0.519) Score 77.659 (AVG: 89.886) mIOU 68.451 mAP 83.235 mAcc 75.995 IOU: 60.459 0.000 80.010 73.800 93.543 90.143 32.894 75.138 98.335 64.303 80.565 85.275 55.402 mAP: 81.402 6.930 92.957 86.393 98.905 98.065 58.799 94.891 99.831 85.557 93.089 94.313 90.918 mAcc: 81.476 0.000 89.116 81.336 94.725 95.574 35.964 88.194 99.012 77.555 90.831 94.965 59.182 [2020-12-31 14:51:00,394][root][INFO] - Finished test. Elapsed time: 39.1432 [2020-12-31 14:51:03,490][root][INFO] - [2020-12-31 14:51:03,490][root][INFO] - ################################################################ [2020-12-31 14:51:03,490][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 14:51:03,490][root][INFO] - ################################################################ [2020-12-31 14:51:03,490][root][INFO] - clutter : nan nan nan [2020-12-31 14:51:03,490][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 14:51:03,490][root][INFO] - board : 0.786 0.844 0.897 [2020-12-31 14:51:03,490][root][INFO] - bookcase : 0.198 0.330 0.503 [2020-12-31 14:51:03,490][root][INFO] - ceiling : 0.595 0.663 0.749 [2020-12-31 14:51:03,490][root][INFO] - chair : 0.783 0.869 0.888 [2020-12-31 14:51:03,490][root][INFO] - column : 0.140 0.216 0.350 [2020-12-31 14:51:03,490][root][INFO] - door : 0.599 0.788 0.884 [2020-12-31 14:51:03,490][root][INFO] - floor : 0.921 0.949 0.960 [2020-12-31 14:51:03,491][root][INFO] - sofa : 0.447 0.636 0.809 [2020-12-31 14:51:03,491][root][INFO] - table : 0.193 0.372 0.622 [2020-12-31 14:51:03,491][root][INFO] - wall : 0.495 0.731 0.815 [2020-12-31 14:51:03,491][root][INFO] - window : 0.455 0.689 0.754 [2020-12-31 14:51:03,491][root][INFO] - ---------------------------------------------------------------- [2020-12-31 14:51:03,491][root][INFO] - average : 0.468 0.591 0.686 [2020-12-31 14:51:03,491][root][INFO] - [2020-12-31 14:51:03,640][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 14:51:04,249][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter10800.pth [2020-12-31 14:51:16,212][root][INFO] - ===> Epoch[2161](10805/5): Loss -0.7735, Sem -0.7735, dir -0.9777, norm 0.1749 LR: 3.177e-02 Score 99.023 Data time: 0.0831, Forward time: 0.9485, Backward time: 1.1899, DDP time: 0.0051, Total iter time: 2.2873 [2020-12-31 14:51:28,751][root][INFO] - ===> Epoch[2162](10810/5): Loss -0.7759, Sem -0.7759, dir -0.9788, norm 0.1739 LR: 3.174e-02 Score 99.033 Data time: 0.1368, Forward time: 0.9485, Backward time: 1.1899, DDP time: 0.0051, Total iter time: 2.4118 [2020-12-31 14:51:40,698][root][INFO] - ===> Epoch[2163](10815/5): Loss -0.7802, Sem -0.7802, dir -0.9780, norm 0.1679 LR: 3.170e-02 Score 99.001 Data time: 0.1561, Forward time: 0.9485, Backward time: 1.1898, DDP time: 0.0051, Total iter time: 2.2969 [2020-12-31 14:51:51,842][root][INFO] - ===> Epoch[2164](10820/5): Loss -0.7767, Sem -0.7767, dir -0.9777, norm 0.1714 LR: 3.167e-02 Score 99.006 Data time: 0.0968, Forward time: 0.9485, Backward time: 1.1897, DDP time: 0.0051, Total iter time: 2.1447 [2020-12-31 14:52:03,389][root][INFO] - ===> Epoch[2165](10825/5): Loss -0.7744, Sem -0.7744, dir -0.9778, norm 0.1738 LR: 3.164e-02 Score 99.012 Data time: 0.1261, Forward time: 0.9485, Backward time: 1.1897, DDP time: 0.0051, Total iter time: 2.2183 [2020-12-31 14:52:15,611][root][INFO] - ===> Epoch[2166](10830/5): Loss -0.7727, Sem -0.7727, dir -0.9783, norm 0.1774 LR: 3.160e-02 Score 99.058 Data time: 0.1890, Forward time: 0.9485, Backward time: 1.1897, DDP time: 0.0051, Total iter time: 2.3482 [2020-12-31 14:52:27,368][root][INFO] - ===> Epoch[2167](10835/5): Loss -0.7741, Sem -0.7741, dir -0.9778, norm 0.1751 LR: 3.157e-02 Score 99.028 Data time: 0.1168, Forward time: 0.9486, Backward time: 1.1896, DDP time: 0.0051, Total iter time: 2.2624 [2020-12-31 14:52:39,402][root][INFO] - ===> Epoch[2168](10840/5): Loss -0.7673, Sem -0.7673, dir -0.9773, norm 0.1804 LR: 3.153e-02 Score 99.008 Data time: 0.0944, Forward time: 0.9486, Backward time: 1.1896, DDP time: 0.0051, Total iter time: 2.3117 [2020-12-31 14:52:51,186][root][INFO] - ===> Epoch[2169](10845/5): Loss -0.7727, Sem -0.7727, dir -0.9787, norm 0.1768 LR: 3.150e-02 Score 99.015 Data time: 0.0985, Forward time: 0.9486, Backward time: 1.1896, DDP time: 0.0051, Total iter time: 2.2615 [2020-12-31 14:53:02,999][root][INFO] - ===> Epoch[2170](10850/5): Loss -0.7733, Sem -0.7733, dir -0.9785, norm 0.1767 LR: 3.147e-02 Score 99.046 Data time: 0.0969, Forward time: 0.9486, Backward time: 1.1896, DDP time: 0.0051, Total iter time: 2.2657 [2020-12-31 14:53:15,191][root][INFO] - ===> Epoch[2171](10855/5): Loss -0.7672, Sem -0.7672, dir -0.9785, norm 0.1831 LR: 3.143e-02 Score 99.054 Data time: 0.0994, Forward time: 0.9487, Backward time: 1.1895, DDP time: 0.0051, Total iter time: 2.3531 [2020-12-31 14:53:28,291][root][INFO] - ===> Epoch[2172](10860/5): Loss -0.7619, Sem -0.7619, dir -0.9781, norm 0.1855 LR: 3.140e-02 Score 98.986 Data time: 0.1202, Forward time: 0.9487, Backward time: 1.1896, DDP time: 0.0050, Total iter time: 2.5284 [2020-12-31 14:53:40,507][root][INFO] - ===> Epoch[2173](10865/5): Loss -0.7619, Sem -0.7619, dir -0.9771, norm 0.1841 LR: 3.136e-02 Score 98.973 Data time: 0.0945, Forward time: 0.9487, Backward time: 1.1896, DDP time: 0.0051, Total iter time: 2.3561 [2020-12-31 14:53:52,419][root][INFO] - ===> Epoch[2174](10870/5): Loss -0.7681, Sem -0.7681, dir -0.9780, norm 0.1787 LR: 3.133e-02 Score 98.969 Data time: 0.0968, Forward time: 0.9487, Backward time: 1.1896, DDP time: 0.0050, Total iter time: 2.2908 [2020-12-31 14:54:05,543][root][INFO] - ===> Epoch[2175](10875/5): Loss -0.7622, Sem -0.7622, dir -0.9774, norm 0.1848 LR: 3.129e-02 Score 98.980 Data time: 0.1113, Forward time: 0.9487, Backward time: 1.1897, DDP time: 0.0050, Total iter time: 2.5313 [2020-12-31 14:54:16,610][root][INFO] - ===> Epoch[2176](10880/5): Loss -0.7743, Sem -0.7743, dir -0.9781, norm 0.1746 LR: 3.126e-02 Score 99.025 Data time: 0.0844, Forward time: 0.9487, Backward time: 1.1897, DDP time: 0.0051, Total iter time: 2.1230 [2020-12-31 14:54:28,809][root][INFO] - ===> Epoch[2177](10885/5): Loss -0.7683, Sem -0.7683, dir -0.9777, norm 0.1800 LR: 3.123e-02 Score 99.009 Data time: 0.0966, Forward time: 0.9487, Backward time: 1.1896, DDP time: 0.0050, Total iter time: 2.3399 [2020-12-31 14:54:40,429][root][INFO] - ===> Epoch[2178](10890/5): Loss -0.7713, Sem -0.7713, dir -0.9778, norm 0.1769 LR: 3.119e-02 Score 99.010 Data time: 0.1040, Forward time: 0.9487, Backward time: 1.1896, DDP time: 0.0050, Total iter time: 2.2341 [2020-12-31 14:54:51,525][root][INFO] - ===> Epoch[2179](10895/5): Loss -0.7674, Sem -0.7674, dir -0.9771, norm 0.1792 LR: 3.116e-02 Score 98.982 Data time: 0.1267, Forward time: 0.9487, Backward time: 1.1895, DDP time: 0.0050, Total iter time: 2.1290 [2020-12-31 14:55:03,906][root][INFO] - ===> Epoch[2180](10900/5): Loss -0.7776, Sem -0.7776, dir -0.9778, norm 0.1706 LR: 3.112e-02 Score 99.003 Data time: 0.0895, Forward time: 0.9487, Backward time: 1.1896, DDP time: 0.0050, Total iter time: 2.3851 [2020-12-31 14:55:15,727][root][INFO] - ===> Epoch[2181](10905/5): Loss -0.7696, Sem -0.7696, dir -0.9773, norm 0.1767 LR: 3.109e-02 Score 98.965 Data time: 0.1048, Forward time: 0.9487, Backward time: 1.1895, DDP time: 0.0050, Total iter time: 2.2603 [2020-12-31 14:55:26,683][root][INFO] - ===> Epoch[2182](10910/5): Loss -0.7812, Sem -0.7812, dir -0.9784, norm 0.1702 LR: 3.106e-02 Score 99.091 Data time: 0.1211, Forward time: 0.9487, Backward time: 1.1894, DDP time: 0.0050, Total iter time: 2.0893 [2020-12-31 14:55:37,796][root][INFO] - ===> Epoch[2183](10915/5): Loss -0.7801, Sem -0.7801, dir -0.9786, norm 0.1692 LR: 3.102e-02 Score 99.023 Data time: 0.1200, Forward time: 0.9487, Backward time: 1.1893, DDP time: 0.0050, Total iter time: 2.1308 [2020-12-31 14:55:49,824][root][INFO] - ===> Epoch[2184](10920/5): Loss -0.7770, Sem -0.7770, dir -0.9788, norm 0.1739 LR: 3.099e-02 Score 99.064 Data time: 0.0847, Forward time: 0.9487, Backward time: 1.1894, DDP time: 0.0050, Total iter time: 2.3148 [2020-12-31 14:56:01,794][root][INFO] - ===> Epoch[2185](10925/5): Loss -0.7768, Sem -0.7768, dir -0.9781, norm 0.1726 LR: 3.095e-02 Score 99.028 Data time: 0.0806, Forward time: 0.9486, Backward time: 1.1894, DDP time: 0.0050, Total iter time: 2.3070 [2020-12-31 14:56:14,111][root][INFO] - ===> Epoch[2186](10930/5): Loss -0.7791, Sem -0.7791, dir -0.9783, norm 0.1714 LR: 3.092e-02 Score 99.067 Data time: 0.1121, Forward time: 0.9487, Backward time: 1.1894, DDP time: 0.0050, Total iter time: 2.3706 [2020-12-31 14:56:26,028][root][INFO] - ===> Epoch[2187](10935/5): Loss -0.7742, Sem -0.7742, dir -0.9787, norm 0.1763 LR: 3.088e-02 Score 99.048 Data time: 0.1132, Forward time: 0.9487, Backward time: 1.1894, DDP time: 0.0050, Total iter time: 2.2963 [2020-12-31 14:56:36,992][root][INFO] - ===> Epoch[2188](10940/5): Loss -0.7731, Sem -0.7731, dir -0.9774, norm 0.1741 LR: 3.085e-02 Score 98.984 Data time: 0.0975, Forward time: 0.9486, Backward time: 1.1893, DDP time: 0.0050, Total iter time: 2.1064 [2020-12-31 14:56:48,282][root][INFO] - ===> Epoch[2189](10945/5): Loss -0.7723, Sem -0.7723, dir -0.9782, norm 0.1773 LR: 3.082e-02 Score 99.037 Data time: 0.0850, Forward time: 0.9487, Backward time: 1.1893, DDP time: 0.0050, Total iter time: 2.1652 [2020-12-31 14:57:00,088][root][INFO] - ===> Epoch[2190](10950/5): Loss -0.7746, Sem -0.7746, dir -0.9777, norm 0.1750 LR: 3.078e-02 Score 99.059 Data time: 0.1076, Forward time: 0.9487, Backward time: 1.1893, DDP time: 0.0050, Total iter time: 2.2755 [2020-12-31 14:57:11,598][root][INFO] - ===> Epoch[2191](10955/5): Loss -0.7736, Sem -0.7736, dir -0.9779, norm 0.1744 LR: 3.075e-02 Score 99.002 Data time: 0.1010, Forward time: 0.9487, Backward time: 1.1892, DDP time: 0.0050, Total iter time: 2.2120 [2020-12-31 14:57:23,493][root][INFO] - ===> Epoch[2192](10960/5): Loss -0.7724, Sem -0.7724, dir -0.9778, norm 0.1757 LR: 3.071e-02 Score 99.017 Data time: 0.1328, Forward time: 0.9487, Backward time: 1.1892, DDP time: 0.0050, Total iter time: 2.2851 [2020-12-31 14:57:35,793][root][INFO] - ===> Epoch[2193](10965/5): Loss -0.7738, Sem -0.7738, dir -0.9787, norm 0.1769 LR: 3.068e-02 Score 99.066 Data time: 0.1700, Forward time: 0.9488, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.3627 [2020-12-31 14:57:47,803][root][INFO] - ===> Epoch[2194](10970/5): Loss -0.7790, Sem -0.7790, dir -0.9786, norm 0.1718 LR: 3.065e-02 Score 99.069 Data time: 0.0938, Forward time: 0.9487, Backward time: 1.1892, DDP time: 0.0050, Total iter time: 2.3169 [2020-12-31 14:57:59,460][root][INFO] - ===> Epoch[2195](10975/5): Loss -0.7759, Sem -0.7759, dir -0.9782, norm 0.1730 LR: 3.061e-02 Score 99.006 Data time: 0.1000, Forward time: 0.9487, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.2394 [2020-12-31 14:58:11,274][root][INFO] - ===> Epoch[2196](10980/5): Loss -0.7712, Sem -0.7712, dir -0.9775, norm 0.1761 LR: 3.058e-02 Score 98.987 Data time: 0.0922, Forward time: 0.9487, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.2733 [2020-12-31 14:58:22,939][root][INFO] - ===> Epoch[2197](10985/5): Loss -0.7766, Sem -0.7766, dir -0.9785, norm 0.1732 LR: 3.054e-02 Score 99.036 Data time: 0.1259, Forward time: 0.9487, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.2379 [2020-12-31 14:58:36,225][root][INFO] - ===> Epoch[2198](10990/5): Loss -0.7772, Sem -0.7772, dir -0.9785, norm 0.1729 LR: 3.051e-02 Score 99.051 Data time: 0.1238, Forward time: 0.9487, Backward time: 1.1892, DDP time: 0.0050, Total iter time: 2.5649 [2020-12-31 14:58:48,206][root][INFO] - ===> Epoch[2199](10995/5): Loss -0.7819, Sem -0.7819, dir -0.9789, norm 0.1686 LR: 3.047e-02 Score 99.038 Data time: 0.1046, Forward time: 0.9487, Backward time: 1.1892, DDP time: 0.0050, Total iter time: 2.2995 [2020-12-31 14:59:00,484][root][INFO] - ===> Epoch[2200](11000/5): Loss -0.7698, Sem -0.7698, dir -0.9774, norm 0.1771 LR: 3.044e-02 Score 98.987 Data time: 0.1686, Forward time: 0.9487, Backward time: 1.1892, DDP time: 0.0050, Total iter time: 2.3561 [2020-12-31 14:59:00,487][root][INFO] - ===> Start testing [2020-12-31 14:59:40,003][root][INFO] - 68/68: Data time: 0.0020, Iter time: 0.0979 Loss 0.100 (AVG: 0.529) Score 97.019 (AVG: 89.840) mIOU 68.353 mAP 83.365 mAcc 76.306 IOU: 59.757 0.000 79.033 71.485 94.292 91.529 31.893 78.234 98.601 62.351 80.336 85.147 55.927 mAP: 81.081 7.964 92.877 85.555 98.969 98.227 59.593 95.900 99.804 86.029 92.935 94.265 90.540 mAcc: 80.009 0.000 90.112 79.207 95.690 95.887 35.072 91.023 99.476 80.387 89.940 94.892 60.288 [2020-12-31 14:59:40,004][root][INFO] - Finished test. Elapsed time: 39.5146 [2020-12-31 14:59:43,817][root][INFO] - [2020-12-31 14:59:43,817][root][INFO] - ################################################################ [2020-12-31 14:59:43,817][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 14:59:43,817][root][INFO] - ################################################################ [2020-12-31 14:59:43,817][root][INFO] - clutter : nan nan nan [2020-12-31 14:59:43,817][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 14:59:43,817][root][INFO] - board : 0.782 0.863 0.889 [2020-12-31 14:59:43,817][root][INFO] - bookcase : 0.196 0.337 0.504 [2020-12-31 14:59:43,817][root][INFO] - ceiling : 0.677 0.788 0.812 [2020-12-31 14:59:43,817][root][INFO] - chair : 0.798 0.886 0.904 [2020-12-31 14:59:43,817][root][INFO] - column : 0.159 0.238 0.343 [2020-12-31 14:59:43,817][root][INFO] - door : 0.615 0.778 0.921 [2020-12-31 14:59:43,817][root][INFO] - floor : 0.836 0.889 0.923 [2020-12-31 14:59:43,817][root][INFO] - sofa : 0.482 0.727 0.809 [2020-12-31 14:59:43,817][root][INFO] - table : 0.196 0.379 0.645 [2020-12-31 14:59:43,817][root][INFO] - wall : 0.475 0.701 0.799 [2020-12-31 14:59:43,817][root][INFO] - window : 0.509 0.721 0.764 [2020-12-31 14:59:43,817][root][INFO] - ---------------------------------------------------------------- [2020-12-31 14:59:43,818][root][INFO] - average : 0.477 0.609 0.693 [2020-12-31 14:59:43,818][root][INFO] - [2020-12-31 14:59:43,968][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 14:59:44,526][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter11000.pth [2020-12-31 14:59:54,747][root][INFO] - ===> Epoch[2201](11005/5): Loss -0.7808, Sem -0.7808, dir -0.9791, norm 0.1695 LR: 3.041e-02 Score 99.035 Data time: 0.1106, Forward time: 0.9487, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 1.9372 [2020-12-31 15:00:06,293][root][INFO] - ===> Epoch[2202](11010/5): Loss -0.7740, Sem -0.7740, dir -0.9776, norm 0.1752 LR: 3.037e-02 Score 99.047 Data time: 0.0906, Forward time: 0.9487, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.2164 [2020-12-31 15:00:18,682][root][INFO] - ===> Epoch[2203](11015/5): Loss -0.7724, Sem -0.7724, dir -0.9780, norm 0.1769 LR: 3.034e-02 Score 99.044 Data time: 0.1266, Forward time: 0.9488, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.3864 [2020-12-31 15:00:30,267][root][INFO] - ===> Epoch[2204](11020/5): Loss -0.7735, Sem -0.7735, dir -0.9780, norm 0.1753 LR: 3.030e-02 Score 99.020 Data time: 0.1077, Forward time: 0.9488, Backward time: 1.1889, DDP time: 0.0050, Total iter time: 2.2180 [2020-12-31 15:00:42,433][root][INFO] - ===> Epoch[2205](11025/5): Loss -0.7700, Sem -0.7700, dir -0.9775, norm 0.1777 LR: 3.027e-02 Score 98.998 Data time: 0.1144, Forward time: 0.9488, Backward time: 1.1889, DDP time: 0.0050, Total iter time: 2.3385 [2020-12-31 15:00:54,182][root][INFO] - ===> Epoch[2206](11030/5): Loss -0.7689, Sem -0.7689, dir -0.9785, norm 0.1803 LR: 3.023e-02 Score 99.038 Data time: 0.1322, Forward time: 0.9488, Backward time: 1.1889, DDP time: 0.0050, Total iter time: 2.2550 [2020-12-31 15:01:05,420][root][INFO] - ===> Epoch[2207](11035/5): Loss -0.7837, Sem -0.7837, dir -0.9788, norm 0.1677 LR: 3.020e-02 Score 99.075 Data time: 0.1131, Forward time: 0.9488, Backward time: 1.1888, DDP time: 0.0050, Total iter time: 2.1568 [2020-12-31 15:01:17,525][root][INFO] - ===> Epoch[2208](11040/5): Loss -0.7777, Sem -0.7777, dir -0.9780, norm 0.1711 LR: 3.017e-02 Score 99.024 Data time: 0.0855, Forward time: 0.9487, Backward time: 1.1889, DDP time: 0.0050, Total iter time: 2.3287 [2020-12-31 15:01:29,337][root][INFO] - ===> Epoch[2209](11045/5): Loss -0.7791, Sem -0.7791, dir -0.9779, norm 0.1685 LR: 3.013e-02 Score 98.999 Data time: 0.1374, Forward time: 0.9487, Backward time: 1.1889, DDP time: 0.0050, Total iter time: 2.2687 [2020-12-31 15:01:42,329][root][INFO] - ===> Epoch[2210](11050/5): Loss -0.7793, Sem -0.7793, dir -0.9787, norm 0.1705 LR: 3.010e-02 Score 99.040 Data time: 0.1267, Forward time: 0.9487, Backward time: 1.1889, DDP time: 0.0050, Total iter time: 2.5005 [2020-12-31 15:01:55,044][root][INFO] - ===> Epoch[2211](11055/5): Loss -0.7676, Sem -0.7676, dir -0.9780, norm 0.1808 LR: 3.006e-02 Score 99.013 Data time: 0.1380, Forward time: 0.9488, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.4403 [2020-12-31 15:02:08,124][root][INFO] - ===> Epoch[2212](11060/5): Loss -0.7793, Sem -0.7793, dir -0.9786, norm 0.1720 LR: 3.003e-02 Score 99.085 Data time: 0.1091, Forward time: 0.9488, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.5249 [2020-12-31 15:02:19,875][root][INFO] - ===> Epoch[2213](11065/5): Loss -0.7765, Sem -0.7765, dir -0.9782, norm 0.1722 LR: 2.999e-02 Score 99.015 Data time: 0.0943, Forward time: 0.9488, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.2618 [2020-12-31 15:02:32,588][root][INFO] - ===> Epoch[2214](11070/5): Loss -0.7763, Sem -0.7763, dir -0.9782, norm 0.1731 LR: 2.996e-02 Score 99.038 Data time: 0.0744, Forward time: 0.9487, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.4591 [2020-12-31 15:02:44,899][root][INFO] - ===> Epoch[2215](11075/5): Loss -0.7812, Sem -0.7812, dir -0.9789, norm 0.1695 LR: 2.993e-02 Score 99.053 Data time: 0.0905, Forward time: 0.9487, Backward time: 1.1892, DDP time: 0.0050, Total iter time: 2.3702 [2020-12-31 15:02:56,470][root][INFO] - ===> Epoch[2216](11080/5): Loss -0.7807, Sem -0.7807, dir -0.9783, norm 0.1688 LR: 2.989e-02 Score 99.037 Data time: 0.1361, Forward time: 0.9487, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.2177 [2020-12-31 15:03:08,089][root][INFO] - ===> Epoch[2217](11085/5): Loss -0.7674, Sem -0.7674, dir -0.9777, norm 0.1786 LR: 2.986e-02 Score 98.996 Data time: 0.1109, Forward time: 0.9488, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.2273 [2020-12-31 15:03:20,842][root][INFO] - ===> Epoch[2218](11090/5): Loss -0.7711, Sem -0.7711, dir -0.9776, norm 0.1758 LR: 2.982e-02 Score 98.975 Data time: 0.1109, Forward time: 0.9487, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.4606 [2020-12-31 15:03:31,991][root][INFO] - ===> Epoch[2219](11095/5): Loss -0.7735, Sem -0.7735, dir -0.9777, norm 0.1728 LR: 2.979e-02 Score 98.973 Data time: 0.0942, Forward time: 0.9487, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.1244 [2020-12-31 15:03:44,596][root][INFO] - ===> Epoch[2220](11100/5): Loss -0.7788, Sem -0.7788, dir -0.9788, norm 0.1703 LR: 2.975e-02 Score 99.003 Data time: 0.1004, Forward time: 0.9487, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.4286 [2020-12-31 15:03:56,207][root][INFO] - ===> Epoch[2221](11105/5): Loss -0.7754, Sem -0.7754, dir -0.9780, norm 0.1708 LR: 2.972e-02 Score 98.958 Data time: 0.1587, Forward time: 0.9488, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.2256 [2020-12-31 15:04:08,296][root][INFO] - ===> Epoch[2222](11110/5): Loss -0.7740, Sem -0.7740, dir -0.9781, norm 0.1736 LR: 2.969e-02 Score 98.983 Data time: 0.0865, Forward time: 0.9487, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.3307 [2020-12-31 15:04:20,242][root][INFO] - ===> Epoch[2223](11115/5): Loss -0.7773, Sem -0.7773, dir -0.9778, norm 0.1703 LR: 2.965e-02 Score 98.992 Data time: 0.1238, Forward time: 0.9488, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.3023 [2020-12-31 15:04:31,421][root][INFO] - ===> Epoch[2224](11120/5): Loss -0.7702, Sem -0.7702, dir -0.9779, norm 0.1774 LR: 2.962e-02 Score 98.998 Data time: 0.1244, Forward time: 0.9488, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.1379 [2020-12-31 15:04:42,190][root][INFO] - ===> Epoch[2225](11125/5): Loss -0.7678, Sem -0.7678, dir -0.9775, norm 0.1773 LR: 2.958e-02 Score 98.946 Data time: 0.1230, Forward time: 0.9487, Backward time: 1.1889, DDP time: 0.0050, Total iter time: 2.0724 [2020-12-31 15:04:54,611][root][INFO] - ===> Epoch[2226](11130/5): Loss -0.7804, Sem -0.7804, dir -0.9785, norm 0.1703 LR: 2.955e-02 Score 99.078 Data time: 0.1517, Forward time: 0.9488, Backward time: 1.1889, DDP time: 0.0050, Total iter time: 2.3779 [2020-12-31 15:05:06,843][root][INFO] - ===> Epoch[2227](11135/5): Loss -0.7787, Sem -0.7787, dir -0.9783, norm 0.1709 LR: 2.951e-02 Score 99.041 Data time: 0.0873, Forward time: 0.9487, Backward time: 1.1889, DDP time: 0.0050, Total iter time: 2.3589 [2020-12-31 15:05:18,645][root][INFO] - ===> Epoch[2228](11140/5): Loss -0.7799, Sem -0.7799, dir -0.9782, norm 0.1693 LR: 2.948e-02 Score 99.040 Data time: 0.0748, Forward time: 0.9487, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.2609 [2020-12-31 15:05:31,102][root][INFO] - ===> Epoch[2229](11145/5): Loss -0.7769, Sem -0.7769, dir -0.9784, norm 0.1723 LR: 2.945e-02 Score 99.034 Data time: 0.1020, Forward time: 0.9487, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.3878 [2020-12-31 15:05:43,380][root][INFO] - ===> Epoch[2230](11150/5): Loss -0.7799, Sem -0.7799, dir -0.9786, norm 0.1702 LR: 2.941e-02 Score 99.050 Data time: 0.1501, Forward time: 0.9487, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.3588 [2020-12-31 15:05:55,037][root][INFO] - ===> Epoch[2231](11155/5): Loss -0.7715, Sem -0.7715, dir -0.9779, norm 0.1765 LR: 2.938e-02 Score 99.011 Data time: 0.1051, Forward time: 0.9487, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.2429 [2020-12-31 15:06:07,271][root][INFO] - ===> Epoch[2232](11160/5): Loss -0.7732, Sem -0.7732, dir -0.9786, norm 0.1763 LR: 2.934e-02 Score 99.026 Data time: 0.1107, Forward time: 0.9487, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.3632 [2020-12-31 15:06:18,929][root][INFO] - ===> Epoch[2233](11165/5): Loss -0.7790, Sem -0.7790, dir -0.9779, norm 0.1694 LR: 2.931e-02 Score 99.014 Data time: 0.0748, Forward time: 0.9486, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.2384 [2020-12-31 15:06:32,010][root][INFO] - ===> Epoch[2234](11170/5): Loss -0.7782, Sem -0.7782, dir -0.9786, norm 0.1723 LR: 2.927e-02 Score 99.052 Data time: 0.1780, Forward time: 0.9487, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.5193 [2020-12-31 15:06:43,714][root][INFO] - ===> Epoch[2235](11175/5): Loss -0.7787, Sem -0.7787, dir -0.9782, norm 0.1701 LR: 2.924e-02 Score 99.021 Data time: 0.0987, Forward time: 0.9487, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.2474 [2020-12-31 15:06:55,768][root][INFO] - ===> Epoch[2236](11180/5): Loss -0.7868, Sem -0.7868, dir -0.9794, norm 0.1652 LR: 2.920e-02 Score 99.080 Data time: 0.1090, Forward time: 0.9487, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.3169 [2020-12-31 15:07:07,202][root][INFO] - ===> Epoch[2237](11185/5): Loss -0.7853, Sem -0.7853, dir -0.9789, norm 0.1664 LR: 2.917e-02 Score 99.088 Data time: 0.1565, Forward time: 0.9487, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.1937 [2020-12-31 15:07:19,299][root][INFO] - ===> Epoch[2238](11190/5): Loss -0.7784, Sem -0.7784, dir -0.9783, norm 0.1709 LR: 2.914e-02 Score 99.036 Data time: 0.0917, Forward time: 0.9487, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.3299 [2020-12-31 15:07:31,431][root][INFO] - ===> Epoch[2239](11195/5): Loss -0.7837, Sem -0.7837, dir -0.9790, norm 0.1680 LR: 2.910e-02 Score 99.087 Data time: 0.1101, Forward time: 0.9486, Backward time: 1.1891, DDP time: 0.0050, Total iter time: 2.3320 [2020-12-31 15:07:42,441][root][INFO] - ===> Epoch[2240](11200/5): Loss -0.7827, Sem -0.7827, dir -0.9785, norm 0.1670 LR: 2.907e-02 Score 99.026 Data time: 0.1219, Forward time: 0.9486, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.1178 [2020-12-31 15:07:42,443][root][INFO] - ===> Start testing [2020-12-31 15:08:21,140][root][INFO] - 68/68: Data time: 0.0020, Iter time: 0.1046 Loss 0.419 (AVG: 0.533) Score 90.598 (AVG: 89.919) mIOU 68.237 mAP 83.226 mAcc 76.090 IOU: 59.859 0.000 81.937 72.534 93.738 90.958 34.985 76.125 98.494 56.309 80.310 85.593 56.242 mAP: 80.256 12.106 93.316 85.174 98.923 98.286 61.287 94.849 99.811 79.791 93.310 93.973 90.854 mAcc: 80.818 0.000 88.079 78.753 94.873 95.878 38.062 88.697 99.353 77.792 90.511 95.780 60.568 [2020-12-31 15:08:21,141][root][INFO] - Finished test. Elapsed time: 38.6958 [2020-12-31 15:08:24,739][root][INFO] - [2020-12-31 15:08:24,739][root][INFO] - ################################################################ [2020-12-31 15:08:24,739][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 15:08:24,739][root][INFO] - ################################################################ [2020-12-31 15:08:24,739][root][INFO] - clutter : nan nan nan [2020-12-31 15:08:24,739][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 15:08:24,739][root][INFO] - board : 0.766 0.875 0.905 [2020-12-31 15:08:24,739][root][INFO] - bookcase : 0.197 0.343 0.484 [2020-12-31 15:08:24,739][root][INFO] - ceiling : 0.698 0.827 0.827 [2020-12-31 15:08:24,739][root][INFO] - chair : 0.782 0.871 0.886 [2020-12-31 15:08:24,739][root][INFO] - column : 0.172 0.268 0.406 [2020-12-31 15:08:24,739][root][INFO] - door : 0.572 0.789 0.909 [2020-12-31 15:08:24,740][root][INFO] - floor : 0.978 0.997 0.999 [2020-12-31 15:08:24,740][root][INFO] - sofa : 0.498 0.727 0.727 [2020-12-31 15:08:24,740][root][INFO] - table : 0.194 0.373 0.632 [2020-12-31 15:08:24,740][root][INFO] - wall : 0.456 0.685 0.782 [2020-12-31 15:08:24,740][root][INFO] - window : 0.487 0.745 0.764 [2020-12-31 15:08:24,740][root][INFO] - ---------------------------------------------------------------- [2020-12-31 15:08:24,740][root][INFO] - average : 0.484 0.625 0.693 [2020-12-31 15:08:24,740][root][INFO] - [2020-12-31 15:08:24,906][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 15:08:25,458][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_mAP.pth [2020-12-31 15:08:25,459][root][INFO] - Current best mAP@0.5: 0.625 at iter 11200 [2020-12-31 15:08:26,009][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter11200.pth [2020-12-31 15:08:37,176][root][INFO] - ===> Epoch[2241](11205/5): Loss -0.7806, Sem -0.7806, dir -0.9784, norm 0.1699 LR: 2.903e-02 Score 99.065 Data time: 0.0796, Forward time: 0.9486, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.1565 [2020-12-31 15:08:48,365][root][INFO] - ===> Epoch[2242](11210/5): Loss -0.7810, Sem -0.7810, dir -0.9787, norm 0.1695 LR: 2.900e-02 Score 99.055 Data time: 0.1608, Forward time: 0.9486, Backward time: 1.1889, DDP time: 0.0050, Total iter time: 2.1390 [2020-12-31 15:09:01,709][root][INFO] - ===> Epoch[2243](11215/5): Loss -0.7753, Sem -0.7753, dir -0.9779, norm 0.1739 LR: 2.896e-02 Score 99.036 Data time: 0.1115, Forward time: 0.9486, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.5849 [2020-12-31 15:09:13,706][root][INFO] - ===> Epoch[2244](11220/5): Loss -0.7837, Sem -0.7837, dir -0.9792, norm 0.1679 LR: 2.893e-02 Score 99.066 Data time: 0.1680, Forward time: 0.9486, Backward time: 1.1889, DDP time: 0.0050, Total iter time: 2.3057 [2020-12-31 15:09:25,145][root][INFO] - ===> Epoch[2245](11225/5): Loss -0.7791, Sem -0.7791, dir -0.9784, norm 0.1711 LR: 2.889e-02 Score 99.053 Data time: 0.1269, Forward time: 0.9486, Backward time: 1.1889, DDP time: 0.0050, Total iter time: 2.2000 [2020-12-31 15:09:37,185][root][INFO] - ===> Epoch[2246](11230/5): Loss -0.7820, Sem -0.7820, dir -0.9787, norm 0.1684 LR: 2.886e-02 Score 99.045 Data time: 0.1145, Forward time: 0.9486, Backward time: 1.1889, DDP time: 0.0050, Total iter time: 2.3107 [2020-12-31 15:09:49,485][root][INFO] - ===> Epoch[2247](11235/5): Loss -0.7735, Sem -0.7735, dir -0.9782, norm 0.1756 LR: 2.883e-02 Score 99.028 Data time: 0.0929, Forward time: 0.9486, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.3679 [2020-12-31 15:10:01,805][root][INFO] - ===> Epoch[2248](11240/5): Loss -0.7776, Sem -0.7776, dir -0.9789, norm 0.1743 LR: 2.879e-02 Score 99.095 Data time: 0.1002, Forward time: 0.9486, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.3752 [2020-12-31 15:10:13,586][root][INFO] - ===> Epoch[2249](11245/5): Loss -0.7788, Sem -0.7788, dir -0.9784, norm 0.1713 LR: 2.876e-02 Score 99.044 Data time: 0.0984, Forward time: 0.9486, Backward time: 1.1890, DDP time: 0.0050, Total iter time: 2.2629 [2020-12-31 15:10:25,538][root][INFO] - ===> Epoch[2250](11250/5): Loss -0.7792, Sem -0.7792, dir -0.9787, norm 0.1696 LR: 2.872e-02 Score 99.009 Data time: 0.1215, Forward time: 0.9486, Backward time: 1.1889, DDP time: 0.0050, Total iter time: 2.2931 [2020-12-31 15:10:37,327][root][INFO] - ===> Epoch[2251](11255/5): Loss -0.7712, Sem -0.7712, dir -0.9776, norm 0.1769 LR: 2.869e-02 Score 99.015 Data time: 0.1376, Forward time: 0.9486, Backward time: 1.1889, DDP time: 0.0050, Total iter time: 2.2533 [2020-12-31 15:10:48,886][root][INFO] - ===> Epoch[2252](11260/5): Loss -0.7816, Sem -0.7816, dir -0.9782, norm 0.1679 LR: 2.865e-02 Score 99.038 Data time: 0.0860, Forward time: 0.9486, Backward time: 1.1889, DDP time: 0.0050, Total iter time: 2.2221 [2020-12-31 15:11:00,372][root][INFO] - ===> Epoch[2253](11265/5): Loss -0.7798, Sem -0.7798, dir -0.9790, norm 0.1715 LR: 2.862e-02 Score 99.077 Data time: 0.1601, Forward time: 0.9487, Backward time: 1.1888, DDP time: 0.0050, Total iter time: 2.2057 [2020-12-31 15:11:12,122][root][INFO] - ===> Epoch[2254](11270/5): Loss -0.7700, Sem -0.7700, dir -0.9777, norm 0.1759 LR: 2.858e-02 Score 98.947 Data time: 0.1424, Forward time: 0.9486, Backward time: 1.1888, DDP time: 0.0050, Total iter time: 2.2759 [2020-12-31 15:11:23,930][root][INFO] - ===> Epoch[2255](11275/5): Loss -0.7837, Sem -0.7837, dir -0.9787, norm 0.1668 LR: 2.855e-02 Score 99.048 Data time: 0.1320, Forward time: 0.9486, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.2624 [2020-12-31 15:11:35,226][root][INFO] - ===> Epoch[2256](11280/5): Loss -0.7760, Sem -0.7760, dir -0.9787, norm 0.1734 LR: 2.852e-02 Score 99.026 Data time: 0.1007, Forward time: 0.9486, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.1670 [2020-12-31 15:11:47,397][root][INFO] - ===> Epoch[2257](11285/5): Loss -0.7807, Sem -0.7807, dir -0.9789, norm 0.1704 LR: 2.848e-02 Score 99.070 Data time: 0.1456, Forward time: 0.9486, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.3416 [2020-12-31 15:11:58,458][root][INFO] - ===> Epoch[2258](11290/5): Loss -0.7873, Sem -0.7873, dir -0.9789, norm 0.1639 LR: 2.845e-02 Score 99.071 Data time: 0.1204, Forward time: 0.9486, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.1277 [2020-12-31 15:12:11,153][root][INFO] - ===> Epoch[2259](11295/5): Loss -0.7793, Sem -0.7793, dir -0.9784, norm 0.1708 LR: 2.841e-02 Score 99.055 Data time: 0.1542, Forward time: 0.9486, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.4495 [2020-12-31 15:12:22,576][root][INFO] - ===> Epoch[2260](11300/5): Loss -0.7880, Sem -0.7880, dir -0.9792, norm 0.1641 LR: 2.838e-02 Score 99.099 Data time: 0.1347, Forward time: 0.9486, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.1879 [2020-12-31 15:12:34,758][root][INFO] - ===> Epoch[2261](11305/5): Loss -0.7737, Sem -0.7737, dir -0.9785, norm 0.1756 LR: 2.834e-02 Score 99.025 Data time: 0.1039, Forward time: 0.9486, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.3594 [2020-12-31 15:12:47,087][root][INFO] - ===> Epoch[2262](11310/5): Loss -0.7810, Sem -0.7810, dir -0.9783, norm 0.1687 LR: 2.831e-02 Score 99.051 Data time: 0.0882, Forward time: 0.9486, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.3884 [2020-12-31 15:12:58,375][root][INFO] - ===> Epoch[2263](11315/5): Loss -0.7826, Sem -0.7826, dir -0.9790, norm 0.1683 LR: 2.827e-02 Score 99.054 Data time: 0.1035, Forward time: 0.9486, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.1594 [2020-12-31 15:13:10,611][root][INFO] - ===> Epoch[2264](11320/5): Loss -0.7817, Sem -0.7817, dir -0.9789, norm 0.1677 LR: 2.824e-02 Score 99.022 Data time: 0.1733, Forward time: 0.9487, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.3417 [2020-12-31 15:13:23,253][root][INFO] - ===> Epoch[2265](11325/5): Loss -0.7792, Sem -0.7792, dir -0.9781, norm 0.1698 LR: 2.821e-02 Score 99.030 Data time: 0.0874, Forward time: 0.9487, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.4364 [2020-12-31 15:13:34,535][root][INFO] - ===> Epoch[2266](11330/5): Loss -0.7734, Sem -0.7734, dir -0.9786, norm 0.1758 LR: 2.817e-02 Score 99.011 Data time: 0.1206, Forward time: 0.9486, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.1752 [2020-12-31 15:13:45,944][root][INFO] - ===> Epoch[2267](11335/5): Loss -0.7774, Sem -0.7774, dir -0.9783, norm 0.1715 LR: 2.814e-02 Score 99.038 Data time: 0.0862, Forward time: 0.9486, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2089 [2020-12-31 15:13:56,897][root][INFO] - ===> Epoch[2268](11340/5): Loss -0.7812, Sem -0.7812, dir -0.9784, norm 0.1679 LR: 2.810e-02 Score 99.009 Data time: 0.0705, Forward time: 0.9485, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.0931 [2020-12-31 15:14:09,088][root][INFO] - ===> Epoch[2269](11345/5): Loss -0.7829, Sem -0.7829, dir -0.9788, norm 0.1678 LR: 2.807e-02 Score 99.067 Data time: 0.0839, Forward time: 0.9486, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3553 [2020-12-31 15:14:20,777][root][INFO] - ===> Epoch[2270](11350/5): Loss -0.7802, Sem -0.7802, dir -0.9786, norm 0.1701 LR: 2.803e-02 Score 99.048 Data time: 0.1133, Forward time: 0.9486, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2375 [2020-12-31 15:14:32,974][root][INFO] - ===> Epoch[2271](11355/5): Loss -0.7802, Sem -0.7802, dir -0.9785, norm 0.1699 LR: 2.800e-02 Score 99.056 Data time: 0.1192, Forward time: 0.9486, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.3536 [2020-12-31 15:14:44,629][root][INFO] - ===> Epoch[2272](11360/5): Loss -0.7854, Sem -0.7854, dir -0.9785, norm 0.1652 LR: 2.796e-02 Score 99.062 Data time: 0.1389, Forward time: 0.9486, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.2509 [2020-12-31 15:14:55,845][root][INFO] - ===> Epoch[2273](11365/5): Loss -0.7851, Sem -0.7851, dir -0.9791, norm 0.1659 LR: 2.793e-02 Score 99.064 Data time: 0.0915, Forward time: 0.9485, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.1604 [2020-12-31 15:15:08,261][root][INFO] - ===> Epoch[2274](11370/5): Loss -0.7819, Sem -0.7819, dir -0.9791, norm 0.1692 LR: 2.789e-02 Score 99.067 Data time: 0.1062, Forward time: 0.9486, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.3909 [2020-12-31 15:15:19,797][root][INFO] - ===> Epoch[2275](11375/5): Loss -0.7819, Sem -0.7819, dir -0.9788, norm 0.1687 LR: 2.786e-02 Score 99.051 Data time: 0.0974, Forward time: 0.9486, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.1964 [2020-12-31 15:15:32,375][root][INFO] - ===> Epoch[2276](11380/5): Loss -0.7850, Sem -0.7850, dir -0.9789, norm 0.1655 LR: 2.783e-02 Score 99.045 Data time: 0.0794, Forward time: 0.9486, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.4251 [2020-12-31 15:15:44,485][root][INFO] - ===> Epoch[2277](11385/5): Loss -0.7831, Sem -0.7831, dir -0.9788, norm 0.1681 LR: 2.779e-02 Score 99.071 Data time: 0.0855, Forward time: 0.9486, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.3242 [2020-12-31 15:15:56,396][root][INFO] - ===> Epoch[2278](11390/5): Loss -0.7814, Sem -0.7814, dir -0.9793, norm 0.1690 LR: 2.776e-02 Score 99.044 Data time: 0.1060, Forward time: 0.9486, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.2895 [2020-12-31 15:16:08,741][root][INFO] - ===> Epoch[2279](11395/5): Loss -0.7795, Sem -0.7795, dir -0.9776, norm 0.1683 LR: 2.772e-02 Score 99.000 Data time: 0.1093, Forward time: 0.9486, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.3689 [2020-12-31 15:16:21,097][root][INFO] - ===> Epoch[2280](11400/5): Loss -0.7870, Sem -0.7870, dir -0.9798, norm 0.1657 LR: 2.769e-02 Score 99.094 Data time: 0.0852, Forward time: 0.9486, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3816 [2020-12-31 15:16:21,100][root][INFO] - ===> Start testing [2020-12-31 15:16:59,701][root][INFO] - 68/68: Data time: 0.1353, Iter time: 0.1938 Loss 0.914 (AVG: 0.550) Score 86.332 (AVG: 89.775) mIOU 67.575 mAP 82.615 mAcc 75.477 IOU: 60.127 0.000 81.951 72.801 93.890 91.684 25.296 75.883 98.397 56.763 80.984 84.901 55.800 mAP: 80.546 9.919 93.113 86.265 98.875 97.780 54.360 94.834 99.816 81.493 92.875 93.915 90.199 mAcc: 80.391 0.000 89.459 80.914 95.060 95.932 26.769 87.353 99.159 81.235 89.769 95.486 59.677 [2020-12-31 15:16:59,702][root][INFO] - Finished test. Elapsed time: 38.6006 [2020-12-31 15:17:02,945][root][INFO] - [2020-12-31 15:17:02,945][root][INFO] - ################################################################ [2020-12-31 15:17:02,945][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 15:17:02,945][root][INFO] - ################################################################ [2020-12-31 15:17:02,945][root][INFO] - clutter : nan nan nan [2020-12-31 15:17:02,945][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 15:17:02,945][root][INFO] - board : 0.784 0.868 0.897 [2020-12-31 15:17:02,945][root][INFO] - bookcase : 0.202 0.336 0.498 [2020-12-31 15:17:02,945][root][INFO] - ceiling : 0.646 0.765 0.765 [2020-12-31 15:17:02,946][root][INFO] - chair : 0.791 0.890 0.902 [2020-12-31 15:17:02,946][root][INFO] - column : 0.148 0.228 0.334 [2020-12-31 15:17:02,946][root][INFO] - door : 0.515 0.744 0.869 [2020-12-31 15:17:02,946][root][INFO] - floor : 0.930 0.952 0.952 [2020-12-31 15:17:02,946][root][INFO] - sofa : 0.495 0.727 0.809 [2020-12-31 15:17:02,946][root][INFO] - table : 0.199 0.391 0.626 [2020-12-31 15:17:02,946][root][INFO] - wall : 0.494 0.710 0.786 [2020-12-31 15:17:02,946][root][INFO] - window : 0.499 0.729 0.762 [2020-12-31 15:17:02,946][root][INFO] - ---------------------------------------------------------------- [2020-12-31 15:17:02,946][root][INFO] - average : 0.475 0.612 0.683 [2020-12-31 15:17:02,946][root][INFO] - [2020-12-31 15:17:03,124][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 15:17:03,678][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter11400.pth [2020-12-31 15:17:14,681][root][INFO] - ===> Epoch[2281](11405/5): Loss -0.7808, Sem -0.7808, dir -0.9787, norm 0.1692 LR: 2.765e-02 Score 99.047 Data time: 0.1062, Forward time: 0.9485, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.0889 [2020-12-31 15:17:25,937][root][INFO] - ===> Epoch[2282](11410/5): Loss -0.7811, Sem -0.7811, dir -0.9786, norm 0.1681 LR: 2.762e-02 Score 99.016 Data time: 0.1158, Forward time: 0.9485, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.1505 [2020-12-31 15:17:37,255][root][INFO] - ===> Epoch[2283](11415/5): Loss -0.7805, Sem -0.7805, dir -0.9785, norm 0.1681 LR: 2.758e-02 Score 98.997 Data time: 0.1437, Forward time: 0.9485, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.1738 [2020-12-31 15:17:49,491][root][INFO] - ===> Epoch[2284](11420/5): Loss -0.7821, Sem -0.7821, dir -0.9791, norm 0.1691 LR: 2.755e-02 Score 99.065 Data time: 0.1190, Forward time: 0.9485, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.3576 [2020-12-31 15:18:02,299][root][INFO] - ===> Epoch[2285](11425/5): Loss -0.7846, Sem -0.7846, dir -0.9785, norm 0.1654 LR: 2.751e-02 Score 99.052 Data time: 0.0807, Forward time: 0.9485, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.4594 [2020-12-31 15:18:13,614][root][INFO] - ===> Epoch[2286](11430/5): Loss -0.7883, Sem -0.7883, dir -0.9790, norm 0.1634 LR: 2.748e-02 Score 99.079 Data time: 0.0984, Forward time: 0.9485, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.1669 [2020-12-31 15:18:25,742][root][INFO] - ===> Epoch[2287](11435/5): Loss -0.7749, Sem -0.7749, dir -0.9786, norm 0.1743 LR: 2.744e-02 Score 99.015 Data time: 0.1367, Forward time: 0.9485, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.3310 [2020-12-31 15:18:38,282][root][INFO] - ===> Epoch[2288](11440/5): Loss -0.7838, Sem -0.7838, dir -0.9789, norm 0.1674 LR: 2.741e-02 Score 99.071 Data time: 0.1173, Forward time: 0.9485, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.4067 [2020-12-31 15:18:50,376][root][INFO] - ===> Epoch[2289](11445/5): Loss -0.7789, Sem -0.7789, dir -0.9789, norm 0.1716 LR: 2.738e-02 Score 99.052 Data time: 0.1024, Forward time: 0.9485, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.3440 [2020-12-31 15:19:02,404][root][INFO] - ===> Epoch[2290](11450/5): Loss -0.7872, Sem -0.7872, dir -0.9793, norm 0.1644 LR: 2.734e-02 Score 99.074 Data time: 0.0917, Forward time: 0.9484, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3313 [2020-12-31 15:19:15,228][root][INFO] - ===> Epoch[2291](11455/5): Loss -0.7762, Sem -0.7762, dir -0.9778, norm 0.1732 LR: 2.731e-02 Score 99.056 Data time: 0.1686, Forward time: 0.9485, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.4814 [2020-12-31 15:19:26,787][root][INFO] - ===> Epoch[2292](11460/5): Loss -0.7755, Sem -0.7755, dir -0.9783, norm 0.1730 LR: 2.727e-02 Score 98.994 Data time: 0.0881, Forward time: 0.9484, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2459 [2020-12-31 15:19:39,486][root][INFO] - ===> Epoch[2293](11465/5): Loss -0.7749, Sem -0.7749, dir -0.9786, norm 0.1749 LR: 2.724e-02 Score 99.029 Data time: 0.1310, Forward time: 0.9484, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.4317 [2020-12-31 15:19:51,656][root][INFO] - ===> Epoch[2294](11470/5): Loss -0.7763, Sem -0.7763, dir -0.9786, norm 0.1733 LR: 2.720e-02 Score 99.036 Data time: 0.0814, Forward time: 0.9484, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.3520 [2020-12-31 15:20:03,385][root][INFO] - ===> Epoch[2295](11475/5): Loss -0.7842, Sem -0.7842, dir -0.9789, norm 0.1668 LR: 2.717e-02 Score 99.064 Data time: 0.0904, Forward time: 0.9484, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.2668 [2020-12-31 15:20:14,812][root][INFO] - ===> Epoch[2296](11480/5): Loss -0.7815, Sem -0.7815, dir -0.9783, norm 0.1681 LR: 2.713e-02 Score 99.036 Data time: 0.0989, Forward time: 0.9484, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.1944 [2020-12-31 15:20:27,182][root][INFO] - ===> Epoch[2297](11485/5): Loss -0.7814, Sem -0.7814, dir -0.9786, norm 0.1695 LR: 2.710e-02 Score 99.072 Data time: 0.1532, Forward time: 0.9484, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.3889 [2020-12-31 15:20:38,610][root][INFO] - ===> Epoch[2298](11490/5): Loss -0.7947, Sem -0.7947, dir -0.9797, norm 0.1580 LR: 2.706e-02 Score 99.088 Data time: 0.0924, Forward time: 0.9484, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.1958 [2020-12-31 15:20:51,455][root][INFO] - ===> Epoch[2299](11495/5): Loss -0.7842, Sem -0.7842, dir -0.9784, norm 0.1664 LR: 2.703e-02 Score 99.069 Data time: 0.0877, Forward time: 0.9484, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.4787 [2020-12-31 15:21:03,360][root][INFO] - ===> Epoch[2300](11500/5): Loss -0.7845, Sem -0.7845, dir -0.9792, norm 0.1668 LR: 2.699e-02 Score 99.063 Data time: 0.1112, Forward time: 0.9484, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.2832 [2020-12-31 15:21:14,712][root][INFO] - ===> Epoch[2301](11505/5): Loss -0.7848, Sem -0.7848, dir -0.9787, norm 0.1653 LR: 2.696e-02 Score 99.045 Data time: 0.1063, Forward time: 0.9484, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.1896 [2020-12-31 15:21:26,724][root][INFO] - ===> Epoch[2302](11510/5): Loss -0.7846, Sem -0.7846, dir -0.9792, norm 0.1670 LR: 2.692e-02 Score 99.072 Data time: 0.0788, Forward time: 0.9484, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.3124 [2020-12-31 15:21:38,270][root][INFO] - ===> Epoch[2303](11515/5): Loss -0.7857, Sem -0.7857, dir -0.9789, norm 0.1657 LR: 2.689e-02 Score 99.071 Data time: 0.0860, Forward time: 0.9483, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.2102 [2020-12-31 15:21:50,183][root][INFO] - ===> Epoch[2304](11520/5): Loss -0.7777, Sem -0.7777, dir -0.9781, norm 0.1703 LR: 2.686e-02 Score 99.005 Data time: 0.1064, Forward time: 0.9484, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.2862 [2020-12-31 15:22:01,751][root][INFO] - ===> Epoch[2305](11525/5): Loss -0.7886, Sem -0.7886, dir -0.9790, norm 0.1629 LR: 2.682e-02 Score 99.071 Data time: 0.1078, Forward time: 0.9483, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.2184 [2020-12-31 15:22:13,299][root][INFO] - ===> Epoch[2306](11530/5): Loss -0.7868, Sem -0.7868, dir -0.9795, norm 0.1648 LR: 2.679e-02 Score 99.070 Data time: 0.0863, Forward time: 0.9483, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.2252 [2020-12-31 15:22:25,260][root][INFO] - ===> Epoch[2307](11535/5): Loss -0.7845, Sem -0.7845, dir -0.9788, norm 0.1664 LR: 2.675e-02 Score 99.064 Data time: 0.0955, Forward time: 0.9483, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.3007 [2020-12-31 15:22:36,635][root][INFO] - ===> Epoch[2308](11540/5): Loss -0.7802, Sem -0.7802, dir -0.9785, norm 0.1702 LR: 2.672e-02 Score 99.054 Data time: 0.1740, Forward time: 0.9483, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.1728 [2020-12-31 15:22:48,543][root][INFO] - ===> Epoch[2309](11545/5): Loss -0.7800, Sem -0.7800, dir -0.9783, norm 0.1704 LR: 2.668e-02 Score 99.056 Data time: 0.1024, Forward time: 0.9483, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2979 [2020-12-31 15:23:00,291][root][INFO] - ===> Epoch[2310](11550/5): Loss -0.7888, Sem -0.7888, dir -0.9797, norm 0.1637 LR: 2.665e-02 Score 99.079 Data time: 0.0806, Forward time: 0.9483, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2762 [2020-12-31 15:23:12,678][root][INFO] - ===> Epoch[2311](11555/5): Loss -0.7782, Sem -0.7782, dir -0.9792, norm 0.1741 LR: 2.661e-02 Score 99.106 Data time: 0.1286, Forward time: 0.9483, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3906 [2020-12-31 15:23:25,233][root][INFO] - ===> Epoch[2312](11560/5): Loss -0.7794, Sem -0.7794, dir -0.9791, norm 0.1721 LR: 2.658e-02 Score 99.065 Data time: 0.1160, Forward time: 0.9483, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.4263 [2020-12-31 15:23:36,663][root][INFO] - ===> Epoch[2313](11565/5): Loss -0.7839, Sem -0.7839, dir -0.9791, norm 0.1673 LR: 2.654e-02 Score 99.071 Data time: 0.1300, Forward time: 0.9483, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.1920 [2020-12-31 15:23:48,233][root][INFO] - ===> Epoch[2314](11570/5): Loss -0.7777, Sem -0.7777, dir -0.9786, norm 0.1719 LR: 2.651e-02 Score 99.036 Data time: 0.0933, Forward time: 0.9483, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2262 [2020-12-31 15:23:59,439][root][INFO] - ===> Epoch[2315](11575/5): Loss -0.7908, Sem -0.7908, dir -0.9795, norm 0.1612 LR: 2.647e-02 Score 99.064 Data time: 0.1559, Forward time: 0.9482, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.1465 [2020-12-31 15:24:11,223][root][INFO] - ===> Epoch[2316](11580/5): Loss -0.7811, Sem -0.7811, dir -0.9785, norm 0.1684 LR: 2.644e-02 Score 99.016 Data time: 0.1200, Forward time: 0.9482, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2739 [2020-12-31 15:24:22,907][root][INFO] - ===> Epoch[2317](11585/5): Loss -0.7813, Sem -0.7813, dir -0.9790, norm 0.1684 LR: 2.640e-02 Score 99.022 Data time: 0.0793, Forward time: 0.9482, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2536 [2020-12-31 15:24:35,333][root][INFO] - ===> Epoch[2318](11590/5): Loss -0.7770, Sem -0.7770, dir -0.9788, norm 0.1732 LR: 2.637e-02 Score 99.058 Data time: 0.1007, Forward time: 0.9482, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3830 [2020-12-31 15:24:46,823][root][INFO] - ===> Epoch[2319](11595/5): Loss -0.7811, Sem -0.7811, dir -0.9780, norm 0.1681 LR: 2.633e-02 Score 99.022 Data time: 0.1373, Forward time: 0.9482, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2113 [2020-12-31 15:24:58,870][root][INFO] - ===> Epoch[2320](11600/5): Loss -0.7868, Sem -0.7868, dir -0.9792, norm 0.1651 LR: 2.630e-02 Score 99.089 Data time: 0.1462, Forward time: 0.9482, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3125 [2020-12-31 15:24:58,873][root][INFO] - ===> Start testing [2020-12-31 15:25:37,484][root][INFO] - 68/68: Data time: 0.0030, Iter time: 0.1401 Loss 0.094 (AVG: 0.539) Score 97.232 (AVG: 89.743) mIOU 67.702 mAP 82.873 mAcc 75.452 IOU: 59.824 0.000 81.502 72.889 93.427 91.068 31.222 75.984 98.321 55.110 80.794 84.890 55.093 mAP: 80.757 9.981 92.840 85.155 98.754 98.357 59.284 94.039 99.825 80.398 93.058 94.269 90.632 mAcc: 80.366 0.000 88.930 80.263 94.668 95.776 33.459 84.289 99.087 78.897 90.913 95.972 58.261 [2020-12-31 15:25:37,485][root][INFO] - Finished test. Elapsed time: 38.6100 [2020-12-31 15:25:41,081][root][INFO] - [2020-12-31 15:25:41,081][root][INFO] - ################################################################ [2020-12-31 15:25:41,081][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 15:25:41,081][root][INFO] - ################################################################ [2020-12-31 15:25:41,081][root][INFO] - clutter : nan nan nan [2020-12-31 15:25:41,081][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 15:25:41,082][root][INFO] - board : 0.789 0.848 0.900 [2020-12-31 15:25:41,082][root][INFO] - bookcase : 0.208 0.338 0.493 [2020-12-31 15:25:41,082][root][INFO] - ceiling : 0.634 0.779 0.811 [2020-12-31 15:25:41,082][root][INFO] - chair : 0.803 0.895 0.911 [2020-12-31 15:25:41,082][root][INFO] - column : 0.172 0.277 0.368 [2020-12-31 15:25:41,082][root][INFO] - door : 0.567 0.808 0.895 [2020-12-31 15:25:41,082][root][INFO] - floor : 0.880 0.929 0.955 [2020-12-31 15:25:41,082][root][INFO] - sofa : 0.467 0.727 0.801 [2020-12-31 15:25:41,082][root][INFO] - table : 0.197 0.382 0.618 [2020-12-31 15:25:41,082][root][INFO] - wall : 0.478 0.699 0.795 [2020-12-31 15:25:41,082][root][INFO] - window : 0.482 0.729 0.765 [2020-12-31 15:25:41,082][root][INFO] - ---------------------------------------------------------------- [2020-12-31 15:25:41,082][root][INFO] - average : 0.473 0.618 0.693 [2020-12-31 15:25:41,082][root][INFO] - [2020-12-31 15:25:41,245][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 15:25:41,859][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter11600.pth [2020-12-31 15:25:53,105][root][INFO] - ===> Epoch[2321](11605/5): Loss -0.7873, Sem -0.7873, dir -0.9795, norm 0.1645 LR: 2.626e-02 Score 99.071 Data time: 0.1083, Forward time: 0.9482, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.1405 [2020-12-31 15:26:04,598][root][INFO] - ===> Epoch[2322](11610/5): Loss -0.7838, Sem -0.7838, dir -0.9788, norm 0.1656 LR: 2.623e-02 Score 99.016 Data time: 0.0899, Forward time: 0.9481, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.2026 [2020-12-31 15:26:16,526][root][INFO] - ===> Epoch[2323](11615/5): Loss -0.7835, Sem -0.7835, dir -0.9790, norm 0.1672 LR: 2.619e-02 Score 99.051 Data time: 0.0898, Forward time: 0.9481, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.2925 [2020-12-31 15:26:28,091][root][INFO] - ===> Epoch[2324](11620/5): Loss -0.7830, Sem -0.7830, dir -0.9788, norm 0.1667 LR: 2.616e-02 Score 99.027 Data time: 0.0949, Forward time: 0.9480, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2203 [2020-12-31 15:26:40,877][root][INFO] - ===> Epoch[2325](11625/5): Loss -0.7840, Sem -0.7840, dir -0.9785, norm 0.1663 LR: 2.612e-02 Score 99.053 Data time: 0.0775, Forward time: 0.9480, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.4858 [2020-12-31 15:26:52,360][root][INFO] - ===> Epoch[2326](11630/5): Loss -0.7870, Sem -0.7870, dir -0.9790, norm 0.1633 LR: 2.609e-02 Score 99.029 Data time: 0.0848, Forward time: 0.9480, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.2137 [2020-12-31 15:27:03,850][root][INFO] - ===> Epoch[2327](11635/5): Loss -0.7854, Sem -0.7854, dir -0.9789, norm 0.1660 LR: 2.606e-02 Score 99.072 Data time: 0.1085, Forward time: 0.9479, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.2060 [2020-12-31 15:27:16,083][root][INFO] - ===> Epoch[2328](11640/5): Loss -0.7838, Sem -0.7838, dir -0.9792, norm 0.1670 LR: 2.602e-02 Score 99.066 Data time: 0.1167, Forward time: 0.9480, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.3523 [2020-12-31 15:27:28,360][root][INFO] - ===> Epoch[2329](11645/5): Loss -0.7845, Sem -0.7845, dir -0.9788, norm 0.1667 LR: 2.599e-02 Score 99.077 Data time: 0.1337, Forward time: 0.9480, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.3660 [2020-12-31 15:27:39,403][root][INFO] - ===> Epoch[2330](11650/5): Loss -0.7776, Sem -0.7776, dir -0.9782, norm 0.1718 LR: 2.595e-02 Score 99.037 Data time: 0.1086, Forward time: 0.9479, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.1244 [2020-12-31 15:27:52,182][root][INFO] - ===> Epoch[2331](11655/5): Loss -0.7774, Sem -0.7774, dir -0.9788, norm 0.1728 LR: 2.592e-02 Score 99.047 Data time: 0.1343, Forward time: 0.9479, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.4648 [2020-12-31 15:28:03,860][root][INFO] - ===> Epoch[2332](11660/5): Loss -0.7862, Sem -0.7862, dir -0.9789, norm 0.1644 LR: 2.588e-02 Score 99.050 Data time: 0.1124, Forward time: 0.9479, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2393 [2020-12-31 15:28:15,832][root][INFO] - ===> Epoch[2333](11665/5): Loss -0.7880, Sem -0.7880, dir -0.9792, norm 0.1639 LR: 2.585e-02 Score 99.079 Data time: 0.1152, Forward time: 0.9479, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.2938 [2020-12-31 15:28:28,151][root][INFO] - ===> Epoch[2334](11670/5): Loss -0.7796, Sem -0.7796, dir -0.9785, norm 0.1702 LR: 2.581e-02 Score 99.038 Data time: 0.1343, Forward time: 0.9479, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.3877 [2020-12-31 15:28:40,508][root][INFO] - ===> Epoch[2335](11675/5): Loss -0.7881, Sem -0.7881, dir -0.9788, norm 0.1634 LR: 2.578e-02 Score 99.090 Data time: 0.1114, Forward time: 0.9479, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.3798 [2020-12-31 15:28:52,647][root][INFO] - ===> Epoch[2336](11680/5): Loss -0.7856, Sem -0.7856, dir -0.9795, norm 0.1664 LR: 2.574e-02 Score 99.067 Data time: 0.1838, Forward time: 0.9480, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3556 [2020-12-31 15:29:06,311][root][INFO] - ===> Epoch[2337](11685/5): Loss -0.7823, Sem -0.7823, dir -0.9790, norm 0.1695 LR: 2.571e-02 Score 99.082 Data time: 0.0817, Forward time: 0.9480, Backward time: 1.1888, DDP time: 0.0050, Total iter time: 2.6327 [2020-12-31 15:29:16,985][root][INFO] - ===> Epoch[2338](11690/5): Loss -0.7959, Sem -0.7959, dir -0.9802, norm 0.1570 LR: 2.567e-02 Score 99.072 Data time: 0.1042, Forward time: 0.9480, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.0351 [2020-12-31 15:29:29,949][root][INFO] - ===> Epoch[2339](11695/5): Loss -0.7814, Sem -0.7814, dir -0.9786, norm 0.1676 LR: 2.564e-02 Score 99.050 Data time: 0.1102, Forward time: 0.9479, Backward time: 1.1888, DDP time: 0.0050, Total iter time: 2.5009 [2020-12-31 15:29:41,696][root][INFO] - ===> Epoch[2340](11700/5): Loss -0.7750, Sem -0.7750, dir -0.9787, norm 0.1748 LR: 2.560e-02 Score 99.035 Data time: 0.1107, Forward time: 0.9479, Backward time: 1.1888, DDP time: 0.0050, Total iter time: 2.2451 [2020-12-31 15:29:53,464][root][INFO] - ===> Epoch[2341](11705/5): Loss -0.7797, Sem -0.7797, dir -0.9790, norm 0.1713 LR: 2.557e-02 Score 99.066 Data time: 0.1030, Forward time: 0.9479, Backward time: 1.1888, DDP time: 0.0050, Total iter time: 2.2572 [2020-12-31 15:30:05,235][root][INFO] - ===> Epoch[2342](11710/5): Loss -0.7843, Sem -0.7843, dir -0.9792, norm 0.1674 LR: 2.553e-02 Score 99.072 Data time: 0.1289, Forward time: 0.9479, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.2619 [2020-12-31 15:30:17,346][root][INFO] - ===> Epoch[2343](11715/5): Loss -0.7847, Sem -0.7847, dir -0.9789, norm 0.1665 LR: 2.550e-02 Score 99.076 Data time: 0.1327, Forward time: 0.9479, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.3089 [2020-12-31 15:30:29,238][root][INFO] - ===> Epoch[2344](11720/5): Loss -0.7810, Sem -0.7810, dir -0.9787, norm 0.1684 LR: 2.546e-02 Score 99.009 Data time: 0.1463, Forward time: 0.9479, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.2933 [2020-12-31 15:30:41,573][root][INFO] - ===> Epoch[2345](11725/5): Loss -0.7788, Sem -0.7788, dir -0.9789, norm 0.1717 LR: 2.543e-02 Score 99.050 Data time: 0.0990, Forward time: 0.9479, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.3750 [2020-12-31 15:30:53,077][root][INFO] - ===> Epoch[2346](11730/5): Loss -0.7830, Sem -0.7830, dir -0.9790, norm 0.1684 LR: 2.539e-02 Score 99.079 Data time: 0.1082, Forward time: 0.9479, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.2115 [2020-12-31 15:31:05,451][root][INFO] - ===> Epoch[2347](11735/5): Loss -0.7794, Sem -0.7794, dir -0.9789, norm 0.1714 LR: 2.536e-02 Score 99.057 Data time: 0.0915, Forward time: 0.9479, Backward time: 1.1888, DDP time: 0.0050, Total iter time: 2.3826 [2020-12-31 15:31:17,322][root][INFO] - ===> Epoch[2348](11740/5): Loss -0.7814, Sem -0.7814, dir -0.9786, norm 0.1683 LR: 2.532e-02 Score 99.019 Data time: 0.2071, Forward time: 0.9479, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.2891 [2020-12-31 15:31:29,457][root][INFO] - ===> Epoch[2349](11745/5): Loss -0.7810, Sem -0.7810, dir -0.9789, norm 0.1685 LR: 2.529e-02 Score 99.020 Data time: 0.1592, Forward time: 0.9479, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.3297 [2020-12-31 15:31:41,278][root][INFO] - ===> Epoch[2350](11750/5): Loss -0.7794, Sem -0.7794, dir -0.9783, norm 0.1699 LR: 2.525e-02 Score 99.031 Data time: 0.1160, Forward time: 0.9479, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.2666 [2020-12-31 15:31:52,224][root][INFO] - ===> Epoch[2351](11755/5): Loss -0.7863, Sem -0.7863, dir -0.9799, norm 0.1646 LR: 2.522e-02 Score 99.036 Data time: 0.1494, Forward time: 0.9479, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.1057 [2020-12-31 15:32:05,696][root][INFO] - ===> Epoch[2352](11760/5): Loss -0.7851, Sem -0.7851, dir -0.9779, norm 0.1647 LR: 2.518e-02 Score 99.051 Data time: 0.1047, Forward time: 0.9479, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.6191 [2020-12-31 15:32:16,885][root][INFO] - ===> Epoch[2353](11765/5): Loss -0.7839, Sem -0.7839, dir -0.9786, norm 0.1661 LR: 2.515e-02 Score 99.036 Data time: 0.0990, Forward time: 0.9479, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.1534 [2020-12-31 15:32:27,818][root][INFO] - ===> Epoch[2354](11770/5): Loss -0.7875, Sem -0.7875, dir -0.9786, norm 0.1628 LR: 2.511e-02 Score 99.048 Data time: 0.1352, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.0958 [2020-12-31 15:32:39,222][root][INFO] - ===> Epoch[2355](11775/5): Loss -0.7838, Sem -0.7838, dir -0.9796, norm 0.1682 LR: 2.508e-02 Score 99.075 Data time: 0.0908, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.1976 [2020-12-31 15:32:50,754][root][INFO] - ===> Epoch[2356](11780/5): Loss -0.7880, Sem -0.7880, dir -0.9795, norm 0.1643 LR: 2.504e-02 Score 99.085 Data time: 0.1053, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2157 [2020-12-31 15:33:04,139][root][INFO] - ===> Epoch[2357](11785/5): Loss -0.7836, Sem -0.7836, dir -0.9789, norm 0.1676 LR: 2.501e-02 Score 99.073 Data time: 0.1080, Forward time: 0.9478, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.5780 [2020-12-31 15:33:15,974][root][INFO] - ===> Epoch[2358](11790/5): Loss -0.7795, Sem -0.7795, dir -0.9789, norm 0.1705 LR: 2.497e-02 Score 99.043 Data time: 0.0942, Forward time: 0.9478, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.2720 [2020-12-31 15:33:27,522][root][INFO] - ===> Epoch[2359](11795/5): Loss -0.7814, Sem -0.7814, dir -0.9793, norm 0.1706 LR: 2.494e-02 Score 99.087 Data time: 0.1228, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2148 [2020-12-31 15:33:39,274][root][INFO] - ===> Epoch[2360](11800/5): Loss -0.7873, Sem -0.7873, dir -0.9793, norm 0.1647 LR: 2.490e-02 Score 99.068 Data time: 0.1543, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2633 [2020-12-31 15:33:39,276][root][INFO] - ===> Start testing [2020-12-31 15:34:19,028][root][INFO] - 68/68: Data time: 0.0030, Iter time: 0.1860 Loss 0.645 (AVG: 0.527) Score 88.970 (AVG: 89.996) mIOU 68.511 mAP 83.207 mAcc 76.096 IOU: 60.545 0.000 81.094 72.876 94.166 91.370 28.739 79.149 98.499 61.494 81.037 85.231 56.450 mAP: 81.209 10.895 92.998 85.747 98.962 98.480 55.105 94.809 99.821 85.894 93.004 93.895 90.872 mAcc: 80.436 0.000 89.389 81.802 95.375 96.282 32.231 87.025 99.282 80.811 90.435 95.011 61.163 [2020-12-31 15:34:19,029][root][INFO] - Finished test. Elapsed time: 39.7502 [2020-12-31 15:34:22,349][root][INFO] - [2020-12-31 15:34:22,349][root][INFO] - ################################################################ [2020-12-31 15:34:22,349][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 15:34:22,349][root][INFO] - ################################################################ [2020-12-31 15:34:22,350][root][INFO] - clutter : nan nan nan [2020-12-31 15:34:22,350][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 15:34:22,350][root][INFO] - board : 0.776 0.846 0.901 [2020-12-31 15:34:22,350][root][INFO] - bookcase : 0.193 0.316 0.504 [2020-12-31 15:34:22,350][root][INFO] - ceiling : 0.680 0.752 0.788 [2020-12-31 15:34:22,350][root][INFO] - chair : 0.803 0.890 0.907 [2020-12-31 15:34:22,350][root][INFO] - column : 0.157 0.226 0.375 [2020-12-31 15:34:22,350][root][INFO] - door : 0.527 0.710 0.855 [2020-12-31 15:34:22,350][root][INFO] - floor : 0.917 0.955 0.985 [2020-12-31 15:34:22,350][root][INFO] - sofa : 0.508 0.727 0.801 [2020-12-31 15:34:22,350][root][INFO] - table : 0.201 0.372 0.646 [2020-12-31 15:34:22,350][root][INFO] - wall : 0.466 0.700 0.780 [2020-12-31 15:34:22,350][root][INFO] - window : 0.492 0.727 0.758 [2020-12-31 15:34:22,350][root][INFO] - ---------------------------------------------------------------- [2020-12-31 15:34:22,350][root][INFO] - average : 0.477 0.602 0.692 [2020-12-31 15:34:22,350][root][INFO] - [2020-12-31 15:34:22,539][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 15:34:23,131][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter11800.pth [2020-12-31 15:34:35,007][root][INFO] - ===> Epoch[2361](11805/5): Loss -0.7822, Sem -0.7822, dir -0.9792, norm 0.1696 LR: 2.487e-02 Score 99.087 Data time: 0.1125, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2774 [2020-12-31 15:34:46,951][root][INFO] - ===> Epoch[2362](11810/5): Loss -0.7931, Sem -0.7931, dir -0.9795, norm 0.1597 LR: 2.483e-02 Score 99.093 Data time: 0.0978, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2937 [2020-12-31 15:34:58,997][root][INFO] - ===> Epoch[2363](11815/5): Loss -0.7873, Sem -0.7873, dir -0.9789, norm 0.1631 LR: 2.480e-02 Score 99.043 Data time: 0.0859, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3254 [2020-12-31 15:35:11,498][root][INFO] - ===> Epoch[2364](11820/5): Loss -0.7848, Sem -0.7848, dir -0.9788, norm 0.1656 LR: 2.476e-02 Score 99.052 Data time: 0.1804, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3978 [2020-12-31 15:35:23,722][root][INFO] - ===> Epoch[2365](11825/5): Loss -0.7893, Sem -0.7893, dir -0.9801, norm 0.1639 LR: 2.473e-02 Score 99.089 Data time: 0.1874, Forward time: 0.9479, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.3462 [2020-12-31 15:35:35,416][root][INFO] - ===> Epoch[2366](11830/5): Loss -0.7878, Sem -0.7878, dir -0.9791, norm 0.1643 LR: 2.469e-02 Score 99.099 Data time: 0.0923, Forward time: 0.9479, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.2487 [2020-12-31 15:35:47,423][root][INFO] - ===> Epoch[2367](11835/5): Loss -0.7878, Sem -0.7878, dir -0.9794, norm 0.1640 LR: 2.466e-02 Score 99.069 Data time: 0.0916, Forward time: 0.9479, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.3168 [2020-12-31 15:35:58,659][root][INFO] - ===> Epoch[2368](11840/5): Loss -0.7766, Sem -0.7766, dir -0.9778, norm 0.1718 LR: 2.462e-02 Score 99.018 Data time: 0.0818, Forward time: 0.9478, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.1625 [2020-12-31 15:36:10,773][root][INFO] - ===> Epoch[2369](11845/5): Loss -0.7889, Sem -0.7889, dir -0.9792, norm 0.1633 LR: 2.459e-02 Score 99.095 Data time: 0.1824, Forward time: 0.9478, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.3433 [2020-12-31 15:36:22,269][root][INFO] - ===> Epoch[2370](11850/5): Loss -0.7872, Sem -0.7872, dir -0.9792, norm 0.1635 LR: 2.455e-02 Score 99.041 Data time: 0.0984, Forward time: 0.9478, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.2223 [2020-12-31 15:36:34,151][root][INFO] - ===> Epoch[2371](11855/5): Loss -0.7895, Sem -0.7895, dir -0.9797, norm 0.1631 LR: 2.452e-02 Score 99.088 Data time: 0.0960, Forward time: 0.9478, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.2863 [2020-12-31 15:36:46,390][root][INFO] - ===> Epoch[2372](11860/5): Loss -0.7874, Sem -0.7874, dir -0.9790, norm 0.1637 LR: 2.448e-02 Score 99.066 Data time: 0.0758, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3562 [2020-12-31 15:36:57,820][root][INFO] - ===> Epoch[2373](11865/5): Loss -0.7911, Sem -0.7911, dir -0.9799, norm 0.1620 LR: 2.445e-02 Score 99.095 Data time: 0.1273, Forward time: 0.9478, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.1873 [2020-12-31 15:37:09,711][root][INFO] - ===> Epoch[2374](11870/5): Loss -0.7849, Sem -0.7849, dir -0.9793, norm 0.1668 LR: 2.441e-02 Score 99.078 Data time: 0.1057, Forward time: 0.9478, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.2839 [2020-12-31 15:37:21,435][root][INFO] - ===> Epoch[2375](11875/5): Loss -0.7879, Sem -0.7879, dir -0.9794, norm 0.1632 LR: 2.438e-02 Score 99.052 Data time: 0.1268, Forward time: 0.9478, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2548 [2020-12-31 15:37:34,641][root][INFO] - ===> Epoch[2376](11880/5): Loss -0.7796, Sem -0.7796, dir -0.9788, norm 0.1699 LR: 2.434e-02 Score 99.045 Data time: 0.1140, Forward time: 0.9478, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.5463 [2020-12-31 15:37:46,531][root][INFO] - ===> Epoch[2377](11885/5): Loss -0.7918, Sem -0.7918, dir -0.9795, norm 0.1607 LR: 2.431e-02 Score 99.083 Data time: 0.1696, Forward time: 0.9478, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.2786 [2020-12-31 15:37:59,564][root][INFO] - ===> Epoch[2378](11890/5): Loss -0.7880, Sem -0.7880, dir -0.9799, norm 0.1640 LR: 2.427e-02 Score 99.063 Data time: 0.1004, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.5344 [2020-12-31 15:38:12,703][root][INFO] - ===> Epoch[2379](11895/5): Loss -0.7901, Sem -0.7901, dir -0.9792, norm 0.1614 LR: 2.424e-02 Score 99.073 Data time: 0.1439, Forward time: 0.9479, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.5229 [2020-12-31 15:38:24,728][root][INFO] - ===> Epoch[2380](11900/5): Loss -0.7834, Sem -0.7834, dir -0.9786, norm 0.1662 LR: 2.420e-02 Score 99.034 Data time: 0.1186, Forward time: 0.9479, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3189 [2020-12-31 15:38:37,004][root][INFO] - ===> Epoch[2381](11905/5): Loss -0.7863, Sem -0.7863, dir -0.9790, norm 0.1647 LR: 2.417e-02 Score 99.082 Data time: 0.1601, Forward time: 0.9479, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3653 [2020-12-31 15:38:49,044][root][INFO] - ===> Epoch[2382](11910/5): Loss -0.7880, Sem -0.7880, dir -0.9791, norm 0.1632 LR: 2.413e-02 Score 99.063 Data time: 0.0976, Forward time: 0.9479, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3250 [2020-12-31 15:39:01,393][root][INFO] - ===> Epoch[2383](11915/5): Loss -0.7917, Sem -0.7917, dir -0.9801, norm 0.1619 LR: 2.410e-02 Score 99.102 Data time: 0.0961, Forward time: 0.9479, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3800 [2020-12-31 15:39:13,355][root][INFO] - ===> Epoch[2384](11920/5): Loss -0.7838, Sem -0.7838, dir -0.9788, norm 0.1669 LR: 2.406e-02 Score 99.060 Data time: 0.1425, Forward time: 0.9479, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2992 [2020-12-31 15:39:24,912][root][INFO] - ===> Epoch[2385](11925/5): Loss -0.7845, Sem -0.7845, dir -0.9789, norm 0.1662 LR: 2.403e-02 Score 99.063 Data time: 0.0833, Forward time: 0.9479, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2245 [2020-12-31 15:39:36,167][root][INFO] - ===> Epoch[2386](11930/5): Loss -0.7857, Sem -0.7857, dir -0.9795, norm 0.1659 LR: 2.399e-02 Score 99.059 Data time: 0.1324, Forward time: 0.9479, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.1621 [2020-12-31 15:39:48,452][root][INFO] - ===> Epoch[2387](11935/5): Loss -0.7890, Sem -0.7890, dir -0.9791, norm 0.1623 LR: 2.396e-02 Score 99.068 Data time: 0.1187, Forward time: 0.9479, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.3705 [2020-12-31 15:40:00,669][root][INFO] - ===> Epoch[2388](11940/5): Loss -0.7903, Sem -0.7903, dir -0.9795, norm 0.1616 LR: 2.392e-02 Score 99.071 Data time: 0.0987, Forward time: 0.9479, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3633 [2020-12-31 15:40:12,286][root][INFO] - ===> Epoch[2389](11945/5): Loss -0.7889, Sem -0.7889, dir -0.9796, norm 0.1637 LR: 2.389e-02 Score 99.093 Data time: 0.1003, Forward time: 0.9479, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.2466 [2020-12-31 15:40:24,971][root][INFO] - ===> Epoch[2390](11950/5): Loss -0.7881, Sem -0.7881, dir -0.9798, norm 0.1635 LR: 2.385e-02 Score 99.058 Data time: 0.1018, Forward time: 0.9479, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.4586 [2020-12-31 15:40:37,236][root][INFO] - ===> Epoch[2391](11955/5): Loss -0.7872, Sem -0.7872, dir -0.9791, norm 0.1649 LR: 2.381e-02 Score 99.094 Data time: 0.1051, Forward time: 0.9479, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3589 [2020-12-31 15:40:48,955][root][INFO] - ===> Epoch[2392](11960/5): Loss -0.7944, Sem -0.7944, dir -0.9794, norm 0.1579 LR: 2.378e-02 Score 99.081 Data time: 0.1109, Forward time: 0.9479, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2430 [2020-12-31 15:41:01,036][root][INFO] - ===> Epoch[2393](11965/5): Loss -0.7872, Sem -0.7872, dir -0.9792, norm 0.1643 LR: 2.374e-02 Score 99.072 Data time: 0.1708, Forward time: 0.9479, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3208 [2020-12-31 15:41:12,959][root][INFO] - ===> Epoch[2394](11970/5): Loss -0.7894, Sem -0.7894, dir -0.9800, norm 0.1639 LR: 2.371e-02 Score 99.096 Data time: 0.1085, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3087 [2020-12-31 15:41:24,520][root][INFO] - ===> Epoch[2395](11975/5): Loss -0.7840, Sem -0.7840, dir -0.9789, norm 0.1666 LR: 2.367e-02 Score 99.052 Data time: 0.0758, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2328 [2020-12-31 15:41:36,580][root][INFO] - ===> Epoch[2396](11980/5): Loss -0.7893, Sem -0.7893, dir -0.9796, norm 0.1625 LR: 2.364e-02 Score 99.066 Data time: 0.0860, Forward time: 0.9478, Backward time: 1.1887, DDP time: 0.0050, Total iter time: 2.3297 [2020-12-31 15:41:48,219][root][INFO] - ===> Epoch[2397](11985/5): Loss -0.7876, Sem -0.7876, dir -0.9788, norm 0.1622 LR: 2.360e-02 Score 99.042 Data time: 0.1316, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2366 [2020-12-31 15:41:59,882][root][INFO] - ===> Epoch[2398](11990/5): Loss -0.7892, Sem -0.7892, dir -0.9796, norm 0.1622 LR: 2.357e-02 Score 99.054 Data time: 0.1337, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2498 [2020-12-31 15:42:11,569][root][INFO] - ===> Epoch[2399](11995/5): Loss -0.7833, Sem -0.7833, dir -0.9787, norm 0.1671 LR: 2.353e-02 Score 99.055 Data time: 0.2081, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2358 [2020-12-31 15:42:23,431][root][INFO] - ===> Epoch[2400](12000/5): Loss -0.7878, Sem -0.7878, dir -0.9796, norm 0.1639 LR: 2.350e-02 Score 99.064 Data time: 0.0797, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.2856 [2020-12-31 15:42:23,434][root][INFO] - ===> Start testing [2020-12-31 15:43:02,915][root][INFO] - 68/68: Data time: 0.0027, Iter time: 0.1467 Loss 0.280 (AVG: 0.553) Score 93.819 (AVG: 89.792) mIOU 67.215 mAP 82.484 mAcc 75.383 IOU: 60.987 0.000 79.846 72.972 93.492 89.344 27.355 74.772 98.058 54.734 81.804 85.190 55.242 mAP: 81.887 10.264 92.889 86.038 98.770 97.990 53.322 95.258 99.802 78.366 93.160 93.966 90.586 mAcc: 81.759 0.000 88.978 79.855 94.463 96.340 29.281 89.200 98.621 75.799 90.880 95.821 58.985 [2020-12-31 15:43:02,917][root][INFO] - Finished test. Elapsed time: 39.4802 [2020-12-31 15:43:06,652][root][INFO] - [2020-12-31 15:43:06,652][root][INFO] - ################################################################ [2020-12-31 15:43:06,652][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 15:43:06,652][root][INFO] - ################################################################ [2020-12-31 15:43:06,652][root][INFO] - clutter : nan nan nan [2020-12-31 15:43:06,652][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 15:43:06,652][root][INFO] - board : 0.794 0.868 0.898 [2020-12-31 15:43:06,652][root][INFO] - bookcase : 0.185 0.317 0.485 [2020-12-31 15:43:06,652][root][INFO] - ceiling : 0.713 0.864 0.874 [2020-12-31 15:43:06,652][root][INFO] - chair : 0.782 0.882 0.893 [2020-12-31 15:43:06,652][root][INFO] - column : 0.160 0.233 0.347 [2020-12-31 15:43:06,652][root][INFO] - door : 0.523 0.716 0.846 [2020-12-31 15:43:06,652][root][INFO] - floor : 0.942 0.966 0.981 [2020-12-31 15:43:06,652][root][INFO] - sofa : 0.444 0.727 0.727 [2020-12-31 15:43:06,653][root][INFO] - table : 0.189 0.379 0.657 [2020-12-31 15:43:06,653][root][INFO] - wall : 0.479 0.694 0.776 [2020-12-31 15:43:06,653][root][INFO] - window : 0.467 0.693 0.747 [2020-12-31 15:43:06,653][root][INFO] - ---------------------------------------------------------------- [2020-12-31 15:43:06,653][root][INFO] - average : 0.473 0.612 0.686 [2020-12-31 15:43:06,653][root][INFO] - [2020-12-31 15:43:06,832][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 15:43:07,460][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter12000.pth [2020-12-31 15:43:19,688][root][INFO] - ===> Epoch[2401](12005/5): Loss -0.7883, Sem -0.7883, dir -0.9793, norm 0.1627 LR: 2.346e-02 Score 99.051 Data time: 0.1372, Forward time: 0.9478, Backward time: 1.1886, DDP time: 0.0050, Total iter time: 2.3372 [2020-12-31 15:43:30,778][root][INFO] - ===> Epoch[2402](12010/5): Loss -0.7898, Sem -0.7898, dir -0.9793, norm 0.1622 LR: 2.343e-02 Score 99.079 Data time: 0.0959, Forward time: 0.9478, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.1382 [2020-12-31 15:43:42,594][root][INFO] - ===> Epoch[2403](12015/5): Loss -0.7881, Sem -0.7881, dir -0.9798, norm 0.1651 LR: 2.339e-02 Score 99.104 Data time: 0.1395, Forward time: 0.9478, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.2632 [2020-12-31 15:43:54,195][root][INFO] - ===> Epoch[2404](12020/5): Loss -0.7859, Sem -0.7859, dir -0.9796, norm 0.1661 LR: 2.336e-02 Score 99.076 Data time: 0.0958, Forward time: 0.9478, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.2274 [2020-12-31 15:44:05,563][root][INFO] - ===> Epoch[2405](12025/5): Loss -0.7874, Sem -0.7874, dir -0.9796, norm 0.1648 LR: 2.332e-02 Score 99.073 Data time: 0.1006, Forward time: 0.9477, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.1813 [2020-12-31 15:44:16,546][root][INFO] - ===> Epoch[2406](12030/5): Loss -0.7845, Sem -0.7845, dir -0.9790, norm 0.1662 LR: 2.329e-02 Score 99.051 Data time: 0.0834, Forward time: 0.9477, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.1024 [2020-12-31 15:44:28,020][root][INFO] - ===> Epoch[2407](12035/5): Loss -0.7830, Sem -0.7830, dir -0.9788, norm 0.1673 LR: 2.325e-02 Score 99.043 Data time: 0.0869, Forward time: 0.9477, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2057 [2020-12-31 15:44:39,418][root][INFO] - ===> Epoch[2408](12040/5): Loss -0.7891, Sem -0.7891, dir -0.9795, norm 0.1630 LR: 2.322e-02 Score 99.076 Data time: 0.1571, Forward time: 0.9477, Backward time: 1.1883, DDP time: 0.0050, Total iter time: 2.1890 [2020-12-31 15:44:49,957][root][INFO] - ===> Epoch[2409](12045/5): Loss -0.7867, Sem -0.7867, dir -0.9795, norm 0.1655 LR: 2.318e-02 Score 99.090 Data time: 0.1345, Forward time: 0.9477, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.0160 [2020-12-31 15:45:02,669][root][INFO] - ===> Epoch[2410](12050/5): Loss -0.7882, Sem -0.7882, dir -0.9791, norm 0.1634 LR: 2.315e-02 Score 99.071 Data time: 0.1033, Forward time: 0.9476, Backward time: 1.1883, DDP time: 0.0050, Total iter time: 2.4506 [2020-12-31 15:45:14,723][root][INFO] - ===> Epoch[2411](12055/5): Loss -0.7831, Sem -0.7831, dir -0.9793, norm 0.1683 LR: 2.311e-02 Score 99.060 Data time: 0.1760, Forward time: 0.9477, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.2966 [2020-12-31 15:45:26,736][root][INFO] - ===> Epoch[2412](12060/5): Loss -0.7850, Sem -0.7850, dir -0.9792, norm 0.1657 LR: 2.307e-02 Score 99.054 Data time: 0.1218, Forward time: 0.9477, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.3055 [2020-12-31 15:45:38,156][root][INFO] - ===> Epoch[2413](12065/5): Loss -0.7894, Sem -0.7894, dir -0.9793, norm 0.1625 LR: 2.304e-02 Score 99.075 Data time: 0.0745, Forward time: 0.9477, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.2152 [2020-12-31 15:45:50,089][root][INFO] - ===> Epoch[2414](12070/5): Loss -0.7883, Sem -0.7883, dir -0.9796, norm 0.1640 LR: 2.300e-02 Score 99.079 Data time: 0.1038, Forward time: 0.9477, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.2982 [2020-12-31 15:46:01,482][root][INFO] - ===> Epoch[2415](12075/5): Loss -0.7868, Sem -0.7868, dir -0.9787, norm 0.1648 LR: 2.297e-02 Score 99.083 Data time: 0.1325, Forward time: 0.9477, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.1835 [2020-12-31 15:46:13,631][root][INFO] - ===> Epoch[2416](12080/5): Loss -0.7938, Sem -0.7938, dir -0.9796, norm 0.1588 LR: 2.293e-02 Score 99.092 Data time: 0.1622, Forward time: 0.9477, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.3348 [2020-12-31 15:46:26,238][root][INFO] - ===> Epoch[2417](12085/5): Loss -0.7851, Sem -0.7851, dir -0.9788, norm 0.1660 LR: 2.290e-02 Score 99.060 Data time: 0.1457, Forward time: 0.9478, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.4190 [2020-12-31 15:46:37,593][root][INFO] - ===> Epoch[2418](12090/5): Loss -0.7886, Sem -0.7886, dir -0.9792, norm 0.1620 LR: 2.286e-02 Score 99.046 Data time: 0.1006, Forward time: 0.9478, Backward time: 1.1880, DDP time: 0.0050, Total iter time: 2.1768 [2020-12-31 15:46:49,162][root][INFO] - ===> Epoch[2419](12095/5): Loss -0.7898, Sem -0.7898, dir -0.9791, norm 0.1613 LR: 2.283e-02 Score 99.063 Data time: 0.1167, Forward time: 0.9478, Backward time: 1.1880, DDP time: 0.0050, Total iter time: 2.2280 [2020-12-31 15:47:01,235][root][INFO] - ===> Epoch[2420](12100/5): Loss -0.7921, Sem -0.7921, dir -0.9795, norm 0.1606 LR: 2.279e-02 Score 99.104 Data time: 0.0958, Forward time: 0.9478, Backward time: 1.1880, DDP time: 0.0050, Total iter time: 2.3326 [2020-12-31 15:47:13,406][root][INFO] - ===> Epoch[2421](12105/5): Loss -0.7914, Sem -0.7914, dir -0.9792, norm 0.1605 LR: 2.276e-02 Score 99.079 Data time: 0.1263, Forward time: 0.9478, Backward time: 1.1880, DDP time: 0.0050, Total iter time: 2.3332 [2020-12-31 15:47:25,711][root][INFO] - ===> Epoch[2422](12110/5): Loss -0.7878, Sem -0.7878, dir -0.9797, norm 0.1640 LR: 2.272e-02 Score 99.073 Data time: 0.1001, Forward time: 0.9477, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.3696 [2020-12-31 15:47:36,692][root][INFO] - ===> Epoch[2423](12115/5): Loss -0.7910, Sem -0.7910, dir -0.9788, norm 0.1602 LR: 2.269e-02 Score 99.078 Data time: 0.0894, Forward time: 0.9477, Backward time: 1.1880, DDP time: 0.0050, Total iter time: 2.1013 [2020-12-31 15:47:49,401][root][INFO] - ===> Epoch[2424](12120/5): Loss -0.7906, Sem -0.7906, dir -0.9798, norm 0.1620 LR: 2.265e-02 Score 99.089 Data time: 0.0811, Forward time: 0.9477, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.4546 [2020-12-31 15:48:01,786][root][INFO] - ===> Epoch[2425](12125/5): Loss -0.7876, Sem -0.7876, dir -0.9794, norm 0.1636 LR: 2.262e-02 Score 99.055 Data time: 0.1747, Forward time: 0.9477, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.3806 [2020-12-31 15:48:15,331][root][INFO] - ===> Epoch[2426](12130/5): Loss -0.7838, Sem -0.7838, dir -0.9794, norm 0.1678 LR: 2.258e-02 Score 99.062 Data time: 0.0839, Forward time: 0.9477, Backward time: 1.1883, DDP time: 0.0050, Total iter time: 2.6208 [2020-12-31 15:48:26,888][root][INFO] - ===> Epoch[2427](12135/5): Loss -0.7924, Sem -0.7924, dir -0.9797, norm 0.1606 LR: 2.254e-02 Score 99.095 Data time: 0.1084, Forward time: 0.9477, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.2246 [2020-12-31 15:48:38,674][root][INFO] - ===> Epoch[2428](12140/5): Loss -0.7911, Sem -0.7911, dir -0.9790, norm 0.1607 LR: 2.251e-02 Score 99.086 Data time: 0.1360, Forward time: 0.9477, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.2707 [2020-12-31 15:48:49,873][root][INFO] - ===> Epoch[2429](12145/5): Loss -0.7865, Sem -0.7865, dir -0.9797, norm 0.1651 LR: 2.247e-02 Score 99.057 Data time: 0.1660, Forward time: 0.9477, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.1408 [2020-12-31 15:49:01,732][root][INFO] - ===> Epoch[2430](12150/5): Loss -0.7887, Sem -0.7887, dir -0.9792, norm 0.1626 LR: 2.244e-02 Score 99.068 Data time: 0.1074, Forward time: 0.9477, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.2835 [2020-12-31 15:49:13,697][root][INFO] - ===> Epoch[2431](12155/5): Loss -0.7852, Sem -0.7852, dir -0.9792, norm 0.1655 LR: 2.240e-02 Score 99.055 Data time: 0.1317, Forward time: 0.9477, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.2967 [2020-12-31 15:49:25,721][root][INFO] - ===> Epoch[2432](12160/5): Loss -0.7936, Sem -0.7936, dir -0.9793, norm 0.1587 LR: 2.237e-02 Score 99.091 Data time: 0.1139, Forward time: 0.9477, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.3122 [2020-12-31 15:49:37,409][root][INFO] - ===> Epoch[2433](12165/5): Loss -0.7882, Sem -0.7882, dir -0.9795, norm 0.1631 LR: 2.233e-02 Score 99.070 Data time: 0.0972, Forward time: 0.9477, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.2505 [2020-12-31 15:49:49,421][root][INFO] - ===> Epoch[2434](12170/5): Loss -0.7932, Sem -0.7932, dir -0.9799, norm 0.1594 LR: 2.230e-02 Score 99.091 Data time: 0.0816, Forward time: 0.9476, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.3279 [2020-12-31 15:50:01,587][root][INFO] - ===> Epoch[2435](12175/5): Loss -0.7956, Sem -0.7956, dir -0.9797, norm 0.1576 LR: 2.226e-02 Score 99.101 Data time: 0.0890, Forward time: 0.9476, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.3395 [2020-12-31 15:50:14,673][root][INFO] - ===> Epoch[2436](12180/5): Loss -0.7828, Sem -0.7828, dir -0.9787, norm 0.1674 LR: 2.223e-02 Score 99.061 Data time: 0.0840, Forward time: 0.9476, Backward time: 1.1883, DDP time: 0.0050, Total iter time: 2.5266 [2020-12-31 15:50:26,954][root][INFO] - ===> Epoch[2437](12185/5): Loss -0.7948, Sem -0.7948, dir -0.9798, norm 0.1571 LR: 2.219e-02 Score 99.074 Data time: 0.0830, Forward time: 0.9476, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.3906 [2020-12-31 15:50:38,789][root][INFO] - ===> Epoch[2438](12190/5): Loss -0.7889, Sem -0.7889, dir -0.9790, norm 0.1611 LR: 2.215e-02 Score 99.030 Data time: 0.1969, Forward time: 0.9476, Backward time: 1.1883, DDP time: 0.0050, Total iter time: 2.2775 [2020-12-31 15:50:50,373][root][INFO] - ===> Epoch[2439](12195/5): Loss -0.7910, Sem -0.7910, dir -0.9793, norm 0.1605 LR: 2.212e-02 Score 99.068 Data time: 0.1122, Forward time: 0.9476, Backward time: 1.1883, DDP time: 0.0050, Total iter time: 2.2279 [2020-12-31 15:51:02,428][root][INFO] - ===> Epoch[2440](12200/5): Loss -0.7945, Sem -0.7945, dir -0.9800, norm 0.1579 LR: 2.208e-02 Score 99.076 Data time: 0.1020, Forward time: 0.9476, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.3255 [2020-12-31 15:51:02,431][root][INFO] - ===> Start testing [2020-12-31 15:51:41,709][root][INFO] - 68/68: Data time: 0.0026, Iter time: 0.1421 Loss 0.301 (AVG: 0.505) Score 93.259 (AVG: 90.223) mIOU 68.838 mAP 83.138 mAcc 76.307 IOU: 61.362 0.000 80.767 73.977 94.345 91.636 34.004 78.385 98.324 58.378 80.536 85.349 57.828 mAP: 81.648 9.409 92.529 86.618 98.936 98.287 59.758 95.570 99.828 80.018 93.153 93.955 91.089 mAcc: 80.057 0.000 88.905 82.650 95.543 95.759 38.824 88.229 99.126 74.724 90.330 94.967 62.875 [2020-12-31 15:51:41,710][root][INFO] - Finished test. Elapsed time: 39.2773 [2020-12-31 15:51:44,757][root][INFO] - [2020-12-31 15:51:44,757][root][INFO] - ################################################################ [2020-12-31 15:51:44,757][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 15:51:44,757][root][INFO] - ################################################################ [2020-12-31 15:51:44,757][root][INFO] - clutter : nan nan nan [2020-12-31 15:51:44,758][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 15:51:44,758][root][INFO] - board : 0.774 0.850 0.897 [2020-12-31 15:51:44,758][root][INFO] - bookcase : 0.196 0.352 0.508 [2020-12-31 15:51:44,758][root][INFO] - ceiling : 0.674 0.814 0.814 [2020-12-31 15:51:44,758][root][INFO] - chair : 0.796 0.886 0.895 [2020-12-31 15:51:44,758][root][INFO] - column : 0.183 0.273 0.390 [2020-12-31 15:51:44,758][root][INFO] - door : 0.614 0.799 0.923 [2020-12-31 15:51:44,758][root][INFO] - floor : 0.948 0.992 0.998 [2020-12-31 15:51:44,758][root][INFO] - sofa : 0.427 0.636 0.818 [2020-12-31 15:51:44,758][root][INFO] - table : 0.207 0.393 0.618 [2020-12-31 15:51:44,758][root][INFO] - wall : 0.494 0.725 0.808 [2020-12-31 15:51:44,758][root][INFO] - window : 0.508 0.743 0.773 [2020-12-31 15:51:44,758][root][INFO] - ---------------------------------------------------------------- [2020-12-31 15:51:44,758][root][INFO] - average : 0.485 0.622 0.704 [2020-12-31 15:51:44,758][root][INFO] - [2020-12-31 15:51:44,927][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 15:51:45,524][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2020-12-31 15:51:45,525][root][INFO] - Current best mIoU: 68.838 at iter 1 [2020-12-31 15:51:46,085][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter12200.pth [2020-12-31 15:51:57,964][root][INFO] - ===> Epoch[2441](12205/5): Loss -0.7915, Sem -0.7915, dir -0.9793, norm 0.1602 LR: 2.205e-02 Score 99.066 Data time: 0.0914, Forward time: 0.9476, Backward time: 1.1883, DDP time: 0.0050, Total iter time: 2.2994 [2020-12-31 15:52:10,022][root][INFO] - ===> Epoch[2442](12210/5): Loss -0.7946, Sem -0.7946, dir -0.9797, norm 0.1585 LR: 2.201e-02 Score 99.102 Data time: 0.1787, Forward time: 0.9476, Backward time: 1.1883, DDP time: 0.0050, Total iter time: 2.3193 [2020-12-31 15:52:21,758][root][INFO] - ===> Epoch[2443](12215/5): Loss -0.7988, Sem -0.7988, dir -0.9803, norm 0.1547 LR: 2.198e-02 Score 99.096 Data time: 0.0895, Forward time: 0.9476, Backward time: 1.1883, DDP time: 0.0050, Total iter time: 2.2703 [2020-12-31 15:52:34,401][root][INFO] - ===> Epoch[2444](12220/5): Loss -0.7944, Sem -0.7944, dir -0.9796, norm 0.1580 LR: 2.194e-02 Score 99.090 Data time: 0.1043, Forward time: 0.9475, Backward time: 1.1883, DDP time: 0.0050, Total iter time: 2.4372 [2020-12-31 15:52:46,982][root][INFO] - ===> Epoch[2445](12225/5): Loss -0.7880, Sem -0.7880, dir -0.9793, norm 0.1634 LR: 2.191e-02 Score 99.065 Data time: 0.1124, Forward time: 0.9475, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.4347 [2020-12-31 15:52:58,276][root][INFO] - ===> Epoch[2446](12230/5): Loss -0.7815, Sem -0.7815, dir -0.9787, norm 0.1678 LR: 2.187e-02 Score 99.016 Data time: 0.1276, Forward time: 0.9475, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.1746 [2020-12-31 15:53:10,745][root][INFO] - ===> Epoch[2447](12235/5): Loss -0.7881, Sem -0.7881, dir -0.9792, norm 0.1636 LR: 2.184e-02 Score 99.079 Data time: 0.0817, Forward time: 0.9475, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.3966 [2020-12-31 15:53:22,479][root][INFO] - ===> Epoch[2448](12240/5): Loss -0.7921, Sem -0.7921, dir -0.9801, norm 0.1608 LR: 2.180e-02 Score 99.095 Data time: 0.0893, Forward time: 0.9475, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2626 [2020-12-31 15:53:35,512][root][INFO] - ===> Epoch[2449](12245/5): Loss -0.7861, Sem -0.7861, dir -0.9790, norm 0.1645 LR: 2.176e-02 Score 99.048 Data time: 0.0963, Forward time: 0.9474, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.5106 [2020-12-31 15:53:46,518][root][INFO] - ===> Epoch[2450](12250/5): Loss -0.7910, Sem -0.7910, dir -0.9797, norm 0.1621 LR: 2.173e-02 Score 99.110 Data time: 0.0931, Forward time: 0.9474, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.0945 [2020-12-31 15:53:58,148][root][INFO] - ===> Epoch[2451](12255/5): Loss -0.7901, Sem -0.7901, dir -0.9795, norm 0.1619 LR: 2.169e-02 Score 99.067 Data time: 0.1544, Forward time: 0.9475, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2331 [2020-12-31 15:54:10,230][root][INFO] - ===> Epoch[2452](12260/5): Loss -0.7956, Sem -0.7956, dir -0.9803, norm 0.1581 LR: 2.166e-02 Score 99.109 Data time: 0.0852, Forward time: 0.9474, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.3433 [2020-12-31 15:54:22,928][root][INFO] - ===> Epoch[2453](12265/5): Loss -0.7950, Sem -0.7950, dir -0.9798, norm 0.1581 LR: 2.162e-02 Score 99.094 Data time: 0.1081, Forward time: 0.9475, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.4449 [2020-12-31 15:54:35,030][root][INFO] - ===> Epoch[2454](12270/5): Loss -0.7915, Sem -0.7915, dir -0.9799, norm 0.1613 LR: 2.159e-02 Score 99.099 Data time: 0.1247, Forward time: 0.9475, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.3393 [2020-12-31 15:54:47,532][root][INFO] - ===> Epoch[2455](12275/5): Loss -0.7884, Sem -0.7884, dir -0.9797, norm 0.1650 LR: 2.155e-02 Score 99.127 Data time: 0.1114, Forward time: 0.9475, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.4068 [2020-12-31 15:54:59,857][root][INFO] - ===> Epoch[2456](12280/5): Loss -0.7886, Sem -0.7886, dir -0.9789, norm 0.1624 LR: 2.152e-02 Score 99.059 Data time: 0.1681, Forward time: 0.9475, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.3672 [2020-12-31 15:55:10,782][root][INFO] - ===> Epoch[2457](12285/5): Loss -0.7882, Sem -0.7882, dir -0.9794, norm 0.1632 LR: 2.148e-02 Score 99.069 Data time: 0.1085, Forward time: 0.9475, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.0880 [2020-12-31 15:55:22,667][root][INFO] - ===> Epoch[2458](12290/5): Loss -0.7868, Sem -0.7868, dir -0.9796, norm 0.1642 LR: 2.144e-02 Score 99.050 Data time: 0.1697, Forward time: 0.9475, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2782 [2020-12-31 15:55:34,154][root][INFO] - ===> Epoch[2459](12295/5): Loss -0.7901, Sem -0.7901, dir -0.9790, norm 0.1614 LR: 2.141e-02 Score 99.075 Data time: 0.1275, Forward time: 0.9475, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.1956 [2020-12-31 15:55:46,292][root][INFO] - ===> Epoch[2460](12300/5): Loss -0.7913, Sem -0.7913, dir -0.9797, norm 0.1611 LR: 2.137e-02 Score 99.088 Data time: 0.1296, Forward time: 0.9475, Backward time: 1.1883, DDP time: 0.0050, Total iter time: 2.3322 [2020-12-31 15:55:57,868][root][INFO] - ===> Epoch[2461](12305/5): Loss -0.7955, Sem -0.7955, dir -0.9800, norm 0.1572 LR: 2.134e-02 Score 99.073 Data time: 0.0839, Forward time: 0.9475, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2308 [2020-12-31 15:56:09,664][root][INFO] - ===> Epoch[2462](12310/5): Loss -0.7898, Sem -0.7898, dir -0.9793, norm 0.1623 LR: 2.130e-02 Score 99.088 Data time: 0.1054, Forward time: 0.9475, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2660 [2020-12-31 15:56:22,623][root][INFO] - ===> Epoch[2463](12315/5): Loss -0.7952, Sem -0.7952, dir -0.9803, norm 0.1592 LR: 2.127e-02 Score 99.118 Data time: 0.1262, Forward time: 0.9474, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.4916 [2020-12-31 15:56:34,339][root][INFO] - ===> Epoch[2464](12320/5): Loss -0.7903, Sem -0.7903, dir -0.9797, norm 0.1611 LR: 2.123e-02 Score 99.053 Data time: 0.1024, Forward time: 0.9474, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2573 [2020-12-31 15:56:46,803][root][INFO] - ===> Epoch[2465](12325/5): Loss -0.7922, Sem -0.7922, dir -0.9798, norm 0.1608 LR: 2.119e-02 Score 99.101 Data time: 0.1487, Forward time: 0.9475, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.3998 [2020-12-31 15:56:59,439][root][INFO] - ===> Epoch[2466](12330/5): Loss -0.7948, Sem -0.7948, dir -0.9797, norm 0.1571 LR: 2.116e-02 Score 99.086 Data time: 0.1308, Forward time: 0.9475, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.4415 [2020-12-31 15:57:12,734][root][INFO] - ===> Epoch[2467](12335/5): Loss -0.7889, Sem -0.7889, dir -0.9792, norm 0.1628 LR: 2.112e-02 Score 99.080 Data time: 0.1455, Forward time: 0.9475, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.5717 [2020-12-31 15:57:24,049][root][INFO] - ===> Epoch[2468](12340/5): Loss -0.7869, Sem -0.7869, dir -0.9794, norm 0.1638 LR: 2.109e-02 Score 99.038 Data time: 0.0684, Forward time: 0.9474, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.1619 [2020-12-31 15:57:35,943][root][INFO] - ===> Epoch[2469](12345/5): Loss -0.7958, Sem -0.7958, dir -0.9796, norm 0.1572 LR: 2.105e-02 Score 99.105 Data time: 0.1603, Forward time: 0.9475, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.2894 [2020-12-31 15:57:48,221][root][INFO] - ===> Epoch[2470](12350/5): Loss -0.7858, Sem -0.7858, dir -0.9791, norm 0.1641 LR: 2.102e-02 Score 99.024 Data time: 0.1422, Forward time: 0.9475, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.3627 [2020-12-31 15:57:59,464][root][INFO] - ===> Epoch[2471](12355/5): Loss -0.7871, Sem -0.7871, dir -0.9790, norm 0.1633 LR: 2.098e-02 Score 99.054 Data time: 0.1289, Forward time: 0.9475, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.1716 [2020-12-31 15:58:11,304][root][INFO] - ===> Epoch[2472](12360/5): Loss -0.7918, Sem -0.7918, dir -0.9797, norm 0.1611 LR: 2.095e-02 Score 99.105 Data time: 0.0933, Forward time: 0.9475, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2868 [2020-12-31 15:58:22,887][root][INFO] - ===> Epoch[2473](12365/5): Loss -0.7989, Sem -0.7989, dir -0.9802, norm 0.1555 LR: 2.091e-02 Score 99.128 Data time: 0.1311, Forward time: 0.9475, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2257 [2020-12-31 15:58:34,060][root][INFO] - ===> Epoch[2474](12370/5): Loss -0.7908, Sem -0.7908, dir -0.9792, norm 0.1600 LR: 2.087e-02 Score 99.058 Data time: 0.0960, Forward time: 0.9474, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.1373 [2020-12-31 15:58:46,669][root][INFO] - ===> Epoch[2475](12375/5): Loss -0.7924, Sem -0.7924, dir -0.9801, norm 0.1606 LR: 2.084e-02 Score 99.089 Data time: 0.0870, Forward time: 0.9474, Backward time: 1.1885, DDP time: 0.0050, Total iter time: 2.4332 [2020-12-31 15:58:58,201][root][INFO] - ===> Epoch[2476](12380/5): Loss -0.7878, Sem -0.7878, dir -0.9788, norm 0.1627 LR: 2.080e-02 Score 99.051 Data time: 0.1194, Forward time: 0.9474, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2059 [2020-12-31 15:59:10,085][root][INFO] - ===> Epoch[2477](12385/5): Loss -0.7888, Sem -0.7888, dir -0.9799, norm 0.1635 LR: 2.077e-02 Score 99.084 Data time: 0.1400, Forward time: 0.9474, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2807 [2020-12-31 15:59:21,777][root][INFO] - ===> Epoch[2478](12390/5): Loss -0.7870, Sem -0.7870, dir -0.9792, norm 0.1649 LR: 2.073e-02 Score 99.077 Data time: 0.0860, Forward time: 0.9473, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2598 [2020-12-31 15:59:33,699][root][INFO] - ===> Epoch[2479](12395/5): Loss -0.7928, Sem -0.7928, dir -0.9795, norm 0.1606 LR: 2.070e-02 Score 99.119 Data time: 0.1099, Forward time: 0.9473, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2859 [2020-12-31 15:59:45,812][root][INFO] - ===> Epoch[2480](12400/5): Loss -0.7932, Sem -0.7932, dir -0.9803, norm 0.1611 LR: 2.066e-02 Score 99.127 Data time: 0.1076, Forward time: 0.9473, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.3261 [2020-12-31 15:59:45,816][root][INFO] - ===> Start testing [2020-12-31 16:00:24,820][root][INFO] - 68/68: Data time: 0.0029, Iter time: 0.1289 Loss 0.949 (AVG: 0.532) Score 86.365 (AVG: 89.884) mIOU 68.202 mAP 83.182 mAcc 75.944 IOU: 60.257 0.000 80.132 72.743 94.122 91.181 31.410 77.185 98.344 59.377 81.234 85.022 55.617 mAP: 80.605 12.054 92.703 85.526 98.988 98.286 59.064 94.814 99.822 82.221 93.100 94.016 90.163 mAcc: 80.514 0.000 88.730 80.172 95.434 96.545 35.559 87.403 99.033 78.404 91.016 95.300 59.168 [2020-12-31 16:00:24,821][root][INFO] - Finished test. Elapsed time: 39.0030 [2020-12-31 16:00:28,648][root][INFO] - [2020-12-31 16:00:28,648][root][INFO] - ################################################################ [2020-12-31 16:00:28,648][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 16:00:28,648][root][INFO] - ################################################################ [2020-12-31 16:00:28,648][root][INFO] - clutter : nan nan nan [2020-12-31 16:00:28,648][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 16:00:28,648][root][INFO] - board : 0.785 0.844 0.900 [2020-12-31 16:00:28,648][root][INFO] - bookcase : 0.195 0.337 0.518 [2020-12-31 16:00:28,648][root][INFO] - ceiling : 0.581 0.690 0.787 [2020-12-31 16:00:28,648][root][INFO] - chair : 0.787 0.888 0.906 [2020-12-31 16:00:28,648][root][INFO] - column : 0.175 0.281 0.371 [2020-12-31 16:00:28,648][root][INFO] - door : 0.612 0.828 0.934 [2020-12-31 16:00:28,648][root][INFO] - floor : 0.934 0.967 0.981 [2020-12-31 16:00:28,649][root][INFO] - sofa : 0.453 0.727 0.794 [2020-12-31 16:00:28,649][root][INFO] - table : 0.215 0.429 0.627 [2020-12-31 16:00:28,649][root][INFO] - wall : 0.494 0.731 0.801 [2020-12-31 16:00:28,649][root][INFO] - window : 0.430 0.646 0.769 [2020-12-31 16:00:28,649][root][INFO] - ---------------------------------------------------------------- [2020-12-31 16:00:28,649][root][INFO] - average : 0.472 0.614 0.699 [2020-12-31 16:00:28,649][root][INFO] - [2020-12-31 16:00:28,816][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 16:00:29,428][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter12400.pth [2020-12-31 16:00:40,535][root][INFO] - ===> Epoch[2481](12405/5): Loss -0.7942, Sem -0.7942, dir -0.9796, norm 0.1582 LR: 2.062e-02 Score 99.092 Data time: 0.0939, Forward time: 0.9473, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.1198 [2020-12-31 16:00:52,409][root][INFO] - ===> Epoch[2482](12410/5): Loss -0.7940, Sem -0.7940, dir -0.9796, norm 0.1578 LR: 2.059e-02 Score 99.063 Data time: 0.1061, Forward time: 0.9473, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2754 [2020-12-31 16:01:05,172][root][INFO] - ===> Epoch[2483](12415/5): Loss -0.7951, Sem -0.7951, dir -0.9803, norm 0.1584 LR: 2.055e-02 Score 99.093 Data time: 0.1136, Forward time: 0.9473, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.4535 [2020-12-31 16:01:16,629][root][INFO] - ===> Epoch[2484](12420/5): Loss -0.7946, Sem -0.7946, dir -0.9804, norm 0.1590 LR: 2.052e-02 Score 99.099 Data time: 0.1348, Forward time: 0.9473, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2035 [2020-12-31 16:01:27,779][root][INFO] - ===> Epoch[2485](12425/5): Loss -0.7909, Sem -0.7909, dir -0.9793, norm 0.1596 LR: 2.048e-02 Score 99.029 Data time: 0.0909, Forward time: 0.9472, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.1400 [2020-12-31 16:01:40,086][root][INFO] - ===> Epoch[2486](12430/5): Loss -0.7980, Sem -0.7980, dir -0.9797, norm 0.1553 LR: 2.044e-02 Score 99.109 Data time: 0.1645, Forward time: 0.9473, Backward time: 1.1883, DDP time: 0.0050, Total iter time: 2.3697 [2020-12-31 16:01:52,280][root][INFO] - ===> Epoch[2487](12435/5): Loss -0.7942, Sem -0.7942, dir -0.9799, norm 0.1584 LR: 2.041e-02 Score 99.089 Data time: 0.0918, Forward time: 0.9473, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.3477 [2020-12-31 16:02:04,346][root][INFO] - ===> Epoch[2488](12440/5): Loss -0.7907, Sem -0.7907, dir -0.9795, norm 0.1607 LR: 2.037e-02 Score 99.062 Data time: 0.0817, Forward time: 0.9473, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.3317 [2020-12-31 16:02:15,850][root][INFO] - ===> Epoch[2489](12445/5): Loss -0.7925, Sem -0.7925, dir -0.9800, norm 0.1599 LR: 2.034e-02 Score 99.077 Data time: 0.1068, Forward time: 0.9472, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2221 [2020-12-31 16:02:27,676][root][INFO] - ===> Epoch[2490](12450/5): Loss -0.7965, Sem -0.7965, dir -0.9806, norm 0.1576 LR: 2.030e-02 Score 99.114 Data time: 0.1071, Forward time: 0.9472, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2678 [2020-12-31 16:02:39,437][root][INFO] - ===> Epoch[2491](12455/5): Loss -0.7915, Sem -0.7915, dir -0.9797, norm 0.1603 LR: 2.027e-02 Score 99.061 Data time: 0.1812, Forward time: 0.9472, Backward time: 1.1884, DDP time: 0.0050, Total iter time: 2.2475 [2020-12-31 16:02:51,673][root][INFO] - ===> Epoch[2492](12460/5): Loss -0.7967, Sem -0.7967, dir -0.9796, norm 0.1570 LR: 2.023e-02 Score 99.128 Data time: 0.1503, Forward time: 0.9473, Backward time: 1.1883, DDP time: 0.0050, Total iter time: 2.3595 [2020-12-31 16:03:03,533][root][INFO] - ===> Epoch[2493](12465/5): Loss -0.8001, Sem -0.8001, dir -0.9805, norm 0.1532 LR: 2.019e-02 Score 99.087 Data time: 0.0966, Forward time: 0.9472, Backward time: 1.1883, DDP time: 0.0050, Total iter time: 2.2831 [2020-12-31 16:03:14,840][root][INFO] - ===> Epoch[2494](12470/5): Loss -0.7893, Sem -0.7893, dir -0.9795, norm 0.1621 LR: 2.016e-02 Score 99.065 Data time: 0.1272, Forward time: 0.9473, Backward time: 1.1883, DDP time: 0.0050, Total iter time: 2.1650 [2020-12-31 16:03:26,349][root][INFO] - ===> Epoch[2495](12475/5): Loss -0.7940, Sem -0.7940, dir -0.9799, norm 0.1591 LR: 2.012e-02 Score 99.102 Data time: 0.0973, Forward time: 0.9473, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.2067 [2020-12-31 16:03:37,929][root][INFO] - ===> Epoch[2496](12480/5): Loss -0.7891, Sem -0.7891, dir -0.9793, norm 0.1625 LR: 2.009e-02 Score 99.080 Data time: 0.0904, Forward time: 0.9473, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.2149 [2020-12-31 16:03:50,095][root][INFO] - ===> Epoch[2497](12485/5): Loss -0.7998, Sem -0.7998, dir -0.9808, norm 0.1554 LR: 2.005e-02 Score 99.136 Data time: 0.1469, Forward time: 0.9473, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.3416 [2020-12-31 16:04:02,282][root][INFO] - ===> Epoch[2498](12490/5): Loss -0.7976, Sem -0.7976, dir -0.9804, norm 0.1553 LR: 2.001e-02 Score 99.093 Data time: 0.0871, Forward time: 0.9473, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.3393 [2020-12-31 16:04:14,055][root][INFO] - ===> Epoch[2499](12495/5): Loss -0.7978, Sem -0.7978, dir -0.9809, norm 0.1568 LR: 1.998e-02 Score 99.118 Data time: 0.1476, Forward time: 0.9473, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.2456 [2020-12-31 16:04:26,656][root][INFO] - ===> Epoch[2500](12500/5): Loss -0.7951, Sem -0.7951, dir -0.9795, norm 0.1581 LR: 1.994e-02 Score 99.110 Data time: 0.1350, Forward time: 0.9473, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.4260 [2020-12-31 16:04:39,169][root][INFO] - ===> Epoch[2501](12505/5): Loss -0.7986, Sem -0.7986, dir -0.9810, norm 0.1554 LR: 1.991e-02 Score 99.090 Data time: 0.0879, Forward time: 0.9473, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.4041 [2020-12-31 16:04:51,675][root][INFO] - ===> Epoch[2502](12510/5): Loss -0.7863, Sem -0.7863, dir -0.9789, norm 0.1643 LR: 1.987e-02 Score 99.057 Data time: 0.1174, Forward time: 0.9473, Backward time: 1.1883, DDP time: 0.0050, Total iter time: 2.4034 [2020-12-31 16:05:03,254][root][INFO] - ===> Epoch[2503](12515/5): Loss -0.7938, Sem -0.7938, dir -0.9800, norm 0.1590 LR: 1.984e-02 Score 99.082 Data time: 0.2071, Forward time: 0.9473, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.2233 [2020-12-31 16:05:14,824][root][INFO] - ===> Epoch[2504](12520/5): Loss -0.7937, Sem -0.7937, dir -0.9796, norm 0.1590 LR: 1.980e-02 Score 99.094 Data time: 0.1686, Forward time: 0.9473, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.2108 [2020-12-31 16:05:26,812][root][INFO] - ===> Epoch[2505](12525/5): Loss -0.7901, Sem -0.7901, dir -0.9795, norm 0.1619 LR: 1.976e-02 Score 99.079 Data time: 0.1246, Forward time: 0.9474, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.3086 [2020-12-31 16:05:38,046][root][INFO] - ===> Epoch[2506](12530/5): Loss -0.7938, Sem -0.7938, dir -0.9796, norm 0.1582 LR: 1.973e-02 Score 99.074 Data time: 0.1298, Forward time: 0.9473, Backward time: 1.1880, DDP time: 0.0050, Total iter time: 2.1629 [2020-12-31 16:05:50,301][root][INFO] - ===> Epoch[2507](12535/5): Loss -0.7903, Sem -0.7903, dir -0.9803, norm 0.1637 LR: 1.969e-02 Score 99.117 Data time: 0.0946, Forward time: 0.9474, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.3475 [2020-12-31 16:06:03,195][root][INFO] - ===> Epoch[2508](12540/5): Loss -0.7930, Sem -0.7930, dir -0.9799, norm 0.1600 LR: 1.966e-02 Score 99.107 Data time: 0.1138, Forward time: 0.9474, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.4994 [2020-12-31 16:06:15,079][root][INFO] - ===> Epoch[2509](12545/5): Loss -0.7937, Sem -0.7937, dir -0.9797, norm 0.1590 LR: 1.962e-02 Score 99.094 Data time: 0.0759, Forward time: 0.9473, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.2924 [2020-12-31 16:06:26,775][root][INFO] - ===> Epoch[2510](12550/5): Loss -0.7934, Sem -0.7934, dir -0.9800, norm 0.1600 LR: 1.958e-02 Score 99.109 Data time: 0.1203, Forward time: 0.9473, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.2539 [2020-12-31 16:06:38,597][root][INFO] - ===> Epoch[2511](12555/5): Loss -0.7924, Sem -0.7924, dir -0.9798, norm 0.1606 LR: 1.955e-02 Score 99.099 Data time: 0.1346, Forward time: 0.9473, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.2683 [2020-12-31 16:06:50,503][root][INFO] - ===> Epoch[2512](12560/5): Loss -0.7928, Sem -0.7928, dir -0.9798, norm 0.1601 LR: 1.951e-02 Score 99.099 Data time: 0.1000, Forward time: 0.9473, Backward time: 1.1882, DDP time: 0.0050, Total iter time: 2.3007 [2020-12-31 16:07:01,592][root][INFO] - ===> Epoch[2513](12565/5): Loss -0.7950, Sem -0.7950, dir -0.9799, norm 0.1566 LR: 1.948e-02 Score 99.056 Data time: 0.1395, Forward time: 0.9473, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.1334 [2020-12-31 16:07:13,421][root][INFO] - ===> Epoch[2514](12570/5): Loss -0.7983, Sem -0.7983, dir -0.9806, norm 0.1565 LR: 1.944e-02 Score 99.124 Data time: 0.1562, Forward time: 0.9473, Backward time: 1.1880, DDP time: 0.0050, Total iter time: 2.2763 [2020-12-31 16:07:25,402][root][INFO] - ===> Epoch[2515](12575/5): Loss -0.7980, Sem -0.7980, dir -0.9802, norm 0.1554 LR: 1.940e-02 Score 99.102 Data time: 0.1184, Forward time: 0.9473, Backward time: 1.1880, DDP time: 0.0050, Total iter time: 2.3113 [2020-12-31 16:07:37,449][root][INFO] - ===> Epoch[2516](12580/5): Loss -0.7954, Sem -0.7954, dir -0.9800, norm 0.1575 LR: 1.937e-02 Score 99.085 Data time: 0.1057, Forward time: 0.9473, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.3322 [2020-12-31 16:07:49,175][root][INFO] - ===> Epoch[2517](12585/5): Loss -0.7910, Sem -0.7910, dir -0.9795, norm 0.1613 LR: 1.933e-02 Score 99.096 Data time: 0.1244, Forward time: 0.9473, Backward time: 1.1880, DDP time: 0.0050, Total iter time: 2.2528 [2020-12-31 16:07:59,796][root][INFO] - ===> Epoch[2518](12590/5): Loss -0.7982, Sem -0.7982, dir -0.9803, norm 0.1550 LR: 1.930e-02 Score 99.089 Data time: 0.0780, Forward time: 0.9473, Backward time: 1.1880, DDP time: 0.0050, Total iter time: 2.0439 [2020-12-31 16:08:11,491][root][INFO] - ===> Epoch[2519](12595/5): Loss -0.7939, Sem -0.7939, dir -0.9799, norm 0.1589 LR: 1.926e-02 Score 99.094 Data time: 0.1656, Forward time: 0.9473, Backward time: 1.1879, DDP time: 0.0050, Total iter time: 2.2464 [2020-12-31 16:08:23,773][root][INFO] - ===> Epoch[2520](12600/5): Loss -0.7939, Sem -0.7939, dir -0.9800, norm 0.1592 LR: 1.922e-02 Score 99.105 Data time: 0.1612, Forward time: 0.9473, Backward time: 1.1879, DDP time: 0.0050, Total iter time: 2.3650 [2020-12-31 16:08:23,776][root][INFO] - ===> Start testing [2020-12-31 16:09:03,891][root][INFO] - 68/68: Data time: 0.0022, Iter time: 0.1305 Loss 0.044 (AVG: 0.545) Score 98.747 (AVG: 89.881) mIOU 67.910 mAP 82.897 mAcc 76.097 IOU: 60.361 0.000 79.424 73.555 93.339 89.604 32.520 74.908 98.307 57.565 81.787 85.191 56.263 mAP: 80.806 8.542 93.027 85.966 98.861 98.461 58.698 94.120 99.821 81.896 93.351 93.818 90.290 mAcc: 80.530 0.000 89.264 80.451 94.603 96.801 35.221 86.141 98.953 79.844 92.116 95.665 59.667 [2020-12-31 16:09:03,892][root][INFO] - Finished test. Elapsed time: 40.1139 [2020-12-31 16:09:06,976][root][INFO] - [2020-12-31 16:09:06,976][root][INFO] - ################################################################ [2020-12-31 16:09:06,976][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 16:09:06,976][root][INFO] - ################################################################ [2020-12-31 16:09:06,976][root][INFO] - clutter : nan nan nan [2020-12-31 16:09:06,976][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 16:09:06,976][root][INFO] - board : 0.790 0.847 0.901 [2020-12-31 16:09:06,976][root][INFO] - bookcase : 0.225 0.375 0.509 [2020-12-31 16:09:06,976][root][INFO] - ceiling : 0.696 0.783 0.825 [2020-12-31 16:09:06,977][root][INFO] - chair : 0.797 0.896 0.912 [2020-12-31 16:09:06,977][root][INFO] - column : 0.172 0.253 0.430 [2020-12-31 16:09:06,977][root][INFO] - door : 0.480 0.693 0.853 [2020-12-31 16:09:06,977][root][INFO] - floor : 0.903 0.909 0.909 [2020-12-31 16:09:06,977][root][INFO] - sofa : 0.458 0.636 0.809 [2020-12-31 16:09:06,977][root][INFO] - table : 0.201 0.412 0.644 [2020-12-31 16:09:06,977][root][INFO] - wall : 0.481 0.694 0.775 [2020-12-31 16:09:06,977][root][INFO] - window : 0.459 0.661 0.770 [2020-12-31 16:09:06,977][root][INFO] - ---------------------------------------------------------------- [2020-12-31 16:09:06,977][root][INFO] - average : 0.472 0.597 0.695 [2020-12-31 16:09:06,977][root][INFO] - [2020-12-31 16:09:07,150][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 16:09:07,771][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter12600.pth [2020-12-31 16:09:18,555][root][INFO] - ===> Epoch[2521](12605/5): Loss -0.7939, Sem -0.7939, dir -0.9797, norm 0.1583 LR: 1.919e-02 Score 99.074 Data time: 0.1110, Forward time: 0.9472, Backward time: 1.1879, DDP time: 0.0050, Total iter time: 2.0532 [2020-12-31 16:09:30,442][root][INFO] - ===> Epoch[2522](12610/5): Loss -0.7950, Sem -0.7950, dir -0.9801, norm 0.1585 LR: 1.915e-02 Score 99.112 Data time: 0.1300, Forward time: 0.9472, Backward time: 1.1879, DDP time: 0.0050, Total iter time: 2.2827 [2020-12-31 16:09:41,319][root][INFO] - ===> Epoch[2523](12615/5): Loss -0.7911, Sem -0.7911, dir -0.9797, norm 0.1602 LR: 1.912e-02 Score 99.041 Data time: 0.1273, Forward time: 0.9472, Backward time: 1.1878, DDP time: 0.0050, Total iter time: 2.0834 [2020-12-31 16:09:54,799][root][INFO] - ===> Epoch[2524](12620/5): Loss -0.7926, Sem -0.7926, dir -0.9796, norm 0.1600 LR: 1.908e-02 Score 99.090 Data time: 0.1097, Forward time: 0.9472, Backward time: 1.1879, DDP time: 0.0050, Total iter time: 2.6186 [2020-12-31 16:10:07,728][root][INFO] - ===> Epoch[2525](12625/5): Loss -0.7990, Sem -0.7990, dir -0.9804, norm 0.1556 LR: 1.904e-02 Score 99.129 Data time: 0.0904, Forward time: 0.9472, Backward time: 1.1880, DDP time: 0.0050, Total iter time: 2.5106 [2020-12-31 16:10:20,259][root][INFO] - ===> Epoch[2526](12630/5): Loss -0.7939, Sem -0.7939, dir -0.9804, norm 0.1599 LR: 1.901e-02 Score 99.108 Data time: 0.1017, Forward time: 0.9472, Backward time: 1.1880, DDP time: 0.0050, Total iter time: 2.4102 [2020-12-31 16:10:31,966][root][INFO] - ===> Epoch[2527](12635/5): Loss -0.8012, Sem -0.8012, dir -0.9805, norm 0.1521 LR: 1.897e-02 Score 99.087 Data time: 0.0749, Forward time: 0.9471, Backward time: 1.1881, DDP time: 0.0050, Total iter time: 2.2691 [2020-12-31 16:10:43,729][root][INFO] - ===> Epoch[2528](12640/5): Loss -0.7940, Sem -0.7940, dir -0.9792, norm 0.1579 LR: 1.894e-02 Score 99.087 Data time: 0.1109, Forward time: 0.9472, Backward time: 1.1880, DDP time: 0.0050, Total iter time: 2.2652 [2020-12-31 16:10:55,184][root][INFO] - ===> Epoch[2529](12645/5): Loss -0.7939, Sem -0.7939, dir -0.9796, norm 0.1583 LR: 1.890e-02 Score 99.083 Data time: 0.1944, Forward time: 0.9472, Backward time: 1.1879, DDP time: 0.0050, Total iter time: 2.1935 [2020-12-31 16:11:06,202][root][INFO] - ===> Epoch[2530](12650/5): Loss -0.8024, Sem -0.8024, dir -0.9810, norm 0.1529 LR: 1.886e-02 Score 99.143 Data time: 0.0809, Forward time: 0.9472, Backward time: 1.1879, DDP time: 0.0050, Total iter time: 2.1149 [2020-12-31 16:11:18,025][root][INFO] - ===> Epoch[2531](12655/5): Loss -0.7957, Sem -0.7957, dir -0.9799, norm 0.1578 LR: 1.883e-02 Score 99.110 Data time: 0.1371, Forward time: 0.9472, Backward time: 1.1879, DDP time: 0.0050, Total iter time: 2.2766 [2020-12-31 16:11:29,803][root][INFO] - ===> Epoch[2532](12660/5): Loss -0.7986, Sem -0.7986, dir -0.9804, norm 0.1545 LR: 1.879e-02 Score 99.089 Data time: 0.1333, Forward time: 0.9472, Backward time: 1.1878, DDP time: 0.0050, Total iter time: 2.2540 [2020-12-31 16:11:42,488][root][INFO] - ===> Epoch[2533](12665/5): Loss -0.7953, Sem -0.7953, dir -0.9797, norm 0.1573 LR: 1.876e-02 Score 99.083 Data time: 0.0888, Forward time: 0.9472, Backward time: 1.1879, DDP time: 0.0050, Total iter time: 2.4527 [2020-12-31 16:11:54,548][root][INFO] - ===> Epoch[2534](12670/5): Loss -0.7891, Sem -0.7891, dir -0.9789, norm 0.1618 LR: 1.872e-02 Score 99.060 Data time: 0.1126, Forward time: 0.9472, Backward time: 1.1879, DDP time: 0.0050, Total iter time: 2.3116 [2020-12-31 16:12:06,770][root][INFO] - ===> Epoch[2535](12675/5): Loss -0.7930, Sem -0.7930, dir -0.9804, norm 0.1605 LR: 1.868e-02 Score 99.104 Data time: 0.1153, Forward time: 0.9472, Backward time: 1.1879, DDP time: 0.0050, Total iter time: 2.3535 [2020-12-31 16:12:18,180][root][INFO] - ===> Epoch[2536](12680/5): Loss -0.7950, Sem -0.7950, dir -0.9801, norm 0.1590 LR: 1.865e-02 Score 99.117 Data time: 0.1481, Forward time: 0.9472, Backward time: 1.1878, DDP time: 0.0050, Total iter time: 2.1902 [2020-12-31 16:12:29,665][root][INFO] - ===> Epoch[2537](12685/5): Loss -0.7909, Sem -0.7909, dir -0.9798, norm 0.1608 LR: 1.861e-02 Score 99.073 Data time: 0.0792, Forward time: 0.9472, Backward time: 1.1878, DDP time: 0.0050, Total iter time: 2.2103 [2020-12-31 16:12:40,959][root][INFO] - ===> Epoch[2538](12690/5): Loss -0.7883, Sem -0.7883, dir -0.9799, norm 0.1637 LR: 1.857e-02 Score 99.060 Data time: 0.0799, Forward time: 0.9472, Backward time: 1.1878, DDP time: 0.0050, Total iter time: 2.1773 [2020-12-31 16:12:53,638][root][INFO] - ===> Epoch[2539](12695/5): Loss -0.7906, Sem -0.7906, dir -0.9795, norm 0.1616 LR: 1.854e-02 Score 99.081 Data time: 0.0874, Forward time: 0.9472, Backward time: 1.1879, DDP time: 0.0050, Total iter time: 2.4453 [2020-12-31 16:13:05,319][root][INFO] - ===> Epoch[2540](12700/5): Loss -0.7942, Sem -0.7942, dir -0.9804, norm 0.1601 LR: 1.850e-02 Score 99.128 Data time: 0.1830, Forward time: 0.9472, Backward time: 1.1878, DDP time: 0.0050, Total iter time: 2.2329 [2020-12-31 16:13:17,046][root][INFO] - ===> Epoch[2541](12705/5): Loss -0.7973, Sem -0.7973, dir -0.9804, norm 0.1573 LR: 1.847e-02 Score 99.123 Data time: 0.1535, Forward time: 0.9472, Backward time: 1.1878, DDP time: 0.0050, Total iter time: 2.2508 [2020-12-31 16:13:29,313][root][INFO] - ===> Epoch[2542](12710/5): Loss -0.7960, Sem -0.7960, dir -0.9799, norm 0.1564 LR: 1.843e-02 Score 99.085 Data time: 0.1075, Forward time: 0.9472, Backward time: 1.1878, DDP time: 0.0050, Total iter time: 2.3673 [2020-12-31 16:13:41,519][root][INFO] - ===> Epoch[2543](12715/5): Loss -0.7979, Sem -0.7979, dir -0.9803, norm 0.1559 LR: 1.839e-02 Score 99.107 Data time: 0.1281, Forward time: 0.9472, Backward time: 1.1878, DDP time: 0.0050, Total iter time: 2.3476 [2020-12-31 16:13:53,440][root][INFO] - ===> Epoch[2544](12720/5): Loss -0.7935, Sem -0.7935, dir -0.9795, norm 0.1587 LR: 1.836e-02 Score 99.082 Data time: 0.0932, Forward time: 0.9473, Backward time: 1.1878, DDP time: 0.0050, Total iter time: 2.3007 [2020-12-31 16:14:06,072][root][INFO] - ===> Epoch[2545](12725/5): Loss -0.7933, Sem -0.7933, dir -0.9801, norm 0.1593 LR: 1.832e-02 Score 99.080 Data time: 0.1023, Forward time: 0.9472, Backward time: 1.1878, DDP time: 0.0050, Total iter time: 2.4372 [2020-12-31 16:14:17,868][root][INFO] - ===> Epoch[2546](12730/5): Loss -0.8000, Sem -0.8000, dir -0.9805, norm 0.1546 LR: 1.828e-02 Score 99.131 Data time: 0.0969, Forward time: 0.9472, Backward time: 1.1878, DDP time: 0.0050, Total iter time: 2.2623 [2020-12-31 16:14:30,511][root][INFO] - ===> Epoch[2547](12735/5): Loss -0.8030, Sem -0.8030, dir -0.9812, norm 0.1526 LR: 1.825e-02 Score 99.144 Data time: 0.1280, Forward time: 0.9473, Backward time: 1.1878, DDP time: 0.0050, Total iter time: 2.4283 [2020-12-31 16:14:41,813][root][INFO] - ===> Epoch[2548](12740/5): Loss -0.7951, Sem -0.7951, dir -0.9797, norm 0.1576 LR: 1.821e-02 Score 99.091 Data time: 0.1146, Forward time: 0.9473, Backward time: 1.1878, DDP time: 0.0050, Total iter time: 2.1638 [2020-12-31 16:14:53,834][root][INFO] - ===> Epoch[2549](12745/5): Loss -0.7945, Sem -0.7945, dir -0.9800, norm 0.1572 LR: 1.818e-02 Score 99.056 Data time: 0.1057, Forward time: 0.9473, Backward time: 1.1878, DDP time: 0.0050, Total iter time: 2.3120 [2020-12-31 16:15:06,105][root][INFO] - ===> Epoch[2550](12750/5): Loss -0.7958, Sem -0.7958, dir -0.9801, norm 0.1568 LR: 1.814e-02 Score 99.077 Data time: 0.1432, Forward time: 0.9473, Backward time: 1.1878, DDP time: 0.0050, Total iter time: 2.3605 [2020-12-31 16:15:17,410][root][INFO] - ===> Epoch[2551](12755/5): Loss -0.8034, Sem -0.8034, dir -0.9804, norm 0.1501 LR: 1.810e-02 Score 99.098 Data time: 0.1017, Forward time: 0.9473, Backward time: 1.1877, DDP time: 0.0050, Total iter time: 2.1906 [2020-12-31 16:15:29,890][root][INFO] - ===> Epoch[2552](12760/5): Loss -0.7930, Sem -0.7930, dir -0.9798, norm 0.1610 LR: 1.807e-02 Score 99.138 Data time: 0.1468, Forward time: 0.9473, Backward time: 1.1877, DDP time: 0.0050, Total iter time: 2.4013 [2020-12-31 16:15:41,027][root][INFO] - ===> Epoch[2553](12765/5): Loss -0.7940, Sem -0.7940, dir -0.9796, norm 0.1586 LR: 1.803e-02 Score 99.095 Data time: 0.0955, Forward time: 0.9473, Backward time: 1.1877, DDP time: 0.0050, Total iter time: 2.1428 [2020-12-31 16:15:53,091][root][INFO] - ===> Epoch[2554](12770/5): Loss -0.7969, Sem -0.7969, dir -0.9804, norm 0.1566 LR: 1.799e-02 Score 99.106 Data time: 0.0968, Forward time: 0.9473, Backward time: 1.1877, DDP time: 0.0050, Total iter time: 2.3228 [2020-12-31 16:16:05,284][root][INFO] - ===> Epoch[2555](12775/5): Loss -0.7946, Sem -0.7946, dir -0.9800, norm 0.1586 LR: 1.796e-02 Score 99.104 Data time: 0.1058, Forward time: 0.9473, Backward time: 1.1877, DDP time: 0.0050, Total iter time: 2.3443 [2020-12-31 16:16:17,000][root][INFO] - ===> Epoch[2556](12780/5): Loss -0.7996, Sem -0.7996, dir -0.9801, norm 0.1537 LR: 1.792e-02 Score 99.103 Data time: 0.0742, Forward time: 0.9472, Backward time: 1.1877, DDP time: 0.0050, Total iter time: 2.2664 [2020-12-31 16:16:29,044][root][INFO] - ===> Epoch[2557](12785/5): Loss -0.7982, Sem -0.7982, dir -0.9805, norm 0.1561 LR: 1.789e-02 Score 99.111 Data time: 0.1100, Forward time: 0.9472, Backward time: 1.1877, DDP time: 0.0050, Total iter time: 2.3074 [2020-12-31 16:16:41,166][root][INFO] - ===> Epoch[2558](12790/5): Loss -0.7973, Sem -0.7973, dir -0.9796, norm 0.1552 LR: 1.785e-02 Score 99.090 Data time: 0.1687, Forward time: 0.9472, Backward time: 1.1877, DDP time: 0.0050, Total iter time: 2.3262 [2020-12-31 16:16:52,931][root][INFO] - ===> Epoch[2559](12795/5): Loss -0.8017, Sem -0.8017, dir -0.9808, norm 0.1537 LR: 1.781e-02 Score 99.153 Data time: 0.1344, Forward time: 0.9472, Backward time: 1.1877, DDP time: 0.0050, Total iter time: 2.2814 [2020-12-31 16:17:04,515][root][INFO] - ===> Epoch[2560](12800/5): Loss -0.7992, Sem -0.7992, dir -0.9803, norm 0.1543 LR: 1.778e-02 Score 99.099 Data time: 0.1235, Forward time: 0.9472, Backward time: 1.1877, DDP time: 0.0050, Total iter time: 2.2243 [2020-12-31 16:17:04,518][root][INFO] - ===> Start testing [2020-12-31 16:17:45,685][root][INFO] - 68/68: Data time: 0.0023, Iter time: 0.1190 Loss 0.040 (AVG: 0.536) Score 98.866 (AVG: 89.963) mIOU 68.449 mAP 83.053 mAcc 75.929 IOU: 60.497 0.000 79.637 73.928 93.630 91.100 30.056 76.888 98.253 62.815 81.851 85.055 56.131 mAP: 81.372 8.305 92.900 86.524 98.862 98.387 58.064 95.303 99.823 81.462 93.334 94.068 91.290 mAcc: 80.446 0.000 89.912 80.986 94.917 96.488 32.697 88.563 98.938 76.697 90.986 95.476 60.973 [2020-12-31 16:17:45,687][root][INFO] - Finished test. Elapsed time: 41.1664 [2020-12-31 16:17:48,900][root][INFO] - [2020-12-31 16:17:48,900][root][INFO] - ################################################################ [2020-12-31 16:17:48,900][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 16:17:48,900][root][INFO] - ################################################################ [2020-12-31 16:17:48,901][root][INFO] - clutter : nan nan nan [2020-12-31 16:17:48,901][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 16:17:48,901][root][INFO] - board : 0.792 0.869 0.898 [2020-12-31 16:17:48,901][root][INFO] - bookcase : 0.205 0.350 0.510 [2020-12-31 16:17:48,901][root][INFO] - ceiling : 0.594 0.717 0.759 [2020-12-31 16:17:48,901][root][INFO] - chair : 0.793 0.888 0.903 [2020-12-31 16:17:48,901][root][INFO] - column : 0.163 0.236 0.360 [2020-12-31 16:17:48,901][root][INFO] - door : 0.615 0.816 0.921 [2020-12-31 16:17:48,901][root][INFO] - floor : 0.905 0.967 0.975 [2020-12-31 16:17:48,901][root][INFO] - sofa : 0.404 0.636 0.809 [2020-12-31 16:17:48,901][root][INFO] - table : 0.219 0.407 0.635 [2020-12-31 16:17:48,901][root][INFO] - wall : 0.480 0.704 0.772 [2020-12-31 16:17:48,901][root][INFO] - window : 0.510 0.744 0.761 [2020-12-31 16:17:48,901][root][INFO] - ---------------------------------------------------------------- [2020-12-31 16:17:48,901][root][INFO] - average : 0.473 0.611 0.692 [2020-12-31 16:17:48,901][root][INFO] - [2020-12-31 16:17:49,066][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 16:17:49,663][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter12800.pth [2020-12-31 16:18:00,714][root][INFO] - ===> Epoch[2561](12805/5): Loss -0.7983, Sem -0.7983, dir -0.9800, norm 0.1553 LR: 1.774e-02 Score 99.116 Data time: 0.1560, Forward time: 0.9472, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.1066 [2020-12-31 16:18:12,411][root][INFO] - ===> Epoch[2562](12810/5): Loss -0.8026, Sem -0.8026, dir -0.9808, norm 0.1522 LR: 1.770e-02 Score 99.123 Data time: 0.1545, Forward time: 0.9473, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.2588 [2020-12-31 16:18:23,527][root][INFO] - ===> Epoch[2563](12815/5): Loss -0.7970, Sem -0.7970, dir -0.9801, norm 0.1561 LR: 1.767e-02 Score 99.096 Data time: 0.0926, Forward time: 0.9472, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.1375 [2020-12-31 16:18:35,136][root][INFO] - ===> Epoch[2564](12820/5): Loss -0.8006, Sem -0.8006, dir -0.9800, norm 0.1528 LR: 1.763e-02 Score 99.107 Data time: 0.0867, Forward time: 0.9472, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.2453 [2020-12-31 16:18:47,360][root][INFO] - ===> Epoch[2565](12825/5): Loss -0.8006, Sem -0.8006, dir -0.9807, norm 0.1543 LR: 1.759e-02 Score 99.126 Data time: 0.1531, Forward time: 0.9472, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.3514 [2020-12-31 16:18:59,503][root][INFO] - ===> Epoch[2566](12830/5): Loss -0.7937, Sem -0.7937, dir -0.9802, norm 0.1595 LR: 1.756e-02 Score 99.102 Data time: 0.1121, Forward time: 0.9472, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.3346 [2020-12-31 16:19:11,582][root][INFO] - ===> Epoch[2567](12835/5): Loss -0.7972, Sem -0.7972, dir -0.9804, norm 0.1564 LR: 1.752e-02 Score 99.096 Data time: 0.1218, Forward time: 0.9472, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.3326 [2020-12-31 16:19:24,392][root][INFO] - ===> Epoch[2568](12840/5): Loss -0.8030, Sem -0.8030, dir -0.9804, norm 0.1517 LR: 1.749e-02 Score 99.133 Data time: 0.2248, Forward time: 0.9472, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.4765 [2020-12-31 16:19:36,060][root][INFO] - ===> Epoch[2569](12845/5): Loss -0.7989, Sem -0.7989, dir -0.9800, norm 0.1549 LR: 1.745e-02 Score 99.126 Data time: 0.0990, Forward time: 0.9472, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.2447 [2020-12-31 16:19:47,332][root][INFO] - ===> Epoch[2570](12850/5): Loss -0.8012, Sem -0.8012, dir -0.9800, norm 0.1523 LR: 1.741e-02 Score 99.104 Data time: 0.1404, Forward time: 0.9472, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.1497 [2020-12-31 16:19:59,169][root][INFO] - ===> Epoch[2571](12855/5): Loss -0.7964, Sem -0.7964, dir -0.9802, norm 0.1575 LR: 1.738e-02 Score 99.113 Data time: 0.1225, Forward time: 0.9472, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.2824 [2020-12-31 16:20:10,927][root][INFO] - ===> Epoch[2572](12860/5): Loss -0.8035, Sem -0.8035, dir -0.9805, norm 0.1509 LR: 1.734e-02 Score 99.135 Data time: 0.1107, Forward time: 0.9472, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.2534 [2020-12-31 16:20:22,666][root][INFO] - ===> Epoch[2573](12865/5): Loss -0.7942, Sem -0.7942, dir -0.9799, norm 0.1581 LR: 1.730e-02 Score 99.075 Data time: 0.1066, Forward time: 0.9472, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.2517 [2020-12-31 16:20:35,171][root][INFO] - ===> Epoch[2574](12870/5): Loss -0.7997, Sem -0.7997, dir -0.9804, norm 0.1546 LR: 1.727e-02 Score 99.122 Data time: 0.0871, Forward time: 0.9472, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.4088 [2020-12-31 16:20:47,593][root][INFO] - ===> Epoch[2575](12875/5): Loss -0.7985, Sem -0.7985, dir -0.9804, norm 0.1557 LR: 1.723e-02 Score 99.113 Data time: 0.0838, Forward time: 0.9472, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.3835 [2020-12-31 16:21:00,223][root][INFO] - ===> Epoch[2576](12880/5): Loss -0.7877, Sem -0.7877, dir -0.9789, norm 0.1628 LR: 1.719e-02 Score 99.055 Data time: 0.0853, Forward time: 0.9472, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.4382 [2020-12-31 16:21:11,909][root][INFO] - ===> Epoch[2577](12885/5): Loss -0.8027, Sem -0.8027, dir -0.9807, norm 0.1521 LR: 1.716e-02 Score 99.114 Data time: 0.1292, Forward time: 0.9472, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.2406 [2020-12-31 16:21:23,534][root][INFO] - ===> Epoch[2578](12890/5): Loss -0.7974, Sem -0.7974, dir -0.9806, norm 0.1568 LR: 1.712e-02 Score 99.114 Data time: 0.1070, Forward time: 0.9471, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.2360 [2020-12-31 16:21:35,281][root][INFO] - ===> Epoch[2579](12895/5): Loss -0.8005, Sem -0.8005, dir -0.9803, norm 0.1532 LR: 1.708e-02 Score 99.108 Data time: 0.0765, Forward time: 0.9471, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.2676 [2020-12-31 16:21:46,908][root][INFO] - ===> Epoch[2580](12900/5): Loss -0.8003, Sem -0.8003, dir -0.9806, norm 0.1538 LR: 1.705e-02 Score 99.111 Data time: 0.1959, Forward time: 0.9471, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.2330 [2020-12-31 16:21:58,424][root][INFO] - ===> Epoch[2581](12905/5): Loss -0.8022, Sem -0.8022, dir -0.9808, norm 0.1525 LR: 1.701e-02 Score 99.130 Data time: 0.0836, Forward time: 0.9471, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.2207 [2020-12-31 16:22:10,058][root][INFO] - ===> Epoch[2582](12910/5): Loss -0.8024, Sem -0.8024, dir -0.9803, norm 0.1512 LR: 1.698e-02 Score 99.104 Data time: 0.1238, Forward time: 0.9471, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.2368 [2020-12-31 16:22:22,636][root][INFO] - ===> Epoch[2583](12915/5): Loss -0.7952, Sem -0.7952, dir -0.9802, norm 0.1584 LR: 1.694e-02 Score 99.106 Data time: 0.1031, Forward time: 0.9471, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.4237 [2020-12-31 16:22:34,564][root][INFO] - ===> Epoch[2584](12920/5): Loss -0.7971, Sem -0.7971, dir -0.9797, norm 0.1554 LR: 1.690e-02 Score 99.086 Data time: 0.0973, Forward time: 0.9471, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.2979 [2020-12-31 16:22:46,082][root][INFO] - ===> Epoch[2585](12925/5): Loss -0.8043, Sem -0.8043, dir -0.9812, norm 0.1513 LR: 1.687e-02 Score 99.135 Data time: 0.1819, Forward time: 0.9470, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.2113 [2020-12-31 16:22:57,861][root][INFO] - ===> Epoch[2586](12930/5): Loss -0.7949, Sem -0.7949, dir -0.9799, norm 0.1581 LR: 1.683e-02 Score 99.106 Data time: 0.1056, Forward time: 0.9471, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.2660 [2020-12-31 16:23:10,188][root][INFO] - ===> Epoch[2587](12935/5): Loss -0.8064, Sem -0.8064, dir -0.9809, norm 0.1493 LR: 1.679e-02 Score 99.152 Data time: 0.1209, Forward time: 0.9471, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.3701 [2020-12-31 16:23:22,444][root][INFO] - ===> Epoch[2588](12940/5): Loss -0.7959, Sem -0.7959, dir -0.9801, norm 0.1574 LR: 1.676e-02 Score 99.097 Data time: 0.0974, Forward time: 0.9470, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.3760 [2020-12-31 16:23:33,829][root][INFO] - ===> Epoch[2589](12945/5): Loss -0.7964, Sem -0.7964, dir -0.9799, norm 0.1561 LR: 1.672e-02 Score 99.082 Data time: 0.0873, Forward time: 0.9470, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.1908 [2020-12-31 16:23:46,432][root][INFO] - ===> Epoch[2590](12950/5): Loss -0.8037, Sem -0.8037, dir -0.9804, norm 0.1513 LR: 1.668e-02 Score 99.141 Data time: 0.1816, Forward time: 0.9470, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.4404 [2020-12-31 16:23:58,263][root][INFO] - ===> Epoch[2591](12955/5): Loss -0.8003, Sem -0.8003, dir -0.9806, norm 0.1538 LR: 1.665e-02 Score 99.109 Data time: 0.1445, Forward time: 0.9470, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.2784 [2020-12-31 16:24:10,944][root][INFO] - ===> Epoch[2592](12960/5): Loss -0.7983, Sem -0.7983, dir -0.9796, norm 0.1542 LR: 1.661e-02 Score 99.090 Data time: 0.1061, Forward time: 0.9470, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.4487 [2020-12-31 16:24:23,348][root][INFO] - ===> Epoch[2593](12965/5): Loss -0.7980, Sem -0.7980, dir -0.9806, norm 0.1561 LR: 1.657e-02 Score 99.109 Data time: 0.0971, Forward time: 0.9470, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.3863 [2020-12-31 16:24:35,199][root][INFO] - ===> Epoch[2594](12970/5): Loss -0.8055, Sem -0.8055, dir -0.9808, norm 0.1490 LR: 1.654e-02 Score 99.118 Data time: 0.0931, Forward time: 0.9470, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.2766 [2020-12-31 16:24:48,032][root][INFO] - ===> Epoch[2595](12975/5): Loss -0.8043, Sem -0.8043, dir -0.9810, norm 0.1507 LR: 1.650e-02 Score 99.122 Data time: 0.0998, Forward time: 0.9470, Backward time: 1.1877, DDP time: 0.0050, Total iter time: 2.4806 [2020-12-31 16:25:00,442][root][INFO] - ===> Epoch[2596](12980/5): Loss -0.7979, Sem -0.7979, dir -0.9804, norm 0.1567 LR: 1.646e-02 Score 99.130 Data time: 0.1410, Forward time: 0.9471, Backward time: 1.1877, DDP time: 0.0050, Total iter time: 2.3817 [2020-12-31 16:25:12,181][root][INFO] - ===> Epoch[2597](12985/5): Loss -0.8011, Sem -0.8011, dir -0.9802, norm 0.1530 LR: 1.643e-02 Score 99.128 Data time: 0.1055, Forward time: 0.9471, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.2439 [2020-12-31 16:25:23,878][root][INFO] - ===> Epoch[2598](12990/5): Loss -0.7945, Sem -0.7945, dir -0.9798, norm 0.1581 LR: 1.639e-02 Score 99.085 Data time: 0.0959, Forward time: 0.9471, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.2617 [2020-12-31 16:25:35,228][root][INFO] - ===> Epoch[2599](12995/5): Loss -0.8034, Sem -0.8034, dir -0.9809, norm 0.1520 LR: 1.635e-02 Score 99.136 Data time: 0.0982, Forward time: 0.9470, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.1756 [2020-12-31 16:25:46,790][root][INFO] - ===> Epoch[2600](13000/5): Loss -0.7991, Sem -0.7991, dir -0.9799, norm 0.1531 LR: 1.632e-02 Score 99.071 Data time: 0.0875, Forward time: 0.9470, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.2095 [2020-12-31 16:25:46,794][root][INFO] - ===> Start testing [2020-12-31 16:26:27,719][root][INFO] - 68/68: Data time: 0.0024, Iter time: 0.1261 Loss 1.035 (AVG: 0.538) Score 85.386 (AVG: 89.950) mIOU 68.550 mAP 83.317 mAcc 76.042 IOU: 60.401 0.000 80.282 72.951 94.131 90.556 30.950 78.159 98.227 63.093 81.139 85.074 56.186 mAP: 80.922 12.953 92.889 85.646 98.980 98.474 57.171 95.324 99.829 82.601 93.125 94.265 90.935 mAcc: 80.894 0.000 89.464 79.480 95.266 96.798 34.223 88.928 98.861 77.753 90.396 95.643 60.834 [2020-12-31 16:26:27,720][root][INFO] - Finished test. Elapsed time: 40.9239 [2020-12-31 16:26:31,240][root][INFO] - [2020-12-31 16:26:31,240][root][INFO] - ################################################################ [2020-12-31 16:26:31,240][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 16:26:31,241][root][INFO] - ################################################################ [2020-12-31 16:26:31,241][root][INFO] - clutter : nan nan nan [2020-12-31 16:26:31,241][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 16:26:31,241][root][INFO] - board : 0.782 0.844 0.894 [2020-12-31 16:26:31,241][root][INFO] - bookcase : 0.199 0.324 0.496 [2020-12-31 16:26:31,241][root][INFO] - ceiling : 0.756 0.874 0.874 [2020-12-31 16:26:31,241][root][INFO] - chair : 0.807 0.896 0.911 [2020-12-31 16:26:31,241][root][INFO] - column : 0.180 0.293 0.407 [2020-12-31 16:26:31,241][root][INFO] - door : 0.581 0.776 0.893 [2020-12-31 16:26:31,241][root][INFO] - floor : 0.961 0.974 0.989 [2020-12-31 16:26:31,241][root][INFO] - sofa : 0.451 0.636 0.809 [2020-12-31 16:26:31,241][root][INFO] - table : 0.194 0.382 0.626 [2020-12-31 16:26:31,241][root][INFO] - wall : 0.471 0.695 0.770 [2020-12-31 16:26:31,241][root][INFO] - window : 0.481 0.730 0.779 [2020-12-31 16:26:31,241][root][INFO] - ---------------------------------------------------------------- [2020-12-31 16:26:31,241][root][INFO] - average : 0.489 0.619 0.704 [2020-12-31 16:26:31,241][root][INFO] - [2020-12-31 16:26:31,404][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 16:26:31,969][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter13000.pth [2020-12-31 16:26:43,228][root][INFO] - ===> Epoch[2601](13005/5): Loss -0.7977, Sem -0.7977, dir -0.9801, norm 0.1559 LR: 1.628e-02 Score 99.106 Data time: 0.1019, Forward time: 0.9470, Backward time: 1.1876, DDP time: 0.0050, Total iter time: 2.1419 [2020-12-31 16:26:54,938][root][INFO] - ===> Epoch[2602](13010/5): Loss -0.8030, Sem -0.8030, dir -0.9806, norm 0.1520 LR: 1.624e-02 Score 99.132 Data time: 0.1159, Forward time: 0.9470, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.2446 [2020-12-31 16:27:06,727][root][INFO] - ===> Epoch[2603](13015/5): Loss -0.7963, Sem -0.7963, dir -0.9799, norm 0.1566 LR: 1.621e-02 Score 99.096 Data time: 0.1865, Forward time: 0.9470, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.2671 [2020-12-31 16:27:19,105][root][INFO] - ===> Epoch[2604](13020/5): Loss -0.8063, Sem -0.8063, dir -0.9813, norm 0.1501 LR: 1.617e-02 Score 99.160 Data time: 0.1307, Forward time: 0.9470, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.3807 [2020-12-31 16:27:30,821][root][INFO] - ===> Epoch[2605](13025/5): Loss -0.7978, Sem -0.7978, dir -0.9802, norm 0.1562 LR: 1.613e-02 Score 99.111 Data time: 0.1630, Forward time: 0.9471, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.2515 [2020-12-31 16:27:41,997][root][INFO] - ===> Epoch[2606](13030/5): Loss -0.7946, Sem -0.7946, dir -0.9800, norm 0.1581 LR: 1.610e-02 Score 99.091 Data time: 0.1011, Forward time: 0.9471, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.1381 [2020-12-31 16:27:53,865][root][INFO] - ===> Epoch[2607](13035/5): Loss -0.7959, Sem -0.7959, dir -0.9800, norm 0.1573 LR: 1.606e-02 Score 99.104 Data time: 0.1928, Forward time: 0.9471, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.2825 [2020-12-31 16:28:05,433][root][INFO] - ===> Epoch[2608](13040/5): Loss -0.7954, Sem -0.7954, dir -0.9800, norm 0.1574 LR: 1.602e-02 Score 99.083 Data time: 0.1593, Forward time: 0.9471, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.2209 [2020-12-31 16:28:17,639][root][INFO] - ===> Epoch[2609](13045/5): Loss -0.7990, Sem -0.7990, dir -0.9804, norm 0.1552 LR: 1.599e-02 Score 99.120 Data time: 0.1138, Forward time: 0.9471, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.3459 [2020-12-31 16:28:30,132][root][INFO] - ===> Epoch[2610](13050/5): Loss -0.8033, Sem -0.8033, dir -0.9811, norm 0.1520 LR: 1.595e-02 Score 99.137 Data time: 0.1130, Forward time: 0.9471, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.4019 [2020-12-31 16:28:41,481][root][INFO] - ===> Epoch[2611](13055/5): Loss -0.7999, Sem -0.7999, dir -0.9802, norm 0.1544 LR: 1.591e-02 Score 99.125 Data time: 0.1831, Forward time: 0.9471, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.1758 [2020-12-31 16:28:54,288][root][INFO] - ===> Epoch[2612](13060/5): Loss -0.7933, Sem -0.7933, dir -0.9801, norm 0.1600 LR: 1.588e-02 Score 99.105 Data time: 0.1107, Forward time: 0.9471, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.4727 [2020-12-31 16:29:06,005][root][INFO] - ===> Epoch[2613](13065/5): Loss -0.7989, Sem -0.7989, dir -0.9803, norm 0.1545 LR: 1.584e-02 Score 99.097 Data time: 0.0737, Forward time: 0.9471, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.2534 [2020-12-31 16:29:18,223][root][INFO] - ===> Epoch[2614](13070/5): Loss -0.8030, Sem -0.8030, dir -0.9807, norm 0.1525 LR: 1.580e-02 Score 99.142 Data time: 0.1116, Forward time: 0.9471, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.3504 [2020-12-31 16:29:29,886][root][INFO] - ===> Epoch[2615](13075/5): Loss -0.8001, Sem -0.8001, dir -0.9800, norm 0.1530 LR: 1.576e-02 Score 99.094 Data time: 0.1316, Forward time: 0.9471, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.2409 [2020-12-31 16:29:42,246][root][INFO] - ===> Epoch[2616](13080/5): Loss -0.8009, Sem -0.8009, dir -0.9806, norm 0.1536 LR: 1.573e-02 Score 99.130 Data time: 0.0812, Forward time: 0.9471, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.3918 [2020-12-31 16:29:53,695][root][INFO] - ===> Epoch[2617](13085/5): Loss -0.8005, Sem -0.8005, dir -0.9804, norm 0.1536 LR: 1.569e-02 Score 99.127 Data time: 0.0863, Forward time: 0.9470, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.1978 [2020-12-31 16:30:05,883][root][INFO] - ===> Epoch[2618](13090/5): Loss -0.8037, Sem -0.8037, dir -0.9807, norm 0.1516 LR: 1.565e-02 Score 99.139 Data time: 0.0877, Forward time: 0.9470, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.3517 [2020-12-31 16:30:18,919][root][INFO] - ===> Epoch[2619](13095/5): Loss -0.7992, Sem -0.7992, dir -0.9804, norm 0.1540 LR: 1.562e-02 Score 99.086 Data time: 0.1478, Forward time: 0.9470, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.5080 [2020-12-31 16:30:31,519][root][INFO] - ===> Epoch[2620](13100/5): Loss -0.8008, Sem -0.8008, dir -0.9808, norm 0.1532 LR: 1.558e-02 Score 99.102 Data time: 0.0999, Forward time: 0.9470, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.4290 [2020-12-31 16:30:44,063][root][INFO] - ===> Epoch[2621](13105/5): Loss -0.8015, Sem -0.8015, dir -0.9804, norm 0.1526 LR: 1.554e-02 Score 99.120 Data time: 0.1044, Forward time: 0.9471, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.4187 [2020-12-31 16:30:55,656][root][INFO] - ===> Epoch[2622](13110/5): Loss -0.7973, Sem -0.7973, dir -0.9806, norm 0.1571 LR: 1.551e-02 Score 99.127 Data time: 0.1165, Forward time: 0.9470, Backward time: 1.1875, DDP time: 0.0050, Total iter time: 2.2234 [2020-12-31 16:31:07,663][root][INFO] - ===> Epoch[2623](13115/5): Loss -0.7958, Sem -0.7958, dir -0.9795, norm 0.1559 LR: 1.547e-02 Score 99.080 Data time: 0.1262, Forward time: 0.9471, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.3137 [2020-12-31 16:31:19,048][root][INFO] - ===> Epoch[2624](13120/5): Loss -0.7981, Sem -0.7981, dir -0.9804, norm 0.1563 LR: 1.543e-02 Score 99.127 Data time: 0.1589, Forward time: 0.9471, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.1829 [2020-12-31 16:31:30,547][root][INFO] - ===> Epoch[2625](13125/5): Loss -0.7964, Sem -0.7964, dir -0.9797, norm 0.1552 LR: 1.540e-02 Score 99.065 Data time: 0.1096, Forward time: 0.9471, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.2046 [2020-12-31 16:31:43,068][root][INFO] - ===> Epoch[2626](13130/5): Loss -0.7948, Sem -0.7948, dir -0.9803, norm 0.1585 LR: 1.536e-02 Score 99.096 Data time: 0.0984, Forward time: 0.9471, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.4196 [2020-12-31 16:31:54,300][root][INFO] - ===> Epoch[2627](13135/5): Loss -0.7999, Sem -0.7999, dir -0.9804, norm 0.1538 LR: 1.532e-02 Score 99.100 Data time: 0.0921, Forward time: 0.9470, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.1605 [2020-12-31 16:32:06,011][root][INFO] - ===> Epoch[2628](13140/5): Loss -0.7987, Sem -0.7987, dir -0.9802, norm 0.1553 LR: 1.528e-02 Score 99.121 Data time: 0.1041, Forward time: 0.9470, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.2563 [2020-12-31 16:32:18,125][root][INFO] - ===> Epoch[2629](13145/5): Loss -0.7983, Sem -0.7983, dir -0.9807, norm 0.1559 LR: 1.525e-02 Score 99.110 Data time: 0.0908, Forward time: 0.9470, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.3373 [2020-12-31 16:32:30,375][root][INFO] - ===> Epoch[2630](13150/5): Loss -0.8034, Sem -0.8034, dir -0.9800, norm 0.1503 LR: 1.521e-02 Score 99.117 Data time: 0.1726, Forward time: 0.9471, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.3545 [2020-12-31 16:32:42,405][root][INFO] - ===> Epoch[2631](13155/5): Loss -0.7944, Sem -0.7944, dir -0.9799, norm 0.1576 LR: 1.517e-02 Score 99.072 Data time: 0.1149, Forward time: 0.9471, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.3169 [2020-12-31 16:32:54,126][root][INFO] - ===> Epoch[2632](13160/5): Loss -0.8046, Sem -0.8046, dir -0.9809, norm 0.1507 LR: 1.514e-02 Score 99.137 Data time: 0.1001, Forward time: 0.9471, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.2500 [2020-12-31 16:33:06,526][root][INFO] - ===> Epoch[2633](13165/5): Loss -0.7972, Sem -0.7972, dir -0.9796, norm 0.1547 LR: 1.510e-02 Score 99.067 Data time: 0.1076, Forward time: 0.9471, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.3710 [2020-12-31 16:33:18,034][root][INFO] - ===> Epoch[2634](13170/5): Loss -0.7992, Sem -0.7992, dir -0.9805, norm 0.1543 LR: 1.506e-02 Score 99.112 Data time: 0.1109, Forward time: 0.9471, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.2069 [2020-12-31 16:33:29,787][root][INFO] - ===> Epoch[2635](13175/5): Loss -0.7961, Sem -0.7961, dir -0.9800, norm 0.1570 LR: 1.503e-02 Score 99.088 Data time: 0.0815, Forward time: 0.9471, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.2560 [2020-12-31 16:33:41,012][root][INFO] - ===> Epoch[2636](13180/5): Loss -0.8035, Sem -0.8035, dir -0.9812, norm 0.1526 LR: 1.499e-02 Score 99.153 Data time: 0.0934, Forward time: 0.9471, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.1623 [2020-12-31 16:33:53,753][root][INFO] - ===> Epoch[2637](13185/5): Loss -0.7965, Sem -0.7965, dir -0.9808, norm 0.1576 LR: 1.495e-02 Score 99.110 Data time: 0.1581, Forward time: 0.9471, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.4520 [2020-12-31 16:34:06,507][root][INFO] - ===> Epoch[2638](13190/5): Loss -0.8070, Sem -0.8070, dir -0.9816, norm 0.1493 LR: 1.491e-02 Score 99.146 Data time: 0.1159, Forward time: 0.9471, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.4682 [2020-12-31 16:34:18,714][root][INFO] - ===> Epoch[2639](13195/5): Loss -0.7992, Sem -0.7992, dir -0.9804, norm 0.1545 LR: 1.488e-02 Score 99.102 Data time: 0.0636, Forward time: 0.9470, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.3731 [2020-12-31 16:34:30,007][root][INFO] - ===> Epoch[2640](13200/5): Loss -0.7991, Sem -0.7991, dir -0.9807, norm 0.1542 LR: 1.484e-02 Score 99.092 Data time: 0.0905, Forward time: 0.9470, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.1753 [2020-12-31 16:34:30,010][root][INFO] - ===> Start testing [2020-12-31 16:35:07,594][root][INFO] - 68/68: Data time: 0.0022, Iter time: 0.1066 Loss 0.029 (AVG: 0.538) Score 99.079 (AVG: 89.854) mIOU 68.452 mAP 83.035 mAcc 75.783 IOU: 59.813 0.000 80.710 73.150 93.506 90.782 30.413 77.268 98.330 63.554 80.649 85.156 56.553 mAP: 80.951 10.772 92.731 86.128 98.801 98.393 56.652 94.387 99.822 82.741 92.905 94.006 91.162 mAcc: 80.683 0.000 89.318 80.049 94.822 96.708 33.539 86.879 99.067 77.743 89.814 95.697 60.860 [2020-12-31 16:35:07,595][root][INFO] - Finished test. Elapsed time: 37.5833 [2020-12-31 16:35:10,921][root][INFO] - [2020-12-31 16:35:10,921][root][INFO] - ################################################################ [2020-12-31 16:35:10,921][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 16:35:10,921][root][INFO] - ################################################################ [2020-12-31 16:35:10,921][root][INFO] - clutter : nan nan nan [2020-12-31 16:35:10,922][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 16:35:10,922][root][INFO] - board : 0.793 0.874 0.903 [2020-12-31 16:35:10,922][root][INFO] - bookcase : 0.214 0.351 0.507 [2020-12-31 16:35:10,922][root][INFO] - ceiling : 0.633 0.744 0.812 [2020-12-31 16:35:10,922][root][INFO] - chair : 0.810 0.906 0.910 [2020-12-31 16:35:10,922][root][INFO] - column : 0.169 0.252 0.374 [2020-12-31 16:35:10,922][root][INFO] - door : 0.588 0.803 0.918 [2020-12-31 16:35:10,922][root][INFO] - floor : 0.956 0.992 0.992 [2020-12-31 16:35:10,922][root][INFO] - sofa : 0.421 0.636 0.809 [2020-12-31 16:35:10,922][root][INFO] - table : 0.202 0.410 0.625 [2020-12-31 16:35:10,922][root][INFO] - wall : 0.488 0.713 0.810 [2020-12-31 16:35:10,922][root][INFO] - window : 0.490 0.729 0.779 [2020-12-31 16:35:10,922][root][INFO] - ---------------------------------------------------------------- [2020-12-31 16:35:10,922][root][INFO] - average : 0.481 0.617 0.703 [2020-12-31 16:35:10,922][root][INFO] - [2020-12-31 16:35:11,102][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 16:35:11,659][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter13200.pth [2020-12-31 16:35:22,804][root][INFO] - ===> Epoch[2641](13205/5): Loss -0.7975, Sem -0.7975, dir -0.9803, norm 0.1564 LR: 1.480e-02 Score 99.112 Data time: 0.1211, Forward time: 0.9470, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.1219 [2020-12-31 16:35:34,634][root][INFO] - ===> Epoch[2642](13210/5): Loss -0.8015, Sem -0.8015, dir -0.9808, norm 0.1528 LR: 1.477e-02 Score 99.110 Data time: 0.1036, Forward time: 0.9470, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.2881 [2020-12-31 16:35:48,180][root][INFO] - ===> Epoch[2643](13215/5): Loss -0.7993, Sem -0.7993, dir -0.9802, norm 0.1551 LR: 1.473e-02 Score 99.134 Data time: 0.1572, Forward time: 0.9471, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.6138 [2020-12-31 16:35:59,823][root][INFO] - ===> Epoch[2644](13220/5): Loss -0.8029, Sem -0.8029, dir -0.9802, norm 0.1508 LR: 1.469e-02 Score 99.106 Data time: 0.1086, Forward time: 0.9471, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.2288 [2020-12-31 16:36:11,798][root][INFO] - ===> Epoch[2645](13225/5): Loss -0.8019, Sem -0.8019, dir -0.9805, norm 0.1520 LR: 1.466e-02 Score 99.103 Data time: 0.1107, Forward time: 0.9471, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.3268 [2020-12-31 16:36:24,568][root][INFO] - ===> Epoch[2646](13230/5): Loss -0.8049, Sem -0.8049, dir -0.9808, norm 0.1502 LR: 1.462e-02 Score 99.131 Data time: 0.0936, Forward time: 0.9471, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.4706 [2020-12-31 16:36:35,932][root][INFO] - ===> Epoch[2647](13235/5): Loss -0.8002, Sem -0.8002, dir -0.9796, norm 0.1519 LR: 1.458e-02 Score 99.077 Data time: 0.1593, Forward time: 0.9471, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.1732 [2020-12-31 16:36:48,289][root][INFO] - ===> Epoch[2648](13240/5): Loss -0.8005, Sem -0.8005, dir -0.9806, norm 0.1535 LR: 1.454e-02 Score 99.104 Data time: 0.1249, Forward time: 0.9471, Backward time: 1.1874, DDP time: 0.0050, Total iter time: 2.3838 [2020-12-31 16:37:00,724][root][INFO] - ===> Epoch[2649](13245/5): Loss -0.8041, Sem -0.8041, dir -0.9813, norm 0.1517 LR: 1.451e-02 Score 99.145 Data time: 0.1384, Forward time: 0.9472, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.3886 [2020-12-31 16:37:12,482][root][INFO] - ===> Epoch[2650](13250/5): Loss -0.7999, Sem -0.7999, dir -0.9802, norm 0.1538 LR: 1.447e-02 Score 99.117 Data time: 0.1468, Forward time: 0.9472, Backward time: 1.1873, DDP time: 0.0050, Total iter time: 2.2519 [2020-12-31 16:37:23,299][root][INFO] - ===> Epoch[2651](13255/5): Loss -0.8015, Sem -0.8015, dir -0.9803, norm 0.1511 LR: 1.443e-02 Score 99.065 Data time: 0.1077, Forward time: 0.9471, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.0687 [2020-12-31 16:37:35,093][root][INFO] - ===> Epoch[2652](13260/5): Loss -0.7972, Sem -0.7972, dir -0.9800, norm 0.1556 LR: 1.439e-02 Score 99.094 Data time: 0.0891, Forward time: 0.9471, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.2642 [2020-12-31 16:37:46,899][root][INFO] - ===> Epoch[2653](13265/5): Loss -0.8010, Sem -0.8010, dir -0.9800, norm 0.1527 LR: 1.436e-02 Score 99.110 Data time: 0.0950, Forward time: 0.9471, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.2663 [2020-12-31 16:37:58,704][root][INFO] - ===> Epoch[2654](13270/5): Loss -0.8034, Sem -0.8034, dir -0.9806, norm 0.1510 LR: 1.432e-02 Score 99.125 Data time: 0.1015, Forward time: 0.9472, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.2873 [2020-12-31 16:38:10,455][root][INFO] - ===> Epoch[2655](13275/5): Loss -0.7996, Sem -0.7996, dir -0.9802, norm 0.1540 LR: 1.428e-02 Score 99.107 Data time: 0.1104, Forward time: 0.9472, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.2562 [2020-12-31 16:38:22,129][root][INFO] - ===> Epoch[2656](13280/5): Loss -0.8074, Sem -0.8074, dir -0.9812, norm 0.1477 LR: 1.425e-02 Score 99.125 Data time: 0.0874, Forward time: 0.9472, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.2513 [2020-12-31 16:38:34,279][root][INFO] - ===> Epoch[2657](13285/5): Loss -0.7960, Sem -0.7960, dir -0.9800, norm 0.1569 LR: 1.421e-02 Score 99.092 Data time: 0.0934, Forward time: 0.9472, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.3464 [2020-12-31 16:38:45,823][root][INFO] - ===> Epoch[2658](13290/5): Loss -0.8020, Sem -0.8020, dir -0.9809, norm 0.1518 LR: 1.417e-02 Score 99.106 Data time: 0.1625, Forward time: 0.9471, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.2281 [2020-12-31 16:38:57,611][root][INFO] - ===> Epoch[2659](13295/5): Loss -0.8068, Sem -0.8068, dir -0.9808, norm 0.1483 LR: 1.413e-02 Score 99.145 Data time: 0.1127, Forward time: 0.9471, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.2701 [2020-12-31 16:39:08,807][root][INFO] - ===> Epoch[2660](13300/5): Loss -0.8014, Sem -0.8014, dir -0.9802, norm 0.1519 LR: 1.410e-02 Score 99.088 Data time: 0.1226, Forward time: 0.9471, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.1543 [2020-12-31 16:39:21,236][root][INFO] - ===> Epoch[2661](13305/5): Loss -0.8027, Sem -0.8027, dir -0.9807, norm 0.1528 LR: 1.406e-02 Score 99.163 Data time: 0.1012, Forward time: 0.9471, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3967 [2020-12-31 16:39:33,155][root][INFO] - ===> Epoch[2662](13310/5): Loss -0.8049, Sem -0.8049, dir -0.9812, norm 0.1504 LR: 1.402e-02 Score 99.133 Data time: 0.1227, Forward time: 0.9472, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2841 [2020-12-31 16:39:46,250][root][INFO] - ===> Epoch[2663](13315/5): Loss -0.8055, Sem -0.8055, dir -0.9812, norm 0.1504 LR: 1.399e-02 Score 99.152 Data time: 0.0979, Forward time: 0.9472, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.5195 [2020-12-31 16:39:57,876][root][INFO] - ===> Epoch[2664](13320/5): Loss -0.8049, Sem -0.8049, dir -0.9810, norm 0.1509 LR: 1.395e-02 Score 99.157 Data time: 0.0949, Forward time: 0.9472, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2337 [2020-12-31 16:40:09,584][root][INFO] - ===> Epoch[2665](13325/5): Loss -0.8054, Sem -0.8054, dir -0.9807, norm 0.1491 LR: 1.391e-02 Score 99.124 Data time: 0.0993, Forward time: 0.9472, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2456 [2020-12-31 16:40:21,547][root][INFO] - ===> Epoch[2666](13330/5): Loss -0.8035, Sem -0.8035, dir -0.9803, norm 0.1507 LR: 1.387e-02 Score 99.121 Data time: 0.0969, Forward time: 0.9471, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.3007 [2020-12-31 16:40:32,933][root][INFO] - ===> Epoch[2667](13335/5): Loss -0.8035, Sem -0.8035, dir -0.9811, norm 0.1518 LR: 1.384e-02 Score 99.143 Data time: 0.1085, Forward time: 0.9471, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.1824 [2020-12-31 16:40:45,367][root][INFO] - ===> Epoch[2668](13340/5): Loss -0.8087, Sem -0.8087, dir -0.9813, norm 0.1472 LR: 1.380e-02 Score 99.141 Data time: 0.1186, Forward time: 0.9471, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.4014 [2020-12-31 16:40:57,921][root][INFO] - ===> Epoch[2669](13345/5): Loss -0.8010, Sem -0.8010, dir -0.9806, norm 0.1540 LR: 1.376e-02 Score 99.146 Data time: 0.0983, Forward time: 0.9472, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.4023 [2020-12-31 16:41:09,476][root][INFO] - ===> Epoch[2670](13350/5): Loss -0.8092, Sem -0.8092, dir -0.9811, norm 0.1467 LR: 1.372e-02 Score 99.145 Data time: 0.2273, Forward time: 0.9472, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2172 [2020-12-31 16:41:20,416][root][INFO] - ===> Epoch[2671](13355/5): Loss -0.8011, Sem -0.8011, dir -0.9799, norm 0.1519 LR: 1.369e-02 Score 99.087 Data time: 0.1163, Forward time: 0.9472, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.0924 [2020-12-31 16:41:32,998][root][INFO] - ===> Epoch[2672](13360/5): Loss -0.8008, Sem -0.8008, dir -0.9805, norm 0.1528 LR: 1.365e-02 Score 99.109 Data time: 0.0961, Forward time: 0.9472, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.4196 [2020-12-31 16:41:44,529][root][INFO] - ===> Epoch[2673](13365/5): Loss -0.7980, Sem -0.7980, dir -0.9804, norm 0.1550 LR: 1.361e-02 Score 99.079 Data time: 0.1059, Forward time: 0.9472, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2165 [2020-12-31 16:41:56,337][root][INFO] - ===> Epoch[2674](13370/5): Loss -0.7962, Sem -0.7962, dir -0.9805, norm 0.1579 LR: 1.357e-02 Score 99.121 Data time: 0.1225, Forward time: 0.9472, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2626 [2020-12-31 16:42:08,403][root][INFO] - ===> Epoch[2675](13375/5): Loss -0.8084, Sem -0.8084, dir -0.9815, norm 0.1476 LR: 1.354e-02 Score 99.148 Data time: 0.0958, Forward time: 0.9472, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3199 [2020-12-31 16:42:20,572][root][INFO] - ===> Epoch[2676](13380/5): Loss -0.8049, Sem -0.8049, dir -0.9809, norm 0.1504 LR: 1.350e-02 Score 99.137 Data time: 0.0946, Forward time: 0.9472, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3481 [2020-12-31 16:42:32,859][root][INFO] - ===> Epoch[2677](13385/5): Loss -0.7993, Sem -0.7993, dir -0.9801, norm 0.1539 LR: 1.346e-02 Score 99.102 Data time: 0.1067, Forward time: 0.9472, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3571 [2020-12-31 16:42:44,882][root][INFO] - ===> Epoch[2678](13390/5): Loss -0.8031, Sem -0.8031, dir -0.9808, norm 0.1517 LR: 1.342e-02 Score 99.130 Data time: 0.0876, Forward time: 0.9472, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3179 [2020-12-31 16:42:57,838][root][INFO] - ===> Epoch[2679](13395/5): Loss -0.8018, Sem -0.8018, dir -0.9807, norm 0.1524 LR: 1.339e-02 Score 99.123 Data time: 0.1073, Forward time: 0.9472, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.5057 [2020-12-31 16:43:09,893][root][INFO] - ===> Epoch[2680](13400/5): Loss -0.8055, Sem -0.8055, dir -0.9807, norm 0.1498 LR: 1.335e-02 Score 99.151 Data time: 0.0799, Forward time: 0.9472, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3258 [2020-12-31 16:43:09,895][root][INFO] - ===> Start testing [2020-12-31 16:43:49,488][root][INFO] - 68/68: Data time: 0.0021, Iter time: 0.1235 Loss 0.505 (AVG: 0.543) Score 89.515 (AVG: 89.889) mIOU 67.985 mAP 82.855 mAcc 75.695 IOU: 60.675 0.000 78.762 73.653 93.621 90.331 29.672 74.557 98.278 62.317 80.947 85.207 55.786 mAP: 81.728 9.095 92.939 86.323 98.869 98.440 56.761 94.319 99.834 80.725 93.067 94.302 90.717 mAcc: 80.605 0.000 89.966 81.168 94.749 96.723 32.069 86.710 98.944 77.072 90.678 95.617 59.738 [2020-12-31 16:43:49,489][root][INFO] - Finished test. Elapsed time: 39.5916 [2020-12-31 16:43:52,598][root][INFO] - [2020-12-31 16:43:52,599][root][INFO] - ################################################################ [2020-12-31 16:43:52,599][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 16:43:52,599][root][INFO] - ################################################################ [2020-12-31 16:43:52,599][root][INFO] - clutter : nan nan nan [2020-12-31 16:43:52,599][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 16:43:52,599][root][INFO] - board : 0.797 0.868 0.896 [2020-12-31 16:43:52,599][root][INFO] - bookcase : 0.210 0.345 0.490 [2020-12-31 16:43:52,599][root][INFO] - ceiling : 0.612 0.739 0.857 [2020-12-31 16:43:52,599][root][INFO] - chair : 0.790 0.885 0.899 [2020-12-31 16:43:52,599][root][INFO] - column : 0.176 0.282 0.371 [2020-12-31 16:43:52,599][root][INFO] - door : 0.603 0.812 0.907 [2020-12-31 16:43:52,599][root][INFO] - floor : 0.949 1.000 1.000 [2020-12-31 16:43:52,599][root][INFO] - sofa : 0.444 0.636 0.809 [2020-12-31 16:43:52,599][root][INFO] - table : 0.196 0.405 0.635 [2020-12-31 16:43:52,599][root][INFO] - wall : 0.484 0.719 0.780 [2020-12-31 16:43:52,599][root][INFO] - window : 0.496 0.729 0.765 [2020-12-31 16:43:52,599][root][INFO] - ---------------------------------------------------------------- [2020-12-31 16:43:52,599][root][INFO] - average : 0.480 0.618 0.701 [2020-12-31 16:43:52,600][root][INFO] - [2020-12-31 16:43:52,761][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 16:43:53,380][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter13400.pth [2020-12-31 16:44:04,545][root][INFO] - ===> Epoch[2681](13405/5): Loss -0.8032, Sem -0.8032, dir -0.9811, norm 0.1524 LR: 1.331e-02 Score 99.141 Data time: 0.0992, Forward time: 0.9472, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.1244 [2020-12-31 16:44:15,508][root][INFO] - ===> Epoch[2682](13410/5): Loss -0.7967, Sem -0.7967, dir -0.9802, norm 0.1574 LR: 1.327e-02 Score 99.121 Data time: 0.0955, Forward time: 0.9472, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.0865 [2020-12-31 16:44:27,752][root][INFO] - ===> Epoch[2683](13415/5): Loss -0.8033, Sem -0.8033, dir -0.9810, norm 0.1520 LR: 1.324e-02 Score 99.141 Data time: 0.0995, Forward time: 0.9472, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3498 [2020-12-31 16:44:38,843][root][INFO] - ===> Epoch[2684](13420/5): Loss -0.8045, Sem -0.8045, dir -0.9802, norm 0.1490 LR: 1.320e-02 Score 99.092 Data time: 0.1080, Forward time: 0.9471, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.1229 [2020-12-31 16:44:50,836][root][INFO] - ===> Epoch[2685](13425/5): Loss -0.7996, Sem -0.7996, dir -0.9806, norm 0.1542 LR: 1.316e-02 Score 99.103 Data time: 0.0820, Forward time: 0.9471, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3102 [2020-12-31 16:45:02,619][root][INFO] - ===> Epoch[2686](13430/5): Loss -0.8068, Sem -0.8068, dir -0.9809, norm 0.1486 LR: 1.312e-02 Score 99.136 Data time: 0.1283, Forward time: 0.9471, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2548 [2020-12-31 16:45:14,198][root][INFO] - ===> Epoch[2687](13435/5): Loss -0.8097, Sem -0.8097, dir -0.9815, norm 0.1467 LR: 1.309e-02 Score 99.151 Data time: 0.0990, Forward time: 0.9471, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2214 [2020-12-31 16:45:25,874][root][INFO] - ===> Epoch[2688](13440/5): Loss -0.8022, Sem -0.8022, dir -0.9804, norm 0.1525 LR: 1.305e-02 Score 99.135 Data time: 0.0893, Forward time: 0.9471, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2446 [2020-12-31 16:45:37,658][root][INFO] - ===> Epoch[2689](13445/5): Loss -0.8038, Sem -0.8038, dir -0.9814, norm 0.1518 LR: 1.301e-02 Score 99.131 Data time: 0.0860, Forward time: 0.9471, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2751 [2020-12-31 16:45:50,430][root][INFO] - ===> Epoch[2690](13450/5): Loss -0.8064, Sem -0.8064, dir -0.9809, norm 0.1486 LR: 1.297e-02 Score 99.122 Data time: 0.0846, Forward time: 0.9471, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.4573 [2020-12-31 16:46:01,956][root][INFO] - ===> Epoch[2691](13455/5): Loss -0.8036, Sem -0.8036, dir -0.9808, norm 0.1517 LR: 1.294e-02 Score 99.146 Data time: 0.1466, Forward time: 0.9471, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2064 [2020-12-31 16:46:14,361][root][INFO] - ===> Epoch[2692](13460/5): Loss -0.8045, Sem -0.8045, dir -0.9808, norm 0.1506 LR: 1.290e-02 Score 99.136 Data time: 0.1600, Forward time: 0.9471, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.4036 [2020-12-31 16:46:25,723][root][INFO] - ===> Epoch[2693](13465/5): Loss -0.8008, Sem -0.8008, dir -0.9804, norm 0.1538 LR: 1.286e-02 Score 99.140 Data time: 0.1165, Forward time: 0.9471, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.1767 [2020-12-31 16:46:37,626][root][INFO] - ===> Epoch[2694](13470/5): Loss -0.7999, Sem -0.7999, dir -0.9809, norm 0.1548 LR: 1.282e-02 Score 99.119 Data time: 0.1350, Forward time: 0.9471, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2870 [2020-12-31 16:46:49,740][root][INFO] - ===> Epoch[2695](13475/5): Loss -0.8064, Sem -0.8064, dir -0.9815, norm 0.1493 LR: 1.278e-02 Score 99.144 Data time: 0.1057, Forward time: 0.9471, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3398 [2020-12-31 16:47:01,354][root][INFO] - ===> Epoch[2696](13480/5): Loss -0.8084, Sem -0.8084, dir -0.9809, norm 0.1470 LR: 1.275e-02 Score 99.140 Data time: 0.1097, Forward time: 0.9471, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2421 [2020-12-31 16:47:13,062][root][INFO] - ===> Epoch[2697](13485/5): Loss -0.8072, Sem -0.8072, dir -0.9813, norm 0.1487 LR: 1.271e-02 Score 99.142 Data time: 0.1484, Forward time: 0.9471, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2461 [2020-12-31 16:47:25,301][root][INFO] - ===> Epoch[2698](13490/5): Loss -0.8036, Sem -0.8036, dir -0.9809, norm 0.1512 LR: 1.267e-02 Score 99.126 Data time: 0.0901, Forward time: 0.9471, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3580 [2020-12-31 16:47:36,679][root][INFO] - ===> Epoch[2699](13495/5): Loss -0.8035, Sem -0.8035, dir -0.9806, norm 0.1511 LR: 1.263e-02 Score 99.125 Data time: 0.1026, Forward time: 0.9471, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.1821 [2020-12-31 16:47:49,477][root][INFO] - ===> Epoch[2700](13500/5): Loss -0.8041, Sem -0.8041, dir -0.9805, norm 0.1507 LR: 1.260e-02 Score 99.148 Data time: 0.1091, Forward time: 0.9471, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.4747 [2020-12-31 16:48:02,263][root][INFO] - ===> Epoch[2701](13505/5): Loss -0.8030, Sem -0.8030, dir -0.9804, norm 0.1510 LR: 1.256e-02 Score 99.120 Data time: 0.1133, Forward time: 0.9471, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.4832 [2020-12-31 16:48:13,645][root][INFO] - ===> Epoch[2702](13510/5): Loss -0.8054, Sem -0.8054, dir -0.9810, norm 0.1505 LR: 1.252e-02 Score 99.152 Data time: 0.1260, Forward time: 0.9470, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.1867 [2020-12-31 16:48:25,658][root][INFO] - ===> Epoch[2703](13515/5): Loss -0.8034, Sem -0.8034, dir -0.9808, norm 0.1508 LR: 1.248e-02 Score 99.103 Data time: 0.1183, Forward time: 0.9470, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3239 [2020-12-31 16:48:38,307][root][INFO] - ===> Epoch[2704](13520/5): Loss -0.8054, Sem -0.8054, dir -0.9812, norm 0.1505 LR: 1.244e-02 Score 99.157 Data time: 0.1136, Forward time: 0.9470, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.4316 [2020-12-31 16:48:49,883][root][INFO] - ===> Epoch[2705](13525/5): Loss -0.8077, Sem -0.8077, dir -0.9808, norm 0.1468 LR: 1.241e-02 Score 99.127 Data time: 0.1204, Forward time: 0.9470, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2229 [2020-12-31 16:49:01,328][root][INFO] - ===> Epoch[2706](13530/5): Loss -0.8048, Sem -0.8048, dir -0.9806, norm 0.1490 LR: 1.237e-02 Score 99.098 Data time: 0.1073, Forward time: 0.9470, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.1957 [2020-12-31 16:49:13,106][root][INFO] - ===> Epoch[2707](13535/5): Loss -0.8012, Sem -0.8012, dir -0.9807, norm 0.1527 LR: 1.233e-02 Score 99.106 Data time: 0.1018, Forward time: 0.9470, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2636 [2020-12-31 16:49:24,615][root][INFO] - ===> Epoch[2708](13540/5): Loss -0.7958, Sem -0.7958, dir -0.9795, norm 0.1560 LR: 1.229e-02 Score 99.073 Data time: 0.1149, Forward time: 0.9469, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2201 [2020-12-31 16:49:37,243][root][INFO] - ===> Epoch[2709](13545/5): Loss -0.8071, Sem -0.8071, dir -0.9811, norm 0.1489 LR: 1.226e-02 Score 99.157 Data time: 0.1075, Forward time: 0.9470, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.4377 [2020-12-31 16:49:48,862][root][INFO] - ===> Epoch[2710](13550/5): Loss -0.8024, Sem -0.8024, dir -0.9805, norm 0.1518 LR: 1.222e-02 Score 99.116 Data time: 0.1183, Forward time: 0.9470, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2258 [2020-12-31 16:50:00,808][root][INFO] - ===> Epoch[2711](13555/5): Loss -0.8045, Sem -0.8045, dir -0.9808, norm 0.1499 LR: 1.218e-02 Score 99.107 Data time: 0.0965, Forward time: 0.9469, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2966 [2020-12-31 16:50:12,606][root][INFO] - ===> Epoch[2712](13560/5): Loss -0.8016, Sem -0.8016, dir -0.9803, norm 0.1522 LR: 1.214e-02 Score 99.107 Data time: 0.1009, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2585 [2020-12-31 16:50:24,055][root][INFO] - ===> Epoch[2713](13565/5): Loss -0.8061, Sem -0.8061, dir -0.9817, norm 0.1493 LR: 1.210e-02 Score 99.125 Data time: 0.1696, Forward time: 0.9469, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.1951 [2020-12-31 16:50:36,385][root][INFO] - ===> Epoch[2714](13570/5): Loss -0.8000, Sem -0.8000, dir -0.9798, norm 0.1527 LR: 1.207e-02 Score 99.081 Data time: 0.0826, Forward time: 0.9468, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3810 [2020-12-31 16:50:47,434][root][INFO] - ===> Epoch[2715](13575/5): Loss -0.8069, Sem -0.8069, dir -0.9811, norm 0.1489 LR: 1.203e-02 Score 99.148 Data time: 0.1476, Forward time: 0.9468, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.1161 [2020-12-31 16:51:00,046][root][INFO] - ===> Epoch[2716](13580/5): Loss -0.8038, Sem -0.8038, dir -0.9806, norm 0.1511 LR: 1.199e-02 Score 99.129 Data time: 0.1154, Forward time: 0.9469, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.4260 [2020-12-31 16:51:11,507][root][INFO] - ===> Epoch[2717](13585/5): Loss -0.8024, Sem -0.8024, dir -0.9805, norm 0.1525 LR: 1.195e-02 Score 99.133 Data time: 0.0856, Forward time: 0.9468, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2030 [2020-12-31 16:51:22,501][root][INFO] - ===> Epoch[2718](13590/5): Loss -0.7993, Sem -0.7993, dir -0.9803, norm 0.1539 LR: 1.191e-02 Score 99.098 Data time: 0.0981, Forward time: 0.9468, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.1140 [2020-12-31 16:51:34,339][root][INFO] - ===> Epoch[2719](13595/5): Loss -0.8006, Sem -0.8006, dir -0.9809, norm 0.1540 LR: 1.188e-02 Score 99.118 Data time: 0.0969, Forward time: 0.9468, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2707 [2020-12-31 16:51:46,682][root][INFO] - ===> Epoch[2720](13600/5): Loss -0.8029, Sem -0.8029, dir -0.9804, norm 0.1510 LR: 1.184e-02 Score 99.114 Data time: 0.0998, Forward time: 0.9468, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3729 [2020-12-31 16:51:46,685][root][INFO] - ===> Start testing [2020-12-31 16:52:26,132][root][INFO] - 68/68: Data time: 0.0031, Iter time: 0.1804 Loss 0.972 (AVG: 0.526) Score 85.915 (AVG: 90.092) mIOU 68.801 mAP 83.134 mAcc 76.181 IOU: 60.897 0.000 80.958 73.469 93.789 89.704 33.780 77.497 98.345 62.332 80.974 85.527 57.145 mAP: 80.813 10.999 92.924 85.871 98.939 98.454 59.388 95.046 99.826 80.099 93.071 94.352 90.960 mAcc: 81.098 0.000 88.495 81.098 94.949 97.010 38.241 87.672 99.039 75.947 89.821 95.453 61.523 [2020-12-31 16:52:26,133][root][INFO] - Finished test. Elapsed time: 39.4455 [2020-12-31 16:52:29,565][root][INFO] - [2020-12-31 16:52:29,565][root][INFO] - ################################################################ [2020-12-31 16:52:29,565][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 16:52:29,565][root][INFO] - ################################################################ [2020-12-31 16:52:29,565][root][INFO] - clutter : nan nan nan [2020-12-31 16:52:29,565][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 16:52:29,566][root][INFO] - board : 0.772 0.848 0.901 [2020-12-31 16:52:29,566][root][INFO] - bookcase : 0.208 0.361 0.506 [2020-12-31 16:52:29,566][root][INFO] - ceiling : 0.693 0.845 0.858 [2020-12-31 16:52:29,566][root][INFO] - chair : 0.802 0.900 0.914 [2020-12-31 16:52:29,566][root][INFO] - column : 0.192 0.305 0.403 [2020-12-31 16:52:29,566][root][INFO] - door : 0.611 0.801 0.923 [2020-12-31 16:52:29,566][root][INFO] - floor : 0.961 1.000 1.000 [2020-12-31 16:52:29,566][root][INFO] - sofa : 0.450 0.636 0.801 [2020-12-31 16:52:29,566][root][INFO] - table : 0.197 0.393 0.627 [2020-12-31 16:52:29,566][root][INFO] - wall : 0.494 0.733 0.810 [2020-12-31 16:52:29,566][root][INFO] - window : 0.506 0.746 0.779 [2020-12-31 16:52:29,566][root][INFO] - ---------------------------------------------------------------- [2020-12-31 16:52:29,566][root][INFO] - average : 0.491 0.631 0.710 [2020-12-31 16:52:29,566][root][INFO] - [2020-12-31 16:52:29,750][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 16:52:30,315][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_mAP.pth [2020-12-31 16:52:30,315][root][INFO] - Current best mAP@0.5: 0.631 at iter 13600 [2020-12-31 16:52:30,867][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter13600.pth [2020-12-31 16:52:42,133][root][INFO] - ===> Epoch[2721](13605/5): Loss -0.8046, Sem -0.8046, dir -0.9809, norm 0.1507 LR: 1.180e-02 Score 99.129 Data time: 0.0770, Forward time: 0.9468, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.1640 [2020-12-31 16:52:54,012][root][INFO] - ===> Epoch[2722](13610/5): Loss -0.8053, Sem -0.8053, dir -0.9814, norm 0.1507 LR: 1.176e-02 Score 99.152 Data time: 0.1275, Forward time: 0.9468, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2801 [2020-12-31 16:53:06,705][root][INFO] - ===> Epoch[2723](13615/5): Loss -0.8053, Sem -0.8053, dir -0.9807, norm 0.1499 LR: 1.172e-02 Score 99.143 Data time: 0.1117, Forward time: 0.9468, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.4567 [2020-12-31 16:53:19,017][root][INFO] - ===> Epoch[2724](13620/5): Loss -0.8073, Sem -0.8073, dir -0.9814, norm 0.1489 LR: 1.169e-02 Score 99.151 Data time: 0.1273, Forward time: 0.9468, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3722 [2020-12-31 16:53:31,164][root][INFO] - ===> Epoch[2725](13625/5): Loss -0.8064, Sem -0.8064, dir -0.9812, norm 0.1484 LR: 1.165e-02 Score 99.117 Data time: 0.1035, Forward time: 0.9468, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3475 [2020-12-31 16:53:43,164][root][INFO] - ===> Epoch[2726](13630/5): Loss -0.8033, Sem -0.8033, dir -0.9802, norm 0.1515 LR: 1.161e-02 Score 99.148 Data time: 0.1759, Forward time: 0.9468, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3067 [2020-12-31 16:53:54,581][root][INFO] - ===> Epoch[2727](13635/5): Loss -0.8138, Sem -0.8138, dir -0.9816, norm 0.1421 LR: 1.157e-02 Score 99.131 Data time: 0.1098, Forward time: 0.9468, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.1902 [2020-12-31 16:54:06,726][root][INFO] - ===> Epoch[2728](13640/5): Loss -0.8039, Sem -0.8039, dir -0.9810, norm 0.1513 LR: 1.153e-02 Score 99.141 Data time: 0.1541, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3361 [2020-12-31 16:54:18,752][root][INFO] - ===> Epoch[2729](13645/5): Loss -0.8048, Sem -0.8048, dir -0.9809, norm 0.1507 LR: 1.150e-02 Score 99.158 Data time: 0.1536, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3051 [2020-12-31 16:54:30,453][root][INFO] - ===> Epoch[2730](13650/5): Loss -0.8045, Sem -0.8045, dir -0.9809, norm 0.1503 LR: 1.146e-02 Score 99.124 Data time: 0.1283, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.2603 [2020-12-31 16:54:43,296][root][INFO] - ===> Epoch[2731](13655/5): Loss -0.8022, Sem -0.8022, dir -0.9808, norm 0.1521 LR: 1.142e-02 Score 99.111 Data time: 0.1388, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.4830 [2020-12-31 16:54:55,114][root][INFO] - ===> Epoch[2732](13660/5): Loss -0.8092, Sem -0.8092, dir -0.9812, norm 0.1469 LR: 1.138e-02 Score 99.150 Data time: 0.1471, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.2773 [2020-12-31 16:55:07,222][root][INFO] - ===> Epoch[2733](13665/5): Loss -0.8012, Sem -0.8012, dir -0.9802, norm 0.1514 LR: 1.134e-02 Score 99.078 Data time: 0.1152, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3344 [2020-12-31 16:55:18,806][root][INFO] - ===> Epoch[2734](13670/5): Loss -0.8067, Sem -0.8067, dir -0.9815, norm 0.1481 LR: 1.130e-02 Score 99.113 Data time: 0.1343, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.2221 [2020-12-31 16:55:31,132][root][INFO] - ===> Epoch[2735](13675/5): Loss -0.8064, Sem -0.8064, dir -0.9811, norm 0.1496 LR: 1.127e-02 Score 99.163 Data time: 0.1085, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3614 [2020-12-31 16:55:43,047][root][INFO] - ===> Epoch[2736](13680/5): Loss -0.8068, Sem -0.8068, dir -0.9811, norm 0.1491 LR: 1.123e-02 Score 99.149 Data time: 0.0861, Forward time: 0.9470, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3050 [2020-12-31 16:55:55,778][root][INFO] - ===> Epoch[2737](13685/5): Loss -0.8057, Sem -0.8057, dir -0.9809, norm 0.1492 LR: 1.119e-02 Score 99.128 Data time: 0.1243, Forward time: 0.9470, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.4473 [2020-12-31 16:56:07,870][root][INFO] - ===> Epoch[2738](13690/5): Loss -0.8018, Sem -0.8018, dir -0.9805, norm 0.1518 LR: 1.115e-02 Score 99.091 Data time: 0.1150, Forward time: 0.9470, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3297 [2020-12-31 16:56:20,078][root][INFO] - ===> Epoch[2739](13695/5): Loss -0.8049, Sem -0.8049, dir -0.9808, norm 0.1498 LR: 1.111e-02 Score 99.125 Data time: 0.0948, Forward time: 0.9470, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3493 [2020-12-31 16:56:31,917][root][INFO] - ===> Epoch[2740](13700/5): Loss -0.8093, Sem -0.8093, dir -0.9810, norm 0.1465 LR: 1.107e-02 Score 99.154 Data time: 0.1162, Forward time: 0.9470, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2788 [2020-12-31 16:56:43,277][root][INFO] - ===> Epoch[2741](13705/5): Loss -0.8091, Sem -0.8091, dir -0.9807, norm 0.1447 LR: 1.104e-02 Score 99.096 Data time: 0.0913, Forward time: 0.9470, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.1747 [2020-12-31 16:56:55,569][root][INFO] - ===> Epoch[2742](13710/5): Loss -0.8081, Sem -0.8081, dir -0.9811, norm 0.1470 LR: 1.100e-02 Score 99.122 Data time: 0.0969, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3887 [2020-12-31 16:57:06,558][root][INFO] - ===> Epoch[2743](13715/5): Loss -0.8051, Sem -0.8051, dir -0.9807, norm 0.1491 LR: 1.096e-02 Score 99.120 Data time: 0.0959, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.1222 [2020-12-31 16:57:18,608][root][INFO] - ===> Epoch[2744](13720/5): Loss -0.8059, Sem -0.8059, dir -0.9805, norm 0.1484 LR: 1.092e-02 Score 99.123 Data time: 0.0994, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2992 [2020-12-31 16:57:30,266][root][INFO] - ===> Epoch[2745](13725/5): Loss -0.8044, Sem -0.8044, dir -0.9809, norm 0.1493 LR: 1.088e-02 Score 99.094 Data time: 0.1057, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2401 [2020-12-31 16:57:42,218][root][INFO] - ===> Epoch[2746](13730/5): Loss -0.8109, Sem -0.8109, dir -0.9817, norm 0.1449 LR: 1.084e-02 Score 99.130 Data time: 0.1310, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3009 [2020-12-31 16:57:55,317][root][INFO] - ===> Epoch[2747](13735/5): Loss -0.8016, Sem -0.8016, dir -0.9803, norm 0.1520 LR: 1.081e-02 Score 99.096 Data time: 0.1109, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.5380 [2020-12-31 16:58:07,064][root][INFO] - ===> Epoch[2748](13740/5): Loss -0.8023, Sem -0.8023, dir -0.9803, norm 0.1518 LR: 1.077e-02 Score 99.128 Data time: 0.1123, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2555 [2020-12-31 16:58:18,538][root][INFO] - ===> Epoch[2749](13745/5): Loss -0.8067, Sem -0.8067, dir -0.9808, norm 0.1476 LR: 1.073e-02 Score 99.109 Data time: 0.0841, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.1946 [2020-12-31 16:58:30,676][root][INFO] - ===> Epoch[2750](13750/5): Loss -0.8046, Sem -0.8046, dir -0.9807, norm 0.1504 LR: 1.069e-02 Score 99.139 Data time: 0.1633, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3325 [2020-12-31 16:58:42,581][root][INFO] - ===> Epoch[2751](13755/5): Loss -0.8001, Sem -0.8001, dir -0.9804, norm 0.1525 LR: 1.065e-02 Score 99.075 Data time: 0.0848, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2996 [2020-12-31 16:58:55,077][root][INFO] - ===> Epoch[2752](13760/5): Loss -0.8077, Sem -0.8077, dir -0.9818, norm 0.1495 LR: 1.061e-02 Score 99.179 Data time: 0.1266, Forward time: 0.9468, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.4017 [2020-12-31 16:59:06,571][root][INFO] - ===> Epoch[2753](13765/5): Loss -0.8025, Sem -0.8025, dir -0.9802, norm 0.1514 LR: 1.058e-02 Score 99.116 Data time: 0.1001, Forward time: 0.9468, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2049 [2020-12-31 16:59:17,906][root][INFO] - ===> Epoch[2754](13770/5): Loss -0.8107, Sem -0.8107, dir -0.9814, norm 0.1454 LR: 1.054e-02 Score 99.150 Data time: 0.1288, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.1660 [2020-12-31 16:59:30,127][root][INFO] - ===> Epoch[2755](13775/5): Loss -0.8033, Sem -0.8033, dir -0.9809, norm 0.1518 LR: 1.050e-02 Score 99.134 Data time: 0.1090, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3730 [2020-12-31 16:59:42,298][root][INFO] - ===> Epoch[2756](13780/5): Loss -0.8057, Sem -0.8057, dir -0.9807, norm 0.1485 LR: 1.046e-02 Score 99.122 Data time: 0.0944, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3504 [2020-12-31 16:59:54,053][root][INFO] - ===> Epoch[2757](13785/5): Loss -0.8076, Sem -0.8076, dir -0.9808, norm 0.1474 LR: 1.042e-02 Score 99.126 Data time: 0.1250, Forward time: 0.9468, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2593 [2020-12-31 17:00:06,001][root][INFO] - ===> Epoch[2758](13790/5): Loss -0.8092, Sem -0.8092, dir -0.9811, norm 0.1461 LR: 1.038e-02 Score 99.147 Data time: 0.1809, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.2883 [2020-12-31 17:00:18,492][root][INFO] - ===> Epoch[2759](13795/5): Loss -0.8064, Sem -0.8064, dir -0.9809, norm 0.1484 LR: 1.034e-02 Score 99.122 Data time: 0.1364, Forward time: 0.9468, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.4011 [2020-12-31 17:00:30,041][root][INFO] - ===> Epoch[2760](13800/5): Loss -0.8045, Sem -0.8045, dir -0.9807, norm 0.1507 LR: 1.031e-02 Score 99.148 Data time: 0.1744, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.2206 [2020-12-31 17:00:30,044][root][INFO] - ===> Start testing [2020-12-31 17:01:09,397][root][INFO] - 68/68: Data time: 0.6136, Iter time: 0.3040 Loss 0.588 (AVG: 0.538) Score 88.463 (AVG: 90.095) mIOU 68.583 mAP 83.252 mAcc 75.924 IOU: 61.139 0.000 82.721 73.981 93.686 90.807 30.572 75.830 98.351 61.898 80.714 85.416 56.459 mAP: 81.479 13.225 92.996 86.563 98.957 98.484 56.371 95.263 99.829 80.977 93.149 93.995 90.988 mAcc: 80.768 0.000 90.123 81.466 94.720 96.517 32.553 88.088 99.064 76.657 90.608 95.858 60.585 [2020-12-31 17:01:09,398][root][INFO] - Finished test. Elapsed time: 39.3523 [2020-12-31 17:01:13,000][root][INFO] - [2020-12-31 17:01:13,000][root][INFO] - ################################################################ [2020-12-31 17:01:13,000][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 17:01:13,000][root][INFO] - ################################################################ [2020-12-31 17:01:13,000][root][INFO] - clutter : nan nan nan [2020-12-31 17:01:13,000][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 17:01:13,000][root][INFO] - board : 0.799 0.872 0.900 [2020-12-31 17:01:13,000][root][INFO] - bookcase : 0.234 0.394 0.521 [2020-12-31 17:01:13,000][root][INFO] - ceiling : 0.699 0.856 0.856 [2020-12-31 17:01:13,000][root][INFO] - chair : 0.802 0.904 0.912 [2020-12-31 17:01:13,000][root][INFO] - column : 0.173 0.271 0.379 [2020-12-31 17:01:13,000][root][INFO] - door : 0.650 0.851 0.919 [2020-12-31 17:01:13,001][root][INFO] - floor : 0.942 1.000 1.000 [2020-12-31 17:01:13,001][root][INFO] - sofa : 0.450 0.636 0.818 [2020-12-31 17:01:13,001][root][INFO] - table : 0.199 0.417 0.655 [2020-12-31 17:01:13,001][root][INFO] - wall : 0.484 0.730 0.800 [2020-12-31 17:01:13,001][root][INFO] - window : 0.497 0.729 0.763 [2020-12-31 17:01:13,001][root][INFO] - ---------------------------------------------------------------- [2020-12-31 17:01:13,001][root][INFO] - average : 0.494 0.638 0.710 [2020-12-31 17:01:13,001][root][INFO] - [2020-12-31 17:01:13,171][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 17:01:13,776][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_mAP.pth [2020-12-31 17:01:13,777][root][INFO] - Current best mAP@0.5: 0.638 at iter 13800 [2020-12-31 17:01:14,377][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter13800.pth [2020-12-31 17:01:25,365][root][INFO] - ===> Epoch[2761](13805/5): Loss -0.8086, Sem -0.8086, dir -0.9815, norm 0.1471 LR: 1.027e-02 Score 99.139 Data time: 0.0801, Forward time: 0.9468, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.1099 [2020-12-31 17:01:37,404][root][INFO] - ===> Epoch[2762](13810/5): Loss -0.8067, Sem -0.8067, dir -0.9808, norm 0.1483 LR: 1.023e-02 Score 99.132 Data time: 0.1063, Forward time: 0.9468, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3160 [2020-12-31 17:01:49,031][root][INFO] - ===> Epoch[2763](13815/5): Loss -0.8078, Sem -0.8078, dir -0.9815, norm 0.1478 LR: 1.019e-02 Score 99.137 Data time: 0.1523, Forward time: 0.9468, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.2274 [2020-12-31 17:02:00,826][root][INFO] - ===> Epoch[2764](13820/5): Loss -0.8121, Sem -0.8121, dir -0.9815, norm 0.1444 LR: 1.015e-02 Score 99.160 Data time: 0.1218, Forward time: 0.9468, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.2680 [2020-12-31 17:02:12,619][root][INFO] - ===> Epoch[2765](13825/5): Loss -0.8060, Sem -0.8060, dir -0.9809, norm 0.1491 LR: 1.011e-02 Score 99.142 Data time: 0.1284, Forward time: 0.9468, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.2770 [2020-12-31 17:02:23,516][root][INFO] - ===> Epoch[2766](13830/5): Loss -0.8042, Sem -0.8042, dir -0.9809, norm 0.1497 LR: 1.007e-02 Score 99.111 Data time: 0.1131, Forward time: 0.9468, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.0857 [2020-12-31 17:02:34,706][root][INFO] - ===> Epoch[2767](13835/5): Loss -0.8102, Sem -0.8102, dir -0.9813, norm 0.1460 LR: 1.004e-02 Score 99.149 Data time: 0.0922, Forward time: 0.9469, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.1533 [2020-12-31 17:02:46,162][root][INFO] - ===> Epoch[2768](13840/5): Loss -0.8083, Sem -0.8083, dir -0.9814, norm 0.1481 LR: 9.996e-03 Score 99.168 Data time: 0.1322, Forward time: 0.9469, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.1869 [2020-12-31 17:02:57,364][root][INFO] - ===> Epoch[2769](13845/5): Loss -0.8087, Sem -0.8087, dir -0.9809, norm 0.1460 LR: 9.958e-03 Score 99.125 Data time: 0.1087, Forward time: 0.9469, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.1687 [2020-12-31 17:03:10,065][root][INFO] - ===> Epoch[2770](13850/5): Loss -0.8097, Sem -0.8097, dir -0.9815, norm 0.1472 LR: 9.919e-03 Score 99.174 Data time: 0.1493, Forward time: 0.9469, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.4541 [2020-12-31 17:03:22,156][root][INFO] - ===> Epoch[2771](13855/5): Loss -0.8064, Sem -0.8064, dir -0.9812, norm 0.1484 LR: 9.880e-03 Score 99.121 Data time: 0.1031, Forward time: 0.9468, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.3226 [2020-12-31 17:03:33,658][root][INFO] - ===> Epoch[2772](13860/5): Loss -0.8095, Sem -0.8095, dir -0.9808, norm 0.1452 LR: 9.841e-03 Score 99.123 Data time: 0.1820, Forward time: 0.9468, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.2040 [2020-12-31 17:03:46,032][root][INFO] - ===> Epoch[2773](13865/5): Loss -0.8080, Sem -0.8080, dir -0.9811, norm 0.1467 LR: 9.802e-03 Score 99.110 Data time: 0.1315, Forward time: 0.9469, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.3866 [2020-12-31 17:03:58,419][root][INFO] - ===> Epoch[2774](13870/5): Loss -0.8088, Sem -0.8088, dir -0.9815, norm 0.1482 LR: 9.764e-03 Score 99.184 Data time: 0.1483, Forward time: 0.9469, Backward time: 1.1864, DDP time: 0.0050, Total iter time: 2.3802 [2020-12-31 17:04:10,189][root][INFO] - ===> Epoch[2775](13875/5): Loss -0.8098, Sem -0.8098, dir -0.9811, norm 0.1455 LR: 9.725e-03 Score 99.128 Data time: 0.1178, Forward time: 0.9469, Backward time: 1.1864, DDP time: 0.0050, Total iter time: 2.2621 [2020-12-31 17:04:22,325][root][INFO] - ===> Epoch[2776](13880/5): Loss -0.8091, Sem -0.8091, dir -0.9809, norm 0.1452 LR: 9.686e-03 Score 99.096 Data time: 0.1155, Forward time: 0.9470, Backward time: 1.1864, DDP time: 0.0050, Total iter time: 2.3302 [2020-12-31 17:04:34,914][root][INFO] - ===> Epoch[2777](13885/5): Loss -0.8100, Sem -0.8100, dir -0.9809, norm 0.1455 LR: 9.647e-03 Score 99.137 Data time: 0.0983, Forward time: 0.9469, Backward time: 1.1864, DDP time: 0.0050, Total iter time: 2.4327 [2020-12-31 17:04:46,502][root][INFO] - ===> Epoch[2778](13890/5): Loss -0.8131, Sem -0.8131, dir -0.9817, norm 0.1441 LR: 9.608e-03 Score 99.180 Data time: 0.0816, Forward time: 0.9469, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.2378 [2020-12-31 17:04:58,752][root][INFO] - ===> Epoch[2779](13895/5): Loss -0.8079, Sem -0.8079, dir -0.9813, norm 0.1487 LR: 9.569e-03 Score 99.162 Data time: 0.0854, Forward time: 0.9469, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.3750 [2020-12-31 17:05:10,438][root][INFO] - ===> Epoch[2780](13900/5): Loss -0.8111, Sem -0.8111, dir -0.9807, norm 0.1441 LR: 9.530e-03 Score 99.139 Data time: 0.0899, Forward time: 0.9469, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.2553 [2020-12-31 17:05:22,355][root][INFO] - ===> Epoch[2781](13905/5): Loss -0.8119, Sem -0.8119, dir -0.9812, norm 0.1450 LR: 9.491e-03 Score 99.179 Data time: 0.1221, Forward time: 0.9469, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.2934 [2020-12-31 17:05:35,332][root][INFO] - ===> Epoch[2782](13910/5): Loss -0.8084, Sem -0.8084, dir -0.9814, norm 0.1483 LR: 9.452e-03 Score 99.169 Data time: 0.0767, Forward time: 0.9469, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.5069 [2020-12-31 17:05:48,549][root][INFO] - ===> Epoch[2783](13915/5): Loss -0.8132, Sem -0.8132, dir -0.9817, norm 0.1437 LR: 9.413e-03 Score 99.158 Data time: 0.0688, Forward time: 0.9468, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.5594 [2020-12-31 17:06:01,895][root][INFO] - ===> Epoch[2784](13920/5): Loss -0.8029, Sem -0.8029, dir -0.9806, norm 0.1517 LR: 9.374e-03 Score 99.130 Data time: 0.2821, Forward time: 0.9468, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.5743 [2020-12-31 17:06:14,841][root][INFO] - ===> Epoch[2785](13925/5): Loss -0.8099, Sem -0.8099, dir -0.9817, norm 0.1475 LR: 9.335e-03 Score 99.182 Data time: 0.1491, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.4784 [2020-12-31 17:06:26,428][root][INFO] - ===> Epoch[2786](13930/5): Loss -0.8077, Sem -0.8077, dir -0.9810, norm 0.1474 LR: 9.296e-03 Score 99.146 Data time: 0.1032, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.2245 [2020-12-31 17:06:38,514][root][INFO] - ===> Epoch[2787](13935/5): Loss -0.8088, Sem -0.8088, dir -0.9810, norm 0.1469 LR: 9.257e-03 Score 99.146 Data time: 0.0795, Forward time: 0.9468, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3185 [2020-12-31 17:06:50,828][root][INFO] - ===> Epoch[2788](13940/5): Loss -0.8076, Sem -0.8076, dir -0.9807, norm 0.1469 LR: 9.218e-03 Score 99.121 Data time: 0.1441, Forward time: 0.9468, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3671 [2020-12-31 17:07:03,190][root][INFO] - ===> Epoch[2789](13945/5): Loss -0.8108, Sem -0.8108, dir -0.9814, norm 0.1463 LR: 9.179e-03 Score 99.183 Data time: 0.0818, Forward time: 0.9468, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3885 [2020-12-31 17:07:16,029][root][INFO] - ===> Epoch[2790](13950/5): Loss -0.8140, Sem -0.8140, dir -0.9816, norm 0.1427 LR: 9.140e-03 Score 99.162 Data time: 0.1024, Forward time: 0.9468, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.4809 [2020-12-31 17:07:28,301][root][INFO] - ===> Epoch[2791](13955/5): Loss -0.8073, Sem -0.8073, dir -0.9807, norm 0.1482 LR: 9.101e-03 Score 99.148 Data time: 0.1073, Forward time: 0.9468, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3591 [2020-12-31 17:07:38,962][root][INFO] - ===> Epoch[2792](13960/5): Loss -0.8067, Sem -0.8067, dir -0.9806, norm 0.1479 LR: 9.061e-03 Score 99.119 Data time: 0.0878, Forward time: 0.9468, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.0519 [2020-12-31 17:07:51,095][root][INFO] - ===> Epoch[2793](13965/5): Loss -0.8064, Sem -0.8064, dir -0.9815, norm 0.1502 LR: 9.022e-03 Score 99.157 Data time: 0.1359, Forward time: 0.9468, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3301 [2020-12-31 17:08:03,378][root][INFO] - ===> Epoch[2794](13970/5): Loss -0.8023, Sem -0.8023, dir -0.9801, norm 0.1514 LR: 8.983e-03 Score 99.107 Data time: 0.0987, Forward time: 0.9468, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3728 [2020-12-31 17:08:15,563][root][INFO] - ===> Epoch[2795](13975/5): Loss -0.8034, Sem -0.8034, dir -0.9805, norm 0.1506 LR: 8.944e-03 Score 99.111 Data time: 0.1431, Forward time: 0.9468, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3431 [2020-12-31 17:08:28,048][root][INFO] - ===> Epoch[2796](13980/5): Loss -0.8078, Sem -0.8078, dir -0.9810, norm 0.1480 LR: 8.905e-03 Score 99.154 Data time: 0.1732, Forward time: 0.9468, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.4056 [2020-12-31 17:08:41,133][root][INFO] - ===> Epoch[2797](13985/5): Loss -0.8147, Sem -0.8147, dir -0.9818, norm 0.1420 LR: 8.865e-03 Score 99.152 Data time: 0.1469, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.5173 [2020-12-31 17:08:53,156][root][INFO] - ===> Epoch[2798](13990/5): Loss -0.8000, Sem -0.8000, dir -0.9805, norm 0.1543 LR: 8.826e-03 Score 99.127 Data time: 0.1112, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3197 [2020-12-31 17:09:04,576][root][INFO] - ===> Epoch[2799](13995/5): Loss -0.8089, Sem -0.8089, dir -0.9807, norm 0.1464 LR: 8.787e-03 Score 99.138 Data time: 0.1326, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.1923 [2020-12-31 17:09:17,079][root][INFO] - ===> Epoch[2800](14000/5): Loss -0.8141, Sem -0.8141, dir -0.9822, norm 0.1434 LR: 8.747e-03 Score 99.179 Data time: 0.0952, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3949 [2020-12-31 17:09:17,081][root][INFO] - ===> Start testing [2020-12-31 17:09:55,511][root][INFO] - 68/68: Data time: 0.0022, Iter time: 0.1077 Loss 0.101 (AVG: 0.537) Score 97.247 (AVG: 90.123) mIOU 68.591 mAP 83.142 mAcc 75.901 IOU: 61.080 0.000 81.400 73.720 94.312 90.562 29.971 77.726 98.348 62.396 80.918 85.217 56.027 mAP: 81.147 11.770 92.820 86.411 99.018 98.454 55.504 96.031 99.828 82.051 93.240 94.040 90.536 mAcc: 80.668 0.000 89.699 80.729 95.515 96.519 32.377 87.848 99.062 77.318 91.256 95.804 59.917 [2020-12-31 17:09:55,512][root][INFO] - Finished test. Elapsed time: 38.4291 [2020-12-31 17:09:59,103][root][INFO] - [2020-12-31 17:09:59,103][root][INFO] - ################################################################ [2020-12-31 17:09:59,103][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 17:09:59,103][root][INFO] - ################################################################ [2020-12-31 17:09:59,103][root][INFO] - clutter : nan nan nan [2020-12-31 17:09:59,104][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 17:09:59,104][root][INFO] - board : 0.791 0.868 0.896 [2020-12-31 17:09:59,104][root][INFO] - bookcase : 0.201 0.338 0.484 [2020-12-31 17:09:59,104][root][INFO] - ceiling : 0.691 0.849 0.873 [2020-12-31 17:09:59,104][root][INFO] - chair : 0.803 0.899 0.908 [2020-12-31 17:09:59,104][root][INFO] - column : 0.168 0.259 0.412 [2020-12-31 17:09:59,104][root][INFO] - door : 0.581 0.768 0.878 [2020-12-31 17:09:59,104][root][INFO] - floor : 0.910 0.954 0.954 [2020-12-31 17:09:59,104][root][INFO] - sofa : 0.456 0.636 0.809 [2020-12-31 17:09:59,104][root][INFO] - table : 0.190 0.379 0.610 [2020-12-31 17:09:59,104][root][INFO] - wall : 0.477 0.708 0.802 [2020-12-31 17:09:59,104][root][INFO] - window : 0.484 0.730 0.780 [2020-12-31 17:09:59,104][root][INFO] - ---------------------------------------------------------------- [2020-12-31 17:09:59,104][root][INFO] - average : 0.479 0.616 0.700 [2020-12-31 17:09:59,104][root][INFO] - [2020-12-31 17:09:59,249][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 17:09:59,798][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter14000.pth [2020-12-31 17:10:10,960][root][INFO] - ===> Epoch[2801](14005/5): Loss -0.8133, Sem -0.8133, dir -0.9818, norm 0.1439 LR: 8.708e-03 Score 99.177 Data time: 0.1049, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.1323 [2020-12-31 17:10:22,743][root][INFO] - ===> Epoch[2802](14010/5): Loss -0.8078, Sem -0.8078, dir -0.9810, norm 0.1468 LR: 8.669e-03 Score 99.114 Data time: 0.2049, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.2640 [2020-12-31 17:10:34,730][root][INFO] - ===> Epoch[2803](14015/5): Loss -0.8059, Sem -0.8059, dir -0.9807, norm 0.1489 LR: 8.629e-03 Score 99.132 Data time: 0.1026, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3014 [2020-12-31 17:10:46,971][root][INFO] - ===> Epoch[2804](14020/5): Loss -0.8136, Sem -0.8136, dir -0.9819, norm 0.1435 LR: 8.590e-03 Score 99.165 Data time: 0.1170, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3602 [2020-12-31 17:10:57,920][root][INFO] - ===> Epoch[2805](14025/5): Loss -0.8084, Sem -0.8084, dir -0.9805, norm 0.1452 LR: 8.551e-03 Score 99.088 Data time: 0.1021, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.1112 [2020-12-31 17:11:10,186][root][INFO] - ===> Epoch[2806](14030/5): Loss -0.8081, Sem -0.8081, dir -0.9811, norm 0.1474 LR: 8.511e-03 Score 99.142 Data time: 0.1097, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.3402 [2020-12-31 17:11:22,623][root][INFO] - ===> Epoch[2807](14035/5): Loss -0.8096, Sem -0.8096, dir -0.9812, norm 0.1461 LR: 8.472e-03 Score 99.142 Data time: 0.1000, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3922 [2020-12-31 17:11:33,965][root][INFO] - ===> Epoch[2808](14040/5): Loss -0.8081, Sem -0.8081, dir -0.9810, norm 0.1469 LR: 8.432e-03 Score 99.130 Data time: 0.1152, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.1784 [2020-12-31 17:11:46,285][root][INFO] - ===> Epoch[2809](14045/5): Loss -0.8076, Sem -0.8076, dir -0.9812, norm 0.1481 LR: 8.393e-03 Score 99.147 Data time: 0.1310, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.3645 [2020-12-31 17:11:58,464][root][INFO] - ===> Epoch[2810](14050/5): Loss -0.8055, Sem -0.8055, dir -0.9809, norm 0.1502 LR: 8.353e-03 Score 99.151 Data time: 0.1246, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.3319 [2020-12-31 17:12:09,879][root][INFO] - ===> Epoch[2811](14055/5): Loss -0.8157, Sem -0.8157, dir -0.9814, norm 0.1411 LR: 8.314e-03 Score 99.168 Data time: 0.1461, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.1826 [2020-12-31 17:12:22,325][root][INFO] - ===> Epoch[2812](14060/5): Loss -0.8027, Sem -0.8027, dir -0.9802, norm 0.1504 LR: 8.274e-03 Score 99.095 Data time: 0.1152, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.3776 [2020-12-31 17:12:35,042][root][INFO] - ===> Epoch[2813](14065/5): Loss -0.8070, Sem -0.8070, dir -0.9816, norm 0.1489 LR: 8.235e-03 Score 99.158 Data time: 0.1097, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.4448 [2020-12-31 17:12:47,747][root][INFO] - ===> Epoch[2814](14070/5): Loss -0.8030, Sem -0.8030, dir -0.9807, norm 0.1514 LR: 8.195e-03 Score 99.122 Data time: 0.0927, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.4451 [2020-12-31 17:13:00,877][root][INFO] - ===> Epoch[2815](14075/5): Loss -0.8100, Sem -0.8100, dir -0.9814, norm 0.1464 LR: 8.155e-03 Score 99.165 Data time: 0.1503, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.5349 [2020-12-31 17:13:12,192][root][INFO] - ===> Epoch[2816](14080/5): Loss -0.8048, Sem -0.8048, dir -0.9812, norm 0.1510 LR: 8.116e-03 Score 99.141 Data time: 0.0999, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.1770 [2020-12-31 17:13:25,300][root][INFO] - ===> Epoch[2817](14085/5): Loss -0.8106, Sem -0.8106, dir -0.9818, norm 0.1454 LR: 8.076e-03 Score 99.146 Data time: 0.1006, Forward time: 0.9470, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.5319 [2020-12-31 17:13:37,599][root][INFO] - ===> Epoch[2818](14090/5): Loss -0.8142, Sem -0.8142, dir -0.9817, norm 0.1419 LR: 8.036e-03 Score 99.138 Data time: 0.1052, Forward time: 0.9469, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3746 [2020-12-31 17:13:50,885][root][INFO] - ===> Epoch[2819](14095/5): Loss -0.8081, Sem -0.8081, dir -0.9813, norm 0.1485 LR: 7.997e-03 Score 99.180 Data time: 0.1111, Forward time: 0.9470, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.5642 [2020-12-31 17:14:03,725][root][INFO] - ===> Epoch[2820](14100/5): Loss -0.8099, Sem -0.8099, dir -0.9810, norm 0.1453 LR: 7.957e-03 Score 99.132 Data time: 0.1213, Forward time: 0.9469, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.4792 [2020-12-31 17:14:15,292][root][INFO] - ===> Epoch[2821](14105/5): Loss -0.8057, Sem -0.8057, dir -0.9811, norm 0.1484 LR: 7.917e-03 Score 99.093 Data time: 0.1192, Forward time: 0.9469, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2160 [2020-12-31 17:14:27,520][root][INFO] - ===> Epoch[2822](14110/5): Loss -0.8124, Sem -0.8124, dir -0.9815, norm 0.1444 LR: 7.877e-03 Score 99.168 Data time: 0.1646, Forward time: 0.9469, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3590 [2020-12-31 17:14:40,261][root][INFO] - ===> Epoch[2823](14115/5): Loss -0.8062, Sem -0.8062, dir -0.9807, norm 0.1486 LR: 7.838e-03 Score 99.143 Data time: 0.1011, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.4595 [2020-12-31 17:14:52,419][root][INFO] - ===> Epoch[2824](14120/5): Loss -0.8121, Sem -0.8121, dir -0.9818, norm 0.1444 LR: 7.798e-03 Score 99.142 Data time: 0.0827, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3451 [2020-12-31 17:15:04,396][root][INFO] - ===> Epoch[2825](14125/5): Loss -0.8094, Sem -0.8094, dir -0.9806, norm 0.1457 LR: 7.758e-03 Score 99.146 Data time: 0.1153, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3057 [2020-12-31 17:15:15,850][root][INFO] - ===> Epoch[2826](14130/5): Loss -0.8089, Sem -0.8089, dir -0.9818, norm 0.1474 LR: 7.718e-03 Score 99.139 Data time: 0.1013, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2031 [2020-12-31 17:15:27,909][root][INFO] - ===> Epoch[2827](14135/5): Loss -0.8105, Sem -0.8105, dir -0.9811, norm 0.1451 LR: 7.678e-03 Score 99.143 Data time: 0.0906, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3171 [2020-12-31 17:15:40,341][root][INFO] - ===> Epoch[2828](14140/5): Loss -0.8060, Sem -0.8060, dir -0.9811, norm 0.1496 LR: 7.638e-03 Score 99.139 Data time: 0.1248, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3875 [2020-12-31 17:15:53,061][root][INFO] - ===> Epoch[2829](14145/5): Loss -0.8102, Sem -0.8102, dir -0.9814, norm 0.1458 LR: 7.598e-03 Score 99.152 Data time: 0.1107, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.4528 [2020-12-31 17:16:05,279][root][INFO] - ===> Epoch[2830](14150/5): Loss -0.8068, Sem -0.8068, dir -0.9811, norm 0.1487 LR: 7.558e-03 Score 99.145 Data time: 0.1279, Forward time: 0.9470, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3512 [2020-12-31 17:16:16,702][root][INFO] - ===> Epoch[2831](14155/5): Loss -0.8177, Sem -0.8177, dir -0.9821, norm 0.1396 LR: 7.518e-03 Score 99.166 Data time: 0.2034, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.1831 [2020-12-31 17:16:29,511][root][INFO] - ===> Epoch[2832](14160/5): Loss -0.8039, Sem -0.8039, dir -0.9810, norm 0.1513 LR: 7.478e-03 Score 99.143 Data time: 0.1009, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.4701 [2020-12-31 17:16:42,077][root][INFO] - ===> Epoch[2833](14165/5): Loss -0.8096, Sem -0.8096, dir -0.9810, norm 0.1465 LR: 7.438e-03 Score 99.162 Data time: 0.1196, Forward time: 0.9469, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.4209 [2020-12-31 17:16:53,537][root][INFO] - ===> Epoch[2834](14170/5): Loss -0.8148, Sem -0.8148, dir -0.9819, norm 0.1427 LR: 7.398e-03 Score 99.169 Data time: 0.1341, Forward time: 0.9470, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2034 [2020-12-31 17:17:05,613][root][INFO] - ===> Epoch[2835](14175/5): Loss -0.8098, Sem -0.8098, dir -0.9812, norm 0.1460 LR: 7.358e-03 Score 99.147 Data time: 0.1109, Forward time: 0.9470, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3224 [2020-12-31 17:17:17,183][root][INFO] - ===> Epoch[2836](14180/5): Loss -0.8065, Sem -0.8065, dir -0.9805, norm 0.1481 LR: 7.318e-03 Score 99.138 Data time: 0.1401, Forward time: 0.9470, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2238 [2020-12-31 17:17:29,928][root][INFO] - ===> Epoch[2837](14185/5): Loss -0.8114, Sem -0.8114, dir -0.9816, norm 0.1449 LR: 7.278e-03 Score 99.150 Data time: 0.1608, Forward time: 0.9470, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.4497 [2020-12-31 17:17:41,959][root][INFO] - ===> Epoch[2838](14190/5): Loss -0.8018, Sem -0.8018, dir -0.9808, norm 0.1524 LR: 7.238e-03 Score 99.110 Data time: 0.1333, Forward time: 0.9470, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3043 [2020-12-31 17:17:54,039][root][INFO] - ===> Epoch[2839](14195/5): Loss -0.8153, Sem -0.8153, dir -0.9814, norm 0.1414 LR: 7.198e-03 Score 99.166 Data time: 0.1095, Forward time: 0.9470, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3372 [2020-12-31 17:18:06,412][root][INFO] - ===> Epoch[2840](14200/5): Loss -0.8105, Sem -0.8105, dir -0.9813, norm 0.1451 LR: 7.157e-03 Score 99.130 Data time: 0.1261, Forward time: 0.9470, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3852 [2020-12-31 17:18:06,416][root][INFO] - ===> Start testing [2020-12-31 17:18:45,595][root][INFO] - 68/68: Data time: 0.0023, Iter time: 0.1132 Loss 0.582 (AVG: 0.541) Score 89.772 (AVG: 90.057) mIOU 68.950 mAP 83.412 mAcc 75.857 IOU: 60.645 0.000 82.590 73.301 94.002 90.885 31.090 77.160 98.339 65.484 81.172 85.338 56.349 mAP: 80.850 15.711 92.856 86.328 98.919 98.479 57.093 95.339 99.830 80.839 93.070 94.125 90.912 mAcc: 81.525 0.000 89.267 80.413 95.161 96.422 33.508 87.686 99.065 76.460 90.751 95.644 60.239 [2020-12-31 17:18:45,596][root][INFO] - Finished test. Elapsed time: 39.1786 [2020-12-31 17:18:48,785][root][INFO] - [2020-12-31 17:18:48,785][root][INFO] - ################################################################ [2020-12-31 17:18:48,785][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 17:18:48,786][root][INFO] - ################################################################ [2020-12-31 17:18:48,786][root][INFO] - clutter : nan nan nan [2020-12-31 17:18:48,786][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 17:18:48,786][root][INFO] - board : 0.780 0.868 0.898 [2020-12-31 17:18:48,786][root][INFO] - bookcase : 0.200 0.339 0.487 [2020-12-31 17:18:48,786][root][INFO] - ceiling : 0.600 0.763 0.791 [2020-12-31 17:18:48,786][root][INFO] - chair : 0.796 0.892 0.900 [2020-12-31 17:18:48,786][root][INFO] - column : 0.161 0.252 0.380 [2020-12-31 17:18:48,786][root][INFO] - door : 0.579 0.768 0.938 [2020-12-31 17:18:48,786][root][INFO] - floor : 0.904 0.963 0.967 [2020-12-31 17:18:48,786][root][INFO] - sofa : 0.450 0.636 0.727 [2020-12-31 17:18:48,786][root][INFO] - table : 0.202 0.396 0.651 [2020-12-31 17:18:48,786][root][INFO] - wall : 0.458 0.671 0.767 [2020-12-31 17:18:48,786][root][INFO] - window : 0.506 0.729 0.780 [2020-12-31 17:18:48,786][root][INFO] - ---------------------------------------------------------------- [2020-12-31 17:18:48,786][root][INFO] - average : 0.470 0.606 0.690 [2020-12-31 17:18:48,786][root][INFO] - [2020-12-31 17:18:48,936][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 17:18:49,548][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_miou.pth [2020-12-31 17:18:49,548][root][INFO] - Current best mIoU: 68.950 at iter 1 [2020-12-31 17:18:50,145][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter14200.pth [2020-12-31 17:19:02,318][root][INFO] - ===> Epoch[2841](14205/5): Loss -0.8124, Sem -0.8124, dir -0.9811, norm 0.1436 LR: 7.117e-03 Score 99.158 Data time: 0.0900, Forward time: 0.9470, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3509 [2020-12-31 17:19:13,965][root][INFO] - ===> Epoch[2842](14210/5): Loss -0.8141, Sem -0.8141, dir -0.9819, norm 0.1432 LR: 7.077e-03 Score 99.166 Data time: 0.1200, Forward time: 0.9470, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2359 [2020-12-31 17:19:25,806][root][INFO] - ===> Epoch[2843](14215/5): Loss -0.8075, Sem -0.8075, dir -0.9811, norm 0.1478 LR: 7.037e-03 Score 99.141 Data time: 0.0914, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2803 [2020-12-31 17:19:37,001][root][INFO] - ===> Epoch[2844](14220/5): Loss -0.8086, Sem -0.8086, dir -0.9815, norm 0.1467 LR: 6.996e-03 Score 99.122 Data time: 0.0916, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.1483 [2020-12-31 17:19:48,569][root][INFO] - ===> Epoch[2845](14225/5): Loss -0.8117, Sem -0.8117, dir -0.9812, norm 0.1433 LR: 6.956e-03 Score 99.126 Data time: 0.1033, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2258 [2020-12-31 17:20:00,361][root][INFO] - ===> Epoch[2846](14230/5): Loss -0.8106, Sem -0.8106, dir -0.9811, norm 0.1456 LR: 6.916e-03 Score 99.160 Data time: 0.1114, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2567 [2020-12-31 17:20:12,296][root][INFO] - ===> Epoch[2847](14235/5): Loss -0.8114, Sem -0.8114, dir -0.9816, norm 0.1448 LR: 6.875e-03 Score 99.149 Data time: 0.0965, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2901 [2020-12-31 17:20:24,684][root][INFO] - ===> Epoch[2848](14240/5): Loss -0.8048, Sem -0.8048, dir -0.9811, norm 0.1506 LR: 6.835e-03 Score 99.134 Data time: 0.1313, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3830 [2020-12-31 17:20:37,220][root][INFO] - ===> Epoch[2849](14245/5): Loss -0.8129, Sem -0.8129, dir -0.9816, norm 0.1437 LR: 6.795e-03 Score 99.161 Data time: 0.0991, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.4337 [2020-12-31 17:20:49,038][root][INFO] - ===> Epoch[2850](14250/5): Loss -0.8026, Sem -0.8026, dir -0.9807, norm 0.1523 LR: 6.754e-03 Score 99.130 Data time: 0.1153, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2717 [2020-12-31 17:21:00,687][root][INFO] - ===> Epoch[2851](14255/5): Loss -0.8140, Sem -0.8140, dir -0.9810, norm 0.1416 LR: 6.714e-03 Score 99.139 Data time: 0.0926, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2505 [2020-12-31 17:21:11,987][root][INFO] - ===> Epoch[2852](14260/5): Loss -0.8057, Sem -0.8057, dir -0.9810, norm 0.1502 LR: 6.673e-03 Score 99.164 Data time: 0.1381, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.1673 [2020-12-31 17:21:23,370][root][INFO] - ===> Epoch[2853](14265/5): Loss -0.8110, Sem -0.8110, dir -0.9810, norm 0.1444 LR: 6.633e-03 Score 99.144 Data time: 0.1157, Forward time: 0.9469, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.1898 [2020-12-31 17:21:35,545][root][INFO] - ===> Epoch[2854](14270/5): Loss -0.8124, Sem -0.8124, dir -0.9815, norm 0.1437 LR: 6.592e-03 Score 99.147 Data time: 0.1141, Forward time: 0.9469, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3537 [2020-12-31 17:21:47,088][root][INFO] - ===> Epoch[2855](14275/5): Loss -0.8096, Sem -0.8096, dir -0.9815, norm 0.1461 LR: 6.551e-03 Score 99.138 Data time: 0.1376, Forward time: 0.9469, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2273 [2020-12-31 17:21:59,426][root][INFO] - ===> Epoch[2856](14280/5): Loss -0.8068, Sem -0.8068, dir -0.9808, norm 0.1485 LR: 6.511e-03 Score 99.142 Data time: 0.1356, Forward time: 0.9469, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3803 [2020-12-31 17:22:11,871][root][INFO] - ===> Epoch[2857](14285/5): Loss -0.8094, Sem -0.8094, dir -0.9809, norm 0.1460 LR: 6.470e-03 Score 99.143 Data time: 0.1318, Forward time: 0.9469, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3980 [2020-12-31 17:22:24,146][root][INFO] - ===> Epoch[2858](14290/5): Loss -0.8140, Sem -0.8140, dir -0.9817, norm 0.1427 LR: 6.429e-03 Score 99.163 Data time: 0.0985, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3717 [2020-12-31 17:22:35,830][root][INFO] - ===> Epoch[2859](14295/5): Loss -0.8111, Sem -0.8111, dir -0.9810, norm 0.1435 LR: 6.389e-03 Score 99.142 Data time: 0.1640, Forward time: 0.9469, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2671 [2020-12-31 17:22:47,562][root][INFO] - ===> Epoch[2860](14300/5): Loss -0.8126, Sem -0.8126, dir -0.9814, norm 0.1437 LR: 6.348e-03 Score 99.150 Data time: 0.0853, Forward time: 0.9468, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2559 [2020-12-31 17:22:59,726][root][INFO] - ===> Epoch[2861](14305/5): Loss -0.8089, Sem -0.8089, dir -0.9812, norm 0.1468 LR: 6.307e-03 Score 99.149 Data time: 0.1135, Forward time: 0.9468, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3390 [2020-12-31 17:23:11,780][root][INFO] - ===> Epoch[2862](14310/5): Loss -0.8134, Sem -0.8134, dir -0.9816, norm 0.1434 LR: 6.266e-03 Score 99.166 Data time: 0.0954, Forward time: 0.9468, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3132 [2020-12-31 17:23:23,510][root][INFO] - ===> Epoch[2863](14315/5): Loss -0.8111, Sem -0.8111, dir -0.9818, norm 0.1456 LR: 6.226e-03 Score 99.157 Data time: 0.1046, Forward time: 0.9468, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2472 [2020-12-31 17:23:34,947][root][INFO] - ===> Epoch[2864](14320/5): Loss -0.8083, Sem -0.8083, dir -0.9808, norm 0.1472 LR: 6.185e-03 Score 99.152 Data time: 0.1228, Forward time: 0.9468, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.1845 [2020-12-31 17:23:46,291][root][INFO] - ===> Epoch[2865](14325/5): Loss -0.8130, Sem -0.8130, dir -0.9819, norm 0.1433 LR: 6.144e-03 Score 99.134 Data time: 0.1863, Forward time: 0.9468, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.1944 [2020-12-31 17:23:58,884][root][INFO] - ===> Epoch[2866](14330/5): Loss -0.8101, Sem -0.8101, dir -0.9814, norm 0.1463 LR: 6.103e-03 Score 99.157 Data time: 0.0940, Forward time: 0.9468, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.4274 [2020-12-31 17:24:09,910][root][INFO] - ===> Epoch[2867](14335/5): Loss -0.8130, Sem -0.8130, dir -0.9815, norm 0.1431 LR: 6.062e-03 Score 99.146 Data time: 0.1269, Forward time: 0.9468, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.1184 [2020-12-31 17:24:21,724][root][INFO] - ===> Epoch[2868](14340/5): Loss -0.8104, Sem -0.8104, dir -0.9814, norm 0.1459 LR: 6.021e-03 Score 99.154 Data time: 0.1330, Forward time: 0.9468, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2817 [2020-12-31 17:24:33,712][root][INFO] - ===> Epoch[2869](14345/5): Loss -0.8148, Sem -0.8148, dir -0.9820, norm 0.1426 LR: 5.980e-03 Score 99.171 Data time: 0.1072, Forward time: 0.9467, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3122 [2020-12-31 17:24:45,467][root][INFO] - ===> Epoch[2870](14350/5): Loss -0.8134, Sem -0.8134, dir -0.9812, norm 0.1425 LR: 5.939e-03 Score 99.155 Data time: 0.1204, Forward time: 0.9468, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2608 [2020-12-31 17:24:57,543][root][INFO] - ===> Epoch[2871](14355/5): Loss -0.8160, Sem -0.8160, dir -0.9815, norm 0.1407 LR: 5.898e-03 Score 99.171 Data time: 0.1166, Forward time: 0.9467, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3323 [2020-12-31 17:25:10,151][root][INFO] - ===> Epoch[2872](14360/5): Loss -0.8119, Sem -0.8119, dir -0.9818, norm 0.1449 LR: 5.857e-03 Score 99.164 Data time: 0.1549, Forward time: 0.9467, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.4221 [2020-12-31 17:25:22,378][root][INFO] - ===> Epoch[2873](14365/5): Loss -0.8125, Sem -0.8125, dir -0.9810, norm 0.1431 LR: 5.816e-03 Score 99.149 Data time: 0.1460, Forward time: 0.9467, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3579 [2020-12-31 17:25:33,657][root][INFO] - ===> Epoch[2874](14370/5): Loss -0.8104, Sem -0.8104, dir -0.9814, norm 0.1453 LR: 5.775e-03 Score 99.140 Data time: 0.1885, Forward time: 0.9467, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.1598 [2020-12-31 17:25:45,294][root][INFO] - ===> Epoch[2875](14375/5): Loss -0.8084, Sem -0.8084, dir -0.9814, norm 0.1472 LR: 5.733e-03 Score 99.146 Data time: 0.1348, Forward time: 0.9467, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2431 [2020-12-31 17:25:57,409][root][INFO] - ===> Epoch[2876](14380/5): Loss -0.8144, Sem -0.8144, dir -0.9818, norm 0.1423 LR: 5.692e-03 Score 99.150 Data time: 0.0914, Forward time: 0.9467, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3249 [2020-12-31 17:26:09,785][root][INFO] - ===> Epoch[2877](14385/5): Loss -0.8108, Sem -0.8108, dir -0.9810, norm 0.1450 LR: 5.651e-03 Score 99.150 Data time: 0.1175, Forward time: 0.9467, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3838 [2020-12-31 17:26:22,640][root][INFO] - ===> Epoch[2878](14390/5): Loss -0.8180, Sem -0.8180, dir -0.9824, norm 0.1405 LR: 5.610e-03 Score 99.197 Data time: 0.1103, Forward time: 0.9468, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.4915 [2020-12-31 17:26:34,966][root][INFO] - ===> Epoch[2879](14395/5): Loss -0.8115, Sem -0.8115, dir -0.9810, norm 0.1433 LR: 5.568e-03 Score 99.122 Data time: 0.1306, Forward time: 0.9468, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3778 [2020-12-31 17:26:46,578][root][INFO] - ===> Epoch[2880](14400/5): Loss -0.8147, Sem -0.8147, dir -0.9813, norm 0.1414 LR: 5.527e-03 Score 99.147 Data time: 0.0888, Forward time: 0.9467, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2306 [2020-12-31 17:26:46,580][root][INFO] - ===> Start testing [2020-12-31 17:27:27,120][root][INFO] - 68/68: Data time: 0.0024, Iter time: 0.1351 Loss 0.440 (AVG: 0.539) Score 90.450 (AVG: 90.007) mIOU 68.723 mAP 83.096 mAcc 76.041 IOU: 60.520 0.000 81.569 73.254 93.827 90.665 30.612 77.625 98.293 64.175 80.995 85.253 56.610 mAP: 80.743 11.463 92.930 85.979 98.880 98.352 56.445 95.650 99.829 81.679 93.157 94.042 91.098 mAcc: 80.809 0.000 90.018 80.310 95.023 96.566 32.779 88.657 98.964 78.187 90.947 95.716 60.551 [2020-12-31 17:27:27,122][root][INFO] - Finished test. Elapsed time: 40.5396 [2020-12-31 17:27:30,098][root][INFO] - [2020-12-31 17:27:30,098][root][INFO] - ################################################################ [2020-12-31 17:27:30,098][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 17:27:30,098][root][INFO] - ################################################################ [2020-12-31 17:27:30,098][root][INFO] - clutter : nan nan nan [2020-12-31 17:27:30,098][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 17:27:30,098][root][INFO] - board : 0.792 0.868 0.896 [2020-12-31 17:27:30,098][root][INFO] - bookcase : 0.210 0.352 0.507 [2020-12-31 17:27:30,098][root][INFO] - ceiling : 0.677 0.829 0.852 [2020-12-31 17:27:30,098][root][INFO] - chair : 0.797 0.898 0.907 [2020-12-31 17:27:30,098][root][INFO] - column : 0.167 0.272 0.391 [2020-12-31 17:27:30,098][root][INFO] - door : 0.624 0.799 0.917 [2020-12-31 17:27:30,098][root][INFO] - floor : 0.899 0.960 0.978 [2020-12-31 17:27:30,098][root][INFO] - sofa : 0.450 0.636 0.801 [2020-12-31 17:27:30,098][root][INFO] - table : 0.207 0.421 0.676 [2020-12-31 17:27:30,098][root][INFO] - wall : 0.466 0.688 0.779 [2020-12-31 17:27:30,098][root][INFO] - window : 0.500 0.729 0.780 [2020-12-31 17:27:30,098][root][INFO] - ---------------------------------------------------------------- [2020-12-31 17:27:30,099][root][INFO] - average : 0.482 0.621 0.707 [2020-12-31 17:27:30,099][root][INFO] - [2020-12-31 17:27:30,264][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 17:27:30,812][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter14400.pth [2020-12-31 17:27:41,699][root][INFO] - ===> Epoch[2881](14405/5): Loss -0.8141, Sem -0.8141, dir -0.9811, norm 0.1408 LR: 5.485e-03 Score 99.116 Data time: 0.1048, Forward time: 0.9467, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.0804 [2020-12-31 17:27:53,787][root][INFO] - ===> Epoch[2882](14410/5): Loss -0.8103, Sem -0.8103, dir -0.9814, norm 0.1458 LR: 5.444e-03 Score 99.149 Data time: 0.1053, Forward time: 0.9467, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.3292 [2020-12-31 17:28:05,639][root][INFO] - ===> Epoch[2883](14415/5): Loss -0.8076, Sem -0.8076, dir -0.9806, norm 0.1462 LR: 5.403e-03 Score 99.111 Data time: 0.1324, Forward time: 0.9466, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2974 [2020-12-31 17:28:18,375][root][INFO] - ===> Epoch[2884](14420/5): Loss -0.8078, Sem -0.8078, dir -0.9815, norm 0.1480 LR: 5.361e-03 Score 99.140 Data time: 0.0785, Forward time: 0.9466, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.4510 [2020-12-31 17:28:29,237][root][INFO] - ===> Epoch[2885](14425/5): Loss -0.8103, Sem -0.8103, dir -0.9815, norm 0.1453 LR: 5.320e-03 Score 99.144 Data time: 0.0755, Forward time: 0.9466, Backward time: 1.1872, DDP time: 0.0050, Total iter time: 2.0835 [2020-12-31 17:28:41,052][root][INFO] - ===> Epoch[2886](14430/5): Loss -0.8106, Sem -0.8106, dir -0.9815, norm 0.1456 LR: 5.278e-03 Score 99.150 Data time: 0.1324, Forward time: 0.9466, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2856 [2020-12-31 17:28:52,051][root][INFO] - ===> Epoch[2887](14435/5): Loss -0.8103, Sem -0.8103, dir -0.9811, norm 0.1450 LR: 5.236e-03 Score 99.133 Data time: 0.0973, Forward time: 0.9466, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.1179 [2020-12-31 17:29:03,665][root][INFO] - ===> Epoch[2888](14440/5): Loss -0.8048, Sem -0.8048, dir -0.9808, norm 0.1494 LR: 5.195e-03 Score 99.109 Data time: 0.1252, Forward time: 0.9466, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.2385 [2020-12-31 17:29:15,075][root][INFO] - ===> Epoch[2889](14445/5): Loss -0.8148, Sem -0.8148, dir -0.9817, norm 0.1425 LR: 5.153e-03 Score 99.178 Data time: 0.1047, Forward time: 0.9466, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.1973 [2020-12-31 17:29:27,197][root][INFO] - ===> Epoch[2890](14450/5): Loss -0.8108, Sem -0.8108, dir -0.9813, norm 0.1449 LR: 5.111e-03 Score 99.140 Data time: 0.0935, Forward time: 0.9466, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3363 [2020-12-31 17:29:38,909][root][INFO] - ===> Epoch[2891](14455/5): Loss -0.8117, Sem -0.8117, dir -0.9811, norm 0.1439 LR: 5.070e-03 Score 99.147 Data time: 0.0994, Forward time: 0.9466, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2512 [2020-12-31 17:29:50,263][root][INFO] - ===> Epoch[2892](14460/5): Loss -0.8157, Sem -0.8157, dir -0.9820, norm 0.1419 LR: 5.028e-03 Score 99.182 Data time: 0.1354, Forward time: 0.9466, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.1801 [2020-12-31 17:30:03,206][root][INFO] - ===> Epoch[2893](14465/5): Loss -0.8140, Sem -0.8140, dir -0.9812, norm 0.1425 LR: 4.986e-03 Score 99.172 Data time: 0.1579, Forward time: 0.9466, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.4868 [2020-12-31 17:30:14,519][root][INFO] - ===> Epoch[2894](14470/5): Loss -0.8077, Sem -0.8077, dir -0.9808, norm 0.1472 LR: 4.944e-03 Score 99.136 Data time: 0.1048, Forward time: 0.9466, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.1653 [2020-12-31 17:30:26,116][root][INFO] - ===> Epoch[2895](14475/5): Loss -0.8154, Sem -0.8154, dir -0.9819, norm 0.1417 LR: 4.902e-03 Score 99.162 Data time: 0.1013, Forward time: 0.9466, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2178 [2020-12-31 17:30:37,413][root][INFO] - ===> Epoch[2896](14480/5): Loss -0.8098, Sem -0.8098, dir -0.9807, norm 0.1446 LR: 4.860e-03 Score 99.115 Data time: 0.1055, Forward time: 0.9466, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.1739 [2020-12-31 17:30:49,998][root][INFO] - ===> Epoch[2897](14485/5): Loss -0.8088, Sem -0.8088, dir -0.9810, norm 0.1459 LR: 4.818e-03 Score 99.116 Data time: 0.0905, Forward time: 0.9465, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.4293 [2020-12-31 17:31:02,798][root][INFO] - ===> Epoch[2898](14490/5): Loss -0.8120, Sem -0.8120, dir -0.9817, norm 0.1444 LR: 4.776e-03 Score 99.158 Data time: 0.0982, Forward time: 0.9465, Backward time: 1.1871, DDP time: 0.0050, Total iter time: 2.4564 [2020-12-31 17:31:14,677][root][INFO] - ===> Epoch[2899](14495/5): Loss -0.8134, Sem -0.8134, dir -0.9815, norm 0.1425 LR: 4.734e-03 Score 99.141 Data time: 0.1012, Forward time: 0.9465, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2877 [2020-12-31 17:31:26,537][root][INFO] - ===> Epoch[2900](14500/5): Loss -0.8154, Sem -0.8154, dir -0.9819, norm 0.1417 LR: 4.692e-03 Score 99.171 Data time: 0.1447, Forward time: 0.9466, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2747 [2020-12-31 17:31:38,114][root][INFO] - ===> Epoch[2901](14505/5): Loss -0.8148, Sem -0.8148, dir -0.9811, norm 0.1410 LR: 4.650e-03 Score 99.146 Data time: 0.1068, Forward time: 0.9466, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2168 [2020-12-31 17:31:49,755][root][INFO] - ===> Epoch[2902](14510/5): Loss -0.8096, Sem -0.8096, dir -0.9812, norm 0.1467 LR: 4.607e-03 Score 99.168 Data time: 0.1486, Forward time: 0.9466, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2276 [2020-12-31 17:32:02,750][root][INFO] - ===> Epoch[2903](14515/5): Loss -0.8135, Sem -0.8135, dir -0.9820, norm 0.1436 LR: 4.565e-03 Score 99.161 Data time: 0.1045, Forward time: 0.9466, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.5196 [2020-12-31 17:32:14,764][root][INFO] - ===> Epoch[2904](14520/5): Loss -0.8167, Sem -0.8167, dir -0.9817, norm 0.1406 LR: 4.523e-03 Score 99.175 Data time: 0.0960, Forward time: 0.9466, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3009 [2020-12-31 17:32:26,297][root][INFO] - ===> Epoch[2905](14525/5): Loss -0.8114, Sem -0.8114, dir -0.9809, norm 0.1437 LR: 4.481e-03 Score 99.139 Data time: 0.0807, Forward time: 0.9465, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2119 [2020-12-31 17:32:37,923][root][INFO] - ===> Epoch[2906](14530/5): Loss -0.8170, Sem -0.8170, dir -0.9821, norm 0.1399 LR: 4.438e-03 Score 99.159 Data time: 0.2415, Forward time: 0.9466, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2276 [2020-12-31 17:32:49,533][root][INFO] - ===> Epoch[2907](14535/5): Loss -0.8097, Sem -0.8097, dir -0.9812, norm 0.1457 LR: 4.396e-03 Score 99.140 Data time: 0.1040, Forward time: 0.9466, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2369 [2020-12-31 17:33:01,528][root][INFO] - ===> Epoch[2908](14540/5): Loss -0.8157, Sem -0.8157, dir -0.9817, norm 0.1419 LR: 4.353e-03 Score 99.195 Data time: 0.1035, Forward time: 0.9466, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3139 [2020-12-31 17:33:13,366][root][INFO] - ===> Epoch[2909](14545/5): Loss -0.8128, Sem -0.8128, dir -0.9813, norm 0.1434 LR: 4.311e-03 Score 99.148 Data time: 0.0878, Forward time: 0.9466, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2746 [2020-12-31 17:33:24,613][root][INFO] - ===> Epoch[2910](14550/5): Loss -0.8154, Sem -0.8154, dir -0.9821, norm 0.1406 LR: 4.268e-03 Score 99.131 Data time: 0.1040, Forward time: 0.9465, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.1629 [2020-12-31 17:33:37,432][root][INFO] - ===> Epoch[2911](14555/5): Loss -0.8116, Sem -0.8116, dir -0.9812, norm 0.1444 LR: 4.226e-03 Score 99.157 Data time: 0.1355, Forward time: 0.9466, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.4709 [2020-12-31 17:33:49,224][root][INFO] - ===> Epoch[2912](14560/5): Loss -0.8137, Sem -0.8137, dir -0.9815, norm 0.1431 LR: 4.183e-03 Score 99.166 Data time: 0.0940, Forward time: 0.9466, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2876 [2020-12-31 17:34:01,482][root][INFO] - ===> Epoch[2913](14565/5): Loss -0.8134, Sem -0.8134, dir -0.9814, norm 0.1432 LR: 4.140e-03 Score 99.167 Data time: 0.2165, Forward time: 0.9466, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3597 [2020-12-31 17:34:13,599][root][INFO] - ===> Epoch[2914](14570/5): Loss -0.8161, Sem -0.8161, dir -0.9816, norm 0.1401 LR: 4.098e-03 Score 99.139 Data time: 0.1037, Forward time: 0.9466, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3302 [2020-12-31 17:34:25,438][root][INFO] - ===> Epoch[2915](14575/5): Loss -0.8078, Sem -0.8078, dir -0.9807, norm 0.1454 LR: 4.055e-03 Score 99.120 Data time: 0.0853, Forward time: 0.9466, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2880 [2020-12-31 17:34:37,798][root][INFO] - ===> Epoch[2916](14580/5): Loss -0.8150, Sem -0.8150, dir -0.9816, norm 0.1422 LR: 4.012e-03 Score 99.176 Data time: 0.0688, Forward time: 0.9466, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3887 [2020-12-31 17:34:49,474][root][INFO] - ===> Epoch[2917](14585/5): Loss -0.8130, Sem -0.8130, dir -0.9815, norm 0.1426 LR: 3.969e-03 Score 99.131 Data time: 0.1069, Forward time: 0.9465, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2363 [2020-12-31 17:35:01,791][root][INFO] - ===> Epoch[2918](14590/5): Loss -0.8098, Sem -0.8098, dir -0.9812, norm 0.1459 LR: 3.926e-03 Score 99.145 Data time: 0.0843, Forward time: 0.9465, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3788 [2020-12-31 17:35:13,816][root][INFO] - ===> Epoch[2919](14595/5): Loss -0.8104, Sem -0.8104, dir -0.9812, norm 0.1447 LR: 3.883e-03 Score 99.126 Data time: 0.0904, Forward time: 0.9465, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.3151 [2020-12-31 17:35:25,666][root][INFO] - ===> Epoch[2920](14600/5): Loss -0.8179, Sem -0.8179, dir -0.9820, norm 0.1397 LR: 3.840e-03 Score 99.171 Data time: 0.1066, Forward time: 0.9465, Backward time: 1.1870, DDP time: 0.0050, Total iter time: 2.2981 [2020-12-31 17:35:25,669][root][INFO] - ===> Start testing [2020-12-31 17:36:05,467][root][INFO] - 68/68: Data time: 0.0019, Iter time: 0.0991 Loss 0.118 (AVG: 0.540) Score 96.931 (AVG: 89.982) mIOU 68.454 mAP 82.913 mAcc 75.844 IOU: 60.518 0.000 80.893 73.244 93.738 89.996 30.316 76.356 98.334 63.215 81.349 85.256 56.692 mAP: 81.040 11.392 92.831 86.135 98.935 98.381 56.308 95.036 99.831 79.596 93.171 94.146 91.074 mAcc: 80.459 0.000 89.758 80.338 94.890 96.651 32.871 87.615 99.011 76.431 91.436 95.856 60.650 [2020-12-31 17:36:05,468][root][INFO] - Finished test. Elapsed time: 39.7963 [2020-12-31 17:36:08,512][root][INFO] - [2020-12-31 17:36:08,512][root][INFO] - ################################################################ [2020-12-31 17:36:08,512][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 17:36:08,512][root][INFO] - ################################################################ [2020-12-31 17:36:08,512][root][INFO] - clutter : nan nan nan [2020-12-31 17:36:08,512][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 17:36:08,512][root][INFO] - board : 0.787 0.872 0.901 [2020-12-31 17:36:08,512][root][INFO] - bookcase : 0.206 0.360 0.492 [2020-12-31 17:36:08,512][root][INFO] - ceiling : 0.670 0.843 0.856 [2020-12-31 17:36:08,512][root][INFO] - chair : 0.793 0.889 0.903 [2020-12-31 17:36:08,512][root][INFO] - column : 0.180 0.302 0.401 [2020-12-31 17:36:08,512][root][INFO] - door : 0.575 0.759 0.862 [2020-12-31 17:36:08,512][root][INFO] - floor : 0.937 0.999 1.000 [2020-12-31 17:36:08,512][root][INFO] - sofa : 0.432 0.636 0.727 [2020-12-31 17:36:08,512][root][INFO] - table : 0.212 0.414 0.644 [2020-12-31 17:36:08,512][root][INFO] - wall : 0.480 0.708 0.790 [2020-12-31 17:36:08,512][root][INFO] - window : 0.475 0.693 0.775 [2020-12-31 17:36:08,512][root][INFO] - ---------------------------------------------------------------- [2020-12-31 17:36:08,512][root][INFO] - average : 0.479 0.623 0.696 [2020-12-31 17:36:08,513][root][INFO] - [2020-12-31 17:36:08,664][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 17:36:09,217][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter14600.pth [2020-12-31 17:36:20,008][root][INFO] - ===> Epoch[2921](14605/5): Loss -0.8162, Sem -0.8162, dir -0.9819, norm 0.1410 LR: 3.797e-03 Score 99.167 Data time: 0.1025, Forward time: 0.9465, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.0508 [2020-12-31 17:36:32,070][root][INFO] - ===> Epoch[2922](14610/5): Loss -0.8113, Sem -0.8113, dir -0.9807, norm 0.1436 LR: 3.754e-03 Score 99.141 Data time: 0.0951, Forward time: 0.9465, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3250 [2020-12-31 17:36:44,016][root][INFO] - ===> Epoch[2923](14615/5): Loss -0.8138, Sem -0.8138, dir -0.9813, norm 0.1423 LR: 3.710e-03 Score 99.150 Data time: 0.1190, Forward time: 0.9465, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3006 [2020-12-31 17:36:56,454][root][INFO] - ===> Epoch[2924](14620/5): Loss -0.8140, Sem -0.8140, dir -0.9811, norm 0.1421 LR: 3.667e-03 Score 99.158 Data time: 0.1638, Forward time: 0.9466, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.4016 [2020-12-31 17:37:08,195][root][INFO] - ===> Epoch[2925](14625/5): Loss -0.8115, Sem -0.8115, dir -0.9811, norm 0.1442 LR: 3.624e-03 Score 99.148 Data time: 0.0990, Forward time: 0.9466, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.2760 [2020-12-31 17:37:19,593][root][INFO] - ===> Epoch[2926](14630/5): Loss -0.8119, Sem -0.8119, dir -0.9812, norm 0.1433 LR: 3.580e-03 Score 99.136 Data time: 0.0857, Forward time: 0.9466, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.1831 [2020-12-31 17:37:31,733][root][INFO] - ===> Epoch[2927](14635/5): Loss -0.8168, Sem -0.8168, dir -0.9816, norm 0.1407 LR: 3.537e-03 Score 99.191 Data time: 0.1118, Forward time: 0.9467, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3405 [2020-12-31 17:37:43,589][root][INFO] - ===> Epoch[2928](14640/5): Loss -0.8100, Sem -0.8100, dir -0.9811, norm 0.1440 LR: 3.493e-03 Score 99.098 Data time: 0.1098, Forward time: 0.9467, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.2888 [2020-12-31 17:37:55,274][root][INFO] - ===> Epoch[2929](14645/5): Loss -0.8161, Sem -0.8161, dir -0.9812, norm 0.1399 LR: 3.450e-03 Score 99.147 Data time: 0.1162, Forward time: 0.9467, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.2373 [2020-12-31 17:38:07,135][root][INFO] - ===> Epoch[2930](14650/5): Loss -0.8086, Sem -0.8086, dir -0.9811, norm 0.1465 LR: 3.406e-03 Score 99.129 Data time: 0.1165, Forward time: 0.9467, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.2668 [2020-12-31 17:38:18,441][root][INFO] - ===> Epoch[2931](14655/5): Loss -0.8118, Sem -0.8118, dir -0.9811, norm 0.1436 LR: 3.362e-03 Score 99.127 Data time: 0.1394, Forward time: 0.9467, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.1616 [2020-12-31 17:38:30,794][root][INFO] - ===> Epoch[2932](14660/5): Loss -0.8138, Sem -0.8138, dir -0.9816, norm 0.1428 LR: 3.319e-03 Score 99.162 Data time: 0.1071, Forward time: 0.9468, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.3865 [2020-12-31 17:38:42,667][root][INFO] - ===> Epoch[2933](14665/5): Loss -0.8150, Sem -0.8150, dir -0.9813, norm 0.1412 LR: 3.275e-03 Score 99.144 Data time: 0.0671, Forward time: 0.9467, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.3039 [2020-12-31 17:38:54,798][root][INFO] - ===> Epoch[2934](14670/5): Loss -0.8152, Sem -0.8152, dir -0.9815, norm 0.1411 LR: 3.231e-03 Score 99.152 Data time: 0.1524, Forward time: 0.9467, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.3341 [2020-12-31 17:39:06,070][root][INFO] - ===> Epoch[2935](14675/5): Loss -0.8139, Sem -0.8139, dir -0.9810, norm 0.1416 LR: 3.187e-03 Score 99.146 Data time: 0.1474, Forward time: 0.9467, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.1633 [2020-12-31 17:39:19,406][root][INFO] - ===> Epoch[2936](14680/5): Loss -0.8160, Sem -0.8160, dir -0.9818, norm 0.1417 LR: 3.143e-03 Score 99.192 Data time: 0.1274, Forward time: 0.9467, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.5675 [2020-12-31 17:39:31,065][root][INFO] - ===> Epoch[2937](14685/5): Loss -0.8165, Sem -0.8165, dir -0.9814, norm 0.1401 LR: 3.099e-03 Score 99.160 Data time: 0.1080, Forward time: 0.9468, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.2324 [2020-12-31 17:39:43,226][root][INFO] - ===> Epoch[2938](14690/5): Loss -0.8155, Sem -0.8155, dir -0.9817, norm 0.1417 LR: 3.055e-03 Score 99.172 Data time: 0.0910, Forward time: 0.9468, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.3378 [2020-12-31 17:39:54,454][root][INFO] - ===> Epoch[2939](14695/5): Loss -0.8166, Sem -0.8166, dir -0.9814, norm 0.1395 LR: 3.011e-03 Score 99.147 Data time: 0.1152, Forward time: 0.9468, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.1623 [2020-12-31 17:40:06,569][root][INFO] - ===> Epoch[2940](14700/5): Loss -0.8089, Sem -0.8089, dir -0.9810, norm 0.1463 LR: 2.966e-03 Score 99.139 Data time: 0.0812, Forward time: 0.9467, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.3462 [2020-12-31 17:40:18,404][root][INFO] - ===> Epoch[2941](14705/5): Loss -0.8182, Sem -0.8182, dir -0.9820, norm 0.1389 LR: 2.922e-03 Score 99.155 Data time: 0.1145, Forward time: 0.9468, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.2755 [2020-12-31 17:40:30,249][root][INFO] - ===> Epoch[2942](14710/5): Loss -0.8101, Sem -0.8101, dir -0.9812, norm 0.1460 LR: 2.877e-03 Score 99.163 Data time: 0.1376, Forward time: 0.9468, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.2880 [2020-12-31 17:40:42,012][root][INFO] - ===> Epoch[2943](14715/5): Loss -0.8153, Sem -0.8153, dir -0.9817, norm 0.1414 LR: 2.833e-03 Score 99.163 Data time: 0.1450, Forward time: 0.9468, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.2682 [2020-12-31 17:40:53,382][root][INFO] - ===> Epoch[2944](14720/5): Loss -0.8165, Sem -0.8165, dir -0.9823, norm 0.1413 LR: 2.788e-03 Score 99.188 Data time: 0.0956, Forward time: 0.9468, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.1949 [2020-12-31 17:41:04,927][root][INFO] - ===> Epoch[2945](14725/5): Loss -0.8156, Sem -0.8156, dir -0.9816, norm 0.1413 LR: 2.744e-03 Score 99.173 Data time: 0.1151, Forward time: 0.9469, Backward time: 1.1864, DDP time: 0.0050, Total iter time: 2.2099 [2020-12-31 17:41:17,165][root][INFO] - ===> Epoch[2946](14730/5): Loss -0.8110, Sem -0.8110, dir -0.9812, norm 0.1449 LR: 2.699e-03 Score 99.149 Data time: 0.0921, Forward time: 0.9469, Backward time: 1.1864, DDP time: 0.0050, Total iter time: 2.3644 [2020-12-31 17:41:29,685][root][INFO] - ===> Epoch[2947](14735/5): Loss -0.8160, Sem -0.8160, dir -0.9816, norm 0.1420 LR: 2.654e-03 Score 99.205 Data time: 0.0731, Forward time: 0.9468, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.4327 [2020-12-31 17:41:41,944][root][INFO] - ===> Epoch[2948](14740/5): Loss -0.8118, Sem -0.8118, dir -0.9813, norm 0.1443 LR: 2.609e-03 Score 99.155 Data time: 0.1357, Forward time: 0.9468, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.3596 [2020-12-31 17:41:54,071][root][INFO] - ===> Epoch[2949](14745/5): Loss -0.8173, Sem -0.8173, dir -0.9820, norm 0.1401 LR: 2.564e-03 Score 99.173 Data time: 0.1602, Forward time: 0.9469, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.3318 [2020-12-31 17:42:05,798][root][INFO] - ===> Epoch[2950](14750/5): Loss -0.8099, Sem -0.8099, dir -0.9810, norm 0.1448 LR: 2.519e-03 Score 99.128 Data time: 0.1120, Forward time: 0.9469, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.2541 [2020-12-31 17:42:18,615][root][INFO] - ===> Epoch[2951](14755/5): Loss -0.8137, Sem -0.8137, dir -0.9812, norm 0.1416 LR: 2.474e-03 Score 99.122 Data time: 0.1195, Forward time: 0.9469, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.4671 [2020-12-31 17:42:30,605][root][INFO] - ===> Epoch[2952](14760/5): Loss -0.8186, Sem -0.8186, dir -0.9820, norm 0.1390 LR: 2.428e-03 Score 99.174 Data time: 0.0697, Forward time: 0.9469, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.3232 [2020-12-31 17:42:42,810][root][INFO] - ===> Epoch[2953](14765/5): Loss -0.8148, Sem -0.8148, dir -0.9816, norm 0.1422 LR: 2.383e-03 Score 99.169 Data time: 0.0798, Forward time: 0.9469, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.3528 [2020-12-31 17:42:55,087][root][INFO] - ===> Epoch[2954](14770/5): Loss -0.8069, Sem -0.8069, dir -0.9813, norm 0.1491 LR: 2.337e-03 Score 99.153 Data time: 0.0942, Forward time: 0.9468, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.3609 [2020-12-31 17:43:07,058][root][INFO] - ===> Epoch[2955](14775/5): Loss -0.8191, Sem -0.8191, dir -0.9820, norm 0.1378 LR: 2.292e-03 Score 99.156 Data time: 0.1587, Forward time: 0.9469, Backward time: 1.1865, DDP time: 0.0050, Total iter time: 2.2958 [2020-12-31 17:43:20,230][root][INFO] - ===> Epoch[2956](14780/5): Loss -0.8135, Sem -0.8135, dir -0.9811, norm 0.1423 LR: 2.246e-03 Score 99.149 Data time: 0.1041, Forward time: 0.9469, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.5438 [2020-12-31 17:43:32,370][root][INFO] - ===> Epoch[2957](14785/5): Loss -0.8179, Sem -0.8179, dir -0.9818, norm 0.1394 LR: 2.200e-03 Score 99.175 Data time: 0.1251, Forward time: 0.9469, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.3349 [2020-12-31 17:43:44,217][root][INFO] - ===> Epoch[2958](14790/5): Loss -0.8137, Sem -0.8137, dir -0.9815, norm 0.1434 LR: 2.154e-03 Score 99.176 Data time: 0.0781, Forward time: 0.9469, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.2872 [2020-12-31 17:43:57,186][root][INFO] - ===> Epoch[2959](14795/5): Loss -0.8160, Sem -0.8160, dir -0.9815, norm 0.1406 LR: 2.108e-03 Score 99.161 Data time: 0.0918, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.4981 [2020-12-31 17:44:09,507][root][INFO] - ===> Epoch[2960](14800/5): Loss -0.8105, Sem -0.8105, dir -0.9813, norm 0.1458 LR: 2.062e-03 Score 99.170 Data time: 0.2050, Forward time: 0.9470, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.3661 [2020-12-31 17:44:09,509][root][INFO] - ===> Start testing [2020-12-31 17:44:48,359][root][INFO] - 68/68: Data time: 0.0024, Iter time: 0.1228 Loss 0.346 (AVG: 0.545) Score 92.056 (AVG: 89.925) mIOU 68.400 mAP 83.041 mAcc 75.814 IOU: 60.325 0.000 81.605 72.927 93.629 90.978 31.204 75.618 98.344 61.956 80.909 85.386 56.313 mAP: 80.814 12.686 92.784 85.666 98.934 98.421 57.258 94.879 99.819 80.261 92.989 94.088 90.938 mAcc: 81.311 0.000 89.694 80.113 94.778 96.593 33.304 87.325 99.052 76.993 90.730 95.683 60.009 [2020-12-31 17:44:48,360][root][INFO] - Finished test. Elapsed time: 38.8489 [2020-12-31 17:44:51,787][root][INFO] - [2020-12-31 17:44:51,787][root][INFO] - ################################################################ [2020-12-31 17:44:51,787][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 17:44:51,787][root][INFO] - ################################################################ [2020-12-31 17:44:51,788][root][INFO] - clutter : nan nan nan [2020-12-31 17:44:51,788][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 17:44:51,788][root][INFO] - board : 0.774 0.868 0.897 [2020-12-31 17:44:51,788][root][INFO] - bookcase : 0.202 0.338 0.488 [2020-12-31 17:44:51,788][root][INFO] - ceiling : 0.675 0.832 0.855 [2020-12-31 17:44:51,788][root][INFO] - chair : 0.794 0.894 0.903 [2020-12-31 17:44:51,788][root][INFO] - column : 0.173 0.266 0.393 [2020-12-31 17:44:51,788][root][INFO] - door : 0.597 0.811 0.888 [2020-12-31 17:44:51,788][root][INFO] - floor : 0.933 1.000 1.000 [2020-12-31 17:44:51,788][root][INFO] - sofa : 0.450 0.636 0.809 [2020-12-31 17:44:51,788][root][INFO] - table : 0.202 0.391 0.646 [2020-12-31 17:44:51,788][root][INFO] - wall : 0.477 0.695 0.794 [2020-12-31 17:44:51,788][root][INFO] - window : 0.500 0.729 0.779 [2020-12-31 17:44:51,788][root][INFO] - ---------------------------------------------------------------- [2020-12-31 17:44:51,788][root][INFO] - average : 0.481 0.622 0.704 [2020-12-31 17:44:51,788][root][INFO] - [2020-12-31 17:44:51,951][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 17:44:52,511][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter14800.pth [2020-12-31 17:45:03,803][root][INFO] - ===> Epoch[2961](14805/5): Loss -0.8140, Sem -0.8140, dir -0.9818, norm 0.1420 LR: 2.016e-03 Score 99.139 Data time: 0.1126, Forward time: 0.9470, Backward time: 1.1866, DDP time: 0.0050, Total iter time: 2.1558 [2020-12-31 17:45:18,258][root][INFO] - ===> Epoch[2962](14810/5): Loss -0.8178, Sem -0.8178, dir -0.9822, norm 0.1395 LR: 1.970e-03 Score 99.160 Data time: 0.0898, Forward time: 0.9470, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.8176 [2020-12-31 17:45:29,802][root][INFO] - ===> Epoch[2963](14815/5): Loss -0.8104, Sem -0.8104, dir -0.9809, norm 0.1445 LR: 1.923e-03 Score 99.136 Data time: 0.1428, Forward time: 0.9470, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.2181 [2020-12-31 17:45:41,890][root][INFO] - ===> Epoch[2964](14820/5): Loss -0.8148, Sem -0.8148, dir -0.9815, norm 0.1411 LR: 1.877e-03 Score 99.138 Data time: 0.1259, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.3213 [2020-12-31 17:45:54,308][root][INFO] - ===> Epoch[2965](14825/5): Loss -0.8164, Sem -0.8164, dir -0.9815, norm 0.1399 LR: 1.830e-03 Score 99.152 Data time: 0.1085, Forward time: 0.9470, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.3913 [2020-12-31 17:46:06,459][root][INFO] - ===> Epoch[2966](14830/5): Loss -0.8134, Sem -0.8134, dir -0.9815, norm 0.1419 LR: 1.783e-03 Score 99.130 Data time: 0.0978, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.3373 [2020-12-31 17:46:19,062][root][INFO] - ===> Epoch[2967](14835/5): Loss -0.8107, Sem -0.8107, dir -0.9812, norm 0.1453 LR: 1.736e-03 Score 99.153 Data time: 0.1612, Forward time: 0.9470, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.4178 [2020-12-31 17:46:30,312][root][INFO] - ===> Epoch[2968](14840/5): Loss -0.8116, Sem -0.8116, dir -0.9807, norm 0.1430 LR: 1.689e-03 Score 99.117 Data time: 0.1323, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.1564 [2020-12-31 17:46:42,378][root][INFO] - ===> Epoch[2969](14845/5): Loss -0.8142, Sem -0.8142, dir -0.9813, norm 0.1420 LR: 1.642e-03 Score 99.157 Data time: 0.0953, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.3201 [2020-12-31 17:46:54,433][root][INFO] - ===> Epoch[2970](14850/5): Loss -0.8141, Sem -0.8141, dir -0.9813, norm 0.1417 LR: 1.594e-03 Score 99.145 Data time: 0.0839, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.3085 [2020-12-31 17:47:06,434][root][INFO] - ===> Epoch[2971](14855/5): Loss -0.8211, Sem -0.8211, dir -0.9822, norm 0.1377 LR: 1.547e-03 Score 99.210 Data time: 0.1095, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3096 [2020-12-31 17:47:19,089][root][INFO] - ===> Epoch[2972](14860/5): Loss -0.8131, Sem -0.8131, dir -0.9816, norm 0.1433 LR: 1.499e-03 Score 99.158 Data time: 0.1478, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.4298 [2020-12-31 17:47:30,679][root][INFO] - ===> Epoch[2973](14865/5): Loss -0.8132, Sem -0.8132, dir -0.9815, norm 0.1426 LR: 1.451e-03 Score 99.139 Data time: 0.1091, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.2380 [2020-12-31 17:47:42,772][root][INFO] - ===> Epoch[2974](14870/5): Loss -0.8187, Sem -0.8187, dir -0.9820, norm 0.1394 LR: 1.403e-03 Score 99.198 Data time: 0.1271, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.3323 [2020-12-31 17:47:54,006][root][INFO] - ===> Epoch[2975](14875/5): Loss -0.8184, Sem -0.8184, dir -0.9815, norm 0.1384 LR: 1.355e-03 Score 99.173 Data time: 0.1141, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.1620 [2020-12-31 17:48:05,784][root][INFO] - ===> Epoch[2976](14880/5): Loss -0.8137, Sem -0.8137, dir -0.9815, norm 0.1431 LR: 1.306e-03 Score 99.166 Data time: 0.0798, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.2758 [2020-12-31 17:48:17,701][root][INFO] - ===> Epoch[2977](14885/5): Loss -0.8151, Sem -0.8151, dir -0.9819, norm 0.1421 LR: 1.257e-03 Score 99.169 Data time: 0.0796, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.2985 [2020-12-31 17:48:29,597][root][INFO] - ===> Epoch[2978](14890/5): Loss -0.8147, Sem -0.8147, dir -0.9815, norm 0.1411 LR: 1.209e-03 Score 99.134 Data time: 0.1555, Forward time: 0.9469, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.2837 [2020-12-31 17:48:42,387][root][INFO] - ===> Epoch[2979](14895/5): Loss -0.8196, Sem -0.8196, dir -0.9817, norm 0.1387 LR: 1.159e-03 Score 99.215 Data time: 0.1133, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.4569 [2020-12-31 17:48:54,810][root][INFO] - ===> Epoch[2980](14900/5): Loss -0.8146, Sem -0.8146, dir -0.9813, norm 0.1422 LR: 1.110e-03 Score 99.181 Data time: 0.1388, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3843 [2020-12-31 17:49:07,211][root][INFO] - ===> Epoch[2981](14905/5): Loss -0.8184, Sem -0.8184, dir -0.9824, norm 0.1400 LR: 1.061e-03 Score 99.187 Data time: 0.0875, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3895 [2020-12-31 17:49:19,136][root][INFO] - ===> Epoch[2982](14910/5): Loss -0.8125, Sem -0.8125, dir -0.9816, norm 0.1435 LR: 1.011e-03 Score 99.142 Data time: 0.1136, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.2985 [2020-12-31 17:49:31,209][root][INFO] - ===> Epoch[2983](14915/5): Loss -0.8145, Sem -0.8145, dir -0.9818, norm 0.1422 LR: 9.606e-04 Score 99.161 Data time: 0.1183, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3158 [2020-12-31 17:49:43,052][root][INFO] - ===> Epoch[2984](14920/5): Loss -0.8166, Sem -0.8166, dir -0.9817, norm 0.1408 LR: 9.102e-04 Score 99.177 Data time: 0.1231, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.2771 [2020-12-31 17:49:54,704][root][INFO] - ===> Epoch[2985](14925/5): Loss -0.8140, Sem -0.8140, dir -0.9817, norm 0.1421 LR: 8.595e-04 Score 99.135 Data time: 0.0858, Forward time: 0.9469, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.2350 [2020-12-31 17:50:06,746][root][INFO] - ===> Epoch[2986](14930/5): Loss -0.8170, Sem -0.8170, dir -0.9815, norm 0.1393 LR: 8.084e-04 Score 99.156 Data time: 0.0965, Forward time: 0.9468, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3186 [2020-12-31 17:50:18,185][root][INFO] - ===> Epoch[2987](14935/5): Loss -0.8138, Sem -0.8138, dir -0.9817, norm 0.1427 LR: 7.570e-04 Score 99.154 Data time: 0.0930, Forward time: 0.9468, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.1991 [2020-12-31 17:50:30,559][root][INFO] - ===> Epoch[2988](14940/5): Loss -0.8174, Sem -0.8174, dir -0.9815, norm 0.1395 LR: 7.052e-04 Score 99.173 Data time: 0.0869, Forward time: 0.9468, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3908 [2020-12-31 17:50:41,769][root][INFO] - ===> Epoch[2989](14945/5): Loss -0.8148, Sem -0.8148, dir -0.9817, norm 0.1421 LR: 6.529e-04 Score 99.172 Data time: 0.1219, Forward time: 0.9468, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.1563 [2020-12-31 17:50:53,931][root][INFO] - ===> Epoch[2990](14950/5): Loss -0.8149, Sem -0.8149, dir -0.9815, norm 0.1412 LR: 6.002e-04 Score 99.141 Data time: 0.0835, Forward time: 0.9468, Backward time: 1.1869, DDP time: 0.0050, Total iter time: 2.3430 [2020-12-31 17:51:05,290][root][INFO] - ===> Epoch[2991](14955/5): Loss -0.8130, Sem -0.8130, dir -0.9815, norm 0.1433 LR: 5.470e-04 Score 99.157 Data time: 0.1079, Forward time: 0.9468, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.1782 [2020-12-31 17:51:17,240][root][INFO] - ===> Epoch[2992](14960/5): Loss -0.8195, Sem -0.8195, dir -0.9818, norm 0.1371 LR: 4.932e-04 Score 99.155 Data time: 0.0801, Forward time: 0.9468, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3086 [2020-12-31 17:51:28,500][root][INFO] - ===> Epoch[2993](14965/5): Loss -0.8131, Sem -0.8131, dir -0.9809, norm 0.1421 LR: 4.387e-04 Score 99.138 Data time: 0.1596, Forward time: 0.9468, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.1688 [2020-12-31 17:51:42,192][root][INFO] - ===> Epoch[2994](14970/5): Loss -0.8119, Sem -0.8119, dir -0.9811, norm 0.1441 LR: 3.835e-04 Score 99.157 Data time: 0.1820, Forward time: 0.9468, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.6313 [2020-12-31 17:51:53,586][root][INFO] - ===> Epoch[2995](14975/5): Loss -0.8180, Sem -0.8180, dir -0.9817, norm 0.1387 LR: 3.273e-04 Score 99.157 Data time: 0.1100, Forward time: 0.9468, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.1849 [2020-12-31 17:52:05,555][root][INFO] - ===> Epoch[2996](14980/5): Loss -0.8105, Sem -0.8105, dir -0.9810, norm 0.1443 LR: 2.701e-04 Score 99.125 Data time: 0.1076, Forward time: 0.9468, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3006 [2020-12-31 17:52:17,530][root][INFO] - ===> Epoch[2997](14985/5): Loss -0.8186, Sem -0.8186, dir -0.9816, norm 0.1391 LR: 2.114e-04 Score 99.193 Data time: 0.1662, Forward time: 0.9468, Backward time: 1.1868, DDP time: 0.0050, Total iter time: 2.3011 [2020-12-31 17:52:28,908][root][INFO] - ===> Epoch[2998](14990/5): Loss -0.8154, Sem -0.8154, dir -0.9813, norm 0.1407 LR: 1.509e-04 Score 99.159 Data time: 0.1738, Forward time: 0.9468, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.1840 [2020-12-31 17:52:41,005][root][INFO] - ===> Epoch[2999](14995/5): Loss -0.8140, Sem -0.8140, dir -0.9815, norm 0.1422 LR: 8.746e-05 Score 99.158 Data time: 0.1164, Forward time: 0.9468, Backward time: 1.1867, DDP time: 0.0050, Total iter time: 2.3399 [2020-12-31 17:52:52,717][root][INFO] - ===> Start testing [2020-12-31 17:53:27,909][root][INFO] - 68/68: Data time: 0.6363, Iter time: 0.3221 Loss 1.639 (AVG: 0.532) Score 71.602 (AVG: 90.073) mIOU 68.871 mAP 83.396 mAcc 76.146 IOU: 60.675 0.000 82.174 73.603 93.791 90.772 31.142 76.685 98.253 64.090 81.331 85.446 57.367 mAP: 81.304 13.386 92.984 86.159 98.918 98.313 58.094 95.027 99.817 81.655 92.962 94.304 91.231 mAcc: 81.008 0.000 89.767 81.132 94.938 96.672 33.875 87.823 98.941 77.792 90.855 95.533 61.563 [2020-12-31 17:53:27,910][root][INFO] - Finished test. Elapsed time: 35.1910 [2020-12-31 17:53:30,626][root][INFO] - [2020-12-31 17:53:30,627][root][INFO] - ################################################################ [2020-12-31 17:53:30,627][root][INFO] - what : AP AP_50% AP_25% [2020-12-31 17:53:30,627][root][INFO] - ################################################################ [2020-12-31 17:53:30,627][root][INFO] - clutter : nan nan nan [2020-12-31 17:53:30,627][root][INFO] - beam : 0.000 0.000 0.000 [2020-12-31 17:53:30,627][root][INFO] - board : 0.789 0.870 0.899 [2020-12-31 17:53:30,627][root][INFO] - bookcase : 0.201 0.329 0.486 [2020-12-31 17:53:30,627][root][INFO] - ceiling : 0.629 0.805 0.852 [2020-12-31 17:53:30,627][root][INFO] - chair : 0.800 0.898 0.907 [2020-12-31 17:53:30,627][root][INFO] - column : 0.173 0.277 0.396 [2020-12-31 17:53:30,627][root][INFO] - door : 0.659 0.850 0.935 [2020-12-31 17:53:30,627][root][INFO] - floor : 0.883 0.957 0.976 [2020-12-31 17:53:30,627][root][INFO] - sofa : 0.444 0.636 0.801 [2020-12-31 17:53:30,627][root][INFO] - table : 0.201 0.403 0.644 [2020-12-31 17:53:30,627][root][INFO] - wall : 0.485 0.712 0.811 [2020-12-31 17:53:30,627][root][INFO] - window : 0.480 0.675 0.773 [2020-12-31 17:53:30,627][root][INFO] - ---------------------------------------------------------------- [2020-12-31 17:53:30,627][root][INFO] - average : 0.479 0.618 0.707 [2020-12-31 17:53:30,627][root][INFO] - [2020-12-31 17:53:30,811][root][INFO] - Summary name val/mAP@0.5 is illegal; using val/mAP_0.5 instead. [2020-12-31 17:53:31,390][root][INFO] - Checkpoint saved to weights/checkpoint_Res16UNet34C_iter15000.pth [2020-12-31 17:53:32,436][submitit][INFO] - Job completed successfully