The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.



This table lists the benchmark results for the 3D semantic instance scenario.




Method Infoavg ap 50%bathtubbedbookshelfcabinetchaircountercurtaindeskdoorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
CRNet0.688 11.000 10.802 60.783 30.644 10.804 40.182 20.874 10.511 40.501 20.592 10.536 30.587 81.000 10.637 40.730 70.683 10.994 90.523 2
OccuSeg+instance0.672 21.000 10.758 100.682 60.576 30.842 10.477 10.504 140.524 30.567 10.585 20.451 50.557 91.000 10.751 10.797 30.563 71.000 10.467 7
Mask-Group0.664 31.000 10.822 30.764 50.616 20.815 20.139 50.694 60.597 20.459 50.566 30.599 10.600 50.516 170.715 20.819 20.635 31.000 10.603 1
PE0.645 41.000 10.773 70.798 20.538 40.786 80.088 110.799 30.350 90.435 90.547 50.545 20.646 40.933 60.562 60.761 50.556 110.997 70.501 5
GICN0.638 51.000 10.895 10.800 10.480 80.676 100.144 40.737 40.354 80.447 60.400 130.365 90.700 11.000 10.569 50.836 10.599 41.000 10.473 6
PointGroup0.636 61.000 10.765 80.624 70.505 70.797 50.116 80.696 50.384 70.441 70.559 40.476 40.596 61.000 10.666 30.756 60.556 100.997 70.513 3
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, Jiaya Jia: PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. CVPR 2020 [oral]
3D-MPA0.611 71.000 10.833 20.765 40.526 50.756 90.136 70.588 110.470 50.438 80.432 110.358 100.650 20.857 80.429 100.765 40.557 91.000 10.430 9
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nie├čner: 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation. CVPR 2020
PCJC0.578 81.000 10.810 40.583 110.449 110.813 30.042 140.603 90.341 100.490 30.465 70.410 70.650 20.835 120.264 170.694 90.561 80.889 150.504 4
SSEN0.575 91.000 10.761 90.473 130.477 90.795 60.066 120.529 120.658 10.460 40.461 80.380 80.331 160.859 70.401 130.692 100.653 21.000 10.348 13
Dongsu Zhang, Junha Chun, Sang Kyun Cha, Young Min Kim: Spatial Semantic Embedding Network: Fast 3D Instance Segmentation with Deep Metric Learning. Arxiv
MTML0.549 101.000 10.807 50.588 100.327 140.647 110.004 200.815 20.180 140.418 100.364 150.182 130.445 111.000 10.442 90.688 110.571 61.000 10.396 10
Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald: 3D Instance Segmentation via Multi-task Metric Learning. ICCV 2019 [oral]
Sparse R-CNN0.515 111.000 10.538 180.282 170.468 100.790 70.173 30.345 160.429 60.413 120.484 60.176 140.595 70.591 150.522 70.668 120.476 140.986 100.327 14
Occipital-SCS0.512 121.000 10.716 120.509 120.506 60.611 130.092 100.602 100.177 150.346 140.383 140.165 150.442 120.850 110.386 140.618 140.543 120.889 150.389 11
3D-BoNet0.488 131.000 10.672 150.590 90.301 150.484 190.098 90.620 70.306 110.341 150.259 170.125 170.434 130.796 130.402 120.499 190.513 130.909 140.439 8
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight
PanopticFusion-inst0.478 140.667 160.712 140.595 80.259 170.550 170.000 230.613 80.175 160.250 180.434 90.437 60.411 150.857 80.485 80.591 170.267 210.944 120.359 12
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
SALoss-ResNet0.459 151.000 10.737 110.159 220.259 160.587 150.138 60.475 150.217 130.416 110.408 120.128 160.315 170.714 140.411 110.536 180.590 50.873 180.304 15
Zhidong Liang, Ming Yang, Hao Li, Chunxiang Wang: 3D Instance Embedding Learning With a Structure-Aware Loss Function for Point Cloud Segmentation. IEEE Robotics and Automation Letters (IROS2020)
MASCpermissive0.447 160.528 200.555 170.381 140.382 120.633 120.002 210.509 130.260 120.361 130.432 100.327 110.451 100.571 160.367 150.639 130.386 150.980 110.276 16
Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.
3D-SISpermissive0.382 171.000 10.432 200.245 180.190 180.577 160.013 180.263 180.033 220.320 160.240 180.075 190.422 140.857 80.117 200.699 80.271 200.883 170.235 18
Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019
UNet-backbone0.319 180.667 160.715 130.233 190.189 190.479 200.008 190.218 190.067 210.201 190.173 190.107 180.123 200.438 180.150 180.615 150.355 160.916 130.093 23
R-PointNet0.306 190.500 210.405 210.311 150.348 130.589 140.054 130.068 220.126 170.283 170.290 160.028 210.219 180.214 210.331 160.396 210.275 190.821 200.245 17
Region0.248 200.667 160.437 190.188 200.153 200.491 180.000 230.208 200.094 190.153 200.099 210.057 200.217 190.119 220.039 230.466 200.302 180.640 210.140 20
3D-BEVIS0.248 200.667 160.566 160.076 230.035 240.394 210.027 160.035 230.098 180.099 220.030 230.025 220.098 210.375 190.126 190.604 160.181 220.854 190.171 19
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.
Seg-Clusterpermissive0.215 220.370 220.337 230.285 160.105 210.325 220.025 170.282 170.085 200.105 210.107 200.007 240.079 220.317 200.114 210.309 230.304 170.587 220.123 22
Sgpn_scannet0.143 230.208 240.390 220.169 210.065 220.275 230.029 150.069 210.000 230.087 230.043 220.014 230.027 240.000 230.112 220.351 220.168 230.438 230.138 21
MaskRCNN 2d->3d Proj0.058 240.333 230.002 240.000 240.053 230.002 240.002 220.021 240.000 230.045 240.024 240.238 120.065 230.000 230.014 240.107 240.020 240.110 240.006 24