The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.



This table lists the benchmark results for the 3D semantic instance scenario.




Method Infoavg ap 50%bathtubbedbookshelfcabinetchaircountercurtaindeskdoorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
HAIS0.699 11.000 10.849 20.820 30.675 10.808 70.279 30.757 50.465 60.517 30.596 10.559 30.600 71.000 10.654 50.767 40.676 20.994 130.560 3
SSTNet0.698 21.000 10.697 190.888 20.556 40.803 80.387 20.626 90.417 90.556 20.585 30.702 10.600 71.000 10.824 10.720 110.692 11.000 10.509 5
OccuSeg+instance0.672 31.000 10.758 140.682 100.576 30.842 10.477 10.504 190.524 30.567 10.585 40.451 70.557 131.000 10.751 20.797 30.563 121.000 10.467 10
Mask-Group0.664 41.000 10.822 50.764 80.616 20.815 40.139 70.694 80.597 20.459 60.566 50.599 20.600 70.516 230.715 30.819 20.635 61.000 10.603 1
CSC-Pretrained0.648 51.000 10.810 60.768 60.523 90.813 50.143 60.819 10.389 100.422 130.511 100.443 80.650 21.000 10.624 60.732 100.634 71.000 10.375 16
PE0.645 61.000 10.773 100.798 50.538 60.786 130.088 140.799 30.350 140.435 120.547 70.545 40.646 60.933 100.562 90.761 70.556 160.997 100.501 7
Biao Zhang, Peter Wonka: Point Cloud Instance Segmentation using Probabilistic Embeddings. CVPR 2021
RPGN0.643 71.000 10.758 130.582 160.539 50.826 30.046 170.765 40.372 120.436 110.588 20.539 50.650 21.000 10.577 70.750 90.653 50.997 100.495 8
Dyco3Dcopyleft0.641 81.000 10.841 30.893 10.531 70.802 90.115 110.588 140.448 70.438 90.537 80.430 100.550 140.857 120.534 100.764 60.657 30.987 140.568 2
GICN0.638 91.000 10.895 10.800 40.480 120.676 150.144 50.737 60.354 130.447 70.400 180.365 130.700 11.000 10.569 80.836 10.599 91.000 10.473 9
PointGroup0.636 101.000 10.765 110.624 110.505 110.797 100.116 100.696 70.384 110.441 80.559 60.476 60.596 101.000 10.666 40.756 80.556 150.997 100.513 4
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, Jiaya Jia: PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. CVPR 2020 [oral]
3D-MPA0.611 111.000 10.833 40.765 70.526 80.756 140.136 90.588 140.470 50.438 100.432 160.358 140.650 20.857 120.429 140.765 50.557 141.000 10.430 13
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nie├čner: 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation. CVPR 2020
SphereNet0.606 121.000 10.776 90.745 90.436 160.834 20.035 190.587 160.518 40.338 200.534 90.352 150.594 121.000 10.391 180.696 140.624 81.000 10.451 11
PCJC0.578 131.000 10.810 70.583 150.449 150.813 60.042 180.603 120.341 150.490 40.465 120.410 110.650 20.835 170.264 220.694 150.561 130.889 200.504 6
SSEN0.575 141.000 10.761 120.473 180.477 130.795 110.066 150.529 170.658 10.460 50.461 130.380 120.331 220.859 110.401 170.692 160.653 41.000 10.348 18
Dongsu Zhang, Junha Chun, Sang Kyun Cha, Young Min Kim: Spatial Semantic Embedding Network: Fast 3D Instance Segmentation with Deep Metric Learning. Arxiv
MTML0.549 151.000 10.807 80.588 140.327 190.647 160.004 240.815 20.180 190.418 140.364 200.182 180.445 171.000 10.442 130.688 170.571 111.000 10.396 14
Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald: 3D Instance Segmentation via Multi-task Metric Learning. ICCV 2019 [oral]
Sparse R-CNN0.515 161.000 10.538 240.282 210.468 140.790 120.173 40.345 210.429 80.413 160.484 110.176 190.595 110.591 210.522 110.668 180.476 190.986 150.327 19
Occipital-SCS0.512 171.000 10.716 160.509 170.506 100.611 180.092 130.602 130.177 200.346 180.383 190.165 200.442 180.850 160.386 190.618 200.543 170.889 200.389 15
3D-BoNet0.488 181.000 10.672 200.590 130.301 200.484 250.098 120.620 100.306 160.341 190.259 220.125 220.434 190.796 180.402 160.499 250.513 180.909 190.439 12
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight
PanopticFusion-inst0.478 190.667 210.712 180.595 120.259 220.550 230.000 270.613 110.175 210.250 240.434 140.437 90.411 210.857 120.485 120.591 230.267 260.944 170.359 17
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
SALoss-ResNet0.459 201.000 10.737 150.159 270.259 210.587 200.138 80.475 200.217 180.416 150.408 170.128 210.315 230.714 190.411 150.536 240.590 100.873 230.304 20
Zhidong Liang, Ming Yang, Hao Li, Chunxiang Wang: 3D Instance Embedding Learning With a Structure-Aware Loss Function for Point Cloud Segmentation. IEEE Robotics and Automation Letters (IROS2020)
MASCpermissive0.447 210.528 260.555 220.381 190.382 170.633 170.002 250.509 180.260 170.361 170.432 150.327 160.451 160.571 220.367 200.639 190.386 200.980 160.276 21
Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.
3D-SISpermissive0.382 221.000 10.432 260.245 230.190 230.577 210.013 220.263 220.033 260.320 210.240 230.075 240.422 200.857 120.117 250.699 120.271 250.883 220.235 23
Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019
Hier3Dcopyleft0.323 230.667 210.542 230.264 220.157 250.550 220.000 270.205 250.009 270.270 230.218 240.075 240.500 150.688 200.007 290.698 130.301 230.459 270.200 24
Tan: HCFS3D: Hierarchical Coupled Feature Selection Network for 3D Semantic and Instance Segmentation.
UNet-backbone0.319 240.667 210.715 170.233 240.189 240.479 260.008 230.218 230.067 250.201 250.173 250.107 230.123 260.438 240.150 230.615 210.355 210.916 180.093 28
R-PointNet0.306 250.500 270.405 270.311 200.348 180.589 190.054 160.068 270.126 220.283 220.290 210.028 270.219 240.214 260.331 210.396 270.275 240.821 250.245 22
3D-BEVIS0.248 260.667 210.566 210.076 280.035 290.394 270.027 210.035 280.098 230.099 270.030 280.025 280.098 270.375 250.126 240.604 220.181 270.854 240.171 25
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.
Region0.248 260.667 210.437 250.188 250.153 260.491 240.000 270.208 240.094 240.153 260.099 260.057 260.217 250.119 270.039 270.466 260.302 220.640 260.140 26
Sgpn_scannet0.143 280.208 290.390 280.169 260.065 270.275 280.029 200.069 260.000 280.087 280.043 270.014 290.027 290.000 280.112 260.351 280.168 280.438 280.138 27
MaskRCNN 2d->3d Proj0.058 290.333 280.002 290.000 290.053 280.002 290.002 260.021 290.000 280.045 290.024 290.238 170.065 280.000 280.014 280.107 290.020 290.110 290.006 29