The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.



This table lists the benchmark results for the 3D semantic instance scenario.




Method Infoavg ap 50%bathtubbedbookshelfcabinetchaircountercurtaindeskdoorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
HAISpermissive0.699 11.000 10.849 20.820 30.675 10.808 70.279 30.757 60.465 60.517 30.596 10.559 30.600 81.000 10.654 50.767 40.676 20.994 140.560 3
Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, Xinggang Wang: Hierarchical Aggregation for 3D Instance Segmentation. ICCV 2021
SSTNet0.698 21.000 10.697 200.888 20.556 50.803 80.387 20.626 100.417 100.556 20.585 30.702 10.600 81.000 10.824 10.720 120.692 11.000 10.509 6
OccuSeg+instance0.672 31.000 10.758 150.682 110.576 30.842 10.477 10.504 200.524 30.567 10.585 40.451 70.557 141.000 10.751 20.797 30.563 131.000 10.467 11
Mask-Group0.664 41.000 10.822 50.764 80.616 20.815 40.139 80.694 90.597 20.459 70.566 50.599 20.600 80.516 240.715 30.819 20.635 61.000 10.603 1
CSC-Pretrained0.648 51.000 10.810 60.768 60.523 100.813 50.143 70.819 10.389 110.422 140.511 110.443 80.650 21.000 10.624 70.732 100.634 71.000 10.375 17
PE0.645 61.000 10.773 110.798 50.538 70.786 130.088 150.799 40.350 150.435 130.547 70.545 40.646 60.933 110.562 100.761 70.556 170.997 110.501 8
Biao Zhang, Peter Wonka: Point Cloud Instance Segmentation using Probabilistic Embeddings. CVPR 2021
RPGN0.643 71.000 10.758 140.582 170.539 60.826 30.046 180.765 50.372 130.436 120.588 20.539 50.650 21.000 10.577 80.750 90.653 50.997 110.495 9
Dyco3Dcopyleft0.641 81.000 10.841 30.893 10.531 80.802 90.115 120.588 150.448 70.438 100.537 90.430 100.550 150.857 130.534 110.764 60.657 30.987 150.568 2
GICN0.638 91.000 10.895 10.800 40.480 130.676 160.144 60.737 70.354 140.447 80.400 190.365 140.700 11.000 10.569 90.836 10.599 101.000 10.473 10
PointGroup0.636 101.000 10.765 120.624 120.505 120.797 100.116 110.696 80.384 120.441 90.559 60.476 60.596 111.000 10.666 40.756 80.556 160.997 110.513 5
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, Jiaya Jia: PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. CVPR 2020 [oral]
DD-UNet+Group0.635 110.667 210.797 90.714 100.562 40.774 140.146 50.810 30.429 90.476 50.546 80.399 120.633 71.000 10.632 60.722 110.609 91.000 10.514 4
H. Liu, R. Liu, K. Yang, J. Zhang, K. Peng, R. Stiefelhagen: HIDA: Towards Holistic Indoor Understanding for the Visually Impaired via Semantic Instance Segmentation with a Wearable Solid-State LiDAR Sensor.
3D-MPA0.611 121.000 10.833 40.765 70.526 90.756 150.136 100.588 150.470 50.438 110.432 170.358 150.650 20.857 130.429 150.765 50.557 151.000 10.430 14
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nie├čner: 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation. CVPR 2020
SphereNet0.606 131.000 10.776 100.745 90.436 170.834 20.035 200.587 170.518 40.338 210.534 100.352 160.594 131.000 10.391 190.696 150.624 81.000 10.451 12
PCJC0.578 141.000 10.810 70.583 160.449 160.813 60.042 190.603 130.341 160.490 40.465 130.410 110.650 20.835 180.264 230.694 160.561 140.889 210.504 7
SSEN0.575 151.000 10.761 130.473 190.477 140.795 110.066 160.529 180.658 10.460 60.461 140.380 130.331 230.859 120.401 180.692 170.653 41.000 10.348 19
Dongsu Zhang, Junha Chun, Sang Kyun Cha, Young Min Kim: Spatial Semantic Embedding Network: Fast 3D Instance Segmentation with Deep Metric Learning. Arxiv
MTML0.549 161.000 10.807 80.588 150.327 200.647 170.004 250.815 20.180 200.418 150.364 210.182 190.445 181.000 10.442 140.688 180.571 121.000 10.396 15
Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald: 3D Instance Segmentation via Multi-task Metric Learning. ICCV 2019 [oral]
Sparse R-CNN0.515 171.000 10.538 260.282 220.468 150.790 120.173 40.345 220.429 80.413 170.484 120.176 200.595 120.591 220.522 120.668 190.476 200.986 160.327 20
Occipital-SCS0.512 181.000 10.716 170.509 180.506 110.611 190.092 140.602 140.177 210.346 190.383 200.165 210.442 190.850 170.386 200.618 210.543 180.889 210.389 16
3D-BoNet0.488 191.000 10.672 210.590 140.301 210.484 270.098 130.620 110.306 170.341 200.259 230.125 230.434 200.796 190.402 170.499 260.513 190.909 200.439 13
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight
PanopticFusion-inst0.478 200.667 210.712 190.595 130.259 230.550 240.000 280.613 120.175 220.250 250.434 150.437 90.411 220.857 130.485 130.591 240.267 280.944 180.359 18
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
SALoss-ResNet0.459 211.000 10.737 160.159 290.259 220.587 210.138 90.475 210.217 190.416 160.408 180.128 220.315 240.714 200.411 160.536 250.590 110.873 240.304 21
Zhidong Liang, Ming Yang, Hao Li, Chunxiang Wang: 3D Instance Embedding Learning With a Structure-Aware Loss Function for Point Cloud Segmentation. IEEE Robotics and Automation Letters (IROS2020)
MASCpermissive0.447 220.528 270.555 240.381 200.382 180.633 180.002 260.509 190.260 180.361 180.432 160.327 170.451 170.571 230.367 210.639 200.386 210.980 170.276 22
Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.
3D-SISpermissive0.382 231.000 10.432 280.245 240.190 240.577 220.013 230.263 240.033 280.320 220.240 240.075 250.422 210.857 130.117 260.699 130.271 270.883 230.235 24
Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019
Hier3Dcopyleft0.323 240.667 210.542 250.264 230.157 270.550 230.000 280.205 270.009 290.270 240.218 250.075 250.500 160.688 210.007 310.698 140.301 250.459 290.200 25
Tan: HCFS3D: Hierarchical Coupled Feature Selection Network for 3D Semantic and Instance Segmentation.
UNet-backbone0.319 250.667 210.715 180.233 250.189 250.479 280.008 240.218 250.067 270.201 260.173 260.107 240.123 280.438 250.150 240.615 220.355 220.916 190.093 30
R-PointNet0.306 260.500 280.405 290.311 210.348 190.589 200.054 170.068 290.126 230.283 230.290 220.028 290.219 260.214 270.331 220.396 290.275 260.821 260.245 23
RandSA0.250 270.333 290.613 220.229 260.163 260.493 250.000 280.304 230.107 240.147 280.100 270.052 280.231 250.119 280.039 280.445 280.325 230.654 270.141 27
Region0.248 280.667 210.437 270.188 270.153 280.491 260.000 280.208 260.094 260.153 270.099 280.057 270.217 270.119 280.039 280.466 270.302 240.640 280.140 28
3D-BEVIS0.248 280.667 210.566 230.076 300.035 310.394 290.027 220.035 300.098 250.099 290.030 300.025 300.098 290.375 260.126 250.604 230.181 290.854 250.171 26
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.
Sgpn_scannet0.143 300.208 310.390 300.169 280.065 290.275 300.029 210.069 280.000 300.087 300.043 290.014 310.027 310.000 300.112 270.351 300.168 300.438 300.138 29
MaskRCNN 2d->3d Proj0.058 310.333 290.002 310.000 310.053 300.002 310.002 270.021 310.000 300.045 310.024 310.238 180.065 300.000 300.014 300.107 310.020 310.110 310.006 31