The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.



This table lists the benchmark results for the 3D semantic instance scenario.




Method Infoavg ap 50%bathtubbedbookshelfcabinetchaircountercurtaindeskdoorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
HAISpermissive0.699 11.000 10.849 20.820 30.675 10.808 70.279 30.757 70.465 60.517 30.596 10.559 30.600 91.000 10.654 60.767 40.676 30.994 150.560 4
Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, Xinggang Wang: Hierarchical Aggregation for 3D Instance Segmentation. ICCV 2021
SSTNetpermissive0.698 21.000 10.697 210.888 20.556 60.803 80.387 20.626 110.417 100.556 20.585 30.702 10.600 91.000 10.824 10.720 120.692 11.000 10.509 7
Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, Kui Jia: Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networks. ICCV2021
OccuSeg+instance0.672 31.000 10.758 160.682 110.576 40.842 10.477 10.504 210.524 30.567 10.585 40.451 70.557 151.000 10.751 20.797 30.563 141.000 10.467 12
Mask-Group0.664 41.000 10.822 50.764 80.616 20.815 40.139 90.694 100.597 20.459 70.566 50.599 20.600 90.516 250.715 30.819 20.635 71.000 10.603 1
CSC-Pretrained0.648 51.000 10.810 60.768 60.523 110.813 50.143 80.819 20.389 110.422 140.511 110.443 80.650 21.000 10.624 80.732 100.634 81.000 10.375 18
PE0.645 61.000 10.773 120.798 50.538 80.786 130.088 160.799 50.350 150.435 130.547 70.545 40.646 70.933 120.562 110.761 70.556 180.997 120.501 9
Biao Zhang, Peter Wonka: Point Cloud Instance Segmentation using Probabilistic Embeddings. CVPR 2021
RPGN0.643 71.000 10.758 150.582 180.539 70.826 30.046 190.765 60.372 130.436 120.588 20.539 50.650 21.000 10.577 90.750 90.653 60.997 120.495 10
Dyco3Dcopyleft0.641 81.000 10.841 30.893 10.531 90.802 90.115 130.588 160.448 70.438 100.537 90.430 100.550 160.857 140.534 120.764 60.657 40.987 160.568 2
Tong He; Chunhua Shen; Anton van den Hengel: DyCo3D: Robust Instance Segmentation of 3D Point Clouds through Dynamic Convolution. CVPR2021
GICN0.638 91.000 10.895 10.800 40.480 140.676 160.144 70.737 80.354 140.447 80.400 190.365 150.700 11.000 10.569 100.836 10.599 111.000 10.473 11
PointGroup0.636 101.000 10.765 130.624 120.505 130.797 100.116 120.696 90.384 120.441 90.559 60.476 60.596 121.000 10.666 40.756 80.556 170.997 120.513 6
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, Jiaya Jia: PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. CVPR 2020 [oral]
DD-UNet+Group0.635 110.667 220.797 100.714 100.562 50.774 140.146 60.810 40.429 90.476 50.546 80.399 120.633 81.000 10.632 70.722 110.609 101.000 10.514 5
H. Liu, R. Liu, K. Yang, J. Zhang, K. Peng, R. Stiefelhagen: HIDA: Towards Holistic Indoor Understanding for the Visually Impaired via Semantic Instance Segmentation with a Wearable Solid-State LiDAR Sensor. ICCVW 2021
DENet0.629 121.000 10.797 90.608 130.589 30.627 190.219 40.882 10.310 170.402 180.383 210.396 130.650 21.000 10.663 50.543 250.691 21.000 10.568 3
3D-MPA0.611 131.000 10.833 40.765 70.526 100.756 150.136 110.588 160.470 50.438 110.432 170.358 160.650 20.857 140.429 160.765 50.557 161.000 10.430 15
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nie├čner: 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation. CVPR 2020
SphereNet0.606 141.000 10.776 110.745 90.436 180.834 20.035 210.587 180.518 40.338 220.534 100.352 170.594 141.000 10.391 200.696 150.624 91.000 10.451 13
PCJC0.578 151.000 10.810 70.583 170.449 170.813 60.042 200.603 140.341 160.490 40.465 130.410 110.650 20.835 190.264 240.694 160.561 150.889 220.504 8
SSEN0.575 161.000 10.761 140.473 200.477 150.795 110.066 170.529 190.658 10.460 60.461 140.380 140.331 240.859 130.401 190.692 170.653 51.000 10.348 20
Dongsu Zhang, Junha Chun, Sang Kyun Cha, Young Min Kim: Spatial Semantic Embedding Network: Fast 3D Instance Segmentation with Deep Metric Learning. Arxiv
MTML0.549 171.000 10.807 80.588 160.327 210.647 170.004 260.815 30.180 210.418 150.364 220.182 200.445 191.000 10.442 150.688 180.571 131.000 10.396 16
Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald: 3D Instance Segmentation via Multi-task Metric Learning. ICCV 2019 [oral]
Sparse R-CNN0.515 181.000 10.538 270.282 230.468 160.790 120.173 50.345 230.429 80.413 170.484 120.176 210.595 130.591 230.522 130.668 190.476 210.986 170.327 21
Occipital-SCS0.512 191.000 10.716 180.509 190.506 120.611 200.092 150.602 150.177 220.346 200.383 200.165 220.442 200.850 180.386 210.618 210.543 190.889 220.389 17
3D-BoNet0.488 201.000 10.672 220.590 150.301 220.484 280.098 140.620 120.306 180.341 210.259 240.125 240.434 210.796 200.402 180.499 270.513 200.909 210.439 14
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight
PanopticFusion-inst0.478 210.667 220.712 200.595 140.259 240.550 250.000 290.613 130.175 230.250 260.434 150.437 90.411 230.857 140.485 140.591 240.267 290.944 190.359 19
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
SALoss-ResNet0.459 221.000 10.737 170.159 300.259 230.587 220.138 100.475 220.217 200.416 160.408 180.128 230.315 250.714 210.411 170.536 260.590 120.873 250.304 22
Zhidong Liang, Ming Yang, Hao Li, Chunxiang Wang: 3D Instance Embedding Learning With a Structure-Aware Loss Function for Point Cloud Segmentation. IEEE Robotics and Automation Letters (IROS2020)
MASCpermissive0.447 230.528 280.555 250.381 210.382 190.633 180.002 270.509 200.260 190.361 190.432 160.327 180.451 180.571 240.367 220.639 200.386 220.980 180.276 23
Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.
3D-SISpermissive0.382 241.000 10.432 290.245 250.190 250.577 230.013 240.263 250.033 290.320 230.240 250.075 260.422 220.857 140.117 270.699 130.271 280.883 240.235 25
Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019
Hier3Dcopyleft0.323 250.667 220.542 260.264 240.157 280.550 240.000 290.205 280.009 300.270 250.218 260.075 260.500 170.688 220.007 320.698 140.301 260.459 300.200 26
Tan: HCFS3D: Hierarchical Coupled Feature Selection Network for 3D Semantic and Instance Segmentation.
UNet-backbone0.319 260.667 220.715 190.233 260.189 260.479 290.008 250.218 260.067 280.201 270.173 270.107 250.123 290.438 260.150 250.615 220.355 230.916 200.093 31
R-PointNet0.306 270.500 290.405 300.311 220.348 200.589 210.054 180.068 300.126 240.283 240.290 230.028 300.219 270.214 280.331 230.396 300.275 270.821 270.245 24
RandSA0.250 280.333 300.613 230.229 270.163 270.493 260.000 290.304 240.107 250.147 290.100 280.052 290.231 260.119 290.039 290.445 290.325 240.654 280.141 28
Region0.248 290.667 220.437 280.188 280.153 290.491 270.000 290.208 270.094 270.153 280.099 290.057 280.217 280.119 290.039 290.466 280.302 250.640 290.140 29
3D-BEVIS0.248 290.667 220.566 240.076 310.035 320.394 300.027 230.035 310.098 260.099 300.030 310.025 310.098 300.375 270.126 260.604 230.181 300.854 260.171 27
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.
Sgpn_scannet0.143 310.208 320.390 310.169 290.065 300.275 310.029 220.069 290.000 310.087 310.043 300.014 320.027 320.000 310.112 280.351 310.168 310.438 310.138 30
MaskRCNN 2d->3d Proj0.058 320.333 300.002 320.000 320.053 310.002 320.002 280.021 320.000 310.045 320.024 320.238 190.065 310.000 310.014 310.107 320.020 320.110 320.006 32