The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.



This table lists the benchmark results for the 3D semantic instance scenario.




Method Infoavg ap 50%bathtubbedbookshelfcabinetchaircountercurtaindeskdoorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
SoftGroup++0.769 11.000 10.803 130.937 10.684 30.865 30.213 110.870 20.664 20.571 40.758 10.702 40.807 11.000 10.653 110.902 10.792 21.000 10.626 1
SoftGrouppermissive0.761 21.000 10.808 110.845 50.716 10.862 50.243 80.824 30.655 40.620 20.734 20.699 50.791 30.981 180.716 40.844 40.769 31.000 10.594 5
Thang Vu, Kookhoi Kim, Tung M. Luu, Xuan Thanh Nguyen, Chang D. Yoo: SoftGroup for 3D Instance Segmentaiton on Point Clouds. CVPR 2022 [Oral]
GraphCut0.732 31.000 10.788 160.724 150.642 50.859 60.248 70.787 80.618 60.596 30.653 30.722 20.583 211.000 10.766 20.861 20.825 11.000 10.504 13
IPCA-Inst0.731 41.000 10.788 170.884 40.698 20.788 190.252 60.760 100.646 50.511 100.637 50.665 60.804 21.000 10.644 120.778 90.747 41.000 10.561 8
DKNet0.718 51.000 10.814 80.782 90.619 60.872 20.224 90.751 120.569 80.677 10.585 80.724 10.633 140.981 180.515 200.819 60.736 51.000 10.617 2
HAISpermissive0.699 61.000 10.849 30.820 60.675 40.808 130.279 40.757 110.465 130.517 90.596 60.559 80.600 161.000 10.654 100.767 100.676 90.994 230.560 9
Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, Xinggang Wang: Hierarchical Aggregation for 3D Instance Segmentation. ICCV 2021
SSTNetpermissive0.698 71.000 10.697 300.888 30.556 140.803 140.387 20.626 190.417 170.556 70.585 90.702 30.600 161.000 10.824 10.720 210.692 71.000 10.509 12
Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, Kui Jia: Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networks. ICCV2021
SphereSeg0.680 81.000 10.856 20.744 140.618 70.893 10.151 130.651 170.713 10.537 80.579 110.430 180.651 61.000 10.389 290.744 170.697 60.991 240.601 4
MaskVoteNet_Coarse0.677 91.000 10.847 40.771 100.509 200.816 90.277 50.558 260.482 100.562 60.640 40.448 140.700 41.000 10.666 60.852 30.578 190.997 190.488 17
OccuSeg+instance0.672 101.000 10.758 240.682 170.576 120.842 70.477 10.504 290.524 90.567 50.585 100.451 130.557 221.000 10.751 30.797 80.563 221.000 10.467 20
Lei Han, Tian Zheng, Lan Xu, Lu Fang: OccuSeg: Occupancy-aware 3D Instance Segmentation. CVPR2020
Mask-Group0.664 111.000 10.822 70.764 130.616 80.815 100.139 170.694 150.597 70.459 150.566 120.599 70.600 160.516 350.715 50.819 70.635 131.000 10.603 3
Min Zhong, Xinghao Chen, Xiaokang Chen, Gang Zeng, Yunhe Wang: MaskGroup: Hierarchical Point Grouping and Masking for 3D Instance Segmentation. ICME 2022
INS-Conv-instance0.657 121.000 10.760 220.667 190.581 100.863 40.323 30.655 160.477 110.473 130.549 140.432 170.650 71.000 10.655 90.738 180.585 180.944 280.472 19
CSC-Pretrained0.648 131.000 10.810 90.768 110.523 190.813 110.143 160.819 40.389 180.422 220.511 180.443 150.650 71.000 10.624 140.732 190.634 141.000 10.375 26
PE0.645 141.000 10.773 190.798 80.538 160.786 200.088 240.799 70.350 220.435 210.547 150.545 90.646 130.933 200.562 170.761 130.556 270.997 190.501 15
Biao Zhang, Peter Wonka: Point Cloud Instance Segmentation using Probabilistic Embeddings. CVPR 2021
RPGN0.643 151.000 10.758 230.582 270.539 150.826 80.046 280.765 90.372 200.436 200.588 70.539 100.650 71.000 10.577 150.750 150.653 120.997 190.495 16
Dyco3Dcopyleft0.641 161.000 10.841 50.893 20.531 170.802 150.115 210.588 240.448 140.438 180.537 170.430 190.550 230.857 220.534 180.764 120.657 100.987 250.568 6
Tong He; Chunhua Shen; Anton van den Hengel: DyCo3D: Robust Instance Segmentation of 3D Point Clouds through Dynamic Convolution. CVPR2021
GICN0.638 171.000 10.895 10.800 70.480 230.676 240.144 150.737 130.354 210.447 160.400 270.365 240.700 41.000 10.569 160.836 50.599 161.000 10.473 18
PointGroup0.636 181.000 10.765 200.624 210.505 220.797 160.116 200.696 140.384 190.441 170.559 130.476 110.596 191.000 10.666 60.756 140.556 260.997 190.513 11
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, Jiaya Jia: PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. CVPR 2020 [oral]
DD-UNet+Group0.635 190.667 290.797 150.714 160.562 130.774 210.146 140.810 60.429 160.476 120.546 160.399 210.633 141.000 10.632 130.722 200.609 151.000 10.514 10
H. Liu, R. Liu, K. Yang, J. Zhang, K. Peng, R. Stiefelhagen: HIDA: Towards Holistic Indoor Understanding for the Visually Impaired via Semantic Instance Segmentation with a Wearable Solid-State LiDAR Sensor. ICCVW 2021
DENet0.629 201.000 10.797 140.608 220.589 90.627 280.219 100.882 10.310 240.402 260.383 290.396 220.650 71.000 10.663 80.543 350.691 81.000 10.568 7
3D-MPA0.611 211.000 10.833 60.765 120.526 180.756 220.136 190.588 240.470 120.438 190.432 250.358 250.650 70.857 220.429 250.765 110.557 251.000 10.430 22
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nie├čner: 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation. CVPR 2020
PCJC0.578 221.000 10.810 100.583 260.449 260.813 120.042 290.603 220.341 230.490 110.465 210.410 200.650 70.835 280.264 340.694 250.561 230.889 320.504 14
SSEN0.575 231.000 10.761 210.473 290.477 240.795 170.066 250.529 270.658 30.460 140.461 220.380 230.331 340.859 210.401 280.692 260.653 111.000 10.348 28
Dongsu Zhang, Junha Chun, Sang Kyun Cha, Young Min Kim: Spatial Semantic Embedding Network: Fast 3D Instance Segmentation with Deep Metric Learning. Arxiv
RWSeg0.567 240.528 370.708 290.626 200.580 110.745 230.063 260.627 180.240 280.400 270.497 190.464 120.515 241.000 10.475 220.745 160.571 201.000 10.429 23
MTML0.549 251.000 10.807 120.588 250.327 300.647 260.004 340.815 50.180 300.418 230.364 300.182 290.445 281.000 10.442 240.688 270.571 211.000 10.396 24
Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald: 3D Instance Segmentation via Multi-task Metric Learning. ICCV 2019 [oral]
Sparse R-CNN0.515 261.000 10.538 370.282 320.468 250.790 180.173 120.345 330.429 150.413 250.484 200.176 300.595 200.591 330.522 190.668 280.476 310.986 260.327 29
Occipital-SCS0.512 271.000 10.716 260.509 280.506 210.611 290.092 230.602 230.177 310.346 300.383 280.165 310.442 290.850 270.386 300.618 310.543 280.889 320.389 25
3D-BoNet0.488 281.000 10.672 320.590 240.301 320.484 380.098 220.620 200.306 250.341 310.259 340.125 330.434 310.796 290.402 270.499 370.513 300.909 310.439 21
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight
PanopticFusion-inst0.478 290.667 290.712 280.595 230.259 340.550 350.000 370.613 210.175 320.250 360.434 230.437 160.411 330.857 220.485 210.591 340.267 400.944 280.359 27
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
SPG_WSIS0.470 300.667 290.685 310.677 180.372 280.562 330.000 370.482 300.244 270.316 330.298 310.052 390.442 300.857 220.267 330.702 220.559 241.000 10.287 31
SALoss-ResNet0.459 311.000 10.737 250.159 410.259 330.587 310.138 180.475 310.217 290.416 240.408 260.128 320.315 350.714 300.411 260.536 360.590 170.873 350.304 30
Zhidong Liang, Ming Yang, Hao Li, Chunxiang Wang: 3D Instance Embedding Learning With a Structure-Aware Loss Function for Point Cloud Segmentation. IEEE Robotics and Automation Letters (IROS2020)
MASCpermissive0.447 320.528 370.555 350.381 300.382 270.633 270.002 350.509 280.260 260.361 290.432 240.327 260.451 270.571 340.367 310.639 290.386 320.980 270.276 32
Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.
SegGroup_inspermissive0.445 330.667 290.773 180.185 380.317 310.656 250.000 370.407 320.134 330.381 280.267 330.217 280.476 260.714 300.452 230.629 300.514 291.000 10.222 35
An Tao, Yueqi Duan, Yi Wei, Jiwen Lu, Jie Zhou: SegGroup: Seg-Level Supervision for 3D Instance and Semantic Segmentation.
3D-SISpermissive0.382 341.000 10.432 390.245 340.190 350.577 320.013 320.263 350.033 390.320 320.240 350.075 350.422 320.857 220.117 370.699 230.271 390.883 340.235 34
Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019
Hier3Dcopyleft0.323 350.667 290.542 360.264 330.157 380.550 340.000 370.205 380.009 400.270 350.218 360.075 350.500 250.688 320.007 430.698 240.301 360.459 400.200 36
Tan: HCFS3D: Hierarchical Coupled Feature Selection Network for 3D Semantic and Instance Segmentation.
UNet-backbone0.319 360.667 290.715 270.233 350.189 360.479 390.008 330.218 360.067 380.201 370.173 370.107 340.123 400.438 360.150 350.615 320.355 330.916 300.093 42
R-PointNet0.306 370.500 390.405 400.311 310.348 290.589 300.054 270.068 410.126 340.283 340.290 320.028 400.219 380.214 390.331 320.396 410.275 370.821 370.245 33
SemRegionNet0.250 380.333 400.613 330.229 360.163 370.493 360.000 370.304 340.107 350.147 390.100 380.052 380.231 360.119 400.039 390.445 390.325 340.654 380.141 38
3D-BEVIS0.248 390.667 290.566 340.076 420.035 430.394 410.027 310.035 420.098 360.099 410.030 420.025 410.098 410.375 380.126 360.604 330.181 410.854 360.171 37
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.
Region0.248 390.667 290.437 380.188 370.153 390.491 370.000 370.208 370.094 370.153 380.099 390.057 370.217 390.119 400.039 390.466 380.302 350.640 390.140 39
ASIS0.199 410.333 400.253 420.167 400.140 400.438 400.000 370.177 390.008 410.121 400.069 400.004 430.231 370.429 370.036 410.445 400.273 380.333 420.119 41
Sgpn_scannet0.143 420.208 430.390 410.169 390.065 410.275 420.029 300.069 400.000 420.087 420.043 410.014 420.027 430.000 420.112 380.351 420.168 420.438 410.138 40
MaskRCNN 2d->3d Proj0.058 430.333 400.002 430.000 430.053 420.002 430.002 360.021 430.000 420.045 430.024 430.238 270.065 420.000 420.014 420.107 430.020 430.110 430.006 43