The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.



This table lists the benchmark results for the 3D semantic instance scenario.




Method Infoavg ap 50%bathtubbedbookshelfcabinetchaircountercurtaindeskdoorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
Mask3D0.780 11.000 10.786 220.716 210.696 40.885 30.500 20.714 170.810 10.672 30.715 30.679 60.809 11.000 10.831 10.833 80.787 31.000 10.602 4
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, Bastian Leibe: Mask3D for 3D Semantic Instance Segmentation.
SPFormerpermissive0.770 20.903 330.903 10.806 90.609 120.886 20.568 10.815 60.705 30.711 10.655 40.652 80.685 81.000 10.789 30.809 110.776 41.000 10.583 7
Sun Jiahao, Qing Chunmei, Tan Junpeng, Xu Xiangmin: Superpoint Transformer for 3D Scene Instance Segmentation. AAAI 2023
SoftGroup++0.769 31.000 10.803 170.937 10.684 50.865 50.213 160.870 20.664 40.571 60.758 10.702 40.807 21.000 10.653 150.902 10.792 21.000 10.626 1
SoftGrouppermissive0.761 41.000 10.808 140.845 60.716 10.862 70.243 130.824 30.655 60.620 40.734 20.699 50.791 40.981 220.716 60.844 40.769 51.000 10.594 6
Thang Vu, Kookhoi Kim, Tung M. Luu, Xuan Thanh Nguyen, Chang D. Yoo: SoftGroup for 3D Instance Segmentaiton on Point Clouds. CVPR 2022 [Oral]
PBNetpermissive0.734 51.000 10.810 110.824 70.704 20.830 100.492 30.696 180.601 110.562 80.593 100.587 110.650 101.000 10.698 90.844 50.762 61.000 10.556 14
GraphCut0.732 61.000 10.788 200.724 200.642 80.859 80.248 120.787 100.618 100.596 50.653 60.722 20.583 261.000 10.766 40.861 20.825 11.000 10.504 18
IPCA-Inst0.731 71.000 10.788 210.884 50.698 30.788 230.252 110.760 120.646 70.511 150.637 80.665 70.804 31.000 10.644 160.778 130.747 81.000 10.561 12
TopoSeg0.725 81.000 10.806 160.933 20.668 70.758 260.272 90.734 160.630 80.549 110.654 50.606 90.697 70.966 240.612 190.839 60.754 71.000 10.573 8
DKNet0.718 91.000 10.814 100.782 120.619 90.872 40.224 140.751 140.569 130.677 20.585 120.724 10.633 180.981 220.515 260.819 90.736 91.000 10.617 2
Yizheng Wu, Min Shi, Shuaiyuan Du, Hao Lu, Zhiguo Cao, Weicai Zhong: 3D Instances as 1D Kernels. ECCV 2022
SSEC0.700 101.000 10.848 50.763 180.609 130.792 210.262 100.824 30.627 90.535 130.547 200.493 160.600 201.000 10.712 80.731 250.689 131.000 10.563 11
HAISpermissive0.699 111.000 10.849 40.820 80.675 60.808 160.279 70.757 130.465 180.517 140.596 90.559 120.600 201.000 10.654 140.767 150.676 140.994 290.560 13
Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, Xinggang Wang: Hierarchical Aggregation for 3D Instance Segmentation. ICCV 2021
SSTNetpermissive0.698 121.000 10.697 370.888 40.556 190.803 170.387 50.626 250.417 220.556 100.585 130.702 30.600 201.000 10.824 20.720 270.692 111.000 10.509 17
Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, Kui Jia: Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networks. ICCV2021
SphereSeg0.680 131.000 10.856 30.744 190.618 100.893 10.151 180.651 230.713 20.537 120.579 150.430 240.651 91.000 10.389 350.744 220.697 100.991 300.601 5
Box2Mask0.677 141.000 10.847 60.771 140.509 250.816 120.277 80.558 320.482 150.562 90.640 70.448 200.700 51.000 10.666 100.852 30.578 250.997 240.488 22
Julian Chibane, Francis Engelmann, Tuan Anh Tran, Gerard Pons-Moll: Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation Using Bounding Boxes. ECCV 2022
OccuSeg+instance0.672 151.000 10.758 290.682 230.576 170.842 90.477 40.504 350.524 140.567 70.585 140.451 190.557 271.000 10.751 50.797 120.563 281.000 10.467 25
Lei Han, Tian Zheng, Lan Xu, Lu Fang: OccuSeg: Occupancy-aware 3D Instance Segmentation. CVPR2020
Mask-Group0.664 161.000 10.822 90.764 170.616 110.815 130.139 220.694 200.597 120.459 200.566 160.599 100.600 200.516 410.715 70.819 100.635 181.000 10.603 3
Min Zhong, Xinghao Chen, Xiaokang Chen, Gang Zeng, Yunhe Wang: MaskGroup: Hierarchical Point Grouping and Masking for 3D Instance Segmentation. ICME 2022
INS-Conv-instance0.657 171.000 10.760 270.667 250.581 150.863 60.323 60.655 220.477 160.473 180.549 180.432 230.650 101.000 10.655 130.738 230.585 240.944 340.472 24
CSC-Pretrained0.648 181.000 10.810 120.768 150.523 240.813 140.143 210.819 50.389 230.422 270.511 230.443 210.650 101.000 10.624 180.732 240.634 191.000 10.375 31
PE0.645 191.000 10.773 240.798 110.538 210.786 240.088 290.799 90.350 270.435 260.547 190.545 130.646 170.933 250.562 220.761 180.556 330.997 240.501 20
Biao Zhang, Peter Wonka: Point Cloud Instance Segmentation using Probabilistic Embeddings. CVPR 2021
RPGN0.643 201.000 10.758 280.582 330.539 200.826 110.046 330.765 110.372 250.436 250.588 110.539 150.650 101.000 10.577 200.750 200.653 170.997 240.495 21
Shichao Dong, Guosheng Lin, Tzu-Yi Hung: Learning Regional Purity for Instance Segmentation on 3D Point Clouds. ECCV 2022
Dyco3Dcopyleft0.641 211.000 10.841 70.893 30.531 220.802 180.115 260.588 300.448 190.438 230.537 220.430 250.550 280.857 270.534 240.764 170.657 150.987 310.568 9
Tong He; Chunhua Shen; Anton van den Hengel: DyCo3D: Robust Instance Segmentation of 3D Point Clouds through Dynamic Convolution. CVPR2021
GICN0.638 221.000 10.895 20.800 100.480 280.676 300.144 200.737 150.354 260.447 210.400 320.365 300.700 51.000 10.569 210.836 70.599 211.000 10.473 23
PointGroup0.636 231.000 10.765 250.624 270.505 270.797 190.116 250.696 180.384 240.441 220.559 170.476 170.596 241.000 10.666 100.756 190.556 320.997 240.513 16
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, Jiaya Jia: PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. CVPR 2020 [oral]
DD-UNet+Group0.635 240.667 340.797 190.714 220.562 180.774 250.146 190.810 80.429 210.476 170.546 210.399 270.633 181.000 10.632 170.722 260.609 201.000 10.514 15
H. Liu, R. Liu, K. Yang, J. Zhang, K. Peng, R. Stiefelhagen: HIDA: Towards Holistic Indoor Understanding for the Visually Impaired via Semantic Instance Segmentation with a Wearable Solid-State LiDAR Sensor. ICCVW 2021
DENet0.629 251.000 10.797 180.608 280.589 140.627 340.219 150.882 10.310 290.402 310.383 340.396 280.650 101.000 10.663 120.543 420.691 121.000 10.568 10
3D-MPA0.611 261.000 10.833 80.765 160.526 230.756 270.136 240.588 300.470 170.438 240.432 300.358 310.650 100.857 270.429 310.765 160.557 311.000 10.430 27
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nie├čner: 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation. CVPR 2020
PCJC0.578 271.000 10.810 130.583 320.449 310.813 150.042 340.603 280.341 280.490 160.465 260.410 260.650 100.835 330.264 400.694 310.561 290.889 380.504 19
SSEN0.575 281.000 10.761 260.473 350.477 290.795 200.066 300.529 330.658 50.460 190.461 270.380 290.331 400.859 260.401 340.692 320.653 161.000 10.348 33
Dongsu Zhang, Junha Chun, Sang Kyun Cha, Young Min Kim: Spatial Semantic Embedding Network: Fast 3D Instance Segmentation with Deep Metric Learning. Arxiv
RWSeg0.567 290.528 430.708 360.626 260.580 160.745 280.063 310.627 240.240 330.400 320.497 240.464 180.515 291.000 10.475 280.745 210.571 261.000 10.429 28
MTML0.549 301.000 10.807 150.588 310.327 360.647 320.004 390.815 70.180 350.418 280.364 360.182 350.445 331.000 10.442 300.688 330.571 271.000 10.396 29
Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald: 3D Instance Segmentation via Multi-task Metric Learning. ICCV 2019 [oral]
One_Thing_One_Clickpermissive0.529 310.667 340.718 320.777 130.399 320.683 290.000 420.669 210.138 380.391 330.374 350.539 140.360 390.641 380.556 230.774 140.593 220.997 240.251 38
Zhengzhe Liu, Xiaojuan Qi, Chi-Wing Fu: One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation. CVPR 2021
Sparse R-CNN0.515 321.000 10.538 440.282 380.468 300.790 220.173 170.345 400.429 200.413 300.484 250.176 360.595 250.591 390.522 250.668 340.476 370.986 320.327 34
Occipital-SCS0.512 331.000 10.716 330.509 340.506 260.611 350.092 280.602 290.177 360.346 360.383 330.165 370.442 340.850 320.386 360.618 380.543 340.889 380.389 30
3D-BoNet0.488 341.000 10.672 390.590 300.301 380.484 450.098 270.620 260.306 300.341 370.259 400.125 390.434 360.796 340.402 330.499 440.513 360.909 370.439 26
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight
PanopticFusion-inst0.478 350.667 340.712 350.595 290.259 410.550 410.000 420.613 270.175 370.250 420.434 280.437 220.411 380.857 270.485 270.591 410.267 470.944 340.359 32
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
SPG_WSIS0.470 360.667 340.685 380.677 240.372 340.562 390.000 420.482 360.244 320.316 390.298 370.052 450.442 350.857 270.267 390.702 280.559 301.000 10.287 36
SALoss-ResNet0.459 371.000 10.737 310.159 480.259 400.587 370.138 230.475 370.217 340.416 290.408 310.128 380.315 410.714 350.411 320.536 430.590 230.873 410.304 35
Zhidong Liang, Ming Yang, Hao Li, Chunxiang Wang: 3D Instance Embedding Learning With a Structure-Aware Loss Function for Point Cloud Segmentation. IEEE Robotics and Automation Letters (IROS2020)
MASCpermissive0.447 380.528 430.555 420.381 360.382 330.633 330.002 400.509 340.260 310.361 350.432 290.327 320.451 320.571 400.367 370.639 360.386 380.980 330.276 37
Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.
SegGroup_inspermissive0.445 390.667 340.773 230.185 450.317 370.656 310.000 420.407 390.134 390.381 340.267 390.217 340.476 310.714 350.452 290.629 370.514 351.000 10.222 41
An Tao, Yueqi Duan, Yi Wei, Jiwen Lu, Jie Zhou: SegGroup: Seg-Level Supervision for 3D Instance and Semantic Segmentation. TIP 2022
3D-SISpermissive0.382 401.000 10.432 460.245 400.190 420.577 380.013 370.263 420.033 450.320 380.240 410.075 410.422 370.857 270.117 440.699 290.271 460.883 400.235 40
Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019
Hier3Dcopyleft0.323 410.667 340.542 430.264 390.157 450.550 400.000 420.205 450.009 460.270 410.218 420.075 410.500 300.688 370.007 500.698 300.301 430.459 470.200 42
Tan: HCFS3D: Hierarchical Coupled Feature Selection Network for 3D Semantic and Instance Segmentation.
UNet-backbone0.319 420.667 340.715 340.233 410.189 430.479 460.008 380.218 430.067 440.201 440.173 430.107 400.123 460.438 420.150 420.615 390.355 390.916 360.093 49
R-PointNet0.306 430.500 450.405 470.311 370.348 350.589 360.054 320.068 480.126 400.283 400.290 380.028 460.219 440.214 450.331 380.396 480.275 440.821 430.245 39
Region-18class0.284 440.250 490.751 300.228 430.270 390.521 420.000 420.468 380.008 480.205 430.127 440.000 500.068 480.070 480.262 410.652 350.323 410.740 440.173 43
SemRegionNet-20cls0.250 450.333 460.613 400.229 420.163 440.493 430.000 420.304 410.107 410.147 460.100 450.052 440.231 420.119 460.039 460.445 460.325 400.654 450.141 45
3D-BEVIS0.248 460.667 340.566 410.076 490.035 500.394 480.027 360.035 490.098 420.099 480.030 490.025 470.098 470.375 440.126 430.604 400.181 480.854 420.171 44
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.
tmp0.248 460.667 340.437 450.188 440.153 460.491 440.000 420.208 440.094 430.153 450.099 460.057 430.217 450.119 460.039 460.466 450.302 420.640 460.140 46
ASIS0.199 480.333 460.253 490.167 470.140 470.438 470.000 420.177 460.008 470.121 470.069 470.004 490.231 430.429 430.036 480.445 470.273 450.333 490.119 48
Sgpn_scannet0.143 490.208 500.390 480.169 460.065 480.275 490.029 350.069 470.000 490.087 490.043 480.014 480.027 500.000 490.112 450.351 490.168 490.438 480.138 47
MaskRCNN 2d->3d Proj0.058 500.333 460.002 500.000 500.053 490.002 500.002 410.021 500.000 490.045 500.024 500.238 330.065 490.000 490.014 490.107 500.020 500.110 500.006 50