The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.



This table lists the benchmark results for the 3D semantic instance scenario.




Method Infoavg ap 50%bathtubbedbookshelfcabinetchaircountercurtaindeskdoorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
OneFormer3D0.801 11.000 10.973 10.909 40.698 60.928 20.582 10.668 260.685 90.780 20.687 70.698 90.702 91.000 10.794 50.900 20.784 70.986 420.635 3
UniPerception0.800 21.000 10.930 40.872 80.727 20.862 140.454 100.764 130.820 10.746 40.706 50.750 10.772 70.926 340.764 80.818 170.826 10.997 320.660 2
Queryformer0.787 31.000 10.933 30.601 380.754 10.886 70.558 30.661 280.767 40.665 90.716 30.639 150.808 21.000 10.844 10.897 30.804 31.000 10.624 5
MAFT0.786 41.000 10.894 90.807 150.694 80.893 50.486 60.674 240.740 50.786 10.704 60.727 30.739 81.000 10.707 140.849 70.756 141.000 10.685 1
Mask3D0.780 51.000 10.786 320.716 290.696 70.885 80.500 50.714 190.810 30.672 80.715 40.679 110.809 11.000 10.831 30.833 110.787 61.000 10.602 11
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, Bastian Leibe: Mask3D for 3D Semantic Instance Segmentation. ICRA 2023
SPFormerpermissive0.770 60.903 450.903 60.806 160.609 200.886 60.568 20.815 60.705 80.711 50.655 90.652 140.685 141.000 10.789 60.809 180.776 101.000 10.583 16
Sun Jiahao, Qing Chunmei, Tan Junpeng, Xu Xiangmin: Superpoint Transformer for 3D Scene Instance Segmentation. AAAI 2023 [Oral]
SoftGroup++0.769 71.000 10.803 250.937 10.684 90.865 110.213 250.870 20.664 120.571 150.758 10.702 70.807 31.000 10.653 210.902 10.792 51.000 10.626 4
SIM3D0.766 81.000 10.948 20.582 440.599 220.882 90.510 40.701 210.632 160.772 30.685 80.687 100.782 61.000 10.833 20.756 280.798 41.000 10.622 6
SoftGrouppermissive0.761 91.000 10.808 210.845 100.716 30.862 130.243 220.824 40.655 140.620 100.734 20.699 80.791 50.981 280.716 120.844 80.769 111.000 10.594 14
Thang Vu, Kookhoi Kim, Tung M. Luu, Xuan Thanh Nguyen, Chang D. Yoo: SoftGroup for 3D Instance Segmentaiton on Point Clouds. CVPR 2022 [Oral]
ISBNetpermissive0.757 101.000 10.904 50.731 270.678 100.895 30.458 80.644 320.670 110.710 60.620 160.732 20.650 161.000 10.756 90.778 210.779 81.000 10.614 8
Tuan Duc Ngo, Binh-Son Hua, Khoi Nguyen: ISBNet: a 3D Point Cloud Instance Segmentation Network with Instance-aware Sampling and Box-aware Dynamic Convolution. CVPR 2023
TD3Dpermissive0.751 111.000 10.774 330.867 90.621 160.934 10.404 110.706 200.812 20.605 130.633 140.626 160.690 131.000 10.640 230.820 140.777 91.000 10.612 9
Maksim Kolodiazhnyi, Anna Vorontsova, Anton Konushin, Danila Rukhovich: Top-Down Beats Bottom-Up in 3D Instance Segmentation. WACV 2024
PBNetpermissive0.747 121.000 10.818 170.837 120.713 40.844 160.457 90.647 310.711 70.614 110.617 170.657 130.650 161.000 10.692 150.822 130.765 131.000 10.595 13
W.Zhao, Y.Yan, C.Yang, J.Ye,X.Yang,K.Huang: Divide and Conquer: 3D Instance Segmentation With Point-Wise Binarization. ICCV 2023
GraphCut0.732 131.000 10.788 300.724 280.642 140.859 150.248 210.787 110.618 180.596 140.653 110.722 50.583 361.000 10.766 70.861 40.825 21.000 10.504 27
IPCA-Inst0.731 141.000 10.788 310.884 70.698 50.788 310.252 200.760 140.646 150.511 230.637 130.665 120.804 41.000 10.644 220.778 220.747 161.000 10.561 20
TopoSeg0.725 151.000 10.806 240.933 20.668 120.758 350.272 190.734 180.630 170.549 190.654 100.606 170.697 120.966 310.612 270.839 90.754 151.000 10.573 17
DKNet0.718 161.000 10.814 180.782 190.619 170.872 100.224 230.751 160.569 220.677 70.585 210.724 40.633 270.981 280.515 370.819 150.736 171.000 10.617 7
Yizheng Wu, Min Shi, Shuaiyuan Du, Hao Lu, Zhiguo Cao, Weicai Zhong: 3D Instances as 1D Kernels. ECCV 2022
SSEC0.707 171.000 10.850 110.924 30.648 130.747 380.162 270.862 30.572 210.520 210.624 150.549 200.649 251.000 10.560 320.706 380.768 121.000 10.591 15
HAISpermissive0.699 181.000 10.849 120.820 130.675 110.808 250.279 170.757 150.465 280.517 220.596 190.559 190.600 301.000 10.654 200.767 240.676 210.994 380.560 21
Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, Xinggang Wang: Hierarchical Aggregation for 3D Instance Segmentation. ICCV 2021
SSTNetpermissive0.698 191.000 10.697 490.888 60.556 280.803 260.387 120.626 340.417 320.556 180.585 220.702 60.600 301.000 10.824 40.720 370.692 191.000 10.509 26
Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, Kui Jia: Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networks. ICCV2021
DualGroup0.694 201.000 10.799 270.811 140.622 150.817 200.376 130.805 90.590 200.487 260.568 250.525 240.650 160.835 440.600 280.829 120.655 231.000 10.526 23
DANCENET0.680 211.000 10.807 220.733 260.600 210.768 340.375 140.543 420.538 230.610 120.599 180.498 250.632 290.981 280.739 110.856 50.633 290.882 530.454 36
SphereSeg0.680 211.000 10.856 100.744 250.618 180.893 40.151 280.651 300.713 60.537 200.579 240.430 340.651 151.000 10.389 470.744 320.697 180.991 400.601 12
Box2Mask0.677 231.000 10.847 130.771 210.509 370.816 210.277 180.558 410.482 250.562 170.640 120.448 300.700 101.000 10.666 160.852 60.578 360.997 320.488 31
Julian Chibane, Francis Engelmann, Tuan Anh Tran, Gerard Pons-Moll: Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation Using Bounding Boxes. ECCV 2022
OccuSeg+instance0.672 241.000 10.758 410.682 320.576 260.842 170.477 70.504 470.524 240.567 160.585 230.451 290.557 381.000 10.751 100.797 190.563 391.000 10.467 35
Lei Han, Tian Zheng, Lan Xu, Lu Fang: OccuSeg: Occupancy-aware 3D Instance Segmentation. CVPR2020
Mask-Group0.664 251.000 10.822 160.764 240.616 190.815 220.139 320.694 230.597 190.459 300.566 260.599 180.600 300.516 540.715 130.819 160.635 271.000 10.603 10
Min Zhong, Xinghao Chen, Xiaokang Chen, Gang Zeng, Yunhe Wang: MaskGroup: Hierarchical Point Grouping and Masking for 3D Instance Segmentation. ICME 2022
INS-Conv-instance0.657 261.000 10.760 390.667 340.581 240.863 120.323 150.655 290.477 260.473 280.549 280.432 330.650 161.000 10.655 190.738 330.585 350.944 450.472 34
CSC-Pretrained0.648 271.000 10.810 190.768 220.523 350.813 230.143 310.819 50.389 350.422 390.511 320.443 310.650 161.000 10.624 250.732 340.634 281.000 10.375 43
PE0.645 281.000 10.773 350.798 180.538 300.786 320.088 400.799 100.350 390.435 370.547 290.545 210.646 260.933 330.562 310.761 270.556 440.997 320.501 29
Biao Zhang, Peter Wonka: Point Cloud Instance Segmentation using Probabilistic Embeddings. CVPR 2021
RPGN0.643 291.000 10.758 400.582 450.539 290.826 190.046 440.765 120.372 370.436 360.588 200.539 230.650 161.000 10.577 290.750 300.653 250.997 320.495 30
Shichao Dong, Guosheng Lin, Tzu-Yi Hung: Learning Regional Purity for Instance Segmentation on 3D Point Clouds. ECCV 2022
Dyco3Dcopyleft0.641 301.000 10.841 140.893 50.531 320.802 270.115 370.588 390.448 290.438 340.537 310.430 350.550 390.857 360.534 350.764 260.657 220.987 410.568 18
Tong He; Chunhua Shen; Anton van den Hengel: DyCo3D: Robust Instance Segmentation of 3D Point Clouds through Dynamic Convolution. CVPR2021
GICN0.638 311.000 10.895 80.800 170.480 410.676 430.144 300.737 170.354 380.447 310.400 450.365 400.700 101.000 10.569 300.836 100.599 311.000 10.473 33
PointGroup0.636 321.000 10.765 360.624 360.505 390.797 280.116 360.696 220.384 360.441 320.559 270.476 270.596 331.000 10.666 160.756 290.556 430.997 320.513 25
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, Jiaya Jia: PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. CVPR 2020 [oral]
DD-UNet+Group0.635 330.667 470.797 290.714 300.562 270.774 330.146 290.810 80.429 310.476 270.546 300.399 370.633 271.000 10.632 240.722 360.609 301.000 10.514 24
H. Liu, R. Liu, K. Yang, J. Zhang, K. Peng, R. Stiefelhagen: HIDA: Towards Holistic Indoor Understanding for the Visually Impaired via Semantic Instance Segmentation with a Wearable Solid-State LiDAR Sensor. ICCVW 2021
DENet0.629 341.000 10.797 280.608 370.589 230.627 470.219 240.882 10.310 410.402 440.383 470.396 380.650 161.000 10.663 180.543 550.691 201.000 10.568 19
3D-MPA0.611 351.000 10.833 150.765 230.526 340.756 360.136 340.588 390.470 270.438 350.432 410.358 420.650 160.857 360.429 430.765 250.557 421.000 10.430 38
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nie├čner: 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation. CVPR 2020
OSIS0.605 361.000 10.801 260.599 390.535 310.728 400.286 160.436 510.679 100.491 240.433 390.256 440.404 510.857 360.620 260.724 350.510 491.000 10.539 22
AOIA0.601 371.000 10.761 380.687 310.485 400.828 180.008 510.663 270.405 340.405 430.425 420.490 260.596 330.714 470.553 340.779 200.597 320.992 390.424 40
PCJC0.578 381.000 10.810 200.583 430.449 440.813 240.042 450.603 370.341 400.490 250.465 360.410 360.650 160.835 440.264 530.694 420.561 400.889 500.504 28
SSEN0.575 391.000 10.761 370.473 470.477 420.795 290.066 410.529 440.658 130.460 290.461 370.380 390.331 530.859 350.401 460.692 440.653 241.000 10.348 45
Dongsu Zhang, Junha Chun, Sang Kyun Cha, Young Min Kim: Spatial Semantic Embedding Network: Fast 3D Instance Segmentation with Deep Metric Learning. Arxiv
RWSeg0.567 400.528 570.708 480.626 350.580 250.745 390.063 420.627 330.240 450.400 450.497 330.464 280.515 401.000 10.475 390.745 310.571 371.000 10.429 39
NeuralBF0.555 410.667 470.896 70.843 110.517 360.751 370.029 460.519 450.414 330.439 330.465 350.000 630.484 420.857 360.287 510.693 430.651 261.000 10.485 32
Weiwei Sun, Daniel Rebain, Renjie Liao, Vladimir Tankovich, Soroosh Yazdani, Kwang Moo Yi, Andrea Tagliasacchi: NeuralBF: Neural Bilateral Filtering for Top-down Instance Segmentation on Point Clouds. WACV 2023
MTML0.549 421.000 10.807 230.588 420.327 490.647 450.004 530.815 70.180 480.418 400.364 490.182 470.445 451.000 10.442 420.688 450.571 381.000 10.396 41
Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald: 3D Instance Segmentation via Multi-task Metric Learning. ICCV 2019 [oral]
ClickSeg_Instance0.539 431.000 10.621 520.300 500.530 330.698 410.127 350.533 430.222 460.430 380.400 440.365 400.574 370.938 320.472 400.659 470.543 450.944 450.347 46
One_Thing_One_Clickpermissive0.529 440.667 470.718 440.777 200.399 450.683 420.000 560.669 250.138 510.391 460.374 480.539 220.360 520.641 510.556 330.774 230.593 330.997 320.251 51
Zhengzhe Liu, Xiaojuan Qi, Chi-Wing Fu: One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation. CVPR 2021
Sparse R-CNN0.515 451.000 10.538 570.282 510.468 430.790 300.173 260.345 530.429 300.413 420.484 340.176 480.595 350.591 520.522 360.668 460.476 500.986 430.327 47
Occipital-SCS0.512 461.000 10.716 450.509 460.506 380.611 480.092 390.602 380.177 490.346 490.383 460.165 490.442 460.850 430.386 480.618 510.543 460.889 500.389 42
3D-BoNet0.488 471.000 10.672 510.590 410.301 510.484 580.098 380.620 350.306 420.341 500.259 530.125 510.434 480.796 460.402 450.499 570.513 480.909 490.439 37
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight
PanopticFusion-inst0.478 480.667 470.712 470.595 400.259 540.550 540.000 560.613 360.175 500.250 550.434 380.437 320.411 500.857 360.485 380.591 540.267 600.944 450.359 44
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
SPG_WSIS0.470 490.667 470.685 500.677 330.372 470.562 520.000 560.482 480.244 440.316 520.298 500.052 580.442 470.857 360.267 520.702 390.559 411.000 10.287 49
SALoss-ResNet0.459 501.000 10.737 430.159 610.259 530.587 500.138 330.475 490.217 470.416 410.408 430.128 500.315 540.714 470.411 440.536 560.590 340.873 540.304 48
Zhidong Liang, Ming Yang, Hao Li, Chunxiang Wang: 3D Instance Embedding Learning With a Structure-Aware Loss Function for Point Cloud Segmentation. IEEE Robotics and Automation Letters (IROS2020)
MASCpermissive0.447 510.528 570.555 550.381 480.382 460.633 460.002 540.509 460.260 430.361 480.432 400.327 430.451 440.571 530.367 490.639 490.386 510.980 440.276 50
Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.
SegGroup_inspermissive0.445 520.667 470.773 340.185 580.317 500.656 440.000 560.407 520.134 520.381 470.267 520.217 460.476 430.714 470.452 410.629 500.514 471.000 10.222 54
An Tao, Yueqi Duan, Yi Wei, Jiwen Lu, Jie Zhou: SegGroup: Seg-Level Supervision for 3D Instance and Semantic Segmentation. TIP 2022
3D-SISpermissive0.382 531.000 10.432 600.245 530.190 550.577 510.013 500.263 550.033 580.320 510.240 540.075 540.422 490.857 360.117 580.699 400.271 590.883 520.235 53
Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019
Hier3Dcopyleft0.323 540.667 470.542 560.264 520.157 580.550 530.000 560.205 580.009 600.270 540.218 550.075 540.500 410.688 500.007 640.698 410.301 560.459 610.200 55
Tan: HCFS3D: Hierarchical Coupled Feature Selection Network for 3D Semantic and Instance Segmentation.
UNet-backbone0.319 550.667 470.715 460.233 540.189 560.479 590.008 510.218 560.067 570.201 570.173 560.107 520.123 590.438 550.150 550.615 520.355 520.916 480.093 63
R-PointNet0.306 560.500 590.405 610.311 490.348 480.589 490.054 430.068 610.126 530.283 530.290 510.028 590.219 570.214 580.331 500.396 610.275 570.821 560.245 52
Region-18class0.284 570.250 630.751 420.228 560.270 520.521 550.000 560.468 500.008 620.205 560.127 570.000 630.068 610.070 620.262 540.652 480.323 540.740 570.173 56
SemRegionNet-20cls0.250 580.333 600.613 530.229 550.163 570.493 560.000 560.304 540.107 540.147 600.100 590.052 570.231 550.119 600.039 600.445 590.325 530.654 580.141 59
tmp0.248 590.667 470.437 590.188 570.153 590.491 570.000 560.208 570.094 560.153 590.099 600.057 560.217 580.119 600.039 600.466 580.302 550.640 590.140 60
3D-BEVIS0.248 590.667 470.566 540.076 620.035 640.394 620.027 480.035 630.098 550.099 620.030 630.025 600.098 600.375 570.126 570.604 530.181 620.854 550.171 57
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.
Sem_Recon_ins0.227 610.764 460.486 580.069 630.098 610.426 610.017 490.067 620.015 590.172 580.100 580.096 530.054 630.183 590.135 560.366 620.260 610.614 600.168 58
ASIS0.199 620.333 600.253 630.167 600.140 600.438 600.000 560.177 590.008 610.121 610.069 610.004 620.231 560.429 560.036 620.445 600.273 580.333 630.119 62
Sgpn_scannet0.143 630.208 640.390 620.169 590.065 620.275 630.029 470.069 600.000 630.087 630.043 620.014 610.027 640.000 630.112 590.351 630.168 630.438 620.138 61
MaskRCNN 2d->3d Proj0.058 640.333 600.002 640.000 640.053 630.002 640.002 550.021 640.000 630.045 640.024 640.238 450.065 620.000 630.014 630.107 640.020 640.110 640.006 64