The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.



This table lists the benchmark results for the 3D semantic instance scenario.




Method Infoavg ap 50%bathtubbedbookshelfcabinetchaircountercurtaindeskdoorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
Queryformer0.787 11.000 10.933 10.601 340.754 10.886 40.558 20.661 250.767 30.665 40.716 30.639 110.808 31.000 10.844 10.897 20.804 21.000 10.624 2
Mask3D0.780 21.000 10.786 270.716 250.696 50.885 50.500 40.714 180.810 20.672 30.715 40.679 70.809 21.000 10.831 20.833 80.787 41.000 10.602 6
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, Bastian Leibe: Mask3D for 3D Semantic Instance Segmentation. ICRA 2023
SPFormerpermissive0.770 30.903 390.903 20.806 130.609 170.886 30.568 10.815 60.705 70.711 10.655 60.652 100.685 111.000 10.789 40.809 140.776 71.000 10.583 11
Sun Jiahao, Qing Chunmei, Tan Junpeng, Xu Xiangmin: Superpoint Transformer for 3D Scene Instance Segmentation. AAAI 2023 [Oral]
SoftGroup++0.769 41.000 10.803 200.937 10.684 60.865 70.213 200.870 20.664 90.571 100.758 10.702 40.807 41.000 10.653 160.902 10.792 31.000 10.626 1
ISBNetpermissive0.763 51.000 10.873 50.717 240.666 90.858 110.508 30.667 230.764 40.643 50.676 50.688 60.825 11.000 10.773 50.741 270.777 61.000 10.556 17
Tuan Duc Ngo, Binh-Son Hua, Khoi Nguyen: ISBNet: a 3D Point Cloud Instance Segmentation Network with Instance-aware Sampling and Box-aware Dynamic Convolution. CVPR 2023
SoftGrouppermissive0.761 61.000 10.808 170.845 80.716 20.862 90.243 170.824 40.655 110.620 60.734 20.699 50.791 60.981 250.716 80.844 50.769 81.000 10.594 9
Thang Vu, Kookhoi Kim, Tung M. Luu, Xuan Thanh Nguyen, Chang D. Yoo: SoftGroup for 3D Instance Segmentaiton on Point Clouds. CVPR 2022 [Oral]
TD3D0.751 71.000 10.774 280.867 70.621 130.934 10.404 70.706 190.812 10.605 80.633 110.626 120.690 101.000 10.640 180.820 110.777 51.000 10.612 4
PBNetpermissive0.747 81.000 10.818 130.837 100.713 30.844 120.457 60.647 280.711 60.614 70.617 130.657 90.650 131.000 10.692 100.822 100.765 101.000 10.595 8
GraphCut0.732 91.000 10.788 250.724 230.642 110.859 100.248 160.787 110.618 140.596 90.653 80.722 20.583 311.000 10.766 60.861 30.825 11.000 10.504 23
IPCA-Inst0.731 101.000 10.788 260.884 60.698 40.788 270.252 150.760 130.646 120.511 180.637 100.665 80.804 51.000 10.644 170.778 170.747 121.000 10.561 15
TopoSeg0.725 111.000 10.806 190.933 20.668 80.758 300.272 140.734 170.630 130.549 140.654 70.606 130.697 90.966 270.612 220.839 60.754 111.000 10.573 12
DKNet0.718 121.000 10.814 140.782 160.619 140.872 60.224 180.751 150.569 180.677 20.585 160.724 10.633 230.981 250.515 320.819 120.736 131.000 10.617 3
Yizheng Wu, Min Shi, Shuaiyuan Du, Hao Lu, Zhiguo Cao, Weicai Zhong: 3D Instances as 1D Kernels. ECCV 2022
SSEC0.707 131.000 10.850 70.924 30.648 100.747 330.162 220.862 30.572 170.520 160.624 120.549 160.649 211.000 10.560 270.706 330.768 91.000 10.591 10
HAISpermissive0.699 141.000 10.849 80.820 110.675 70.808 210.279 120.757 140.465 230.517 170.596 140.559 150.600 251.000 10.654 150.767 190.676 170.994 350.560 16
Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, Xinggang Wang: Hierarchical Aggregation for 3D Instance Segmentation. ICCV 2021
SSTNetpermissive0.698 151.000 10.697 440.888 50.556 230.803 220.387 80.626 300.417 270.556 130.585 170.702 30.600 251.000 10.824 30.720 320.692 151.000 10.509 22
Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, Kui Jia: Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networks. ICCV2021
DualGroup0.694 161.000 10.799 220.811 120.622 120.817 160.376 90.805 90.590 160.487 210.568 200.525 200.650 130.835 380.600 230.829 90.655 191.000 10.526 19
SphereSeg0.680 171.000 10.856 60.744 220.618 150.893 20.151 230.651 270.713 50.537 150.579 190.430 290.651 121.000 10.389 410.744 260.697 140.991 370.601 7
Box2Mask0.677 181.000 10.847 90.771 180.509 310.816 170.277 130.558 370.482 200.562 120.640 90.448 250.700 71.000 10.666 110.852 40.578 310.997 300.488 27
Julian Chibane, Francis Engelmann, Tuan Anh Tran, Gerard Pons-Moll: Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation Using Bounding Boxes. ECCV 2022
OccuSeg+instance0.672 191.000 10.758 360.682 280.576 210.842 130.477 50.504 410.524 190.567 110.585 180.451 240.557 321.000 10.751 70.797 150.563 341.000 10.467 31
Lei Han, Tian Zheng, Lan Xu, Lu Fang: OccuSeg: Occupancy-aware 3D Instance Segmentation. CVPR2020
Mask-Group0.664 201.000 10.822 120.764 210.616 160.815 180.139 270.694 210.597 150.459 250.566 210.599 140.600 250.516 480.715 90.819 130.635 231.000 10.603 5
Min Zhong, Xinghao Chen, Xiaokang Chen, Gang Zeng, Yunhe Wang: MaskGroup: Hierarchical Point Grouping and Masking for 3D Instance Segmentation. ICME 2022
INS-Conv-instance0.657 211.000 10.760 340.667 300.581 190.863 80.323 100.655 260.477 210.473 230.549 230.432 280.650 131.000 10.655 140.738 280.585 300.944 410.472 30
CSC-Pretrained0.648 221.000 10.810 150.768 190.523 290.813 190.143 260.819 50.389 300.422 330.511 270.443 260.650 131.000 10.624 200.732 290.634 241.000 10.375 38
PE0.645 231.000 10.773 300.798 150.538 250.786 280.088 340.799 100.350 340.435 320.547 240.545 170.646 220.933 280.562 260.761 220.556 390.997 300.501 25
Biao Zhang, Peter Wonka: Point Cloud Instance Segmentation using Probabilistic Embeddings. CVPR 2021
RPGN0.643 241.000 10.758 350.582 400.539 240.826 150.046 380.765 120.372 320.436 310.588 150.539 190.650 131.000 10.577 240.750 240.653 210.997 300.495 26
Shichao Dong, Guosheng Lin, Tzu-Yi Hung: Learning Regional Purity for Instance Segmentation on 3D Point Clouds. ECCV 2022
Dyco3Dcopyleft0.641 251.000 10.841 100.893 40.531 270.802 230.115 310.588 350.448 240.438 290.537 260.430 300.550 330.857 300.534 300.764 210.657 180.987 380.568 13
Tong He; Chunhua Shen; Anton van den Hengel: DyCo3D: Robust Instance Segmentation of 3D Point Clouds through Dynamic Convolution. CVPR2021
GICN0.638 261.000 10.895 40.800 140.480 350.676 370.144 250.737 160.354 330.447 260.400 390.365 350.700 71.000 10.569 250.836 70.599 261.000 10.473 29
PointGroup0.636 271.000 10.765 310.624 320.505 330.797 240.116 300.696 200.384 310.441 270.559 220.476 220.596 281.000 10.666 110.756 230.556 380.997 300.513 21
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, Jiaya Jia: PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. CVPR 2020 [oral]
DD-UNet+Group0.635 280.667 400.797 240.714 260.562 220.774 290.146 240.810 80.429 260.476 220.546 250.399 320.633 231.000 10.632 190.722 310.609 251.000 10.514 20
H. Liu, R. Liu, K. Yang, J. Zhang, K. Peng, R. Stiefelhagen: HIDA: Towards Holistic Indoor Understanding for the Visually Impaired via Semantic Instance Segmentation with a Wearable Solid-State LiDAR Sensor. ICCVW 2021
DENet0.629 291.000 10.797 230.608 330.589 180.627 410.219 190.882 10.310 360.402 380.383 410.396 330.650 131.000 10.663 130.543 490.691 161.000 10.568 14
3D-MPA0.611 301.000 10.833 110.765 200.526 280.756 310.136 290.588 350.470 220.438 300.432 360.358 360.650 130.857 300.429 370.765 200.557 371.000 10.430 33
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nießner: 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation. CVPR 2020
OSIS0.605 311.000 10.801 210.599 350.535 260.728 350.286 110.436 450.679 80.491 190.433 340.256 380.404 450.857 300.620 210.724 300.510 431.000 10.539 18
AOIA0.601 321.000 10.761 330.687 270.485 340.828 140.008 440.663 240.405 290.405 370.425 370.490 210.596 280.714 410.553 290.779 160.597 270.992 360.424 35
PCJC0.578 331.000 10.810 160.583 390.449 380.813 200.042 390.603 330.341 350.490 200.465 310.410 310.650 130.835 380.264 470.694 370.561 350.889 450.504 24
SSEN0.575 341.000 10.761 320.473 420.477 360.795 250.066 350.529 380.658 100.460 240.461 320.380 340.331 470.859 290.401 400.692 390.653 201.000 10.348 40
Dongsu Zhang, Junha Chun, Sang Kyun Cha, Young Min Kim: Spatial Semantic Embedding Network: Fast 3D Instance Segmentation with Deep Metric Learning. Arxiv
RWSeg0.567 350.528 500.708 430.626 310.580 200.745 340.063 360.627 290.240 400.400 390.497 280.464 230.515 341.000 10.475 340.745 250.571 321.000 10.429 34
NeuralBF0.555 360.667 400.896 30.843 90.517 300.751 320.029 400.519 390.414 280.439 280.465 300.000 560.484 360.857 300.287 450.693 380.651 221.000 10.485 28
Weiwei Sun, Daniel Rebain, Renjie Liao, Vladimir Tankovich, Soroosh Yazdani, Kwang Moo Yi, Andrea Tagliasacchi: NeuralBF: Neural Bilateral Filtering for Top-down Instance Segmentation on Point Clouds. WACV 2023
MTML0.549 371.000 10.807 180.588 380.327 430.647 390.004 460.815 70.180 420.418 340.364 430.182 410.445 391.000 10.442 360.688 400.571 331.000 10.396 36
Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald: 3D Instance Segmentation via Multi-task Metric Learning. ICCV 2019 [oral]
One_Thing_One_Clickpermissive0.529 380.667 400.718 390.777 170.399 390.683 360.000 490.669 220.138 450.391 400.374 420.539 180.360 460.641 450.556 280.774 180.593 280.997 300.251 45
Zhengzhe Liu, Xiaojuan Qi, Chi-Wing Fu: One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation. CVPR 2021
Sparse R-CNN0.515 391.000 10.538 510.282 450.468 370.790 260.173 210.345 470.429 250.413 360.484 290.176 420.595 300.591 460.522 310.668 410.476 440.986 390.327 41
Occipital-SCS0.512 401.000 10.716 400.509 410.506 320.611 420.092 330.602 340.177 430.346 430.383 400.165 430.442 400.850 370.386 420.618 450.543 400.889 450.389 37
3D-BoNet0.488 411.000 10.672 460.590 370.301 450.484 520.098 320.620 310.306 370.341 440.259 470.125 450.434 420.796 400.402 390.499 510.513 420.909 440.439 32
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight
PanopticFusion-inst0.478 420.667 400.712 420.595 360.259 480.550 480.000 490.613 320.175 440.250 490.434 330.437 270.411 440.857 300.485 330.591 480.267 540.944 410.359 39
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
SPG_WSIS0.470 430.667 400.685 450.677 290.372 410.562 460.000 490.482 420.244 390.316 460.298 440.052 510.442 410.857 300.267 460.702 340.559 361.000 10.287 43
SALoss-ResNet0.459 441.000 10.737 380.159 550.259 470.587 440.138 280.475 430.217 410.416 350.408 380.128 440.315 480.714 410.411 380.536 500.590 290.873 480.304 42
Zhidong Liang, Ming Yang, Hao Li, Chunxiang Wang: 3D Instance Embedding Learning With a Structure-Aware Loss Function for Point Cloud Segmentation. IEEE Robotics and Automation Letters (IROS2020)
MASCpermissive0.447 450.528 500.555 490.381 430.382 400.633 400.002 470.509 400.260 380.361 420.432 350.327 370.451 380.571 470.367 430.639 430.386 450.980 400.276 44
Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.
SegGroup_inspermissive0.445 460.667 400.773 290.185 520.317 440.656 380.000 490.407 460.134 460.381 410.267 460.217 400.476 370.714 410.452 350.629 440.514 411.000 10.222 48
An Tao, Yueqi Duan, Yi Wei, Jiwen Lu, Jie Zhou: SegGroup: Seg-Level Supervision for 3D Instance and Semantic Segmentation. TIP 2022
3D-SISpermissive0.382 471.000 10.432 530.245 470.190 490.577 450.013 430.263 490.033 520.320 450.240 480.075 470.422 430.857 300.117 510.699 350.271 530.883 470.235 47
Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019
Hier3Dcopyleft0.323 480.667 400.542 500.264 460.157 520.550 470.000 490.205 520.009 530.270 480.218 490.075 470.500 350.688 440.007 570.698 360.301 500.459 540.200 49
Tan: HCFS3D: Hierarchical Coupled Feature Selection Network for 3D Semantic and Instance Segmentation.
UNet-backbone0.319 490.667 400.715 410.233 480.189 500.479 530.008 440.218 500.067 510.201 510.173 500.107 460.123 530.438 490.150 490.615 460.355 460.916 430.093 56
R-PointNet0.306 500.500 520.405 540.311 440.348 420.589 430.054 370.068 550.126 470.283 470.290 450.028 520.219 510.214 520.331 440.396 550.275 510.821 500.245 46
Region-18class0.284 510.250 560.751 370.228 500.270 460.521 490.000 490.468 440.008 550.205 500.127 510.000 560.068 550.070 550.262 480.652 420.323 480.740 510.173 50
SemRegionNet-20cls0.250 520.333 530.613 470.229 490.163 510.493 500.000 490.304 480.107 480.147 530.100 520.052 500.231 490.119 530.039 530.445 530.325 470.654 520.141 52
tmp0.248 530.667 400.437 520.188 510.153 530.491 510.000 490.208 510.094 500.153 520.099 530.057 490.217 520.119 530.039 530.466 520.302 490.640 530.140 53
3D-BEVIS0.248 530.667 400.566 480.076 560.035 570.394 550.027 420.035 560.098 490.099 550.030 560.025 530.098 540.375 510.126 500.604 470.181 550.854 490.171 51
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.
ASIS0.199 550.333 530.253 560.167 540.140 540.438 540.000 490.177 530.008 540.121 540.069 540.004 550.231 500.429 500.036 550.445 540.273 520.333 560.119 55
Sgpn_scannet0.143 560.208 570.390 550.169 530.065 550.275 560.029 410.069 540.000 560.087 560.043 550.014 540.027 570.000 560.112 520.351 560.168 560.438 550.138 54
MaskRCNN 2d->3d Proj0.058 570.333 530.002 570.000 570.053 560.002 570.002 480.021 570.000 560.045 570.024 570.238 390.065 560.000 560.014 560.107 570.020 570.110 570.006 57