The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.



This table lists the benchmark results for the 3D semantic instance scenario.




Method Infoavg apbathtubbedbookshelfcabinetchaircountercurtaindeskdoorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
Spherical Mask(CtoF)0.616 10.946 30.654 90.555 40.434 70.769 30.271 60.604 70.447 30.505 40.549 10.698 10.716 10.775 100.480 50.747 30.575 60.925 70.436 3
ExtMask3D0.598 20.852 120.692 40.433 220.461 40.791 10.264 70.488 290.493 10.508 30.528 90.594 60.706 30.791 50.483 30.734 60.595 20.911 110.437 2
MAFT0.596 30.889 90.721 10.448 160.460 50.768 40.251 80.558 160.408 40.504 50.539 50.616 40.618 70.858 20.482 40.684 130.551 100.931 60.450 1
UniPerception0.588 40.963 20.667 70.493 90.472 30.750 70.229 110.528 220.468 20.498 70.542 30.643 20.530 160.661 300.463 100.695 120.599 10.972 10.420 5
Queryformer0.583 50.926 50.702 20.393 280.504 10.733 130.276 50.527 230.373 100.479 80.534 70.533 150.697 40.720 220.436 140.745 40.592 30.958 30.363 14
SIM3D0.575 60.889 90.675 60.284 440.401 120.762 60.329 20.531 210.408 50.521 20.541 40.587 70.646 50.744 180.467 80.665 150.579 50.886 210.425 4
PBNetpermissive0.573 70.926 50.575 180.619 10.472 20.736 110.239 100.487 300.383 90.459 110.506 120.533 140.585 90.767 110.404 160.717 70.559 90.969 20.381 10
W.Zhao, Y.Yan, C.Yang, J.Ye,X.Yang,K.Huang: Divide and Conquer: 3D Instance Segmentation With Point-Wise Binarization. ICCV 2023
TST3D0.569 80.778 190.675 50.598 20.451 60.727 140.280 40.476 320.395 60.472 90.457 200.583 80.580 110.777 70.462 120.735 50.547 120.919 90.333 20
Mask3D0.566 90.926 50.597 130.408 250.420 100.737 100.239 90.598 90.386 80.458 120.549 10.568 120.716 10.601 360.480 50.646 180.575 60.922 80.364 13
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, Bastian Leibe: Mask3D for 3D Semantic Instance Segmentation. ICRA 2023
OneFormer3Dcopyleft0.566 90.781 180.697 30.562 30.431 80.770 20.331 10.400 380.373 110.529 10.504 130.568 110.475 210.732 200.470 70.762 10.550 110.871 270.379 11
Maxim Kolodiazhnyi, Anna Vorontsova, Anton Konushin, Danila Rukhovich: OneFormer3D: One Transformer for Unified Point Cloud Segmentation.
ISBNetpermissive0.559 110.939 40.655 80.383 310.426 90.763 50.180 130.534 200.386 70.499 60.509 110.621 30.427 310.704 250.467 90.649 170.571 80.948 40.401 6
Tuan Duc Ngo, Binh-Son Hua, Khoi Nguyen: ISBNet: a 3D Point Cloud Instance Segmentation Network with Instance-aware Sampling and Box-aware Dynamic Convolution. CVPR 2023
GraphCut0.552 121.000 10.611 120.438 190.392 140.714 150.139 160.598 100.327 140.389 150.510 100.598 50.427 320.754 140.463 110.761 20.588 40.903 140.329 21
SPFormerpermissive0.549 130.745 220.640 100.484 100.395 130.739 90.311 30.566 140.335 130.468 100.492 140.555 130.478 200.747 160.436 130.712 80.540 130.893 180.343 19
Sun Jiahao, Qing Chunmei, Tan Junpeng, Xu Xiangmin: Superpoint Transformer for 3D Scene Instance Segmentation. AAAI 2023 [Oral]
DKNet0.532 140.815 150.624 110.517 60.377 160.749 80.107 180.509 260.304 160.437 130.475 150.581 90.539 140.775 90.339 210.640 200.506 160.901 150.385 9
Yizheng Wu, Min Shi, Shuaiyuan Du, Hao Lu, Zhiguo Cao, Weicai Zhong: 3D Instances as 1D Kernels. ECCV 2022
IPCA-Inst0.520 150.889 90.551 220.548 50.418 110.665 250.064 270.585 110.260 240.277 290.471 170.500 160.644 60.785 60.369 170.591 260.511 140.878 240.362 15
SoftGroup++0.513 160.704 280.578 170.398 270.363 220.704 160.061 280.647 40.297 210.378 180.537 60.343 190.614 80.828 40.295 260.710 100.505 180.875 260.394 7
SSTNetpermissive0.506 170.738 250.549 230.497 80.316 270.693 190.178 140.377 410.198 300.330 200.463 190.576 100.515 170.857 30.494 10.637 210.457 220.943 50.290 30
Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, Kui Jia: Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networks. ICCV2021
SoftGrouppermissive0.504 180.667 350.579 150.372 330.381 150.694 180.072 240.677 20.303 170.387 160.531 80.319 230.582 100.754 130.318 220.643 190.492 190.907 130.388 8
Thang Vu, Kookhoi Kim, Tung M. Luu, Xuan Thanh Nguyen, Chang D. Yoo: SoftGroup for 3D Instance Segmentaiton on Point Clouds. CVPR 2022 [Oral]
DANCENET0.504 180.926 50.579 140.472 120.367 190.626 350.165 150.432 330.221 260.408 140.449 220.411 170.564 120.746 170.421 150.707 110.438 250.846 350.288 31
TD3Dpermissive0.489 200.852 120.511 320.434 200.322 260.735 120.101 210.512 250.355 120.349 190.468 180.283 270.514 180.676 290.268 310.671 140.510 150.908 120.329 22
Maksim Kolodiazhnyi, Anna Vorontsova, Anton Konushin, Danila Rukhovich: Top-Down Beats Bottom-Up in 3D Instance Segmentation. WACV 2024
OccuSeg+instance0.486 210.802 170.536 250.428 230.369 180.702 170.205 120.331 460.301 180.379 170.474 160.327 200.437 260.862 10.485 20.601 240.394 330.846 370.273 34
Lei Han, Tian Zheng, Lan Xu, Lu Fang: OccuSeg: Occupancy-aware 3D Instance Segmentation. CVPR2020
TopoSeg0.479 220.704 280.564 190.467 140.366 200.633 330.068 250.554 170.262 230.328 210.447 230.323 210.534 150.722 210.288 280.614 220.482 200.912 100.358 17
DualGroup0.469 230.815 150.552 210.398 260.374 170.683 210.130 170.539 190.310 150.327 220.407 260.276 280.447 250.535 400.342 200.659 160.455 230.900 170.301 26
SSEC0.465 240.667 350.578 160.502 70.362 230.641 320.035 370.605 60.291 220.323 230.451 210.296 250.417 350.677 280.245 350.501 440.506 170.900 160.366 12
HAISpermissive0.457 250.704 280.561 200.457 150.364 210.673 220.046 360.547 180.194 310.308 240.426 240.288 260.454 240.711 230.262 320.563 340.434 270.889 200.344 18
Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, Xinggang Wang: Hierarchical Aggregation for 3D Instance Segmentation. ICCV 2021
DD-UNet+Group0.436 260.630 430.508 350.480 110.310 290.624 370.065 260.638 50.174 320.256 330.384 300.194 400.428 290.759 120.289 270.574 310.400 310.849 340.291 29
H. Liu, R. Liu, K. Yang, J. Zhang, K. Peng, R. Stiefelhagen: HIDA: Towards Holistic Indoor Understanding for the Visually Impaired via Semantic Instance Segmentation with a Wearable Solid-State LiDAR Sensor. ICCVW 2021
INS-Conv-instance0.435 270.716 270.495 370.355 350.331 240.689 200.102 200.394 400.208 290.280 270.395 280.250 310.544 130.741 190.309 240.536 400.391 340.842 400.258 38
Mask-Group0.434 280.778 190.516 300.471 130.330 250.658 260.029 390.526 240.249 250.256 320.400 270.309 240.384 390.296 560.368 180.575 300.425 280.877 250.362 16
Min Zhong, Xinghao Chen, Xiaokang Chen, Gang Zeng, Yunhe Wang: MaskGroup: Hierarchical Point Grouping and Masking for 3D Instance Segmentation. ICME 2022
Box2Mask0.433 290.741 230.463 420.433 210.283 320.625 360.103 190.298 510.125 410.260 310.424 250.322 220.472 220.701 260.363 190.711 90.309 500.882 220.272 36
Julian Chibane, Francis Engelmann, Tuan Anh Tran, Gerard Pons-Moll: Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation Using Bounding Boxes. ECCV 2022
RPGN0.428 300.630 430.508 340.367 340.249 390.658 270.016 470.673 30.131 390.234 360.383 310.270 290.434 270.748 150.274 300.609 230.406 300.842 390.267 37
Shichao Dong, Guosheng Lin, Tzu-Yi Hung: Learning Regional Purity for Instance Segmentation on 3D Point Clouds. ECCV 2022
DENet0.413 310.741 230.520 270.237 470.284 310.523 460.097 220.691 10.138 360.209 460.229 480.238 340.390 370.707 240.310 230.448 510.470 210.892 190.310 24
PointGroup0.407 320.639 420.496 360.415 240.243 410.645 310.021 440.570 130.114 420.211 440.359 330.217 380.428 300.660 310.256 330.562 350.341 420.860 300.291 28
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, Jiaya Jia: PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. CVPR 2020 [oral]
CSC-Pretrained0.405 330.738 250.465 410.331 390.205 450.655 280.051 320.601 80.092 460.211 450.329 360.198 390.459 230.775 80.195 420.524 420.400 320.878 230.184 47
PE0.396 340.667 350.467 400.446 180.243 400.624 380.022 430.577 120.106 430.219 390.340 340.239 330.487 190.475 470.225 370.541 390.350 400.818 420.273 35
Biao Zhang, Peter Wonka: Point Cloud Instance Segmentation using Probabilistic Embeddings. CVPR 2021
Dyco3Dcopyleft0.395 350.642 410.518 290.447 170.259 380.666 240.050 330.251 560.166 330.231 370.362 320.232 350.331 420.535 390.229 360.587 270.438 260.850 320.317 23
Tong He; Chunhua Shen; Anton van den Hengel: DyCo3D: Robust Instance Segmentation of 3D Point Clouds through Dynamic Convolution. CVPR2021
OSIS0.392 360.778 190.530 260.220 490.278 330.567 430.083 230.330 470.299 190.270 300.310 390.143 460.260 460.624 340.277 290.568 330.361 380.865 290.301 25
AOIA0.387 370.704 280.515 310.385 300.225 440.669 230.005 540.482 310.126 400.181 490.269 450.221 370.426 330.478 460.218 380.592 250.371 360.851 310.242 40
SSEN0.384 380.852 120.494 380.192 500.226 430.648 300.022 420.398 390.299 200.277 280.317 380.231 360.194 530.514 430.196 400.586 280.444 240.843 380.184 46
Dongsu Zhang, Junha Chun, Sang Kyun Cha, Young Min Kim: Spatial Semantic Embedding Network: Fast 3D Instance Segmentation with Deep Metric Learning. Arxiv
Mask3D_evaluation0.382 390.593 450.520 280.390 290.314 280.600 390.018 460.287 540.151 350.281 260.387 290.169 440.429 280.654 320.172 460.578 290.384 350.670 530.278 33
PCJC0.375 400.704 280.542 240.284 430.197 470.649 290.006 510.426 340.138 370.242 340.304 400.183 430.388 380.629 330.141 530.546 380.344 410.738 480.283 32
ClickSeg_Instance0.366 410.654 390.375 460.184 510.302 300.592 410.050 340.300 500.093 450.283 250.277 420.249 320.426 340.615 350.299 250.504 430.367 370.832 410.191 45
SphereSeg0.357 420.651 400.411 440.345 360.264 370.630 340.059 290.289 530.212 270.240 350.336 350.158 450.305 430.557 370.159 490.455 500.341 430.726 500.294 27
3D-MPA0.355 430.457 550.484 390.299 410.277 340.591 420.047 350.332 440.212 280.217 400.278 410.193 410.413 360.410 500.195 410.574 320.352 390.849 330.213 43
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nießner: 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation. CVPR 2020
NeuralBF0.353 440.593 450.511 330.375 320.264 360.597 400.008 490.332 450.160 340.229 380.274 440.000 670.206 500.678 270.155 500.485 460.422 290.816 430.254 39
Weiwei Sun, Daniel Rebain, Renjie Liao, Vladimir Tankovich, Soroosh Yazdani, Kwang Moo Yi, Andrea Tagliasacchi: NeuralBF: Neural Bilateral Filtering for Top-down Instance Segmentation on Point Clouds. WACV 2023
RWSeg0.348 450.475 520.456 430.320 400.275 350.476 480.020 450.491 280.056 530.212 430.320 370.261 300.302 440.520 410.182 440.557 360.285 520.867 280.197 44
GICN0.341 460.580 470.371 470.344 370.198 460.469 490.052 310.564 150.093 440.212 420.212 500.127 480.347 410.537 380.206 390.525 410.329 450.729 490.241 41
One_Thing_One_Clickpermissive0.326 470.472 530.361 480.232 480.183 480.555 440.000 600.498 270.038 550.195 470.226 490.362 180.168 540.469 480.251 340.553 370.335 440.846 360.117 55
Zhengzhe Liu, Xiaojuan Qi, Chi-Wing Fu: One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation. CVPR 2021
Occipital-SCS0.320 480.679 340.352 490.334 380.229 420.436 500.025 400.412 370.058 510.161 540.240 470.085 500.262 450.496 450.187 430.467 480.328 460.775 440.231 42
Sparse R-CNN0.292 490.704 280.213 590.153 530.154 500.551 450.053 300.212 570.132 380.174 510.274 430.070 520.363 400.441 490.176 450.424 530.234 540.758 460.161 51
MTML0.282 500.577 480.380 450.182 520.107 560.430 510.001 570.422 350.057 520.179 500.162 530.070 530.229 480.511 440.161 470.491 450.313 470.650 560.162 49
Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald: 3D Instance Segmentation via Multi-task Metric Learning. ICCV 2019 [oral]
SALoss-ResNet0.262 510.667 350.335 500.067 600.123 540.427 520.022 410.280 550.058 500.216 410.211 510.039 560.142 560.519 420.106 570.338 570.310 490.721 510.138 52
Zhidong Liang, Ming Yang, Hao Li, Chunxiang Wang: 3D Instance Embedding Learning With a Structure-Aware Loss Function for Point Cloud Segmentation. IEEE Robotics and Automation Letters (IROS2020)
MASCpermissive0.254 520.463 540.249 580.113 540.167 490.412 540.000 590.374 420.073 470.173 520.243 460.130 470.228 490.368 520.160 480.356 550.208 550.711 520.136 53
Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.
3D-BoNet0.253 530.519 500.324 530.251 460.137 530.345 590.031 380.419 360.069 480.162 530.131 550.052 540.202 520.338 540.147 520.301 600.303 510.651 550.178 48
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight
SPG_WSIS0.251 540.380 570.274 560.289 420.144 510.413 530.000 600.311 480.065 490.113 560.130 560.029 590.204 510.388 510.108 560.459 490.311 480.769 450.127 54
SegGroup_inspermissive0.246 550.556 490.335 510.062 620.115 550.490 470.000 600.297 520.018 590.186 480.142 540.083 510.233 470.216 580.153 510.469 470.251 530.744 470.083 58
An Tao, Yueqi Duan, Yi Wei, Jiwen Lu, Jie Zhou: SegGroup: Seg-Level Supervision for 3D Instance and Semantic Segmentation. TIP 2022
PanopticFusion-inst0.214 560.250 620.330 520.275 450.103 570.228 650.000 600.345 430.024 570.088 580.203 520.186 420.167 550.367 530.125 540.221 630.112 650.666 540.162 50
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
UNet-backbone0.161 570.519 500.259 570.084 560.059 590.325 610.002 550.093 620.009 610.077 600.064 590.045 550.044 630.161 600.045 590.331 580.180 570.566 570.033 67
3D-SISpermissive0.161 570.407 560.155 640.068 590.043 630.346 580.001 560.134 590.005 620.088 570.106 580.037 570.135 580.321 550.028 630.339 560.116 640.466 600.093 57
Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019
R-PointNet0.158 590.356 580.173 620.113 550.140 520.359 550.012 480.023 650.039 540.134 550.123 570.008 630.089 590.149 610.117 550.221 620.128 620.563 580.094 56
Region-18class0.146 600.175 660.321 540.080 570.062 580.357 560.000 600.307 490.002 640.066 610.044 610.000 670.018 650.036 660.054 580.447 520.133 600.472 590.060 62
SemRegionNet-20cls0.121 610.296 600.203 600.071 580.058 600.349 570.000 600.150 580.019 580.054 630.034 640.017 620.052 610.042 650.013 660.209 640.183 560.371 610.057 63
3D-BEVIS0.117 620.250 620.308 550.020 660.009 680.269 640.006 520.008 660.029 560.037 660.014 670.003 650.036 640.147 620.042 610.381 540.118 630.362 620.069 61
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.
Hier3Dcopyleft0.117 620.222 640.161 630.054 640.027 650.289 620.000 600.124 600.001 660.079 590.061 600.027 600.141 570.240 570.005 670.310 590.129 610.153 670.081 59
Tan: HCFS3D: Hierarchical Coupled Feature Selection Network for 3D Semantic and Instance Segmentation.
tmp0.113 640.333 590.151 650.056 630.053 610.344 600.000 600.105 610.016 600.049 640.035 630.020 610.053 600.048 640.013 650.183 660.173 580.344 640.054 64
Sem_Recon_ins0.098 650.295 610.187 610.015 670.036 640.213 660.005 530.038 640.003 630.056 620.037 620.036 580.015 660.051 630.044 600.209 650.098 660.354 630.071 60
ASIS0.085 660.037 670.080 670.066 610.047 620.282 630.000 600.052 630.002 650.047 650.026 650.001 660.046 620.194 590.031 620.264 610.140 590.167 660.047 66
Sgpn_scannet0.049 670.023 680.134 660.031 650.013 670.144 670.006 500.008 670.000 670.028 670.017 660.003 640.009 680.000 670.021 640.122 670.095 670.175 650.054 65
MaskRCNN 2d->3d Proj0.022 680.185 650.000 680.000 680.015 660.000 680.000 580.006 680.000 670.010 680.006 680.107 490.012 670.000 670.002 680.027 680.004 680.022 680.001 68