The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).



This table lists the benchmark results for the 3D semantic label scenario.


Method Infoavg ioubathtubbedbookshelfcabinetchaircountercurtaindeskdoorfloorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwallwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
OccuSeg+Semantic0.764 10.758 310.796 70.839 40.746 20.907 10.562 10.850 50.680 20.672 10.978 10.610 10.335 30.777 10.819 160.847 10.830 10.691 30.972 10.885 10.727 2
BPNetcopyleft0.749 20.909 10.818 40.811 80.752 10.839 60.485 140.842 70.673 30.644 40.957 30.528 80.305 90.773 20.859 40.788 20.818 30.693 20.916 60.856 50.723 4
Wenbo Hu, Hengshuang Zhao, Li Jiang, Jiaya Jia, Tien-Tsin Wong: Bidirectional Projection Network for Cross Dimension Scene Understanding. CVPR 2021 (Oral)
Virtual MVFusion0.746 30.771 260.819 30.848 20.702 80.865 30.397 450.899 10.699 10.664 20.948 200.588 20.330 40.746 50.851 80.764 30.796 70.704 10.935 40.866 20.728 1
Abhijit Kundu, Xiaoqi Yin, Alireza Fathi, David Ross, Brian Brewington, Thomas Funkhouser, Caroline Pantofaru: Virtual Multi-view Fusion for 3D Semantic Segmentation. ECCV 2020
VMNetpermissive0.746 30.870 60.838 10.858 10.729 40.850 40.501 80.874 20.587 210.658 30.956 40.564 40.299 100.765 30.900 10.716 80.812 40.631 110.939 20.858 40.709 5
Zeyu HU, Xuyang Bai, Jiaxiang Shang, Runze Zhang, Jiayu Dong, Xin Wang, Guangyuan Sun, Hongbo Fu, Chiew-Lan Tai: VMNet: Voxel-Mesh Network for Geodesic-Aware 3D Semantic Segmentation. ICCV 2021 (Oral)
MinkowskiNetpermissive0.736 50.859 80.818 40.832 50.709 70.840 50.521 50.853 40.660 40.643 50.951 110.544 50.286 150.731 60.893 20.675 190.772 140.683 40.874 310.852 60.727 2
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet0.725 60.647 500.821 20.846 30.721 50.869 20.533 30.754 230.603 180.614 60.955 50.572 30.325 50.710 70.870 30.724 70.823 20.628 120.934 50.865 30.683 8
MatchingNet0.724 70.812 180.812 60.810 90.735 30.834 80.495 110.860 30.572 270.602 120.954 60.512 100.280 160.757 40.845 110.725 60.780 110.606 200.937 30.851 70.700 7
RFCR0.702 80.889 30.745 230.813 70.672 120.818 190.493 120.815 100.623 100.610 80.947 240.470 200.249 280.594 200.848 90.705 130.779 120.646 70.892 180.823 150.611 24
Jingyu Gong, Jiachen Xu, Xin Tan, Haichuan Song, Yanyun Qu, Yuan Xie, Lizhuang Ma: Omni-Supervised Point Cloud Segmentation via Gradual Receptive Field Component Reasoning. CVPR2021
One Thing One Click0.701 90.825 160.796 70.723 290.716 60.832 90.433 350.816 80.634 80.609 90.969 20.418 430.344 20.559 310.833 120.715 90.808 50.560 340.902 100.847 80.680 9
JSENetpermissive0.699 100.881 50.762 150.821 60.667 130.800 310.522 40.792 150.613 110.607 100.935 460.492 140.205 390.576 260.853 60.691 140.758 190.652 60.872 340.828 120.649 15
Zeyu HU, Mingmin Zhen, Xuyang BAI, Hongbo Fu, Chiew-lan Tai: JSENet: Joint Semantic Segmentation and Edge Detection Network for 3D Point Clouds. ECCV 2020
Feature-Geometry Netpermissive0.694 110.894 20.741 260.768 210.677 100.827 130.491 130.811 110.612 120.612 70.948 200.464 230.250 260.554 320.828 130.708 120.781 100.614 150.884 220.822 170.593 35
Kangcheng Liu, Ben M. Chen: https://arxiv.org/abs/2012.09439. arXiv Preprint
CU-Hybrid Net0.693 120.596 560.789 90.803 110.677 100.800 310.469 180.846 60.554 350.591 150.948 200.500 120.316 70.609 170.847 100.732 40.808 50.593 250.894 160.839 90.652 14
Feature_GeometricNetpermissive0.690 130.884 40.754 200.795 140.647 180.818 190.422 360.802 140.612 120.604 110.945 290.462 240.189 450.563 300.853 60.726 50.765 160.632 100.904 80.821 180.606 28
Kangcheng Liu, Ben M. Chen: https://arxiv.org/abs/2012.09439. arXiv Preprint
FusionNet0.688 140.704 430.741 260.754 260.656 140.829 110.501 80.741 280.609 150.548 220.950 150.522 90.371 10.633 150.756 220.715 90.771 150.623 130.861 390.814 200.658 12
Feihu Zhang, Jin Fang, Benjamin Wah, Philip Torr: Deep FusionNet for Point Cloud Semantic Segmentation. ECCV 2020
VACNN++0.684 150.728 390.757 190.776 180.690 90.804 290.464 220.816 80.577 250.587 160.945 290.508 110.276 180.671 80.710 300.663 240.750 220.589 260.881 260.832 110.653 13
KP-FCNN0.684 150.847 110.758 180.784 160.647 180.814 220.473 160.772 180.605 160.594 140.935 460.450 310.181 480.587 210.805 180.690 150.785 90.614 150.882 240.819 190.632 19
H. Thomas, C. Qi, J. Deschaud, B. Marcotegui, F. Goulette, L. Guibas.: KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV 2019
Superpoint Network0.683 170.851 100.728 330.800 130.653 160.806 280.468 190.804 120.572 270.602 120.946 260.453 300.239 310.519 400.822 140.689 170.762 180.595 240.895 150.827 130.630 20
PointContrast_LA_SEM0.683 170.757 320.784 100.786 150.639 210.824 160.408 400.775 170.604 170.541 240.934 500.532 60.269 210.552 330.777 200.645 320.793 80.640 80.913 70.824 140.671 10
VI-PointConv0.676 190.770 280.754 200.783 170.621 240.814 220.552 20.758 210.571 290.557 200.954 60.529 70.268 230.530 380.682 350.675 190.719 280.603 210.888 200.833 100.665 11
Xingyi Li, Wenxuan Wu, Xiaoli Z. Fern, Li Fuxin: The Devils in the Point Clouds: Studying the Robustness of Point Cloud Convolutions.
ROSMRF3D0.673 200.789 190.748 220.763 240.635 220.814 220.407 420.747 250.581 240.573 170.950 150.484 150.271 200.607 180.754 230.649 280.774 130.596 230.883 230.823 150.606 28
SALANet0.670 210.816 170.770 140.768 210.652 170.807 270.451 240.747 250.659 50.545 230.924 530.473 190.149 590.571 280.811 170.635 350.746 230.623 130.892 180.794 310.570 41
PointConvpermissive0.666 220.781 210.759 170.699 340.644 200.822 170.475 150.779 160.564 320.504 360.953 80.428 380.203 410.586 230.754 230.661 250.753 200.588 270.902 100.813 220.642 16
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
PointASNLpermissive0.666 220.703 440.781 110.751 280.655 150.830 100.471 170.769 190.474 500.537 250.951 110.475 180.279 170.635 130.698 340.675 190.751 210.553 390.816 490.806 230.703 6
Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, Shuguang Cui: PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Sampling. CVPR 2020
DCM-Net0.658 240.778 220.702 380.806 100.619 250.813 250.468 190.693 400.494 460.524 300.941 380.449 320.298 110.510 420.821 150.675 190.727 270.568 310.826 460.803 250.637 18
Jonas Schult*, Francis Engelmann*, Theodora Kontogianni, Bastian Leibe: DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes. CVPR 2020 [Oral]
HPGCNN0.656 250.698 450.743 250.650 480.564 400.820 180.505 70.758 210.631 90.479 420.945 290.480 160.226 320.572 270.774 210.690 150.735 260.614 150.853 420.776 430.597 33
Jisheng Dang, Qingyong Hu, Yulan Guo, Jun Yang: HPGCNN.
SAFNet-segpermissive0.654 260.752 330.734 290.664 460.583 340.815 210.399 440.754 230.639 70.535 260.942 360.470 200.309 80.665 90.539 440.650 270.708 330.635 90.857 410.793 320.642 16
Linqing Zhao, Jiwen Lu, Jie Zhou: Similarity-Aware Fusion Network for 3D Semantic Segmentation. IROS 2021
RandLA-Netpermissive0.645 270.778 220.731 300.699 340.577 350.829 110.446 280.736 290.477 490.523 320.945 290.454 280.269 210.484 490.749 260.618 390.738 240.599 220.827 450.792 340.621 22
MVPNetpermissive0.641 280.831 130.715 340.671 430.590 310.781 410.394 460.679 430.642 60.553 210.937 430.462 240.256 240.649 100.406 560.626 360.691 400.666 50.877 270.792 340.608 27
Maximilian Jaritz, Jiayuan Gu, Hao Su: Multi-view PointNet for 3D Scene Understanding. GMDL Workshop, ICCV 2019
PointConv-SFPN0.641 280.776 240.703 370.721 300.557 430.826 140.451 240.672 450.563 330.483 410.943 350.425 410.162 540.644 110.726 270.659 260.709 320.572 290.875 290.786 380.559 46
PointMRNet0.640 300.717 420.701 390.692 360.576 360.801 300.467 210.716 340.563 330.459 460.953 80.429 370.169 510.581 240.854 50.605 400.710 310.550 400.894 160.793 320.575 39
FPConvpermissive0.639 310.785 200.760 160.713 330.603 280.798 340.392 470.534 580.603 180.524 300.948 200.457 260.250 260.538 360.723 280.598 430.696 390.614 150.872 340.799 260.567 43
Yiqun Lin, Zizheng Yan, Haibin Huang, Dong Du, Ligang Liu, Shuguang Cui, Xiaoguang Han: FPConv: Learning Local Flattening for Point Convolution. CVPR 2020
PointSPNet0.637 320.734 360.692 440.714 320.576 360.797 350.446 280.743 270.598 200.437 500.942 360.403 450.150 580.626 160.800 190.649 280.697 380.557 370.846 430.777 420.563 44
SConv0.636 330.830 140.697 420.752 270.572 390.780 420.445 300.716 340.529 390.530 270.951 110.446 330.170 500.507 440.666 370.636 340.682 420.541 450.886 210.799 260.594 34
Supervoxel-CNN0.635 340.656 480.711 350.719 310.613 260.757 480.444 320.765 200.534 380.566 180.928 510.478 170.272 190.636 120.531 460.664 230.645 500.508 490.864 380.792 340.611 24
joint point-basedpermissive0.634 350.614 530.778 120.667 450.633 230.825 150.420 370.804 120.467 520.561 190.951 110.494 130.291 120.566 290.458 500.579 480.764 170.559 360.838 440.814 200.598 32
Hung-Yueh Chiang, Yen-Liang Lin, Yueh-Cheng Liu, Winston H. Hsu: A Unified Point-Based Framework for 3D Segmentation. 3DV 2019
MCCNNpermissive0.633 360.866 70.731 300.771 190.576 360.809 260.410 390.684 410.497 450.491 380.949 170.466 220.105 640.581 240.646 380.620 370.680 430.542 440.817 480.795 290.618 23
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
PointMTL0.632 370.731 370.688 470.675 400.591 300.784 400.444 320.565 560.610 140.492 370.949 170.456 270.254 250.587 210.706 310.599 420.665 470.612 190.868 370.791 370.579 38
APCF-Net0.631 380.742 340.687 490.672 410.557 430.792 370.408 400.665 460.545 360.508 340.952 100.428 380.186 460.634 140.702 320.620 370.706 340.555 380.873 330.798 280.581 37
Haojia, Lin: Adaptive Pyramid Context Fusion for Point Cloud Perception. GRSL
PointNet2-SFPN0.631 380.771 260.692 440.672 410.524 460.837 70.440 340.706 380.538 370.446 480.944 330.421 420.219 350.552 330.751 250.591 450.737 250.543 430.901 120.768 460.557 47
3DSM_DMMF0.631 380.626 520.745 230.801 120.607 270.751 490.506 60.729 320.565 310.491 380.866 640.434 340.197 430.595 190.630 390.709 110.705 350.560 340.875 290.740 530.491 56
FusionAwareConv0.630 410.604 550.741 260.766 230.590 310.747 500.501 80.734 300.503 440.527 280.919 570.454 280.323 60.550 350.420 550.678 180.688 410.544 420.896 140.795 290.627 21
Jiazhao Zhang, Chenyang Zhu, Lintao Zheng, Kai Xu: Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentation. CVPR 2020
SIConv0.625 420.830 140.694 430.757 250.563 410.772 440.448 260.647 490.520 410.509 330.949 170.431 360.191 440.496 470.614 400.647 310.672 450.535 470.876 280.783 390.571 40
HPEIN0.618 430.729 380.668 500.647 500.597 290.766 450.414 380.680 420.520 410.525 290.946 260.432 350.215 360.493 480.599 410.638 330.617 550.570 300.897 130.806 230.605 30
Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, Jiaya Jia: Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation. ICCV 2019
SPH3D-GCNpermissive0.610 440.858 90.772 130.489 630.532 450.792 370.404 430.643 500.570 300.507 350.935 460.414 440.046 690.510 420.702 320.602 410.705 350.549 410.859 400.773 440.534 49
Huan Lei, Naveed Akhtar, and Ajmal Mian: Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds. TPAMI 2020
AttAN0.609 450.760 300.667 510.649 490.521 470.793 360.457 230.648 480.528 400.434 520.947 240.401 460.153 570.454 500.721 290.648 300.717 290.536 460.904 80.765 470.485 57
Gege Zhang, Qinghua Ma, Licheng Jiao, Fang Liu and Qigong Sun: AttAN: Attention Adversarial Networks for 3D Point Cloud Semantic Segmentation. IJCAI2020
LAP-D0.594 460.720 400.692 440.637 520.456 540.773 430.391 490.730 310.587 210.445 490.940 400.381 490.288 130.434 530.453 510.591 450.649 480.581 280.777 530.749 520.610 26
DPC0.592 470.720 400.700 400.602 550.480 510.762 470.380 520.713 360.585 230.437 500.940 400.369 510.288 130.434 530.509 480.590 470.639 530.567 320.772 540.755 500.592 36
Francis Engelmann, Theodora Kontogianni, Bastian Leibe: Dilated Point Convolutions: On the Receptive Field Size of Point Convolutions on 3D Point Clouds. ICRA 2020
SegGCNpermissive0.589 480.833 120.731 300.539 600.514 480.789 390.448 260.467 600.573 260.484 400.936 440.396 470.061 680.501 450.507 490.594 440.700 370.563 330.874 310.771 450.493 55
Huan Lei, Naveed Akhtar, and Ajmal Mian: SegGCN: Efficient 3D Point Cloud Segmentation with Fuzzy Spherical Kernel. CVPR 2020
CCRFNet0.589 480.766 290.659 540.683 380.470 530.740 520.387 510.620 520.490 470.476 430.922 550.355 550.245 290.511 410.511 470.571 490.643 510.493 520.872 340.762 480.600 31
ROSMRF0.580 500.772 250.707 360.681 390.563 410.764 460.362 530.515 590.465 530.465 450.936 440.427 400.207 380.438 510.577 420.536 530.675 440.486 530.723 580.779 400.524 51
TextureNetpermissive0.566 510.672 470.664 520.671 430.494 490.719 530.445 300.678 440.411 600.396 550.935 460.356 540.225 330.412 550.535 450.565 500.636 540.464 560.794 520.680 600.568 42
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
DVVNet0.562 520.648 490.700 400.770 200.586 330.687 570.333 560.650 470.514 430.475 440.906 610.359 530.223 340.340 590.442 540.422 620.668 460.501 500.708 590.779 400.534 49
Pointnet++ & Featurepermissive0.557 530.735 350.661 530.686 370.491 500.744 510.392 470.539 570.451 550.375 580.946 260.376 500.205 390.403 560.356 590.553 520.643 510.497 510.824 470.756 490.515 52
PointMRNet-lite0.553 540.633 510.648 560.659 470.430 560.800 310.390 500.592 530.454 540.371 590.939 420.368 520.136 610.368 570.448 520.560 510.715 300.486 530.882 240.720 550.462 60
PanopticFusion-label0.529 550.491 640.688 470.604 540.386 590.632 620.225 670.705 390.434 580.293 630.815 650.348 560.241 300.499 460.669 360.507 550.649 480.442 610.796 510.602 660.561 45
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
Online SegFusion0.515 560.607 540.644 590.579 570.434 550.630 630.353 540.628 510.440 560.410 530.762 680.307 580.167 520.520 390.403 570.516 540.565 580.447 600.678 610.701 570.514 53
Davide Menini, Suryansh Kumar, Martin R. Oswald, Erik Sandstroem, Cristian Sminchisescu, Luc van Gool: A Real-Time Learning Framework for Joint 3D Reconstruction and Semantic Segmentation. Robotics and Automation Letters Submission
3DMV, FTSDF0.501 570.558 600.608 630.424 680.478 520.690 560.246 630.586 540.468 510.450 470.911 590.394 480.160 550.438 510.212 650.432 610.541 630.475 550.742 560.727 540.477 58
PCNN0.498 580.559 590.644 590.560 590.420 580.711 550.229 650.414 610.436 570.352 600.941 380.324 570.155 560.238 650.387 580.493 560.529 640.509 480.813 500.751 510.504 54
3DMV0.484 590.484 650.538 660.643 510.424 570.606 660.310 570.574 550.433 590.378 570.796 660.301 590.214 370.537 370.208 660.472 600.507 670.413 640.693 600.602 660.539 48
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGBpermissive0.458 600.577 580.611 620.356 700.321 660.715 540.299 590.376 640.328 670.319 610.944 330.285 610.164 530.216 680.229 640.484 580.545 620.456 580.755 550.709 560.475 59
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPNpermissive0.447 610.679 460.604 640.578 580.380 600.682 580.291 600.106 700.483 480.258 680.920 560.258 640.025 700.231 670.325 600.480 590.560 600.463 570.725 570.666 620.231 70
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
PNET20.442 620.548 610.548 650.597 560.363 620.628 640.300 580.292 650.374 630.307 620.881 630.268 630.186 460.238 650.204 670.407 630.506 680.449 590.667 620.620 640.462 60
SurfaceConvPF0.442 620.505 630.622 610.380 690.342 640.654 600.227 660.397 630.367 640.276 650.924 530.240 660.198 420.359 580.262 620.366 640.581 570.435 620.640 630.668 610.398 62
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
Tangent Convolutionspermissive0.438 640.437 680.646 580.474 650.369 610.645 610.353 540.258 670.282 690.279 640.918 580.298 600.147 600.283 620.294 610.487 570.562 590.427 630.619 640.633 630.352 65
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
3DWSSS0.425 650.525 620.647 570.522 610.324 650.488 700.077 710.712 370.353 650.401 540.636 700.281 620.176 490.340 590.565 430.175 710.551 610.398 650.370 710.602 660.361 64
subcloud_weak0.411 660.479 660.650 550.475 640.285 690.519 690.087 700.725 330.396 620.386 560.621 710.250 650.117 620.338 610.443 530.188 700.594 560.369 680.377 700.616 650.306 66
SPLAT Netcopyleft0.393 670.472 670.511 670.606 530.311 670.656 590.245 640.405 620.328 670.197 690.927 520.227 680.000 720.001 720.249 630.271 690.510 650.383 670.593 650.699 580.267 68
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF0.383 680.297 700.491 680.432 670.358 630.612 650.274 610.116 690.411 600.265 660.904 620.229 670.079 660.250 630.185 680.320 670.510 650.385 660.548 660.597 690.394 63
PointNet++permissive0.339 690.584 570.478 690.458 660.256 700.360 710.250 620.247 680.278 700.261 670.677 690.183 690.117 620.212 690.145 700.364 650.346 710.232 710.548 660.523 700.252 69
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNetpermissive0.308 700.353 690.290 710.278 710.166 710.553 670.169 690.286 660.147 710.148 710.908 600.182 700.064 670.023 710.018 720.354 660.363 690.345 690.546 680.685 590.278 67
ScanNetpermissive0.306 710.203 710.366 700.501 620.311 670.524 680.211 680.002 720.342 660.189 700.786 670.145 710.102 650.245 640.152 690.318 680.348 700.300 700.460 690.437 710.182 71
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
ERROR0.054 720.000 720.041 720.172 720.030 720.062 720.001 720.035 710.004 720.051 720.143 720.019 720.003 710.041 700.050 710.003 720.054 720.018 720.005 720.264 720.082 72