The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).



This table lists the benchmark results for the 3D semantic label scenario.


Method Infoavg ioubathtubbedbookshelfcabinetchaircountercurtaindeskdoorfloorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwallwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
OccuSeg+Semantic0.764 10.758 280.796 70.839 40.746 20.907 10.562 10.850 50.680 20.672 10.978 10.610 10.335 30.777 10.819 130.847 10.830 10.691 30.972 10.885 10.727 2
BPNet0.749 20.909 10.818 40.811 80.752 10.839 60.485 120.842 70.673 30.644 40.957 30.528 70.305 90.773 20.859 40.788 20.818 30.693 20.916 60.856 50.723 4
Wenbo Hu, Hengshuang Zhao, Li Jiang, Jiaya Jia, Tien-Tsin Wong: Bidirectional Projection Network for Cross Dimension Scene Understanding. CVPR 2021
Virtual MVFusion0.746 30.771 240.819 30.848 20.702 80.865 30.397 420.899 10.699 10.664 20.948 200.588 20.330 40.746 50.851 70.764 30.796 70.704 10.935 40.866 20.728 1
Abhijit Kundu, Xiaoqi Yin, Alireza Fathi, David Ross, Brian Brewington, Thomas Funkhouser, Caroline Pantofaru: Virtual Multi-view Fusion for 3D Semantic Segmentation. ECCV 2020
VMNet0.746 30.870 50.838 10.858 10.729 40.850 40.501 70.874 20.587 190.658 30.956 40.564 40.299 100.765 30.900 10.716 70.812 40.631 100.939 20.858 40.709 5
MinkowskiNetpermissive0.736 50.859 70.818 40.832 50.709 70.840 50.521 40.853 40.660 40.643 50.951 100.544 50.286 150.731 60.893 20.675 160.772 130.683 40.874 250.852 60.727 2
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet0.725 60.647 460.821 20.846 30.721 50.869 20.533 20.754 180.603 160.614 60.955 50.572 30.325 50.710 70.870 30.724 60.823 20.628 110.934 50.865 30.683 8
MatchingNet0.724 70.812 160.812 60.810 90.735 30.834 80.495 100.860 30.572 240.602 100.954 60.512 90.280 160.757 40.845 100.725 50.780 100.606 180.937 30.851 70.700 7
RFCR0.702 80.889 30.745 200.813 70.672 100.818 180.493 110.815 90.623 100.610 70.947 230.470 190.249 250.594 210.848 80.705 110.779 110.646 70.892 160.823 120.611 22
One Thing One Click0.701 90.825 140.796 70.723 240.716 60.832 90.433 320.816 80.634 80.609 80.969 20.418 400.344 20.559 310.833 110.715 80.808 50.560 300.902 90.847 80.680 9
JSENet0.699 100.881 40.762 150.821 60.667 110.800 280.522 30.792 110.613 110.607 90.935 420.492 130.205 360.576 270.853 60.691 120.758 160.652 60.872 280.828 100.649 14
Zeyu HU, Mingmin Zhen, Xuyang BAI, Hongbo Fu, Chiew-lan Tai: JSENet: Joint Semantic Segmentation and Edge Detection Network for 3D Point Clouds. ECCV 2020
CU-Hybrid Net0.693 110.596 510.789 90.803 110.677 90.800 280.469 160.846 60.554 310.591 120.948 200.500 100.316 70.609 180.847 90.732 40.808 50.593 210.894 140.839 90.652 13
FusionNet0.688 120.704 390.741 230.754 210.656 120.829 110.501 70.741 230.609 130.548 170.950 140.522 80.371 10.633 140.756 200.715 80.771 140.623 120.861 340.814 160.658 12
Feihu Zhang, Jin Fang, Benjamin Wah, Philip Torr: Deep FusionNet for Point Cloud Semantic Segmentation. ECCV 2020
KP-FCNN0.684 130.847 90.758 180.784 140.647 150.814 200.473 140.772 140.605 140.594 110.935 420.450 270.181 440.587 220.805 150.690 130.785 90.614 140.882 200.819 140.632 18
H. Thomas, C. Qi, J. Deschaud, B. Marcotegui, F. Goulette, L. Guibas.: KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV 2019
PointContrast_LA_SEM0.683 140.757 290.784 100.786 130.639 170.824 150.408 360.775 130.604 150.541 190.934 460.532 60.269 200.552 320.777 180.645 280.793 80.640 80.913 70.824 110.671 10
ROSMRF3D0.673 150.789 170.748 190.763 190.635 190.814 200.407 390.747 200.581 220.573 130.950 140.484 140.271 190.607 190.754 210.649 240.774 120.596 200.883 190.823 120.606 26
SALANet0.670 160.816 150.770 140.768 170.652 140.807 250.451 200.747 200.659 50.545 180.924 490.473 180.149 540.571 290.811 140.635 310.746 190.623 120.892 160.794 270.570 37
PointConvpermissive0.666 170.781 190.759 170.699 290.644 160.822 160.475 130.779 120.564 270.504 320.953 70.428 350.203 380.586 240.754 210.661 210.753 170.588 220.902 90.813 180.642 15
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
PointASNLpermissive0.666 170.703 400.781 110.751 230.655 130.830 100.471 150.769 150.474 470.537 200.951 100.475 170.279 170.635 120.698 320.675 160.751 180.553 350.816 450.806 190.703 6
Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, Shuguang Cui: PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Sampling. CVPR 2020
VACNN++0.664 190.891 20.724 290.680 360.636 180.814 200.438 310.629 460.553 320.537 200.950 140.499 110.247 260.626 150.786 170.666 190.700 320.566 280.860 350.816 150.665 11
DCM-Net0.658 200.778 200.702 350.806 100.619 210.813 230.468 170.693 340.494 430.524 260.941 350.449 280.298 110.510 380.821 120.675 160.727 240.568 260.826 420.803 210.637 17
Jonas Schult*, Francis Engelmann*, Theodora Kontogianni, Bastian Leibe: DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes. CVPR 2020 [Oral]
HPGCNN0.656 210.698 410.743 220.650 440.564 370.820 170.505 60.758 170.631 90.479 390.945 270.480 150.226 290.572 280.774 190.690 130.735 230.614 140.853 380.776 400.597 30
Jisheng Dang, Qingyong Hu, Yulan Guo, Jun Yang: HPGCNN.
SAFNet-segcopyleft0.654 220.752 300.734 250.664 430.583 300.815 190.399 410.754 180.639 70.535 220.942 330.470 190.309 80.665 80.539 400.650 230.708 280.635 90.857 370.793 280.642 15
RandLA-Netpermissive0.645 230.778 200.731 260.699 290.577 320.829 110.446 240.736 240.477 460.523 280.945 270.454 250.269 200.484 450.749 240.618 350.738 210.599 190.827 410.792 300.621 20
PointConv-SFPN0.641 240.776 220.703 340.721 250.557 400.826 130.451 200.672 390.563 280.483 380.943 320.425 380.162 480.644 100.726 250.659 220.709 270.572 240.875 230.786 340.559 42
MVPNetpermissive0.641 240.831 110.715 300.671 400.590 270.781 370.394 430.679 370.642 60.553 160.937 390.462 220.256 220.649 90.406 510.626 320.691 360.666 50.877 210.792 300.608 25
Maximilian Jaritz, Jiayuan Gu, Hao Su: Multi-view PointNet for 3D Scene Understanding. GMDL Workshop, ICCV 2019
PointMRNet0.640 260.717 380.701 360.692 320.576 330.801 260.467 180.716 290.563 280.459 430.953 70.429 340.169 460.581 250.854 50.605 360.710 260.550 360.894 140.793 280.575 35
FPConvpermissive0.639 270.785 180.760 160.713 280.603 240.798 300.392 440.534 520.603 160.524 260.948 200.457 230.250 240.538 350.723 260.598 400.696 350.614 140.872 280.799 220.567 39
Yiqun Lin, Zizheng Yan, Haibin Huang, Dong Du, Ligang Liu, Shuguang Cui, Xiaoguang Han: FPConv: Learning Local Flattening for Point Convolution. CVPR 2020
PointSPNet0.637 280.734 330.692 410.714 270.576 330.797 310.446 240.743 220.598 180.437 470.942 330.403 420.150 530.626 150.800 160.649 240.697 340.557 330.846 390.777 390.563 40
SConv0.636 290.830 120.697 390.752 220.572 360.780 380.445 260.716 290.529 360.530 230.951 100.446 290.170 450.507 400.666 340.636 300.682 380.541 410.886 180.799 220.594 31
Supervoxel-CNN0.635 300.656 440.711 310.719 260.613 220.757 440.444 280.765 160.534 350.566 140.928 470.478 160.272 180.636 110.531 420.664 200.645 460.508 460.864 330.792 300.611 22
joint point-basedpermissive0.634 310.614 490.778 120.667 420.633 200.825 140.420 330.804 100.467 490.561 150.951 100.494 120.291 120.566 300.458 460.579 450.764 150.559 320.838 400.814 160.598 29
Hung-Yueh Chiang, Yen-Liang Lin, Yueh-Cheng Liu, Winston H. Hsu: A Unified Point-Based Framework for 3D Segmentation. 3DV 2019
MCCNNpermissive0.633 320.866 60.731 260.771 150.576 330.809 240.410 350.684 350.497 420.491 350.949 170.466 210.105 580.581 250.646 350.620 330.680 390.542 400.817 440.795 250.618 21
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
PointMTL0.632 330.731 340.688 440.675 370.591 260.784 360.444 280.565 500.610 120.492 340.949 170.456 240.254 230.587 220.706 290.599 390.665 430.612 170.868 320.791 330.579 34
APCF-Net0.631 340.742 310.687 460.672 380.557 400.792 330.408 360.665 410.545 330.508 300.952 90.428 350.186 420.634 130.702 300.620 330.706 290.555 340.873 270.798 240.581 33
Haojia, Lin: Adaptive Pyramid Context Fusion for Point Cloud Perception. GRSL
PointNet2-SFPN0.631 340.771 240.692 410.672 380.524 430.837 70.440 300.706 320.538 340.446 450.944 290.421 390.219 320.552 320.751 230.591 420.737 220.543 390.901 110.768 430.557 43
3DSM_DMMF0.631 340.626 480.745 200.801 120.607 230.751 450.506 50.729 270.565 260.491 350.866 600.434 310.197 400.595 200.630 360.709 100.705 300.560 300.875 230.740 500.491 52
FusionAwareConv0.630 370.604 500.741 230.766 180.590 270.747 460.501 70.734 250.503 410.527 240.919 530.454 250.323 60.550 340.420 500.678 150.688 370.544 380.896 130.795 250.627 19
Jiazhao Zhang, Chenyang Zhu, Lintao Zheng, Kai Xu: Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentation. CVPR 2020
PointMRNet-lite0.625 380.643 470.711 310.697 310.581 310.801 260.408 360.670 400.558 300.497 330.944 290.436 300.152 520.617 170.708 280.603 370.743 200.532 440.870 310.784 350.545 44
SIConv0.625 380.830 120.694 400.757 200.563 380.772 400.448 220.647 440.520 380.509 290.949 170.431 330.191 410.496 430.614 370.647 270.672 410.535 430.876 220.783 360.571 36
HPEIN0.618 400.729 350.668 470.647 460.597 250.766 410.414 340.680 360.520 380.525 250.946 250.432 320.215 330.493 440.599 380.638 290.617 510.570 250.897 120.806 190.605 27
Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, Jiaya Jia: Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation. ICCV 2019
SPH3D-GCNpermissive0.610 410.858 80.772 130.489 570.532 420.792 330.404 400.643 450.570 250.507 310.935 420.414 410.046 630.510 380.702 300.602 380.705 300.549 370.859 360.773 410.534 46
Huan Lei, Naveed Akhtar, and Ajmal Mian: Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds. TPAMI 2020
AttAN0.609 420.760 270.667 480.649 450.521 440.793 320.457 190.648 430.528 370.434 490.947 230.401 430.153 510.454 460.721 270.648 260.717 250.536 420.904 80.765 440.485 53
Gege Zhang, Qinghua Ma, Licheng Jiao, Fang Liu and Qigong Sun: AttAN: Attention Adversarial Networks for 3D Point Cloud Semantic Segmentation. IJCAI2020
LAP-D0.594 430.720 360.692 410.637 480.456 510.773 390.391 460.730 260.587 190.445 460.940 370.381 460.288 130.434 490.453 470.591 420.649 440.581 230.777 490.749 490.610 24
DPC0.592 440.720 360.700 370.602 510.480 480.762 430.380 480.713 310.585 210.437 470.940 370.369 480.288 130.434 490.509 440.590 440.639 490.567 270.772 500.755 470.592 32
Francis Engelmann, Theodora Kontogianni, Bastian Leibe: Dilated Point Convolutions: On the Receptive Field Size of Point Convolutions on 3D Point Clouds. ICRA 2020
CCRFNet0.589 450.766 260.659 510.683 340.470 500.740 480.387 470.620 470.490 440.476 400.922 510.355 510.245 270.511 370.511 430.571 460.643 470.493 490.872 280.762 450.600 28
SegGCNpermissive0.589 450.833 100.731 260.539 550.514 450.789 350.448 220.467 540.573 230.484 370.936 400.396 440.061 620.501 410.507 450.594 410.700 320.563 290.874 250.771 420.493 51
Huan Lei, Naveed Akhtar, and Ajmal Mian: SegGCN: Efficient 3D Point Cloud Segmentation with Fuzzy Spherical Kernel. CVPR 2020
ROSMRF0.580 470.772 230.707 330.681 350.563 380.764 420.362 490.515 530.465 500.465 420.936 400.427 370.207 350.438 470.577 390.536 490.675 400.486 500.723 540.779 370.524 48
TextureNetpermissive0.566 480.672 430.664 490.671 400.494 460.719 490.445 260.678 380.411 550.396 500.935 420.356 500.225 300.412 510.535 410.565 470.636 500.464 520.794 480.680 550.568 38
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
DVVNet0.562 490.648 450.700 370.770 160.586 290.687 530.333 510.650 420.514 400.475 410.906 570.359 490.223 310.340 540.442 490.422 570.668 420.501 470.708 550.779 370.534 46
Pointnet++ & Featurepermissive0.557 500.735 320.661 500.686 330.491 470.744 470.392 440.539 510.451 510.375 530.946 250.376 470.205 360.403 520.356 530.553 480.643 470.497 480.824 430.756 460.515 49
PanopticFusion-label0.529 510.491 580.688 440.604 500.386 540.632 580.225 620.705 330.434 530.293 570.815 610.348 520.241 280.499 420.669 330.507 500.649 440.442 560.796 470.602 610.561 41
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
3DMV, FTSDF0.501 520.558 550.608 570.424 620.478 490.690 520.246 580.586 480.468 480.450 440.911 550.394 450.160 490.438 470.212 590.432 560.541 570.475 510.742 520.727 510.477 54
PCNN0.498 530.559 540.644 540.560 540.420 530.711 510.229 600.414 550.436 520.352 540.941 350.324 530.155 500.238 590.387 520.493 510.529 580.509 450.813 460.751 480.504 50
3DMV0.484 540.484 590.538 600.643 470.424 520.606 610.310 520.574 490.433 540.378 520.796 620.301 540.214 340.537 360.208 600.472 550.507 610.413 590.693 560.602 610.539 45
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGBpermissive0.458 550.577 530.611 560.356 640.321 600.715 500.299 540.376 580.328 610.319 550.944 290.285 560.164 470.216 620.229 580.484 530.545 560.456 540.755 510.709 520.475 55
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPNpermissive0.447 560.679 420.604 580.578 530.380 550.682 540.291 550.106 640.483 450.258 620.920 520.258 580.025 640.231 610.325 540.480 540.560 550.463 530.725 530.666 570.231 64
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
PNET20.442 570.548 560.548 590.597 520.363 570.628 590.300 530.292 590.374 580.307 560.881 590.268 570.186 420.238 590.204 610.407 580.506 620.449 550.667 570.620 590.462 56
SurfaceConvPF0.442 570.505 570.622 550.380 630.342 590.654 560.227 610.397 570.367 590.276 590.924 490.240 600.198 390.359 530.262 560.366 590.581 530.435 570.640 580.668 560.398 57
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
Tangent Convolutionspermissive0.438 590.437 620.646 530.474 590.369 560.645 570.353 500.258 610.282 630.279 580.918 540.298 550.147 550.283 560.294 550.487 520.562 540.427 580.619 590.633 580.352 59
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
subcloud_weak0.411 600.479 600.650 520.475 580.285 630.519 640.087 650.725 280.396 570.386 510.621 650.250 590.117 560.338 550.443 480.188 650.594 520.369 620.377 650.616 600.306 60
SPLAT Netcopyleft0.393 610.472 610.511 610.606 490.311 610.656 550.245 590.405 560.328 610.197 630.927 480.227 620.000 660.001 660.249 570.271 640.510 590.383 610.593 600.699 530.267 62
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF0.383 620.297 640.491 620.432 610.358 580.612 600.274 560.116 630.411 550.265 600.904 580.229 610.079 600.250 570.185 620.320 620.510 590.385 600.548 610.597 630.394 58
PointNet++permissive0.339 630.584 520.478 630.458 600.256 640.360 650.250 570.247 620.278 640.261 610.677 640.183 630.117 560.212 630.145 640.364 600.346 650.232 650.548 610.523 640.252 63
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNetpermissive0.308 640.353 630.290 650.278 650.166 650.553 620.169 640.286 600.147 650.148 650.908 560.182 640.064 610.023 650.018 660.354 610.363 630.345 630.546 630.685 540.278 61
ScanNetpermissive0.306 650.203 650.366 640.501 560.311 610.524 630.211 630.002 660.342 600.189 640.786 630.145 650.102 590.245 580.152 630.318 630.348 640.300 640.460 640.437 650.182 65
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
ERROR0.054 660.000 660.041 660.172 660.030 660.062 660.001 660.035 650.004 660.051 660.143 660.019 660.003 650.041 640.050 650.003 660.054 660.018 660.005 660.264 660.082 66