The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).



This table lists the benchmark results for the 3D semantic label scenario.


Method Infoavg ioubathtubbedbookshelfcabinetchaircountercurtaindeskdoorfloorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwallwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
OccuSeg+Semantic0.764 10.758 230.796 60.839 30.746 20.907 10.562 10.850 40.680 20.672 10.978 10.610 10.335 20.777 10.819 110.847 10.830 10.691 30.972 10.885 10.727 2
BPNet0.749 20.909 10.818 30.811 70.752 10.839 50.485 110.842 60.673 30.644 30.957 20.528 50.305 70.773 20.859 30.788 20.818 30.693 20.916 50.856 40.723 4
Virtual MVFusion0.746 30.771 200.819 20.848 10.702 60.865 30.397 360.899 10.699 10.664 20.948 170.588 20.330 30.746 40.851 60.764 30.796 50.704 10.935 30.866 20.728 1
Abhijit Kundu, Xiaoqi Yin, Alireza Fathi, David Ross, Brian Brewington, Thomas Funkhouser, Caroline Pantofaru: Virtual Multi-view Fusion for 3D Semantic Segmentation. ECCV 2020
MinkowskiNetpermissive0.736 40.859 50.818 30.832 40.709 50.840 40.521 40.853 30.660 40.643 40.951 80.544 40.286 120.731 50.893 10.675 140.772 90.683 40.874 200.852 50.727 2
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet0.725 50.647 390.821 10.846 20.721 40.869 20.533 20.754 150.603 130.614 50.955 30.572 30.325 40.710 60.870 20.724 60.823 20.628 80.934 40.865 30.683 7
MatchingNet0.724 60.812 140.812 50.810 80.735 30.834 60.495 90.860 20.572 190.602 80.954 40.512 70.280 130.757 30.845 90.725 50.780 70.606 150.937 20.851 60.700 6
RFCR0.702 70.889 20.745 160.813 60.672 80.818 140.493 100.815 70.623 80.610 60.947 200.470 150.249 210.594 170.848 70.705 90.779 80.646 70.892 120.823 90.611 18
JSENet0.699 80.881 30.762 120.821 50.667 90.800 210.522 30.792 90.613 90.607 70.935 360.492 100.205 290.576 230.853 50.691 100.758 120.652 60.872 230.828 80.649 11
Zeyu HU, Mingmin Zhen, Xuyang BAI, Hongbo Fu, Chiew-lan Tai: JSENet: Joint Semantic Segmentation and Edge Detection Network for 3D Point Clouds. ECCV 2020
CU-Hybrid Net0.693 90.596 440.789 70.803 100.677 70.800 210.469 150.846 50.554 270.591 100.948 170.500 80.316 60.609 150.847 80.732 40.808 40.593 170.894 100.839 70.652 10
FusionNet0.688 100.704 320.741 190.754 180.656 100.829 80.501 70.741 180.609 110.548 140.950 120.522 60.371 10.633 120.756 160.715 70.771 100.623 90.861 290.814 110.658 9
Feihu Zhang, Jin Fang, Benjamin Wah, Philip Torr: Deep FusionNet for Point Cloud Semantic Segmentation. ECCV 2020
KP-FCNN0.684 110.847 70.758 150.784 120.647 130.814 150.473 130.772 110.605 120.594 90.935 360.450 230.181 370.587 180.805 130.690 110.785 60.614 110.882 150.819 100.632 14
H. Thomas, C. Qi, J. Deschaud, B. Marcotegui, F. Goulette, L. Guibas.: KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV 2019
SALANet0.670 120.816 130.770 110.768 150.652 120.807 180.451 190.747 160.659 50.545 150.924 420.473 140.149 470.571 250.811 120.635 250.746 150.623 90.892 120.794 230.570 32
PointConvpermissive0.666 130.781 160.759 140.699 250.644 140.822 120.475 120.779 100.564 230.504 270.953 50.428 310.203 310.586 200.754 170.661 180.753 130.588 180.902 70.813 130.642 12
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
PointASNLpermissive0.666 130.703 330.781 80.751 200.655 110.830 70.471 140.769 120.474 410.537 160.951 80.475 130.279 140.635 100.698 270.675 140.751 140.553 300.816 380.806 140.703 5
Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, Shuguang Cui: PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Sampling. CVPR 2020
DCM-Net0.658 150.778 170.702 280.806 90.619 160.813 160.468 160.693 280.494 370.524 210.941 300.449 240.298 80.510 320.821 100.675 140.727 190.568 230.826 350.803 160.637 13
Jonas Schult*, Francis Engelmann*, Theodora Kontogianni, Bastian Leibe: DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes. CVPR 2020 [Oral]
HPGCNN0.656 160.698 340.743 180.650 370.564 320.820 130.505 60.758 140.631 70.479 340.945 240.480 110.226 240.572 240.774 150.690 110.735 180.614 110.853 310.776 340.597 25
Jisheng Dang, Qingyong Hu, Yulan Guo, Jun Yang: HPGCNN.
RandLA-Netpermissive0.645 170.778 170.731 210.699 250.577 270.829 80.446 230.736 190.477 400.523 230.945 240.454 210.269 160.484 390.749 180.618 290.738 170.599 160.827 340.792 250.621 16
MVPNetpermissive0.641 180.831 90.715 240.671 340.590 230.781 310.394 370.679 310.642 60.553 130.937 340.462 180.256 170.649 70.406 440.626 260.691 300.666 50.877 160.792 250.608 21
Maximilian Jaritz, Jiayuan Gu, Hao Su: Multi-view PointNet for 3D Scene Understanding. GMDL Workshop, ICCV 2019
PointConv-SFPN0.641 180.776 190.703 270.721 210.557 340.826 100.451 190.672 330.563 240.483 330.943 280.425 330.162 410.644 80.726 190.659 190.709 220.572 200.875 180.786 290.559 37
PointMRNet0.640 200.717 310.701 290.692 280.576 280.801 190.467 170.716 240.563 240.459 370.953 50.429 300.169 390.581 210.854 40.605 310.710 210.550 310.894 100.793 240.575 30
FPConvpermissive0.639 210.785 150.760 130.713 240.603 190.798 230.392 380.534 460.603 130.524 210.948 170.457 190.250 200.538 280.723 200.598 350.696 280.614 110.872 230.799 180.567 34
Yiqun Lin, Zizheng Yan, Haibin Huang, Dong Du, Ligang Liu, Shuguang Cui, Xiaoguang Han: FPConv: Learning Local Flattening for Point Convolution. CVPR 2020
VACNN++0.638 220.820 120.701 290.687 290.594 210.791 280.430 290.587 410.569 210.529 180.950 120.467 160.253 190.524 300.722 210.618 290.694 290.570 210.793 420.802 170.659 8
PointSPNet0.637 230.734 260.692 350.714 230.576 280.797 240.446 230.743 170.598 150.437 400.942 290.403 350.150 460.626 130.800 140.649 200.697 270.557 280.846 320.777 330.563 35
SConv0.636 240.830 100.697 330.752 190.572 310.780 320.445 250.716 240.529 300.530 170.951 80.446 250.170 380.507 340.666 290.636 240.682 320.541 350.886 140.799 180.594 26
Supervoxel-CNN0.635 250.656 370.711 250.719 220.613 170.757 370.444 270.765 130.534 290.566 110.928 400.478 120.272 150.636 90.531 350.664 170.645 390.508 400.864 280.792 250.611 18
joint point-basedpermissive0.634 260.614 420.778 90.667 360.633 150.825 110.420 300.804 80.467 430.561 120.951 80.494 90.291 90.566 260.458 390.579 390.764 110.559 270.838 330.814 110.598 24
Hung-Yueh Chiang, Yen-Liang Lin, Yueh-Cheng Liu, Winston H. Hsu: A Unified Point-Based Framework for 3D Segmentation. 3DV 2019
MCCNNpermissive0.633 270.866 40.731 210.771 130.576 280.809 170.410 320.684 290.497 360.491 300.949 140.466 170.105 510.581 210.646 300.620 270.680 330.542 340.817 370.795 210.618 17
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
PointMTL0.632 280.731 270.688 370.675 320.591 220.784 300.444 270.565 440.610 100.492 290.949 140.456 200.254 180.587 180.706 240.599 340.665 360.612 140.868 270.791 280.579 29
3DSM_DMMF0.631 290.626 410.745 160.801 110.607 180.751 380.506 50.729 220.565 220.491 300.866 530.434 270.197 330.595 160.630 310.709 80.705 240.560 260.875 180.740 430.491 45
APCF-Net0.631 290.742 240.687 390.672 330.557 340.792 260.408 330.665 350.545 280.508 250.952 70.428 310.186 350.634 110.702 250.620 270.706 230.555 290.873 220.798 200.581 28
Haojia, Lin: Adaptive Pyramid Context Fusion for Point Cloud Perception. GRSL
FusionAwareConv0.630 310.604 430.741 190.766 160.590 230.747 390.501 70.734 200.503 350.527 190.919 460.454 210.323 50.550 270.420 430.678 130.688 310.544 330.896 90.795 210.627 15
Jiazhao Zhang, Chenyang Zhu, Lintao Zheng, Kai Xu: Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentation. CVPR 2020
PointMRNet-lite0.625 320.643 400.711 250.697 270.581 260.801 190.408 330.670 340.558 260.497 280.944 260.436 260.152 450.617 140.708 230.603 320.743 160.532 380.870 260.784 300.545 38
SIConv0.625 320.830 100.694 340.757 170.563 330.772 340.448 210.647 380.520 320.509 240.949 140.431 290.191 340.496 370.614 320.647 220.672 340.535 370.876 170.783 310.571 31
HPEIN0.618 340.729 280.668 400.647 390.597 200.766 350.414 310.680 300.520 320.525 200.946 220.432 280.215 270.493 380.599 330.638 230.617 440.570 210.897 80.806 140.605 22
Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, Jiaya Jia: Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation. ICCV 2019
SPH3D-GCNpermissive0.610 350.858 60.772 100.489 500.532 360.792 260.404 350.643 390.570 200.507 260.935 360.414 340.046 560.510 320.702 250.602 330.705 240.549 320.859 300.773 350.534 40
Huan Lei, Naveed Akhtar, and Ajmal Mian: Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds. TPAMI 2020
AttAN0.609 360.760 220.667 410.649 380.521 370.793 250.457 180.648 370.528 310.434 420.947 200.401 360.153 440.454 400.721 220.648 210.717 200.536 360.904 60.765 370.485 46
Gege Zhang, Qinghua Ma, Licheng Jiao, Fang Liu and Qigong Sun: AttAN: Attention Adversarial Networks for 3D Point Cloud Semantic Segmentation. IJCAI2020
LAP-D0.594 370.720 290.692 350.637 410.456 440.773 330.391 400.730 210.587 160.445 390.940 320.381 390.288 100.434 420.453 400.591 370.649 370.581 190.777 430.749 420.610 20
DPC0.592 380.720 290.700 310.602 440.480 410.762 360.380 420.713 260.585 170.437 400.940 320.369 410.288 100.434 420.509 370.590 380.639 420.567 240.772 440.755 400.592 27
Francis Engelmann, Theodora Kontogianni, Bastian Leibe: Dilated Point Convolutions: On the Receptive Field Size of Point Convolutions on 3D Point Clouds. ICRA 2020
CCRFNet0.589 390.766 210.659 440.683 310.470 430.740 410.387 410.620 400.490 380.476 350.922 440.355 440.245 220.511 310.511 360.571 400.643 400.493 430.872 230.762 380.600 23
SegGCNpermissive0.589 390.833 80.731 210.539 480.514 380.789 290.448 210.467 470.573 180.484 320.936 350.396 370.061 550.501 350.507 380.594 360.700 260.563 250.874 200.771 360.493 44
Huan Lei, Naveed Akhtar, and Ajmal Mian: SegGCN: Efficient 3D Point Cloud Segmentation with Fuzzy Spherical Kernel. CVPR 2020
TextureNetpermissive0.566 410.672 360.664 420.671 340.494 390.719 420.445 250.678 320.411 480.396 430.935 360.356 430.225 250.412 440.535 340.565 410.636 430.464 450.794 410.680 480.568 33
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
DVVNet0.562 420.648 380.700 310.770 140.586 250.687 460.333 440.650 360.514 340.475 360.906 500.359 420.223 260.340 470.442 420.422 500.668 350.501 410.708 480.779 320.534 40
Pointnet++ & Featurepermissive0.557 430.735 250.661 430.686 300.491 400.744 400.392 380.539 450.451 440.375 460.946 220.376 400.205 290.403 450.356 460.553 420.643 400.497 420.824 360.756 390.515 42
PanopticFusion-label0.529 440.491 510.688 370.604 430.386 470.632 510.225 550.705 270.434 460.293 500.815 540.348 450.241 230.499 360.669 280.507 430.649 370.442 490.796 400.602 540.561 36
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
3DMV, FTSDF0.501 450.558 480.608 500.424 550.478 420.690 450.246 510.586 420.468 420.450 380.911 480.394 380.160 420.438 410.212 520.432 490.541 500.475 440.742 460.727 440.477 47
PCNN0.498 460.559 470.644 470.560 470.420 460.711 440.229 530.414 480.436 450.352 470.941 300.324 460.155 430.238 520.387 450.493 440.529 510.509 390.813 390.751 410.504 43
3DMV0.484 470.484 520.538 530.643 400.424 450.606 540.310 450.574 430.433 470.378 450.796 550.301 470.214 280.537 290.208 530.472 480.507 540.413 520.693 490.602 540.539 39
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGBpermissive0.458 480.577 460.611 490.356 570.321 530.715 430.299 470.376 510.328 540.319 480.944 260.285 490.164 400.216 550.229 510.484 460.545 490.456 470.755 450.709 450.475 48
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPNpermissive0.447 490.679 350.604 510.578 460.380 480.682 470.291 480.106 570.483 390.258 550.920 450.258 510.025 570.231 540.325 470.480 470.560 480.463 460.725 470.666 500.231 57
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
PNET20.442 500.548 490.548 520.597 450.363 500.628 520.300 460.292 520.374 510.307 490.881 520.268 500.186 350.238 520.204 540.407 510.506 550.449 480.667 500.620 520.462 49
SurfaceConvPF0.442 500.505 500.622 480.380 560.342 520.654 490.227 540.397 500.367 520.276 520.924 420.240 530.198 320.359 460.262 490.366 520.581 460.435 500.640 510.668 490.398 50
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
Tangent Convolutionspermissive0.438 520.437 550.646 460.474 520.369 490.645 500.353 430.258 540.282 560.279 510.918 470.298 480.147 480.283 490.294 480.487 450.562 470.427 510.619 520.633 510.352 52
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
subcloud_weak0.411 530.479 530.650 450.475 510.285 560.519 570.087 580.725 230.396 500.386 440.621 580.250 520.117 490.338 480.443 410.188 580.594 450.369 550.377 580.616 530.306 53
SPLAT Netcopyleft0.393 540.472 540.511 540.606 420.311 540.656 480.245 520.405 490.328 540.197 560.927 410.227 550.000 590.001 590.249 500.271 570.510 520.383 540.593 530.699 460.267 55
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF0.383 550.297 570.491 550.432 540.358 510.612 530.274 490.116 560.411 480.265 530.904 510.229 540.079 530.250 500.185 550.320 550.510 520.385 530.548 540.597 560.394 51
PointNet++permissive0.339 560.584 450.478 560.458 530.256 570.360 580.250 500.247 550.278 570.261 540.677 570.183 560.117 490.212 560.145 570.364 530.346 580.232 580.548 540.523 570.252 56
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNetpermissive0.308 570.353 560.290 580.278 580.166 580.553 550.169 570.286 530.147 580.148 580.908 490.182 570.064 540.023 580.018 590.354 540.363 560.345 560.546 560.685 470.278 54
ScanNetpermissive0.306 580.203 580.366 570.501 490.311 540.524 560.211 560.002 590.342 530.189 570.786 560.145 580.102 520.245 510.152 560.318 560.348 570.300 570.460 570.437 580.182 58
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
ERROR0.054 590.000 590.041 590.172 590.030 590.062 590.001 590.035 580.004 590.051 590.143 590.019 590.003 580.041 570.050 580.003 590.054 590.018 590.005 590.264 590.082 59