The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).



This table lists the benchmark results for the 3D semantic label scenario.


Method Infoavg ioubathtubbedbookshelfcabinetchaircountercurtaindeskdoorfloorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwallwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
OccuSeg+Semantic0.764 10.758 180.796 60.839 30.746 20.907 10.562 10.850 40.680 20.672 10.978 10.610 10.335 20.777 10.819 100.847 10.830 10.691 30.972 10.885 10.727 2
BPNet0.749 20.909 10.818 30.811 60.752 10.839 50.485 90.842 60.673 30.644 30.957 20.528 50.305 70.773 20.859 30.788 20.818 30.693 20.916 50.856 40.723 4
Virtual MVFusion0.746 30.771 150.819 20.848 10.702 60.865 30.397 280.899 10.699 10.664 20.948 160.588 20.330 30.746 40.851 60.764 30.796 50.704 10.935 30.866 20.728 1
Abhijit Kundu, Xiaoqi Yin, Alireza Fathi, David Ross, Brian Brewington, Thomas Funkhouser, Caroline Pantofaru: Virtual Multi-view Fusion for 3D Semantic Segmentation. ECCV 2020
MinkowskiNetpermissive0.736 40.859 40.818 30.832 40.709 50.840 40.521 40.853 30.660 40.643 40.951 80.544 40.286 120.731 50.893 10.675 120.772 80.683 40.874 170.852 50.727 2
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet0.725 50.647 310.821 10.846 20.721 40.869 20.533 20.754 120.603 100.614 50.955 30.572 30.325 40.710 60.870 20.724 60.823 20.628 70.934 40.865 30.683 7
MatchingNet0.724 60.812 110.812 50.810 70.735 30.834 60.495 80.860 20.572 150.602 70.954 40.512 70.280 130.757 30.845 80.725 50.780 70.606 120.937 20.851 60.700 6
JSENet0.699 70.881 20.762 110.821 50.667 80.800 160.522 30.792 80.613 60.607 60.935 300.492 100.205 240.576 190.853 50.691 90.758 110.652 60.872 200.828 80.649 10
Zeyu HU, Mingmin Zhen, Xuyang BAI, Hongbo Fu, Chiew-lan Tai: JSENet: Joint Semantic Segmentation and Edge Detection Network for 3D Point Clouds. ECCV 2020
CU-Hybrid Net0.693 80.596 360.789 70.803 90.677 70.800 160.469 130.846 50.554 210.591 90.948 160.500 80.316 60.609 120.847 70.732 40.808 40.593 130.894 100.839 70.652 9
FusionNet0.688 90.704 260.741 160.754 160.656 90.829 80.501 60.741 130.609 80.548 120.950 120.522 60.371 10.633 100.756 120.715 70.771 90.623 80.861 250.814 100.658 8
Feihu Zhang, Jin Fang, Benjamin Wah, Philip Torr: Deep FusionNet for Point Cloud Semantic Segmentation. ECCV 2020
KP-FCNN0.684 100.847 60.758 140.784 110.647 110.814 110.473 110.772 100.605 90.594 80.935 300.450 170.181 320.587 140.805 110.690 100.785 60.614 90.882 130.819 90.632 13
H. Thomas, C. Qi, J. Deschaud, B. Marcotegui, F. Goulette, L. Guibas.: KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV 2019
PointConvpermissive0.666 110.781 130.759 130.699 200.644 120.822 100.475 100.779 90.564 180.504 220.953 50.428 250.203 260.586 160.754 130.661 150.753 120.588 140.902 70.813 120.642 11
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
PointASNLpermissive0.666 110.703 270.781 80.751 180.655 100.830 70.471 120.769 110.474 330.537 130.951 80.475 110.279 140.635 80.698 200.675 120.751 130.553 230.816 310.806 130.703 5
Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, Shuguang Cui: PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Sampling. CVPR 2020
DCM-Net0.658 130.778 140.702 220.806 80.619 140.813 120.468 140.693 220.494 300.524 170.941 240.449 180.298 80.510 250.821 90.675 120.727 150.568 170.826 280.803 150.637 12
Jonas Schult*, Francis Engelmann*, Theodora Kontogianni, Bastian Leibe: DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes. CVPR 2020 [Oral]
MVPNetpermissive0.641 140.831 80.715 200.671 270.590 190.781 240.394 290.679 250.642 50.553 110.937 280.462 130.256 150.649 70.406 360.626 200.691 230.666 50.877 140.792 220.608 17
Maximilian Jaritz, Jiayuan Gu, Hao Su: Multi-view PointNet for 3D Scene Understanding. GMDL Workshop, ICCV 2019
PointMRNet0.640 150.717 250.701 230.692 220.576 230.801 140.467 150.716 180.563 190.459 300.953 50.429 240.169 340.581 170.854 40.605 230.710 170.550 240.894 100.793 210.575 25
FPConvpermissive0.639 160.785 120.760 120.713 190.603 160.798 180.392 300.534 380.603 100.524 170.948 160.457 140.250 170.538 220.723 140.598 270.696 220.614 90.872 200.799 160.567 28
Yiqun Lin, Zizheng Yan, Haibin Huang, Dong Du, Ligang Liu, Shuguang Cui, Xiaoguang Han: FPConv: Learning Local Flattening for Point Convolution. CVPR 2020
SConv0.636 170.830 90.697 260.752 170.572 250.780 250.445 190.716 180.529 230.530 140.951 80.446 190.170 330.507 270.666 220.636 190.682 250.541 280.886 120.799 160.594 21
joint point-basedpermissive0.634 180.614 340.778 90.667 290.633 130.825 90.420 220.804 70.467 350.561 100.951 80.494 90.291 90.566 200.458 310.579 310.764 100.559 210.838 270.814 100.598 20
Hung-Yueh Chiang, Yen-Liang Lin, Yueh-Cheng Liu, Winston H. Hsu: A Unified Point-Based Framework for 3D Segmentation. 3DV 2019
MCCNNpermissive0.633 190.866 30.731 180.771 120.576 230.809 130.410 240.684 230.497 290.491 250.949 130.466 120.105 430.581 170.646 230.620 210.680 260.542 270.817 300.795 190.618 15
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
PointMTL0.632 200.731 210.688 290.675 250.591 180.784 230.444 210.565 360.610 70.492 240.949 130.456 150.254 160.587 140.706 170.599 260.665 290.612 110.868 240.791 230.579 24
3DSM_DMMF0.631 210.626 330.745 150.801 100.607 150.751 300.506 50.729 160.565 170.491 250.866 450.434 210.197 280.595 130.630 240.709 80.705 190.560 200.875 160.740 350.491 37
APCF-Net0.631 210.742 190.687 310.672 260.557 270.792 200.408 250.665 280.545 220.508 200.952 70.428 250.186 300.634 90.702 180.620 210.706 180.555 220.873 190.798 180.581 23
FusionAwareConv0.630 230.604 350.741 160.766 140.590 190.747 310.501 60.734 140.503 280.527 150.919 380.454 160.323 50.550 210.420 350.678 110.688 240.544 260.896 90.795 190.627 14
Jiazhao Zhang, Chenyang Zhu, Lintao Zheng, Kai Xu: Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentation. CVPR 2020
SIConv0.625 240.830 90.694 270.757 150.563 260.772 270.448 170.647 310.520 250.509 190.949 130.431 230.191 290.496 300.614 250.647 170.672 270.535 300.876 150.783 250.571 26
PointMRNet-lite0.625 240.643 320.711 210.697 210.581 220.801 140.408 250.670 270.558 200.497 230.944 220.436 200.152 390.617 110.708 160.603 240.743 140.532 310.870 230.784 240.545 30
HPEIN0.618 260.729 220.668 320.647 310.597 170.766 280.414 230.680 240.520 250.525 160.946 200.432 220.215 220.493 310.599 260.638 180.617 360.570 160.897 80.806 130.605 18
Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, Jiaya Jia: Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation. ICCV 2019
SPH3D-GCNpermissive0.610 270.858 50.772 100.489 420.532 280.792 200.404 270.643 320.570 160.507 210.935 300.414 270.046 480.510 250.702 180.602 250.705 190.549 250.859 260.773 270.534 32
Huan Lei, Naveed Akhtar, and Ajmal Mian: Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds. TPAMI 2020
AttAN0.609 280.760 170.667 330.649 300.521 290.793 190.457 160.648 300.528 240.434 340.947 190.401 280.153 380.454 320.721 150.648 160.717 160.536 290.904 60.765 290.485 38
Gege Zhang, Qinghua Ma, Licheng Jiao, Fang Liu and Qigong Sun: AttAN: Attention Adversarial Networks for 3D Point Cloud Semantic Segmentation. IJCAI2020
LAP-D0.594 290.720 230.692 280.637 330.456 360.773 260.391 320.730 150.587 120.445 320.940 260.381 310.288 100.434 340.453 320.591 290.649 300.581 150.777 350.749 340.610 16
DPC0.592 300.720 230.700 240.602 360.480 330.762 290.380 340.713 200.585 130.437 330.940 260.369 330.288 100.434 340.509 290.590 300.639 340.567 180.772 360.755 320.592 22
Francis Engelmann, Theodora Kontogianni, Bastian Leibe: Dilated Point Convolutions: On the Receptive Field Size of Point Convolutions on 3D Point Clouds. ICRA 2020
CCRFNet0.589 310.766 160.659 360.683 240.470 350.740 330.387 330.620 330.490 310.476 280.922 360.355 360.245 180.511 240.511 280.571 320.643 320.493 350.872 200.762 300.600 19
SegGCNpermissive0.589 310.833 70.731 180.539 400.514 300.789 220.448 170.467 390.573 140.484 270.936 290.396 290.061 470.501 280.507 300.594 280.700 210.563 190.874 170.771 280.493 36
Huan Lei, Naveed Akhtar, and Ajmal Mian: SegGCN: Efficient 3D Point Cloud Segmentation with Fuzzy Spherical Kernel. CVPR 2020
TextureNetpermissive0.566 330.672 290.664 340.671 270.494 310.719 340.445 190.678 260.411 400.396 350.935 300.356 350.225 200.412 360.535 270.565 330.636 350.464 370.794 340.680 400.568 27
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
DVVNet0.562 340.648 300.700 240.770 130.586 210.687 380.333 360.650 290.514 270.475 290.906 420.359 340.223 210.340 390.442 340.422 420.668 280.501 330.708 400.779 260.534 32
Pointnet++ & Featurepermissive0.557 350.735 200.661 350.686 230.491 320.744 320.392 300.539 370.451 360.375 380.946 200.376 320.205 240.403 370.356 380.553 340.643 320.497 340.824 290.756 310.515 34
PanopticFusion-label0.529 360.491 430.688 290.604 350.386 390.632 430.225 470.705 210.434 380.293 420.815 460.348 370.241 190.499 290.669 210.507 350.649 300.442 410.796 330.602 460.561 29
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
3DMV, FTSDF0.501 370.558 400.608 420.424 470.478 340.690 370.246 430.586 340.468 340.450 310.911 400.394 300.160 360.438 330.212 440.432 410.541 420.475 360.742 380.727 360.477 39
PCNN0.498 380.559 390.644 390.560 390.420 380.711 360.229 450.414 400.436 370.352 390.941 240.324 380.155 370.238 440.387 370.493 360.529 430.509 320.813 320.751 330.504 35
3DMV0.484 390.484 440.538 450.643 320.424 370.606 460.310 370.574 350.433 390.378 370.796 470.301 390.214 230.537 230.208 450.472 400.507 460.413 440.693 410.602 460.539 31
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGBpermissive0.458 400.577 380.611 410.356 490.321 450.715 350.299 390.376 430.328 460.319 400.944 220.285 410.164 350.216 470.229 430.484 380.545 410.456 390.755 370.709 370.475 40
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPNpermissive0.447 410.679 280.604 430.578 380.380 400.682 390.291 400.106 490.483 320.258 470.920 370.258 430.025 490.231 460.325 390.480 390.560 400.463 380.725 390.666 420.231 49
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
PNET20.442 420.548 410.548 440.597 370.363 420.628 440.300 380.292 440.374 430.307 410.881 440.268 420.186 300.238 440.204 460.407 430.506 470.449 400.667 420.620 440.462 41
SurfaceConvPF0.442 420.505 420.622 400.380 480.342 440.654 410.227 460.397 420.367 440.276 440.924 350.240 450.198 270.359 380.262 410.366 440.581 380.435 420.640 430.668 410.398 42
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
Tangent Convolutionspermissive0.438 440.437 470.646 380.474 440.369 410.645 420.353 350.258 460.282 480.279 430.918 390.298 400.147 400.283 410.294 400.487 370.562 390.427 430.619 440.633 430.352 44
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
subcloud_weak0.411 450.479 450.650 370.475 430.285 480.519 490.087 500.725 170.396 420.386 360.621 500.250 440.117 410.338 400.443 330.188 500.594 370.369 470.377 500.616 450.306 45
SPLAT Netcopyleft0.393 460.472 460.511 460.606 340.311 460.656 400.245 440.405 410.328 460.197 480.927 340.227 470.000 510.001 510.249 420.271 490.510 440.383 460.593 450.699 380.267 47
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF0.383 470.297 490.491 470.432 460.358 430.612 450.274 410.116 480.411 400.265 450.904 430.229 460.079 450.250 420.185 470.320 470.510 440.385 450.548 460.597 480.394 43
PointNet++permissive0.339 480.584 370.478 480.458 450.256 490.360 500.250 420.247 470.278 490.261 460.677 490.183 480.117 410.212 480.145 490.364 450.346 500.232 500.548 460.523 490.252 48
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNetpermissive0.308 490.353 480.290 500.278 500.166 500.553 470.169 490.286 450.147 500.148 500.908 410.182 490.064 460.023 500.018 510.354 460.363 480.345 480.546 480.685 390.278 46
ScanNetpermissive0.306 500.203 500.366 490.501 410.311 460.524 480.211 480.002 510.342 450.189 490.786 480.145 500.102 440.245 430.152 480.318 480.348 490.300 490.460 490.437 500.182 50
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
ERROR0.054 510.000 510.041 510.172 510.030 510.062 510.001 510.035 500.004 510.051 510.143 510.019 510.003 500.041 490.050 500.003 510.054 510.018 510.005 510.264 510.082 51