The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).



This table lists the benchmark results for the 3D semantic label scenario.


Method Infoavg ioubathtubbedbookshelfcabinetchaircountercurtaindeskdoorfloorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwallwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
OccuSeg+Semantic0.764 10.758 160.796 40.839 20.746 10.907 10.562 10.850 30.680 10.672 10.978 10.610 10.335 10.777 10.819 70.847 10.830 10.691 10.972 10.885 10.727 1
MinkowskiNetpermissive0.736 20.859 30.818 20.832 30.709 40.840 30.521 40.853 20.660 20.643 20.951 60.544 30.286 90.731 30.893 10.675 90.772 60.683 20.874 140.852 30.727 1
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet0.725 30.647 250.821 10.846 10.721 30.869 20.533 20.754 100.603 60.614 30.955 20.572 20.325 20.710 40.870 20.724 40.823 20.628 50.934 30.865 20.683 5
MatchingNet0.724 40.812 100.812 30.810 50.735 20.834 40.495 70.860 10.572 120.602 50.954 30.512 40.280 100.757 20.845 50.725 30.780 50.606 80.937 20.851 40.700 4
JSENet0.699 50.881 10.762 90.821 40.667 60.800 110.522 30.792 60.613 40.607 40.935 230.492 70.205 200.576 120.853 30.691 60.758 80.652 40.872 160.828 60.649 7
CU-Hybrid Net0.693 60.596 290.789 50.803 70.677 50.800 110.469 110.846 40.554 160.591 70.948 120.500 50.316 40.609 70.847 40.732 20.808 30.593 90.894 80.839 50.652 6
KP-FCNN0.684 70.847 50.758 120.784 90.647 80.814 80.473 90.772 80.605 50.594 60.935 230.450 130.181 270.587 90.805 80.690 70.785 40.614 60.882 100.819 70.632 10
H. Thomas, C. Qi, J. Deschaud, B. Marcotegui, F. Goulette, L. Guibas.: KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV 2019
PointConvpermissive0.666 80.781 120.759 110.699 170.644 90.822 70.475 80.779 70.564 150.504 180.953 40.428 190.203 220.586 100.754 90.661 120.753 90.588 100.902 40.813 90.642 8
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
PointASNLpermissive0.666 80.703 210.781 60.751 150.655 70.830 50.471 100.769 90.474 260.537 100.951 60.475 80.279 110.635 60.698 130.675 90.751 100.553 190.816 240.806 100.703 3
Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, Shuguang Cui: PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Sampling. CVPR 2020
DCM-Net0.658 100.778 130.702 180.806 60.619 110.813 90.468 120.693 180.494 230.524 140.941 170.449 140.298 50.510 180.821 60.675 90.727 110.568 130.826 210.803 120.637 9
Jonas Schult*, Francis Engelmann*, Theodora Kontogianni, Bastian Leibe: DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes. CVPR 2020 [Oral]
MVPNetpermissive0.641 110.831 70.715 170.671 200.590 150.781 170.394 220.679 210.642 30.553 90.937 210.462 100.256 120.649 50.406 290.626 170.691 170.666 30.877 110.792 170.608 14
Maximilian Jaritz, Jiayuan Gu, Hao Su: Multi-view PointNet for 3D Scene Understanding. GMDL Workshop, ICCV 2019
FPConvpermissive0.639 120.785 110.760 100.713 160.603 130.798 130.392 230.534 310.603 60.524 140.948 120.457 110.250 130.538 150.723 110.598 200.696 160.614 60.872 160.799 130.567 22
Yiqun Lin, Zizheng Yan, Haibin Huang, Dong Du, Ligang Liu, Shuguang Cui, Xiaoguang Han: FPConv: Learning Local Flattening for Point Convolution. CVPR 2020
SConv0.636 130.830 80.697 210.752 140.572 190.780 180.445 150.716 150.529 170.530 110.951 60.446 150.170 280.507 200.666 150.636 160.682 190.541 230.886 90.799 130.594 18
joint point-basedpermissive0.634 140.614 270.778 70.667 220.633 100.825 60.420 180.804 50.467 280.561 80.951 60.494 60.291 60.566 130.458 240.579 240.764 70.559 180.838 200.814 80.598 17
Hung-Yueh Chiang, Yen-Liang Lin, Yueh-Cheng Liu, Winston H. Hsu: A Unified Point-Based Framework for 3D Segmentation. 3DV 2019
MCCNNpermissive0.633 150.866 20.731 150.771 100.576 180.809 100.410 200.684 190.497 220.491 190.949 100.466 90.105 360.581 110.646 160.620 180.680 200.542 220.817 230.795 150.618 12
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
3DSM_DMMF0.631 160.626 260.745 130.801 80.607 120.751 230.506 50.729 130.565 140.491 190.866 380.434 160.197 240.595 80.630 170.709 50.705 130.560 170.875 130.740 280.491 31
FusionAwareConv0.630 170.604 280.741 140.766 120.590 150.747 240.501 60.734 110.503 210.527 120.919 310.454 120.323 30.550 140.420 280.678 80.688 180.544 210.896 70.795 150.627 11
Jiazhao Zhang, Chenyang Zhu, Lintao Zheng, Kai Xu: Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentation. CVPR 2020
SIConv0.625 180.830 80.694 230.757 130.563 200.772 200.448 130.647 250.520 180.509 160.949 100.431 180.191 250.496 230.614 180.647 130.672 210.535 240.876 120.783 180.571 20
FRPointConv0.619 190.762 150.697 210.602 280.547 210.782 160.432 170.650 230.581 100.458 240.952 50.428 190.130 330.496 230.747 100.641 140.710 120.568 130.902 40.782 190.511 28
HPEIN0.618 200.729 180.668 260.647 230.597 140.766 210.414 190.680 200.520 180.525 130.946 140.432 170.215 180.493 250.599 190.638 150.617 290.570 120.897 60.806 100.605 15
Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, Jiaya Jia: Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation. ICCV 2019
SPH3D-GCNpermissive0.610 210.858 40.772 80.489 350.532 220.792 140.404 210.643 260.570 130.507 170.935 230.414 210.046 410.510 180.702 120.602 190.705 130.549 200.859 190.773 210.534 25
Huan Lei, Naveed Akhtar, and Ajmal Mian: Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds. TPAMI 2020
LAP-D0.594 220.720 190.692 240.637 250.456 290.773 190.391 250.730 120.587 80.445 260.940 190.381 240.288 70.434 270.453 250.591 220.649 230.581 110.777 280.749 270.610 13
DPC0.592 230.720 190.700 190.602 280.480 260.762 220.380 270.713 160.585 90.437 270.940 190.369 260.288 70.434 270.509 220.590 230.639 270.567 150.772 290.755 250.592 19
Francis Engelmann, Theodora Kontogianni, Bastian Leibe: Dilated Point Convolutions: On the Receptive Field Size of Point Convolutions on 3D Point Clouds. ICRA 2020
SegGCNpermissive0.589 240.833 60.731 150.539 330.514 230.789 150.448 130.467 320.573 110.484 210.936 220.396 220.061 400.501 210.507 230.594 210.700 150.563 160.874 140.771 220.493 30
Huan Lei, Naveed Akhtar, and Ajmal Mian: SegGCN: Efficient 3D Point Cloud Segmentation with Fuzzy Spherical Kernel. CVPR 2020
CCRFNet0.589 240.766 140.659 290.683 190.470 280.740 260.387 260.620 270.490 240.476 220.922 290.355 290.245 140.511 170.511 210.571 250.643 250.493 280.872 160.762 230.600 16
TextureNetpermissive0.566 260.672 230.664 270.671 200.494 240.719 270.445 150.678 220.411 330.396 280.935 230.356 280.225 160.412 290.535 200.565 260.636 280.464 300.794 270.680 330.568 21
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
DVVNet0.562 270.648 240.700 190.770 110.586 170.687 310.333 290.650 230.514 200.475 230.906 350.359 270.223 170.340 320.442 270.422 350.668 220.501 260.708 330.779 200.534 25
Pointnet++ & Featurepermissive0.557 280.735 170.661 280.686 180.491 250.744 250.392 230.539 300.451 290.375 310.946 140.376 250.205 200.403 300.356 310.553 270.643 250.497 270.824 220.756 240.515 27
PanopticFusion-label0.529 290.491 360.688 250.604 270.386 320.632 360.225 400.705 170.434 310.293 350.815 390.348 300.241 150.499 220.669 140.507 280.649 230.442 340.796 260.602 390.561 23
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
3DMV, FTSDF0.501 300.558 330.608 350.424 400.478 270.690 300.246 360.586 280.468 270.450 250.911 330.394 230.160 300.438 260.212 370.432 340.541 350.475 290.742 310.727 290.477 32
PCNN0.498 310.559 320.644 320.560 320.420 310.711 290.229 380.414 330.436 300.352 320.941 170.324 310.155 310.238 370.387 300.493 290.529 360.509 250.813 250.751 260.504 29
3DMV0.484 320.484 370.538 380.643 240.424 300.606 390.310 300.574 290.433 320.378 300.796 400.301 320.214 190.537 160.208 380.472 330.507 390.413 370.693 340.602 390.539 24
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGBpermissive0.458 330.577 310.611 340.356 420.321 380.715 280.299 320.376 360.328 390.319 330.944 160.285 340.164 290.216 400.229 360.484 310.545 340.456 320.755 300.709 300.475 33
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPNpermissive0.447 340.679 220.604 360.578 310.380 330.682 320.291 330.106 420.483 250.258 400.920 300.258 360.025 420.231 390.325 320.480 320.560 330.463 310.725 320.666 350.231 42
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
SurfaceConvPF0.442 350.505 350.622 330.380 410.342 370.654 340.227 390.397 350.367 370.276 370.924 280.240 380.198 230.359 310.262 340.366 370.581 310.435 350.640 360.668 340.398 35
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
PNET20.442 350.548 340.548 370.597 300.363 350.628 370.300 310.292 370.374 360.307 340.881 370.268 350.186 260.238 370.204 390.407 360.506 400.449 330.667 350.620 370.462 34
Tangent Convolutionspermissive0.438 370.437 400.646 310.474 370.369 340.645 350.353 280.258 390.282 410.279 360.918 320.298 330.147 320.283 340.294 330.487 300.562 320.427 360.619 370.633 360.352 37
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
subcloud_weak0.411 380.479 380.650 300.475 360.285 410.519 420.087 430.725 140.396 350.386 290.621 430.250 370.117 340.338 330.443 260.188 430.594 300.369 400.377 430.616 380.306 38
SPLAT Netcopyleft0.393 390.472 390.511 390.606 260.311 390.656 330.245 370.405 340.328 390.197 410.927 270.227 400.000 440.001 440.249 350.271 420.510 370.383 390.593 380.699 310.267 40
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF0.383 400.297 420.491 400.432 390.358 360.612 380.274 340.116 410.411 330.265 380.904 360.229 390.079 380.250 350.185 400.320 400.510 370.385 380.548 390.597 410.394 36
PointNet++permissive0.339 410.584 300.478 410.458 380.256 420.360 430.250 350.247 400.278 420.261 390.677 420.183 410.117 340.212 410.145 420.364 380.346 430.232 430.548 390.523 420.252 41
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNetpermissive0.308 420.353 410.290 430.278 430.166 430.553 400.169 420.286 380.147 430.148 430.908 340.182 420.064 390.023 430.018 440.354 390.363 410.345 410.546 410.685 320.278 39
ScanNetpermissive0.306 430.203 430.366 420.501 340.311 390.524 410.211 410.002 440.342 380.189 420.786 410.145 430.102 370.245 360.152 410.318 410.348 420.300 420.460 420.437 430.182 43
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
ERROR0.054 440.000 440.041 440.172 440.030 440.062 440.001 440.035 430.004 440.051 440.143 440.019 440.003 430.041 420.050 430.003 440.054 440.018 440.005 440.264 440.082 44