The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).



This table lists the benchmark results for the 3D semantic label scenario.


Method Infoavg ioubathtubbedbookshelfcabinetchaircountercurtaindeskdoorfloorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwallwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
MinkowskiNetpermissive0.736 10.859 20.818 20.832 20.709 20.840 20.521 20.853 10.660 10.643 10.951 30.544 20.286 70.731 10.893 10.675 50.772 40.683 10.874 80.852 20.727 1
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet0.725 20.647 180.821 10.846 10.721 10.869 10.533 10.754 60.603 40.614 20.955 10.572 10.325 10.710 20.870 20.724 20.823 10.628 30.934 10.865 10.683 2
CU-Hybrid Net0.693 30.596 210.789 30.803 30.677 30.800 80.469 60.846 20.554 100.591 40.948 70.500 30.316 20.609 40.847 30.732 10.808 20.593 50.894 40.839 30.652 3
KP-FCNN0.684 40.847 40.758 70.784 60.647 40.814 50.473 50.772 50.605 30.594 30.935 170.450 80.181 220.587 60.805 50.690 40.785 30.614 40.882 50.819 40.632 6
H. Thomas, C. Qi, J. Deschaud, B. Marcotegui, F. Goulette, L. Guibas.: KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV 2019
PointConvpermissive0.666 50.781 60.759 60.699 100.644 50.822 40.475 40.779 40.564 90.504 110.953 20.428 110.203 180.586 70.754 60.661 70.753 60.588 60.902 20.813 60.642 4
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
DMC-Net0.653 60.771 70.701 120.801 40.619 80.807 70.463 70.680 120.495 160.520 90.940 130.452 70.301 30.496 150.816 40.664 60.719 70.563 110.822 150.799 90.638 5
MVPNetpermissive0.641 70.831 50.715 110.671 130.590 110.781 100.394 150.679 140.642 20.553 60.937 160.462 60.256 90.649 30.406 200.626 100.691 100.666 20.877 60.792 110.608 10
Maximilian Jaritz, Jiayuan Gu, Hao Su: Multi-view PointNet for 3D Scene Understanding. GMDL Workshop, ICCV 2019
joint point-basedpermissive0.634 80.614 200.778 40.667 150.633 70.825 30.420 90.804 30.467 200.561 50.951 30.494 40.291 40.566 90.458 170.579 160.764 50.559 130.838 130.814 50.598 13
Hung-Yueh Chiang, Yen-Liang Lin, Yueh-Cheng Liu, Winston H. Hsu: A Unified Point-Based Framework for 3D Segmentation. 3DV 2019
MCCNNpermissive0.633 90.866 10.731 90.771 70.576 130.809 60.410 130.684 110.497 150.491 120.949 60.466 50.105 280.581 80.646 90.620 110.680 130.542 160.817 160.795 100.618 8
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
3DSM_DMMF0.631 100.626 190.745 80.801 40.607 90.751 160.506 30.729 80.565 80.491 120.866 300.434 90.197 200.595 50.630 100.709 30.705 80.560 120.875 70.740 200.491 23
PointASNL0.630 110.738 100.729 100.764 90.637 60.779 110.416 110.626 180.518 120.530 70.951 30.398 140.260 80.518 110.576 130.590 140.687 110.568 90.872 90.810 70.631 7
HPEIN0.618 120.729 120.668 170.647 160.597 100.766 140.414 120.680 120.520 110.525 80.946 80.432 100.215 140.493 160.599 110.638 80.617 210.570 80.897 30.806 80.605 11
Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, Jiaya Jia: Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation. ICCV 2019
SPH3D-GCNpermissive0.610 130.858 30.772 50.489 270.532 150.792 90.404 140.643 170.570 70.507 100.935 170.414 120.046 320.510 130.702 70.602 120.705 80.549 140.859 110.773 140.534 18
Huan Lei, Naveed Akhtar, and Ajmal Mian: Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds.
SIConv0.594 140.768 80.639 230.616 190.544 140.768 130.419 100.601 200.513 140.474 160.946 80.402 130.213 160.387 220.581 120.633 90.683 120.549 140.843 120.774 130.521 20
LAP-D0.594 140.720 130.692 150.637 180.456 210.773 120.391 170.730 70.587 50.445 180.940 130.381 160.288 50.434 180.453 180.591 130.649 150.581 70.777 200.749 190.610 9
DPC0.592 160.720 130.700 130.602 220.480 180.762 150.380 190.713 90.585 60.437 190.940 130.369 180.288 50.434 180.509 160.590 140.639 190.567 100.772 210.755 170.592 14
Francis Engelmann, Theodora Kontogianni, Bastian Leibe: Dilated Point Convolutions. arXiv
CCRFNet0.589 170.766 90.659 200.683 120.470 200.740 180.387 180.620 190.490 170.476 140.922 220.355 210.245 100.511 120.511 150.571 170.643 170.493 200.872 90.762 150.600 12
TextureNetpermissive0.566 180.672 160.664 180.671 130.494 160.719 190.445 80.678 150.411 250.396 200.935 170.356 200.225 120.412 200.535 140.565 180.636 200.464 220.794 190.680 250.568 15
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
DVVNet0.562 190.648 170.700 130.770 80.586 120.687 230.333 210.650 160.514 130.475 150.906 270.359 190.223 130.340 240.442 190.422 270.668 140.501 180.708 250.779 120.534 18
Pointnet++ & Featurepermissive0.557 200.735 110.661 190.686 110.491 170.744 170.392 160.539 230.451 210.375 220.946 80.376 170.205 170.403 210.356 220.553 190.643 170.497 190.824 140.756 160.515 21
PanopticFusion-label0.529 210.491 280.688 160.604 210.386 240.632 280.225 320.705 100.434 230.293 260.815 310.348 220.241 110.499 140.669 80.507 200.649 150.442 260.796 180.602 300.561 16
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
3DMV, FTSDF0.501 220.558 250.608 260.424 310.478 190.690 220.246 280.586 210.468 190.450 170.911 250.394 150.160 240.438 170.212 280.432 260.541 260.475 210.742 230.727 210.477 24
PCNN0.498 230.559 240.644 220.560 250.420 230.711 210.229 300.414 240.436 220.352 230.941 120.324 230.155 250.238 280.387 210.493 210.529 270.509 170.813 170.751 180.504 22
3DMV0.484 240.484 290.538 290.643 170.424 220.606 310.310 220.574 220.433 240.378 210.796 320.301 240.214 150.537 100.208 290.472 250.507 300.413 290.693 260.602 300.539 17
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGBpermissive0.458 250.577 230.611 250.356 330.321 300.715 200.299 240.376 270.328 300.319 240.944 110.285 260.164 230.216 310.229 270.484 230.545 250.456 240.755 220.709 220.475 25
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPNpermissive0.447 260.679 150.604 270.578 240.380 250.682 240.291 250.106 330.483 180.258 310.920 230.258 280.025 330.231 300.325 230.480 240.560 240.463 230.725 240.666 270.231 33
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
SurfaceConvPF0.442 270.505 270.622 240.380 320.342 290.654 260.227 310.397 260.367 280.276 280.924 210.240 290.198 190.359 230.262 250.366 290.581 220.435 270.640 280.668 260.398 27
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
PNET20.442 270.548 260.548 280.597 230.363 270.628 290.300 230.292 280.374 270.307 250.881 290.268 270.186 210.238 280.204 300.407 280.506 310.449 250.667 270.620 290.462 26
Tangent Convolutionspermissive0.438 290.437 310.646 210.474 280.369 260.645 270.353 200.258 300.282 320.279 270.918 240.298 250.147 260.283 250.294 240.487 220.562 230.427 280.619 290.633 280.352 29
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
SPLAT Netcopyleft0.393 300.472 300.511 300.606 200.311 310.656 250.245 290.405 250.328 300.197 320.927 200.227 310.000 350.001 350.249 260.271 340.510 280.383 310.593 300.699 230.267 31
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF0.383 310.297 330.491 310.432 300.358 280.612 300.274 260.116 320.411 250.265 290.904 280.229 300.079 300.250 260.185 310.320 320.510 280.385 300.548 310.597 320.394 28
PointNet++permissive0.339 320.584 220.478 320.458 290.256 330.360 340.250 270.247 310.278 330.261 300.677 340.183 320.117 270.212 320.145 330.364 300.346 340.232 340.548 310.523 330.252 32
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNetpermissive0.308 330.353 320.290 340.278 340.166 340.553 320.169 340.286 290.147 340.148 340.908 260.182 330.064 310.023 340.018 350.354 310.363 320.345 320.546 330.685 240.278 30
ScanNetpermissive0.306 340.203 340.366 330.501 260.311 310.524 330.211 330.002 350.342 290.189 330.786 330.145 340.102 290.245 270.152 320.318 330.348 330.300 330.460 340.437 340.182 34
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
ERROR0.054 350.000 350.041 350.172 350.030 350.062 350.001 350.035 340.004 350.051 350.143 350.019 350.003 340.041 330.050 340.003 350.054 350.018 350.005 350.264 350.082 35