The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).



This table lists the benchmark results for the 3D semantic label scenario.


Method Infoavg ioubathtubbedbookshelfcabinetchaircountercurtaindeskdoorfloorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwallwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
MinkowskiNetpermissive0.736 10.859 20.818 20.832 20.709 20.840 20.521 20.853 10.660 10.643 10.951 30.544 20.286 60.731 10.893 10.675 40.772 30.683 10.874 70.852 20.727 1
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet0.725 20.647 180.821 10.846 10.721 10.869 10.533 10.754 50.603 40.614 20.955 10.572 10.325 10.710 20.870 20.724 10.823 10.628 30.934 10.865 10.683 2
KP-FCNN0.684 30.847 40.758 60.784 40.647 30.814 50.473 50.772 40.605 30.594 30.935 160.450 70.181 210.587 40.805 40.690 30.785 20.614 40.882 50.819 30.632 5
H. Thomas, C. Qi, J. Deschaud, B. Marcotegui, F. Goulette, L. Guibas.: KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV 2019
PointConvpermissive0.666 40.781 60.759 50.699 90.644 40.822 40.475 40.779 30.564 80.504 100.953 20.428 90.203 170.586 50.754 50.661 60.753 50.588 50.902 20.813 50.642 3
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
DMC-Net0.653 50.771 70.701 110.801 30.619 70.807 70.463 60.680 110.495 150.520 80.940 120.452 60.301 20.496 140.816 30.664 50.719 60.563 110.822 140.799 80.638 4
MVPNet0.641 60.831 50.715 100.671 120.590 100.781 90.394 140.679 130.642 20.553 50.937 150.462 50.256 80.649 30.406 190.626 90.691 90.666 20.877 60.792 100.608 9
Maximilian Jaritz, Jiayuan Gu, Hao Su: Multi-view PointNet for 3D Scene Understanding. GMDL Workshop, ICCV 2019
joint point-basedpermissive0.634 70.614 190.778 30.667 140.633 60.825 30.420 80.804 20.467 190.561 40.951 30.494 30.291 30.566 80.458 160.579 150.764 40.559 120.838 120.814 40.598 12
Hung-Yueh Chiang, Yen-Liang Lin, Yueh-Cheng Liu, Winston H. Hsu: A Unified Point-Based Framework for 3D Segmentation. 3DV 2019
MCCNNpermissive0.633 80.866 10.731 80.771 50.576 120.809 60.410 120.684 100.497 140.491 120.949 60.466 40.105 270.581 70.646 80.620 100.680 120.542 150.817 150.795 90.618 7
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
PointASNL0.630 90.738 100.729 90.764 80.637 50.779 100.416 100.626 170.518 110.530 60.951 30.398 130.260 70.518 100.576 120.590 130.687 100.568 80.872 80.810 60.631 6
CDF-SM3D0.626 100.592 200.746 70.767 70.607 80.761 150.501 30.738 60.546 90.503 110.864 290.421 100.198 180.584 60.579 110.694 20.706 70.566 100.885 40.745 190.523 19
HPEIN0.618 110.729 120.668 160.647 150.597 90.766 130.414 110.680 110.520 100.525 70.946 70.432 80.215 130.493 150.599 90.638 70.617 200.570 70.897 30.806 70.605 10
Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, Jiaya Jia: Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation. ICCV 2019
SPH3D-GCNpermissive0.610 120.858 30.772 40.489 260.532 140.792 80.404 130.643 160.570 70.507 90.935 160.414 110.046 310.510 120.702 60.602 110.705 80.549 130.859 100.773 130.534 17
Huan Lei, Naveed Akhtar, and Ajmal Mian: Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds.
LAP-D0.594 130.720 130.692 140.637 170.456 200.773 110.391 160.730 70.587 50.445 170.940 120.381 150.288 40.434 170.453 170.591 120.649 140.581 60.777 190.749 180.610 8
SIConv0.594 130.768 80.639 220.616 180.544 130.768 120.419 90.601 190.513 130.474 150.946 70.402 120.213 150.387 210.581 100.633 80.683 110.549 130.843 110.774 120.521 20
DPC0.592 150.720 130.700 120.602 210.480 170.762 140.380 180.713 80.585 60.437 180.940 120.369 170.288 40.434 170.509 150.590 130.639 180.567 90.772 200.755 160.592 13
Francis Engelmann, Theodora Kontogianni, Bastian Leibe: Dilated Point Convolutions. arXiv
CCRFNet0.589 160.766 90.659 190.683 110.470 190.740 170.387 170.620 180.490 160.476 130.922 210.355 200.245 90.511 110.511 140.571 160.643 160.493 190.872 80.762 140.600 11
TextureNetpermissive0.566 170.672 160.664 170.671 120.494 150.719 180.445 70.678 140.411 240.396 190.935 160.356 190.225 110.412 190.535 130.565 170.636 190.464 210.794 180.680 240.568 14
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
DVVNet0.562 180.648 170.700 120.770 60.586 110.687 220.333 200.650 150.514 120.475 140.906 260.359 180.223 120.340 230.442 180.422 260.668 130.501 170.708 240.779 110.534 17
Pointnet++ & Featurepermissive0.557 190.735 110.661 180.686 100.491 160.744 160.392 150.539 220.451 200.375 210.946 70.376 160.205 160.403 200.356 210.553 180.643 160.497 180.824 130.756 150.515 21
PanopticFusion-label0.529 200.491 270.688 150.604 200.386 230.632 270.225 310.705 90.434 220.293 250.815 300.348 210.241 100.499 130.669 70.507 190.649 140.442 250.796 170.602 290.561 15
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
3DMV, FTSDF0.501 210.558 240.608 250.424 300.478 180.690 210.246 270.586 200.468 180.450 160.911 240.394 140.160 230.438 160.212 270.432 250.541 250.475 200.742 220.727 200.477 23
PCNN0.498 220.559 230.644 210.560 240.420 220.711 200.229 290.414 230.436 210.352 220.941 110.324 220.155 240.238 270.387 200.493 200.529 260.509 160.813 160.751 170.504 22
3DMV0.484 230.484 280.538 280.643 160.424 210.606 300.310 210.574 210.433 230.378 200.796 310.301 230.214 140.537 90.208 280.472 240.507 290.413 280.693 250.602 290.539 16
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGBpermissive0.458 240.577 220.611 240.356 320.321 290.715 190.299 230.376 260.328 290.319 230.944 100.285 250.164 220.216 300.229 260.484 220.545 240.456 230.755 210.709 210.475 24
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPNpermissive0.447 250.679 150.604 260.578 230.380 240.682 230.291 240.106 320.483 170.258 300.920 220.258 270.025 320.231 290.325 220.480 230.560 230.463 220.725 230.666 260.231 32
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
SurfaceConvPF0.442 260.505 260.622 230.380 310.342 280.654 250.227 300.397 250.367 270.276 270.924 200.240 280.198 180.359 220.262 240.366 280.581 210.435 260.640 270.668 250.398 26
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
PNET20.442 260.548 250.548 270.597 220.363 260.628 280.300 220.292 270.374 260.307 240.881 280.268 260.186 200.238 270.204 290.407 270.506 300.449 240.667 260.620 280.462 25
Tangent Convolutionspermissive0.438 280.437 300.646 200.474 270.369 250.645 260.353 190.258 290.282 310.279 260.918 230.298 240.147 250.283 240.294 230.487 210.562 220.427 270.619 280.633 270.352 28
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
SPLAT Netcopyleft0.393 290.472 290.511 290.606 190.311 300.656 240.245 280.405 240.328 290.197 310.927 190.227 300.000 340.001 340.249 250.271 330.510 270.383 300.593 290.699 220.267 30
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF0.383 300.297 320.491 300.432 290.358 270.612 290.274 250.116 310.411 240.265 280.904 270.229 290.079 290.250 250.185 300.320 310.510 270.385 290.548 300.597 310.394 27
PointNet++permissive0.339 310.584 210.478 310.458 280.256 320.360 330.250 260.247 300.278 320.261 290.677 330.183 310.117 260.212 310.145 320.364 290.346 330.232 330.548 300.523 320.252 31
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNetpermissive0.308 320.353 310.290 330.278 330.166 330.553 310.169 330.286 280.147 330.148 330.908 250.182 320.064 300.023 330.018 340.354 300.363 310.345 310.546 320.685 230.278 29
ScanNetpermissive0.306 330.203 330.366 320.501 250.311 300.524 320.211 320.002 340.342 280.189 320.786 320.145 330.102 280.245 260.152 310.318 320.348 320.300 320.460 330.437 330.182 33
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
ERROR0.054 340.000 340.041 340.172 340.030 340.062 340.001 340.035 330.004 340.051 340.143 340.019 340.003 330.041 320.050 330.003 340.054 340.018 340.005 340.264 340.082 34