The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).



This table lists the benchmark results for the 3D semantic label scenario.


Method Infoavg ioubathtubbedbookshelfcabinetchaircountercurtaindeskdoorfloorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwallwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
MinkowskiNetpermissive0.734 10.858 20.833 10.834 20.716 20.855 20.459 30.836 10.639 20.641 10.953 20.541 20.302 20.743 10.865 20.726 10.771 30.664 20.891 30.851 20.694 1
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet0.725 20.647 120.821 20.846 10.721 10.869 10.533 10.754 40.603 40.614 20.955 10.572 10.325 10.710 20.870 10.724 20.823 10.628 30.934 10.865 10.683 2
KP-FCNN0.684 30.847 30.758 40.784 30.647 30.814 40.473 20.772 30.605 30.594 30.935 120.450 60.181 160.587 40.805 30.690 30.785 20.614 40.882 40.819 30.632 3
MVPNet0.641 40.831 40.715 60.671 70.590 60.781 60.394 90.679 100.642 10.553 50.937 110.462 50.256 60.649 30.406 140.626 50.691 50.666 10.877 50.792 70.608 6
joint point-based0.634 50.614 140.778 30.667 90.633 40.825 30.420 60.804 20.467 130.561 40.951 30.494 30.291 30.566 60.458 110.579 90.764 40.559 80.838 70.814 40.598 9
MCCNNpermissive0.633 60.866 10.731 50.771 40.576 80.809 50.410 80.684 80.497 90.491 70.949 40.466 40.105 220.581 50.646 50.620 60.680 60.542 90.817 90.795 60.618 4
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
HPEIN0.618 70.729 60.668 110.647 100.597 50.766 80.414 70.680 90.520 70.525 60.946 50.432 70.215 110.493 100.599 60.638 40.617 140.570 60.897 20.806 50.605 7
LAP-D0.594 80.720 70.692 90.637 120.456 140.773 70.391 100.730 50.587 50.445 110.940 90.381 90.288 40.434 130.453 120.591 70.649 80.581 50.777 130.749 130.610 5
DPC0.592 90.720 70.700 70.602 150.480 100.762 90.380 120.713 60.585 60.437 130.940 90.369 110.288 40.434 130.509 100.590 80.639 110.567 70.772 140.755 110.592 10
CCRFNet0.589 100.766 50.659 130.683 60.470 130.740 100.387 110.620 130.490 100.476 80.922 160.355 140.245 70.511 80.511 90.571 100.643 100.493 130.872 60.762 90.600 8
TextureNetpermissive0.566 110.672 100.664 120.671 70.494 90.719 120.445 40.678 110.411 180.396 140.935 120.356 130.225 90.412 150.535 80.565 110.636 130.464 150.794 120.680 180.568 11
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
DVVNet0.562 120.648 110.700 70.770 50.586 70.687 160.333 140.650 120.514 80.475 90.906 210.359 120.223 100.340 170.442 130.422 200.668 70.501 120.708 180.779 80.534 14
PointConv0.556 130.636 130.640 160.574 180.472 120.739 110.430 50.433 160.418 170.445 110.944 60.372 100.185 150.464 110.575 70.540 120.639 110.505 110.827 80.762 90.515 15
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
PanopticFusion-label0.529 140.491 210.688 100.604 140.386 170.632 210.225 250.705 70.434 150.293 190.815 240.348 150.241 80.499 90.669 40.507 130.649 80.442 190.796 110.602 230.561 12
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. arXiv
3DMV, FTSDF0.501 150.558 180.608 190.424 240.478 110.690 150.246 210.586 140.468 120.450 100.911 190.394 80.160 180.438 120.212 210.432 190.541 190.475 140.742 160.727 140.477 17
PCNN0.498 160.559 170.644 150.560 190.420 160.711 140.229 230.414 170.436 140.352 160.941 80.324 160.155 190.238 210.387 150.493 140.529 200.509 100.813 100.751 120.504 16
3DMV0.484 170.484 220.538 220.643 110.424 150.606 240.310 150.574 150.433 160.378 150.796 250.301 170.214 120.537 70.208 220.472 180.507 230.413 220.693 190.602 230.539 13
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGBpermissive0.458 180.577 160.611 180.356 260.321 230.715 130.299 170.376 200.328 230.319 170.944 60.285 190.164 170.216 240.229 200.484 160.545 180.456 170.755 150.709 150.475 18
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPNpermissive0.447 190.679 90.604 200.578 170.380 180.682 170.291 180.106 260.483 110.258 240.920 170.258 210.025 260.231 230.325 160.480 170.560 170.463 160.725 170.666 200.231 26
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
PNET20.442 200.548 190.548 210.597 160.363 200.628 220.300 160.292 210.374 200.307 180.881 230.268 200.186 140.238 210.204 230.407 210.506 240.449 180.667 200.620 220.462 19
SurfaceConvPF0.442 200.505 200.622 170.380 250.342 220.654 190.227 240.397 190.367 210.276 210.924 150.240 220.198 130.359 160.262 180.366 220.581 150.435 200.640 210.668 190.398 20
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
Tangent Convolutionspermissive0.438 220.437 240.646 140.474 210.369 190.645 200.353 130.258 230.282 250.279 200.918 180.298 180.147 200.283 180.294 170.487 150.562 160.427 210.619 220.633 210.352 22
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
SPLAT Netcopyleft0.393 230.472 230.511 230.606 130.311 240.656 180.245 220.405 180.328 230.197 250.927 140.227 240.000 280.001 280.249 190.271 270.510 210.383 240.593 230.699 160.267 24
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF0.383 240.297 260.491 240.432 230.358 210.612 230.274 190.116 250.411 180.265 220.904 220.229 230.079 240.250 190.185 240.320 250.510 210.385 230.548 240.597 250.394 21
PointNet++permissive0.339 250.584 150.478 250.458 220.256 260.360 270.250 200.247 240.278 260.261 230.677 270.183 250.117 210.212 250.145 260.364 230.346 270.232 270.548 240.523 260.252 25
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNetpermissive0.308 260.353 250.290 270.278 270.166 270.553 250.169 270.286 220.147 270.148 270.908 200.182 260.064 250.023 270.018 280.354 240.363 250.345 250.546 260.685 170.278 23
ScanNetpermissive0.306 270.203 270.366 260.501 200.311 240.524 260.211 260.002 280.342 220.189 260.786 260.145 270.102 230.245 200.152 250.318 260.348 260.300 260.460 270.437 270.182 27
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
ERROR0.054 280.000 280.041 280.172 280.030 280.062 280.001 280.035 270.004 280.051 280.143 280.019 280.003 270.041 260.050 270.003 280.054 280.018 280.005 280.264 280.082 28