The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).



This table lists the benchmark results for the 3D semantic label scenario.


Method Infoavg ioubathtubbedbookshelfcabinetchaircountercurtaindeskdoorfloorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwallwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
SparseConvNet0.725 10.647 80.821 10.846 10.721 10.869 10.533 10.754 40.603 40.614 10.955 10.572 10.325 10.710 20.870 20.724 10.823 10.628 30.934 10.865 10.683 1
MinkowskiNet0.721 20.837 20.804 20.800 20.721 10.843 20.460 30.835 10.647 10.597 20.953 20.542 20.214 80.746 10.912 10.705 20.771 30.640 20.876 50.842 20.672 2
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
KP-FCNN0.684 30.847 10.758 40.784 30.647 30.814 40.473 20.772 30.605 30.594 30.935 90.450 50.181 130.587 40.805 30.690 30.785 20.614 40.882 30.819 30.632 3
MVPNet0.641 40.831 30.715 50.671 50.590 60.781 50.394 80.679 70.642 20.553 50.937 80.462 40.256 30.649 30.406 100.626 50.691 50.666 10.877 40.792 60.608 4
joint point-based0.634 50.614 100.778 30.667 70.633 40.825 30.420 60.804 20.467 100.561 40.951 30.494 30.291 20.566 50.458 80.579 60.764 40.559 60.838 60.814 40.598 6
HPEIN0.618 60.729 40.668 90.647 80.597 50.766 60.414 70.680 60.520 50.525 60.946 40.432 60.215 70.493 80.599 50.638 40.617 110.570 50.897 20.806 50.605 5
TextureNet0.566 70.672 60.664 100.671 50.494 80.719 80.445 40.678 80.411 150.396 110.935 90.356 100.225 50.412 110.535 70.565 80.636 90.464 120.794 100.680 150.568 7
DVVNet0.562 80.648 70.700 60.770 40.586 70.687 120.333 100.650 90.514 60.475 70.906 180.359 90.223 60.340 130.442 90.422 170.668 60.501 90.708 150.779 70.534 10
PointConv0.556 90.636 90.640 130.574 150.472 100.739 70.430 50.433 120.418 140.445 90.944 50.372 80.185 120.464 90.575 60.540 90.639 80.505 80.827 70.762 80.515 11
PanopticFusion-label0.529 100.491 180.688 70.604 120.386 140.632 180.225 220.705 50.434 120.293 160.815 210.348 110.241 40.499 70.669 40.507 100.649 70.442 160.796 90.602 200.561 8
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. arXiv
LAP-D0.504 110.604 110.679 80.608 100.464 110.678 140.308 120.386 160.500 70.397 100.935 90.332 120.086 200.212 210.228 170.579 60.628 100.499 100.769 110.730 100.452 16
3DMV, FTSDF0.501 120.558 150.608 160.424 210.478 90.690 110.246 180.586 100.468 90.450 80.911 160.394 70.160 150.438 100.212 180.432 160.541 160.475 110.742 130.727 110.477 13
PCNN0.498 130.559 140.644 120.560 160.420 130.711 100.229 200.414 130.436 110.352 130.941 70.324 130.155 160.238 170.387 110.493 110.529 170.509 70.813 80.751 90.504 12
3DMV0.484 140.484 190.538 190.643 90.424 120.606 210.310 110.574 110.433 130.378 120.796 220.301 140.214 80.537 60.208 190.472 150.507 200.413 190.693 160.602 200.539 9
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGBpermissive0.458 150.577 130.611 150.356 230.321 200.715 90.299 140.376 170.328 200.319 140.944 50.285 160.164 140.216 200.229 160.484 130.545 150.456 140.755 120.709 120.475 14
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPNpermissive0.447 160.679 50.604 170.578 140.380 150.682 130.291 150.106 230.483 80.258 210.920 140.258 180.025 230.231 190.325 120.480 140.560 140.463 130.725 140.666 170.231 23
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
SurfaceConvPF0.442 170.505 170.622 140.380 220.342 190.654 160.227 210.397 150.367 180.276 180.924 130.240 190.198 100.359 120.262 140.366 190.581 120.435 170.640 180.668 160.398 17
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
PNET20.442 170.548 160.548 180.597 130.363 170.628 190.300 130.292 180.374 170.307 150.881 200.268 170.186 110.238 170.204 200.407 180.506 210.449 150.667 170.620 190.462 15
Tangent Convolutionspermissive0.438 190.437 210.646 110.474 180.369 160.645 170.353 90.258 200.282 220.279 170.918 150.298 150.147 170.283 140.294 130.487 120.562 130.427 180.619 190.633 180.352 19
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
SPLAT Netcopyleft0.393 200.472 200.511 200.606 110.311 210.656 150.245 190.405 140.328 200.197 220.927 120.227 210.000 240.001 240.249 150.271 240.510 180.383 210.593 200.699 130.267 21
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF0.383 210.297 230.491 210.432 200.358 180.612 200.274 160.116 220.411 150.265 190.904 190.229 200.079 210.250 150.185 210.320 220.510 180.385 200.548 210.597 220.394 18
PointNet++permissive0.339 220.584 120.478 220.458 190.256 230.360 240.250 170.247 210.278 230.261 200.677 240.183 220.117 180.212 210.145 230.364 200.346 240.232 240.548 210.523 230.252 22
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNetpermissive0.308 230.353 220.290 240.278 240.166 240.553 220.169 240.286 190.147 240.148 240.908 170.182 230.064 220.023 230.018 240.354 210.363 220.345 220.546 230.685 140.278 20
ScanNetpermissive0.306 240.203 240.366 230.501 170.311 210.524 230.211 230.002 240.342 190.189 230.786 230.145 240.102 190.245 160.152 220.318 230.348 230.300 230.460 240.437 240.182 24
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17