3D Semantic Label Results - ScanNet Benchmark

The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).

This table lists the benchmark results for the 3D semantic label scenario.

Method	avg iou	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	floor	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	wall	window

MinkowskiNet	0.734 1	0.858 2	0.833 1	0.834 2	0.716 2	0.855 2	0.459 3	0.836 1	0.639 2	0.641 1	0.953 2	0.541 2	0.302 2	0.743 1	0.865 2	0.726 1	0.771 3	0.664 2	0.891 3	0.851 2	0.694 1
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet	0.725 2	0.647 12	0.821 2	0.846 1	0.721 1	0.869 1	0.533 1	0.754 4	0.603 4	0.614 2	0.955 1	0.572 1	0.325 1	0.710 2	0.870 1	0.724 2	0.823 1	0.628 3	0.934 1	0.865 1	0.683 2

KP-FCNN	0.684 3	0.847 3	0.758 4	0.784 3	0.647 3	0.814 4	0.473 2	0.772 3	0.605 3	0.594 3	0.935 12	0.450 6	0.181 16	0.587 4	0.805 3	0.690 3	0.785 2	0.614 4	0.882 4	0.819 3	0.632 3

MVPNet	0.641 4	0.831 4	0.715 6	0.671 7	0.590 6	0.781 6	0.394 9	0.679 10	0.642 1	0.553 5	0.937 11	0.462 5	0.256 6	0.649 3	0.406 14	0.626 5	0.691 5	0.666 1	0.877 5	0.792 7	0.608 6

joint point-based	0.634 5	0.614 14	0.778 3	0.667 9	0.633 4	0.825 3	0.420 6	0.804 2	0.467 13	0.561 4	0.951 3	0.494 3	0.291 3	0.566 6	0.458 11	0.579 9	0.764 4	0.559 8	0.838 7	0.814 4	0.598 9

MCCNN	0.633 6	0.866 1	0.731 5	0.771 4	0.576 8	0.809 5	0.410 8	0.684 8	0.497 9	0.491 7	0.949 4	0.466 4	0.105 22	0.581 5	0.646 5	0.620 6	0.680 6	0.542 9	0.817 9	0.795 6	0.618 4
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
HPEIN	0.618 7	0.729 6	0.668 11	0.647 10	0.597 5	0.766 8	0.414 7	0.680 9	0.520 7	0.525 6	0.946 5	0.432 7	0.215 11	0.493 10	0.599 6	0.638 4	0.617 14	0.570 6	0.897 2	0.806 5	0.605 7

LAP-D	0.594 8	0.720 7	0.692 9	0.637 12	0.456 14	0.773 7	0.391 10	0.730 5	0.587 5	0.445 11	0.940 9	0.381 9	0.288 4	0.434 13	0.453 12	0.591 7	0.649 8	0.581 5	0.777 13	0.749 13	0.610 5

DPC	0.592 9	0.720 7	0.700 7	0.602 15	0.480 10	0.762 9	0.380 12	0.713 6	0.585 6	0.437 13	0.940 9	0.369 11	0.288 4	0.434 13	0.509 10	0.590 8	0.639 11	0.567 7	0.772 14	0.755 11	0.592 10

CCRFNet	0.589 10	0.766 5	0.659 13	0.683 6	0.470 13	0.740 10	0.387 11	0.620 13	0.490 10	0.476 8	0.922 16	0.355 14	0.245 7	0.511 8	0.511 9	0.571 10	0.643 10	0.493 13	0.872 6	0.762 9	0.600 8

TextureNet	0.566 11	0.672 10	0.664 12	0.671 7	0.494 9	0.719 12	0.445 4	0.678 11	0.411 18	0.396 14	0.935 12	0.356 13	0.225 9	0.412 15	0.535 8	0.565 11	0.636 13	0.464 15	0.794 12	0.680 18	0.568 11
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
DVVNet	0.562 12	0.648 11	0.700 7	0.770 5	0.586 7	0.687 16	0.333 14	0.650 12	0.514 8	0.475 9	0.906 21	0.359 12	0.223 10	0.340 17	0.442 13	0.422 20	0.668 7	0.501 12	0.708 18	0.779 8	0.534 14

PointConv	0.556 13	0.636 13	0.640 16	0.574 18	0.472 12	0.739 11	0.430 5	0.433 16	0.418 17	0.445 11	0.944 6	0.372 10	0.185 15	0.464 11	0.575 7	0.540 12	0.639 11	0.505 11	0.827 8	0.762 9	0.515 15
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
PanopticFusion-label	0.529 14	0.491 21	0.688 10	0.604 14	0.386 17	0.632 21	0.225 25	0.705 7	0.434 15	0.293 19	0.815 24	0.348 15	0.241 8	0.499 9	0.669 4	0.507 13	0.649 8	0.442 19	0.796 11	0.602 23	0.561 12
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. arXiv
3DMV, FTSDF	0.501 15	0.558 18	0.608 19	0.424 24	0.478 11	0.690 15	0.246 21	0.586 14	0.468 12	0.450 10	0.911 19	0.394 8	0.160 18	0.438 12	0.212 21	0.432 19	0.541 19	0.475 14	0.742 16	0.727 14	0.477 17

PCNN	0.498 16	0.559 17	0.644 15	0.560 19	0.420 16	0.711 14	0.229 23	0.414 17	0.436 14	0.352 16	0.941 8	0.324 16	0.155 19	0.238 21	0.387 15	0.493 14	0.529 20	0.509 10	0.813 10	0.751 12	0.504 16

3DMV	0.484 17	0.484 22	0.538 22	0.643 11	0.424 15	0.606 24	0.310 15	0.574 15	0.433 16	0.378 15	0.796 25	0.301 17	0.214 12	0.537 7	0.208 22	0.472 18	0.507 23	0.413 22	0.693 19	0.602 23	0.539 13
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGB	0.458 18	0.577 16	0.611 18	0.356 26	0.321 23	0.715 13	0.299 17	0.376 20	0.328 23	0.319 17	0.944 6	0.285 19	0.164 17	0.216 24	0.229 20	0.484 16	0.545 18	0.456 17	0.755 15	0.709 15	0.475 18
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPN	0.447 19	0.679 9	0.604 20	0.578 17	0.380 18	0.682 17	0.291 18	0.106 26	0.483 11	0.258 24	0.920 17	0.258 21	0.025 26	0.231 23	0.325 16	0.480 17	0.560 17	0.463 16	0.725 17	0.666 20	0.231 26
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
PNET2	0.442 20	0.548 19	0.548 21	0.597 16	0.363 20	0.628 22	0.300 16	0.292 21	0.374 20	0.307 18	0.881 23	0.268 20	0.186 14	0.238 21	0.204 23	0.407 21	0.506 24	0.449 18	0.667 20	0.620 22	0.462 19

SurfaceConvPF	0.442 20	0.505 20	0.622 17	0.380 25	0.342 22	0.654 19	0.227 24	0.397 19	0.367 21	0.276 21	0.924 15	0.240 22	0.198 13	0.359 16	0.262 18	0.366 22	0.581 15	0.435 20	0.640 21	0.668 19	0.398 20
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
Tangent Convolutions	0.438 22	0.437 24	0.646 14	0.474 21	0.369 19	0.645 20	0.353 13	0.258 23	0.282 25	0.279 20	0.918 18	0.298 18	0.147 20	0.283 18	0.294 17	0.487 15	0.562 16	0.427 21	0.619 22	0.633 21	0.352 22
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
SPLAT Net	0.393 23	0.472 23	0.511 23	0.606 13	0.311 24	0.656 18	0.245 22	0.405 18	0.328 23	0.197 25	0.927 14	0.227 24	0.000 28	0.001 28	0.249 19	0.271 27	0.510 21	0.383 24	0.593 23	0.699 16	0.267 24
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF	0.383 24	0.297 26	0.491 24	0.432 23	0.358 21	0.612 23	0.274 19	0.116 25	0.411 18	0.265 22	0.904 22	0.229 23	0.079 24	0.250 19	0.185 24	0.320 25	0.510 21	0.385 23	0.548 24	0.597 25	0.394 21

PointNet++	0.339 25	0.584 15	0.478 25	0.458 22	0.256 26	0.360 27	0.250 20	0.247 24	0.278 26	0.261 23	0.677 27	0.183 25	0.117 21	0.212 25	0.145 26	0.364 23	0.346 27	0.232 27	0.548 24	0.523 26	0.252 25
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNet	0.308 26	0.353 25	0.290 27	0.278 27	0.166 27	0.553 25	0.169 27	0.286 22	0.147 27	0.148 27	0.908 20	0.182 26	0.064 25	0.023 27	0.018 28	0.354 24	0.363 25	0.345 25	0.546 26	0.685 17	0.278 23

ScanNet	0.306 27	0.203 27	0.366 26	0.501 20	0.311 24	0.524 26	0.211 26	0.002 28	0.342 22	0.189 26	0.786 26	0.145 27	0.102 23	0.245 20	0.152 25	0.318 26	0.348 26	0.300 26	0.460 27	0.437 27	0.182 27
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
ERROR	0.054 28	0.000 28	0.041 28	0.172 28	0.030 28	0.062 28	0.001 28	0.035 27	0.004 28	0.051 28	0.143 28	0.019 28	0.003 27	0.041 26	0.050 27	0.003 28	0.054 28	0.018 28	0.005 28	0.264 28	0.082 28

3D Semantic label benchmark