3D Semantic Label Results - ScanNet Benchmark

The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).

This table lists the benchmark results for the 3D semantic label scenario.

Method	avg iou	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	floor	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	wall	window

MinkowskiNet	0.734 1	0.858 2	0.833 1	0.834 2	0.716 2	0.855 2	0.459 3	0.836 1	0.639 2	0.641 1	0.953 2	0.541 2	0.302 2	0.743 1	0.865 2	0.726 1	0.771 3	0.664 2	0.891 3	0.851 2	0.694 1
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet	0.725 2	0.647 14	0.821 2	0.846 1	0.721 1	0.869 1	0.533 1	0.754 4	0.603 4	0.614 2	0.955 1	0.572 1	0.325 1	0.710 2	0.870 1	0.724 2	0.823 1	0.628 3	0.934 1	0.865 1	0.683 2

KP-FCNN	0.684 3	0.847 3	0.758 4	0.784 3	0.647 3	0.814 4	0.473 2	0.772 3	0.605 3	0.594 3	0.935 13	0.450 6	0.181 18	0.587 4	0.805 3	0.690 3	0.785 2	0.614 4	0.882 4	0.819 3	0.632 3
H. Thomas, C. Qi, J. Deschaud, B. Marcotegui, F. Goulette, L. Guibas.: KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV 2019
MVPNet	0.641 4	0.831 4	0.715 7	0.671 9	0.590 6	0.781 7	0.394 10	0.679 10	0.642 1	0.553 5	0.937 12	0.462 5	0.256 7	0.649 3	0.406 15	0.626 5	0.691 6	0.666 1	0.877 5	0.792 7	0.608 6

joint point-based	0.634 5	0.614 16	0.778 3	0.667 11	0.633 4	0.825 3	0.420 7	0.804 2	0.467 13	0.561 4	0.951 3	0.494 3	0.291 3	0.566 6	0.458 12	0.579 10	0.764 4	0.559 8	0.838 7	0.814 4	0.598 9
Hung-Yueh Chiang, Yen-Liang Lin, Yueh-Cheng Liu, Winston H. Hsu: A Unified Point-Based Framework for 3D Segmentation. 3DV 2019
MCCNN	0.633 6	0.866 1	0.731 5	0.771 4	0.576 8	0.809 5	0.410 9	0.684 8	0.497 9	0.491 7	0.949 4	0.466 4	0.105 24	0.581 5	0.646 6	0.620 6	0.680 7	0.542 9	0.817 10	0.795 6	0.618 4
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
HPEIN	0.618 7	0.729 8	0.668 12	0.647 12	0.597 5	0.766 9	0.414 8	0.680 9	0.520 7	0.525 6	0.946 5	0.432 7	0.215 12	0.493 10	0.599 7	0.638 4	0.617 16	0.570 6	0.897 2	0.806 5	0.605 7

DMC-Net	0.608 8	0.732 7	0.729 6	0.694 6	0.536 9	0.783 6	0.427 6	0.639 13	0.438 15	0.450 10	0.934 15	0.379 10	0.289 4	0.492 11	0.674 4	0.608 7	0.709 5	0.503 12	0.800 12	0.779 8	0.567 12

LAP-D	0.594 9	0.720 9	0.692 10	0.637 14	0.456 16	0.773 8	0.391 12	0.730 5	0.587 5	0.445 12	0.940 10	0.381 9	0.288 5	0.434 14	0.453 13	0.591 8	0.649 9	0.581 5	0.777 15	0.749 15	0.610 5

DPC	0.592 10	0.720 9	0.700 8	0.602 17	0.480 12	0.762 10	0.380 14	0.713 6	0.585 6	0.437 14	0.940 10	0.369 13	0.288 5	0.434 14	0.509 11	0.590 9	0.639 13	0.567 7	0.772 16	0.755 13	0.592 10
Francis Engelmann, Theodora Kontogianni, Bastian Leibe: Dilated Point Convolutions. arXiv
CCRFNet	0.589 11	0.766 5	0.659 15	0.683 8	0.470 15	0.740 12	0.387 13	0.620 14	0.490 10	0.476 8	0.922 18	0.355 16	0.245 8	0.511 8	0.511 10	0.571 11	0.643 11	0.493 15	0.872 6	0.762 10	0.600 8

TextureNet	0.566 12	0.672 12	0.664 13	0.671 9	0.494 10	0.719 14	0.445 4	0.678 11	0.411 20	0.396 15	0.935 13	0.356 15	0.225 10	0.412 16	0.535 9	0.565 12	0.636 15	0.464 17	0.794 14	0.680 20	0.568 11
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
DVVNet	0.562 13	0.648 13	0.700 8	0.770 5	0.586 7	0.687 18	0.333 16	0.650 12	0.514 8	0.475 9	0.906 23	0.359 14	0.223 11	0.340 19	0.442 14	0.422 22	0.668 8	0.501 13	0.708 20	0.779 8	0.534 15

Pointnet++ & Feature	0.557 14	0.735 6	0.661 14	0.686 7	0.491 11	0.744 11	0.392 11	0.539 17	0.451 14	0.375 17	0.946 5	0.376 11	0.205 14	0.403 17	0.356 17	0.553 13	0.643 11	0.497 14	0.824 9	0.756 12	0.515 16

PointConv	0.556 15	0.636 15	0.640 18	0.574 20	0.472 14	0.739 13	0.430 5	0.433 18	0.418 19	0.445 12	0.944 7	0.372 12	0.185 17	0.464 12	0.575 8	0.540 14	0.639 13	0.505 11	0.827 8	0.762 10	0.515 16
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
PanopticFusion-label	0.529 16	0.491 23	0.688 11	0.604 16	0.386 19	0.632 23	0.225 27	0.705 7	0.434 17	0.293 21	0.815 26	0.348 17	0.241 9	0.499 9	0.669 5	0.507 15	0.649 9	0.442 21	0.796 13	0.602 25	0.561 13
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
3DMV, FTSDF	0.501 17	0.558 20	0.608 21	0.424 26	0.478 13	0.690 17	0.246 23	0.586 15	0.468 12	0.450 10	0.911 21	0.394 8	0.160 20	0.438 13	0.212 23	0.432 21	0.541 21	0.475 16	0.742 18	0.727 16	0.477 19

PCNN	0.498 18	0.559 19	0.644 17	0.560 21	0.420 18	0.711 16	0.229 25	0.414 19	0.436 16	0.352 18	0.941 9	0.324 18	0.155 21	0.238 23	0.387 16	0.493 16	0.529 22	0.509 10	0.813 11	0.751 14	0.504 18

3DMV	0.484 19	0.484 24	0.538 24	0.643 13	0.424 17	0.606 26	0.310 17	0.574 16	0.433 18	0.378 16	0.796 27	0.301 19	0.214 13	0.537 7	0.208 24	0.472 20	0.507 25	0.413 24	0.693 21	0.602 25	0.539 14
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGB	0.458 20	0.577 18	0.611 20	0.356 28	0.321 25	0.715 15	0.299 19	0.376 22	0.328 25	0.319 19	0.944 7	0.285 21	0.164 19	0.216 26	0.229 22	0.484 18	0.545 20	0.456 19	0.755 17	0.709 17	0.475 20
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPN	0.447 21	0.679 11	0.604 22	0.578 19	0.380 20	0.682 19	0.291 20	0.106 28	0.483 11	0.258 26	0.920 19	0.258 23	0.025 28	0.231 25	0.325 18	0.480 19	0.560 19	0.463 18	0.725 19	0.666 22	0.231 28
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
PNET2	0.442 22	0.548 21	0.548 23	0.597 18	0.363 22	0.628 24	0.300 18	0.292 23	0.374 22	0.307 20	0.881 25	0.268 22	0.186 16	0.238 23	0.204 25	0.407 23	0.506 26	0.449 20	0.667 22	0.620 24	0.462 21

SurfaceConvPF	0.442 22	0.505 22	0.622 19	0.380 27	0.342 24	0.654 21	0.227 26	0.397 21	0.367 23	0.276 23	0.924 17	0.240 24	0.198 15	0.359 18	0.262 20	0.366 24	0.581 17	0.435 22	0.640 23	0.668 21	0.398 22
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
Tangent Convolutions	0.438 24	0.437 26	0.646 16	0.474 23	0.369 21	0.645 22	0.353 15	0.258 25	0.282 27	0.279 22	0.918 20	0.298 20	0.147 22	0.283 20	0.294 19	0.487 17	0.562 18	0.427 23	0.619 24	0.633 23	0.352 24
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
SPLAT Net	0.393 25	0.472 25	0.511 25	0.606 15	0.311 26	0.656 20	0.245 24	0.405 20	0.328 25	0.197 27	0.927 16	0.227 26	0.000 30	0.001 30	0.249 21	0.271 29	0.510 23	0.383 26	0.593 25	0.699 18	0.267 26
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF	0.383 26	0.297 28	0.491 26	0.432 25	0.358 23	0.612 25	0.274 21	0.116 27	0.411 20	0.265 24	0.904 24	0.229 25	0.079 26	0.250 21	0.185 26	0.320 27	0.510 23	0.385 25	0.548 26	0.597 27	0.394 23

PointNet++	0.339 27	0.584 17	0.478 27	0.458 24	0.256 28	0.360 29	0.250 22	0.247 26	0.278 28	0.261 25	0.677 29	0.183 27	0.117 23	0.212 27	0.145 28	0.364 25	0.346 29	0.232 29	0.548 26	0.523 28	0.252 27
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNet	0.308 28	0.353 27	0.290 29	0.278 29	0.166 29	0.553 27	0.169 29	0.286 24	0.147 29	0.148 29	0.908 22	0.182 28	0.064 27	0.023 29	0.018 30	0.354 26	0.363 27	0.345 27	0.546 28	0.685 19	0.278 25

ScanNet	0.306 29	0.203 29	0.366 28	0.501 22	0.311 26	0.524 28	0.211 28	0.002 30	0.342 24	0.189 28	0.786 28	0.145 29	0.102 25	0.245 22	0.152 27	0.318 28	0.348 28	0.300 28	0.460 29	0.437 29	0.182 29
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
ERROR	0.054 30	0.000 30	0.041 30	0.172 30	0.030 30	0.062 30	0.001 30	0.035 29	0.004 30	0.051 30	0.143 30	0.019 30	0.003 29	0.041 28	0.050 29	0.003 30	0.054 30	0.018 30	0.005 30	0.264 30	0.082 30

3D Semantic label benchmark