3D Semantic Label Results - ScanNet Benchmark

The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).

This table lists the benchmark results for the 3D semantic label scenario.

Method	avg iou	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	floor	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	wall	window

MinkowskiNet	0.736 1	0.859 2	0.818 2	0.832 2	0.709 2	0.840 2	0.521 2	0.853 1	0.660 1	0.643 1	0.951 4	0.544 2	0.286 6	0.731 1	0.893 1	0.675 4	0.772 3	0.683 1	0.874 7	0.852 2	0.727 1
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet	0.725 2	0.647 17	0.821 1	0.846 1	0.721 1	0.869 1	0.533 1	0.754 5	0.603 4	0.614 2	0.955 1	0.572 1	0.325 2	0.710 2	0.870 2	0.724 1	0.823 1	0.628 3	0.934 1	0.865 1	0.683 2

KP-FCNN	0.684 3	0.847 4	0.758 6	0.784 3	0.647 3	0.814 5	0.473 4	0.772 4	0.605 3	0.594 3	0.935 15	0.450 6	0.181 19	0.587 4	0.805 3	0.690 2	0.785 2	0.614 4	0.882 4	0.819 3	0.632 5
H. Thomas, C. Qi, J. Deschaud, B. Marcotegui, F. Goulette, L. Guibas.: KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV 2019
PointConv	0.666 4	0.781 6	0.759 5	0.699 9	0.644 4	0.822 4	0.475 3	0.779 3	0.564 8	0.504 9	0.953 2	0.428 8	0.203 15	0.586 5	0.754 4	0.661 5	0.753 5	0.588 5	0.902 2	0.813 5	0.642 3
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
MVPNet	0.641 5	0.831 5	0.715 10	0.671 12	0.590 7	0.781 9	0.394 12	0.679 13	0.642 2	0.553 5	0.937 14	0.462 5	0.256 7	0.649 3	0.406 18	0.626 8	0.691 9	0.666 2	0.877 6	0.792 9	0.608 8
Maximilian Jaritz, Jiayuan Gu, Hao Su: Multi-view PointNet for 3D Scene Understanding. GMDL Workshop, ICCV 2019
joint point-based	0.634 6	0.614 18	0.778 3	0.667 14	0.633 5	0.825 3	0.420 8	0.804 2	0.467 18	0.561 4	0.951 4	0.494 3	0.291 3	0.566 7	0.458 15	0.579 14	0.764 4	0.559 9	0.838 11	0.814 4	0.598 11
Hung-Yueh Chiang, Yen-Liang Lin, Yueh-Cheng Liu, Winston H. Hsu: A Unified Point-Based Framework for 3D Segmentation. 3DV 2019
MCCNN	0.633 7	0.866 1	0.731 8	0.771 4	0.576 9	0.809 6	0.410 10	0.684 11	0.497 14	0.491 10	0.949 6	0.466 4	0.105 26	0.581 6	0.646 8	0.620 9	0.680 11	0.542 13	0.817 13	0.795 8	0.618 6
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
DMC-Net	0.630 8	0.706 13	0.738 7	0.745 6	0.535 12	0.787 8	0.335 18	0.742 6	0.512 12	0.546 6	0.941 10	0.427 9	0.330 1	0.496 12	0.653 7	0.634 7	0.719 6	0.550 11	0.754 20	0.801 7	0.642 3

HPEIN	0.618 9	0.729 10	0.668 16	0.647 15	0.597 6	0.766 11	0.414 9	0.680 12	0.520 10	0.525 7	0.946 7	0.432 7	0.215 12	0.493 14	0.599 9	0.638 6	0.617 19	0.570 7	0.897 3	0.806 6	0.605 9
Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, Jiaya Jia: Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation. ICCV 2019
SPH3D-GCN	0.610 10	0.858 3	0.772 4	0.489 25	0.532 13	0.792 7	0.404 11	0.643 16	0.570 7	0.507 8	0.935 15	0.414 11	0.046 30	0.510 10	0.702 5	0.602 11	0.705 8	0.549 12	0.859 10	0.773 11	0.534 17
Huan Lei, Naveed Akhtar, and Ajmal Mian: Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds.
FLConv	0.605 11	0.751 8	0.695 13	0.701 8	0.545 11	0.758 13	0.448 5	0.596 18	0.538 9	0.428 17	0.953 2	0.398 12	0.160 21	0.457 15	0.598 10	0.612 10	0.686 10	0.551 10	0.880 5	0.768 12	0.585 13

CDF-SM3D	0.597 12	0.557 23	0.728 9	0.703 7	0.572 10	0.730 16	0.435 7	0.715 8	0.504 13	0.467 13	0.863 28	0.427 9	0.184 18	0.495 13	0.569 11	0.685 3	0.717 7	0.516 14	0.866 9	0.726 19	0.479 21

LAP-D	0.594 13	0.720 11	0.692 14	0.637 17	0.456 19	0.773 10	0.391 14	0.730 7	0.587 5	0.445 15	0.940 12	0.381 14	0.288 4	0.434 17	0.453 16	0.591 12	0.649 13	0.581 6	0.777 17	0.749 17	0.610 7

DPC	0.592 14	0.720 11	0.700 11	0.602 20	0.480 16	0.762 12	0.380 16	0.713 9	0.585 6	0.437 16	0.940 12	0.369 16	0.288 4	0.434 17	0.509 14	0.590 13	0.639 17	0.567 8	0.772 18	0.755 15	0.592 12
Francis Engelmann, Theodora Kontogianni, Bastian Leibe: Dilated Point Convolutions. arXiv
CCRFNet	0.589 15	0.766 7	0.659 19	0.683 11	0.470 18	0.740 15	0.387 15	0.620 17	0.490 15	0.476 11	0.922 20	0.355 19	0.245 8	0.511 9	0.511 13	0.571 15	0.643 15	0.493 18	0.872 8	0.762 13	0.600 10

TextureNet	0.566 16	0.672 15	0.664 17	0.671 12	0.494 14	0.719 17	0.445 6	0.678 14	0.411 23	0.396 18	0.935 15	0.356 18	0.225 10	0.412 19	0.535 12	0.565 16	0.636 18	0.464 20	0.794 16	0.680 23	0.568 14
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
DVVNet	0.562 17	0.648 16	0.700 11	0.770 5	0.586 8	0.687 21	0.333 19	0.650 15	0.514 11	0.475 12	0.906 25	0.359 17	0.223 11	0.340 22	0.442 17	0.422 25	0.668 12	0.501 16	0.708 23	0.779 10	0.534 17

Pointnet++ & Feature	0.557 18	0.735 9	0.661 18	0.686 10	0.491 15	0.744 14	0.392 13	0.539 21	0.451 19	0.375 20	0.946 7	0.376 15	0.205 14	0.403 20	0.356 20	0.553 17	0.643 15	0.497 17	0.824 12	0.756 14	0.515 19

PanopticFusion-label	0.529 19	0.491 26	0.688 15	0.604 19	0.386 22	0.632 26	0.225 30	0.705 10	0.434 21	0.293 24	0.815 29	0.348 20	0.241 9	0.499 11	0.669 6	0.507 18	0.649 13	0.442 24	0.796 15	0.602 28	0.561 15
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
3DMV, FTSDF	0.501 20	0.558 22	0.608 24	0.424 29	0.478 17	0.690 20	0.246 26	0.586 19	0.468 17	0.450 14	0.911 23	0.394 13	0.160 21	0.438 16	0.212 26	0.432 24	0.541 24	0.475 19	0.742 21	0.727 18	0.477 22

PCNN	0.498 21	0.559 21	0.644 21	0.560 23	0.420 21	0.711 19	0.229 28	0.414 22	0.436 20	0.352 21	0.941 10	0.324 21	0.155 23	0.238 26	0.387 19	0.493 19	0.529 25	0.509 15	0.813 14	0.751 16	0.504 20

3DMV	0.484 22	0.484 27	0.538 27	0.643 16	0.424 20	0.606 29	0.310 20	0.574 20	0.433 22	0.378 19	0.796 30	0.301 22	0.214 13	0.537 8	0.208 27	0.472 23	0.507 28	0.413 27	0.693 24	0.602 28	0.539 16
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGB	0.458 23	0.577 20	0.611 23	0.356 31	0.321 28	0.715 18	0.299 22	0.376 25	0.328 28	0.319 22	0.944 9	0.285 24	0.164 20	0.216 29	0.229 25	0.484 21	0.545 23	0.456 22	0.755 19	0.709 20	0.475 23
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPN	0.447 24	0.679 14	0.604 25	0.578 22	0.380 23	0.682 22	0.291 23	0.106 31	0.483 16	0.258 29	0.920 21	0.258 26	0.025 31	0.231 28	0.325 21	0.480 22	0.560 22	0.463 21	0.725 22	0.666 25	0.231 31
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
SurfaceConvPF	0.442 25	0.505 25	0.622 22	0.380 30	0.342 27	0.654 24	0.227 29	0.397 24	0.367 26	0.276 26	0.924 19	0.240 27	0.198 16	0.359 21	0.262 23	0.366 27	0.581 20	0.435 25	0.640 26	0.668 24	0.398 25
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
PNET2	0.442 25	0.548 24	0.548 26	0.597 21	0.363 25	0.628 27	0.300 21	0.292 26	0.374 25	0.307 23	0.881 27	0.268 25	0.186 17	0.238 26	0.204 28	0.407 26	0.506 29	0.449 23	0.667 25	0.620 27	0.462 24

Tangent Convolutions	0.438 27	0.437 29	0.646 20	0.474 26	0.369 24	0.645 25	0.353 17	0.258 28	0.282 30	0.279 25	0.918 22	0.298 23	0.147 24	0.283 23	0.294 22	0.487 20	0.562 21	0.427 26	0.619 27	0.633 26	0.352 27
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
SPLAT Net	0.393 28	0.472 28	0.511 28	0.606 18	0.311 29	0.656 23	0.245 27	0.405 23	0.328 28	0.197 30	0.927 18	0.227 29	0.000 33	0.001 33	0.249 24	0.271 32	0.510 26	0.383 29	0.593 28	0.699 21	0.267 29
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF	0.383 29	0.297 31	0.491 29	0.432 28	0.358 26	0.612 28	0.274 24	0.116 30	0.411 23	0.265 27	0.904 26	0.229 28	0.079 28	0.250 24	0.185 29	0.320 30	0.510 26	0.385 28	0.548 29	0.597 30	0.394 26

PointNet++	0.339 30	0.584 19	0.478 30	0.458 27	0.256 31	0.360 32	0.250 25	0.247 29	0.278 31	0.261 28	0.677 32	0.183 30	0.117 25	0.212 30	0.145 31	0.364 28	0.346 32	0.232 32	0.548 29	0.523 31	0.252 30
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNet	0.308 31	0.353 30	0.290 32	0.278 32	0.166 32	0.553 30	0.169 32	0.286 27	0.147 32	0.148 32	0.908 24	0.182 31	0.064 29	0.023 32	0.018 33	0.354 29	0.363 30	0.345 30	0.546 31	0.685 22	0.278 28

ScanNet	0.306 32	0.203 32	0.366 31	0.501 24	0.311 29	0.524 31	0.211 31	0.002 33	0.342 27	0.189 31	0.786 31	0.145 32	0.102 27	0.245 25	0.152 30	0.318 31	0.348 31	0.300 31	0.460 32	0.437 32	0.182 32
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
ERROR	0.054 33	0.000 33	0.041 33	0.172 33	0.030 33	0.062 33	0.001 33	0.035 32	0.004 33	0.051 33	0.143 33	0.019 33	0.003 32	0.041 31	0.050 32	0.003 33	0.054 33	0.018 33	0.005 33	0.264 33	0.082 33

3D Semantic label benchmark