3D Semantic Label Results - ScanNet Benchmark

The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).

This table lists the benchmark results for the 3D semantic label scenario.

Method	avg iou	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	floor	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	wall	window

MinkowskiNet	0.736 1	0.859 2	0.818 2	0.832 2	0.709 2	0.840 2	0.521 2	0.853 1	0.660 1	0.643 1	0.951 3	0.544 2	0.286 7	0.731 1	0.893 1	0.675 5	0.772 4	0.683 1	0.874 8	0.852 2	0.727 1
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet	0.725 2	0.647 18	0.821 1	0.846 1	0.721 1	0.869 1	0.533 1	0.754 6	0.603 4	0.614 2	0.955 1	0.572 1	0.325 1	0.710 2	0.870 2	0.724 2	0.823 1	0.628 3	0.934 1	0.865 1	0.683 2

CU-Hybrid Net	0.693 3	0.596 21	0.789 3	0.803 3	0.677 3	0.800 8	0.469 6	0.846 2	0.554 10	0.591 4	0.948 7	0.500 3	0.316 2	0.609 4	0.847 3	0.732 1	0.808 2	0.593 5	0.894 4	0.839 3	0.652 3

KP-FCNN	0.684 4	0.847 4	0.758 7	0.784 6	0.647 4	0.814 5	0.473 5	0.772 5	0.605 3	0.594 3	0.935 17	0.450 8	0.181 22	0.587 6	0.805 5	0.690 4	0.785 3	0.614 4	0.882 5	0.819 4	0.632 6
H. Thomas, C. Qi, J. Deschaud, B. Marcotegui, F. Goulette, L. Guibas.: KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV 2019
PointConv	0.666 5	0.781 6	0.759 6	0.699 10	0.644 5	0.822 4	0.475 4	0.779 4	0.564 9	0.504 11	0.953 2	0.428 11	0.203 18	0.586 7	0.754 6	0.661 7	0.753 6	0.588 6	0.902 2	0.813 6	0.642 4
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
DMC-Net	0.653 6	0.771 7	0.701 12	0.801 4	0.619 8	0.807 7	0.463 7	0.680 12	0.495 16	0.520 9	0.940 13	0.452 7	0.301 3	0.496 15	0.816 4	0.664 6	0.719 7	0.563 11	0.822 15	0.799 9	0.638 5

MVPNet	0.641 7	0.831 5	0.715 11	0.671 13	0.590 11	0.781 10	0.394 15	0.679 14	0.642 2	0.553 6	0.937 16	0.462 6	0.256 9	0.649 3	0.406 20	0.626 10	0.691 10	0.666 2	0.877 6	0.792 11	0.608 10
Maximilian Jaritz, Jiayuan Gu, Hao Su: Multi-view PointNet for 3D Scene Understanding. GMDL Workshop, ICCV 2019
joint point-based	0.634 8	0.614 20	0.778 4	0.667 15	0.633 7	0.825 3	0.420 9	0.804 3	0.467 20	0.561 5	0.951 3	0.494 4	0.291 4	0.566 9	0.458 17	0.579 16	0.764 5	0.559 13	0.838 13	0.814 5	0.598 13
Hung-Yueh Chiang, Yen-Liang Lin, Yueh-Cheng Liu, Winston H. Hsu: A Unified Point-Based Framework for 3D Segmentation. 3DV 2019
MCCNN	0.633 9	0.866 1	0.731 9	0.771 7	0.576 13	0.809 6	0.410 13	0.684 11	0.497 15	0.491 12	0.949 6	0.466 5	0.105 28	0.581 8	0.646 9	0.620 11	0.680 13	0.542 16	0.817 16	0.795 10	0.618 8
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
3DSM_DMMF	0.631 10	0.626 19	0.745 8	0.801 4	0.607 9	0.751 16	0.506 3	0.729 8	0.565 8	0.491 12	0.866 30	0.434 9	0.197 20	0.595 5	0.630 10	0.709 3	0.705 8	0.560 12	0.875 7	0.740 20	0.491 23

PointASNL	0.630 11	0.738 10	0.729 10	0.764 9	0.637 6	0.779 11	0.416 11	0.626 18	0.518 12	0.530 7	0.951 3	0.398 14	0.260 8	0.518 11	0.576 13	0.590 14	0.687 11	0.568 9	0.872 9	0.810 7	0.631 7

HPEIN	0.618 12	0.729 12	0.668 17	0.647 16	0.597 10	0.766 14	0.414 12	0.680 12	0.520 11	0.525 8	0.946 8	0.432 10	0.215 14	0.493 16	0.599 11	0.638 8	0.617 21	0.570 8	0.897 3	0.806 8	0.605 11
Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, Jiaya Jia: Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation. ICCV 2019
SPH3D-GCN	0.610 13	0.858 3	0.772 5	0.489 27	0.532 15	0.792 9	0.404 14	0.643 17	0.570 7	0.507 10	0.935 17	0.414 12	0.046 32	0.510 13	0.702 7	0.602 12	0.705 8	0.549 14	0.859 11	0.773 14	0.534 18
Huan Lei, Naveed Akhtar, and Ajmal Mian: Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds.
SIConv	0.594 14	0.768 8	0.639 23	0.616 19	0.544 14	0.768 13	0.419 10	0.601 20	0.513 14	0.474 16	0.946 8	0.402 13	0.213 16	0.387 22	0.581 12	0.633 9	0.683 12	0.549 14	0.843 12	0.774 13	0.521 20

LAP-D	0.594 14	0.720 13	0.692 15	0.637 18	0.456 21	0.773 12	0.391 17	0.730 7	0.587 5	0.445 18	0.940 13	0.381 16	0.288 5	0.434 18	0.453 18	0.591 13	0.649 15	0.581 7	0.777 20	0.749 19	0.610 9

DPC	0.592 16	0.720 13	0.700 13	0.602 22	0.480 18	0.762 15	0.380 19	0.713 9	0.585 6	0.437 19	0.940 13	0.369 18	0.288 5	0.434 18	0.509 16	0.590 14	0.639 19	0.567 10	0.772 21	0.755 17	0.592 14
Francis Engelmann, Theodora Kontogianni, Bastian Leibe: Dilated Point Convolutions. arXiv
CCRFNet	0.589 17	0.766 9	0.659 20	0.683 12	0.470 20	0.740 18	0.387 18	0.620 19	0.490 17	0.476 14	0.922 22	0.355 21	0.245 10	0.511 12	0.511 15	0.571 17	0.643 17	0.493 20	0.872 9	0.762 15	0.600 12

TextureNet	0.566 18	0.672 16	0.664 18	0.671 13	0.494 16	0.719 19	0.445 8	0.678 15	0.411 25	0.396 20	0.935 17	0.356 20	0.225 12	0.412 20	0.535 14	0.565 18	0.636 20	0.464 22	0.794 19	0.680 25	0.568 15
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
DVVNet	0.562 19	0.648 17	0.700 13	0.770 8	0.586 12	0.687 23	0.333 21	0.650 16	0.514 13	0.475 15	0.906 27	0.359 19	0.223 13	0.340 24	0.442 19	0.422 27	0.668 14	0.501 18	0.708 25	0.779 12	0.534 18

Pointnet++ & Feature	0.557 20	0.735 11	0.661 19	0.686 11	0.491 17	0.744 17	0.392 16	0.539 23	0.451 21	0.375 22	0.946 8	0.376 17	0.205 17	0.403 21	0.356 22	0.553 19	0.643 17	0.497 19	0.824 14	0.756 16	0.515 21

PanopticFusion-label	0.529 21	0.491 28	0.688 16	0.604 21	0.386 24	0.632 28	0.225 32	0.705 10	0.434 23	0.293 26	0.815 31	0.348 22	0.241 11	0.499 14	0.669 8	0.507 20	0.649 15	0.442 26	0.796 18	0.602 30	0.561 16
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
3DMV, FTSDF	0.501 22	0.558 25	0.608 26	0.424 31	0.478 19	0.690 22	0.246 28	0.586 21	0.468 19	0.450 17	0.911 25	0.394 15	0.160 24	0.438 17	0.212 28	0.432 26	0.541 26	0.475 21	0.742 23	0.727 21	0.477 24

PCNN	0.498 23	0.559 24	0.644 22	0.560 25	0.420 23	0.711 21	0.229 30	0.414 24	0.436 22	0.352 23	0.941 12	0.324 23	0.155 25	0.238 28	0.387 21	0.493 21	0.529 27	0.509 17	0.813 17	0.751 18	0.504 22

3DMV	0.484 24	0.484 29	0.538 29	0.643 17	0.424 22	0.606 31	0.310 22	0.574 22	0.433 24	0.378 21	0.796 32	0.301 24	0.214 15	0.537 10	0.208 29	0.472 25	0.507 30	0.413 29	0.693 26	0.602 30	0.539 17
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGB	0.458 25	0.577 23	0.611 25	0.356 33	0.321 30	0.715 20	0.299 24	0.376 27	0.328 30	0.319 24	0.944 11	0.285 26	0.164 23	0.216 31	0.229 27	0.484 23	0.545 25	0.456 24	0.755 22	0.709 22	0.475 25
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPN	0.447 26	0.679 15	0.604 27	0.578 24	0.380 25	0.682 24	0.291 25	0.106 33	0.483 18	0.258 31	0.920 23	0.258 28	0.025 33	0.231 30	0.325 23	0.480 24	0.560 24	0.463 23	0.725 24	0.666 27	0.231 33
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
SurfaceConvPF	0.442 27	0.505 27	0.622 24	0.380 32	0.342 29	0.654 26	0.227 31	0.397 26	0.367 28	0.276 28	0.924 21	0.240 29	0.198 19	0.359 23	0.262 25	0.366 29	0.581 22	0.435 27	0.640 28	0.668 26	0.398 27
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
PNET2	0.442 27	0.548 26	0.548 28	0.597 23	0.363 27	0.628 29	0.300 23	0.292 28	0.374 27	0.307 25	0.881 29	0.268 27	0.186 21	0.238 28	0.204 30	0.407 28	0.506 31	0.449 25	0.667 27	0.620 29	0.462 26

Tangent Convolutions	0.438 29	0.437 31	0.646 21	0.474 28	0.369 26	0.645 27	0.353 20	0.258 30	0.282 32	0.279 27	0.918 24	0.298 25	0.147 26	0.283 25	0.294 24	0.487 22	0.562 23	0.427 28	0.619 29	0.633 28	0.352 29
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
SPLAT Net	0.393 30	0.472 30	0.511 30	0.606 20	0.311 31	0.656 25	0.245 29	0.405 25	0.328 30	0.197 32	0.927 20	0.227 31	0.000 35	0.001 35	0.249 26	0.271 34	0.510 28	0.383 31	0.593 30	0.699 23	0.267 31
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF	0.383 31	0.297 33	0.491 31	0.432 30	0.358 28	0.612 30	0.274 26	0.116 32	0.411 25	0.265 29	0.904 28	0.229 30	0.079 30	0.250 26	0.185 31	0.320 32	0.510 28	0.385 30	0.548 31	0.597 32	0.394 28

PointNet++	0.339 32	0.584 22	0.478 32	0.458 29	0.256 33	0.360 34	0.250 27	0.247 31	0.278 33	0.261 30	0.677 34	0.183 32	0.117 27	0.212 32	0.145 33	0.364 30	0.346 34	0.232 34	0.548 31	0.523 33	0.252 32
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNet	0.308 33	0.353 32	0.290 34	0.278 34	0.166 34	0.553 32	0.169 34	0.286 29	0.147 34	0.148 34	0.908 26	0.182 33	0.064 31	0.023 34	0.018 35	0.354 31	0.363 32	0.345 32	0.546 33	0.685 24	0.278 30

ScanNet	0.306 34	0.203 34	0.366 33	0.501 26	0.311 31	0.524 33	0.211 33	0.002 35	0.342 29	0.189 33	0.786 33	0.145 34	0.102 29	0.245 27	0.152 32	0.318 33	0.348 33	0.300 33	0.460 34	0.437 34	0.182 34
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
ERROR	0.054 35	0.000 35	0.041 35	0.172 35	0.030 35	0.062 35	0.001 35	0.035 34	0.004 35	0.051 35	0.143 35	0.019 35	0.003 34	0.041 33	0.050 34	0.003 35	0.054 35	0.018 35	0.005 35	0.264 35	0.082 35

3D Semantic label benchmark