3D Semantic Label Results - ScanNet Benchmark

The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).

This table lists the benchmark results for the 3D semantic label scenario.

Method	avg iou	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	floor	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	wall	window

MinkowskiNet	0.736 1	0.859 2	0.818 2	0.832 2	0.709 2	0.840 2	0.521 2	0.853 1	0.660 1	0.643 1	0.951 3	0.544 2	0.286 6	0.731 1	0.893 1	0.675 4	0.772 3	0.683 1	0.874 7	0.852 2	0.727 1
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet	0.725 2	0.647 18	0.821 1	0.846 1	0.721 1	0.869 1	0.533 1	0.754 5	0.603 4	0.614 2	0.955 1	0.572 1	0.325 1	0.710 2	0.870 2	0.724 1	0.823 1	0.628 3	0.934 1	0.865 1	0.683 2

KP-FCNN	0.684 3	0.847 4	0.758 6	0.784 4	0.647 3	0.814 5	0.473 5	0.772 4	0.605 3	0.594 3	0.935 16	0.450 7	0.181 21	0.587 4	0.805 4	0.690 3	0.785 2	0.614 4	0.882 5	0.819 3	0.632 5
H. Thomas, C. Qi, J. Deschaud, B. Marcotegui, F. Goulette, L. Guibas.: KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV 2019
PointConv	0.666 4	0.781 6	0.759 5	0.699 9	0.644 4	0.822 4	0.475 4	0.779 3	0.564 8	0.504 10	0.953 2	0.428 9	0.203 17	0.586 5	0.754 5	0.661 6	0.753 5	0.588 5	0.902 2	0.813 5	0.642 3
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
DMC-Net	0.653 5	0.771 7	0.701 11	0.801 3	0.619 7	0.807 7	0.463 6	0.680 11	0.495 15	0.520 8	0.940 12	0.452 6	0.301 2	0.496 14	0.816 3	0.664 5	0.719 6	0.563 11	0.822 14	0.799 8	0.638 4

MVPNet	0.641 6	0.831 5	0.715 10	0.671 12	0.590 10	0.781 9	0.394 14	0.679 13	0.642 2	0.553 5	0.937 15	0.462 5	0.256 8	0.649 3	0.406 19	0.626 9	0.691 9	0.666 2	0.877 6	0.792 10	0.608 9
Maximilian Jaritz, Jiayuan Gu, Hao Su: Multi-view PointNet for 3D Scene Understanding. GMDL Workshop, ICCV 2019
joint point-based	0.634 7	0.614 19	0.778 3	0.667 14	0.633 6	0.825 3	0.420 8	0.804 2	0.467 19	0.561 4	0.951 3	0.494 3	0.291 3	0.566 8	0.458 16	0.579 15	0.764 4	0.559 12	0.838 12	0.814 4	0.598 12
Hung-Yueh Chiang, Yen-Liang Lin, Yueh-Cheng Liu, Winston H. Hsu: A Unified Point-Based Framework for 3D Segmentation. 3DV 2019
MCCNN	0.633 8	0.866 1	0.731 8	0.771 5	0.576 12	0.809 6	0.410 12	0.684 10	0.497 14	0.491 12	0.949 6	0.466 4	0.105 27	0.581 7	0.646 8	0.620 10	0.680 12	0.542 15	0.817 15	0.795 9	0.618 7
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
PointASNL	0.630 9	0.738 10	0.729 9	0.764 8	0.637 5	0.779 10	0.416 10	0.626 17	0.518 11	0.530 6	0.951 3	0.398 13	0.260 7	0.518 10	0.576 12	0.590 13	0.687 10	0.568 8	0.872 8	0.810 6	0.631 6

CDF-SM3D	0.626 10	0.592 20	0.746 7	0.767 7	0.607 8	0.761 15	0.501 3	0.738 6	0.546 9	0.503 11	0.864 29	0.421 10	0.198 18	0.584 6	0.579 11	0.694 2	0.706 7	0.566 10	0.885 4	0.745 19	0.523 19

HPEIN	0.618 11	0.729 12	0.668 16	0.647 15	0.597 9	0.766 13	0.414 11	0.680 11	0.520 10	0.525 7	0.946 7	0.432 8	0.215 13	0.493 15	0.599 9	0.638 7	0.617 20	0.570 7	0.897 3	0.806 7	0.605 10
Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, Jiaya Jia: Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation. ICCV 2019
SPH3D-GCN	0.610 12	0.858 3	0.772 4	0.489 26	0.532 14	0.792 8	0.404 13	0.643 16	0.570 7	0.507 9	0.935 16	0.414 11	0.046 31	0.510 12	0.702 6	0.602 11	0.705 8	0.549 13	0.859 10	0.773 13	0.534 17
Huan Lei, Naveed Akhtar, and Ajmal Mian: Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds.
LAP-D	0.594 13	0.720 13	0.692 14	0.637 17	0.456 20	0.773 11	0.391 16	0.730 7	0.587 5	0.445 17	0.940 12	0.381 15	0.288 4	0.434 17	0.453 17	0.591 12	0.649 14	0.581 6	0.777 19	0.749 18	0.610 8

SIConv	0.594 13	0.768 8	0.639 22	0.616 18	0.544 13	0.768 12	0.419 9	0.601 19	0.513 13	0.474 15	0.946 7	0.402 12	0.213 15	0.387 21	0.581 10	0.633 8	0.683 11	0.549 13	0.843 11	0.774 12	0.521 20

DPC	0.592 15	0.720 13	0.700 12	0.602 21	0.480 17	0.762 14	0.380 18	0.713 8	0.585 6	0.437 18	0.940 12	0.369 17	0.288 4	0.434 17	0.509 15	0.590 13	0.639 18	0.567 9	0.772 20	0.755 16	0.592 13
Francis Engelmann, Theodora Kontogianni, Bastian Leibe: Dilated Point Convolutions. arXiv
CCRFNet	0.589 16	0.766 9	0.659 19	0.683 11	0.470 19	0.740 17	0.387 17	0.620 18	0.490 16	0.476 13	0.922 21	0.355 20	0.245 9	0.511 11	0.511 14	0.571 16	0.643 16	0.493 19	0.872 8	0.762 14	0.600 11

TextureNet	0.566 17	0.672 16	0.664 17	0.671 12	0.494 15	0.719 18	0.445 7	0.678 14	0.411 24	0.396 19	0.935 16	0.356 19	0.225 11	0.412 19	0.535 13	0.565 17	0.636 19	0.464 21	0.794 18	0.680 24	0.568 14
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
DVVNet	0.562 18	0.648 17	0.700 12	0.770 6	0.586 11	0.687 22	0.333 20	0.650 15	0.514 12	0.475 14	0.906 26	0.359 18	0.223 12	0.340 23	0.442 18	0.422 26	0.668 13	0.501 17	0.708 24	0.779 11	0.534 17

Pointnet++ & Feature	0.557 19	0.735 11	0.661 18	0.686 10	0.491 16	0.744 16	0.392 15	0.539 22	0.451 20	0.375 21	0.946 7	0.376 16	0.205 16	0.403 20	0.356 21	0.553 18	0.643 16	0.497 18	0.824 13	0.756 15	0.515 21

PanopticFusion-label	0.529 20	0.491 27	0.688 15	0.604 20	0.386 23	0.632 27	0.225 31	0.705 9	0.434 22	0.293 25	0.815 30	0.348 21	0.241 10	0.499 13	0.669 7	0.507 19	0.649 14	0.442 25	0.796 17	0.602 29	0.561 15
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
3DMV, FTSDF	0.501 21	0.558 24	0.608 25	0.424 30	0.478 18	0.690 21	0.246 27	0.586 20	0.468 18	0.450 16	0.911 24	0.394 14	0.160 23	0.438 16	0.212 27	0.432 25	0.541 25	0.475 20	0.742 22	0.727 20	0.477 23

PCNN	0.498 22	0.559 23	0.644 21	0.560 24	0.420 22	0.711 20	0.229 29	0.414 23	0.436 21	0.352 22	0.941 11	0.324 22	0.155 24	0.238 27	0.387 20	0.493 20	0.529 26	0.509 16	0.813 16	0.751 17	0.504 22

3DMV	0.484 23	0.484 28	0.538 28	0.643 16	0.424 21	0.606 30	0.310 21	0.574 21	0.433 23	0.378 20	0.796 31	0.301 23	0.214 14	0.537 9	0.208 28	0.472 24	0.507 29	0.413 28	0.693 25	0.602 29	0.539 16
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGB	0.458 24	0.577 22	0.611 24	0.356 32	0.321 29	0.715 19	0.299 23	0.376 26	0.328 29	0.319 23	0.944 10	0.285 25	0.164 22	0.216 30	0.229 26	0.484 22	0.545 24	0.456 23	0.755 21	0.709 21	0.475 24
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPN	0.447 25	0.679 15	0.604 26	0.578 23	0.380 24	0.682 23	0.291 24	0.106 32	0.483 17	0.258 30	0.920 22	0.258 27	0.025 32	0.231 29	0.325 22	0.480 23	0.560 23	0.463 22	0.725 23	0.666 26	0.231 32
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
SurfaceConvPF	0.442 26	0.505 26	0.622 23	0.380 31	0.342 28	0.654 25	0.227 30	0.397 25	0.367 27	0.276 27	0.924 20	0.240 28	0.198 18	0.359 22	0.262 24	0.366 28	0.581 21	0.435 26	0.640 27	0.668 25	0.398 26
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
PNET2	0.442 26	0.548 25	0.548 27	0.597 22	0.363 26	0.628 28	0.300 22	0.292 27	0.374 26	0.307 24	0.881 28	0.268 26	0.186 20	0.238 27	0.204 29	0.407 27	0.506 30	0.449 24	0.667 26	0.620 28	0.462 25

Tangent Convolutions	0.438 28	0.437 30	0.646 20	0.474 27	0.369 25	0.645 26	0.353 19	0.258 29	0.282 31	0.279 26	0.918 23	0.298 24	0.147 25	0.283 24	0.294 23	0.487 21	0.562 22	0.427 27	0.619 28	0.633 27	0.352 28
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
SPLAT Net	0.393 29	0.472 29	0.511 29	0.606 19	0.311 30	0.656 24	0.245 28	0.405 24	0.328 29	0.197 31	0.927 19	0.227 30	0.000 34	0.001 34	0.249 25	0.271 33	0.510 27	0.383 30	0.593 29	0.699 22	0.267 30
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF	0.383 30	0.297 32	0.491 30	0.432 29	0.358 27	0.612 29	0.274 25	0.116 31	0.411 24	0.265 28	0.904 27	0.229 29	0.079 29	0.250 25	0.185 30	0.320 31	0.510 27	0.385 29	0.548 30	0.597 31	0.394 27

PointNet++	0.339 31	0.584 21	0.478 31	0.458 28	0.256 32	0.360 33	0.250 26	0.247 30	0.278 32	0.261 29	0.677 33	0.183 31	0.117 26	0.212 31	0.145 32	0.364 29	0.346 33	0.232 33	0.548 30	0.523 32	0.252 31
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNet	0.308 32	0.353 31	0.290 33	0.278 33	0.166 33	0.553 31	0.169 33	0.286 28	0.147 33	0.148 33	0.908 25	0.182 32	0.064 30	0.023 33	0.018 34	0.354 30	0.363 31	0.345 31	0.546 32	0.685 23	0.278 29

ScanNet	0.306 33	0.203 33	0.366 32	0.501 25	0.311 30	0.524 32	0.211 32	0.002 34	0.342 28	0.189 32	0.786 32	0.145 33	0.102 28	0.245 26	0.152 31	0.318 32	0.348 32	0.300 32	0.460 33	0.437 33	0.182 33
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
ERROR	0.054 34	0.000 34	0.041 34	0.172 34	0.030 34	0.062 34	0.001 34	0.035 33	0.004 34	0.051 34	0.143 34	0.019 34	0.003 33	0.041 32	0.050 33	0.003 34	0.054 34	0.018 34	0.005 34	0.264 34	0.082 34

3D Semantic label benchmark