3D Semantic Label Results - ScanNet Benchmark

The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).

This table lists the benchmark results for the 3D semantic label scenario.

Method	avg iou	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	floor	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	wall	window

OccuSeg+Semantic	0.764 1	0.758 13	0.796 3	0.839 2	0.746 1	0.907 1	0.562 1	0.850 2	0.680 1	0.672 1	0.978 1	0.610 1	0.335 1	0.777 1	0.819 6	0.847 1	0.830 1	0.691 1	0.972 1	0.885 1	0.727 1

MinkowskiNet	0.736 2	0.859 3	0.818 2	0.832 3	0.709 3	0.840 3	0.521 4	0.853 1	0.660 2	0.643 2	0.951 4	0.544 3	0.286 9	0.731 2	0.893 1	0.675 8	0.772 5	0.683 2	0.874 12	0.852 3	0.727 1
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet	0.725 3	0.647 22	0.821 1	0.846 1	0.721 2	0.869 2	0.533 2	0.754 9	0.603 6	0.614 3	0.955 2	0.572 2	0.325 2	0.710 3	0.870 2	0.724 3	0.823 2	0.628 5	0.934 2	0.865 2	0.683 4

JSENet	0.699 4	0.881 1	0.762 8	0.821 4	0.667 5	0.800 10	0.522 3	0.792 5	0.613 4	0.607 4	0.935 20	0.492 6	0.205 19	0.576 11	0.853 3	0.691 5	0.758 7	0.652 4	0.872 13	0.828 5	0.649 6

CU-Hybrid Net	0.693 5	0.596 26	0.789 4	0.803 6	0.677 4	0.800 10	0.469 10	0.846 3	0.554 13	0.591 6	0.948 10	0.500 4	0.316 4	0.609 6	0.847 4	0.732 2	0.808 3	0.593 8	0.894 6	0.839 4	0.652 5

KP-FCNN	0.684 6	0.847 5	0.758 11	0.784 8	0.647 7	0.814 7	0.473 8	0.772 7	0.605 5	0.594 5	0.935 20	0.450 12	0.181 26	0.587 8	0.805 7	0.690 6	0.785 4	0.614 6	0.882 8	0.819 6	0.632 9
H. Thomas, C. Qi, J. Deschaud, B. Marcotegui, F. Goulette, L. Guibas.: KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV 2019
PointConv	0.666 7	0.781 10	0.759 10	0.699 16	0.644 8	0.822 6	0.475 7	0.779 6	0.564 12	0.504 17	0.953 3	0.428 18	0.203 21	0.586 9	0.754 8	0.661 11	0.753 8	0.588 9	0.902 3	0.813 8	0.642 7
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
PointASNL	0.666 7	0.703 18	0.781 5	0.751 14	0.655 6	0.830 4	0.471 9	0.769 8	0.474 23	0.537 9	0.951 4	0.475 7	0.279 10	0.635 5	0.698 11	0.675 8	0.751 9	0.553 16	0.816 21	0.806 9	0.703 3
Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, Shuguang Cui: PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Sampling. CVPR 2020
DualMeshNet	0.658 9	0.778 11	0.702 16	0.806 5	0.619 10	0.813 8	0.468 11	0.693 16	0.494 20	0.524 13	0.941 15	0.449 13	0.298 5	0.510 17	0.821 5	0.675 8	0.727 10	0.568 12	0.826 18	0.803 11	0.637 8
Jonas Schult, Francis Engelmann, Theodora Kontogianni, Bastian Leibe: DualMeshNet: Joint Geodesic and Euclidean Convolutions for 3D Semantic Segmentation. CVPR 2020
MVPNet	0.641 10	0.831 6	0.715 15	0.671 19	0.590 14	0.781 14	0.394 19	0.679 19	0.642 3	0.553 8	0.937 19	0.462 9	0.256 11	0.649 4	0.406 25	0.626 15	0.691 14	0.666 3	0.877 9	0.792 16	0.608 13
Maximilian Jaritz, Jiayuan Gu, Hao Su: Multi-view PointNet for 3D Scene Understanding. GMDL Workshop, ICCV 2019
FPConv	0.639 11	0.785 9	0.760 9	0.713 15	0.603 12	0.798 12	0.392 20	0.534 28	0.603 6	0.524 13	0.948 10	0.457 10	0.250 12	0.538 14	0.723 9	0.598 18	0.696 13	0.614 6	0.872 13	0.799 12	0.567 21
Yiqun Lin, Zizheng Yan, Haibin Huang, Dong Du, Ligang Liu, Shuguang Cui, Xiaoguang Han: FPConv: Learning Local Flattening for Point Convolution. CVPR 2020
SConv	0.636 12	0.830 7	0.697 19	0.752 13	0.572 18	0.780 15	0.445 13	0.716 13	0.529 14	0.530 10	0.951 4	0.446 14	0.170 27	0.507 19	0.666 13	0.636 14	0.682 16	0.541 20	0.886 7	0.799 12	0.594 17

joint point-based	0.634 13	0.614 24	0.778 6	0.667 21	0.633 9	0.825 5	0.420 15	0.804 4	0.467 25	0.561 7	0.951 4	0.494 5	0.291 6	0.566 12	0.458 21	0.579 21	0.764 6	0.559 15	0.838 17	0.814 7	0.598 16
Hung-Yueh Chiang, Yen-Liang Lin, Yueh-Cheng Liu, Winston H. Hsu: A Unified Point-Based Framework for 3D Segmentation. 3DV 2019
MCCNN	0.633 14	0.866 2	0.731 14	0.771 9	0.576 17	0.809 9	0.410 17	0.684 17	0.497 19	0.491 18	0.949 8	0.466 8	0.105 33	0.581 10	0.646 14	0.620 16	0.680 17	0.542 19	0.817 20	0.795 14	0.618 11
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
3DSM_DMMF	0.631 15	0.626 23	0.745 12	0.801 7	0.607 11	0.751 20	0.506 5	0.729 12	0.565 11	0.491 18	0.866 35	0.434 15	0.197 23	0.595 7	0.630 15	0.709 4	0.705 11	0.560 14	0.875 11	0.740 25	0.491 28

FusionAwareConv	0.630 16	0.604 25	0.741 13	0.766 11	0.590 14	0.747 21	0.501 6	0.734 10	0.503 18	0.527 11	0.919 28	0.454 11	0.323 3	0.550 13	0.420 24	0.678 7	0.688 15	0.544 18	0.896 5	0.795 14	0.627 10
Jiazhao Zhang, Chenyang Zhu, Lintao Zheng, Kai Xu: Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentation. CVPR2020
SIConv	0.625 17	0.830 7	0.694 20	0.757 12	0.563 19	0.772 17	0.448 12	0.647 22	0.520 15	0.509 15	0.949 8	0.431 17	0.191 24	0.496 21	0.614 16	0.647 12	0.672 18	0.535 21	0.876 10	0.783 17	0.571 19

HPEIN	0.618 18	0.729 15	0.668 23	0.647 22	0.597 13	0.766 18	0.414 16	0.680 18	0.520 15	0.525 12	0.946 12	0.432 16	0.215 17	0.493 22	0.599 17	0.638 13	0.617 26	0.570 11	0.897 4	0.806 9	0.605 14
Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, Jiaya Jia: Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation. ICCV 2019
SPH3D-GCN	0.610 19	0.858 4	0.772 7	0.489 32	0.532 20	0.792 13	0.404 18	0.643 23	0.570 10	0.507 16	0.935 20	0.414 19	0.046 37	0.510 17	0.702 10	0.602 17	0.705 11	0.549 17	0.859 16	0.773 19	0.534 24
Huan Lei, Naveed Akhtar, and Ajmal Mian: Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds.
LAP-D	0.594 20	0.720 16	0.692 21	0.637 24	0.456 26	0.773 16	0.391 22	0.730 11	0.587 8	0.445 23	0.940 17	0.381 21	0.288 7	0.434 24	0.453 22	0.591 19	0.649 20	0.581 10	0.777 25	0.749 24	0.610 12

DPC	0.592 21	0.720 16	0.700 17	0.602 27	0.480 23	0.762 19	0.380 24	0.713 14	0.585 9	0.437 24	0.940 17	0.369 23	0.288 7	0.434 24	0.509 20	0.590 20	0.639 24	0.567 13	0.772 26	0.755 22	0.592 18
Francis Engelmann, Theodora Kontogianni, Bastian Leibe: Dilated Point Convolutions. arXiv
CCRFNet	0.589 22	0.766 12	0.659 26	0.683 18	0.470 25	0.740 23	0.387 23	0.620 24	0.490 21	0.476 20	0.922 26	0.355 26	0.245 13	0.511 16	0.511 19	0.571 22	0.643 22	0.493 25	0.872 13	0.762 20	0.600 15

TextureNet	0.566 23	0.672 20	0.664 24	0.671 19	0.494 21	0.719 24	0.445 13	0.678 20	0.411 30	0.396 25	0.935 20	0.356 25	0.225 15	0.412 26	0.535 18	0.565 23	0.636 25	0.464 27	0.794 24	0.680 30	0.568 20
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
DVVNet	0.562 24	0.648 21	0.700 17	0.770 10	0.586 16	0.687 28	0.333 26	0.650 21	0.514 17	0.475 21	0.906 32	0.359 24	0.223 16	0.340 29	0.442 23	0.422 32	0.668 19	0.501 23	0.708 30	0.779 18	0.534 24

Pointnet++ & Feature	0.557 25	0.735 14	0.661 25	0.686 17	0.491 22	0.744 22	0.392 20	0.539 27	0.451 26	0.375 27	0.946 12	0.376 22	0.205 19	0.403 27	0.356 27	0.553 24	0.643 22	0.497 24	0.824 19	0.756 21	0.515 26

PanopticFusion-label	0.529 26	0.491 33	0.688 22	0.604 26	0.386 29	0.632 33	0.225 37	0.705 15	0.434 28	0.293 31	0.815 36	0.348 27	0.241 14	0.499 20	0.669 12	0.507 25	0.649 20	0.442 31	0.796 23	0.602 35	0.561 22
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
3DMV, FTSDF	0.501 27	0.558 30	0.608 31	0.424 36	0.478 24	0.690 27	0.246 33	0.586 25	0.468 24	0.450 22	0.911 30	0.394 20	0.160 29	0.438 23	0.212 33	0.432 31	0.541 31	0.475 26	0.742 28	0.727 26	0.477 29

PCNN	0.498 28	0.559 29	0.644 28	0.560 30	0.420 28	0.711 26	0.229 35	0.414 29	0.436 27	0.352 28	0.941 15	0.324 28	0.155 30	0.238 33	0.387 26	0.493 26	0.529 32	0.509 22	0.813 22	0.751 23	0.504 27

3DMV	0.484 29	0.484 34	0.538 34	0.643 23	0.424 27	0.606 36	0.310 27	0.574 26	0.433 29	0.378 26	0.796 37	0.301 29	0.214 18	0.537 15	0.208 34	0.472 30	0.507 35	0.413 34	0.693 31	0.602 35	0.539 23
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGB	0.458 30	0.577 28	0.611 30	0.356 38	0.321 35	0.715 25	0.299 29	0.376 32	0.328 35	0.319 29	0.944 14	0.285 31	0.164 28	0.216 36	0.229 32	0.484 28	0.545 30	0.456 29	0.755 27	0.709 27	0.475 30
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPN	0.447 31	0.679 19	0.604 32	0.578 29	0.380 30	0.682 29	0.291 30	0.106 38	0.483 22	0.258 36	0.920 27	0.258 33	0.025 38	0.231 35	0.325 28	0.480 29	0.560 29	0.463 28	0.725 29	0.666 32	0.231 38
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
PNET2	0.442 32	0.548 31	0.548 33	0.597 28	0.363 32	0.628 34	0.300 28	0.292 33	0.374 32	0.307 30	0.881 34	0.268 32	0.186 25	0.238 33	0.204 35	0.407 33	0.506 36	0.449 30	0.667 32	0.620 34	0.462 31

SurfaceConvPF	0.442 32	0.505 32	0.622 29	0.380 37	0.342 34	0.654 31	0.227 36	0.397 31	0.367 33	0.276 33	0.924 25	0.240 34	0.198 22	0.359 28	0.262 30	0.366 34	0.581 27	0.435 32	0.640 33	0.668 31	0.398 32
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
Tangent Convolutions	0.438 34	0.437 36	0.646 27	0.474 33	0.369 31	0.645 32	0.353 25	0.258 35	0.282 37	0.279 32	0.918 29	0.298 30	0.147 31	0.283 30	0.294 29	0.487 27	0.562 28	0.427 33	0.619 34	0.633 33	0.352 34
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
SPLAT Net	0.393 35	0.472 35	0.511 35	0.606 25	0.311 36	0.656 30	0.245 34	0.405 30	0.328 35	0.197 37	0.927 24	0.227 36	0.000 40	0.001 40	0.249 31	0.271 39	0.510 33	0.383 36	0.593 35	0.699 28	0.267 36
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF	0.383 36	0.297 38	0.491 36	0.432 35	0.358 33	0.612 35	0.274 31	0.116 37	0.411 30	0.265 34	0.904 33	0.229 35	0.079 35	0.250 31	0.185 36	0.320 37	0.510 33	0.385 35	0.548 36	0.597 37	0.394 33

PointNet++	0.339 37	0.584 27	0.478 37	0.458 34	0.256 38	0.360 39	0.250 32	0.247 36	0.278 38	0.261 35	0.677 39	0.183 37	0.117 32	0.212 37	0.145 38	0.364 35	0.346 39	0.232 39	0.548 36	0.523 38	0.252 37
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNet	0.308 38	0.353 37	0.290 39	0.278 39	0.166 39	0.553 37	0.169 39	0.286 34	0.147 39	0.148 39	0.908 31	0.182 38	0.064 36	0.023 39	0.018 40	0.354 36	0.363 37	0.345 37	0.546 38	0.685 29	0.278 35

ScanNet	0.306 39	0.203 39	0.366 38	0.501 31	0.311 36	0.524 38	0.211 38	0.002 40	0.342 34	0.189 38	0.786 38	0.145 39	0.102 34	0.245 32	0.152 37	0.318 38	0.348 38	0.300 38	0.460 39	0.437 39	0.182 39
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
ERROR	0.054 40	0.000 40	0.041 40	0.172 40	0.030 40	0.062 40	0.001 40	0.035 39	0.004 40	0.051 40	0.143 40	0.019 40	0.003 39	0.041 38	0.050 39	0.003 40	0.054 40	0.018 40	0.005 40	0.264 40	0.082 40

3D Semantic label benchmark