3D Semantic Label Results - ScanNet Benchmark

The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).

This table lists the benchmark results for the 3D semantic label scenario.

Method	avg iou	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	floor	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	wall	window

OccuSeg+Semantic	0.764 1	0.758 16	0.796 4	0.839 2	0.746 1	0.907 1	0.562 1	0.850 3	0.680 1	0.672 1	0.978 1	0.610 1	0.335 1	0.777 1	0.819 7	0.847 1	0.830 1	0.691 1	0.972 1	0.885 1	0.727 1

MinkowskiNet	0.736 2	0.859 3	0.818 2	0.832 3	0.709 4	0.840 3	0.521 4	0.853 2	0.660 2	0.643 2	0.951 6	0.544 3	0.286 9	0.731 3	0.893 1	0.675 9	0.772 6	0.683 2	0.874 14	0.852 3	0.727 1
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet	0.725 3	0.647 25	0.821 1	0.846 1	0.721 3	0.869 2	0.533 2	0.754 10	0.603 6	0.614 3	0.955 2	0.572 2	0.325 2	0.710 4	0.870 2	0.724 4	0.823 2	0.628 5	0.934 3	0.865 2	0.683 5

MatchingNet	0.724 4	0.812 10	0.812 3	0.810 5	0.735 2	0.834 4	0.495 7	0.860 1	0.572 12	0.602 5	0.954 3	0.512 4	0.280 10	0.757 2	0.845 5	0.725 3	0.780 5	0.606 8	0.937 2	0.851 4	0.700 4

JSENet	0.699 5	0.881 1	0.762 9	0.821 4	0.667 6	0.800 11	0.522 3	0.792 6	0.613 4	0.607 4	0.935 23	0.492 7	0.205 20	0.576 12	0.853 3	0.691 6	0.758 8	0.652 4	0.872 16	0.828 6	0.649 7

CU-Hybrid Net	0.693 6	0.596 29	0.789 5	0.803 7	0.677 5	0.800 11	0.469 11	0.846 4	0.554 16	0.591 7	0.948 12	0.500 5	0.316 4	0.609 7	0.847 4	0.732 2	0.808 3	0.593 9	0.894 8	0.839 5	0.652 6

KP-FCNN	0.684 7	0.847 5	0.758 12	0.784 9	0.647 8	0.814 8	0.473 9	0.772 8	0.605 5	0.594 6	0.935 23	0.450 13	0.181 27	0.587 9	0.805 8	0.690 7	0.785 4	0.614 6	0.882 10	0.819 7	0.632 10
H. Thomas, C. Qi, J. Deschaud, B. Marcotegui, F. Goulette, L. Guibas.: KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV 2019
PointConv	0.666 8	0.781 12	0.759 11	0.699 17	0.644 9	0.822 7	0.475 8	0.779 7	0.564 15	0.504 18	0.953 4	0.428 19	0.203 22	0.586 10	0.754 9	0.661 12	0.753 9	0.588 10	0.902 4	0.813 9	0.642 8
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
PointASNL	0.666 8	0.703 21	0.781 6	0.751 15	0.655 7	0.830 5	0.471 10	0.769 9	0.474 26	0.537 10	0.951 6	0.475 8	0.279 11	0.635 6	0.698 13	0.675 9	0.751 10	0.553 19	0.816 24	0.806 10	0.703 3
Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, Shuguang Cui: PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Sampling. CVPR 2020
DCM-Net	0.658 10	0.778 13	0.702 18	0.806 6	0.619 11	0.813 9	0.468 12	0.693 18	0.494 23	0.524 14	0.941 17	0.449 14	0.298 5	0.510 18	0.821 6	0.675 9	0.727 11	0.568 13	0.826 21	0.803 12	0.637 9
Jonas Schult, Francis Engelmann, Theodora Kontogianni, Bastian Leibe: DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes. CVPR 2020 [Oral]
MVPNet	0.641 11	0.831 7	0.715 17	0.671 20	0.590 15	0.781 17	0.394 22	0.679 21	0.642 3	0.553 9	0.937 21	0.462 10	0.256 12	0.649 5	0.406 29	0.626 17	0.691 17	0.666 3	0.877 11	0.792 17	0.608 14
Maximilian Jaritz, Jiayuan Gu, Hao Su: Multi-view PointNet for 3D Scene Understanding. GMDL Workshop, ICCV 2019
FPConv	0.639 12	0.785 11	0.760 10	0.713 16	0.603 13	0.798 13	0.392 23	0.534 31	0.603 6	0.524 14	0.948 12	0.457 11	0.250 13	0.538 15	0.723 11	0.598 20	0.696 16	0.614 6	0.872 16	0.799 13	0.567 22
Yiqun Lin, Zizheng Yan, Haibin Huang, Dong Du, Ligang Liu, Shuguang Cui, Xiaoguang Han: FPConv: Learning Local Flattening for Point Convolution. CVPR 2020
SConv	0.636 13	0.830 8	0.697 21	0.752 14	0.572 19	0.780 18	0.445 15	0.716 15	0.529 17	0.530 11	0.951 6	0.446 15	0.170 28	0.507 20	0.666 15	0.636 16	0.682 19	0.541 23	0.886 9	0.799 13	0.594 18

joint point-based	0.634 14	0.614 27	0.778 7	0.667 22	0.633 10	0.825 6	0.420 18	0.804 5	0.467 28	0.561 8	0.951 6	0.494 6	0.291 6	0.566 13	0.458 24	0.579 24	0.764 7	0.559 18	0.838 20	0.814 8	0.598 17
Hung-Yueh Chiang, Yen-Liang Lin, Yueh-Cheng Liu, Winston H. Hsu: A Unified Point-Based Framework for 3D Segmentation. 3DV 2019
MCCNN	0.633 15	0.866 2	0.731 15	0.771 10	0.576 18	0.809 10	0.410 20	0.684 19	0.497 22	0.491 19	0.949 10	0.466 9	0.105 36	0.581 11	0.646 16	0.620 18	0.680 20	0.542 22	0.817 23	0.795 15	0.618 12
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
3DSM_DMMF	0.631 16	0.626 26	0.745 13	0.801 8	0.607 12	0.751 23	0.506 5	0.729 13	0.565 14	0.491 19	0.866 38	0.434 16	0.197 24	0.595 8	0.630 17	0.709 5	0.705 13	0.560 17	0.875 13	0.740 28	0.491 31

FusionAwareConv	0.630 17	0.604 28	0.741 14	0.766 12	0.590 15	0.747 24	0.501 6	0.734 11	0.503 21	0.527 12	0.919 31	0.454 12	0.323 3	0.550 14	0.420 28	0.678 8	0.688 18	0.544 21	0.896 7	0.795 15	0.627 11
Jiazhao Zhang, Chenyang Zhu, Lintao Zheng, Kai Xu: Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentation. CVPR 2020
SIConv	0.625 18	0.830 8	0.694 23	0.757 13	0.563 20	0.772 20	0.448 13	0.647 25	0.520 18	0.509 16	0.949 10	0.431 18	0.191 25	0.496 23	0.614 18	0.647 13	0.672 21	0.535 24	0.876 12	0.783 18	0.571 20

FRPointConv	0.619 19	0.762 15	0.697 21	0.602 28	0.547 21	0.782 16	0.432 17	0.650 23	0.581 10	0.458 24	0.952 5	0.428 19	0.130 33	0.496 23	0.747 10	0.641 14	0.710 12	0.568 13	0.902 4	0.782 19	0.511 28

HPEIN	0.618 20	0.729 18	0.668 26	0.647 23	0.597 14	0.766 21	0.414 19	0.680 20	0.520 18	0.525 13	0.946 14	0.432 17	0.215 18	0.493 25	0.599 19	0.638 15	0.617 29	0.570 12	0.897 6	0.806 10	0.605 15
Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, Jiaya Jia: Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation. ICCV 2019
SPH3D-GCN	0.610 21	0.858 4	0.772 8	0.489 35	0.532 22	0.792 14	0.404 21	0.643 26	0.570 13	0.507 17	0.935 23	0.414 21	0.046 41	0.510 18	0.702 12	0.602 19	0.705 13	0.549 20	0.859 19	0.773 21	0.534 25
Huan Lei, Naveed Akhtar, and Ajmal Mian: Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds. TPAMI 2020
LAP-D	0.594 22	0.720 19	0.692 24	0.637 25	0.456 29	0.773 19	0.391 25	0.730 12	0.587 8	0.445 26	0.940 19	0.381 24	0.288 7	0.434 27	0.453 25	0.591 22	0.649 23	0.581 11	0.777 28	0.749 27	0.610 13

DPC	0.592 23	0.720 19	0.700 19	0.602 28	0.480 26	0.762 22	0.380 27	0.713 16	0.585 9	0.437 27	0.940 19	0.369 26	0.288 7	0.434 27	0.509 22	0.590 23	0.639 27	0.567 15	0.772 29	0.755 25	0.592 19
Francis Engelmann, Theodora Kontogianni, Bastian Leibe: Dilated Point Convolutions: On the Receptive Field Size of Point Convolutions on 3D Point Clouds. ICRA 2020
SegGCN	0.589 24	0.833 6	0.731 15	0.539 33	0.514 23	0.789 15	0.448 13	0.467 32	0.573 11	0.484 21	0.936 22	0.396 22	0.061 40	0.501 21	0.507 23	0.594 21	0.700 15	0.563 16	0.874 14	0.771 22	0.493 30
Huan Lei, Naveed Akhtar, and Ajmal Mian: SegGCN: Efficient 3D Point Cloud Segmentation with Fuzzy Spherical Kernel. CVPR 2020
CCRFNet	0.589 24	0.766 14	0.659 29	0.683 19	0.470 28	0.740 26	0.387 26	0.620 27	0.490 24	0.476 22	0.922 29	0.355 29	0.245 14	0.511 17	0.511 21	0.571 25	0.643 25	0.493 28	0.872 16	0.762 23	0.600 16

TextureNet	0.566 26	0.672 23	0.664 27	0.671 20	0.494 24	0.719 27	0.445 15	0.678 22	0.411 33	0.396 28	0.935 23	0.356 28	0.225 16	0.412 29	0.535 20	0.565 26	0.636 28	0.464 30	0.794 27	0.680 33	0.568 21
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
DVVNet	0.562 27	0.648 24	0.700 19	0.770 11	0.586 17	0.687 31	0.333 29	0.650 23	0.514 20	0.475 23	0.906 35	0.359 27	0.223 17	0.340 32	0.442 27	0.422 35	0.668 22	0.501 26	0.708 33	0.779 20	0.534 25

Pointnet++ & Feature	0.557 28	0.735 17	0.661 28	0.686 18	0.491 25	0.744 25	0.392 23	0.539 30	0.451 29	0.375 31	0.946 14	0.376 25	0.205 20	0.403 30	0.356 31	0.553 27	0.643 25	0.497 27	0.824 22	0.756 24	0.515 27

PanopticFusion-label	0.529 29	0.491 36	0.688 25	0.604 27	0.386 32	0.632 36	0.225 40	0.705 17	0.434 31	0.293 35	0.815 39	0.348 30	0.241 15	0.499 22	0.669 14	0.507 28	0.649 23	0.442 34	0.796 26	0.602 39	0.561 23
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
3DMV, FTSDF	0.501 30	0.558 33	0.608 35	0.424 40	0.478 27	0.690 30	0.246 36	0.586 28	0.468 27	0.450 25	0.911 33	0.394 23	0.160 30	0.438 26	0.212 37	0.432 34	0.541 35	0.475 29	0.742 31	0.727 29	0.477 32

PCNN	0.498 31	0.559 32	0.644 32	0.560 32	0.420 31	0.711 29	0.229 38	0.414 33	0.436 30	0.352 32	0.941 17	0.324 31	0.155 31	0.238 37	0.387 30	0.493 29	0.529 36	0.509 25	0.813 25	0.751 26	0.504 29

3DMV	0.484 32	0.484 37	0.538 38	0.643 24	0.424 30	0.606 39	0.310 30	0.574 29	0.433 32	0.378 30	0.796 40	0.301 32	0.214 19	0.537 16	0.208 38	0.472 33	0.507 39	0.413 37	0.693 34	0.602 39	0.539 24
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGB	0.458 33	0.577 31	0.611 34	0.356 42	0.321 38	0.715 28	0.299 32	0.376 36	0.328 39	0.319 33	0.944 16	0.285 34	0.164 29	0.216 40	0.229 36	0.484 31	0.545 34	0.456 32	0.755 30	0.709 30	0.475 33
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPN	0.447 34	0.679 22	0.604 36	0.578 31	0.380 33	0.682 32	0.291 33	0.106 42	0.483 25	0.258 40	0.920 30	0.258 36	0.025 42	0.231 39	0.325 32	0.480 32	0.560 33	0.463 31	0.725 32	0.666 35	0.231 42
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
SurfaceConvPF	0.442 35	0.505 35	0.622 33	0.380 41	0.342 37	0.654 34	0.227 39	0.397 35	0.367 37	0.276 37	0.924 28	0.240 38	0.198 23	0.359 31	0.262 34	0.366 37	0.581 31	0.435 35	0.640 36	0.668 34	0.398 35
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
PNET2	0.442 35	0.548 34	0.548 37	0.597 30	0.363 35	0.628 37	0.300 31	0.292 37	0.374 36	0.307 34	0.881 37	0.268 35	0.186 26	0.238 37	0.204 39	0.407 36	0.506 40	0.449 33	0.667 35	0.620 37	0.462 34

Tangent Convolutions	0.438 37	0.437 40	0.646 31	0.474 37	0.369 34	0.645 35	0.353 28	0.258 39	0.282 41	0.279 36	0.918 32	0.298 33	0.147 32	0.283 34	0.294 33	0.487 30	0.562 32	0.427 36	0.619 37	0.633 36	0.352 37
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
subcloud_weak	0.411 38	0.479 38	0.650 30	0.475 36	0.285 41	0.519 42	0.087 43	0.725 14	0.396 35	0.386 29	0.621 43	0.250 37	0.117 34	0.338 33	0.443 26	0.188 43	0.594 30	0.369 40	0.377 43	0.616 38	0.306 38

SPLAT Net	0.393 39	0.472 39	0.511 39	0.606 26	0.311 39	0.656 33	0.245 37	0.405 34	0.328 39	0.197 41	0.927 27	0.227 40	0.000 44	0.001 44	0.249 35	0.271 42	0.510 37	0.383 39	0.593 38	0.699 31	0.267 40
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF	0.383 40	0.297 42	0.491 40	0.432 39	0.358 36	0.612 38	0.274 34	0.116 41	0.411 33	0.265 38	0.904 36	0.229 39	0.079 38	0.250 35	0.185 40	0.320 40	0.510 37	0.385 38	0.548 39	0.597 41	0.394 36

PointNet++	0.339 41	0.584 30	0.478 41	0.458 38	0.256 42	0.360 43	0.250 35	0.247 40	0.278 42	0.261 39	0.677 42	0.183 41	0.117 34	0.212 41	0.145 42	0.364 38	0.346 43	0.232 43	0.548 39	0.523 42	0.252 41
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNet	0.308 42	0.353 41	0.290 43	0.278 43	0.166 43	0.553 40	0.169 42	0.286 38	0.147 43	0.148 43	0.908 34	0.182 42	0.064 39	0.023 43	0.018 44	0.354 39	0.363 41	0.345 41	0.546 41	0.685 32	0.278 39

ScanNet	0.306 43	0.203 43	0.366 42	0.501 34	0.311 39	0.524 41	0.211 41	0.002 44	0.342 38	0.189 42	0.786 41	0.145 43	0.102 37	0.245 36	0.152 41	0.318 41	0.348 42	0.300 42	0.460 42	0.437 43	0.182 43
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
ERROR	0.054 44	0.000 44	0.041 44	0.172 44	0.030 44	0.062 44	0.001 44	0.035 43	0.004 44	0.051 44	0.143 44	0.019 44	0.003 43	0.041 42	0.050 43	0.003 44	0.054 44	0.018 44	0.005 44	0.264 44	0.082 44

3D Semantic label benchmark