3D Semantic Label Results - ScanNet Benchmark

The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).

This table lists the benchmark results for the 3D semantic label scenario.

Method	avg iou	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	floor	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	wall	window

OccuSeg+Semantic	0.764 1	0.758 18	0.796 6	0.839 3	0.746 2	0.907 1	0.562 1	0.850 4	0.680 2	0.672 1	0.978 1	0.610 1	0.335 2	0.777 1	0.819 10	0.847 1	0.830 1	0.691 3	0.972 1	0.885 1	0.727 2

BPNet	0.749 2	0.909 1	0.818 3	0.811 6	0.752 1	0.839 5	0.485 9	0.842 6	0.673 3	0.644 3	0.957 2	0.528 5	0.305 7	0.773 2	0.859 3	0.788 2	0.818 3	0.693 2	0.916 5	0.856 4	0.723 4

Virtual MVFusion	0.746 3	0.771 15	0.819 2	0.848 1	0.702 6	0.865 3	0.397 28	0.899 1	0.699 1	0.664 2	0.948 16	0.588 2	0.330 3	0.746 4	0.851 6	0.764 3	0.796 5	0.704 1	0.935 3	0.866 2	0.728 1
Abhijit Kundu, Xiaoqi Yin, Alireza Fathi, David Ross, Brian Brewington, Thomas Funkhouser, Caroline Pantofaru: Virtual Multi-view Fusion for 3D Semantic Segmentation. ECCV 2020
MinkowskiNet	0.736 4	0.859 4	0.818 3	0.832 4	0.709 5	0.840 4	0.521 4	0.853 3	0.660 4	0.643 4	0.951 8	0.544 4	0.286 12	0.731 5	0.893 1	0.675 12	0.772 8	0.683 4	0.874 17	0.852 5	0.727 2
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet	0.725 5	0.647 31	0.821 1	0.846 2	0.721 4	0.869 2	0.533 2	0.754 12	0.603 10	0.614 5	0.955 3	0.572 3	0.325 4	0.710 6	0.870 2	0.724 6	0.823 2	0.628 7	0.934 4	0.865 3	0.683 7

MatchingNet	0.724 6	0.812 11	0.812 5	0.810 7	0.735 3	0.834 6	0.495 8	0.860 2	0.572 15	0.602 7	0.954 4	0.512 7	0.280 13	0.757 3	0.845 8	0.725 5	0.780 7	0.606 12	0.937 2	0.851 6	0.700 6

JSENet	0.699 7	0.881 2	0.762 11	0.821 5	0.667 8	0.800 16	0.522 3	0.792 8	0.613 6	0.607 6	0.935 30	0.492 10	0.205 24	0.576 19	0.853 5	0.691 9	0.758 11	0.652 6	0.872 20	0.828 8	0.649 10
Zeyu HU, Mingmin Zhen, Xuyang BAI, Hongbo Fu, Chiew-lan Tai: JSENet: Joint Semantic Segmentation and Edge Detection Network for 3D Point Clouds. ECCV 2020
CU-Hybrid Net	0.693 8	0.596 36	0.789 7	0.803 9	0.677 7	0.800 16	0.469 13	0.846 5	0.554 21	0.591 9	0.948 16	0.500 8	0.316 6	0.609 12	0.847 7	0.732 4	0.808 4	0.593 13	0.894 10	0.839 7	0.652 9

FusionNet	0.688 9	0.704 26	0.741 16	0.754 16	0.656 9	0.829 8	0.501 6	0.741 13	0.609 8	0.548 12	0.950 12	0.522 6	0.371 1	0.633 10	0.756 12	0.715 7	0.771 9	0.623 8	0.861 25	0.814 10	0.658 8
Feihu Zhang, Jin Fang, Benjamin Wah, Philip Torr: Deep FusionNet for Point Cloud Semantic Segmentation. ECCV 2020
KP-FCNN	0.684 10	0.847 6	0.758 14	0.784 11	0.647 11	0.814 11	0.473 11	0.772 10	0.605 9	0.594 8	0.935 30	0.450 17	0.181 32	0.587 14	0.805 11	0.690 10	0.785 6	0.614 9	0.882 13	0.819 9	0.632 13
H. Thomas, C. Qi, J. Deschaud, B. Marcotegui, F. Goulette, L. Guibas.: KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV 2019
PointConv	0.666 11	0.781 13	0.759 13	0.699 20	0.644 12	0.822 10	0.475 10	0.779 9	0.564 18	0.504 22	0.953 5	0.428 25	0.203 26	0.586 16	0.754 13	0.661 15	0.753 12	0.588 14	0.902 7	0.813 12	0.642 11
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
PointASNL	0.666 11	0.703 27	0.781 8	0.751 18	0.655 10	0.830 7	0.471 12	0.769 11	0.474 33	0.537 13	0.951 8	0.475 11	0.279 14	0.635 8	0.698 20	0.675 12	0.751 13	0.553 23	0.816 31	0.806 13	0.703 5
Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, Shuguang Cui: PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Sampling. CVPR 2020
DCM-Net	0.658 13	0.778 14	0.702 22	0.806 8	0.619 14	0.813 12	0.468 14	0.693 22	0.494 30	0.524 17	0.941 24	0.449 18	0.298 8	0.510 25	0.821 9	0.675 12	0.727 15	0.568 17	0.826 28	0.803 15	0.637 12
Jonas Schult, Francis Engelmann, Theodora Kontogianni, Bastian Leibe: DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes. CVPR 2020 [Oral]
MVPNet	0.641 14	0.831 8	0.715 20	0.671 27	0.590 19	0.781 24	0.394 29	0.679 25	0.642 5	0.553 11	0.937 28	0.462 13	0.256 15	0.649 7	0.406 36	0.626 20	0.691 23	0.666 5	0.877 14	0.792 22	0.608 17
Maximilian Jaritz, Jiayuan Gu, Hao Su: Multi-view PointNet for 3D Scene Understanding. GMDL Workshop, ICCV 2019
PointMRNet	0.640 15	0.717 25	0.701 23	0.692 22	0.576 23	0.801 14	0.467 15	0.716 18	0.563 19	0.459 30	0.953 5	0.429 24	0.169 34	0.581 17	0.854 4	0.605 23	0.710 17	0.550 24	0.894 10	0.793 21	0.575 25

FPConv	0.639 16	0.785 12	0.760 12	0.713 19	0.603 16	0.798 18	0.392 30	0.534 38	0.603 10	0.524 17	0.948 16	0.457 14	0.250 17	0.538 22	0.723 14	0.598 27	0.696 22	0.614 9	0.872 20	0.799 16	0.567 28
Yiqun Lin, Zizheng Yan, Haibin Huang, Dong Du, Ligang Liu, Shuguang Cui, Xiaoguang Han: FPConv: Learning Local Flattening for Point Convolution. CVPR 2020
SConv	0.636 17	0.830 9	0.697 26	0.752 17	0.572 25	0.780 25	0.445 19	0.716 18	0.529 23	0.530 14	0.951 8	0.446 19	0.170 33	0.507 27	0.666 22	0.636 19	0.682 25	0.541 28	0.886 12	0.799 16	0.594 21

joint point-based	0.634 18	0.614 34	0.778 9	0.667 29	0.633 13	0.825 9	0.420 22	0.804 7	0.467 35	0.561 10	0.951 8	0.494 9	0.291 9	0.566 20	0.458 31	0.579 31	0.764 10	0.559 21	0.838 27	0.814 10	0.598 20
Hung-Yueh Chiang, Yen-Liang Lin, Yueh-Cheng Liu, Winston H. Hsu: A Unified Point-Based Framework for 3D Segmentation. 3DV 2019
MCCNN	0.633 19	0.866 3	0.731 18	0.771 12	0.576 23	0.809 13	0.410 24	0.684 23	0.497 29	0.491 25	0.949 13	0.466 12	0.105 43	0.581 17	0.646 23	0.620 21	0.680 26	0.542 27	0.817 30	0.795 19	0.618 15
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
PointMTL	0.632 20	0.731 21	0.688 29	0.675 25	0.591 18	0.784 23	0.444 21	0.565 36	0.610 7	0.492 24	0.949 13	0.456 15	0.254 16	0.587 14	0.706 17	0.599 26	0.665 29	0.612 11	0.868 24	0.791 23	0.579 24

3DSM_DMMF	0.631 21	0.626 33	0.745 15	0.801 10	0.607 15	0.751 30	0.506 5	0.729 16	0.565 17	0.491 25	0.866 45	0.434 21	0.197 28	0.595 13	0.630 24	0.709 8	0.705 19	0.560 20	0.875 16	0.740 35	0.491 37

APCF-Net	0.631 21	0.742 19	0.687 31	0.672 26	0.557 27	0.792 20	0.408 25	0.665 28	0.545 22	0.508 20	0.952 7	0.428 25	0.186 30	0.634 9	0.702 18	0.620 21	0.706 18	0.555 22	0.873 19	0.798 18	0.581 23

FusionAwareConv	0.630 23	0.604 35	0.741 16	0.766 14	0.590 19	0.747 31	0.501 6	0.734 14	0.503 28	0.527 15	0.919 38	0.454 16	0.323 5	0.550 21	0.420 35	0.678 11	0.688 24	0.544 26	0.896 9	0.795 19	0.627 14
Jiazhao Zhang, Chenyang Zhu, Lintao Zheng, Kai Xu: Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentation. CVPR 2020
SIConv	0.625 24	0.830 9	0.694 27	0.757 15	0.563 26	0.772 27	0.448 17	0.647 31	0.520 25	0.509 19	0.949 13	0.431 23	0.191 29	0.496 30	0.614 25	0.647 17	0.672 27	0.535 30	0.876 15	0.783 25	0.571 26

PointMRNet-lite	0.625 24	0.643 32	0.711 21	0.697 21	0.581 22	0.801 14	0.408 25	0.670 27	0.558 20	0.497 23	0.944 22	0.436 20	0.152 39	0.617 11	0.708 16	0.603 24	0.743 14	0.532 31	0.870 23	0.784 24	0.545 30

HPEIN	0.618 26	0.729 22	0.668 32	0.647 31	0.597 17	0.766 28	0.414 23	0.680 24	0.520 25	0.525 16	0.946 20	0.432 22	0.215 22	0.493 31	0.599 26	0.638 18	0.617 36	0.570 16	0.897 8	0.806 13	0.605 18
Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, Jiaya Jia: Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation. ICCV 2019
SPH3D-GCN	0.610 27	0.858 5	0.772 10	0.489 42	0.532 28	0.792 20	0.404 27	0.643 32	0.570 16	0.507 21	0.935 30	0.414 27	0.046 48	0.510 25	0.702 18	0.602 25	0.705 19	0.549 25	0.859 26	0.773 27	0.534 32
Huan Lei, Naveed Akhtar, and Ajmal Mian: Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds. TPAMI 2020
AttAN	0.609 28	0.760 17	0.667 33	0.649 30	0.521 29	0.793 19	0.457 16	0.648 30	0.528 24	0.434 34	0.947 19	0.401 28	0.153 38	0.454 32	0.721 15	0.648 16	0.717 16	0.536 29	0.904 6	0.765 29	0.485 38
Gege Zhang, Qinghua Ma, Licheng Jiao, Fang Liu and Qigong Sun: AttAN: Attention Adversarial Networks for 3D Point Cloud Semantic Segmentation. IJCAI2020
LAP-D	0.594 29	0.720 23	0.692 28	0.637 33	0.456 36	0.773 26	0.391 32	0.730 15	0.587 12	0.445 32	0.940 26	0.381 31	0.288 10	0.434 34	0.453 32	0.591 29	0.649 30	0.581 15	0.777 35	0.749 34	0.610 16

DPC	0.592 30	0.720 23	0.700 24	0.602 36	0.480 33	0.762 29	0.380 34	0.713 20	0.585 13	0.437 33	0.940 26	0.369 33	0.288 10	0.434 34	0.509 29	0.590 30	0.639 34	0.567 18	0.772 36	0.755 32	0.592 22
Francis Engelmann, Theodora Kontogianni, Bastian Leibe: Dilated Point Convolutions: On the Receptive Field Size of Point Convolutions on 3D Point Clouds. ICRA 2020
CCRFNet	0.589 31	0.766 16	0.659 36	0.683 24	0.470 35	0.740 33	0.387 33	0.620 33	0.490 31	0.476 28	0.922 36	0.355 36	0.245 18	0.511 24	0.511 28	0.571 32	0.643 32	0.493 35	0.872 20	0.762 30	0.600 19

SegGCN	0.589 31	0.833 7	0.731 18	0.539 40	0.514 30	0.789 22	0.448 17	0.467 39	0.573 14	0.484 27	0.936 29	0.396 29	0.061 47	0.501 28	0.507 30	0.594 28	0.700 21	0.563 19	0.874 17	0.771 28	0.493 36
Huan Lei, Naveed Akhtar, and Ajmal Mian: SegGCN: Efficient 3D Point Cloud Segmentation with Fuzzy Spherical Kernel. CVPR 2020
TextureNet	0.566 33	0.672 29	0.664 34	0.671 27	0.494 31	0.719 34	0.445 19	0.678 26	0.411 40	0.396 35	0.935 30	0.356 35	0.225 20	0.412 36	0.535 27	0.565 33	0.636 35	0.464 37	0.794 34	0.680 40	0.568 27
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
DVVNet	0.562 34	0.648 30	0.700 24	0.770 13	0.586 21	0.687 38	0.333 36	0.650 29	0.514 27	0.475 29	0.906 42	0.359 34	0.223 21	0.340 39	0.442 34	0.422 42	0.668 28	0.501 33	0.708 40	0.779 26	0.534 32

Pointnet++ & Feature	0.557 35	0.735 20	0.661 35	0.686 23	0.491 32	0.744 32	0.392 30	0.539 37	0.451 36	0.375 38	0.946 20	0.376 32	0.205 24	0.403 37	0.356 38	0.553 34	0.643 32	0.497 34	0.824 29	0.756 31	0.515 34

PanopticFusion-label	0.529 36	0.491 43	0.688 29	0.604 35	0.386 39	0.632 43	0.225 47	0.705 21	0.434 38	0.293 42	0.815 46	0.348 37	0.241 19	0.499 29	0.669 21	0.507 35	0.649 30	0.442 41	0.796 33	0.602 46	0.561 29
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
3DMV, FTSDF	0.501 37	0.558 40	0.608 42	0.424 47	0.478 34	0.690 37	0.246 43	0.586 34	0.468 34	0.450 31	0.911 40	0.394 30	0.160 36	0.438 33	0.212 44	0.432 41	0.541 42	0.475 36	0.742 38	0.727 36	0.477 39

PCNN	0.498 38	0.559 39	0.644 39	0.560 39	0.420 38	0.711 36	0.229 45	0.414 40	0.436 37	0.352 39	0.941 24	0.324 38	0.155 37	0.238 44	0.387 37	0.493 36	0.529 43	0.509 32	0.813 32	0.751 33	0.504 35

3DMV	0.484 39	0.484 44	0.538 45	0.643 32	0.424 37	0.606 46	0.310 37	0.574 35	0.433 39	0.378 37	0.796 47	0.301 39	0.214 23	0.537 23	0.208 45	0.472 40	0.507 46	0.413 44	0.693 41	0.602 46	0.539 31
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGB	0.458 40	0.577 38	0.611 41	0.356 49	0.321 45	0.715 35	0.299 39	0.376 43	0.328 46	0.319 40	0.944 22	0.285 41	0.164 35	0.216 47	0.229 43	0.484 38	0.545 41	0.456 39	0.755 37	0.709 37	0.475 40
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPN	0.447 41	0.679 28	0.604 43	0.578 38	0.380 40	0.682 39	0.291 40	0.106 49	0.483 32	0.258 47	0.920 37	0.258 43	0.025 49	0.231 46	0.325 39	0.480 39	0.560 40	0.463 38	0.725 39	0.666 42	0.231 49
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
PNET2	0.442 42	0.548 41	0.548 44	0.597 37	0.363 42	0.628 44	0.300 38	0.292 44	0.374 43	0.307 41	0.881 44	0.268 42	0.186 30	0.238 44	0.204 46	0.407 43	0.506 47	0.449 40	0.667 42	0.620 44	0.462 41

SurfaceConvPF	0.442 42	0.505 42	0.622 40	0.380 48	0.342 44	0.654 41	0.227 46	0.397 42	0.367 44	0.276 44	0.924 35	0.240 45	0.198 27	0.359 38	0.262 41	0.366 44	0.581 38	0.435 42	0.640 43	0.668 41	0.398 42
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
Tangent Convolutions	0.438 44	0.437 47	0.646 38	0.474 44	0.369 41	0.645 42	0.353 35	0.258 46	0.282 48	0.279 43	0.918 39	0.298 40	0.147 40	0.283 41	0.294 40	0.487 37	0.562 39	0.427 43	0.619 44	0.633 43	0.352 44
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
subcloud_weak	0.411 45	0.479 45	0.650 37	0.475 43	0.285 48	0.519 49	0.087 50	0.725 17	0.396 42	0.386 36	0.621 50	0.250 44	0.117 41	0.338 40	0.443 33	0.188 50	0.594 37	0.369 47	0.377 50	0.616 45	0.306 45

SPLAT Net	0.393 46	0.472 46	0.511 46	0.606 34	0.311 46	0.656 40	0.245 44	0.405 41	0.328 46	0.197 48	0.927 34	0.227 47	0.000 51	0.001 51	0.249 42	0.271 49	0.510 44	0.383 46	0.593 45	0.699 38	0.267 47
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF	0.383 47	0.297 49	0.491 47	0.432 46	0.358 43	0.612 45	0.274 41	0.116 48	0.411 40	0.265 45	0.904 43	0.229 46	0.079 45	0.250 42	0.185 47	0.320 47	0.510 44	0.385 45	0.548 46	0.597 48	0.394 43

PointNet++	0.339 48	0.584 37	0.478 48	0.458 45	0.256 49	0.360 50	0.250 42	0.247 47	0.278 49	0.261 46	0.677 49	0.183 48	0.117 41	0.212 48	0.145 49	0.364 45	0.346 50	0.232 50	0.548 46	0.523 49	0.252 48
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNet	0.308 49	0.353 48	0.290 50	0.278 50	0.166 50	0.553 47	0.169 49	0.286 45	0.147 50	0.148 50	0.908 41	0.182 49	0.064 46	0.023 50	0.018 51	0.354 46	0.363 48	0.345 48	0.546 48	0.685 39	0.278 46

ScanNet	0.306 50	0.203 50	0.366 49	0.501 41	0.311 46	0.524 48	0.211 48	0.002 51	0.342 45	0.189 49	0.786 48	0.145 50	0.102 44	0.245 43	0.152 48	0.318 48	0.348 49	0.300 49	0.460 49	0.437 50	0.182 50
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
ERROR	0.054 51	0.000 51	0.041 51	0.172 51	0.030 51	0.062 51	0.001 51	0.035 50	0.004 51	0.051 51	0.143 51	0.019 51	0.003 50	0.041 49	0.050 50	0.003 51	0.054 51	0.018 51	0.005 51	0.264 51	0.082 51

3D Semantic label benchmark