Benchmark Results - ScanNet Benchmark

This table lists the benchmark results for the 3D semantic label scenario.

Method	avg iou	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	floor	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	wall	window

MinkowskiNet	0.734 1	0.858 2	0.833 1	0.834 2	0.716 2	0.855 2	0.459 3	0.836 1	0.639 2	0.641 1	0.953 2	0.541 2	0.302 2	0.743 1	0.865 2	0.726 1	0.771 3	0.664 2	0.891 3	0.851 2	0.694 1
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet	0.725 2	0.647 14	0.821 2	0.846 1	0.721 1	0.869 1	0.533 1	0.754 4	0.603 4	0.614 2	0.955 1	0.572 1	0.325 1	0.710 2	0.870 1	0.724 2	0.823 1	0.628 3	0.934 1	0.865 1	0.683 2

KP-FCNN	0.684 3	0.847 3	0.758 4	0.784 3	0.647 3	0.814 4	0.473 2	0.772 3	0.605 3	0.594 3	0.935 13	0.450 6	0.181 18	0.587 4	0.805 3	0.690 3	0.785 2	0.614 4	0.882 4	0.819 3	0.632 3
H. Thomas, C. Qi, J. Deschaud, B. Marcotegui, F. Goulette, L. Guibas.: KPConv: Flexible and Deformable Convolution for Point Clouds. ICCV 2019
joint point-based	0.634 5	0.614 16	0.778 3	0.667 11	0.633 4	0.825 3	0.420 7	0.804 2	0.467 13	0.561 4	0.951 3	0.494 3	0.291 3	0.566 6	0.458 12	0.579 10	0.764 4	0.559 8	0.838 7	0.814 4	0.598 9
Hung-Yueh Chiang, Yen-Liang Lin, Yueh-Cheng Liu, Winston H. Hsu: A Unified Point-Based Framework for 3D Segmentation. 3DV 2019
MVPNet	0.641 4	0.831 4	0.715 7	0.671 9	0.590 6	0.781 7	0.394 10	0.679 10	0.642 1	0.553 5	0.937 12	0.462 5	0.256 7	0.649 3	0.406 15	0.626 5	0.691 6	0.666 1	0.877 5	0.792 7	0.608 6

HPEIN	0.618 7	0.729 8	0.668 12	0.647 12	0.597 5	0.766 9	0.414 8	0.680 9	0.520 7	0.525 6	0.946 5	0.432 7	0.215 12	0.493 10	0.599 7	0.638 4	0.617 16	0.570 6	0.897 2	0.806 5	0.605 7

MCCNN	0.633 6	0.866 1	0.731 5	0.771 4	0.576 8	0.809 5	0.410 9	0.684 8	0.497 9	0.491 7	0.949 4	0.466 4	0.105 24	0.581 5	0.646 6	0.620 6	0.680 7	0.542 9	0.817 10	0.795 6	0.618 4
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
CCRFNet	0.589 11	0.766 5	0.659 15	0.683 8	0.470 15	0.740 12	0.387 13	0.620 14	0.490 10	0.476 8	0.922 18	0.355 16	0.245 8	0.511 8	0.511 10	0.571 11	0.643 11	0.493 15	0.872 6	0.762 10	0.600 8

DVVNet	0.562 13	0.648 13	0.700 8	0.770 5	0.586 7	0.687 18	0.333 16	0.650 12	0.514 8	0.475 9	0.906 23	0.359 14	0.223 11	0.340 19	0.442 14	0.422 22	0.668 8	0.501 13	0.708 20	0.779 8	0.534 15

DMC-Net	0.608 8	0.732 7	0.729 6	0.694 6	0.536 9	0.783 6	0.427 6	0.639 13	0.438 15	0.450 10	0.934 15	0.379 10	0.289 4	0.492 11	0.674 4	0.608 7	0.709 5	0.503 12	0.800 12	0.779 8	0.567 12

3DMV, FTSDF	0.501 17	0.558 20	0.608 21	0.424 26	0.478 13	0.690 17	0.246 23	0.586 15	0.468 12	0.450 10	0.911 21	0.394 8	0.160 20	0.438 13	0.212 23	0.432 21	0.541 21	0.475 16	0.742 18	0.727 16	0.477 19

LAP-D	0.594 9	0.720 9	0.692 10	0.637 14	0.456 16	0.773 8	0.391 12	0.730 5	0.587 5	0.445 12	0.940 10	0.381 9	0.288 5	0.434 14	0.453 13	0.591 8	0.649 9	0.581 5	0.777 15	0.749 15	0.610 5

PointConv	0.556 15	0.636 15	0.640 18	0.574 20	0.472 14	0.739 13	0.430 5	0.433 18	0.418 19	0.445 12	0.944 7	0.372 12	0.185 17	0.464 12	0.575 8	0.540 14	0.639 13	0.505 11	0.827 8	0.762 10	0.515 16
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
DPC	0.592 10	0.720 9	0.700 8	0.602 17	0.480 12	0.762 10	0.380 14	0.713 6	0.585 6	0.437 14	0.940 10	0.369 13	0.288 5	0.434 14	0.509 11	0.590 9	0.639 13	0.567 7	0.772 16	0.755 13	0.592 10
Francis Engelmann, Theodora Kontogianni, Bastian Leibe: Dilated Point Convolutions. arXiv
TextureNet	0.566 12	0.672 12	0.664 13	0.671 9	0.494 10	0.719 14	0.445 4	0.678 11	0.411 20	0.396 15	0.935 13	0.356 15	0.225 10	0.412 16	0.535 9	0.565 12	0.636 15	0.464 17	0.794 14	0.680 20	0.568 11
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
3DMV	0.484 19	0.484 24	0.538 24	0.643 13	0.424 17	0.606 26	0.310 17	0.574 16	0.433 18	0.378 16	0.796 27	0.301 19	0.214 13	0.537 7	0.208 24	0.472 20	0.507 25	0.413 24	0.693 21	0.602 25	0.539 14
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
Pointnet++ & Feature	0.557 14	0.735 6	0.661 14	0.686 7	0.491 11	0.744 11	0.392 11	0.539 17	0.451 14	0.375 17	0.946 5	0.376 11	0.205 14	0.403 17	0.356 17	0.553 13	0.643 11	0.497 14	0.824 9	0.756 12	0.515 16

PCNN	0.498 18	0.559 19	0.644 17	0.560 21	0.420 18	0.711 16	0.229 25	0.414 19	0.436 16	0.352 18	0.941 9	0.324 18	0.155 21	0.238 23	0.387 16	0.493 16	0.529 22	0.509 10	0.813 11	0.751 14	0.504 18

PointCNN with RGB	0.458 20	0.577 18	0.611 20	0.356 28	0.321 25	0.715 15	0.299 19	0.376 22	0.328 25	0.319 19	0.944 7	0.285 21	0.164 19	0.216 26	0.229 22	0.484 18	0.545 20	0.456 19	0.755 17	0.709 17	0.475 20
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
PNET2	0.442 22	0.548 21	0.548 23	0.597 18	0.363 22	0.628 24	0.300 18	0.292 23	0.374 22	0.307 20	0.881 25	0.268 22	0.186 16	0.238 23	0.204 25	0.407 23	0.506 26	0.449 20	0.667 22	0.620 24	0.462 21

PanopticFusion-label	0.529 16	0.491 23	0.688 11	0.604 16	0.386 19	0.632 23	0.225 27	0.705 7	0.434 17	0.293 21	0.815 26	0.348 17	0.241 9	0.499 9	0.669 5	0.507 15	0.649 9	0.442 21	0.796 13	0.602 25	0.561 13
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
Tangent Convolutions	0.438 24	0.437 26	0.646 16	0.474 23	0.369 21	0.645 22	0.353 15	0.258 25	0.282 27	0.279 22	0.918 20	0.298 20	0.147 22	0.283 20	0.294 19	0.487 17	0.562 18	0.427 23	0.619 24	0.633 23	0.352 24
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
SurfaceConvPF	0.442 22	0.505 22	0.622 19	0.380 27	0.342 24	0.654 21	0.227 26	0.397 21	0.367 23	0.276 23	0.924 17	0.240 24	0.198 15	0.359 18	0.262 20	0.366 24	0.581 17	0.435 22	0.640 23	0.668 21	0.398 22
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
ScanNet+FTSDF	0.383 26	0.297 28	0.491 26	0.432 25	0.358 23	0.612 25	0.274 21	0.116 27	0.411 20	0.265 24	0.904 24	0.229 25	0.079 26	0.250 21	0.185 26	0.320 27	0.510 23	0.385 25	0.548 26	0.597 27	0.394 23

PointNet++	0.339 27	0.584 17	0.478 27	0.458 24	0.256 28	0.360 29	0.250 22	0.247 26	0.278 28	0.261 25	0.677 29	0.183 27	0.117 23	0.212 27	0.145 28	0.364 25	0.346 29	0.232 29	0.548 26	0.523 28	0.252 27
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
FCPN	0.447 21	0.679 11	0.604 22	0.578 19	0.380 20	0.682 19	0.291 20	0.106 28	0.483 11	0.258 26	0.920 19	0.258 23	0.025 28	0.231 25	0.325 18	0.480 19	0.560 19	0.463 18	0.725 19	0.666 22	0.231 28
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
SPLAT Net	0.393 25	0.472 25	0.511 25	0.606 15	0.311 26	0.656 20	0.245 24	0.405 20	0.328 25	0.197 27	0.927 16	0.227 26	0.000 30	0.001 30	0.249 21	0.271 29	0.510 23	0.383 26	0.593 25	0.699 18	0.267 26
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet	0.306 29	0.203 29	0.366 28	0.501 22	0.311 26	0.524 28	0.211 28	0.002 30	0.342 24	0.189 28	0.786 28	0.145 29	0.102 25	0.245 22	0.152 27	0.318 28	0.348 28	0.300 28	0.460 29	0.437 29	0.182 29
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
SSC-UNet	0.308 28	0.353 27	0.290 29	0.278 29	0.166 29	0.553 27	0.169 29	0.286 24	0.147 29	0.148 29	0.908 22	0.182 28	0.064 27	0.023 29	0.018 30	0.354 26	0.363 27	0.345 27	0.546 28	0.685 19	0.278 25

ERROR	0.054 30	0.000 30	0.041 30	0.172 30	0.030 30	0.062 30	0.001 30	0.035 29	0.004 30	0.051 30	0.143 30	0.019 30	0.003 29	0.041 28	0.050 29	0.003 30	0.054 30	0.018 30	0.005 30	0.264 30	0.082 30

This table lists the benchmark results for the 3D semantic instance scenario.

Method	avg ap 50%	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	window

MTML	0.549 2	1.000 1	0.807 2	0.588 4	0.327 5	0.647 2	0.004 12	0.815 1	0.180 5	0.418 1	0.364 6	0.182 5	0.445 3	1.000 1	0.442 4	0.688 3	0.571 2	1.000 1	0.396 2

ResNet-backbone	0.459 6	1.000 1	0.737 3	0.159 13	0.259 7	0.587 6	0.138 2	0.475 7	0.217 4	0.416 2	0.408 4	0.128 7	0.315 8	0.714 7	0.411 5	0.536 9	0.590 1	0.873 10	0.304 6

DCNet	0.607 1	1.000 1	0.907 1	0.792 1	0.462 2	0.788 1	0.151 1	0.535 5	0.292 2	0.395 3	0.501 1	0.263 3	0.600 1	1.000 1	0.598 1	0.857 1	0.502 5	0.918 5	0.368 4

MASC	0.447 7	0.528 10	0.555 9	0.381 7	0.382 3	0.633 3	0.002 13	0.509 6	0.260 3	0.361 4	0.432 3	0.327 2	0.451 2	0.571 8	0.367 8	0.639 4	0.386 7	0.980 2	0.276 7
Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.
Occipital-SCS	0.512 3	1.000 1	0.716 4	0.509 5	0.506 1	0.611 4	0.092 5	0.602 4	0.177 6	0.346 5	0.383 5	0.165 6	0.442 4	0.850 5	0.386 7	0.618 5	0.543 3	0.889 8	0.389 3

3D-BoNet	0.488 4	1.000 1	0.672 7	0.590 3	0.301 6	0.484 9	0.098 4	0.620 2	0.306 1	0.341 6	0.259 8	0.125 8	0.434 5	0.796 6	0.402 6	0.499 10	0.513 4	0.909 7	0.439 1

3D-SIS	0.382 8	1.000 1	0.432 11	0.245 10	0.190 10	0.577 7	0.013 10	0.263 10	0.033 13	0.320 7	0.240 10	0.075 11	0.422 6	0.857 3	0.117 12	0.699 2	0.271 11	0.883 9	0.235 10
Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019
R-PointNet	0.306 11	0.500 11	0.405 12	0.311 8	0.348 4	0.589 5	0.054 6	0.068 13	0.126 8	0.283 8	0.290 7	0.028 12	0.219 10	0.214 13	0.331 9	0.396 12	0.275 10	0.821 12	0.245 8

PanopticFusion-inst	0.478 5	0.667 7	0.712 6	0.595 2	0.259 8	0.550 8	0.000 15	0.613 3	0.175 7	0.250 9	0.434 2	0.437 1	0.411 7	0.857 3	0.485 2	0.591 8	0.267 12	0.944 4	0.359 5
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
DPC-instance	0.355 9	0.500 11	0.517 10	0.467 6	0.228 9	0.422 11	0.133 3	0.405 8	0.111 9	0.205 10	0.241 9	0.075 10	0.233 9	0.306 12	0.445 3	0.439 11	0.457 6	0.974 3	0.239 9
Francis Engelmann, Theodora Kontogianni, Bastian Leibe: Dilated Point Convolutions. arXiv
UNet-backbone	0.319 10	0.667 7	0.715 5	0.233 11	0.189 11	0.479 10	0.008 11	0.218 11	0.067 12	0.201 11	0.173 11	0.107 9	0.123 11	0.438 9	0.150 10	0.615 6	0.355 8	0.916 6	0.093 14

Seg-Cluster	0.215 13	0.370 13	0.337 14	0.285 9	0.105 12	0.325 13	0.025 9	0.282 9	0.085 11	0.105 12	0.107 12	0.007 15	0.079 13	0.317 11	0.114 13	0.309 14	0.304 9	0.587 13	0.123 13

3D-BEVIS	0.248 12	0.667 7	0.566 8	0.076 14	0.035 15	0.394 12	0.027 8	0.035 14	0.098 10	0.099 13	0.030 14	0.025 13	0.098 12	0.375 10	0.126 11	0.604 7	0.181 13	0.854 11	0.171 11
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.
Sgpn_scannet	0.143 14	0.208 15	0.390 13	0.169 12	0.065 13	0.275 14	0.029 7	0.069 12	0.000 14	0.087 14	0.043 13	0.014 14	0.027 15	0.000 14	0.112 14	0.351 13	0.168 14	0.438 14	0.138 12

MaskRCNN 2d->3d Proj	0.058 15	0.333 14	0.002 15	0.000 15	0.053 14	0.002 15	0.002 14	0.021 15	0.000 14	0.045 15	0.024 15	0.238 4	0.065 14	0.000 14	0.014 15	0.107 15	0.020 15	0.110 15	0.006 15

This table lists the benchmark results for the 2D semantic label scenario.

Method	avg iou	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	floor	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	wall	window

RFBNet	0.592 1	0.616 3	0.758 1	0.659 1	0.581 1	0.330 3	0.469 1	0.655 4	0.543 3	0.524 1	0.924 1	0.355 2	0.336 4	0.572 2	0.479 2	0.671 3	0.648 2	0.480 1	0.814 4	0.814 1	0.614 2

DCRedNet	0.583 2	0.682 2	0.723 2	0.542 4	0.510 4	0.310 5	0.451 2	0.668 2	0.549 2	0.520 2	0.920 2	0.375 1	0.446 1	0.528 4	0.417 3	0.670 4	0.577 6	0.478 2	0.862 2	0.806 2	0.628 1

SSMA	0.577 3	0.695 1	0.716 4	0.439 6	0.563 2	0.314 4	0.444 3	0.719 1	0.551 1	0.503 3	0.887 5	0.346 3	0.348 3	0.603 1	0.353 5	0.709 1	0.600 4	0.457 4	0.901 1	0.786 3	0.599 3
Abhinav Valada, Rohit Mohan, Wolfram Burgard: Self-Supervised Model Adaptation for Multimodal Semantic Segmentation. International Journal of Computer Vision, 2019
FuseNet	0.535 4	0.570 5	0.681 5	0.182 9	0.512 3	0.290 6	0.431 4	0.659 3	0.504 5	0.495 4	0.903 4	0.308 4	0.428 2	0.523 5	0.365 4	0.676 2	0.621 3	0.470 3	0.762 6	0.779 4	0.541 5
Caner Hazirbas, Lingni Ma, Csaba Domokos, Daniel Cremers: FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-based CNN Architecture. ACCV 2016
ILC-PSPNet	0.475 7	0.490 6	0.581 7	0.289 8	0.507 5	0.067 9	0.379 6	0.610 6	0.417 8	0.435 5	0.822 8	0.278 5	0.267 5	0.503 6	0.228 6	0.616 6	0.533 7	0.375 7	0.820 3	0.729 5	0.560 4

3DMV (2d proj)	0.498 6	0.481 7	0.612 6	0.579 3	0.456 6	0.343 1	0.384 5	0.623 5	0.525 4	0.381 6	0.845 6	0.254 6	0.264 6	0.557 3	0.182 7	0.581 7	0.598 5	0.429 6	0.760 7	0.661 8	0.446 7
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
AdapNet++	0.503 5	0.613 4	0.722 3	0.418 7	0.358 9	0.337 2	0.370 7	0.479 7	0.443 6	0.368 7	0.907 3	0.207 7	0.213 8	0.464 7	0.525 1	0.618 5	0.657 1	0.450 5	0.788 5	0.721 6	0.408 8
Abhinav Valada, Rohit Mohan, Wolfram Burgard: Self-Supervised Model Adaptation for Multimodal Semantic Segmentation. International Journal of Computer Vision, 2019
Enet (reimpl)	0.376 8	0.264 9	0.452 9	0.452 5	0.365 7	0.181 7	0.143 9	0.456 8	0.409 9	0.346 8	0.769 9	0.164 8	0.218 7	0.359 8	0.123 9	0.403 9	0.381 9	0.313 9	0.571 8	0.685 7	0.472 6
Re-implementation of Adam Paszke, Abhishek Chaurasia, Sangpil Kim, Eugenio Culurciello: ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation.
ScanNet (2d proj)	0.330 9	0.293 8	0.521 8	0.657 2	0.361 8	0.161 8	0.250 8	0.004 9	0.440 7	0.183 9	0.836 7	0.125 9	0.060 9	0.319 9	0.132 8	0.417 8	0.412 8	0.344 8	0.541 9	0.427 9	0.109 9
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17

This table lists the benchmark results for the 2D semantic instance scenario.

Method	avg ap	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	window

MaskRCNN_ScanNet	0.119 1	0.129 1	0.212 1	0.002 1	0.112 1	0.148 1	0.014 1	0.205 1	0.044 1	0.066 1	0.078 1	0.095 1	0.142 1	0.030 1	0.128 1	0.139 1	0.080 1	0.459 1	0.057 1
Re-implementation of Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick: Mask R-CNN. ICCV'17

This table lists the benchmark results for the scene type classification scenario.

Method	avg recall	apartment	bathroom	bedroom / hotel	bookstore / library	conference room	copy/mail room	hallway	kitchen	laundry room	living room / lounge	misc	office	storage / basement / garage

SE-ResNeXt-SSMA	0.498 1	0.000 2	0.812 1	0.941 1	0.500 1	0.500 1	0.500 1	0.500 1	0.429 2	0.500 1	0.667 1	0.500 1	0.625 1	0.000 1
Abhinav Valada, Rohit Mohan, Wolfram Burgard: Self-Supervised Model Adaptation for Multimodal Semantic Segmentation. arXiv
resnet50_scannet	0.353 2	0.250 1	0.812 1	0.529 2	0.500 1	0.500 1	0.000 2	0.500 1	0.571 1	0.000 2	0.556 2	0.000 2	0.375 2	0.000 1

3D Semantic label benchmark

3D Semantic instance benchmark

2D Semantic label benchmark

2D Semantic instance benchmark

Scene type classification benchmark