Benchmark Results - ScanNet Benchmark

This table lists the benchmark results for the 3D semantic label scenario.

Method	avg iou	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	floor	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	wall	window

MinkowskiNet	0.734 1	0.858 2	0.833 1	0.834 2	0.716 2	0.855 2	0.459 3	0.836 1	0.639 2	0.641 1	0.953 2	0.541 2	0.302 2	0.743 1	0.865 2	0.726 1	0.771 3	0.664 2	0.891 3	0.851 2	0.694 1
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
SparseConvNet	0.725 2	0.647 12	0.821 2	0.846 1	0.721 1	0.869 1	0.533 1	0.754 4	0.603 4	0.614 2	0.955 1	0.572 1	0.325 1	0.710 2	0.870 1	0.724 2	0.823 1	0.628 3	0.934 1	0.865 1	0.683 2

KP-FCNN	0.684 3	0.847 3	0.758 4	0.784 3	0.647 3	0.814 4	0.473 2	0.772 3	0.605 3	0.594 3	0.935 12	0.450 6	0.181 16	0.587 4	0.805 3	0.690 3	0.785 2	0.614 4	0.882 4	0.819 3	0.632 3

MVPNet	0.641 4	0.831 4	0.715 6	0.671 7	0.590 6	0.781 6	0.394 9	0.679 10	0.642 1	0.553 5	0.937 11	0.462 5	0.256 6	0.649 3	0.406 14	0.626 5	0.691 5	0.666 1	0.877 5	0.792 7	0.608 6

joint point-based	0.634 5	0.614 14	0.778 3	0.667 9	0.633 4	0.825 3	0.420 6	0.804 2	0.467 13	0.561 4	0.951 3	0.494 3	0.291 3	0.566 6	0.458 11	0.579 9	0.764 4	0.559 8	0.838 7	0.814 4	0.598 9

MCCNN	0.633 6	0.866 1	0.731 5	0.771 4	0.576 8	0.809 5	0.410 8	0.684 8	0.497 9	0.491 7	0.949 4	0.466 4	0.105 22	0.581 5	0.646 5	0.620 6	0.680 6	0.542 9	0.817 9	0.795 6	0.618 4
P. Hermosilla, T. Ritschel, P.P. Vazquez, A. Vinacua, T. Ropinski: Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds. SIGGRAPH Asia 2018
HPEIN	0.618 7	0.729 6	0.668 11	0.647 10	0.597 5	0.766 8	0.414 7	0.680 9	0.520 7	0.525 6	0.946 5	0.432 7	0.215 11	0.493 10	0.599 6	0.638 4	0.617 14	0.570 6	0.897 2	0.806 5	0.605 7

LAP-D	0.594 8	0.720 7	0.692 9	0.637 12	0.456 14	0.773 7	0.391 10	0.730 5	0.587 5	0.445 11	0.940 9	0.381 9	0.288 4	0.434 13	0.453 12	0.591 7	0.649 8	0.581 5	0.777 13	0.749 13	0.610 5

DPC	0.592 9	0.720 7	0.700 7	0.602 15	0.480 10	0.762 9	0.380 12	0.713 6	0.585 6	0.437 13	0.940 9	0.369 11	0.288 4	0.434 13	0.509 10	0.590 8	0.639 11	0.567 7	0.772 14	0.755 11	0.592 10

CCRFNet	0.589 10	0.766 5	0.659 13	0.683 6	0.470 13	0.740 10	0.387 11	0.620 13	0.490 10	0.476 8	0.922 16	0.355 14	0.245 7	0.511 8	0.511 9	0.571 10	0.643 10	0.493 13	0.872 6	0.762 9	0.600 8

TextureNet	0.566 11	0.672 10	0.664 12	0.671 7	0.494 9	0.719 12	0.445 4	0.678 11	0.411 18	0.396 14	0.935 12	0.356 13	0.225 9	0.412 15	0.535 8	0.565 11	0.636 13	0.464 15	0.794 12	0.680 18	0.568 11
Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkerhouser, Matthias Niessner, Leonidas Guibas: TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes. CVPR
DVVNet	0.562 12	0.648 11	0.700 7	0.770 5	0.586 7	0.687 16	0.333 14	0.650 12	0.514 8	0.475 9	0.906 21	0.359 12	0.223 10	0.340 17	0.442 13	0.422 20	0.668 7	0.501 12	0.708 18	0.779 8	0.534 14

PointConv	0.556 13	0.636 13	0.640 16	0.574 18	0.472 12	0.739 11	0.430 5	0.433 16	0.418 17	0.445 11	0.944 6	0.372 10	0.185 15	0.464 11	0.575 7	0.540 12	0.639 11	0.505 11	0.827 8	0.762 9	0.515 15
Wenxuan Wu, Zhongang Qi, Li Fuxin: PointConv: Deep Convolutional Networks on 3D Point Clouds. CVPR 2019
PanopticFusion-label	0.529 14	0.491 21	0.688 10	0.604 14	0.386 17	0.632 21	0.225 25	0.705 7	0.434 15	0.293 19	0.815 24	0.348 15	0.241 8	0.499 9	0.669 4	0.507 13	0.649 8	0.442 19	0.796 11	0.602 23	0.561 12
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. arXiv
3DMV, FTSDF	0.501 15	0.558 18	0.608 19	0.424 24	0.478 11	0.690 15	0.246 21	0.586 14	0.468 12	0.450 10	0.911 19	0.394 8	0.160 18	0.438 12	0.212 21	0.432 19	0.541 19	0.475 14	0.742 16	0.727 14	0.477 17

PCNN	0.498 16	0.559 17	0.644 15	0.560 19	0.420 16	0.711 14	0.229 23	0.414 17	0.436 14	0.352 16	0.941 8	0.324 16	0.155 19	0.238 21	0.387 15	0.493 14	0.529 20	0.509 10	0.813 10	0.751 12	0.504 16

3DMV	0.484 17	0.484 22	0.538 22	0.643 11	0.424 15	0.606 24	0.310 15	0.574 15	0.433 16	0.378 15	0.796 25	0.301 17	0.214 12	0.537 7	0.208 22	0.472 18	0.507 23	0.413 22	0.693 19	0.602 23	0.539 13
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGB	0.458 18	0.577 16	0.611 18	0.356 26	0.321 23	0.715 13	0.299 17	0.376 20	0.328 23	0.319 17	0.944 6	0.285 19	0.164 17	0.216 24	0.229 20	0.484 16	0.545 18	0.456 17	0.755 15	0.709 15	0.475 18
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
FCPN	0.447 19	0.679 9	0.604 20	0.578 17	0.380 18	0.682 17	0.291 18	0.106 26	0.483 11	0.258 24	0.920 17	0.258 21	0.025 26	0.231 23	0.325 16	0.480 17	0.560 17	0.463 16	0.725 17	0.666 20	0.231 26
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
PNET2	0.442 20	0.548 19	0.548 21	0.597 16	0.363 20	0.628 22	0.300 16	0.292 21	0.374 20	0.307 18	0.881 23	0.268 20	0.186 14	0.238 21	0.204 23	0.407 21	0.506 24	0.449 18	0.667 20	0.620 22	0.462 19

SurfaceConvPF	0.442 20	0.505 20	0.622 17	0.380 25	0.342 22	0.654 19	0.227 24	0.397 19	0.367 21	0.276 21	0.924 15	0.240 22	0.198 13	0.359 16	0.262 18	0.366 22	0.581 15	0.435 20	0.640 21	0.668 19	0.398 20
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
Tangent Convolutions	0.438 22	0.437 24	0.646 14	0.474 21	0.369 19	0.645 20	0.353 13	0.258 23	0.282 25	0.279 20	0.918 18	0.298 18	0.147 20	0.283 18	0.294 17	0.487 15	0.562 16	0.427 21	0.619 22	0.633 21	0.352 22
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
SPLAT Net	0.393 23	0.472 23	0.511 23	0.606 13	0.311 24	0.656 18	0.245 22	0.405 18	0.328 23	0.197 25	0.927 14	0.227 24	0.000 28	0.001 28	0.249 19	0.271 27	0.510 21	0.383 24	0.593 23	0.699 16	0.267 24
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF	0.383 24	0.297 26	0.491 24	0.432 23	0.358 21	0.612 23	0.274 19	0.116 25	0.411 18	0.265 22	0.904 22	0.229 23	0.079 24	0.250 19	0.185 24	0.320 25	0.510 21	0.385 23	0.548 24	0.597 25	0.394 21

PointNet++	0.339 25	0.584 15	0.478 25	0.458 22	0.256 26	0.360 27	0.250 20	0.247 24	0.278 26	0.261 23	0.677 27	0.183 25	0.117 21	0.212 25	0.145 26	0.364 23	0.346 27	0.232 27	0.548 24	0.523 26	0.252 25
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNet	0.308 26	0.353 25	0.290 27	0.278 27	0.166 27	0.553 25	0.169 27	0.286 22	0.147 27	0.148 27	0.908 20	0.182 26	0.064 25	0.023 27	0.018 28	0.354 24	0.363 25	0.345 25	0.546 26	0.685 17	0.278 23

ScanNet	0.306 27	0.203 27	0.366 26	0.501 20	0.311 24	0.524 26	0.211 26	0.002 28	0.342 22	0.189 26	0.786 26	0.145 27	0.102 23	0.245 20	0.152 25	0.318 26	0.348 26	0.300 26	0.460 27	0.437 27	0.182 27
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17
ERROR	0.054 28	0.000 28	0.041 28	0.172 28	0.030 28	0.062 28	0.001 28	0.035 27	0.004 28	0.051 28	0.143 28	0.019 28	0.003 27	0.041 26	0.050 27	0.003 28	0.054 28	0.018 28	0.005 28	0.264 28	0.082 28

Method

Info

avg ap 50%

bathtub

bed

bookshelf

cabinet

chair

counter

curtain

desk

door

otherfurniture

picture

refrigerator

shower curtain

sink

sofa

table

toilet

window

Occipital-SCS

0.512 1

1.000 1

0.716 2

0.509 3

0.506 1

0.611 3

0.092 3

0.602 3

0.177 5

0.346 4

0.383 4

0.165 4

0.442 2

0.850 4

0.386 4

0.618 3

0.543 2

0.889 6

0.389 2

3D-BoNet

0.488 2

1.000 1

0.672 5

0.590 2

0.301 4

0.484 8

0.098 2

0.620 1

0.306 1

0.341 5

0.259 7

0.125 6

0.434 3

0.796 5

0.402 3

0.499 9

0.513 3

0.909 5

0.439 1

MTML

0.481 3

1.000 1

0.666 6

0.377 5

0.272 5

0.709 1

0.001 12

0.579 4

0.254 3

0.361 3

0.318 5

0.095 8

0.432 4

1.000 1

0.184 7

0.601 6

0.487 4

0.938 3

0.384 3

PanopticFusion-inst

0.478 4

0.667 6

0.712 4

0.595 1

0.259 7

0.550 7

0.000 13

0.613 2

0.175 6

0.250 8

0.434 1

0.437 1

0.411 6

0.857 2

0.485 1

0.591 7

0.267 10

0.944 2

0.359 4

Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. arXiv

ResNet-backbone

0.459 5

1.000 1

0.737 1

0.159 11

0.259 6

0.587 5

0.138 1

0.475 6

0.217 4

0.416 1

0.408 3

0.128 5

0.315 7

0.714 6

0.411 2

0.536 8

0.590 1

0.873 8

0.304 5

MASC

0.447 6

0.528 9

0.555 8

0.381 4

0.382 2

0.633 2

0.002 10

0.509 5

0.260 2

0.361 2

0.432 2

0.327 2

0.451 1

0.571 7

0.367 5

0.639 2

0.386 5

0.980 1

0.276 6

Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.

3D-SIS

0.382 7

1.000 1

0.432 9

0.245 8

0.190 8

0.577 6

0.013 8

0.263 8

0.033 11

0.320 6

0.240 8

0.075 9

0.422 5

0.857 2

0.117 10

0.699 1

0.271 9

0.883 7

0.235 8

Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019

UNet-backbone

0.319 8

0.667 6

0.715 3

0.233 9

0.189 9

0.479 9

0.008 9

0.218 9

0.067 10

0.201 9

0.173 9

0.107 7

0.123 9

0.438 8

0.150 8

0.615 4

0.355 6

0.916 4

0.093 12

R-PointNet

0.306 9

0.500 10

0.405 10

0.311 6

0.348 3

0.589 4

0.054 4

0.068 11

0.126 7

0.283 7

0.290 6

0.028 10

0.219 8

0.214 11

0.331 6

0.396 10

0.275 8

0.821 10

0.245 7

3D-BEVIS

0.248 10

0.667 6

0.566 7

0.076 12

0.035 13

0.394 10

0.027 6

0.035 12

0.098 8

0.099 11

0.030 12

0.025 11

0.098 10

0.375 9

0.126 9

0.604 5

0.181 11

0.854 9

0.171 9

Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.

Seg-Cluster

0.215 11

0.370 11

0.337 12

0.285 7

0.105 10

0.325 11

0.025 7

0.282 7

0.085 9

0.105 10

0.107 10

0.007 13

0.079 11

0.317 10

0.114 11

0.309 12

0.304 7

0.587 11

0.123 11

Sgpn_scannet

0.143 12

0.208 13

0.390 11

0.169 10

0.065 11

0.275 12

0.029 5

0.069 10

0.000 12

0.087 12

0.043 11

0.014 12

0.027 13

0.000 12

0.112 12

0.351 11

0.168 12

0.438 12

0.138 10

MaskRCNN 2d->3d Proj

0.058 13

0.333 12

0.002 13

0.000 13

0.053 12

0.002 13

0.002 11

0.021 13

0.000 12

0.045 13

0.024 13

0.238 3

0.065 12

0.000 12

0.014 13

0.107 13

0.020 13

0.110 13

0.006 13

Method

Info

avg iou

bathtub

bed

bookshelf

cabinet

chair

counter

curtain

desk

door

floor

otherfurniture

picture

refrigerator

shower curtain

sink

sofa

table

toilet

wall

window

RFBNet

0.592 1

0.616 3

0.758 1

0.659 1

0.581 1

0.330 3

0.469 1

0.655 4

0.543 3

0.524 1

0.924 1

0.355 2

0.336 4

0.572 2

0.479 2

0.671 3

0.648 2

0.480 1

0.814 4

0.814 1

0.614 2

DCRedNet

0.583 2

0.682 2

0.723 2

0.542 4

0.510 4

0.310 5

0.451 2

0.668 2

0.549 2

0.520 2

0.920 2

0.375 1

0.446 1

0.528 4

0.417 3

0.670 4

0.577 6

0.478 2

0.862 2

0.806 2

0.628 1

SSMA

0.577 3

0.695 1

0.716 4

0.439 6

0.563 2

0.314 4

0.444 3

0.719 1

0.551 1

0.503 3

0.887 5

0.346 3

0.348 3

0.603 1

0.353 5

0.709 1

0.600 4

0.457 4

0.901 1

0.786 3

0.599 3

Abhinav Valada, Rohit Mohan, Wolfram Burgard: Self-Supervised Model Adaptation for Multimodal Semantic Segmentation. arXiv

FuseNet

0.535 4

0.570 5

0.681 5

0.182 9

0.512 3

0.290 6

0.431 4

0.659 3

0.504 5

0.495 4

0.903 4

0.308 4

0.428 2

0.523 5

0.365 4

0.676 2

0.621 3

0.470 3

0.762 6

0.779 4

0.541 5

Caner Hazirbas, Lingni Ma, Csaba Domokos, Daniel Cremers: FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-based CNN Architecture. ACCV 2016

AdapNet++

0.503 5

0.613 4

0.722 3

0.418 7

0.358 9

0.337 2

0.370 7

0.479 7

0.443 6

0.368 7

0.907 3

0.207 7

0.213 8

0.464 7

0.525 1

0.618 5

0.657 1

0.450 5

0.788 5

0.721 6

0.408 8

Abhinav Valada, Rohit Mohan, Wolfram Burgard: Self-Supervised Model Adaptation for Multimodal Semantic Segmentation. arXiv

3DMV (2d proj)

0.498 6

0.481 7

0.612 6

0.579 3

0.456 6

0.343 1

0.384 5

0.623 5

0.525 4

0.381 6

0.845 6

0.254 6

0.264 6

0.557 3

0.182 7

0.581 7

0.598 5

0.429 6

0.760 7

0.661 8

0.446 7

Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18

ILC-PSPNet

0.475 7

0.490 6

0.581 7

0.289 8

0.507 5

0.067 9

0.379 6

0.610 6

0.417 8

0.435 5

0.822 8

0.278 5

0.267 5

0.503 6

0.228 6

0.616 6

0.533 7

0.375 7

0.820 3

0.729 5

0.560 4

Enet (reimpl)

0.376 8

0.264 9

0.452 9

0.452 5

0.365 7

0.181 7

0.143 9

0.456 8

0.409 9

0.346 8

0.769 9

0.164 8

0.218 7

0.359 8

0.123 9

0.403 9

0.381 9

0.313 9

0.571 8

0.685 7

0.472 6

Re-implementation of Adam Paszke, Abhishek Chaurasia, Sangpil Kim, Eugenio Culurciello: ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation.

ScanNet (2d proj)

0.330 9

0.293 8

0.521 8

0.657 2

0.361 8

0.161 8

0.250 8

0.004 9

0.440 7

0.183 9

0.836 7

0.125 9

0.060 9

0.319 9

0.132 8

0.417 8

0.412 8

0.344 8

0.541 9

0.427 9

0.109 9

Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17

Method

Info

avg ap

bathtub

bed

bookshelf

cabinet

chair

counter

curtain

desk

door

otherfurniture

picture

refrigerator

shower curtain

sink

sofa

table

toilet

window

MaskRCNN_ScanNet

0.119 1

0.129 1

0.212 1

0.002 1

0.112 1

0.148 1

0.014 1

0.205 1

0.044 1

0.066 1

0.078 1

0.095 1

0.142 1

0.030 1

0.128 1

0.139 1

0.080 1

0.459 1

0.057 1

Re-implementation of Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick: Mask R-CNN. ICCV'17

Method	avg recall	apartment	bathroom	bedroom / hotel	bookstore / library	conference room	copy/mail room	hallway	kitchen	laundry room	living room / lounge	misc	office	storage / basement / garage
SE-ResNeXt-SSMA	0.498 1	0.000 2	0.812 1	0.941 1	0.500 1	0.500 1	0.500 1	0.500 1	0.429 2	0.500 1	0.667 1	0.500 1	0.625 1	0.000 1
Abhinav Valada, Rohit Mohan, Wolfram Burgard: Self-Supervised Model Adaptation for Multimodal Semantic Segmentation. arXiv
resnet50_scannet	0.353 2	0.250 1	0.812 1	0.529 2	0.500 1	0.500 1	0.000 2	0.500 1	0.571 1	0.000 2	0.556 2	0.000 2	0.375 2	0.000 1

Method

Info

avg recall

apartment

bathroom

bedroom / hotel

bookstore / library

conference room

copy/mail room

hallway

kitchen

laundry room

living room / lounge

misc

office

storage / basement / garage

SE-ResNeXt-SSMA

0.498 1

0.000 2

0.812 1

0.941 1

0.500 1

0.429 2

0.500 1

0.667 1

0.500 1

0.625 1

0.000 1

Abhinav Valada, Rohit Mohan, Wolfram Burgard: Self-Supervised Model Adaptation for Multimodal Semantic Segmentation. arXiv

resnet50_scannet

0.353 2

0.250 1

0.812 1

0.529 2

0.500 1

0.000 2

0.500 1

0.571 1

0.000 2

0.556 2

0.000 2

0.375 2

0.000 1

3D Semantic label benchmark

3D Semantic instance benchmark

2D Semantic label benchmark

2D Semantic instance benchmark

Scene type classification benchmark