Benchmark Results - ScanNet Benchmark

This table lists the benchmark results for the 3D semantic label scenario.

Method	avg iou	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	floor	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	wall	window

SparseConvNet	0.725 1	0.647 8	0.821 1	0.846 1	0.721 1	0.869 1	0.533 1	0.754 4	0.603 4	0.614 1	0.955 1	0.572 1	0.325 1	0.710 2	0.870 2	0.724 1	0.823 1	0.628 3	0.934 1	0.865 1	0.683 1

MinkowskiNet	0.721 2	0.837 2	0.804 2	0.800 2	0.721 1	0.843 2	0.460 3	0.835 1	0.647 1	0.597 2	0.953 2	0.542 2	0.214 8	0.746 1	0.912 1	0.705 2	0.771 3	0.640 2	0.876 5	0.842 2	0.672 2
C. Choy, J. Gwak, S. Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019
KP-FCNN	0.684 3	0.847 1	0.758 4	0.784 3	0.647 3	0.814 4	0.473 2	0.772 3	0.605 3	0.594 3	0.935 9	0.450 5	0.181 13	0.587 4	0.805 3	0.690 3	0.785 2	0.614 4	0.882 3	0.819 3	0.632 3

MVPNet	0.641 4	0.831 3	0.715 5	0.671 5	0.590 6	0.781 5	0.394 8	0.679 7	0.642 2	0.553 5	0.937 8	0.462 4	0.256 3	0.649 3	0.406 10	0.626 5	0.691 5	0.666 1	0.877 4	0.792 6	0.608 4

HPEIN	0.618 6	0.729 4	0.668 9	0.647 8	0.597 5	0.766 6	0.414 7	0.680 6	0.520 5	0.525 6	0.946 4	0.432 6	0.215 7	0.493 8	0.599 5	0.638 4	0.617 11	0.570 5	0.897 2	0.806 5	0.605 5

joint point-based	0.634 5	0.614 10	0.778 3	0.667 7	0.633 4	0.825 3	0.420 6	0.804 2	0.467 10	0.561 4	0.951 3	0.494 3	0.291 2	0.566 5	0.458 8	0.579 6	0.764 4	0.559 6	0.838 6	0.814 4	0.598 6

TextureNet	0.566 7	0.672 6	0.664 10	0.671 5	0.494 8	0.719 8	0.445 4	0.678 8	0.411 15	0.396 11	0.935 9	0.356 10	0.225 5	0.412 11	0.535 7	0.565 8	0.636 9	0.464 12	0.794 10	0.680 15	0.568 7

PanopticFusion-label	0.529 10	0.491 18	0.688 7	0.604 12	0.386 14	0.632 18	0.225 22	0.705 5	0.434 12	0.293 16	0.815 21	0.348 11	0.241 4	0.499 7	0.669 4	0.507 10	0.649 7	0.442 16	0.796 9	0.602 20	0.561 8
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. arXiv
3DMV	0.484 14	0.484 19	0.538 19	0.643 9	0.424 12	0.606 21	0.310 11	0.574 11	0.433 13	0.378 12	0.796 22	0.301 14	0.214 8	0.537 6	0.208 19	0.472 15	0.507 20	0.413 19	0.693 16	0.602 20	0.539 9
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
DVVNet	0.562 8	0.648 7	0.700 6	0.770 4	0.586 7	0.687 12	0.333 10	0.650 9	0.514 6	0.475 7	0.906 18	0.359 9	0.223 6	0.340 13	0.442 9	0.422 17	0.668 6	0.501 9	0.708 15	0.779 7	0.534 10

PointConv	0.556 9	0.636 9	0.640 13	0.574 15	0.472 10	0.739 7	0.430 5	0.433 12	0.418 14	0.445 9	0.944 5	0.372 8	0.185 12	0.464 9	0.575 6	0.540 9	0.639 8	0.505 8	0.827 7	0.762 8	0.515 11

PCNN	0.498 13	0.559 14	0.644 12	0.560 16	0.420 13	0.711 10	0.229 20	0.414 13	0.436 11	0.352 13	0.941 7	0.324 13	0.155 16	0.238 17	0.387 11	0.493 11	0.529 17	0.509 7	0.813 8	0.751 9	0.504 12

3DMV, FTSDF	0.501 12	0.558 15	0.608 16	0.424 21	0.478 9	0.690 11	0.246 18	0.586 10	0.468 9	0.450 8	0.911 16	0.394 7	0.160 15	0.438 10	0.212 18	0.432 16	0.541 16	0.475 11	0.742 13	0.727 11	0.477 13

PointCNN with RGB	0.458 15	0.577 13	0.611 15	0.356 23	0.321 20	0.715 9	0.299 14	0.376 17	0.328 20	0.319 14	0.944 5	0.285 16	0.164 14	0.216 20	0.229 16	0.484 13	0.545 15	0.456 14	0.755 12	0.709 12	0.475 14
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
PNET2	0.442 17	0.548 16	0.548 18	0.597 13	0.363 17	0.628 19	0.300 13	0.292 18	0.374 17	0.307 15	0.881 20	0.268 17	0.186 11	0.238 17	0.204 20	0.407 18	0.506 21	0.449 15	0.667 17	0.620 19	0.462 15

LAP-D	0.504 11	0.604 11	0.679 8	0.608 10	0.464 11	0.678 14	0.308 12	0.386 16	0.500 7	0.397 10	0.935 9	0.332 12	0.086 20	0.212 21	0.228 17	0.579 6	0.628 10	0.499 10	0.769 11	0.730 10	0.452 16

SurfaceConvPF	0.442 17	0.505 17	0.622 14	0.380 22	0.342 19	0.654 16	0.227 21	0.397 15	0.367 18	0.276 18	0.924 13	0.240 19	0.198 10	0.359 12	0.262 14	0.366 19	0.581 12	0.435 17	0.640 18	0.668 16	0.398 17
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
ScanNet+FTSDF	0.383 21	0.297 23	0.491 21	0.432 20	0.358 18	0.612 20	0.274 16	0.116 22	0.411 15	0.265 19	0.904 19	0.229 20	0.079 21	0.250 15	0.185 21	0.320 22	0.510 18	0.385 20	0.548 21	0.597 22	0.394 18

Tangent Convolutions	0.438 19	0.437 21	0.646 11	0.474 18	0.369 16	0.645 17	0.353 9	0.258 20	0.282 22	0.279 17	0.918 15	0.298 15	0.147 17	0.283 14	0.294 13	0.487 12	0.562 13	0.427 18	0.619 19	0.633 18	0.352 19
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
SSC-UNet	0.308 23	0.353 22	0.290 24	0.278 24	0.166 24	0.553 22	0.169 24	0.286 19	0.147 24	0.148 24	0.908 17	0.182 23	0.064 22	0.023 23	0.018 24	0.354 21	0.363 22	0.345 22	0.546 23	0.685 14	0.278 20

SPLAT Net	0.393 20	0.472 20	0.511 20	0.606 11	0.311 21	0.656 15	0.245 19	0.405 14	0.328 20	0.197 22	0.927 12	0.227 21	0.000 24	0.001 24	0.249 15	0.271 24	0.510 18	0.383 21	0.593 20	0.699 13	0.267 21
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
PointNet++	0.339 22	0.584 12	0.478 22	0.458 19	0.256 23	0.360 24	0.250 17	0.247 21	0.278 23	0.261 20	0.677 24	0.183 22	0.117 18	0.212 21	0.145 23	0.364 20	0.346 24	0.232 24	0.548 21	0.523 23	0.252 22
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
FCPN	0.447 16	0.679 5	0.604 17	0.578 14	0.380 15	0.682 13	0.291 15	0.106 23	0.483 8	0.258 21	0.920 14	0.258 18	0.025 23	0.231 19	0.325 12	0.480 14	0.560 14	0.463 13	0.725 14	0.666 17	0.231 23
Dario Rethage, Johanna Wald, Jürgen Sturm, Nassir Navab, Federico Tombari: Fully-Convolutional Point Networks for Large-Scale Point Clouds. ECCV 2018
ScanNet	0.306 24	0.203 24	0.366 23	0.501 17	0.311 21	0.524 23	0.211 23	0.002 24	0.342 19	0.189 23	0.786 23	0.145 24	0.102 19	0.245 16	0.152 22	0.318 23	0.348 23	0.300 23	0.460 24	0.437 24	0.182 24
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17

Method

Info

avg ap 50%

bathtub

bed

bookshelf

cabinet

chair

counter

curtain

desk

door

otherfurniture

picture

refrigerator

shower curtain

sink

sofa

table

toilet

window

MTML

0.481 1

1.000 1

0.666 4

0.377 3

0.272 3

0.709 1

0.001 10

0.579 2

0.254 2

0.361 3

0.318 4

0.095 6

0.432 2

1.000 1

0.184 5

0.601 5

0.487 2

0.938 3

0.384 1

PanopticFusion-inst

0.478 2

0.667 4

0.712 3

0.595 1

0.259 5

0.550 6

0.000 11

0.613 1

0.175 4

0.250 6

0.434 1

0.437 1

0.411 4

0.857 2

0.485 1

0.591 6

0.267 8

0.944 2

0.359 2

Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. arXiv

ResNet-backbone

0.459 3

1.000 1

0.737 1

0.159 9

0.259 4

0.587 4

0.138 1

0.475 4

0.217 3

0.416 1

0.408 3

0.128 4

0.315 5

0.714 4

0.411 2

0.536 7

0.590 1

0.873 6

0.304 3

MASC

0.447 4

0.528 7

0.555 6

0.381 2

0.382 1

0.633 2

0.002 8

0.509 3

0.260 1

0.361 2

0.432 2

0.327 2

0.451 1

0.571 5

0.367 3

0.639 2

0.386 3

0.980 1

0.276 4

Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.

R-PointNet

0.306 7

0.500 8

0.405 8

0.311 4

0.348 2

0.589 3

0.054 2

0.068 9

0.126 5

0.283 5

0.290 5

0.028 8

0.219 6

0.214 9

0.331 4

0.396 8

0.275 6

0.821 8

0.245 5

3D-SIS

0.382 5

1.000 1

0.432 7

0.245 6

0.190 6

0.577 5

0.013 6

0.263 6

0.033 9

0.320 4

0.240 6

0.075 7

0.422 3

0.857 2

0.117 8

0.699 1

0.271 7

0.883 5

0.235 6

3D-BEVIS

0.248 8

0.667 4

0.566 5

0.076 10

0.035 11

0.394 8

0.027 4

0.035 10

0.098 6

0.099 9

0.030 10

0.025 9

0.098 8

0.375 7

0.126 7

0.604 4

0.181 9

0.854 7

0.171 7

Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.

Sgpn_scannet

0.143 10

0.208 11

0.390 9

0.169 8

0.065 9

0.275 10

0.029 3

0.069 8

0.000 10

0.087 10

0.043 9

0.014 10

0.027 11

0.000 10

0.112 10

0.351 9

0.168 10

0.438 10

0.138 8

Seg-Cluster

0.215 9

0.370 9

0.337 10

0.285 5

0.105 8

0.325 9

0.025 5

0.282 5

0.085 7

0.105 8

0.107 8

0.007 11

0.079 9

0.317 8

0.114 9

0.309 10

0.304 5

0.587 9

0.123 9

UNet-backbone

0.319 6

0.667 4

0.715 2

0.233 7

0.189 7

0.479 7

0.008 7

0.218 7

0.067 8

0.201 7

0.173 7

0.107 5

0.123 7

0.438 6

0.150 6

0.615 3

0.355 4

0.916 4

0.093 10

MaskRCNN 2d->3d Proj

0.058 11

0.333 10

0.002 11

0.000 11

0.053 10

0.002 11

0.002 9

0.021 11

0.000 10

0.045 11

0.024 11

0.238 3

0.065 10

0.000 10

0.014 11

0.107 11

0.020 11

0.110 11

0.006 11

Method

Info

avg iou

bathtub

bed

bookshelf

cabinet

chair

counter

curtain

desk

door

floor

otherfurniture

picture

refrigerator

shower curtain

sink

sofa

table

toilet

wall

window

SSMA

0.577 1

0.695 1

0.716 2

0.439 4

0.563 1

0.314 3

0.444 1

0.719 1

0.551 1

0.503 1

0.887 3

0.346 1

0.348 2

0.603 1

0.353 3

0.709 1

0.600 2

0.457 1

0.901 1

0.786 1

0.599 1

Abhinav Valada, Rohit Mohan, Wolfram Burgard: Self-Supervised Model Adaptation for Multimodal Semantic Segmentation. arXiv

FuseNet

0.521 2

0.591 3

0.682 3

0.220 7

0.488 3

0.279 4

0.344 5

0.610 3

0.461 3

0.475 2

0.910 1

0.293 2

0.447 1

0.512 3

0.397 2

0.618 2

0.567 4

0.452 2

0.734 5

0.782 2

0.566 2

Caner Hazirbas, Lingni Ma, Csaba Domokos, Daniel Cremers: FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-based CNN Architecture. ACCV 2016

ILC-PSPNet

0.475 5

0.490 4

0.581 5

0.289 6

0.507 2

0.067 7

0.379 3

0.610 3

0.417 6

0.435 3

0.822 6

0.278 3

0.267 3

0.503 4

0.228 4

0.616 4

0.533 5

0.375 5

0.820 2

0.729 3

0.560 3

Enet (reimpl)

0.376 6

0.264 7

0.452 7

0.452 3

0.365 5

0.181 5

0.143 7

0.456 6

0.409 7

0.346 6

0.769 7

0.164 6

0.218 5

0.359 6

0.123 7

0.403 7

0.381 7

0.313 7

0.571 6

0.685 5

0.472 4

Re-implementation of Adam Paszke, Abhishek Chaurasia, Sangpil Kim, Eugenio Culurciello: ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation.

3DMV (2d proj)

0.498 4

0.481 5

0.612 4

0.579 2

0.456 4

0.343 1

0.384 2

0.623 2

0.525 2

0.381 4

0.845 4

0.254 4

0.264 4

0.557 2

0.182 5

0.581 5

0.598 3

0.429 4

0.760 4

0.661 6

0.446 5

Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18

AdapNet++

0.503 3

0.613 2

0.722 1

0.418 5

0.358 7

0.337 2

0.370 4

0.479 5

0.443 4

0.368 5

0.907 2

0.207 5

0.213 6

0.464 5

0.525 1

0.618 2

0.657 1

0.450 3

0.788 3

0.721 4

0.408 6

Abhinav Valada, Rohit Mohan, Wolfram Burgard: Self-Supervised Model Adaptation for Multimodal Semantic Segmentation. arXiv

ScanNet (2d proj)

0.330 7

0.293 6

0.521 6

0.657 1

0.361 6

0.161 6

0.250 6

0.004 7

0.440 5

0.183 7

0.836 5

0.125 7

0.060 7

0.319 7

0.132 6

0.417 6

0.412 6

0.344 6

0.541 7

0.427 7

0.109 7

Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17

Method

Info

avg ap

bathtub

bed

bookshelf

cabinet

chair

counter

curtain

desk

door

otherfurniture

picture

refrigerator

shower curtain

sink

sofa

table

toilet

window

MaskRCNN_ScanNet

0.119 1

0.129 1

0.212 1

0.002 1

0.112 1

0.148 1

0.014 1

0.205 1

0.044 1

0.066 1

0.078 1

0.095 1

0.142 1

0.030 1

0.128 1

0.139 1

0.080 1

0.459 1

0.057 1

Re-implementation of Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick: Mask R-CNN. ICCV'17

Method	avg recall	apartment	bathroom	bedroom / hotel	bookstore / library	conference room	copy/mail room	hallway	kitchen	laundry room	living room / lounge	misc	office	storage / basement / garage
SE-ResNeXt-SSMA	0.498 1	0.000 2	0.812 1	0.941 1	0.500 1	0.500 1	0.500 1	0.500 1	0.429 2	0.500 1	0.667 1	0.500 1	0.625 1	0.000 1
Abhinav Valada, Rohit Mohan, Wolfram Burgard: Self-Supervised Model Adaptation for Multimodal Semantic Segmentation. arXiv
resnet50_scannet	0.353 2	0.250 1	0.812 1	0.529 2	0.500 1	0.500 1	0.000 2	0.500 1	0.571 1	0.000 2	0.556 2	0.000 2	0.375 2	0.000 1

Method

Info

avg recall

apartment

bathroom

bedroom / hotel

bookstore / library

conference room

copy/mail room

hallway

kitchen

laundry room

living room / lounge

misc

office

storage / basement / garage

SE-ResNeXt-SSMA

0.498 1

0.000 2

0.812 1

0.941 1

0.500 1

0.429 2

0.500 1

0.667 1

0.500 1

0.625 1

0.000 1

Abhinav Valada, Rohit Mohan, Wolfram Burgard: Self-Supervised Model Adaptation for Multimodal Semantic Segmentation. arXiv

resnet50_scannet

0.353 2

0.250 1

0.812 1

0.529 2

0.500 1

0.000 2

0.500 1

0.571 1

0.000 2

0.556 2

0.000 2

0.375 2

0.000 1

3D Semantic label benchmark

3D Semantic instance benchmark

2D Semantic label benchmark

2D Semantic instance benchmark

Scene type classification benchmark