Benchmark Results - ScanNet Benchmark

This table lists the benchmark results for the 3D semantic label scenario.

Method	avg iou	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	floor	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	wall	window

SparseConvNet	0.725 1	0.647 5	0.821 1	0.846 1	0.721 1	0.869 1	0.533 1	0.754 4	0.603 2	0.614 2	0.955 1	0.572 1	0.325 1	0.710 1	0.870 2	0.724 1	0.823 1	0.628 2	0.934 1	0.865 1	0.683 1

KP-FCNN	0.694 2	0.849 1	0.770 3	0.810 2	0.685 2	0.813 3	0.438 3	0.791 2	0.566 3	0.616 1	0.944 3	0.500 3	0.216 6	0.559 4	0.880 1	0.690 2	0.758 3	0.627 3	0.922 2	0.832 2	0.613 3

MinkowskiNet34	0.679 3	0.811 2	0.734 4	0.739 4	0.641 3	0.804 4	0.413 6	0.759 3	0.696 1	0.545 4	0.938 7	0.518 2	0.141 15	0.623 2	0.757 3	0.680 3	0.723 4	0.684 1	0.896 3	0.821 3	0.651 2

joint point-based	0.634 4	0.614 7	0.778 2	0.667 6	0.633 4	0.825 2	0.420 5	0.804 1	0.467 6	0.561 3	0.951 2	0.494 4	0.291 2	0.566 3	0.458 7	0.579 4	0.764 2	0.559 4	0.838 4	0.814 4	0.598 4

TextureNet	0.566 5	0.672 3	0.664 7	0.671 5	0.494 6	0.719 6	0.445 2	0.678 6	0.411 11	0.396 8	0.935 8	0.356 8	0.225 4	0.412 9	0.535 6	0.565 5	0.636 8	0.464 9	0.794 8	0.680 12	0.568 5

DVVNet	0.562 6	0.648 4	0.700 5	0.770 3	0.586 5	0.687 10	0.333 8	0.650 7	0.514 4	0.475 5	0.906 14	0.359 7	0.223 5	0.340 11	0.442 8	0.422 13	0.668 5	0.501 7	0.708 11	0.779 5	0.534 8

PointConv	0.556 7	0.636 6	0.640 10	0.574 11	0.472 8	0.739 5	0.430 4	0.433 10	0.418 10	0.445 7	0.944 3	0.372 6	0.185 10	0.464 7	0.575 5	0.540 6	0.639 7	0.505 6	0.827 5	0.762 6	0.515 9

PanopticFusion-label	0.529 8	0.491 14	0.688 6	0.604 9	0.386 11	0.632 14	0.225 18	0.705 5	0.434 8	0.293 13	0.815 17	0.348 9	0.241 3	0.499 6	0.669 4	0.507 7	0.649 6	0.442 12	0.796 7	0.602 16	0.561 6

3DMV, FTSDF	0.501 9	0.558 11	0.608 13	0.424 17	0.478 7	0.690 9	0.246 14	0.586 8	0.468 5	0.450 6	0.911 12	0.394 5	0.160 12	0.438 8	0.212 14	0.432 12	0.541 12	0.475 8	0.742 10	0.727 8	0.477 11

PCNN	0.498 10	0.559 10	0.644 9	0.560 12	0.420 10	0.711 8	0.229 16	0.414 11	0.436 7	0.352 10	0.941 6	0.324 10	0.155 13	0.238 15	0.387 9	0.493 8	0.529 13	0.509 5	0.813 6	0.751 7	0.504 10

3DMV	0.484 11	0.484 15	0.538 15	0.643 7	0.424 9	0.606 17	0.310 9	0.574 9	0.433 9	0.378 9	0.796 18	0.301 11	0.214 7	0.537 5	0.208 15	0.472 11	0.507 16	0.413 15	0.693 12	0.602 16	0.539 7
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGB	0.458 12	0.577 9	0.611 12	0.356 19	0.321 16	0.715 7	0.299 11	0.376 14	0.328 16	0.319 11	0.944 3	0.285 13	0.164 11	0.216 17	0.229 13	0.484 10	0.545 11	0.456 10	0.755 9	0.709 9	0.475 12
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
SurfaceConvPF	0.442 13	0.505 13	0.622 11	0.380 18	0.342 15	0.654 12	0.227 17	0.397 13	0.367 14	0.276 15	0.924 10	0.240 15	0.198 8	0.359 10	0.262 11	0.366 15	0.581 9	0.435 13	0.640 14	0.668 13	0.398 14
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
PNET2	0.442 13	0.548 12	0.548 14	0.597 10	0.363 13	0.628 15	0.300 10	0.292 15	0.374 13	0.307 12	0.881 16	0.268 14	0.186 9	0.238 15	0.204 16	0.407 14	0.506 17	0.449 11	0.667 13	0.620 15	0.462 13

Tangent Convolutions	0.438 15	0.437 17	0.646 8	0.474 14	0.369 12	0.645 13	0.353 7	0.258 17	0.282 18	0.279 14	0.918 11	0.298 12	0.147 14	0.283 12	0.294 10	0.487 9	0.562 10	0.427 14	0.619 15	0.633 14	0.352 16
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
SPLAT Net	0.393 16	0.472 16	0.511 16	0.606 8	0.311 17	0.656 11	0.245 15	0.405 12	0.328 16	0.197 18	0.927 9	0.227 17	0.000 20	0.001 20	0.249 12	0.271 20	0.510 14	0.383 17	0.593 16	0.699 10	0.267 18
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF	0.383 17	0.297 19	0.491 17	0.432 16	0.358 14	0.612 16	0.274 12	0.116 19	0.411 11	0.265 16	0.904 15	0.229 16	0.079 18	0.250 13	0.185 17	0.320 18	0.510 14	0.385 16	0.548 17	0.597 18	0.394 15

PointNet++	0.339 18	0.584 8	0.478 18	0.458 15	0.256 19	0.360 20	0.250 13	0.247 18	0.278 19	0.261 17	0.677 20	0.183 18	0.117 16	0.212 18	0.145 19	0.364 16	0.346 20	0.232 20	0.548 17	0.523 19	0.252 19
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNet	0.308 19	0.353 18	0.290 20	0.278 20	0.166 20	0.553 18	0.169 20	0.286 16	0.147 20	0.148 20	0.908 13	0.182 19	0.064 19	0.023 19	0.018 20	0.354 17	0.363 18	0.345 18	0.546 19	0.685 11	0.278 17

ScanNet	0.306 20	0.203 20	0.366 19	0.501 13	0.311 17	0.524 19	0.211 19	0.002 20	0.342 15	0.189 19	0.786 19	0.145 20	0.102 17	0.245 14	0.152 18	0.318 19	0.348 19	0.300 19	0.460 20	0.437 20	0.182 20
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17

Method

Info

avg ap 50%

bathtub

bed

bookshelf

cabinet

chair

counter

curtain

desk

door

otherfurniture

picture

refrigerator

shower curtain

sink

sofa

table

toilet

window

PanopticFusion-inst

0.478 1

0.667 3

0.712 3

0.595 1

0.259 4

0.550 5

0.000 10

0.613 1

0.175 3

0.250 5

0.434 1

0.437 1

0.411 3

0.857 1

0.485 1

0.591 5

0.267 7

0.944 2

0.359 1

ResNet-backbone

0.459 2

1.000 1

0.737 1

0.159 9

0.259 3

0.587 3

0.138 1

0.475 3

0.217 2

0.416 1

0.408 3

0.128 4

0.315 4

0.714 3

0.411 2

0.536 6

0.590 1

0.873 5

0.304 2

MASC

0.447 3

0.528 7

0.555 6

0.381 2

0.382 1

0.633 1

0.002 8

0.509 2

0.260 1

0.361 2

0.432 2

0.327 2

0.451 1

0.571 4

0.367 3

0.639 2

0.386 2

0.980 1

0.276 3

Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.

3D-SIS

0.382 4

1.000 1

0.432 7

0.245 6

0.190 5

0.577 4

0.013 6

0.263 5

0.033 8

0.320 3

0.240 5

0.075 6

0.422 2

0.857 1

0.117 7

0.699 1

0.271 6

0.883 4

0.235 5

UNet-backbone

0.319 5

0.667 3

0.715 2

0.233 7

0.189 6

0.479 6

0.008 7

0.218 6

0.067 7

0.201 6

0.173 6

0.107 5

0.123 7

0.438 5

0.150 5

0.615 3

0.355 3

0.916 3

0.093 9

R-PointNet

0.306 6

0.500 8

0.405 8

0.311 4

0.348 2

0.589 2

0.054 2

0.068 9

0.126 4

0.283 4

0.290 4

0.028 7

0.219 5

0.214 8

0.331 4

0.396 8

0.275 5

0.821 7

0.245 4

3D-BEVIS

0.248 7

0.667 3

0.566 5

0.076 10

0.035 10

0.394 7

0.027 4

0.035 10

0.098 5

0.099 9

0.030 9

0.025 8

0.098 8

0.375 6

0.126 6

0.604 4

0.181 8

0.854 6

0.171 6

Seg-Cluster

0.215 8

0.370 9

0.337 10

0.285 5

0.105 7

0.325 9

0.025 5

0.282 4

0.085 6

0.105 7

0.107 7

0.007 10

0.079 9

0.317 7

0.114 8

0.309 10

0.304 4

0.587 8

0.123 8

MTML

0.212 9

0.667 3

0.614 4

0.337 3

0.027 11

0.390 8

0.000 10

0.118 7

0.001 9

0.100 8

0.028 10

0.000 11

0.167 6

0.143 9

0.046 10

0.500 7

0.105 10

0.570 9

0.003 11

Sgpn_scannet

0.143 10

0.208 11

0.390 9

0.169 8

0.065 8

0.275 10

0.029 3

0.069 8

0.000 10

0.087 10

0.043 8

0.014 9

0.027 11

0.000 10

0.112 9

0.351 9

0.168 9

0.438 10

0.138 7

MaskRCNN 2d->3d Proj

0.058 11

0.333 10

0.002 11

0.000 11

0.053 9

0.002 11

0.002 9

0.021 11

0.000 10

0.045 11

0.024 11

0.238 3

0.065 10

0.000 10

0.014 11

0.107 11

0.020 11

0.110 11

0.006 10

Method

Info

avg iou

bathtub

bed

bookshelf

cabinet

chair

counter

curtain

desk

door

floor

otherfurniture

picture

refrigerator

shower curtain

sink

sofa

table

toilet

wall

window

SSMA

0.577 1

0.695 1

0.716 2

0.439 4

0.563 1

0.314 3

0.444 1

0.719 1

0.551 1

0.503 1

0.887 3

0.346 1

0.348 2

0.603 1

0.353 3

0.709 1

0.600 2

0.457 1

0.901 1

0.786 1

0.599 1

Abhinav Valada, Rohit Mohan, Wolfram Burgard: Self-Supervised Model Adaptation for Multimodal Semantic Segmentation. arXiv

FuseNet

0.521 2

0.591 3

0.682 3

0.220 7

0.488 3

0.279 4

0.344 5

0.610 3

0.461 3

0.475 2

0.910 1

0.293 2

0.447 1

0.512 3

0.397 2

0.618 2

0.567 4

0.452 2

0.734 5

0.782 2

0.566 2

Caner Hazirbas, Lingni Ma, Csaba Domokos, Daniel Cremers: FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-based CNN Architecture. ACCV 2016

AdapNet++

0.503 3

0.613 2

0.722 1

0.418 5

0.358 7

0.337 2

0.370 4

0.479 5

0.443 4

0.368 5

0.907 2

0.207 5

0.213 6

0.464 5

0.525 1

0.618 2

0.657 1

0.450 3

0.788 3

0.721 4

0.408 6

Abhinav Valada, Rohit Mohan, Wolfram Burgard: Self-Supervised Model Adaptation for Multimodal Semantic Segmentation. arXiv

3DMV (2d proj)

0.498 4

0.481 5

0.612 4

0.579 2

0.456 4

0.343 1

0.384 2

0.623 2

0.525 2

0.381 4

0.845 4

0.254 4

0.264 4

0.557 2

0.182 5

0.581 5

0.598 3

0.429 4

0.760 4

0.661 6

0.446 5

Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18

ILC-PSPNet

0.475 5

0.490 4

0.581 5

0.289 6

0.507 2

0.067 7

0.379 3

0.610 3

0.417 6

0.435 3

0.822 6

0.278 3

0.267 3

0.503 4

0.228 4

0.616 4

0.533 5

0.375 5

0.820 2

0.729 3

0.560 3

Enet (reimpl)

0.376 6

0.264 7

0.452 7

0.452 3

0.365 5

0.181 5

0.143 7

0.456 6

0.409 7

0.346 6

0.769 7

0.164 6

0.218 5

0.359 6

0.123 7

0.403 7

0.381 7

0.313 7

0.571 6

0.685 5

0.472 4

Re-implementation of Adam Paszke, Abhishek Chaurasia, Sangpil Kim, Eugenio Culurciello: ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation.

ScanNet (2d proj)

0.330 7

0.293 6

0.521 6

0.657 1

0.361 6

0.161 6

0.250 6

0.004 7

0.440 5

0.183 7

0.836 5

0.125 7

0.060 7

0.319 7

0.132 6

0.417 6

0.412 6

0.344 6

0.541 7

0.427 7

0.109 7

Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17

Method

Info

avg ap

bathtub

bed

bookshelf

cabinet

chair

counter

curtain

desk

door

otherfurniture

picture

refrigerator

shower curtain

sink

sofa

table

toilet

window

MaskRCNN_ScanNet

0.119 1

0.129 1

0.212 1

0.002 1

0.112 1

0.148 1

0.014 1

0.205 1

0.044 1

0.066 1

0.078 1

0.095 1

0.142 1

0.030 1

0.128 1

0.139 1

0.080 1

0.459 1

0.057 1

Re-implementation of Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick: Mask R-CNN. ICCV'17

Method	avg recall	apartment	bathroom	bedroom / hotel	bookstore / library	conference room	copy/mail room	hallway	kitchen	laundry room	living room / lounge	misc	office	storage / basement / garage
SE-ResNeXt-SSMA	0.498 1	0.000 2	0.812 1	0.941 1	0.500 1	0.500 1	0.500 1	0.500 1	0.429 2	0.500 1	0.667 1	0.500 1	0.625 1	0.000 1
Abhinav Valada, Rohit Mohan, Wolfram Burgard: Self-Supervised Model Adaptation for Multimodal Semantic Segmentation. arXiv
resnet50_scannet	0.353 2	0.250 1	0.812 1	0.529 2	0.500 1	0.500 1	0.000 2	0.500 1	0.571 1	0.000 2	0.556 2	0.000 2	0.375 2	0.000 1

Method

Info

avg recall

apartment

bathroom

bedroom / hotel

bookstore / library

conference room

copy/mail room

hallway

kitchen

laundry room

living room / lounge

misc

office

storage / basement / garage

SE-ResNeXt-SSMA

0.498 1

0.000 2

0.812 1

0.941 1

0.500 1

0.429 2

0.500 1

0.667 1

0.500 1

0.625 1

0.000 1

Abhinav Valada, Rohit Mohan, Wolfram Burgard: Self-Supervised Model Adaptation for Multimodal Semantic Segmentation. arXiv

resnet50_scannet

0.353 2

0.250 1

0.812 1

0.529 2

0.500 1

0.000 2

0.500 1

0.571 1

0.000 2

0.556 2

0.000 2

0.375 2

0.000 1

3D Semantic label benchmark

3D Semantic instance benchmark

2D Semantic label benchmark

2D Semantic instance benchmark

Scene type classification benchmark