Benchmark Results - ScanNet Benchmark

This table lists the benchmark results for the 3D semantic label scenario.

Method	avg iou	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	floor	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	wall	window

SparseConvNet	0.726 1	0.629 6	0.801 1	0.858 1	0.713 1	0.884 1	0.505 1	0.799 1	0.636 2	0.628 1	0.956 1	0.602 1	0.299 1	0.712 1	0.858 1	0.717 1	0.808 1	0.629 2	0.929 1	0.858 1	0.694 1

MinkowskiNet34	0.679 2	0.811 1	0.734 3	0.739 3	0.641 2	0.804 2	0.413 4	0.759 3	0.696 1	0.545 2	0.938 5	0.518 2	0.141 10	0.623 2	0.757 2	0.680 2	0.723 3	0.684 1	0.896 2	0.821 2	0.651 2

joint point-based	0.621 3	0.645 4	0.746 2	0.612 6	0.571 4	0.795 3	0.386 5	0.798 2	0.485 4	0.539 3	0.943 4	0.445 3	0.287 2	0.520 4	0.418 6	0.635 3	0.744 2	0.570 3	0.859 3	0.795 3	0.628 3

TextureNet	0.566 4	0.672 2	0.664 5	0.671 4	0.494 5	0.719 5	0.445 2	0.678 4	0.411 8	0.396 7	0.935 6	0.356 7	0.225 3	0.412 7	0.535 4	0.565 4	0.636 6	0.464 8	0.794 6	0.680 10	0.568 4

DVVNet	0.562 5	0.648 3	0.700 4	0.770 2	0.586 3	0.687 8	0.333 7	0.650 5	0.514 3	0.475 4	0.906 12	0.359 6	0.223 4	0.340 9	0.442 5	0.422 10	0.668 4	0.501 6	0.708 8	0.779 4	0.534 6

PointConv	0.556 6	0.636 5	0.640 7	0.574 8	0.472 7	0.739 4	0.430 3	0.433 8	0.418 7	0.445 6	0.944 2	0.372 5	0.185 7	0.464 5	0.575 3	0.540 5	0.639 5	0.505 5	0.827 4	0.762 5	0.515 7

3DMV, FTSDF	0.501 7	0.558 8	0.608 9	0.424 13	0.478 6	0.690 7	0.246 11	0.586 6	0.468 5	0.450 5	0.911 10	0.394 4	0.160 8	0.438 6	0.212 11	0.432 9	0.541 9	0.475 7	0.742 7	0.727 7	0.477 9

3DMV	0.484 8	0.484 11	0.538 11	0.643 5	0.424 8	0.606 13	0.310 8	0.574 7	0.433 6	0.378 8	0.796 14	0.301 8	0.214 5	0.537 3	0.208 12	0.472 8	0.507 13	0.413 11	0.693 9	0.602 13	0.539 5
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGB	0.479 9	0.510 9	0.583 10	0.417 14	0.414 9	0.708 6	0.241 13	0.367 11	0.405 10	0.323 9	0.944 2	0.300 9	0.132 11	0.226 13	0.417 7	0.534 6	0.525 10	0.511 4	0.806 5	0.743 6	0.479 8
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NIPS 2018
SurfaceConvPF	0.442 10	0.505 10	0.622 8	0.380 15	0.342 12	0.654 10	0.227 14	0.397 10	0.367 11	0.276 11	0.924 8	0.240 11	0.198 6	0.359 8	0.262 9	0.366 11	0.581 7	0.435 9	0.640 10	0.668 11	0.398 10
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
Tangent Convolutions	0.438 11	0.437 13	0.646 6	0.474 10	0.369 10	0.645 11	0.353 6	0.258 13	0.282 14	0.279 10	0.918 9	0.298 10	0.147 9	0.283 10	0.294 8	0.487 7	0.562 8	0.427 10	0.619 11	0.633 12	0.352 12
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
SPLAT Net	0.393 12	0.472 12	0.511 12	0.606 7	0.311 13	0.656 9	0.245 12	0.405 9	0.328 13	0.197 14	0.927 7	0.227 13	0.000 16	0.001 16	0.249 10	0.271 16	0.510 11	0.383 13	0.593 12	0.699 8	0.267 14
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF	0.383 13	0.297 15	0.491 13	0.432 12	0.358 11	0.612 12	0.274 9	0.116 15	0.411 8	0.265 12	0.904 13	0.229 12	0.079 14	0.250 11	0.185 13	0.320 14	0.510 11	0.385 12	0.548 13	0.597 14	0.394 11

PointNet++	0.339 14	0.584 7	0.478 14	0.458 11	0.256 15	0.360 16	0.250 10	0.247 14	0.278 15	0.261 13	0.677 16	0.183 14	0.117 12	0.212 14	0.145 15	0.364 12	0.346 16	0.232 16	0.548 13	0.523 15	0.252 15
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNet	0.308 15	0.353 14	0.290 16	0.278 16	0.166 16	0.553 14	0.169 16	0.286 12	0.147 16	0.148 16	0.908 11	0.182 15	0.064 15	0.023 15	0.018 16	0.354 13	0.363 14	0.345 14	0.546 15	0.685 9	0.278 13

ScanNet	0.306 16	0.203 16	0.366 15	0.501 9	0.311 13	0.524 15	0.211 15	0.002 16	0.342 12	0.189 15	0.786 15	0.145 16	0.102 13	0.245 12	0.152 14	0.318 15	0.348 15	0.300 15	0.460 16	0.437 16	0.182 16
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17

Method

Info

avg ap 50%

bathtub

bed

bookshelf

cabinet

chair

counter

curtain

desk

door

otherfurniture

picture

refrigerator

shower curtain

sink

sofa

table

toilet

window

R-PointNet

0.306 1

0.500 4

0.405 4

0.311 2

0.348 1

0.589 1

0.054 1

0.068 4

0.126 2

0.283 1

0.290 1

0.028 3

0.219 1

0.214 3

0.331 1

0.396 4

0.275 1

0.821 2

0.245 1

3D-SIS

0.271 2

0.528 3

0.558 3

0.241 3

0.272 2

0.488 2

0.000 5

0.093 2

0.167 1

0.179 2

0.231 2

0.040 2

0.182 2

0.286 2

0.168 2

0.546 2

0.155 4

0.555 4

0.185 2

3D-BEVIS

0.248 3

0.667 1

0.566 2

0.076 5

0.035 5

0.394 3

0.027 3

0.035 5

0.098 3

0.099 4

0.030 4

0.025 4

0.098 4

0.375 1

0.126 3

0.604 1

0.181 2

0.854 1

0.171 3

MTML

0.212 4

0.667 1

0.614 1

0.337 1

0.027 6

0.390 4

0.000 5

0.118 1

0.001 4

0.100 3

0.028 5

0.000 6

0.167 3

0.143 4

0.046 5

0.500 3

0.105 5

0.570 3

0.003 6

Sgpn_scannet

0.143 5

0.208 6

0.390 5

0.169 4

0.065 3

0.275 5

0.029 2

0.069 3

0.000 5

0.087 5

0.043 3

0.014 5

0.027 6

0.000 5

0.112 4

0.351 5

0.168 3

0.438 5

0.138 4

MaskRCNN 2d->3d Proj

0.058 6

0.333 5

0.002 6

0.000 6

0.053 4

0.002 6

0.002 4

0.021 6

0.000 5

0.045 6

0.024 6

0.238 1

0.065 5

0.000 5

0.014 6

0.107 6

0.020 6

0.110 6

0.006 5

Method

Info

avg iou

bathtub

bed

bookshelf

cabinet

chair

counter

curtain

desk

door

floor

otherfurniture

picture

refrigerator

shower curtain

sink

sofa

table

toilet

wall

window

FuseNet

0.521 1

0.591 1

0.682 1

0.220 5

0.488 2

0.279 2

0.344 3

0.610 2

0.461 2

0.475 1

0.910 1

0.293 1

0.447 1

0.512 2

0.397 1

0.618 1

0.567 2

0.452 1

0.734 3

0.782 1

0.566 1

Caner Hazirbas, Lingni Ma, Csaba Domokos, Daniel Cremers: FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-based CNN Architecture. ACCV 2016

3DMV (2d proj)

0.498 2

0.481 3

0.612 2

0.579 2

0.456 3

0.343 1

0.384 1

0.623 1

0.525 1

0.381 3

0.845 2

0.254 3

0.264 3

0.557 1

0.182 3

0.581 3

0.598 1

0.429 2

0.760 2

0.661 4

0.446 4

Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18

ILC-PSPNet

0.475 3

0.490 2

0.581 3

0.289 4

0.507 1

0.067 5

0.379 2

0.610 2

0.417 4

0.435 2

0.822 4

0.278 2

0.267 2

0.503 3

0.228 2

0.616 2

0.533 3

0.375 3

0.820 1

0.729 2

0.560 2

Enet (reimpl)

0.376 4

0.264 5

0.452 5

0.452 3

0.365 4

0.181 3

0.143 5

0.456 4

0.409 5

0.346 4

0.769 5

0.164 4

0.218 4

0.359 4

0.123 5

0.403 5

0.381 5

0.313 5

0.571 4

0.685 3

0.472 3

Re-implementation of Adam Paszke, Abhishek Chaurasia, Sangpil Kim, Eugenio Culurciello: ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation.

ScanNet (2d proj)

0.330 5

0.293 4

0.521 4

0.657 1

0.361 5

0.161 4

0.250 4

0.004 5

0.440 3

0.183 5

0.836 3

0.125 5

0.060 5

0.319 5

0.132 4

0.417 4

0.412 4

0.344 4

0.541 5

0.427 5

0.109 5

Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17

Method

Info

avg ap

bathtub

bed

bookshelf

cabinet

chair

counter

curtain

desk

door

otherfurniture

picture

refrigerator

shower curtain

sink

sofa

table

toilet

window

MaskRCNN_ScanNet

0.119 1

0.129 1

0.212 1

0.002 1

0.112 1

0.148 1

0.014 1

0.205 1

0.044 1

0.066 1

0.078 1

0.095 1

0.142 1

0.030 1

0.128 1

0.139 1

0.080 1

0.459 1

0.057 1

Re-implementation of Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick: Mask R-CNN. ICCV'17

Method	Info	avg recall	apartment	bathroom	bedroom / hotel	bookstore / library	conference room	copy/mail room	hallway	kitchen	laundry room	living room / lounge	misc	office	storage / basement / garage
resnet50_scannet		0.353 1	0.250 1	0.812 1	0.529 1	0.500 1	0.500 1	0.000 1	0.500 1	0.571 1	0.000 1	0.556 1	0.000 1	0.375 1	0.000 1

Method

Info

avg recall

apartment

bathroom

bedroom / hotel

bookstore / library

conference room

copy/mail room

hallway

kitchen

laundry room

living room / lounge

misc

office

storage / basement / garage

resnet50_scannet

0.353 1

0.250 1

0.812 1

0.529 1

0.500 1

0.000 1

0.500 1

0.571 1

0.000 1

0.556 1

0.000 1

0.375 1

0.000 1

3D Semantic label benchmark

3D Semantic instance benchmark

2D Semantic label benchmark

2D Semantic instance benchmark

Scene type classification benchmark