3D Semantic Label Results - ScanNet Benchmark

The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).

This table lists the benchmark results for the 3D semantic label scenario.

Method	avg iou	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	floor	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	wall	window

SparseConvNet	0.726 1	0.629 6	0.801 1	0.858 1	0.713 1	0.884 1	0.505 1	0.799 1	0.636 2	0.628 1	0.956 1	0.602 1	0.299 1	0.712 1	0.858 1	0.717 1	0.808 1	0.629 2	0.929 1	0.858 1	0.694 1

MinkowskiNet34	0.679 2	0.811 1	0.734 3	0.739 3	0.641 2	0.804 2	0.413 4	0.759 3	0.696 1	0.545 2	0.938 5	0.518 2	0.141 10	0.623 2	0.757 2	0.680 2	0.723 3	0.684 1	0.896 2	0.821 2	0.651 2

joint point-based	0.621 3	0.645 4	0.746 2	0.612 6	0.571 4	0.795 3	0.386 5	0.798 2	0.485 4	0.539 3	0.943 4	0.445 3	0.287 2	0.520 4	0.418 6	0.635 3	0.744 2	0.570 3	0.859 3	0.795 3	0.628 3

TextureNet	0.566 4	0.672 2	0.664 5	0.671 4	0.494 5	0.719 5	0.445 2	0.678 4	0.411 8	0.396 7	0.935 6	0.356 7	0.225 3	0.412 7	0.535 4	0.565 4	0.636 6	0.464 8	0.794 6	0.680 10	0.568 4

DVVNet	0.562 5	0.648 3	0.700 4	0.770 2	0.586 3	0.687 8	0.333 7	0.650 5	0.514 3	0.475 4	0.906 12	0.359 6	0.223 4	0.340 9	0.442 5	0.422 10	0.668 4	0.501 6	0.708 8	0.779 4	0.534 6

PointConv	0.556 6	0.636 5	0.640 7	0.574 8	0.472 7	0.739 4	0.430 3	0.433 8	0.418 7	0.445 6	0.944 2	0.372 5	0.185 7	0.464 5	0.575 3	0.540 5	0.639 5	0.505 5	0.827 4	0.762 5	0.515 7

3DMV, FTSDF	0.501 7	0.558 8	0.608 9	0.424 13	0.478 6	0.690 7	0.246 11	0.586 6	0.468 5	0.450 5	0.911 10	0.394 4	0.160 8	0.438 6	0.212 11	0.432 9	0.541 9	0.475 7	0.742 7	0.727 7	0.477 9

3DMV	0.484 8	0.484 11	0.538 11	0.643 5	0.424 8	0.606 13	0.310 8	0.574 7	0.433 6	0.378 8	0.796 14	0.301 8	0.214 5	0.537 3	0.208 12	0.472 8	0.507 13	0.413 11	0.693 9	0.602 13	0.539 5
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGB	0.479 9	0.510 9	0.583 10	0.417 14	0.414 9	0.708 6	0.241 13	0.367 11	0.405 10	0.323 9	0.944 2	0.300 9	0.132 11	0.226 13	0.417 7	0.534 6	0.525 10	0.511 4	0.806 5	0.743 6	0.479 8
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NIPS 2018
SurfaceConvPF	0.442 10	0.505 10	0.622 8	0.380 15	0.342 12	0.654 10	0.227 14	0.397 10	0.367 11	0.276 11	0.924 8	0.240 11	0.198 6	0.359 8	0.262 9	0.366 11	0.581 7	0.435 9	0.640 10	0.668 11	0.398 10
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
Tangent Convolutions	0.438 11	0.437 13	0.646 6	0.474 10	0.369 10	0.645 11	0.353 6	0.258 13	0.282 14	0.279 10	0.918 9	0.298 10	0.147 9	0.283 10	0.294 8	0.487 7	0.562 8	0.427 10	0.619 11	0.633 12	0.352 12
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
SPLAT Net	0.393 12	0.472 12	0.511 12	0.606 7	0.311 13	0.656 9	0.245 12	0.405 9	0.328 13	0.197 14	0.927 7	0.227 13	0.000 16	0.001 16	0.249 10	0.271 16	0.510 11	0.383 13	0.593 12	0.699 8	0.267 14
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF	0.383 13	0.297 15	0.491 13	0.432 12	0.358 11	0.612 12	0.274 9	0.116 15	0.411 8	0.265 12	0.904 13	0.229 12	0.079 14	0.250 11	0.185 13	0.320 14	0.510 11	0.385 12	0.548 13	0.597 14	0.394 11

PointNet++	0.339 14	0.584 7	0.478 14	0.458 11	0.256 15	0.360 16	0.250 10	0.247 14	0.278 15	0.261 13	0.677 16	0.183 14	0.117 12	0.212 14	0.145 15	0.364 12	0.346 16	0.232 16	0.548 13	0.523 15	0.252 15
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNet	0.308 15	0.353 14	0.290 16	0.278 16	0.166 16	0.553 14	0.169 16	0.286 12	0.147 16	0.148 16	0.908 11	0.182 15	0.064 15	0.023 15	0.018 16	0.354 13	0.363 14	0.345 14	0.546 15	0.685 9	0.278 13

ScanNet	0.306 16	0.203 16	0.366 15	0.501 9	0.311 13	0.524 15	0.211 15	0.002 16	0.342 12	0.189 15	0.786 15	0.145 16	0.102 13	0.245 12	0.152 14	0.318 15	0.348 15	0.300 15	0.460 16	0.437 16	0.182 16
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17

3D Semantic label benchmark