3D Semantic Label Results - ScanNet Benchmark

The 3D semantic labeling task involves predicting a semantic labeling of a 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the PASCAL VOC intersection-over-union metric (IoU). IoU = TP/(TP+FP+FN), where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively. Predicted labels are evaluated per-vertex over the respective 3D scan mesh; for 3D approaches that operate on other representations like grids or points, the predicted labels should be mapped onto the mesh vertices (e.g., one such example for grid to mesh vertices is provided in the evaluation helpers).

This table lists the benchmark results for the 3D semantic label scenario.

Method	avg iou	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	floor	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	wall	window

SparseConvNet	0.725 1	0.647 5	0.821 1	0.846 1	0.721 1	0.869 1	0.533 1	0.754 4	0.603 2	0.614 2	0.955 1	0.572 1	0.325 1	0.710 1	0.870 2	0.724 1	0.823 1	0.628 2	0.934 1	0.865 1	0.683 1

KP-FCNN	0.694 2	0.849 1	0.770 3	0.810 2	0.685 2	0.813 3	0.438 3	0.791 2	0.566 3	0.616 1	0.944 3	0.500 3	0.216 6	0.559 4	0.880 1	0.690 2	0.758 3	0.627 3	0.922 2	0.832 2	0.613 3

MinkowskiNet34	0.679 3	0.811 2	0.734 4	0.739 4	0.641 3	0.804 4	0.413 6	0.759 3	0.696 1	0.545 4	0.938 7	0.518 2	0.141 15	0.623 2	0.757 3	0.680 3	0.723 4	0.684 1	0.896 3	0.821 3	0.651 2

joint point-based	0.634 4	0.614 7	0.778 2	0.667 6	0.633 4	0.825 2	0.420 5	0.804 1	0.467 6	0.561 3	0.951 2	0.494 4	0.291 2	0.566 3	0.458 7	0.579 4	0.764 2	0.559 4	0.838 4	0.814 4	0.598 4

TextureNet	0.566 5	0.672 3	0.664 7	0.671 5	0.494 6	0.719 6	0.445 2	0.678 6	0.411 11	0.396 8	0.935 8	0.356 8	0.225 4	0.412 9	0.535 6	0.565 5	0.636 8	0.464 9	0.794 8	0.680 12	0.568 5

DVVNet	0.562 6	0.648 4	0.700 5	0.770 3	0.586 5	0.687 10	0.333 8	0.650 7	0.514 4	0.475 5	0.906 14	0.359 7	0.223 5	0.340 11	0.442 8	0.422 13	0.668 5	0.501 7	0.708 11	0.779 5	0.534 8

PointConv	0.556 7	0.636 6	0.640 10	0.574 11	0.472 8	0.739 5	0.430 4	0.433 10	0.418 10	0.445 7	0.944 3	0.372 6	0.185 10	0.464 7	0.575 5	0.540 6	0.639 7	0.505 6	0.827 5	0.762 6	0.515 9

PanopticFusion-label	0.529 8	0.491 14	0.688 6	0.604 9	0.386 11	0.632 14	0.225 18	0.705 5	0.434 8	0.293 13	0.815 17	0.348 9	0.241 3	0.499 6	0.669 4	0.507 7	0.649 6	0.442 12	0.796 7	0.602 16	0.561 6

3DMV, FTSDF	0.501 9	0.558 11	0.608 13	0.424 17	0.478 7	0.690 9	0.246 14	0.586 8	0.468 5	0.450 6	0.911 12	0.394 5	0.160 12	0.438 8	0.212 14	0.432 12	0.541 12	0.475 8	0.742 10	0.727 8	0.477 11

PCNN	0.498 10	0.559 10	0.644 9	0.560 12	0.420 10	0.711 8	0.229 16	0.414 11	0.436 7	0.352 10	0.941 6	0.324 10	0.155 13	0.238 15	0.387 9	0.493 8	0.529 13	0.509 5	0.813 6	0.751 7	0.504 10

3DMV	0.484 11	0.484 15	0.538 15	0.643 7	0.424 9	0.606 17	0.310 9	0.574 9	0.433 9	0.378 9	0.796 18	0.301 11	0.214 7	0.537 5	0.208 15	0.472 11	0.507 16	0.413 15	0.693 12	0.602 16	0.539 7
Angela Dai, Matthias Niessner: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation. ECCV'18
PointCNN with RGB	0.458 12	0.577 9	0.611 12	0.356 19	0.321 16	0.715 7	0.299 11	0.376 14	0.328 16	0.319 11	0.944 3	0.285 13	0.164 11	0.216 17	0.229 13	0.484 10	0.545 11	0.456 10	0.755 9	0.709 9	0.475 12
Yangyan Li, Rui Bu, Mingchao Sun, Baoquan Chen: PointCNN. NeurIPS 2018
SurfaceConvPF	0.442 13	0.505 13	0.622 11	0.380 18	0.342 15	0.654 12	0.227 17	0.397 13	0.367 14	0.276 15	0.924 10	0.240 15	0.198 8	0.359 10	0.262 11	0.366 15	0.581 9	0.435 13	0.640 14	0.668 13	0.398 14
Hao Pan, Shilin Liu, Yang Liu, Xin Tong: Convolutional Neural Networks on 3D Surfaces Using Parallel Frames.
PNET2	0.442 13	0.548 12	0.548 14	0.597 10	0.363 13	0.628 15	0.300 10	0.292 15	0.374 13	0.307 12	0.881 16	0.268 14	0.186 9	0.238 15	0.204 16	0.407 14	0.506 17	0.449 11	0.667 13	0.620 15	0.462 13

Tangent Convolutions	0.438 15	0.437 17	0.646 8	0.474 14	0.369 12	0.645 13	0.353 7	0.258 17	0.282 18	0.279 14	0.918 11	0.298 12	0.147 14	0.283 12	0.294 10	0.487 9	0.562 10	0.427 14	0.619 15	0.633 14	0.352 16
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, Qian-Yi Zhou: Tangent convolutions for dense prediction in 3d. CVPR 2018
SPLAT Net	0.393 16	0.472 16	0.511 16	0.606 8	0.311 17	0.656 11	0.245 15	0.405 12	0.328 16	0.197 18	0.927 9	0.227 17	0.000 20	0.001 20	0.249 12	0.271 20	0.510 14	0.383 17	0.593 16	0.699 10	0.267 18
Hang Su, Varun Jampani, Deqing Sun, Subhransu Maji, Evangelos Kalogerakis, Ming-Hsuan Yang, Jan Kautz: SPLATNet: Sparse Lattice Networks for Point Cloud Processing. CVPR 2018
ScanNet+FTSDF	0.383 17	0.297 19	0.491 17	0.432 16	0.358 14	0.612 16	0.274 12	0.116 19	0.411 11	0.265 16	0.904 15	0.229 16	0.079 18	0.250 13	0.185 17	0.320 18	0.510 14	0.385 16	0.548 17	0.597 18	0.394 15

PointNet++	0.339 18	0.584 8	0.478 18	0.458 15	0.256 19	0.360 20	0.250 13	0.247 18	0.278 19	0.261 17	0.677 20	0.183 18	0.117 16	0.212 18	0.145 19	0.364 16	0.346 20	0.232 20	0.548 17	0.523 19	0.252 19
Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: pointnet++: deep hierarchical feature learning on point sets in a metric space.
SSC-UNet	0.308 19	0.353 18	0.290 20	0.278 20	0.166 20	0.553 18	0.169 20	0.286 16	0.147 20	0.148 20	0.908 13	0.182 19	0.064 19	0.023 19	0.018 20	0.354 17	0.363 18	0.345 18	0.546 19	0.685 11	0.278 17

ScanNet	0.306 20	0.203 20	0.366 19	0.501 13	0.311 17	0.524 19	0.211 19	0.002 20	0.342 15	0.189 19	0.786 19	0.145 20	0.102 17	0.245 14	0.152 18	0.318 19	0.348 19	0.300 19	0.460 20	0.437 20	0.182 20
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. CVPR'17

3D Semantic label benchmark