3D Semantic Instance Results - ScanNet Benchmark

The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.

This table lists the benchmark results for the 3D semantic instance scenario.

Method	avg ap 50%	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	window

OccuSeg+instance	0.672 1	1.000 1	0.758 6	0.682 3	0.576 1	0.842 1	0.477 1	0.504 10	0.524 2	0.567 1	0.585 1	0.451 2	0.557 4	1.000 1	0.751 1	0.797 2	0.563 5	1.000 1	0.467 3

GICN	0.638 2	1.000 1	0.895 1	0.800 1	0.480 5	0.676 5	0.144 2	0.737 2	0.354 5	0.447 3	0.400 8	0.365 5	0.700 1	1.000 1	0.569 3	0.836 1	0.599 2	1.000 1	0.473 2

PointGroup	0.636 3	1.000 1	0.765 4	0.624 4	0.505 4	0.797 2	0.116 5	0.696 3	0.384 4	0.441 4	0.559 2	0.476 1	0.596 3	1.000 1	0.666 2	0.756 4	0.556 7	0.997 6	0.513 1

MPA	0.611 4	1.000 1	0.833 2	0.765 2	0.526 2	0.756 4	0.136 4	0.588 7	0.470 3	0.438 5	0.432 6	0.358 6	0.650 2	0.857 6	0.429 6	0.765 3	0.557 6	1.000 1	0.430 5

SSEN	0.575 5	1.000 1	0.761 5	0.473 9	0.477 6	0.795 3	0.066 8	0.529 8	0.658 1	0.460 2	0.461 3	0.380 4	0.331 11	0.859 5	0.401 9	0.692 6	0.653 1	1.000 1	0.348 9

MTML	0.549 6	1.000 1	0.807 3	0.588 7	0.327 9	0.647 6	0.004 15	0.815 1	0.180 9	0.418 6	0.364 10	0.182 9	0.445 6	1.000 1	0.442 5	0.688 7	0.571 4	1.000 1	0.396 6
Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald: 3D Instance Segmentation via Multi-task Metric Learning. ICCV 2019 [oral]
Occipital-SCS	0.512 7	1.000 1	0.716 8	0.509 8	0.506 3	0.611 8	0.092 7	0.602 6	0.177 10	0.346 9	0.383 9	0.165 10	0.442 7	0.850 9	0.386 10	0.618 9	0.543 8	0.889 11	0.389 7

3D-BoNet	0.488 8	1.000 1	0.672 11	0.590 6	0.301 10	0.484 14	0.098 6	0.620 4	0.306 6	0.341 10	0.259 12	0.125 12	0.434 8	0.796 10	0.402 8	0.499 14	0.513 9	0.909 10	0.439 4
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight
PanopticFusion-inst	0.478 9	0.667 11	0.712 10	0.595 5	0.259 12	0.550 12	0.000 18	0.613 5	0.175 11	0.250 13	0.434 4	0.437 3	0.411 10	0.857 6	0.485 4	0.591 12	0.267 16	0.944 8	0.359 8
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
ResNet-backbone	0.459 10	1.000 1	0.737 7	0.159 17	0.259 11	0.587 10	0.138 3	0.475 11	0.217 8	0.416 7	0.408 7	0.128 11	0.315 12	0.714 11	0.411 7	0.536 13	0.590 3	0.873 13	0.304 10

MASC	0.447 11	0.528 15	0.555 13	0.381 10	0.382 7	0.633 7	0.002 16	0.509 9	0.260 7	0.361 8	0.432 5	0.327 7	0.451 5	0.571 12	0.367 11	0.639 8	0.386 10	0.980 7	0.276 11
Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.
3D-SIS	0.382 12	1.000 1	0.432 15	0.245 13	0.190 13	0.577 11	0.013 13	0.263 13	0.033 17	0.320 11	0.240 13	0.075 14	0.422 9	0.857 6	0.117 15	0.699 5	0.271 15	0.883 12	0.235 13
Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019
UNet-backbone	0.319 13	0.667 11	0.715 9	0.233 14	0.189 14	0.479 15	0.008 14	0.218 14	0.067 16	0.201 14	0.173 14	0.107 13	0.123 15	0.438 13	0.150 13	0.615 10	0.355 11	0.916 9	0.093 18

R-PointNet	0.306 14	0.500 16	0.405 16	0.311 11	0.348 8	0.589 9	0.054 9	0.068 17	0.126 12	0.283 12	0.290 11	0.028 16	0.219 13	0.214 16	0.331 12	0.396 16	0.275 14	0.821 15	0.245 12

RegionNet	0.248 15	0.667 11	0.437 14	0.188 15	0.153 15	0.491 13	0.000 18	0.208 15	0.094 14	0.153 15	0.099 16	0.057 15	0.217 14	0.119 17	0.039 18	0.466 15	0.302 13	0.640 16	0.140 15

3D-BEVIS	0.248 15	0.667 11	0.566 12	0.076 18	0.035 19	0.394 16	0.027 11	0.035 18	0.098 13	0.099 17	0.030 18	0.025 17	0.098 16	0.375 14	0.126 14	0.604 11	0.181 17	0.854 14	0.171 14
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.
Seg-Cluster	0.215 17	0.370 17	0.337 18	0.285 12	0.105 16	0.325 17	0.025 12	0.282 12	0.085 15	0.105 16	0.107 15	0.007 19	0.079 17	0.317 15	0.114 16	0.309 18	0.304 12	0.587 17	0.123 17

Sgpn_scannet	0.143 18	0.208 19	0.390 17	0.169 16	0.065 17	0.275 18	0.029 10	0.069 16	0.000 18	0.087 18	0.043 17	0.014 18	0.027 19	0.000 18	0.112 17	0.351 17	0.168 18	0.438 18	0.138 16

MaskRCNN 2d->3d Proj	0.058 19	0.333 18	0.002 19	0.000 19	0.053 18	0.002 19	0.002 17	0.021 19	0.000 18	0.045 19	0.024 19	0.238 8	0.065 18	0.000 18	0.014 19	0.107 19	0.020 19	0.110 19	0.006 19

3D Semantic instance benchmark