3D Semantic Instance Results - ScanNet Benchmark

The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.

This table lists the benchmark results for the 3D semantic instance scenario.

Method	avg ap 50%	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	window

OccuSeg+instance	0.672 1	1.000 1	0.758 7	0.682 3	0.576 1	0.842 1	0.477 1	0.504 11	0.524 2	0.567 1	0.585 1	0.451 2	0.557 6	1.000 1	0.751 1	0.797 2	0.563 5	1.000 1	0.467 4

GICN	0.638 2	1.000 1	0.895 1	0.800 1	0.480 5	0.676 7	0.144 3	0.737 2	0.354 6	0.447 4	0.400 10	0.365 6	0.700 1	1.000 1	0.569 3	0.836 1	0.599 2	1.000 1	0.473 3

PointGroup	0.636 3	1.000 1	0.765 5	0.624 4	0.505 4	0.797 3	0.116 6	0.696 3	0.384 5	0.441 5	0.559 2	0.476 1	0.596 4	1.000 1	0.666 2	0.756 4	0.556 8	0.997 6	0.513 1
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, Jiaya Jia: PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. CVPR 2020 [oral]
3D-MPA	0.611 4	1.000 1	0.833 2	0.765 2	0.526 2	0.756 6	0.136 5	0.588 8	0.470 3	0.438 6	0.432 8	0.358 7	0.650 2	0.857 6	0.429 7	0.765 3	0.557 7	1.000 1	0.430 6
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nießner: 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation. CVPR 2020
PCJC	0.578 5	1.000 1	0.810 3	0.583 8	0.449 8	0.813 2	0.042 11	0.603 6	0.341 7	0.490 2	0.465 4	0.410 4	0.650 2	0.835 10	0.264 14	0.694 6	0.561 6	0.889 13	0.504 2

SSEN	0.575 6	1.000 1	0.761 6	0.473 10	0.477 6	0.795 4	0.066 9	0.529 9	0.658 1	0.460 3	0.461 5	0.380 5	0.331 13	0.859 5	0.401 10	0.692 7	0.653 1	1.000 1	0.348 10

MTML	0.549 7	1.000 1	0.807 4	0.588 7	0.327 11	0.647 8	0.004 17	0.815 1	0.180 11	0.418 7	0.364 12	0.182 10	0.445 8	1.000 1	0.442 6	0.688 8	0.571 4	1.000 1	0.396 7
Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald: 3D Instance Segmentation via Multi-task Metric Learning. ICCV 2019 [oral]
Sparse R-CNN	0.515 8	1.000 1	0.538 15	0.282 14	0.468 7	0.790 5	0.173 2	0.345 13	0.429 4	0.413 9	0.484 3	0.176 11	0.595 5	0.591 13	0.522 4	0.668 9	0.476 11	0.986 7	0.327 11

Occipital-SCS	0.512 9	1.000 1	0.716 9	0.509 9	0.506 3	0.611 10	0.092 8	0.602 7	0.177 12	0.346 11	0.383 11	0.165 12	0.442 9	0.850 9	0.386 11	0.618 11	0.543 9	0.889 13	0.389 8

3D-BoNet	0.488 10	1.000 1	0.672 12	0.590 6	0.301 12	0.484 17	0.098 7	0.620 4	0.306 8	0.341 12	0.259 14	0.125 14	0.434 10	0.796 11	0.402 9	0.499 16	0.513 10	0.909 12	0.439 5
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight
PanopticFusion-inst	0.478 11	0.667 13	0.712 11	0.595 5	0.259 14	0.550 15	0.000 20	0.613 5	0.175 13	0.250 15	0.434 6	0.437 3	0.411 12	0.857 6	0.485 5	0.591 14	0.267 18	0.944 9	0.359 9
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
ResNet-backbone	0.459 12	1.000 1	0.737 8	0.159 19	0.259 13	0.587 13	0.138 4	0.475 12	0.217 10	0.416 8	0.408 9	0.128 13	0.315 14	0.714 12	0.411 8	0.536 15	0.590 3	0.873 16	0.304 12

MASC	0.447 13	0.528 17	0.555 14	0.381 11	0.382 9	0.633 9	0.002 18	0.509 10	0.260 9	0.361 10	0.432 7	0.327 8	0.451 7	0.571 14	0.367 12	0.639 10	0.386 12	0.980 8	0.276 13
Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.
3D-SIS	0.382 14	1.000 1	0.432 17	0.245 15	0.190 15	0.577 14	0.013 15	0.263 15	0.033 19	0.320 13	0.240 15	0.075 16	0.422 11	0.857 6	0.117 17	0.699 5	0.271 17	0.883 15	0.235 15
Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019
UNet-backbone	0.319 15	0.667 13	0.715 10	0.233 16	0.189 16	0.479 18	0.008 16	0.218 16	0.067 18	0.201 16	0.173 16	0.107 15	0.123 17	0.438 16	0.150 15	0.615 12	0.355 13	0.916 11	0.093 20

R-PointNet	0.306 16	0.500 19	0.405 18	0.311 12	0.348 10	0.589 12	0.054 10	0.068 19	0.126 14	0.283 14	0.290 13	0.028 18	0.219 15	0.214 19	0.331 13	0.396 18	0.275 16	0.821 18	0.245 14

Region	0.248 17	0.667 13	0.437 16	0.188 17	0.153 17	0.491 16	0.000 20	0.208 17	0.094 16	0.153 18	0.099 18	0.057 17	0.217 16	0.119 20	0.039 21	0.466 17	0.302 15	0.640 19	0.140 17

3D-BEVIS	0.248 17	0.667 13	0.566 13	0.076 20	0.035 22	0.394 19	0.027 13	0.035 20	0.098 15	0.099 20	0.030 21	0.025 19	0.098 18	0.375 17	0.126 16	0.604 13	0.181 20	0.854 17	0.171 16
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.
Seg-Cluster	0.215 19	0.370 20	0.337 20	0.285 13	0.105 18	0.325 20	0.025 14	0.282 14	0.085 17	0.105 19	0.107 17	0.007 22	0.079 19	0.317 18	0.114 18	0.309 20	0.304 14	0.587 20	0.123 19

PE	0.188 20	0.528 17	0.046 21	0.017 21	0.062 20	0.600 11	0.000 20	0.003 22	0.030 20	0.177 17	0.032 20	0.013 21	0.000 22	0.487 15	0.090 20	0.093 22	0.261 19	0.916 10	0.029 21

Sgpn_scannet	0.143 21	0.208 22	0.390 19	0.169 18	0.065 19	0.275 21	0.029 12	0.069 18	0.000 21	0.087 21	0.043 19	0.014 20	0.027 21	0.000 21	0.112 19	0.351 19	0.168 21	0.438 21	0.138 18

MaskRCNN 2d->3d Proj	0.058 22	0.333 21	0.002 22	0.000 22	0.053 21	0.002 22	0.002 19	0.021 21	0.000 21	0.045 22	0.024 22	0.238 9	0.065 20	0.000 21	0.014 22	0.107 21	0.020 22	0.110 22	0.006 22

3D Semantic instance benchmark