3D Semantic Instance Results - ScanNet Benchmark

The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.

This table lists the benchmark results for the 3D semantic instance scenario.

Method	avg ap 50%	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	window

OccuSeg+instance	0.672 1	1.000 1	0.758 7	0.682 3	0.576 2	0.842 1	0.477 1	0.504 12	0.524 2	0.567 1	0.585 1	0.451 2	0.557 6	1.000 1	0.751 1	0.797 2	0.563 6	1.000 1	0.467 4

GICN	0.638 2	1.000 1	0.895 1	0.800 1	0.480 6	0.676 8	0.144 4	0.737 2	0.354 7	0.447 5	0.400 11	0.365 6	0.700 1	1.000 1	0.569 3	0.836 1	0.599 3	1.000 1	0.473 3

PointGroup	0.636 3	1.000 1	0.765 5	0.624 5	0.505 5	0.797 4	0.116 7	0.696 3	0.384 6	0.441 6	0.559 2	0.476 1	0.596 4	1.000 1	0.666 2	0.756 5	0.556 9	0.997 6	0.513 1
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, Jiaya Jia: PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. CVPR 2020 [oral]
3D-MPA	0.611 4	1.000 1	0.833 2	0.765 2	0.526 3	0.756 7	0.136 6	0.588 8	0.470 4	0.438 7	0.432 9	0.358 7	0.650 2	0.857 6	0.429 8	0.765 4	0.557 8	1.000 1	0.430 6
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nießner: 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation. CVPR 2020
PCJC	0.578 5	1.000 1	0.810 3	0.583 9	0.449 9	0.813 2	0.042 12	0.603 6	0.341 8	0.490 3	0.465 4	0.410 4	0.650 2	0.835 10	0.264 15	0.694 7	0.561 7	0.889 13	0.504 2

SSEN	0.575 6	1.000 1	0.761 6	0.473 11	0.477 7	0.795 5	0.066 10	0.529 9	0.658 1	0.460 4	0.461 5	0.380 5	0.331 14	0.859 5	0.401 11	0.692 8	0.653 1	1.000 1	0.348 10
Dongsu Zhang, Junha Chun, Sang Kyun Cha, Young Min Kim: Spatial Semantic Embedding Network: Fast 3D Instance Segmentation with Deep Metric Learning. Arxiv
PE	0.560 7	0.667 13	0.722 9	0.673 4	0.577 1	0.806 3	0.202 2	0.520 10	0.524 3	0.491 2	0.450 6	0.061 17	0.553 7	0.688 13	0.507 5	0.777 3	0.604 2	0.941 10	0.322 12

MTML	0.549 8	1.000 1	0.807 4	0.588 8	0.327 12	0.647 9	0.004 18	0.815 1	0.180 12	0.418 8	0.364 13	0.182 10	0.445 9	1.000 1	0.442 7	0.688 9	0.571 5	1.000 1	0.396 7
Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald: 3D Instance Segmentation via Multi-task Metric Learning. ICCV 2019 [oral]
Sparse R-CNN	0.515 9	1.000 1	0.538 16	0.282 15	0.468 8	0.790 6	0.173 3	0.345 14	0.429 5	0.413 10	0.484 3	0.176 11	0.595 5	0.591 14	0.522 4	0.668 10	0.476 12	0.986 7	0.327 11

Occipital-SCS	0.512 10	1.000 1	0.716 10	0.509 10	0.506 4	0.611 11	0.092 9	0.602 7	0.177 13	0.346 12	0.383 12	0.165 12	0.442 10	0.850 9	0.386 12	0.618 12	0.543 10	0.889 13	0.389 8

3D-BoNet	0.488 11	1.000 1	0.672 13	0.590 7	0.301 13	0.484 17	0.098 8	0.620 4	0.306 9	0.341 13	0.259 15	0.125 14	0.434 11	0.796 11	0.402 10	0.499 17	0.513 11	0.909 12	0.439 5
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight
PanopticFusion-inst	0.478 12	0.667 13	0.712 12	0.595 6	0.259 15	0.550 15	0.000 21	0.613 5	0.175 14	0.250 16	0.434 7	0.437 3	0.411 13	0.857 6	0.485 6	0.591 15	0.267 19	0.944 9	0.359 9
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
SALoss-ResNet	0.459 13	1.000 1	0.737 8	0.159 20	0.259 14	0.587 13	0.138 5	0.475 13	0.217 11	0.416 9	0.408 10	0.128 13	0.315 15	0.714 12	0.411 9	0.536 16	0.590 4	0.873 16	0.304 13
Zhidong Liang, Ming Yang, Hao Li, Chunxiang Wang: 3D Instance Embedding Learning With a Structure-Aware Loss Function for Point Cloud Segmentation. IEEE Robotics and Automation Letters (IROS2020)
MASC	0.447 14	0.528 18	0.555 15	0.381 12	0.382 10	0.633 10	0.002 19	0.509 11	0.260 10	0.361 11	0.432 8	0.327 8	0.451 8	0.571 15	0.367 13	0.639 11	0.386 13	0.980 8	0.276 14
Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.
3D-SIS	0.382 15	1.000 1	0.432 18	0.245 16	0.190 16	0.577 14	0.013 16	0.263 16	0.033 20	0.320 14	0.240 16	0.075 16	0.422 12	0.857 6	0.117 18	0.699 6	0.271 18	0.883 15	0.235 16
Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019
UNet-backbone	0.319 16	0.667 13	0.715 11	0.233 17	0.189 17	0.479 18	0.008 17	0.218 17	0.067 19	0.201 17	0.173 17	0.107 15	0.123 18	0.438 16	0.150 16	0.615 13	0.355 14	0.916 11	0.093 21

R-PointNet	0.306 17	0.500 19	0.405 19	0.311 13	0.348 11	0.589 12	0.054 11	0.068 20	0.126 15	0.283 15	0.290 14	0.028 19	0.219 16	0.214 19	0.331 14	0.396 19	0.275 17	0.821 18	0.245 15

Region	0.248 18	0.667 13	0.437 17	0.188 18	0.153 18	0.491 16	0.000 21	0.208 18	0.094 17	0.153 18	0.099 19	0.057 18	0.217 17	0.119 20	0.039 21	0.466 18	0.302 16	0.640 19	0.140 18

3D-BEVIS	0.248 18	0.667 13	0.566 14	0.076 21	0.035 22	0.394 19	0.027 14	0.035 21	0.098 16	0.099 20	0.030 21	0.025 20	0.098 19	0.375 17	0.126 17	0.604 14	0.181 20	0.854 17	0.171 17
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.
Seg-Cluster	0.215 20	0.370 20	0.337 21	0.285 14	0.105 19	0.325 20	0.025 15	0.282 15	0.085 18	0.105 19	0.107 18	0.007 22	0.079 20	0.317 18	0.114 19	0.309 21	0.304 15	0.587 20	0.123 20

Sgpn_scannet	0.143 21	0.208 22	0.390 20	0.169 19	0.065 20	0.275 21	0.029 13	0.069 19	0.000 21	0.087 21	0.043 20	0.014 21	0.027 22	0.000 21	0.112 20	0.351 20	0.168 21	0.438 21	0.138 19

MaskRCNN 2d->3d Proj	0.058 22	0.333 21	0.002 22	0.000 22	0.053 21	0.002 22	0.002 20	0.021 22	0.000 21	0.045 22	0.024 22	0.238 9	0.065 21	0.000 21	0.014 22	0.107 22	0.020 22	0.110 22	0.006 22

3D Semantic instance benchmark