3D Semantic Instance Results - ScanNet Benchmark

The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.

This table lists the benchmark results for the 3D semantic instance scenario.

Method	avg ap 50%	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	window

HAIS	0.699 1	1.000 1	0.849 2	0.820 3	0.675 1	0.808 7	0.279 3	0.757 6	0.465 6	0.517 3	0.596 1	0.559 3	0.600 8	1.000 1	0.654 5	0.767 4	0.676 2	0.994 14	0.560 3
Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, Xinggang Wang: Hierarchical Aggregation for 3D Instance Segmentation. ICCV 2021
SSTNet	0.698 2	1.000 1	0.697 20	0.888 2	0.556 5	0.803 8	0.387 2	0.626 10	0.417 10	0.556 2	0.585 3	0.702 1	0.600 8	1.000 1	0.824 1	0.720 12	0.692 1	1.000 1	0.509 6

OccuSeg+instance	0.672 3	1.000 1	0.758 15	0.682 11	0.576 3	0.842 1	0.477 1	0.504 20	0.524 3	0.567 1	0.585 4	0.451 7	0.557 14	1.000 1	0.751 2	0.797 3	0.563 13	1.000 1	0.467 11

Mask-Group	0.664 4	1.000 1	0.822 5	0.764 8	0.616 2	0.815 4	0.139 8	0.694 9	0.597 2	0.459 7	0.566 5	0.599 2	0.600 8	0.516 24	0.715 3	0.819 2	0.635 6	1.000 1	0.603 1

CSC-Pretrained	0.648 5	1.000 1	0.810 6	0.768 6	0.523 10	0.813 5	0.143 7	0.819 1	0.389 11	0.422 14	0.511 11	0.443 8	0.650 2	1.000 1	0.624 7	0.732 10	0.634 7	1.000 1	0.375 17

PE	0.645 6	1.000 1	0.773 11	0.798 5	0.538 7	0.786 13	0.088 15	0.799 4	0.350 15	0.435 13	0.547 7	0.545 4	0.646 6	0.933 11	0.562 10	0.761 7	0.556 17	0.997 11	0.501 8
Biao Zhang, Peter Wonka: Point Cloud Instance Segmentation using Probabilistic Embeddings. CVPR 2021
RPGN	0.643 7	1.000 1	0.758 14	0.582 17	0.539 6	0.826 3	0.046 18	0.765 5	0.372 13	0.436 12	0.588 2	0.539 5	0.650 2	1.000 1	0.577 8	0.750 9	0.653 5	0.997 11	0.495 9

Dyco3D	0.641 8	1.000 1	0.841 3	0.893 1	0.531 8	0.802 9	0.115 12	0.588 15	0.448 7	0.438 10	0.537 9	0.430 10	0.550 15	0.857 13	0.534 11	0.764 6	0.657 3	0.987 15	0.568 2

GICN	0.638 9	1.000 1	0.895 1	0.800 4	0.480 13	0.676 16	0.144 6	0.737 7	0.354 14	0.447 8	0.400 19	0.365 14	0.700 1	1.000 1	0.569 9	0.836 1	0.599 10	1.000 1	0.473 10

PointGroup	0.636 10	1.000 1	0.765 12	0.624 12	0.505 12	0.797 10	0.116 11	0.696 8	0.384 12	0.441 9	0.559 6	0.476 6	0.596 11	1.000 1	0.666 4	0.756 8	0.556 16	0.997 11	0.513 5
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, Jiaya Jia: PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. CVPR 2020 [oral]
DD-UNet+Group	0.635 11	0.667 21	0.797 9	0.714 10	0.562 4	0.774 14	0.146 5	0.810 3	0.429 9	0.476 5	0.546 8	0.399 12	0.633 7	1.000 1	0.632 6	0.722 11	0.609 9	1.000 1	0.514 4
H. Liu, R. Liu, K. Yang, J. Zhang, K. Peng, R. Stiefelhagen: HIDA: Towards Holistic Indoor Understanding for the Visually Impaired via Semantic Instance Segmentation with a Wearable Solid-State LiDAR Sensor.
3D-MPA	0.611 12	1.000 1	0.833 4	0.765 7	0.526 9	0.756 15	0.136 10	0.588 15	0.470 5	0.438 11	0.432 17	0.358 15	0.650 2	0.857 13	0.429 15	0.765 5	0.557 15	1.000 1	0.430 14
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nießner: 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation. CVPR 2020
SphereNet	0.606 13	1.000 1	0.776 10	0.745 9	0.436 17	0.834 2	0.035 20	0.587 17	0.518 4	0.338 21	0.534 10	0.352 16	0.594 13	1.000 1	0.391 19	0.696 15	0.624 8	1.000 1	0.451 12

PCJC	0.578 14	1.000 1	0.810 7	0.583 16	0.449 16	0.813 6	0.042 19	0.603 13	0.341 16	0.490 4	0.465 13	0.410 11	0.650 2	0.835 18	0.264 23	0.694 16	0.561 14	0.889 21	0.504 7

SSEN	0.575 15	1.000 1	0.761 13	0.473 19	0.477 14	0.795 11	0.066 16	0.529 18	0.658 1	0.460 6	0.461 14	0.380 13	0.331 23	0.859 12	0.401 18	0.692 17	0.653 4	1.000 1	0.348 19
Dongsu Zhang, Junha Chun, Sang Kyun Cha, Young Min Kim: Spatial Semantic Embedding Network: Fast 3D Instance Segmentation with Deep Metric Learning. Arxiv
MTML	0.549 16	1.000 1	0.807 8	0.588 15	0.327 20	0.647 17	0.004 25	0.815 2	0.180 20	0.418 15	0.364 21	0.182 19	0.445 18	1.000 1	0.442 14	0.688 18	0.571 12	1.000 1	0.396 15
Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald: 3D Instance Segmentation via Multi-task Metric Learning. ICCV 2019 [oral]
Sparse R-CNN	0.515 17	1.000 1	0.538 26	0.282 22	0.468 15	0.790 12	0.173 4	0.345 22	0.429 8	0.413 17	0.484 12	0.176 20	0.595 12	0.591 22	0.522 12	0.668 19	0.476 20	0.986 16	0.327 20

Occipital-SCS	0.512 18	1.000 1	0.716 17	0.509 18	0.506 11	0.611 19	0.092 14	0.602 14	0.177 21	0.346 19	0.383 20	0.165 21	0.442 19	0.850 17	0.386 20	0.618 21	0.543 18	0.889 21	0.389 16

3D-BoNet	0.488 19	1.000 1	0.672 21	0.590 14	0.301 21	0.484 27	0.098 13	0.620 11	0.306 17	0.341 20	0.259 23	0.125 23	0.434 20	0.796 19	0.402 17	0.499 26	0.513 19	0.909 20	0.439 13
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight
PanopticFusion-inst	0.478 20	0.667 21	0.712 19	0.595 13	0.259 23	0.550 24	0.000 28	0.613 12	0.175 22	0.250 25	0.434 15	0.437 9	0.411 22	0.857 13	0.485 13	0.591 24	0.267 28	0.944 18	0.359 18
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
SALoss-ResNet	0.459 21	1.000 1	0.737 16	0.159 29	0.259 22	0.587 21	0.138 9	0.475 21	0.217 19	0.416 16	0.408 18	0.128 22	0.315 24	0.714 20	0.411 16	0.536 25	0.590 11	0.873 24	0.304 21
Zhidong Liang, Ming Yang, Hao Li, Chunxiang Wang: 3D Instance Embedding Learning With a Structure-Aware Loss Function for Point Cloud Segmentation. IEEE Robotics and Automation Letters (IROS2020)
MASC	0.447 22	0.528 27	0.555 24	0.381 20	0.382 18	0.633 18	0.002 26	0.509 19	0.260 18	0.361 18	0.432 16	0.327 17	0.451 17	0.571 23	0.367 21	0.639 20	0.386 21	0.980 17	0.276 22
Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.
3D-SIS	0.382 23	1.000 1	0.432 28	0.245 24	0.190 24	0.577 22	0.013 23	0.263 24	0.033 28	0.320 22	0.240 24	0.075 25	0.422 21	0.857 13	0.117 26	0.699 13	0.271 27	0.883 23	0.235 24
Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019
Hier3D	0.323 24	0.667 21	0.542 25	0.264 23	0.157 27	0.550 23	0.000 28	0.205 27	0.009 29	0.270 24	0.218 25	0.075 25	0.500 16	0.688 21	0.007 31	0.698 14	0.301 25	0.459 29	0.200 25
Tan: HCFS3D: Hierarchical Coupled Feature Selection Network for 3D Semantic and Instance Segmentation.
UNet-backbone	0.319 25	0.667 21	0.715 18	0.233 25	0.189 25	0.479 28	0.008 24	0.218 25	0.067 27	0.201 26	0.173 26	0.107 24	0.123 28	0.438 25	0.150 24	0.615 22	0.355 22	0.916 19	0.093 30

R-PointNet	0.306 26	0.500 28	0.405 29	0.311 21	0.348 19	0.589 20	0.054 17	0.068 29	0.126 23	0.283 23	0.290 22	0.028 29	0.219 26	0.214 27	0.331 22	0.396 29	0.275 26	0.821 26	0.245 23

RandSA	0.250 27	0.333 29	0.613 22	0.229 26	0.163 26	0.493 25	0.000 28	0.304 23	0.107 24	0.147 28	0.100 27	0.052 28	0.231 25	0.119 28	0.039 28	0.445 28	0.325 23	0.654 27	0.141 27

Region	0.248 28	0.667 21	0.437 27	0.188 27	0.153 28	0.491 26	0.000 28	0.208 26	0.094 26	0.153 27	0.099 28	0.057 27	0.217 27	0.119 28	0.039 28	0.466 27	0.302 24	0.640 28	0.140 28

3D-BEVIS	0.248 28	0.667 21	0.566 23	0.076 30	0.035 31	0.394 29	0.027 22	0.035 30	0.098 25	0.099 29	0.030 30	0.025 30	0.098 29	0.375 26	0.126 25	0.604 23	0.181 29	0.854 25	0.171 26
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.
Sgpn_scannet	0.143 30	0.208 31	0.390 30	0.169 28	0.065 29	0.275 30	0.029 21	0.069 28	0.000 30	0.087 30	0.043 29	0.014 31	0.027 31	0.000 30	0.112 27	0.351 30	0.168 30	0.438 30	0.138 29

MaskRCNN 2d->3d Proj	0.058 31	0.333 29	0.002 31	0.000 31	0.053 30	0.002 31	0.002 27	0.021 31	0.000 30	0.045 31	0.024 31	0.238 18	0.065 30	0.000 30	0.014 30	0.107 31	0.020 31	0.110 31	0.006 31

3D Semantic instance benchmark