3D Semantic Instance Results - ScanNet Benchmark

The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.

This table lists the benchmark results for the 3D semantic instance scenario.

Method	avg ap 50%	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	window

HAIS	0.699 1	1.000 1	0.849 2	0.820 3	0.675 1	0.808 7	0.279 3	0.757 7	0.465 6	0.517 3	0.596 1	0.559 3	0.600 9	1.000 1	0.654 6	0.767 4	0.676 3	0.994 15	0.560 4
Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, Xinggang Wang: Hierarchical Aggregation for 3D Instance Segmentation. ICCV 2021
SSTNet	0.698 2	1.000 1	0.697 21	0.888 2	0.556 6	0.803 8	0.387 2	0.626 11	0.417 10	0.556 2	0.585 3	0.702 1	0.600 9	1.000 1	0.824 1	0.720 12	0.692 1	1.000 1	0.509 7
Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, Kui Jia: Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networks. ICCV2021
OccuSeg+instance	0.672 3	1.000 1	0.758 16	0.682 11	0.576 4	0.842 1	0.477 1	0.504 21	0.524 3	0.567 1	0.585 4	0.451 7	0.557 15	1.000 1	0.751 2	0.797 3	0.563 14	1.000 1	0.467 12

Mask-Group	0.664 4	1.000 1	0.822 5	0.764 8	0.616 2	0.815 4	0.139 9	0.694 10	0.597 2	0.459 7	0.566 5	0.599 2	0.600 9	0.516 25	0.715 3	0.819 2	0.635 7	1.000 1	0.603 1

CSC-Pretrained	0.648 5	1.000 1	0.810 6	0.768 6	0.523 11	0.813 5	0.143 8	0.819 2	0.389 11	0.422 14	0.511 11	0.443 8	0.650 2	1.000 1	0.624 8	0.732 10	0.634 8	1.000 1	0.375 18

PE	0.645 6	1.000 1	0.773 12	0.798 5	0.538 8	0.786 13	0.088 16	0.799 5	0.350 15	0.435 13	0.547 7	0.545 4	0.646 7	0.933 12	0.562 11	0.761 7	0.556 18	0.997 12	0.501 9
Biao Zhang, Peter Wonka: Point Cloud Instance Segmentation using Probabilistic Embeddings. CVPR 2021
RPGN	0.643 7	1.000 1	0.758 15	0.582 18	0.539 7	0.826 3	0.046 19	0.765 6	0.372 13	0.436 12	0.588 2	0.539 5	0.650 2	1.000 1	0.577 9	0.750 9	0.653 6	0.997 12	0.495 10

Dyco3D	0.641 8	1.000 1	0.841 3	0.893 1	0.531 9	0.802 9	0.115 13	0.588 16	0.448 7	0.438 10	0.537 9	0.430 10	0.550 16	0.857 14	0.534 12	0.764 6	0.657 4	0.987 16	0.568 2
Tong He; Chunhua Shen; Anton van den Hengel: DyCo3D: Robust Instance Segmentation of 3D Point Clouds through Dynamic Convolution. CVPR2021
GICN	0.638 9	1.000 1	0.895 1	0.800 4	0.480 14	0.676 16	0.144 7	0.737 8	0.354 14	0.447 8	0.400 19	0.365 15	0.700 1	1.000 1	0.569 10	0.836 1	0.599 11	1.000 1	0.473 11

PointGroup	0.636 10	1.000 1	0.765 13	0.624 12	0.505 13	0.797 10	0.116 12	0.696 9	0.384 12	0.441 9	0.559 6	0.476 6	0.596 12	1.000 1	0.666 4	0.756 8	0.556 17	0.997 12	0.513 6
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, Jiaya Jia: PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. CVPR 2020 [oral]
DD-UNet+Group	0.635 11	0.667 22	0.797 10	0.714 10	0.562 5	0.774 14	0.146 6	0.810 4	0.429 9	0.476 5	0.546 8	0.399 12	0.633 8	1.000 1	0.632 7	0.722 11	0.609 10	1.000 1	0.514 5
H. Liu, R. Liu, K. Yang, J. Zhang, K. Peng, R. Stiefelhagen: HIDA: Towards Holistic Indoor Understanding for the Visually Impaired via Semantic Instance Segmentation with a Wearable Solid-State LiDAR Sensor. ICCVW 2021
DENet	0.629 12	1.000 1	0.797 9	0.608 13	0.589 3	0.627 19	0.219 4	0.882 1	0.310 17	0.402 18	0.383 21	0.396 13	0.650 2	1.000 1	0.663 5	0.543 25	0.691 2	1.000 1	0.568 3

3D-MPA	0.611 13	1.000 1	0.833 4	0.765 7	0.526 10	0.756 15	0.136 11	0.588 16	0.470 5	0.438 11	0.432 17	0.358 16	0.650 2	0.857 14	0.429 16	0.765 5	0.557 16	1.000 1	0.430 15
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nießner: 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation. CVPR 2020
SphereNet	0.606 14	1.000 1	0.776 11	0.745 9	0.436 18	0.834 2	0.035 21	0.587 18	0.518 4	0.338 22	0.534 10	0.352 17	0.594 14	1.000 1	0.391 20	0.696 15	0.624 9	1.000 1	0.451 13

PCJC	0.578 15	1.000 1	0.810 7	0.583 17	0.449 17	0.813 6	0.042 20	0.603 14	0.341 16	0.490 4	0.465 13	0.410 11	0.650 2	0.835 19	0.264 24	0.694 16	0.561 15	0.889 22	0.504 8

SSEN	0.575 16	1.000 1	0.761 14	0.473 20	0.477 15	0.795 11	0.066 17	0.529 19	0.658 1	0.460 6	0.461 14	0.380 14	0.331 24	0.859 13	0.401 19	0.692 17	0.653 5	1.000 1	0.348 20
Dongsu Zhang, Junha Chun, Sang Kyun Cha, Young Min Kim: Spatial Semantic Embedding Network: Fast 3D Instance Segmentation with Deep Metric Learning. Arxiv
MTML	0.549 17	1.000 1	0.807 8	0.588 16	0.327 21	0.647 17	0.004 26	0.815 3	0.180 21	0.418 15	0.364 22	0.182 20	0.445 19	1.000 1	0.442 15	0.688 18	0.571 13	1.000 1	0.396 16
Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald: 3D Instance Segmentation via Multi-task Metric Learning. ICCV 2019 [oral]
Sparse R-CNN	0.515 18	1.000 1	0.538 27	0.282 23	0.468 16	0.790 12	0.173 5	0.345 23	0.429 8	0.413 17	0.484 12	0.176 21	0.595 13	0.591 23	0.522 13	0.668 19	0.476 21	0.986 17	0.327 21

Occipital-SCS	0.512 19	1.000 1	0.716 18	0.509 19	0.506 12	0.611 20	0.092 15	0.602 15	0.177 22	0.346 20	0.383 20	0.165 22	0.442 20	0.850 18	0.386 21	0.618 21	0.543 19	0.889 22	0.389 17

3D-BoNet	0.488 20	1.000 1	0.672 22	0.590 15	0.301 22	0.484 28	0.098 14	0.620 12	0.306 18	0.341 21	0.259 24	0.125 24	0.434 21	0.796 20	0.402 18	0.499 27	0.513 20	0.909 21	0.439 14
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight
PanopticFusion-inst	0.478 21	0.667 22	0.712 20	0.595 14	0.259 24	0.550 25	0.000 29	0.613 13	0.175 23	0.250 26	0.434 15	0.437 9	0.411 23	0.857 14	0.485 14	0.591 24	0.267 29	0.944 19	0.359 19
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
SALoss-ResNet	0.459 22	1.000 1	0.737 17	0.159 30	0.259 23	0.587 22	0.138 10	0.475 22	0.217 20	0.416 16	0.408 18	0.128 23	0.315 25	0.714 21	0.411 17	0.536 26	0.590 12	0.873 25	0.304 22
Zhidong Liang, Ming Yang, Hao Li, Chunxiang Wang: 3D Instance Embedding Learning With a Structure-Aware Loss Function for Point Cloud Segmentation. IEEE Robotics and Automation Letters (IROS2020)
MASC	0.447 23	0.528 28	0.555 25	0.381 21	0.382 19	0.633 18	0.002 27	0.509 20	0.260 19	0.361 19	0.432 16	0.327 18	0.451 18	0.571 24	0.367 22	0.639 20	0.386 22	0.980 18	0.276 23
Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.
3D-SIS	0.382 24	1.000 1	0.432 29	0.245 25	0.190 25	0.577 23	0.013 24	0.263 25	0.033 29	0.320 23	0.240 25	0.075 26	0.422 22	0.857 14	0.117 27	0.699 13	0.271 28	0.883 24	0.235 25
Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019
Hier3D	0.323 25	0.667 22	0.542 26	0.264 24	0.157 28	0.550 24	0.000 29	0.205 28	0.009 30	0.270 25	0.218 26	0.075 26	0.500 17	0.688 22	0.007 32	0.698 14	0.301 26	0.459 30	0.200 26
Tan: HCFS3D: Hierarchical Coupled Feature Selection Network for 3D Semantic and Instance Segmentation.
UNet-backbone	0.319 26	0.667 22	0.715 19	0.233 26	0.189 26	0.479 29	0.008 25	0.218 26	0.067 28	0.201 27	0.173 27	0.107 25	0.123 29	0.438 26	0.150 25	0.615 22	0.355 23	0.916 20	0.093 31

R-PointNet	0.306 27	0.500 29	0.405 30	0.311 22	0.348 20	0.589 21	0.054 18	0.068 30	0.126 24	0.283 24	0.290 23	0.028 30	0.219 27	0.214 28	0.331 23	0.396 30	0.275 27	0.821 27	0.245 24

RandSA	0.250 28	0.333 30	0.613 23	0.229 27	0.163 27	0.493 26	0.000 29	0.304 24	0.107 25	0.147 29	0.100 28	0.052 29	0.231 26	0.119 29	0.039 29	0.445 29	0.325 24	0.654 28	0.141 28

Region	0.248 29	0.667 22	0.437 28	0.188 28	0.153 29	0.491 27	0.000 29	0.208 27	0.094 27	0.153 28	0.099 29	0.057 28	0.217 28	0.119 29	0.039 29	0.466 28	0.302 25	0.640 29	0.140 29

3D-BEVIS	0.248 29	0.667 22	0.566 24	0.076 31	0.035 32	0.394 30	0.027 23	0.035 31	0.098 26	0.099 30	0.030 31	0.025 31	0.098 30	0.375 27	0.126 26	0.604 23	0.181 30	0.854 26	0.171 27
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.
Sgpn_scannet	0.143 31	0.208 32	0.390 31	0.169 29	0.065 30	0.275 31	0.029 22	0.069 29	0.000 31	0.087 31	0.043 30	0.014 32	0.027 32	0.000 31	0.112 28	0.351 31	0.168 31	0.438 31	0.138 30

MaskRCNN 2d->3d Proj	0.058 32	0.333 30	0.002 32	0.000 32	0.053 31	0.002 32	0.002 28	0.021 32	0.000 31	0.045 32	0.024 32	0.238 19	0.065 31	0.000 31	0.014 31	0.107 32	0.020 32	0.110 32	0.006 32

3D Semantic instance benchmark