3D Semantic Instance Results - ScanNet Benchmark

The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.

This table lists the benchmark results for the 3D semantic instance scenario.

Method	avg ap 50%	bathtub	bed	bookshelf	cabinet	chair	counter	curtain	desk	door	otherfurniture	picture	refrigerator	shower curtain	sink	sofa	table	toilet	window

Queryformer	0.787 1	1.000 1	0.933 1	0.601 34	0.754 1	0.886 4	0.558 2	0.661 25	0.767 3	0.665 4	0.716 3	0.639 11	0.808 3	1.000 1	0.844 1	0.897 2	0.804 2	1.000 1	0.624 2

Mask3D	0.780 2	1.000 1	0.786 27	0.716 25	0.696 5	0.885 5	0.500 4	0.714 18	0.810 2	0.672 3	0.715 4	0.679 7	0.809 2	1.000 1	0.831 2	0.833 8	0.787 4	1.000 1	0.602 6
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, Bastian Leibe: Mask3D for 3D Semantic Instance Segmentation. ICRA 2023
SPFormer	0.770 3	0.903 39	0.903 2	0.806 13	0.609 17	0.886 3	0.568 1	0.815 6	0.705 7	0.711 1	0.655 6	0.652 10	0.685 11	1.000 1	0.789 4	0.809 14	0.776 7	1.000 1	0.583 11
Sun Jiahao, Qing Chunmei, Tan Junpeng, Xu Xiangmin: Superpoint Transformer for 3D Scene Instance Segmentation. AAAI 2023 [Oral]
SoftGroup++	0.769 4	1.000 1	0.803 20	0.937 1	0.684 6	0.865 7	0.213 20	0.870 2	0.664 9	0.571 10	0.758 1	0.702 4	0.807 4	1.000 1	0.653 16	0.902 1	0.792 3	1.000 1	0.626 1

ISBNet	0.763 5	1.000 1	0.873 5	0.717 24	0.666 9	0.858 11	0.508 3	0.667 23	0.764 4	0.643 5	0.676 5	0.688 6	0.825 1	1.000 1	0.773 5	0.741 27	0.777 6	1.000 1	0.556 17
Tuan Duc Ngo, Binh-Son Hua, Khoi Nguyen: ISBNet: a 3D Point Cloud Instance Segmentation Network with Instance-aware Sampling and Box-aware Dynamic Convolution. CVPR 2023
SoftGroup	0.761 6	1.000 1	0.808 17	0.845 8	0.716 2	0.862 9	0.243 17	0.824 4	0.655 11	0.620 6	0.734 2	0.699 5	0.791 6	0.981 25	0.716 8	0.844 5	0.769 8	1.000 1	0.594 9
Thang Vu, Kookhoi Kim, Tung M. Luu, Xuan Thanh Nguyen, Chang D. Yoo: SoftGroup for 3D Instance Segmentaiton on Point Clouds. CVPR 2022 [Oral]
TD3D	0.751 7	1.000 1	0.774 28	0.867 7	0.621 13	0.934 1	0.404 7	0.706 19	0.812 1	0.605 8	0.633 11	0.626 12	0.690 10	1.000 1	0.640 18	0.820 11	0.777 5	1.000 1	0.612 4

PBNet	0.747 8	1.000 1	0.818 13	0.837 10	0.713 3	0.844 12	0.457 6	0.647 28	0.711 6	0.614 7	0.617 13	0.657 9	0.650 13	1.000 1	0.692 10	0.822 10	0.765 10	1.000 1	0.595 8

GraphCut	0.732 9	1.000 1	0.788 25	0.724 23	0.642 11	0.859 10	0.248 16	0.787 11	0.618 14	0.596 9	0.653 8	0.722 2	0.583 31	1.000 1	0.766 6	0.861 3	0.825 1	1.000 1	0.504 23

IPCA-Inst	0.731 10	1.000 1	0.788 26	0.884 6	0.698 4	0.788 27	0.252 15	0.760 13	0.646 12	0.511 18	0.637 10	0.665 8	0.804 5	1.000 1	0.644 17	0.778 17	0.747 12	1.000 1	0.561 15

TopoSeg	0.725 11	1.000 1	0.806 19	0.933 2	0.668 8	0.758 30	0.272 14	0.734 17	0.630 13	0.549 14	0.654 7	0.606 13	0.697 9	0.966 27	0.612 22	0.839 6	0.754 11	1.000 1	0.573 12

DKNet	0.718 12	1.000 1	0.814 14	0.782 16	0.619 14	0.872 6	0.224 18	0.751 15	0.569 18	0.677 2	0.585 16	0.724 1	0.633 23	0.981 25	0.515 32	0.819 12	0.736 13	1.000 1	0.617 3
Yizheng Wu, Min Shi, Shuaiyuan Du, Hao Lu, Zhiguo Cao, Weicai Zhong: 3D Instances as 1D Kernels. ECCV 2022
SSEC	0.707 13	1.000 1	0.850 7	0.924 3	0.648 10	0.747 33	0.162 22	0.862 3	0.572 17	0.520 16	0.624 12	0.549 16	0.649 21	1.000 1	0.560 27	0.706 33	0.768 9	1.000 1	0.591 10

HAIS	0.699 14	1.000 1	0.849 8	0.820 11	0.675 7	0.808 21	0.279 12	0.757 14	0.465 23	0.517 17	0.596 14	0.559 15	0.600 25	1.000 1	0.654 15	0.767 19	0.676 17	0.994 35	0.560 16
Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, Xinggang Wang: Hierarchical Aggregation for 3D Instance Segmentation. ICCV 2021
SSTNet	0.698 15	1.000 1	0.697 44	0.888 5	0.556 23	0.803 22	0.387 8	0.626 30	0.417 27	0.556 13	0.585 17	0.702 3	0.600 25	1.000 1	0.824 3	0.720 32	0.692 15	1.000 1	0.509 22
Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, Kui Jia: Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networks. ICCV2021
DualGroup	0.694 16	1.000 1	0.799 22	0.811 12	0.622 12	0.817 16	0.376 9	0.805 9	0.590 16	0.487 21	0.568 20	0.525 20	0.650 13	0.835 38	0.600 23	0.829 9	0.655 19	1.000 1	0.526 19

SphereSeg	0.680 17	1.000 1	0.856 6	0.744 22	0.618 15	0.893 2	0.151 23	0.651 27	0.713 5	0.537 15	0.579 19	0.430 29	0.651 12	1.000 1	0.389 41	0.744 26	0.697 14	0.991 37	0.601 7

Box2Mask	0.677 18	1.000 1	0.847 9	0.771 18	0.509 31	0.816 17	0.277 13	0.558 37	0.482 20	0.562 12	0.640 9	0.448 25	0.700 7	1.000 1	0.666 11	0.852 4	0.578 31	0.997 30	0.488 27
Julian Chibane, Francis Engelmann, Tuan Anh Tran, Gerard Pons-Moll: Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation Using Bounding Boxes. ECCV 2022
OccuSeg+instance	0.672 19	1.000 1	0.758 36	0.682 28	0.576 21	0.842 13	0.477 5	0.504 41	0.524 19	0.567 11	0.585 18	0.451 24	0.557 32	1.000 1	0.751 7	0.797 15	0.563 34	1.000 1	0.467 31
Lei Han, Tian Zheng, Lan Xu, Lu Fang: OccuSeg: Occupancy-aware 3D Instance Segmentation. CVPR2020
Mask-Group	0.664 20	1.000 1	0.822 12	0.764 21	0.616 16	0.815 18	0.139 27	0.694 21	0.597 15	0.459 25	0.566 21	0.599 14	0.600 25	0.516 48	0.715 9	0.819 13	0.635 23	1.000 1	0.603 5
Min Zhong, Xinghao Chen, Xiaokang Chen, Gang Zeng, Yunhe Wang: MaskGroup: Hierarchical Point Grouping and Masking for 3D Instance Segmentation. ICME 2022
INS-Conv-instance	0.657 21	1.000 1	0.760 34	0.667 30	0.581 19	0.863 8	0.323 10	0.655 26	0.477 21	0.473 23	0.549 23	0.432 28	0.650 13	1.000 1	0.655 14	0.738 28	0.585 30	0.944 41	0.472 30

CSC-Pretrained	0.648 22	1.000 1	0.810 15	0.768 19	0.523 29	0.813 19	0.143 26	0.819 5	0.389 30	0.422 33	0.511 27	0.443 26	0.650 13	1.000 1	0.624 20	0.732 29	0.634 24	1.000 1	0.375 38

PE	0.645 23	1.000 1	0.773 30	0.798 15	0.538 25	0.786 28	0.088 34	0.799 10	0.350 34	0.435 32	0.547 24	0.545 17	0.646 22	0.933 28	0.562 26	0.761 22	0.556 39	0.997 30	0.501 25
Biao Zhang, Peter Wonka: Point Cloud Instance Segmentation using Probabilistic Embeddings. CVPR 2021
RPGN	0.643 24	1.000 1	0.758 35	0.582 40	0.539 24	0.826 15	0.046 38	0.765 12	0.372 32	0.436 31	0.588 15	0.539 19	0.650 13	1.000 1	0.577 24	0.750 24	0.653 21	0.997 30	0.495 26
Shichao Dong, Guosheng Lin, Tzu-Yi Hung: Learning Regional Purity for Instance Segmentation on 3D Point Clouds. ECCV 2022
Dyco3D	0.641 25	1.000 1	0.841 10	0.893 4	0.531 27	0.802 23	0.115 31	0.588 35	0.448 24	0.438 29	0.537 26	0.430 30	0.550 33	0.857 30	0.534 30	0.764 21	0.657 18	0.987 38	0.568 13
Tong He; Chunhua Shen; Anton van den Hengel: DyCo3D: Robust Instance Segmentation of 3D Point Clouds through Dynamic Convolution. CVPR2021
GICN	0.638 26	1.000 1	0.895 4	0.800 14	0.480 35	0.676 37	0.144 25	0.737 16	0.354 33	0.447 26	0.400 39	0.365 35	0.700 7	1.000 1	0.569 25	0.836 7	0.599 26	1.000 1	0.473 29

PointGroup	0.636 27	1.000 1	0.765 31	0.624 32	0.505 33	0.797 24	0.116 30	0.696 20	0.384 31	0.441 27	0.559 22	0.476 22	0.596 28	1.000 1	0.666 11	0.756 23	0.556 38	0.997 30	0.513 21
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, Jiaya Jia: PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. CVPR 2020 [oral]
DD-UNet+Group	0.635 28	0.667 40	0.797 24	0.714 26	0.562 22	0.774 29	0.146 24	0.810 8	0.429 26	0.476 22	0.546 25	0.399 32	0.633 23	1.000 1	0.632 19	0.722 31	0.609 25	1.000 1	0.514 20
H. Liu, R. Liu, K. Yang, J. Zhang, K. Peng, R. Stiefelhagen: HIDA: Towards Holistic Indoor Understanding for the Visually Impaired via Semantic Instance Segmentation with a Wearable Solid-State LiDAR Sensor. ICCVW 2021
DENet	0.629 29	1.000 1	0.797 23	0.608 33	0.589 18	0.627 41	0.219 19	0.882 1	0.310 36	0.402 38	0.383 41	0.396 33	0.650 13	1.000 1	0.663 13	0.543 49	0.691 16	1.000 1	0.568 14

3D-MPA	0.611 30	1.000 1	0.833 11	0.765 20	0.526 28	0.756 31	0.136 29	0.588 35	0.470 22	0.438 30	0.432 36	0.358 36	0.650 13	0.857 30	0.429 37	0.765 20	0.557 37	1.000 1	0.430 33
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nießner: 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation. CVPR 2020
OSIS	0.605 31	1.000 1	0.801 21	0.599 35	0.535 26	0.728 35	0.286 11	0.436 45	0.679 8	0.491 19	0.433 34	0.256 38	0.404 45	0.857 30	0.620 21	0.724 30	0.510 43	1.000 1	0.539 18

AOIA	0.601 32	1.000 1	0.761 33	0.687 27	0.485 34	0.828 14	0.008 44	0.663 24	0.405 29	0.405 37	0.425 37	0.490 21	0.596 28	0.714 41	0.553 29	0.779 16	0.597 27	0.992 36	0.424 35

PCJC	0.578 33	1.000 1	0.810 16	0.583 39	0.449 38	0.813 20	0.042 39	0.603 33	0.341 35	0.490 20	0.465 31	0.410 31	0.650 13	0.835 38	0.264 47	0.694 37	0.561 35	0.889 45	0.504 24

SSEN	0.575 34	1.000 1	0.761 32	0.473 42	0.477 36	0.795 25	0.066 35	0.529 38	0.658 10	0.460 24	0.461 32	0.380 34	0.331 47	0.859 29	0.401 40	0.692 39	0.653 20	1.000 1	0.348 40
Dongsu Zhang, Junha Chun, Sang Kyun Cha, Young Min Kim: Spatial Semantic Embedding Network: Fast 3D Instance Segmentation with Deep Metric Learning. Arxiv
RWSeg	0.567 35	0.528 50	0.708 43	0.626 31	0.580 20	0.745 34	0.063 36	0.627 29	0.240 40	0.400 39	0.497 28	0.464 23	0.515 34	1.000 1	0.475 34	0.745 25	0.571 32	1.000 1	0.429 34

NeuralBF	0.555 36	0.667 40	0.896 3	0.843 9	0.517 30	0.751 32	0.029 40	0.519 39	0.414 28	0.439 28	0.465 30	0.000 56	0.484 36	0.857 30	0.287 45	0.693 38	0.651 22	1.000 1	0.485 28
Weiwei Sun, Daniel Rebain, Renjie Liao, Vladimir Tankovich, Soroosh Yazdani, Kwang Moo Yi, Andrea Tagliasacchi: NeuralBF: Neural Bilateral Filtering for Top-down Instance Segmentation on Point Clouds. WACV 2023
MTML	0.549 37	1.000 1	0.807 18	0.588 38	0.327 43	0.647 39	0.004 46	0.815 7	0.180 42	0.418 34	0.364 43	0.182 41	0.445 39	1.000 1	0.442 36	0.688 40	0.571 33	1.000 1	0.396 36
Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald: 3D Instance Segmentation via Multi-task Metric Learning. ICCV 2019 [oral]
One_Thing_One_Click	0.529 38	0.667 40	0.718 39	0.777 17	0.399 39	0.683 36	0.000 49	0.669 22	0.138 45	0.391 40	0.374 42	0.539 18	0.360 46	0.641 45	0.556 28	0.774 18	0.593 28	0.997 30	0.251 45
Zhengzhe Liu, Xiaojuan Qi, Chi-Wing Fu: One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation. CVPR 2021
Sparse R-CNN	0.515 39	1.000 1	0.538 51	0.282 45	0.468 37	0.790 26	0.173 21	0.345 47	0.429 25	0.413 36	0.484 29	0.176 42	0.595 30	0.591 46	0.522 31	0.668 41	0.476 44	0.986 39	0.327 41

Occipital-SCS	0.512 40	1.000 1	0.716 40	0.509 41	0.506 32	0.611 42	0.092 33	0.602 34	0.177 43	0.346 43	0.383 40	0.165 43	0.442 40	0.850 37	0.386 42	0.618 45	0.543 40	0.889 45	0.389 37

3D-BoNet	0.488 41	1.000 1	0.672 46	0.590 37	0.301 45	0.484 52	0.098 32	0.620 31	0.306 37	0.341 44	0.259 47	0.125 45	0.434 42	0.796 40	0.402 39	0.499 51	0.513 42	0.909 44	0.439 32
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight
PanopticFusion-inst	0.478 42	0.667 40	0.712 42	0.595 36	0.259 48	0.550 48	0.000 49	0.613 32	0.175 44	0.250 49	0.434 33	0.437 27	0.411 44	0.857 30	0.485 33	0.591 48	0.267 54	0.944 41	0.359 39
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear)
SPG_WSIS	0.470 43	0.667 40	0.685 45	0.677 29	0.372 41	0.562 46	0.000 49	0.482 42	0.244 39	0.316 46	0.298 44	0.052 51	0.442 41	0.857 30	0.267 46	0.702 34	0.559 36	1.000 1	0.287 43

SALoss-ResNet	0.459 44	1.000 1	0.737 38	0.159 55	0.259 47	0.587 44	0.138 28	0.475 43	0.217 41	0.416 35	0.408 38	0.128 44	0.315 48	0.714 41	0.411 38	0.536 50	0.590 29	0.873 48	0.304 42
Zhidong Liang, Ming Yang, Hao Li, Chunxiang Wang: 3D Instance Embedding Learning With a Structure-Aware Loss Function for Point Cloud Segmentation. IEEE Robotics and Automation Letters (IROS2020)
MASC	0.447 45	0.528 50	0.555 49	0.381 43	0.382 40	0.633 40	0.002 47	0.509 40	0.260 38	0.361 42	0.432 35	0.327 37	0.451 38	0.571 47	0.367 43	0.639 43	0.386 45	0.980 40	0.276 44
Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.
SegGroup_ins	0.445 46	0.667 40	0.773 29	0.185 52	0.317 44	0.656 38	0.000 49	0.407 46	0.134 46	0.381 41	0.267 46	0.217 40	0.476 37	0.714 41	0.452 35	0.629 44	0.514 41	1.000 1	0.222 48
An Tao, Yueqi Duan, Yi Wei, Jiwen Lu, Jie Zhou: SegGroup: Seg-Level Supervision for 3D Instance and Semantic Segmentation. TIP 2022
3D-SIS	0.382 47	1.000 1	0.432 53	0.245 47	0.190 49	0.577 45	0.013 43	0.263 49	0.033 52	0.320 45	0.240 48	0.075 47	0.422 43	0.857 30	0.117 51	0.699 35	0.271 53	0.883 47	0.235 47
Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019
Hier3D	0.323 48	0.667 40	0.542 50	0.264 46	0.157 52	0.550 47	0.000 49	0.205 52	0.009 53	0.270 48	0.218 49	0.075 47	0.500 35	0.688 44	0.007 57	0.698 36	0.301 50	0.459 54	0.200 49
Tan: HCFS3D: Hierarchical Coupled Feature Selection Network for 3D Semantic and Instance Segmentation.
UNet-backbone	0.319 49	0.667 40	0.715 41	0.233 48	0.189 50	0.479 53	0.008 44	0.218 50	0.067 51	0.201 51	0.173 50	0.107 46	0.123 53	0.438 49	0.150 49	0.615 46	0.355 46	0.916 43	0.093 56

R-PointNet	0.306 50	0.500 52	0.405 54	0.311 44	0.348 42	0.589 43	0.054 37	0.068 55	0.126 47	0.283 47	0.290 45	0.028 52	0.219 51	0.214 52	0.331 44	0.396 55	0.275 51	0.821 50	0.245 46

Region-18class	0.284 51	0.250 56	0.751 37	0.228 50	0.270 46	0.521 49	0.000 49	0.468 44	0.008 55	0.205 50	0.127 51	0.000 56	0.068 55	0.070 55	0.262 48	0.652 42	0.323 48	0.740 51	0.173 50

SemRegionNet-20cls	0.250 52	0.333 53	0.613 47	0.229 49	0.163 51	0.493 50	0.000 49	0.304 48	0.107 48	0.147 53	0.100 52	0.052 50	0.231 49	0.119 53	0.039 53	0.445 53	0.325 47	0.654 52	0.141 52

tmp	0.248 53	0.667 40	0.437 52	0.188 51	0.153 53	0.491 51	0.000 49	0.208 51	0.094 50	0.153 52	0.099 53	0.057 49	0.217 52	0.119 53	0.039 53	0.466 52	0.302 49	0.640 53	0.140 53

3D-BEVIS	0.248 53	0.667 40	0.566 48	0.076 56	0.035 57	0.394 55	0.027 42	0.035 56	0.098 49	0.099 55	0.030 56	0.025 53	0.098 54	0.375 51	0.126 50	0.604 47	0.181 55	0.854 49	0.171 51
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.
ASIS	0.199 55	0.333 53	0.253 56	0.167 54	0.140 54	0.438 54	0.000 49	0.177 53	0.008 54	0.121 54	0.069 54	0.004 55	0.231 50	0.429 50	0.036 55	0.445 54	0.273 52	0.333 56	0.119 55

Sgpn_scannet	0.143 56	0.208 57	0.390 55	0.169 53	0.065 55	0.275 56	0.029 41	0.069 54	0.000 56	0.087 56	0.043 55	0.014 54	0.027 57	0.000 56	0.112 52	0.351 56	0.168 56	0.438 55	0.138 54

MaskRCNN 2d->3d Proj	0.058 57	0.333 53	0.002 57	0.000 57	0.053 56	0.002 57	0.002 48	0.021 57	0.000 56	0.045 57	0.024 57	0.238 39	0.065 56	0.000 56	0.014 56	0.107 57	0.020 57	0.110 57	0.006 57

3D Semantic Instance Benchmark