3D Semantic instance benchmark
The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.
Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.
This table lists the benchmark results for the 3D semantic instance scenario.
| Method | Info | avg ap 50% | bathtub | bed | bookshelf | cabinet | chair | counter | curtain | desk | door | otherfurniture | picture | refrigerator | shower curtain | sink | sofa | table | toilet | window |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
 |  | | | | | | | | | | | | | | | | | | ||
| CRNet | 0.688 1 | 1.000 1 | 0.802 6 | 0.783 3 | 0.644 1 | 0.804 4 | 0.182 2 | 0.874 1 | 0.511 4 | 0.501 2 | 0.592 1 | 0.536 3 | 0.587 8 | 1.000 1 | 0.637 4 | 0.730 7 | 0.683 1 | 0.994 9 | 0.523 2 | |
| OccuSeg+instance | 0.672 2 | 1.000 1 | 0.758 10 | 0.682 6 | 0.576 3 | 0.842 1 | 0.477 1 | 0.504 14 | 0.524 3 | 0.567 1 | 0.585 2 | 0.451 5 | 0.557 9 | 1.000 1 | 0.751 1 | 0.797 3 | 0.563 7 | 1.000 1 | 0.467 7 | |
| Mask-Group | 0.664 3 | 1.000 1 | 0.822 3 | 0.764 5 | 0.616 2 | 0.815 2 | 0.139 5 | 0.694 6 | 0.597 2 | 0.459 5 | 0.566 3 | 0.599 1 | 0.600 5 | 0.516 17 | 0.715 2 | 0.819 2 | 0.635 3 | 1.000 1 | 0.603 1 | |
| PE | 0.645 4 | 1.000 1 | 0.773 7 | 0.798 2 | 0.538 4 | 0.786 8 | 0.088 11 | 0.799 3 | 0.350 9 | 0.435 9 | 0.547 5 | 0.545 2 | 0.646 4 | 0.933 6 | 0.562 6 | 0.761 5 | 0.556 11 | 0.997 7 | 0.501 5 | |
| GICN | 0.638 5 | 1.000 1 | 0.895 1 | 0.800 1 | 0.480 8 | 0.676 10 | 0.144 4 | 0.737 4 | 0.354 8 | 0.447 6 | 0.400 13 | 0.365 9 | 0.700 1 | 1.000 1 | 0.569 5 | 0.836 1 | 0.599 4 | 1.000 1 | 0.473 6 | |
| PointGroup | 0.636 6 | 1.000 1 | 0.765 8 | 0.624 7 | 0.505 7 | 0.797 5 | 0.116 8 | 0.696 5 | 0.384 7 | 0.441 7 | 0.559 4 | 0.476 4 | 0.596 6 | 1.000 1 | 0.666 3 | 0.756 6 | 0.556 10 | 0.997 7 | 0.513 3 | |
| Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, Jiaya Jia: PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. CVPR 2020 [oral] | ||||||||||||||||||||
| 3D-MPA | 0.611 7 | 1.000 1 | 0.833 2 | 0.765 4 | 0.526 5 | 0.756 9 | 0.136 7 | 0.588 11 | 0.470 5 | 0.438 8 | 0.432 11 | 0.358 10 | 0.650 2 | 0.857 8 | 0.429 10 | 0.765 4 | 0.557 9 | 1.000 1 | 0.430 9 | |
| Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias Nießner: 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation. CVPR 2020 | ||||||||||||||||||||
| PCJC | 0.578 8 | 1.000 1 | 0.810 4 | 0.583 11 | 0.449 11 | 0.813 3 | 0.042 14 | 0.603 9 | 0.341 10 | 0.490 3 | 0.465 7 | 0.410 7 | 0.650 2 | 0.835 12 | 0.264 17 | 0.694 9 | 0.561 8 | 0.889 15 | 0.504 4 | |
| SSEN | 0.575 9 | 1.000 1 | 0.761 9 | 0.473 13 | 0.477 9 | 0.795 6 | 0.066 12 | 0.529 12 | 0.658 1 | 0.460 4 | 0.461 8 | 0.380 8 | 0.331 16 | 0.859 7 | 0.401 13 | 0.692 10 | 0.653 2 | 1.000 1 | 0.348 13 | |
| Dongsu Zhang, Junha Chun, Sang Kyun Cha, Young Min Kim: Spatial Semantic Embedding Network: Fast 3D Instance Segmentation with Deep Metric Learning. Arxiv | ||||||||||||||||||||
| MTML | 0.549 10 | 1.000 1 | 0.807 5 | 0.588 10 | 0.327 14 | 0.647 11 | 0.004 20 | 0.815 2 | 0.180 14 | 0.418 10 | 0.364 15 | 0.182 13 | 0.445 11 | 1.000 1 | 0.442 9 | 0.688 11 | 0.571 6 | 1.000 1 | 0.396 10 | |
| Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald: 3D Instance Segmentation via Multi-task Metric Learning. ICCV 2019 [oral] | ||||||||||||||||||||
| Sparse R-CNN | 0.515 11 | 1.000 1 | 0.538 18 | 0.282 17 | 0.468 10 | 0.790 7 | 0.173 3 | 0.345 16 | 0.429 6 | 0.413 12 | 0.484 6 | 0.176 14 | 0.595 7 | 0.591 15 | 0.522 7 | 0.668 12 | 0.476 14 | 0.986 10 | 0.327 14 | |
| Occipital-SCS | 0.512 12 | 1.000 1 | 0.716 12 | 0.509 12 | 0.506 6 | 0.611 13 | 0.092 10 | 0.602 10 | 0.177 15 | 0.346 14 | 0.383 14 | 0.165 15 | 0.442 12 | 0.850 11 | 0.386 14 | 0.618 14 | 0.543 12 | 0.889 15 | 0.389 11 | |
| 3D-BoNet | 0.488 13 | 1.000 1 | 0.672 15 | 0.590 9 | 0.301 15 | 0.484 19 | 0.098 9 | 0.620 7 | 0.306 11 | 0.341 15 | 0.259 17 | 0.125 17 | 0.434 13 | 0.796 13 | 0.402 12 | 0.499 19 | 0.513 13 | 0.909 14 | 0.439 8 | |
| Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight | ||||||||||||||||||||
| PanopticFusion-inst | 0.478 14 | 0.667 16 | 0.712 14 | 0.595 8 | 0.259 17 | 0.550 17 | 0.000 23 | 0.613 8 | 0.175 16 | 0.250 18 | 0.434 9 | 0.437 6 | 0.411 15 | 0.857 8 | 0.485 8 | 0.591 17 | 0.267 21 | 0.944 12 | 0.359 12 | |
| Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear) | ||||||||||||||||||||
| SALoss-ResNet | 0.459 15 | 1.000 1 | 0.737 11 | 0.159 22 | 0.259 16 | 0.587 15 | 0.138 6 | 0.475 15 | 0.217 13 | 0.416 11 | 0.408 12 | 0.128 16 | 0.315 17 | 0.714 14 | 0.411 11 | 0.536 18 | 0.590 5 | 0.873 18 | 0.304 15 | |
| Zhidong Liang, Ming Yang, Hao Li, Chunxiang Wang: 3D Instance Embedding Learning With a Structure-Aware Loss Function for Point Cloud Segmentation. IEEE Robotics and Automation Letters (IROS2020) | ||||||||||||||||||||
| MASC |  | 0.447 16 | 0.528 20 | 0.555 17 | 0.381 14 | 0.382 12 | 0.633 12 | 0.002 21 | 0.509 13 | 0.260 12 | 0.361 13 | 0.432 10 | 0.327 11 | 0.451 10 | 0.571 16 | 0.367 15 | 0.639 13 | 0.386 15 | 0.980 11 | 0.276 16 |
| Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation. | ||||||||||||||||||||
| 3D-SIS | | 0.382 17 | 1.000 1 | 0.432 20 | 0.245 18 | 0.190 18 | 0.577 16 | 0.013 18 | 0.263 18 | 0.033 22 | 0.320 16 | 0.240 18 | 0.075 19 | 0.422 14 | 0.857 8 | 0.117 20 | 0.699 8 | 0.271 20 | 0.883 17 | 0.235 18 |
| Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019 | ||||||||||||||||||||
| UNet-backbone | 0.319 18 | 0.667 16 | 0.715 13 | 0.233 19 | 0.189 19 | 0.479 20 | 0.008 19 | 0.218 19 | 0.067 21 | 0.201 19 | 0.173 19 | 0.107 18 | 0.123 20 | 0.438 18 | 0.150 18 | 0.615 15 | 0.355 16 | 0.916 13 | 0.093 23 | |
| R-PointNet | 0.306 19 | 0.500 21 | 0.405 21 | 0.311 15 | 0.348 13 | 0.589 14 | 0.054 13 | 0.068 22 | 0.126 17 | 0.283 17 | 0.290 16 | 0.028 21 | 0.219 18 | 0.214 21 | 0.331 16 | 0.396 21 | 0.275 19 | 0.821 20 | 0.245 17 | |
| Region | 0.248 20 | 0.667 16 | 0.437 19 | 0.188 20 | 0.153 20 | 0.491 18 | 0.000 23 | 0.208 20 | 0.094 19 | 0.153 20 | 0.099 21 | 0.057 20 | 0.217 19 | 0.119 22 | 0.039 23 | 0.466 20 | 0.302 18 | 0.640 21 | 0.140 20 | |
| 3D-BEVIS | 0.248 20 | 0.667 16 | 0.566 16 | 0.076 23 | 0.035 24 | 0.394 21 | 0.027 16 | 0.035 23 | 0.098 18 | 0.099 22 | 0.030 23 | 0.025 22 | 0.098 21 | 0.375 19 | 0.126 19 | 0.604 16 | 0.181 22 | 0.854 19 | 0.171 19 | |
| Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation. | ||||||||||||||||||||
| Seg-Cluster | | 0.215 22 | 0.370 22 | 0.337 23 | 0.285 16 | 0.105 21 | 0.325 22 | 0.025 17 | 0.282 17 | 0.085 20 | 0.105 21 | 0.107 20 | 0.007 24 | 0.079 22 | 0.317 20 | 0.114 21 | 0.309 23 | 0.304 17 | 0.587 22 | 0.123 22 |
| Sgpn_scannet | 0.143 23 | 0.208 24 | 0.390 22 | 0.169 21 | 0.065 22 | 0.275 23 | 0.029 15 | 0.069 21 | 0.000 23 | 0.087 23 | 0.043 22 | 0.014 23 | 0.027 24 | 0.000 23 | 0.112 22 | 0.351 22 | 0.168 23 | 0.438 23 | 0.138 21 | |
| MaskRCNN 2d->3d Proj | 0.058 24 | 0.333 23 | 0.002 24 | 0.000 24 | 0.053 23 | 0.002 24 | 0.002 22 | 0.021 24 | 0.000 23 | 0.045 24 | 0.024 24 | 0.238 12 | 0.065 23 | 0.000 23 | 0.014 24 | 0.107 24 | 0.020 24 | 0.110 24 | 0.006 24 | |