3D Semantic instance benchmark
The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.
Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.
This table lists the benchmark results for the 3D semantic instance scenario.
| Method | Info | avg ap 50% | bathtub | bed | bookshelf | cabinet | chair | counter | curtain | desk | door | otherfurniture | picture | refrigerator | shower curtain | sink | sofa | table | toilet | window |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
 |  | | | | | | | | | | | | | | | | | | ||
| PointGroup | 0.636 1 | 1.000 1 | 0.765 4 | 0.624 4 | 0.505 3 | 0.797 2 | 0.116 5 | 0.696 3 | 0.384 2 | 0.441 2 | 0.559 1 | 0.476 1 | 0.596 2 | 1.000 1 | 0.666 1 | 0.756 2 | 0.556 5 | 0.997 5 | 0.513 2 | |
| OccuSeg | 0.634 2 | 1.000 1 | 0.902 1 | 0.771 1 | 0.461 4 | 0.814 1 | 0.282 1 | 0.583 7 | 0.328 3 | 0.472 1 | 0.471 2 | 0.295 6 | 0.600 1 | 1.000 1 | 0.650 2 | 0.664 6 | 0.587 2 | 1.000 1 | 0.537 1 | |
| MPA | 0.595 3 | 1.000 1 | 0.833 2 | 0.668 3 | 0.517 1 | 0.748 3 | 0.131 4 | 0.539 8 | 0.466 1 | 0.423 3 | 0.420 5 | 0.362 3 | 0.589 3 | 0.857 5 | 0.424 6 | 0.755 3 | 0.558 4 | 1.000 1 | 0.420 4 | |
| GICN | 0.586 4 | 1.000 1 | 0.716 6 | 0.769 2 | 0.438 5 | 0.644 5 | 0.140 2 | 0.714 2 | 0.292 5 | 0.371 6 | 0.305 9 | 0.353 4 | 0.550 4 | 1.000 1 | 0.525 3 | 0.789 1 | 0.531 7 | 1.000 1 | 0.412 5 | |
| MTML | 0.549 5 | 1.000 1 | 0.807 3 | 0.588 7 | 0.327 8 | 0.647 4 | 0.004 14 | 0.815 1 | 0.180 8 | 0.418 4 | 0.364 8 | 0.182 8 | 0.445 6 | 1.000 1 | 0.442 5 | 0.688 5 | 0.571 3 | 1.000 1 | 0.396 6 | |
| Jean Lahoud, Bernard Ghanem, Marc Pollefeys, Martin R. Oswald: 3D Instance Segmentation via Multi-task Metric Learning. ICCV 2019 [oral] | ||||||||||||||||||||
| Occipital-SCS | 0.512 6 | 1.000 1 | 0.716 6 | 0.509 8 | 0.506 2 | 0.611 7 | 0.092 7 | 0.602 6 | 0.177 9 | 0.346 8 | 0.383 7 | 0.165 9 | 0.442 7 | 0.850 8 | 0.386 9 | 0.618 8 | 0.543 6 | 0.889 10 | 0.389 7 | |
| 3D-BoNet | 0.488 7 | 1.000 1 | 0.672 10 | 0.590 6 | 0.301 9 | 0.484 12 | 0.098 6 | 0.620 4 | 0.306 4 | 0.341 9 | 0.259 11 | 0.125 11 | 0.434 8 | 0.796 9 | 0.402 8 | 0.499 13 | 0.513 8 | 0.909 9 | 0.439 3 | |
| Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight | ||||||||||||||||||||
| PanopticFusion-inst | 0.478 8 | 0.667 10 | 0.712 9 | 0.595 5 | 0.259 11 | 0.550 11 | 0.000 17 | 0.613 5 | 0.175 10 | 0.250 12 | 0.434 3 | 0.437 2 | 0.411 10 | 0.857 5 | 0.485 4 | 0.591 11 | 0.267 14 | 0.944 7 | 0.359 8 | |
| Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear) | ||||||||||||||||||||
| ResNet-backbone | 0.459 9 | 1.000 1 | 0.737 5 | 0.159 15 | 0.259 10 | 0.587 9 | 0.138 3 | 0.475 10 | 0.217 7 | 0.416 5 | 0.408 6 | 0.128 10 | 0.315 11 | 0.714 10 | 0.411 7 | 0.536 12 | 0.590 1 | 0.873 12 | 0.304 9 | |
| MASC |  | 0.447 10 | 0.528 13 | 0.555 12 | 0.381 9 | 0.382 6 | 0.633 6 | 0.002 15 | 0.509 9 | 0.260 6 | 0.361 7 | 0.432 4 | 0.327 5 | 0.451 5 | 0.571 11 | 0.367 10 | 0.639 7 | 0.386 9 | 0.980 6 | 0.276 10 |
| Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation. | ||||||||||||||||||||
| 3D-SIS | | 0.382 11 | 1.000 1 | 0.432 13 | 0.245 12 | 0.190 12 | 0.577 10 | 0.013 12 | 0.263 12 | 0.033 15 | 0.320 10 | 0.240 12 | 0.075 13 | 0.422 9 | 0.857 5 | 0.117 14 | 0.699 4 | 0.271 13 | 0.883 11 | 0.235 12 |
| Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019 | ||||||||||||||||||||
| UNet-backbone | 0.319 12 | 0.667 10 | 0.715 8 | 0.233 13 | 0.189 13 | 0.479 13 | 0.008 13 | 0.218 13 | 0.067 14 | 0.201 13 | 0.173 13 | 0.107 12 | 0.123 13 | 0.438 12 | 0.150 12 | 0.615 9 | 0.355 10 | 0.916 8 | 0.093 16 | |
| R-PointNet | 0.306 13 | 0.500 14 | 0.405 14 | 0.311 10 | 0.348 7 | 0.589 8 | 0.054 8 | 0.068 15 | 0.126 11 | 0.283 11 | 0.290 10 | 0.028 14 | 0.219 12 | 0.214 15 | 0.331 11 | 0.396 14 | 0.275 12 | 0.821 14 | 0.245 11 | |
| 3D-BEVIS | 0.248 14 | 0.667 10 | 0.566 11 | 0.076 16 | 0.035 17 | 0.394 14 | 0.027 10 | 0.035 16 | 0.098 12 | 0.099 15 | 0.030 16 | 0.025 15 | 0.098 14 | 0.375 13 | 0.126 13 | 0.604 10 | 0.181 15 | 0.854 13 | 0.171 13 | |
| Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation. | ||||||||||||||||||||
| Seg-Cluster | | 0.215 15 | 0.370 15 | 0.337 16 | 0.285 11 | 0.105 14 | 0.325 15 | 0.025 11 | 0.282 11 | 0.085 13 | 0.105 14 | 0.107 14 | 0.007 17 | 0.079 15 | 0.317 14 | 0.114 15 | 0.309 16 | 0.304 11 | 0.587 15 | 0.123 15 |
| Sgpn_scannet | 0.143 16 | 0.208 17 | 0.390 15 | 0.169 14 | 0.065 15 | 0.275 16 | 0.029 9 | 0.069 14 | 0.000 16 | 0.087 16 | 0.043 15 | 0.014 16 | 0.027 17 | 0.000 16 | 0.112 16 | 0.351 15 | 0.168 16 | 0.438 16 | 0.138 14 | |
| MaskRCNN 2d->3d Proj | 0.058 17 | 0.333 16 | 0.002 17 | 0.000 17 | 0.053 16 | 0.002 17 | 0.002 16 | 0.021 17 | 0.000 16 | 0.045 17 | 0.024 17 | 0.238 7 | 0.065 16 | 0.000 16 | 0.014 17 | 0.107 17 | 0.020 17 | 0.110 17 | 0.006 17 | |