3D Semantic instance benchmark
The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.
Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.
This table lists the benchmark results for the 3D semantic instance scenario.
| Method | Info | avg ap 50% | bathtub | bed | bookshelf | cabinet | chair | counter | curtain | desk | door | otherfurniture | picture | refrigerator | shower curtain | sink | sofa | table | toilet | window |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
 |  | | | | | | | | | | | | | | | | | | ||
| DCNet | 0.634 1 | 1.000 1 | 0.902 1 | 0.771 1 | 0.461 2 | 0.814 1 | 0.282 1 | 0.583 5 | 0.328 1 | 0.472 1 | 0.471 1 | 0.295 3 | 0.600 1 | 1.000 1 | 0.650 1 | 0.664 3 | 0.587 2 | 1.000 1 | 0.537 1 | |
| MTML | 0.549 2 | 1.000 1 | 0.807 2 | 0.588 4 | 0.327 5 | 0.647 2 | 0.004 12 | 0.815 1 | 0.180 5 | 0.418 2 | 0.364 6 | 0.182 5 | 0.445 3 | 1.000 1 | 0.442 4 | 0.688 2 | 0.571 3 | 1.000 1 | 0.396 3 | |
| Occipital-SCS | 0.512 3 | 1.000 1 | 0.716 4 | 0.509 5 | 0.506 1 | 0.611 4 | 0.092 5 | 0.602 4 | 0.177 6 | 0.346 5 | 0.383 5 | 0.165 6 | 0.442 4 | 0.850 5 | 0.386 7 | 0.618 5 | 0.543 4 | 0.889 8 | 0.389 4 | |
| 3D-BoNet | 0.488 4 | 1.000 1 | 0.672 7 | 0.590 3 | 0.301 6 | 0.484 9 | 0.098 4 | 0.620 2 | 0.306 2 | 0.341 6 | 0.259 8 | 0.125 8 | 0.434 5 | 0.796 6 | 0.402 6 | 0.499 10 | 0.513 5 | 0.909 7 | 0.439 2 | |
| Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, Niki Trigoni: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. NeurIPS 2019 Spotlight | ||||||||||||||||||||
| PanopticFusion-inst | 0.478 5 | 0.667 7 | 0.712 6 | 0.595 2 | 0.259 8 | 0.550 8 | 0.000 15 | 0.613 3 | 0.175 7 | 0.250 9 | 0.434 2 | 0.437 1 | 0.411 7 | 0.857 3 | 0.485 2 | 0.591 8 | 0.267 12 | 0.944 5 | 0.359 5 | |
| Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. IROS 2019 (to appear) | ||||||||||||||||||||
| ResNet-backbone | 0.459 6 | 1.000 1 | 0.737 3 | 0.159 13 | 0.259 7 | 0.587 6 | 0.138 2 | 0.475 7 | 0.217 4 | 0.416 3 | 0.408 4 | 0.128 7 | 0.315 8 | 0.714 7 | 0.411 5 | 0.536 9 | 0.590 1 | 0.873 10 | 0.304 6 | |
| MASC |  | 0.447 7 | 0.528 10 | 0.555 9 | 0.381 7 | 0.382 3 | 0.633 3 | 0.002 13 | 0.509 6 | 0.260 3 | 0.361 4 | 0.432 3 | 0.327 2 | 0.451 2 | 0.571 8 | 0.367 8 | 0.639 4 | 0.386 7 | 0.980 3 | 0.276 7 |
| Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation. | ||||||||||||||||||||
| 3D-SIS | | 0.382 8 | 1.000 1 | 0.432 11 | 0.245 10 | 0.190 10 | 0.577 7 | 0.013 10 | 0.263 10 | 0.033 13 | 0.320 7 | 0.240 10 | 0.075 11 | 0.422 6 | 0.857 3 | 0.117 12 | 0.699 1 | 0.271 11 | 0.883 9 | 0.235 10 |
| Ji Hou, Angela Dai, Matthias Niessner: 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. CVPR 2019 | ||||||||||||||||||||
| DPC-instance | 0.355 9 | 0.500 11 | 0.517 10 | 0.467 6 | 0.228 9 | 0.422 11 | 0.133 3 | 0.405 8 | 0.111 9 | 0.205 10 | 0.241 9 | 0.075 10 | 0.233 9 | 0.306 12 | 0.445 3 | 0.439 11 | 0.457 6 | 0.974 4 | 0.239 9 | |
| Francis Engelmann, Theodora Kontogianni, Bastian Leibe: Dilated Point Convolutions. arXiv | ||||||||||||||||||||
| UNet-backbone | 0.319 10 | 0.667 7 | 0.715 5 | 0.233 11 | 0.189 11 | 0.479 10 | 0.008 11 | 0.218 11 | 0.067 12 | 0.201 11 | 0.173 11 | 0.107 9 | 0.123 11 | 0.438 9 | 0.150 10 | 0.615 6 | 0.355 8 | 0.916 6 | 0.093 14 | |
| R-PointNet | 0.306 11 | 0.500 11 | 0.405 12 | 0.311 8 | 0.348 4 | 0.589 5 | 0.054 6 | 0.068 13 | 0.126 8 | 0.283 8 | 0.290 7 | 0.028 12 | 0.219 10 | 0.214 13 | 0.331 9 | 0.396 12 | 0.275 10 | 0.821 12 | 0.245 8 | |
| 3D-BEVIS | 0.248 12 | 0.667 7 | 0.566 8 | 0.076 14 | 0.035 15 | 0.394 12 | 0.027 8 | 0.035 14 | 0.098 10 | 0.099 13 | 0.030 14 | 0.025 13 | 0.098 12 | 0.375 10 | 0.126 11 | 0.604 7 | 0.181 13 | 0.854 11 | 0.171 11 | |
| Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation. | ||||||||||||||||||||
| Seg-Cluster | | 0.215 13 | 0.370 13 | 0.337 14 | 0.285 9 | 0.105 12 | 0.325 13 | 0.025 9 | 0.282 9 | 0.085 11 | 0.105 12 | 0.107 12 | 0.007 15 | 0.079 13 | 0.317 11 | 0.114 13 | 0.309 14 | 0.304 9 | 0.587 13 | 0.123 13 |
| Sgpn_scannet | 0.143 14 | 0.208 15 | 0.390 13 | 0.169 12 | 0.065 13 | 0.275 14 | 0.029 7 | 0.069 12 | 0.000 14 | 0.087 14 | 0.043 13 | 0.014 14 | 0.027 15 | 0.000 14 | 0.112 14 | 0.351 13 | 0.168 14 | 0.438 14 | 0.138 12 | |
| MaskRCNN 2d->3d Proj | 0.058 15 | 0.333 14 | 0.002 15 | 0.000 15 | 0.053 14 | 0.002 15 | 0.002 14 | 0.021 15 | 0.000 14 | 0.045 15 | 0.024 15 | 0.238 4 | 0.065 14 | 0.000 14 | 0.014 15 | 0.107 15 | 0.020 15 | 0.110 15 | 0.006 15 | |