3D Semantic instance benchmark

The 3D semantic instance prediction task involves detecting and segmenting the object in an 3D scan mesh.

Evaluation and metrics

Our evaluation ranks all methods according to the average precision for each class. We report the mean average precision AP at overlap 0.25 (AP 25%), overlap 0.5 (AP 50%), and over overlaps in the range [0.5:0.95:0.05] (AP). Note that multiple predictions of the same ground truth instance are penalized as false positives.


This table lists the benchmark results for the 3D semantic instance scenario.




Method Infoavg ap 50%bathtubbedbookshelfcabinetchaircountercurtaindeskdoorotherfurniturepicturerefrigeratorshower curtainsinksofatabletoiletwindow
sorted bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort bysort by
MTML0.481 11.000 10.666 40.377 30.272 30.709 10.001 100.579 20.254 20.361 30.318 40.095 60.432 21.000 10.184 50.601 50.487 20.938 30.384 1
PanopticFusion-inst0.478 20.667 40.712 30.595 10.259 50.550 60.000 110.613 10.175 40.250 60.434 10.437 10.411 40.857 20.485 10.591 60.267 80.944 20.359 2
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji: PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things. arXiv
ResNet-backbone0.459 31.000 10.737 10.159 90.259 40.587 40.138 10.475 40.217 30.416 10.408 30.128 40.315 50.714 40.411 20.536 70.590 10.873 60.304 3
MASCpermissive0.447 40.528 70.555 60.381 20.382 10.633 20.002 80.509 30.260 10.361 20.432 20.327 20.451 10.571 50.367 30.639 20.386 30.980 10.276 4
Chen Liu, Yasutaka Furukawa: MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation.
3D-SIS0.382 51.000 10.432 70.245 60.190 60.577 50.013 60.263 60.033 90.320 40.240 60.075 70.422 30.857 20.117 80.699 10.271 70.883 50.235 6
UNet-backbone0.319 60.667 40.715 20.233 70.189 70.479 70.008 70.218 70.067 80.201 70.173 70.107 50.123 70.438 60.150 60.615 30.355 40.916 40.093 10
R-PointNet0.306 70.500 80.405 80.311 40.348 20.589 30.054 20.068 90.126 50.283 50.290 50.028 80.219 60.214 90.331 40.396 80.275 60.821 80.245 5
3D-BEVIS0.248 80.667 40.566 50.076 100.035 110.394 80.027 40.035 100.098 60.099 90.030 100.025 90.098 80.375 70.126 70.604 40.181 90.854 70.171 7
Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian Leibe: 3D-BEVIS: Birds-Eye-View Instance Segmentation.
Seg-Clusterpermissive0.215 90.370 90.337 100.285 50.105 80.325 90.025 50.282 50.085 70.105 80.107 80.007 110.079 90.317 80.114 90.309 100.304 50.587 90.123 9
Sgpn_scannet0.143 100.208 110.390 90.169 80.065 90.275 100.029 30.069 80.000 100.087 100.043 90.014 100.027 110.000 100.112 100.351 90.168 100.438 100.138 8
MaskRCNN 2d->3d Proj0.058 110.333 100.002 110.000 110.053 100.002 110.002 90.021 110.000 100.045 110.024 110.238 30.065 100.000 100.014 110.107 110.020 110.110 110.006 11