Documentation
Submission Policy
The 1506 annotated scans of the Scan2CAD dataset release may be used for learning the parameters of the algorithms. The test data should be used strictly for reporting the final results -- this benchmark is not meant for iterative testing sessions or parameter tweaking.
Parameter tuning is only allowed on the training data. Evaluating on the test data via this evaluation server must only be done once for the final system. It is not permitted to use it to train systems, for example by trying out different parameter values and choosing the best. Only one version must be evaluated (which performed best on the training data). This is to avoid overfitting on the test data. Results of different parameter settings of an algorithm can therefore only be reported on the training set. To help enforcing this policy, we block updates to the test set results of a method for two weeks after a test set submission. You can split up the training data into training and validation sets yourself as you wish.
It is not permitted to register on this webpage with multiple e-mail addresses. We will ban users or domains if required.
General Info
Code page: Detailed information about our dataset and file formats is provided on our github page, please see our git repo.
Download Scan2CAD: If you would like to download the public Scan2CAD dataset (for training purposes), please fill out this google-form and we will reply to your request.
Download ScanNet: The ScanNet project page can be found here. If you would like to download the ScanNet data, please fill out an agreement to the ScanNet Terms of Use and send it to the the scannet group email.
Tasks Data Requirements: For all tasks, both 2D and 3D data modalities can be used as input.
Evaluation scripts: In our git repo, we provide the used evaluation script.Benchmark Input Files
If you want to participate to this benchmark, in the Scan2CAD dataset folder you will find a subfolder named benchmark_input_files. This contains few necessary input files:
- CAD Appearances: This file contains information on the number of appearances of a CAD model appears per scene.
- Scene List: This file contains a list of all scenes that appear in the hidden test set which are used in this benchmark.
Submission Format
Results for a method must be uploaded as a single .zip or .7z file, which when unzipped must contain a single folder in which all results are saved as .csv file format. With other words: There must not be any additional files or folders in the archive except those specified below.
unzip_root/
my_folder/
|-- scene0707_00.csv
|-- scene0708_00.csv
|-- scene0709_00.csv
⋮
|-- scene0806_00.csv
An alignment file ( .csv ) has to be structured as following. Note: category-id and id are ShapeNet model ids. Transformations are applied in this order x' = T * R * S * x.
| # category-id | id | tx | ty | tz | qw | qx | qy | qz | sx | sy | sz |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 03001627 | 38bdba5f6455c5ff91663a74ccd2338 | 4.82 | 7.43 | 0.57 | 0.31 | 0.29 | 0.57 | 0.69 | 1.40 | 1.25 | 1.41 |
| ... | |||||||||||
| 02747177 | 85d8a1ad55fa646878725384d6baf445 | 4.27 | 9.94 | 0.88 | 0.59 | 0.69 | 0.30 | 0.27 | 0.52 | 0.44 | 0.52 |
| 04256520 | 23833969c0011b8e98494085d68ad6a0 | 3.36 | 5.10 | 0.48 | 0.94 | 0.00 | -0.0 | 0.31 | 3.25 | 3.11 | 2.87 |
How many times one CAD model appears in a scene is given by cad_appearances_hidden_testset.json (see benchmark input files). It is important to know that in your results .csv file one CAD model should not appear more often than allowed. An example: The afromentioned .json states that for the hypothetical scene scene0999_00 the CAD model category-id=00000001, id=000000000000abc appears 3 times. However in your .csv file that CAD model appears 5 times. Hence, for the benchmark we will only take the first 3 into consideration, the excess will be ignored.
Evaluation Logic
A model alignment is considered successful only if the category of the CAD model matches that of the scan object and the pose error is within translation, rotational, and scale bounds relative to the ground truth CAD. We do not enforce strict instance matching (i.e., matching the exact CAD model of the ground truth annotation). Instead, we treat CAD models of the same category as interchangeable (according to the ShapeNetCorev2 top-level synset). Once a CAD model is determined to be aligned correctly, the ground truth counterpart is removed from the candidate pool in order to prevent multiple alignments to the same object. Alignments are fully parameterized by 9 pose parameters.
The error thresholds for a successful alignment are set to
- t ≤ 20cm for translation
- r ≤ 20deg for rotation
- s ≤ 20% for scale
Evaluation Metric
The metrics for this benchmark are accuracies. Accuracies are calculated for the top-8 most frequent classes: bathtub, bookshelf, cabinet, chair, display, other, sofa, table, trash bin.
avg (per CAD) is the average accuracy over all annotated CAD models in the test set.
avg (per class) is the average accuracy over the top-8 classes.