The benchmark is currently evaluated on the v2 (Feb 02, 2025) version of the NeRSemble dataset.
Benchmark results are evaluated on the hidden test sets, which are not publicly available. Users must not attempt to optimize results on the hidden test set through repeated submissions or to access test set data in any way.
A user may create upto 10 submissions for each task. Creating multiple accounts to bypass this limit is prohibited, and will cause the user to get banned from the submission system.
The interval between two submissions must be at least 2 hours. It may take upto 24 hours after a valid submission for the results to appear on the public leaderboard.
Upon submission, a shallow check is performed to ensure that folder structure, file names, video resolutions, and number of frames are correct. However, note that submissions that pass this shallow check but have other formal flaws will still count towards a user's submission quota.
The hidden test sets (and hence the submission system) must not be used for ablation studies.
The benchmark data can be downloaded via the NeRSemble Benchmark Toolkit.
Upon installation, it provides a command nersemble-benchmark-download ${benchmark_folder} nvs that downloads the training data for the dynamic novel view synthesis task to your local ${benchmark_folder}.
A submission must include algorithm-generated videos (supported format: mp4) for each of the 3 hold-out cameras and for each of the 5 individuals for a total of 15 videos.
Generated videos must have the same resolution as the train videos (i.e., 1604x1100). For more details of evaluation see the dynamic novel view synthesis page.
The .zip file should be uploaded to the submission system. A submission should have the following structure (following the structure of the training data):
nvs_submission.zip
├── 388
│ └── GLASSES
│ ├── cam_222200037.mp4 # <- Video predictions from your method
│ ├── cam_222200039.mp4
│ └── cam_222200046.mp4
├── 422
│ └── EXP-2-eyes
│ ├── cam_222200037.mp4
│ ├── cam_222200039.mp4
│ └── cam_222200046.mp4
┆
└── 475
└── ...
NVSSubmissionDataWriter Python class that facilitates creating the submission .zip file.
The benchmark data can be downloaded via the NeRSemble Benchmark Toolkit.
Upon installation, it provides a command nersemble-benchmark-download ${benchmark_folder} mono_flame_avatar that downloads the training data for the monocular FLAME avatar task to your local ${benchmark_folder}.
A submission must include algorithm-generated videos (supported format: mp4) for all 4 cameras (3 hold-out + 1 train viewpoint), for each of the 4 hold-out sequences, and for each of the 5 individuals for a total of 80 videos.
Generated videos must have the same resolution as the train videos (i.e., 512x512). For more details of evaluation see the monocular FLAME avatar page.
The .zip file should be uploaded to the submission system. A submission should have the following structure (following the structure of the training data):
mono_flame_avatar_submission.zip
├── 393
│ ├── EMO-1-shout+laugh
│ │ ├── cam_220700191.mp4 # <- Video predictions from your method
│ │ ├── cam_222200037.mp4
│ │ ├── cam_222200039.mp4
│ │ └── cam_222200046.mp4
│ ┆
│ └── SEN-10-port_strong_smokey
│ ├── cam_220700191.mp4
│ ├── cam_222200037.mp4
│ ├── cam_222200039.mp4
│ └── cam_222200046.mp4
┆
└── 486
└── ...
MonoFlameAvatarSubmissionDataWriter Python class that facilitates creating the submission .zip file.