Novel View Synthesis on DSLR Images
The novel view synthesis task is to render images from novel viewpoints given a dense RGB capture of the scene. The images are captured by a fisheye DSLR camera, and camera poses from COLMAP are provided for every training and test image.
We also provide the undistorted evaluation track: rendering undistorted perspective (pinhole) images of the given poses. The training images and the GT are generated from the raw fisheye images using the ScanNet++ Toolbox.
The complete testing set for NVS consists of 50 scenes (referred to as the full set). To facilitate per-scene optimization methods, we also offer a smaller subset containing 12 scenes (referred to as the small set). Users have the option to submit results for the small set alone or for the full set. Submissions for the full set will include evaluations for both the small set and the full set.
Evaluation and Metrics
We evaluate the similarity beween the ground truth and generated RGB images. Our evaluation metrics are peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and learned perceptual image patch similarity (LPIPS). For each pair of generated and ground-truth images, we compute these three metrics, and the numbers reported in the table are the average over all the images across all the scenes.
Evaluation is carried out on GT images with resolution 1752 x 1168. Submitted images will be automatically resized if their resolutions differ from this.
Evaluation excludes the pixels which are anonymized. Anonymized pixels are specified in resized_anon_masks and original_anon_masks.
The benchmark is currently evaluated on the v1 (Nov 2023) version of the dataset.
Results (small set)
Small set is the subset of the full testing set of NVS, which contains 12 scenes.
| Methods | PSNR | SSIM | LPIPS |
|---|---|---|---|
| Zip-NeRF | 24.630 | 0.887 | 0.320 |
| Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, Peter Hedman. Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields. ICCV 2023 | |||
| RPBG | 24.005 | 0.882 | 0.271 |
| Zizhuang Wei, Qingtian Zhu, et al. RPBG: Towards Robust Neural Point-based Graphics in the Wild. ECCV 2024 | |||
| FeatSplat | 24.177 | 0.880 | 0.303 |
| Tomas Berriel Martins, Javier Civera. Feature Splatting for Better Novel View Synthesis with Low Overlap. BMVC 2024 | |||
| Nerfacto | 23.498 | 0.868 | 0.340 |
| Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, Angjoo Kanazawa. Nerfstudio: A Modular Framework for Neural Radiance Field Development. SIGGRAPH 2023 | |||
| TensoRF | 23.524 | 0.857 | 0.404 |
| Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, Hao Su. TensoRF: Tensorial Radiance Fields. ECCV 2022 | |||
| Gaussian Splatting | 23.389 | 0.876 | 0.312 |
| Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George Drettakis. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. SIGGRAPH 2023 | |||
| Instant-NGP | 23.695 | 0.871 | 0.363 |
| Thomas Müller, Alex Evans, Christoph Schied, Alexander Keller. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. SIGGRAPH 2022 | |||
| Plenoxels | 22.177 | 0.841 | 0.399 |
| Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, Angjoo Kanazawa. Plenoxels: Radiance Fields without Neural Networks. CVPR 2022 | |||
Results (full set)
Full set is the complete testing set of NVS, which contains 50 scenes.
| Methods | PSNR | SSIM | LPIPS |
|---|---|---|---|
| Zip-NeRF | 25.007 | 0.879 | 0.326 |
| Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, Peter Hedman. Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields. ICCV 2023 | |||
| RPBG | 24.355 | 0.873 | 0.280 |
| Zizhuang Wei, Qingtian Zhu, et al. RPBG: Towards Robust Neural Point-based Graphics in the Wild. ECCV 2024 | |||
| FeatSplat | 24.247 | 0.869 | 0.314 |
| Tomas Berriel Martins, Javier Civera. Feature Splatting for Better Novel View Synthesis with Low Overlap. BMVC 2024 | |||
| Nerfacto | 24.049 | 0.861 | 0.342 |
| Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, Angjoo Kanazawa. Nerfstudio: A Modular Framework for Neural Radiance Field Development. SIGGRAPH 2023 | |||
| TensoRF | 23.978 | 0.849 | 0.407 |
| Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, Hao Su. TensoRF: Tensorial Radiance Fields. ECCV 2022 | |||
| Gaussian Splatting | 23.891 | 0.871 | 0.319 |
| Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George Drettakis. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. SIGGRAPH 2023 | |||
| Instant-NGP | 23.812 | 0.859 | 0.375 |
| Thomas Müller, Alex Evans, Christoph Schied, Alexander Keller. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. SIGGRAPH 2022 | |||
| Plenoxels | 22.549 | 0.836 | 0.407 |
| Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, Angjoo Kanazawa. Plenoxels: Radiance Fields without Neural Networks. CVPR 2022 | |||
Please refer to the submission instructions before making a submission
Submit results