Benchmark Results - ScanRefer Benchmark

This table lists the benchmark results for the ScanRefer Localization Benchmark scenario.

		Unique	Unique	Multiple	Multiple	Overall	Overall
Method	Info	acc@0.25IoU	acc@0.5IoU	acc@0.25IoU	acc@0.5IoU	acc@0.25IoU	acc@0.5IoU
Method	Info
Chat-Scene		0.8887 1	0.8005 1	0.5421 1	0.4861 1	0.6198 1	0.5566 1
Haifeng Huang, Yilun Chen, Zehan Wang, et al.: Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers. NeurIPS 2024
ConcreteNet		0.8607 2	0.7923 2	0.4746 7	0.4091 2	0.5612 6	0.4950 2
Ozan Unal, Christos Sakaridis, Suman Saha, Luc Van Gool: Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Grounding. ECCV 2024
cus3d		0.8384 4	0.7073 6	0.4908 5	0.4000 3	0.5688 4	0.4689 3

D-LISA		0.8195 6	0.6900 8	0.4975 3	0.3967 5	0.5697 3	0.4625 4
Haomeng Zhang, Chiao-An Yang, Raymond A. Yeh: Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attention. NeurIPS 2024
M3DRef-test		0.7865 15	0.6793 14	0.4963 4	0.3977 4	0.5614 5	0.4608 5

pointclip		0.8211 5	0.7082 5	0.4803 6	0.3884 6	0.5567 7	0.4601 6

M3DRef-SCLIP		0.7997 11	0.7123 3	0.4708 8	0.3805 9	0.5445 8	0.4549 7

M3DRef-CLIP		0.7980 12	0.7085 4	0.4692 9	0.3807 8	0.5433 9	0.4545 8
Yiming Zhang, ZeMing Gong, Angel X. Chang: Multi3DRefer: Grounding Text Description to Multiple 3D Objects. ICCV 2023
CORE-3DVG		0.8557 3	0.6867 9	0.5275 2	0.3850 7	0.6011 2	0.4527 9

3DInsVG		0.8170 7	0.6925 7	0.4582 11	0.3617 10	0.5386 10	0.4359 10

RG-SAN		0.7964 13	0.6785 15	0.4591 10	0.3600 11	0.5348 12	0.4314 11

HAM		0.7799 19	0.6373 20	0.4148 22	0.3324 12	0.4967 22	0.4007 12
Jiaming Chen, Weixin Luo, Ran Song, Xiaolin Wei, Lin Ma, Wei Zhang: Learning Point-Language Hierarchical Alignment for 3D Visual Grounding.
CSA-M3LM		0.8137 8	0.6241 21	0.4544 13	0.3317 13	0.5349 11	0.3972 13

D3Net		0.7923 14	0.6843 10	0.3905 26	0.3074 20	0.4806 25	0.3919 14
Dave Zhenyu Chen, Qirui Wu, Matthias Niessner, Angel X. Chang: D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding. 17th European Conference on Computer Vision (ECCV), 2022
ContraRefer		0.7832 18	0.6801 13	0.3850 27	0.2947 22	0.4743 26	0.3811 15

Graph-VG-2		0.8021 10	0.5829 28	0.4546 12	0.3217 14	0.5325 13	0.3802 16

Clip		0.7733 24	0.6810 12	0.3619 36	0.2919 27	0.4542 31	0.3791 17

Clip-pre		0.7766 22	0.6843 10	0.3617 38	0.2904 28	0.4547 30	0.3787 18

3DJCG(Grounding)		0.7675 27	0.6059 23	0.4389 17	0.3117 18	0.5126 17	0.3776 19
Daigang Cai, Lichen Zhao, Jing Zhang†, Lu Sheng, Dong Xu: 3DJCG: A Unified Framework for Joint Dense Captioning and Visual Grounding on 3D Point Clouds. CVPR2022 Oral
GALA-Grounder + 2D		0.8104 9	0.5754 30	0.4479 16	0.3176 16	0.5292 14	0.3754 20

3DVG-Trans +		0.7733 24	0.5787 29	0.4370 18	0.3102 19	0.5124 18	0.3704 21
Lichen Zhao∗, Daigang Cai∗, Lu Sheng†, Dong Xu: 3DVG-Transformer: Relation Modeling for Visual Grounding on Point Clouds. ICCV2021
GALA-Grounder		0.7659 28	0.5400 37	0.4532 15	0.3190 15	0.5233 15	0.3686 22

bo3d-1		0.7469 34	0.5606 33	0.4539 14	0.3124 17	0.5196 16	0.3680 23

Se2d		0.7799 19	0.6628 17	0.3636 34	0.2823 30	0.4569 28	0.3677 24

secg		0.7288 36	0.6175 22	0.3696 33	0.2933 24	0.4501 34	0.3660 25

SAF		0.6348 43	0.5647 32	0.3726 31	0.3009 21	0.4314 37	0.3601 26

FE-3DGQA		0.7857 16	0.5862 27	0.4317 19	0.2935 23	0.5111 19	0.3592 27

D3Net - Pretrained		0.7659 28	0.6579 18	0.3619 36	0.2726 31	0.4525 33	0.3590 28
Dave Zhenyu Chen, Qirui Wu, Matthias Niessner, Angel X. Chang: D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding. 17th European Conference on Computer Vision (ECCV), 2022
HGT		0.7692 26	0.5886 26	0.4141 23	0.2924 26	0.4937 23	0.3588 29

InstanceRefer		0.7782 21	0.6669 16	0.3457 41	0.2688 32	0.4427 36	0.3580 30
Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Zhen Li*, Shuguang Cui: InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring. ICCV 2021
3DVG-Transformer		0.7576 30	0.5515 34	0.4224 21	0.2933 24	0.4976 21	0.3512 31
Lichen Zhao∗, Daigang Cai∗, Lu Sheng†, Dong Xu: 3DVG-Transformer: Relation Modeling for Visual Grounding on Point Clouds. ICCV2021
SAVG		0.7758 23	0.5664 31	0.4236 20	0.2826 29	0.5026 20	0.3462 32

PointGroup_MCAN		0.7510 31	0.6397 19	0.3271 43	0.2535 34	0.4222 39	0.3401 33

TransformerVG		0.7502 32	0.5977 24	0.3712 32	0.2628 33	0.4562 29	0.3379 34

TGNN		0.6834 40	0.5894 25	0.3312 42	0.2526 35	0.4102 42	0.3281 35
Pin-Hao Huang, Han-Hung Lee, Hwann-Tzong Chen, Tyng-Luh Liu: Text-Guided Graph Neural Network for Referring 3D Instance Segmentation. AAAI 2021
BEAUTY-DETR		0.7848 17	0.5499 35	0.3934 25	0.2480 36	0.4811 24	0.3157 36
Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, Katerina Fragkiadaki: Looking Outside the Box to Ground Language in 3D Scenes.
grounding		0.7298 35	0.5458 36	0.3822 29	0.2421 38	0.4538 32	0.3046 37

henet		0.7110 37	0.5180 39	0.3936 24	0.2472 37	0.4590 27	0.3030 38

SRGA		0.7494 33	0.5128 40	0.3631 35	0.2218 39	0.4497 35	0.2871 39

SR-GAB		0.7016 38	0.5202 38	0.3233 45	0.1959 42	0.4081 43	0.2686 40

SPANet		0.5614 47	0.4641 42	0.2800 49	0.2071 41	0.3431 50	0.2647 41

ScanRefer		0.6859 39	0.4353 43	0.3488 40	0.2097 40	0.4244 38	0.2603 42
Dave Zhenyu Chen, Angel X. Chang, Matthias Nießner: ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language. 16th European Conference on Computer Vision (ECCV), 2020
scanrefer2		0.6340 44	0.4353 43	0.3193 46	0.1947 43	0.3898 45	0.2486 43

TransformerRefer		0.6010 45	0.4658 41	0.2540 51	0.1730 48	0.3318 51	0.2386 44

ScanRefer Baseline		0.6422 42	0.4196 45	0.3090 47	0.1832 44	0.3837 46	0.2362 45

ScanRefer_vanilla		0.6488 41	0.4056 46	0.3052 48	0.1782 46	0.3823 47	0.2292 46

pairwisemethod		0.5779 46	0.3603 47	0.2792 50	0.1746 47	0.3462 49	0.2163 47

bo3d		0.5400 48	0.1550 48	0.3817 30	0.1785 45	0.4172 41	0.1732 48

Co3d3		0.5326 49	0.1369 49	0.3848 28	0.1651 49	0.4179 40	0.1588 49

Co3d2		0.5070 50	0.1195 51	0.3569 39	0.1511 50	0.3906 44	0.1440 50

bo3d0		0.4823 51	0.1278 50	0.3271 43	0.1394 51	0.3619 48	0.1368 51

Co3d		0.0000 52	0.0000 52	0.0000 52	0.0000 52	0.0000 52	0.0000 52

		Captioning F1-Score				Dense Captioning	Object Detection
Method	Info	CIDEr@0.5IoU	BLEU-4@0.5IoU	Rouge-L@0.5IoU	METEOR@0.5IoU	DCmAP	mAP@0.5
Method	Info
Chat-Scene-thres0.5		0.3456 1	0.1859 2	0.3162 1	0.1527 1	0.1415 8	0.4856 4
Haifeng Huang, Yilun Chen, Zehan Wang, et al.: Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers. NeurIPS 2024
Vote2Cap-DETR++		0.3360 2	0.1908 1	0.3012 2	0.1386 2	0.1864 1	0.5090 1
Sijin Chen, Hongyuan Zhu, Mingsheng Li, Xin Chen, Peng Guo, Yinjie Lei, Gang Yu, Taihao Li, Tao Chen: Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioning.
vote2cap-detr		0.3128 3	0.1778 3	0.2842 4	0.1316 4	0.1825 2	0.4454 6
Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Tao Chen, Gang YU, Taihao Li: End-to-End 3D Dense Captioning with Vote2Cap-DETR. CVPR 2023
TMP		0.3029 4	0.1728 4	0.2898 3	0.1332 3	0.1801 3	0.4605 5

CFM		0.2360 5	0.1417 5	0.2253 5	0.1034 5	0.1379 10	0.3008 10

CM3D-Trans+		0.2348 6	0.1383 6	0.2250 7	0.1030 6	0.1398 9	0.2966 12
Yufeng Zhong, Long Xu, Jiebo Luo, Lin Ma: Contextual Modeling for 3D Dense Captioning on Point Clouds.
Forest-xyz		0.2266 7	0.1363 7	0.2250 6	0.1027 7	0.1161 15	0.2825 15

D3Net - Speaker		0.2088 8	0.1335 9	0.2237 8	0.1022 8	0.1481 7	0.4198 7
Dave Zhenyu Chen, Qirui Wu, Matthias Niessner, Angel X. Chang: D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding. 17th European Conference on Computer Vision (ECCV), 2022
Chat-Scene-thres0.01		0.2053 9	0.1103 10	0.1884 10	0.0907 10	0.1527 5	0.5076 2

3DJCG(Captioning)		0.1918 10	0.1350 8	0.2207 9	0.1013 9	0.1506 6	0.3867 8
Daigang Cai, Lichen Zhao, Jing Zhang†, Lu Sheng, Dong Xu: 3DJCG: A Unified Framework for Joint Dense Captioning and Visual Grounding on 3D Point Clouds. CVPR2022 Oral
REMAN		0.1662 11	0.1070 11	0.1790 11	0.0815 11	0.1235 13	0.2927 14

NOAH		0.1382 12	0.0901 12	0.1598 12	0.0747 12	0.1359 11	0.2977 11

SpaCap3D		0.1359 13	0.0883 13	0.1591 13	0.0738 13	0.1182 14	0.3275 9
Heng Wang, Chaoyi Zhang, Jianhui Yu, Weidong Cai: Spatiality-guided Transformer for 3D Dense Captioning on Point Clouds. the 31st International Joint Conference on Artificial Intelligence (IJCAI), 2022
X-Trans2Cap		0.1274 14	0.0808 15	0.1392 15	0.0653 15	0.1244 12	0.2795 16
Yuan, Zhihao and Yan, Xu and Liao, Yinghong and Guo, Yao and Li, Guanbin and Cui, Shuguang and Li, Zhen: X-Trans2Cap: Cross-Modal Knowledge Transfer Using Transformer for 3D Dense Captioning. CVPR 2022
Chat-Scene-all		0.1257 15	0.0671 17	0.1150 17	0.0554 17	0.1539 4	0.5076 2

MORE-xyz		0.1239 16	0.0796 16	0.1362 16	0.0631 16	0.1116 17	0.2648 17
Yang Jiao, Shaoxiang Chen, Zequn Jie, Jingjing Chen, Lin Ma, Yu-Gang Jiang: MORE: Multi_ORder RElation Mining for Dense Captioning in 3D Scenes. ECCV 2022
SUN+		0.1148 17	0.0846 14	0.1564 14	0.0711 14	0.1143 16	0.2958 13

Scan2Cap		0.0849 18	0.0576 18	0.1073 18	0.0492 18	0.0970 18	0.2481 18
Dave Zhenyu Chen, Ali Gholami, Matthias Nießner and Angel X. Chang: Scan2Cap: Context-aware Dense Captioning in RGB-D Scans. CVPR 2021

ScanRefer Benchmark

Scan2Cap Benchmark