Benchmark Results - ScanRefer Benchmark

This table lists the benchmark results for the ScanRefer Localization Benchmark scenario.

		Unique	Unique	Multiple	Multiple	Overall	Overall
Method	Info	acc@0.25IoU	acc@0.5IoU	acc@0.25IoU	acc@0.5IoU	acc@0.25IoU	acc@0.5IoU
Method	Info
UniVLG		0.8895 1	0.8236 1	0.5921 1	0.5030 1	0.6588 1	0.5749 1
Ayush Jain, Alexander Swerdlow, Yuzhou Wang, Alexander Sax, Franziska Meier, Katerina Fragkiadaki: Unifying 2D and 3D Vision-Language Understanding.
Chat-Scene		0.8887 2	0.8005 2	0.5421 2	0.4861 2	0.6198 2	0.5566 2
Haifeng Huang, Yilun Chen, Zehan Wang, et al.: Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers. NeurIPS 2024
ConcreteNet		0.8607 3	0.7923 3	0.4746 9	0.4091 3	0.5612 8	0.4950 3
Ozan Unal, Christos Sakaridis, Suman Saha, Luc Van Gool: Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Grounding. ECCV 2024
cus3d		0.8384 5	0.7073 7	0.4908 7	0.4000 4	0.5688 6	0.4689 4

D-LISA		0.8195 7	0.6900 9	0.4975 5	0.3967 6	0.5697 5	0.4625 5
Haomeng Zhang, Chiao-An Yang, Raymond A. Yeh: Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attention. NeurIPS 2024
M3DRef-test		0.7865 21	0.6793 15	0.4963 6	0.3977 5	0.5614 7	0.4608 6

pointclip		0.8211 6	0.7082 6	0.4803 8	0.3884 7	0.5567 9	0.4601 7

M3DRef-SCLIP		0.7997 14	0.7123 4	0.4708 10	0.3805 10	0.5445 10	0.4549 8

M3DRef-CLIP		0.7980 15	0.7085 5	0.4692 11	0.3807 9	0.5433 11	0.4545 9
Yiming Zhang, ZeMing Gong, Angel X. Chang: Multi3DRefer: Grounding Text Description to Multiple 3D Objects. ICCV 2023
CORE-3DVG		0.8557 4	0.6867 10	0.5275 3	0.3850 8	0.6011 3	0.4527 10

3DInsVG		0.8170 8	0.6925 8	0.4582 14	0.3617 11	0.5386 12	0.4359 11

RG-SAN		0.7964 16	0.6785 16	0.4591 13	0.3600 12	0.5348 15	0.4314 12

3DVLP-baseline		0.7766 32	0.6373 22	0.4572 16	0.3469 13	0.5288 22	0.4120 13

3dvlp-with-judge		0.7807 28	0.6472 20	0.4498 25	0.3407 14	0.5240 25	0.4094 14

Jung		0.8096 11	0.6331 24	0.5113 4	0.3398 16	0.5782 4	0.4055 15

ScanRefer-3dvlp-test		0.7824 26	0.6298 25	0.4532 23	0.3405 15	0.5270 24	0.4054 16

HAM		0.7799 29	0.6373 22	0.4148 33	0.3324 18	0.4967 33	0.4007 17
Jiaming Chen, Weixin Luo, Ran Song, Xiaolin Wei, Lin Ma, Wei Zhang: Learning Point-Language Hierarchical Alignment for 3D Visual Grounding.
CSA-M3LM		0.8137 9	0.6241 26	0.4544 21	0.3317 19	0.5349 14	0.3972 18

3dvlp-judge-h		0.7552 41	0.6051 29	0.4458 27	0.3340 17	0.5152 27	0.3948 19

D3Net		0.7923 19	0.6843 11	0.3905 37	0.3074 31	0.4806 36	0.3919 20
Dave Zhenyu Chen, Qirui Wu, Matthias Niessner, Angel X. Chang: D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding. 17th European Conference on Computer Vision (ECCV), 2022
GALA-Grounder-D3		0.7939 18	0.5952 31	0.4625 12	0.3229 21	0.5368 13	0.3839 21

LAG-3D-2		0.7964 16	0.5812 37	0.4572 16	0.3245 20	0.5333 16	0.3821 22

ContraRefer		0.7832 25	0.6801 14	0.3850 38	0.2947 33	0.4743 37	0.3811 23

LAG-3D-3		0.7815 27	0.5837 35	0.4556 19	0.3219 22	0.5287 23	0.3806 24

Graph-VG-2		0.8021 13	0.5829 36	0.4546 20	0.3217 23	0.5325 17	0.3802 25

Clip		0.7733 35	0.6810 13	0.3619 48	0.2919 38	0.4542 43	0.3791 26

Clip-pre		0.7766 32	0.6843 11	0.3617 50	0.2904 39	0.4547 42	0.3787 27

3DJCG(Grounding)		0.7675 38	0.6059 28	0.4389 28	0.3117 29	0.5126 28	0.3776 28
Daigang Cai, Lichen Zhao, Jing Zhang†, Lu Sheng, Dong Xu: 3DJCG: A Unified Framework for Joint Dense Captioning and Visual Grounding on 3D Point Clouds. CVPR2022 Oral
Graph-VG-3		0.8038 12	0.5812 37	0.4515 24	0.3169 25	0.5305 19	0.3762 29

GALA-Grounder-D1		0.8104 10	0.5754 40	0.4479 26	0.3176 24	0.5292 21	0.3754 30

Graph-VG-4		0.7848 23	0.5631 43	0.4560 18	0.3164 27	0.5298 20	0.3717 31

LAG-3D		0.7881 20	0.5606 44	0.4579 15	0.3169 25	0.5320 18	0.3715 32

3DVG-Trans +		0.7733 35	0.5787 39	0.4370 29	0.3102 30	0.5124 29	0.3704 33
Lichen Zhao∗, Daigang Cai∗, Lu Sheng†, Dong Xu: 3DVG-Transformer: Relation Modeling for Visual Grounding on Point Clouds. ICCV2021
bo3d-1		0.7469 45	0.5606 44	0.4539 22	0.3124 28	0.5196 26	0.3680 34

Se2d		0.7799 29	0.6628 18	0.3636 46	0.2823 41	0.4569 40	0.3677 35

secg		0.7288 48	0.6175 27	0.3696 45	0.2933 35	0.4501 46	0.3660 36

SAF		0.6348 56	0.5647 42	0.3726 43	0.3009 32	0.4314 49	0.3601 37

FE-3DGQA		0.7857 22	0.5862 34	0.4317 30	0.2935 34	0.5111 30	0.3592 38

D3Net - Pretrained		0.7659 39	0.6579 19	0.3619 48	0.2726 42	0.4525 45	0.3590 39
Dave Zhenyu Chen, Qirui Wu, Matthias Niessner, Angel X. Chang: D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding. 17th European Conference on Computer Vision (ECCV), 2022
HGT		0.7692 37	0.5886 33	0.4141 34	0.2924 37	0.4937 34	0.3588 40

InstanceRefer		0.7782 31	0.6669 17	0.3457 53	0.2688 44	0.4427 48	0.3580 41
Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Zhen Li*, Shuguang Cui: InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring. ICCV 2021
3DVG-Transformer		0.7576 40	0.5515 46	0.4224 32	0.2933 35	0.4976 32	0.3512 42
Lichen Zhao∗, Daigang Cai∗, Lu Sheng†, Dong Xu: 3DVG-Transformer: Relation Modeling for Visual Grounding on Point Clouds. ICCV2021
SAVG		0.7758 34	0.5664 41	0.4236 31	0.2826 40	0.5026 31	0.3462 43

PointGroup_MCAN		0.7510 42	0.6397 21	0.3271 56	0.2535 46	0.4222 51	0.3401 44

TransformerVG		0.7502 43	0.5977 30	0.3712 44	0.2628 45	0.4562 41	0.3379 45

TFVG3D ++		0.7453 46	0.5458 49	0.3793 42	0.2690 43	0.4614 38	0.3311 46
Ali Solgi, Mehdi Ezoji: A Transformer-based Framework for Visual Grounding on 3D Point Clouds. AISP 2024
TGNN		0.6834 53	0.5894 32	0.3312 54	0.2526 47	0.4102 55	0.3281 47
Pin-Hao Huang, Han-Hung Lee, Hwann-Tzong Chen, Tyng-Luh Liu: Text-Guided Graph Neural Network for Referring 3D Instance Segmentation. AAAI 2021
BEAUTY-DETR		0.7848 23	0.5499 47	0.3934 36	0.2480 48	0.4811 35	0.3157 48
Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, Katerina Fragkiadaki: Looking Outside the Box to Ground Language in 3D Scenes.
grounding		0.7298 47	0.5458 48	0.3822 40	0.2421 50	0.4538 44	0.3046 49

henet		0.7110 49	0.5180 51	0.3936 35	0.2472 49	0.4590 39	0.3030 50

SRGA		0.7494 44	0.5128 52	0.3631 47	0.2218 51	0.4497 47	0.2871 51

SR-GAB		0.7016 50	0.5202 50	0.3233 58	0.1959 54	0.4081 56	0.2686 52

SPANet		0.5614 60	0.4641 54	0.2800 62	0.2071 53	0.3431 63	0.2647 53

ScanRefer		0.6859 52	0.4353 56	0.3488 52	0.2097 52	0.4244 50	0.2603 54
Dave Zhenyu Chen, Angel X. Chang, Matthias Nießner: ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language. 16th European Conference on Computer Vision (ECCV), 2020
scanrefer2		0.6340 57	0.4353 56	0.3193 59	0.1947 55	0.3898 58	0.2486 55

TransformerRefer		0.6010 58	0.4658 53	0.2540 64	0.1730 60	0.3318 64	0.2386 56

ScanRefer Baseline		0.6422 55	0.4196 58	0.3090 60	0.1832 56	0.3837 59	0.2362 57

ScanRefer-test		0.6999 51	0.4361 55	0.3274 55	0.1725 61	0.4109 54	0.2316 58

ScanRefer_vanilla		0.6488 54	0.4056 59	0.3052 61	0.1782 58	0.3823 60	0.2292 59

pairwisemethod		0.5779 59	0.3603 60	0.2792 63	0.1746 59	0.3462 62	0.2163 60

bo3d		0.5400 61	0.1550 61	0.3817 41	0.1785 57	0.4172 53	0.1732 61

Co3d3		0.5326 62	0.1369 62	0.3848 39	0.1651 62	0.4179 52	0.1588 62

Co3d2		0.5070 63	0.1195 64	0.3569 51	0.1511 63	0.3906 57	0.1440 63

bo3d0		0.4823 64	0.1278 63	0.3271 56	0.1394 64	0.3619 61	0.1368 64

3DVLP		0.0038 65	0.0019 65	0.0049 65	0.0023 65	0.0047 65	0.0022 65

Co3d		0.0000 66	0.0000 66	0.0000 66	0.0000 66	0.0000 66	0.0000 66

		Captioning F1-Score				Dense Captioning	Object Detection
Method	Info	CIDEr@0.5IoU	BLEU-4@0.5IoU	Rouge-L@0.5IoU	METEOR@0.5IoU	DCmAP	mAP@0.5
Method	Info
Chat-Scene-thres0.5		0.3456 1	0.1859 2	0.3162 1	0.1527 1	0.1415 8	0.4856 4
Haifeng Huang, Yilun Chen, Zehan Wang, et al.: Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers. NeurIPS 2024
CM3D-Trans+		0.2348 6	0.1383 6	0.2250 7	0.1030 6	0.1398 9	0.2966 12
Yufeng Zhong, Long Xu, Jiebo Luo, Lin Ma: Contextual Modeling for 3D Dense Captioning on Point Clouds.
Scan2Cap		0.0849 18	0.0576 18	0.1073 18	0.0492 18	0.0970 18	0.2481 18
Dave Zhenyu Chen, Ali Gholami, Matthias Nießner and Angel X. Chang: Scan2Cap: Context-aware Dense Captioning in RGB-D Scans. CVPR 2021
X-Trans2Cap		0.1274 14	0.0808 15	0.1392 15	0.0653 15	0.1244 12	0.2795 16
Yuan, Zhihao and Yan, Xu and Liao, Yinghong and Guo, Yao and Li, Guanbin and Cui, Shuguang and Li, Zhen: X-Trans2Cap: Cross-Modal Knowledge Transfer Using Transformer for 3D Dense Captioning. CVPR 2022
SpaCap3D		0.1359 13	0.0883 13	0.1591 13	0.0738 13	0.1182 14	0.3275 9
Heng Wang, Chaoyi Zhang, Jianhui Yu, Weidong Cai: Spatiality-guided Transformer for 3D Dense Captioning on Point Clouds. the 31st International Joint Conference on Artificial Intelligence (IJCAI), 2022
MORE-xyz		0.1239 16	0.0796 16	0.1362 16	0.0631 16	0.1116 17	0.2648 17
Yang Jiao, Shaoxiang Chen, Zequn Jie, Jingjing Chen, Lin Ma, Yu-Gang Jiang: MORE: Multi_ORder RElation Mining for Dense Captioning in 3D Scenes. ECCV 2022
REMAN		0.1662 11	0.1070 11	0.1790 11	0.0815 11	0.1235 13	0.2927 14

3DJCG(Captioning)		0.1918 10	0.1350 8	0.2207 9	0.1013 9	0.1506 6	0.3867 8
Daigang Cai, Lichen Zhao, Jing Zhang†, Lu Sheng, Dong Xu: 3DJCG: A Unified Framework for Joint Dense Captioning and Visual Grounding on 3D Point Clouds. CVPR2022 Oral
SUN+		0.1148 17	0.0846 14	0.1564 14	0.0711 14	0.1143 16	0.2958 13

Chat-Scene-thres0.01		0.2053 9	0.1103 10	0.1884 10	0.0907 10	0.1527 5	0.5076 2

NOAH		0.1382 12	0.0901 12	0.1598 12	0.0747 12	0.1359 11	0.2977 11

Forest-xyz		0.2266 7	0.1363 7	0.2250 6	0.1027 7	0.1161 15	0.2825 15

CFM		0.2360 5	0.1417 5	0.2253 5	0.1034 5	0.1379 10	0.3008 10

vote2cap-detr		0.3128 3	0.1778 3	0.2842 4	0.1316 4	0.1825 2	0.4454 6
Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Tao Chen, Gang YU, Taihao Li: End-to-End 3D Dense Captioning with Vote2Cap-DETR. CVPR 2023
Vote2Cap-DETR++		0.3360 2	0.1908 1	0.3012 2	0.1386 2	0.1864 1	0.5090 1
Sijin Chen, Hongyuan Zhu, Mingsheng Li, Xin Chen, Peng Guo, Yinjie Lei, Gang Yu, Taihao Li, Tao Chen: Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioning.
TMP		0.3029 4	0.1728 4	0.2898 3	0.1332 3	0.1801 3	0.4605 5

Chat-Scene-all		0.1257 15	0.0671 17	0.1150 17	0.0554 17	0.1539 4	0.5076 2

D3Net - Speaker		0.2088 8	0.1335 9	0.2237 8	0.1022 8	0.1481 7	0.4198 7
Dave Zhenyu Chen, Qirui Wu, Matthias Niessner, Angel X. Chang: D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding. 17th European Conference on Computer Vision (ECCV), 2022

ScanRefer Benchmark

Scan2Cap Benchmark