Benchmark Results - ScanRefer Benchmark

This table lists the benchmark results for the ScanRefer Localization Benchmark scenario.

		Unique	Unique	Multiple	Multiple	Overall	Overall
Method	Info	acc@0.25IoU	acc@0.5IoU	acc@0.25IoU	acc@0.5IoU	acc@0.25IoU	acc@0.5IoU
Method	Info
3DVLP-baseline		0.7766 31	0.6373 22	0.4572 15	0.3469 13	0.5288 21	0.4120 13

TransformerVG		0.7502 42	0.5977 29	0.3712 43	0.2628 44	0.4562 40	0.3379 44

ConcreteNet		0.8607 3	0.7923 3	0.4746 8	0.4091 3	0.5612 7	0.4950 3
Ozan Unal, Christos Sakaridis, Suman Saha, Luc Van Gool: Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Grounding. ECCV 2024
scanrefer2		0.6340 56	0.4353 55	0.3193 58	0.1947 54	0.3898 57	0.2486 54

CSA-M3LM		0.8137 9	0.6241 25	0.4544 20	0.3317 18	0.5349 13	0.3972 17

ScanRefer_vanilla		0.6488 53	0.4056 58	0.3052 60	0.1782 57	0.3823 59	0.2292 58

HAM		0.7799 28	0.6373 22	0.4148 32	0.3324 17	0.4967 32	0.4007 16
Jiaming Chen, Weixin Luo, Ran Song, Xiaolin Wei, Lin Ma, Wei Zhang: Learning Point-Language Hierarchical Alignment for 3D Visual Grounding.
SPANet		0.5614 59	0.4641 53	0.2800 61	0.2071 52	0.3431 62	0.2647 52

henet		0.7110 48	0.5180 50	0.3936 34	0.2472 48	0.4590 38	0.3030 49

grounding		0.7298 46	0.5458 47	0.3822 39	0.2421 49	0.4538 43	0.3046 48

SAVG		0.7758 33	0.5664 40	0.4236 30	0.2826 39	0.5026 30	0.3462 42

HGT		0.7692 36	0.5886 32	0.4141 33	0.2924 36	0.4937 33	0.3588 39

BEAUTY-DETR		0.7848 22	0.5499 46	0.3934 35	0.2480 47	0.4811 34	0.3157 47
Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, Katerina Fragkiadaki: Looking Outside the Box to Ground Language in 3D Scenes.
Clip-pre		0.7766 31	0.6843 11	0.3617 49	0.2904 38	0.4547 41	0.3787 26

Clip		0.7733 34	0.6810 13	0.3619 47	0.2919 37	0.4542 42	0.3791 25

FE-3DGQA		0.7857 21	0.5862 33	0.4317 29	0.2935 33	0.5111 29	0.3592 37

3DInsVG		0.8170 8	0.6925 8	0.4582 13	0.3617 11	0.5386 11	0.4359 11

ContraRefer		0.7832 24	0.6801 14	0.3850 37	0.2947 32	0.4743 36	0.3811 22

D3Net		0.7923 18	0.6843 11	0.3905 36	0.3074 30	0.4806 35	0.3919 19
Dave Zhenyu Chen, Qirui Wu, Matthias Niessner, Angel X. Chang: D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding. 17th European Conference on Computer Vision (ECCV), 2022
D3Net - Pretrained		0.7659 38	0.6579 19	0.3619 47	0.2726 41	0.4525 44	0.3590 38
Dave Zhenyu Chen, Qirui Wu, Matthias Niessner, Angel X. Chang: D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding. 17th European Conference on Computer Vision (ECCV), 2022
TransformerRefer		0.6010 57	0.4658 52	0.2540 63	0.1730 59	0.3318 63	0.2386 55

SR-GAB		0.7016 49	0.5202 49	0.3233 57	0.1959 53	0.4081 55	0.2686 51

pairwisemethod		0.5779 58	0.3603 59	0.2792 62	0.1746 58	0.3462 61	0.2163 59

PointGroup_MCAN		0.7510 41	0.6397 21	0.3271 55	0.2535 45	0.4222 50	0.3401 43

3DJCG(Grounding)		0.7675 37	0.6059 27	0.4389 27	0.3117 28	0.5126 27	0.3776 27
Daigang Cai, Lichen Zhao, Jing Zhang†, Lu Sheng, Dong Xu: 3DJCG: A Unified Framework for Joint Dense Captioning and Visual Grounding on 3D Point Clouds. CVPR2022 Oral
3DVG-Transformer		0.7576 39	0.5515 45	0.4224 31	0.2933 34	0.4976 31	0.3512 41
Lichen Zhao∗, Daigang Cai∗, Lu Sheng†, Dong Xu: 3DVG-Transformer: Relation Modeling for Visual Grounding on Point Clouds. ICCV2021
3DVG-Trans +		0.7733 34	0.5787 38	0.4370 28	0.3102 29	0.5124 28	0.3704 32
Lichen Zhao∗, Daigang Cai∗, Lu Sheng†, Dong Xu: 3DVG-Transformer: Relation Modeling for Visual Grounding on Point Clouds. ICCV2021
SRGA		0.7494 43	0.5128 51	0.3631 46	0.2218 50	0.4497 46	0.2871 50

InstanceRefer		0.7782 30	0.6669 17	0.3457 52	0.2688 43	0.4427 47	0.3580 40
Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Zhen Li*, Shuguang Cui: InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring. ICCV 2021
ScanRefer Baseline		0.6422 54	0.4196 57	0.3090 59	0.1832 55	0.3837 58	0.2362 56

TGNN		0.6834 52	0.5894 31	0.3312 53	0.2526 46	0.4102 54	0.3281 46
Pin-Hao Huang, Han-Hung Lee, Hwann-Tzong Chen, Tyng-Luh Liu: Text-Guided Graph Neural Network for Referring 3D Instance Segmentation. AAAI 2021
M3DRef-CLIP		0.7980 14	0.7085 5	0.4692 10	0.3807 9	0.5433 10	0.4545 9
Yiming Zhang, ZeMing Gong, Angel X. Chang: Multi3DRefer: Grounding Text Description to Multiple 3D Objects. ICCV 2023
Co3d		0.0000 65	0.0000 65	0.0000 65	0.0000 65	0.0000 65	0.0000 65

3dvlp-with-judge		0.7807 27	0.6472 20	0.4498 24	0.3407 14	0.5240 24	0.4094 14

TFVG3D ++		0.7453 45	0.5458 48	0.3793 41	0.2690 42	0.4614 37	0.3311 45
Ali Solgi, Mehdi Ezoji: A Transformer-based Framework for Visual Grounding on 3D Point Clouds. AISP 2024
3dvlp-judge-h		0.7552 40	0.6051 28	0.4458 26	0.3340 16	0.5152 26	0.3948 18

ScanRefer-3dvlp-test		0.7824 25	0.6298 24	0.4532 22	0.3405 15	0.5270 23	0.4054 15

ScanRefer-test		0.6999 50	0.4361 54	0.3274 54	0.1725 60	0.4109 53	0.2316 57

3DVLP		0.0038 64	0.0019 64	0.0049 64	0.0023 64	0.0047 64	0.0022 64

UniVLG		0.8895 1	0.8236 1	0.5921 1	0.5030 1	0.6588 1	0.5749 1
Ayush Jain, Alexander Swerdlow, Yuzhou Wang, Alexander Sax, Franziska Meier, Katerina Fragkiadaki: Unifying 2D and 3D Vision-Language Understanding.
GALA-Grounder-D3		0.7939 17	0.5952 30	0.4625 11	0.3229 20	0.5368 12	0.3839 20

LAG-3D-3		0.7815 26	0.5837 34	0.4556 18	0.3219 21	0.5287 22	0.3806 23

LAG-3D-2		0.7964 15	0.5812 36	0.4572 15	0.3245 19	0.5333 15	0.3821 21

LAG-3D		0.7881 19	0.5606 43	0.4579 14	0.3169 24	0.5320 17	0.3715 31

Graph-VG-4		0.7848 22	0.5631 42	0.4560 17	0.3164 26	0.5298 19	0.3717 30

Graph-VG-3		0.8038 11	0.5812 36	0.4515 23	0.3169 24	0.5305 18	0.3762 28

Graph-VG-2		0.8021 12	0.5829 35	0.4546 19	0.3217 22	0.5325 16	0.3802 24

GALA-Grounder-D1		0.8104 10	0.5754 39	0.4479 25	0.3176 23	0.5292 20	0.3754 29

Chat-Scene		0.8887 2	0.8005 2	0.5421 2	0.4861 2	0.6198 2	0.5566 2
Haifeng Huang, Yilun Chen, Zehan Wang, et al.: Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers. NeurIPS 2024
Co3d2		0.5070 62	0.1195 63	0.3569 50	0.1511 62	0.3906 56	0.1440 62

D-LISA		0.8195 7	0.6900 9	0.4975 4	0.3967 6	0.5697 4	0.4625 5
Haomeng Zhang, Chiao-An Yang, Raymond A. Yeh: Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attention. NeurIPS 2024
M3DRef-test		0.7865 20	0.6793 15	0.4963 5	0.3977 5	0.5614 6	0.4608 6

RG-SAN		0.7964 15	0.6785 16	0.4591 12	0.3600 12	0.5348 14	0.4314 12

SAF		0.6348 55	0.5647 41	0.3726 42	0.3009 31	0.4314 48	0.3601 36

M3DRef-SCLIP		0.7997 13	0.7123 4	0.4708 9	0.3805 10	0.5445 9	0.4549 8

cus3d		0.8384 5	0.7073 7	0.4908 6	0.4000 4	0.5688 5	0.4689 4

pointclip		0.8211 6	0.7082 6	0.4803 7	0.3884 7	0.5567 8	0.4601 7

Se2d		0.7799 28	0.6628 18	0.3636 45	0.2823 40	0.4569 39	0.3677 34

secg		0.7288 47	0.6175 26	0.3696 44	0.2933 34	0.4501 45	0.3660 35

CORE-3DVG		0.8557 4	0.6867 10	0.5275 3	0.3850 8	0.6011 3	0.4527 10

bo3d-1		0.7469 44	0.5606 43	0.4539 21	0.3124 27	0.5196 25	0.3680 33

Co3d3		0.5326 61	0.1369 61	0.3848 38	0.1651 61	0.4179 51	0.1588 61

bo3d0		0.4823 63	0.1278 62	0.3271 55	0.1394 63	0.3619 60	0.1368 63

bo3d		0.5400 60	0.1550 60	0.3817 40	0.1785 56	0.4172 52	0.1732 60

ScanRefer		0.6859 51	0.4353 55	0.3488 51	0.2097 51	0.4244 49	0.2603 53
Dave Zhenyu Chen, Angel X. Chang, Matthias Nießner: ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language. 16th European Conference on Computer Vision (ECCV), 2020

		Captioning F1-Score				Dense Captioning	Object Detection
Method	Info	CIDEr@0.5IoU	BLEU-4@0.5IoU	Rouge-L@0.5IoU	METEOR@0.5IoU	DCmAP	mAP@0.5
Method	Info
Vote2Cap-DETR++		0.3360 2	0.1908 1	0.3012 2	0.1386 2	0.1864 1	0.5090 1
Sijin Chen, Hongyuan Zhu, Mingsheng Li, Xin Chen, Peng Guo, Yinjie Lei, Gang Yu, Taihao Li, Tao Chen: Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioning.
Chat-Scene-thres0.5		0.3456 1	0.1859 2	0.3162 1	0.1527 1	0.1415 8	0.4856 4
Haifeng Huang, Yilun Chen, Zehan Wang, et al.: Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers. NeurIPS 2024
vote2cap-detr		0.3128 3	0.1778 3	0.2842 4	0.1316 4	0.1825 2	0.4454 6
Sijin Chen, Hongyuan Zhu, Xin Chen, Yinjie Lei, Tao Chen, Gang YU, Taihao Li: End-to-End 3D Dense Captioning with Vote2Cap-DETR. CVPR 2023
TMP		0.3029 4	0.1728 4	0.2898 3	0.1332 3	0.1801 3	0.4605 5

CFM		0.2360 5	0.1417 5	0.2253 5	0.1034 5	0.1379 10	0.3008 10

CM3D-Trans+		0.2348 6	0.1383 6	0.2250 7	0.1030 6	0.1398 9	0.2966 12
Yufeng Zhong, Long Xu, Jiebo Luo, Lin Ma: Contextual Modeling for 3D Dense Captioning on Point Clouds.
Forest-xyz		0.2266 7	0.1363 7	0.2250 6	0.1027 7	0.1161 15	0.2825 15

3DJCG(Captioning)		0.1918 10	0.1350 8	0.2207 9	0.1013 9	0.1506 6	0.3867 8
Daigang Cai, Lichen Zhao, Jing Zhang†, Lu Sheng, Dong Xu: 3DJCG: A Unified Framework for Joint Dense Captioning and Visual Grounding on 3D Point Clouds. CVPR2022 Oral
D3Net - Speaker		0.2088 8	0.1335 9	0.2237 8	0.1022 8	0.1481 7	0.4198 7
Dave Zhenyu Chen, Qirui Wu, Matthias Niessner, Angel X. Chang: D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding. 17th European Conference on Computer Vision (ECCV), 2022
Chat-Scene-thres0.01		0.2053 9	0.1103 10	0.1884 10	0.0907 10	0.1527 5	0.5076 2

REMAN		0.1662 11	0.1070 11	0.1790 11	0.0815 11	0.1235 13	0.2927 14

NOAH		0.1382 12	0.0901 12	0.1598 12	0.0747 12	0.1359 11	0.2977 11

SpaCap3D		0.1359 13	0.0883 13	0.1591 13	0.0738 13	0.1182 14	0.3275 9
Heng Wang, Chaoyi Zhang, Jianhui Yu, Weidong Cai: Spatiality-guided Transformer for 3D Dense Captioning on Point Clouds. the 31st International Joint Conference on Artificial Intelligence (IJCAI), 2022
SUN+		0.1148 17	0.0846 14	0.1564 14	0.0711 14	0.1143 16	0.2958 13

X-Trans2Cap		0.1274 14	0.0808 15	0.1392 15	0.0653 15	0.1244 12	0.2795 16
Yuan, Zhihao and Yan, Xu and Liao, Yinghong and Guo, Yao and Li, Guanbin and Cui, Shuguang and Li, Zhen: X-Trans2Cap: Cross-Modal Knowledge Transfer Using Transformer for 3D Dense Captioning. CVPR 2022
MORE-xyz		0.1239 16	0.0796 16	0.1362 16	0.0631 16	0.1116 17	0.2648 17
Yang Jiao, Shaoxiang Chen, Zequn Jie, Jingjing Chen, Lin Ma, Yu-Gang Jiang: MORE: Multi_ORder RElation Mining for Dense Captioning in 3D Scenes. ECCV 2022
Chat-Scene-all		0.1257 15	0.0671 17	0.1150 17	0.0554 17	0.1539 4	0.5076 2

Scan2Cap		0.0849 18	0.0576 18	0.1073 18	0.0492 18	0.0970 18	0.2481 18
Dave Zhenyu Chen, Ali Gholami, Matthias Nießner and Angel X. Chang: Scan2Cap: Context-aware Dense Captioning in RGB-D Scans. CVPR 2021

ScanRefer Benchmark

Scan2Cap Benchmark