VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generation

Lim, Hyeonseok; Shin, Dongjae; Song, Seohyun; Won, Inho; Kim, Minjun; Yuk, Junghun; Jang, Haneol; Lim, KyungTae

A multilingual (English, Chinese, Korean) VQA benchmark for retrieval-augmented vision-language models, with gold, silver, and bronze passages

COLING 2025

Hyeonseok Lim^1,*, Dongjae Shin^1,2,*, Seohyun Song¹, Inho Won^1,2, Minjun Kim¹, Junghun Yuk³, Haneol Jang^3,†, KyungTae Lim^1,†

¹Seoul National University of Science and Technology (SeoulTech) ²Teddysum ³Hanbat National University

^*Equal contribution ^†Corresponding authors

Abstract

We propose the VLR-Bench, a visual question answering (VQA) benchmark for evaluating vision language models (VLMs) based on retrieval augmented generation (RAG). Unlike existing evaluation datasets for external knowledge-based VQA, the proposed VLR-Bench includes five input passages. This allows testing of the ability to determine which passage is useful for answering a given query, a capability lacking in previous research. In this context, we constructed a dataset of 32,000 automatically generated instruction-following examples, which we denote as VLR-IF. This dataset is specifically designed to enhance the RAG capabilities of VLMs by enabling them to learn how to generate appropriate answers based on input passages. We evaluated the validity of the proposed benchmark and training data and verified its performance using the state-of-the-art Llama3-based VLM, the Llava-Llama-3 model. The proposed VLR-Bench and VLR-IF datasets are publicly available online.

Keywords Vision-language RAG Multimodal retrieval-augmented generation Knowledge-based VQA Vision-language models (VLMs) Multilingual benchmark Korean Chinese Instruction tuning LLaVA

Benchmark Construction

Three-step construction diagram. Step 1: an image sampled from BOK-VQA plus few-shot samples go to GPT-4o, which generates 10 question-answer-keywords-gold passage candidates. Step 2: annotators accept the best candidate or reject and return to step 1. Step 3: GPT-4o generates two silver passages and one bronze passage, followed by review and revision into the final data.

Figure 6. Overview of the VLR-Bench dataset construction process (Appendix A.2 of the paper).

Real RAG systems feed several retrieved passages (e.g. top-5) to the model, some of which look relevant but are wrong. VLR-Bench therefore tests both choosing the right external knowledge and generating a correct answer despite distractors.

Images. 150 images manually selected from the 10 categories of BOK-VQA (general knowledge) and 150 images of English, Chinese, and Korean cultural backgrounds from Wikimedia Commons, giving a parallel corpus of 300 samples.
Queries and gold passages. GPT-4o proposes ten query–knowledge–answer candidates per image; queries must be unanswerable from the image alone and must not reveal the image content. The best candidate is kept after review.
Distractors and review. GPT-4o adds two silver passages and one bronze passage; three annotators check every sample, replace erroneous or unsourced knowledge, and mark two answer keywords.
VLR-IF training data. From 9K COCO images, a valid passage is generated for each image and invalid passages are taken from other samples, producing contrastive instruction-following data: 32K examples per language for English, Chinese, and Korean.

Long-form answers are scored with the keyword-matching score (KMS: correct only if both keywords appear), ROUGE-2, ROUGE-L, BLEU, and BERTScore. Because KMS rises with answer length, it is used as a reference indicator.

Main Results

VLR-IF helps models use passages. Training LLaVA-Llama-3 on VLR-IF improves it by 22.67% over the baseline when passages are given (e.g. ROUGE-L 38.8 → 46.4, BLEU 13.5 → 20.9 in English). On Korean, X-LLaVA + VLR-IF (Ko) rises from 27.0 to 35.7 ROUGE-2 and from 15.2 to 24.9 BLEU.
External knowledge is required. Without the passages, X-LLaVA's English ROUGE-2 drops by 37.72% on average.
Passages narrow the gap to GPT-4o. Without passages GPT-4o beats Qwen-VL-Chat by 17.33 points on average; with passages the gap shrinks to 7.36 points.
KMS alone can mislead. LLaVA-1.5 has the top English KMS (88.4) by copying passages into long answers, but low ROUGE-L (37.2) and BLEU (14.7).
Transfers to InfoSeek. VLR-IF training also gives a 3.6% improvement on the InfoSeek benchmark.

**Table 1.** Overall results on VLR-Bench by language. The VLR-IF columns mark which language subsets of VLR-IF the model was trained on (highlighted rows). KMS = keyword-matching score, R = ROUGE, B-Score = BERTScore. Bold marks the best score within each language.
Lang.	Model	VLR-IF			With passages					Without passages
Lang.	Model	EN	ZH	KO	KMS	R-2	R-L	BLEU	B-Score	KMS	R-2	R-L	BLEU	B-Score
EN	LLaVA-1.5	✗	✗	✗	88.4	26.0	37.2	14.7	76.3	25.6	17.8	30.4	9.1	73.4
	LLaVA-Llama-3	✗	✗	✗	79.2	25.4	38.8	13.5	79.3	20.4	12.2	23.8	6.1	73.4
	LLaVA-Llama-3 + VLR-IF (En)	✓	✗	✗	85.6	30.1	46.4	20.9	81.5	20.4	19.1	29.9	8.6	69.1
	X-LLaVA	✗	✗	✗	80.4	28.1	42.2	16.9	80.1	20.8	17.5	31.9	9.6	74.3
	X-LLaVA + VLR-IF (En)	✓	✗	✗	82.4	29.4	44.2	20.1	80.7	20.4	19.7	35.4	12.7	77.5
	X-LLaVA + VLR-IF (En+Ko)	✓	✗	✓	83.2	30.2	45.2	20.6	81.0	18.4	20.4	36.3	14.5	77.1
	Qwen-VL-Chat	✗	✗	✗	84.8	32.8	47.4	20.6	82.1	31.2	20.0	34.1	9.7	77.5
	GPT-4o	✗	✗	✗	85.6	42.6	57.9	32.8	85.6	61.6	35.6	52.1	26.2	83.7
ZH	Qwen-VL-Chat	✗	✗	✗	75.6	51.6	56.3	33.8	84.0	10.8	28.9	37.2	18.2	75.4
	Qwen-VL-Chat + VLR-IF (Zh)	✗	✓	✗	72.4	59.0	63.4	42.9	86.2	16.0	30.4	37.8	18.1	77.4
	GPT-4o	✗	✗	✗	80.4	56.9	62.3	41.6	86.2	36.0	36.6	42.9	24.6	80.3
KO	X-LLaVA	✗	✗	✗	59.6	27.0	35.2	15.2	78.4	6.0	18.0	28.0	8.6	74.2
	X-LLaVA + VLR-IF (Ko)	✗	✗	✓	63.6	35.7	44.1	24.9	81.0	6.8	22.4	32.9	14.9	77.0
	X-LLaVA + VLR-IF (En+Ko)	✓	✗	✓	62.4	36.0	44.6	24.2	81.7	0.8	4.7	15.2	5.14	64.5
	GPT-4o	✗	✗	✗	83.6	51.9	55.2	37.2	84.4	31.6	35.9	39.0	24.9	79.7

@inproceedings{lim-etal-2025-vlr, title = "{VLR}-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generation", author = "Lim, Hyeonseok and Shin, Dongjae and Song, Seohyun and Won, Inho and Kim, Minjun and Yuk, Junghun and Jang, Haneol and Lim, KyungTae", editor = "Rambow, Owen and Wanner, Leo and Apidianaki, Marianna and Al-Khalifa, Hend and Eugenio, Barbara Di and Schockaert, Steven", booktitle = "Proceedings of the 31st International Conference on Computational Linguistics", month = jan, year = "2025", address = "Abu Dhabi, UAE", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2025.coling-main.411/", pages = "6150--6168" }

VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generation

Abstract

Benchmark Construction

Main Results

BibTeX