X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment

A Korean-English multimodal LLM and a 91K English-Korean-Chinese visual instruction dataset built with GPT-4V

Findings of NAACL 2024

DongJae Shin1,2,*, HyeonSeok Lim1,*, Inho Won1,2, ChangSu Choi1, Minjun Kim1, SeungWoo Song1, HanGyeol Yoo1, SangMin Kim1, KyungTae Lim1,†

1Seoul National University of Science and Technology 2Teddysum

*These authors contributed equally   †Corresponding author

Paper (ACL Anthology) PDF arXiv
Left: an input image of a bus parking lot with double-decker buses, and the prompt given to GPT-4V (system message, two in-context samples with image, scene graph, questions and answers, and the main objects sign, parking lot, lamp, man, bus). Right: GPT-4V output for the object-centric case, a question and a detailed answer about the buses, a van and people, each written in English, Korean and Chinese.
Figure 2 (paper). Building the multilingual visual instruction-following dataset (mvif). GPT-4V receives an image, its main objects, a system message and two in-context samples, and generates questions and answers in English, Korean and Chinese. Four data types are produced (object-centric, location-centric, atmosphere-centric and 8-turn conversation), 91K items in total from 23,496 Visual Genome images, for about $3,200.

Abstract

The impressive development of large language models (LLMs) is expanding into the realm of large multimodal models (LMMs), which incorporate multiple types of data beyond text. However, the nature of multimodal models leads to significant expenses in the creation of training data. Furthermore, constructing multilingual data for LMMs presents its own set of challenges due to language diversity and complexity. Therefore, in this study, we propose two cost-effective methods to solve this problem: (1) vocabulary expansion and pretraining of multilingual LLM for specific languages, and (2) automatic and elaborate construction of multimodal datasets using GPT4-V. Based on these methods, we constructed a 91K English-Korean-Chinese multilingual, multimodal training dataset. Additionally, we developed a bilingual multimodal model that exhibits excellent performance in both Korean and English, surpassing existing approaches.

Keywords Large multimodal models (LMM) Vision-language models Bilingual LMM Korean VQA Visual instruction tuning Multilingual visual instruction dataset GPT-4V data generation Vocabulary expansion LLaVA1.5

Method

Left (a): LLaVA1.5 architecture, where an input image passes through a visual encoder H and a projection P, a tokenized question passes through the word embedding layer G, and the two representations are concatenated and fed to the language model F. Right: (b) vocabulary expansion of LLaMA2 by adding the KoBERT vocabulary to the LLaMA2 vocabulary in the embedding layer, and (c) multilingual pretraining on English and Korean pretraining data.
Figure 3 (paper). (a) The LLaVA1.5 architecture that X-LLaVA keeps: a CLIP visual encoder H, a projection P into the word-embedding space (576×1024 → 576×5120), and the LLaMA2-13B language model F. (b, c) The proposed language-model preparation: KoBERT vocabulary added to LLaMA2 and further pretraining on English and Korean Wikipedia.
  1. Vocabulary expansion. Korean is only 0.37% of the LLaMA2 tokenizer (89.7% Latin script), which causes English answers or code-switching to Korean queries. 7,478 KoBERT tokens are added (39,478 entries in total) with randomly initialized embeddings.
  2. Bilingual pretraining. The expanded LLaMA2 is further trained with causal language modeling on English and Korean Wikipedia (7.8 GB) to link knowledge across the two languages.
  3. Stage 1: image-text alignment. Only the projection layer is trained, on LLaVA-CC3M captions and their machine-translated Korean counterpart, learning English and Korean simultaneously.
  4. Stage 2: multilingual visual instruction tuning. The projection and language model are trained on the LLaVA instruct data (158K), its Korean translation (158K), and the new mvif dataset (91K).

The whole pipeline trains in 7.5 days on a single A6000 GPU. Evaluation covers Korean (BVQAk, KoViz, KoLiv) and English (BVQAe, VQA2.0, GQA, LLaVA-in-the-wild (LV), POPE) benchmarks, plus GPT-4V and human preference judgments.

Main Results

  • Best Korean performance. X-LLaVA averages 57.0 on the three Korean benchmarks (BVQAk 57.9, KoViz 51.3, KoLiv 61.7), about 5.2 points higher than KoLLaVA.
  • English is not sacrificed. X-LLaVA has the best English BVQAe (34.3) and GQA (64.0) scores, 8.2 and 0.7 points above LLaVA1.5-13B; LLaVA1.5 is 4.5 points higher on VQA2.0.
  • Vocabulary expansion and pretraining. Compared with the variant without them, X-LLaVA gains 6.1 points on average in Korean and 0.8 points in English; adding mvif data (+O) improves scores by 0.2 to 3 points.
  • Preference evaluation. In Korean GPT-4V preference tests, X-LLaVA’s answers are preferred over BLIP-2 in 93% and over InstructBLIP in 83% of comparisons; in English it receives about 25% higher preference than LLaVA1.5.
  • Low cost. The 91K mvif data cost about $3,200 to generate with GPT-4V.
Table 1. Extrinsic evaluation on Korean (BVQAk, KoViz, KoLiv) and English (BVQAe, VQA2.0, GQA, LV, POPE) benchmarks. #PT / #VIT: number of pretraining / visual instruction tuning examples; (O) and (B): LLaVA1.5 additionally trained with mvif or BVQA data; POPE measures hallucination. Bold marks the best score per column as in the paper (Table 3 in the paper).
LMMLLM#PT#VITBVQAkKoVizKoLivBVQAeVQAGQALVPOPE
BLIP-2Vicuna13B129M-----4141-85.3
InstructBLIPVicuna7B129M1.2M-----49.2--
InstructBLIPVicuna13B129M1.2M-----49.5-78.9
LLaVA1.5Vicuna7B558K665K16.233.944.925.178.562.064.785.9
LLaVA1.5Vicuna13B558K665K27.924.433.426.180.063.365.785.9
LLaVA1.5(O)Vicuna13B558K756K32.624.623.229.178.145.370.485.8
LLaVA1.5(B)Vicuna13B558K857K54.550.352.133.576.463.022.885.8
KoLLaVASynatra7B595K612k45.355.954.25.5----
X-LLaVA (ours)Ours1.2M407K57.951.361.734.375.564.057.585.5

BibTeX

@inproceedings{shin-etal-2024-x,
    title = "{X}-{LL}a{VA}: Optimizing Bilingual Large Vision-Language Alignment",
    author = "Shin, DongJae  and
      Lim, HyeonSeok  and
      Won, Inho  and
      Choi, ChangSu  and
      Kim, Minjun  and
      Song, SeungWoo  and
      Yoo, HanGyeol  and
      Kim, SangMin  and
      Lim, KyungTae",
    editor = "Duh, Kevin  and
      Gomez, Helena  and
      Bethard, Steven",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {NAACL} 2024",
    month = jun,
    year = "2024",
    address = "Mexico City, Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-naacl.158/",
    doi = "10.18653/v1/2024.findings-naacl.158",
    pages = "2463--2473"
}
More projects from KAIST MLP Lab →