BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretraining

A Korean–English knowledge-based VQA dataset (17K images, 280K knowledge triples) and a model that injects pretrained knowledge graph embeddings into visual question answering

AAAI 2024

MinJun Kim1,*, SeungWoo Song1,*, YouHan Lee2, Haneol Jang1, KyungTae Lim3,†

1Hanbat National University 2Kakao Brain 3Seoul National University of Science and Technology

*These authors contributed equally   †Corresponding author

Paper (AAAI) PDF arXiv
Example BOK-VQA sample. Left: photo of a jellyfish in an aquarium. Right: English question 'Which phylum does an animal belonging to Scyphozoa in the image belong to?', the same question in Korean, the knowledge-base triple [Jellyfish, phylum, Cnidaria], and the answer 'Cnidaria' in English and its Korean equivalent.
Figure 1. Example of a BOK-VQA sample. The question cannot be answered from the image alone; the external knowledge triple [Jellyfish, phylum, Cnidaria] supplies the missing fact. Every sample has the question and answer in both English (en) and Korean (ko), while the knowledge base is in English only.

Abstract

The current research direction in generative models, such as the recently developed GPT4, aims to find relevant knowledge information for multimodal and multilingual inputs to provide answers. Under these research circumstances, the demand for multilingual evaluation of visual question answering (VQA) tasks, a representative task of multimodal systems, has increased. Accordingly, we propose a bilingual outside-knowledge VQA (BOK-VQA) dataset in this study that can be extended to multilingualism. The proposed data include 17K images, 17K question-answer pairs for both Korean and English and 280K instances of knowledge information related to question-answer content. We also present a framework that can effectively inject knowledge information into a VQA system by pretraining the knowledge information of BOK-VQA data in the form of graph embeddings. Finally, through in-depth analysis, we demonstrated the actual effect of the knowledge information contained in the constructed training data on VQA.

Keywords Knowledge-based VQA (KB-VQA) Outside-knowledge VQA (OK-VQA) Multilingual / bilingual VQA Korean–English dataset Knowledge graph embedding (KGE) ConvKB Triple prediction Multimodal dataset Multi-task learning

Dataset and Method

GEL-VQA architecture. (a) VQA module: image v_i goes through ResNet and question q_i through XLM-RoBERTa; their features are multiplied element-wise into f_i(u), which is multiplied with a knowledge embedding emb_kb_i and passed through an FCNN and softmax to predict the answer y_i. (b) Triple prediction module: three separate FCNN and softmax heads predict the head h_i, relation r_i, and tail t_i from the fused feature. (c) Pretrained knowledge graph embedding table: the embeddings of the predicted head, relation, and tail from the previous step are concatenated and passed through an FCNN to form emb_kb_i.
Figure 4. GEL-VQA architecture: (a) VQA module, (b) triple prediction module, and (c) pretrained knowledge graph embeddings (KGE). The embeddings of the triple predicted at the previous training step are fused with the image–question feature to answer the question.

Dataset. BOK-VQA contains 17,836 samples, each with an image, a question, an answer, and the IDs of the external knowledge needed to answer it, plus 282,533 knowledge triples covering 1,579 objects and 42 relations taken from English ConceptNet and DBpedia (based on the 500 objects and 10 relations of FVQA, extended with 1,079 ImageNet objects and 32 more relations). 87 skilled annotators wrote the Korean questions and answers; English versions were produced by neural translation and reviewed by two professional translators. A question was kept only if three reviewers agreed that external knowledge was needed to answer it.

Key idea. Entities and relations do not depend on language, so knowledge graph embeddings trained on English triples can be reused for Korean (or any other language) questions. No Korean knowledge graph is needed; the model only has to link a Korean question to the right English triple.

  1. KGE pretraining. ConvKB is trained on the full English knowledge graph (about 280K triples) for 50,000 iterations, learning to separate valid triples from corrupted ones (random head or tail replacement).
  2. VQA module. ResNet-50 encodes the image and multilingual XLM-RoBERTa encodes the question; the two features are combined by element-wise multiplication and classified with an MLP. Used alone, this is the BASELINE.
  3. Triple prediction. From the same image–question feature, three separate classifiers predict the head, relation, and tail of the needed triple. The targets are language-independent IDs.
  4. Knowledge injection (GEL-VQA). The pretrained embeddings of the triple predicted at the previous step are concatenated, projected, and multiplied with the image–question feature before answer classification. VQA and triple prediction are trained jointly as a multi-task loss. Variants add teacher forcing with ground-truth triples during training (+TF) and multi-head attention over the head, relation, and tail embeddings (+ATTN).

Main Results

  • External knowledge matters. VQA accuracy ranges from 21.16 to 55.48 depending on how knowledge graph embeddings and triple prediction are used. the paper reports GEL-VQA 25.97 points above the BASELINE, and the oracle GEL-VQA (Ideal), which is given the correct triples, reaches 66.01–72.25.
  • No drop for the less-resourced language. Even though the knowledge graph is English only, Korean accuracy was on average 3.7% higher than English (best Korean result 55.48 with GEL-VQA + TF + ATTN).
  • Triple prediction is language-independent. Overall triple prediction accuracy is 50.43 for Korean, 47.59 for English, and 47.17 for bilingual training (relation accuracy 85.39 / 81.93 / 82.76).
  • Robustness. Replacing nouns with synonyms keeps GEL-VQA close to the original (42.79 vs. 45.08), while hypernyms drop it to 36.24. Under ontological transformation the accuracy drop on BOK-VQA was 7.85%, versus 12.03% reported for VQA v2.0.
  • Choice of KGE. TransE gives the best GEL-VQA accuracy (51.22, vs. 45.08 with ConvKB); KGE quality on FB15K (Hits@10) correlates with GEL-VQA (Ideal) accuracy (Pearson 0.85).
Table 2. VQA accuracy (mean ± std over five folds) on BOK-VQA when training on bilingual, English, or Korean questions, always with the English knowledge base. GEL-VQA (Ideal) is an oracle that is given the correct triples. Rows and columns are transposed from the paper (Table 2 in the paper).
ModelBilingualEnglishKorean
BASELINE (no external knowledge)21.51±1.8121.90±2.3421.16±1.50
GEL-VQA (Ideal, oracle triples)66.01±1.8366.68±1.1572.25±1.29
GEL-VQA45.08±0.9447.83±0.5650.30±2.24
GEL-VQA + TF48.07±1.3351.64±0.8853.40±2.73
GEL-VQA + TF + ATTN48.11±1.5050.74±0.9055.48±1.89

BibTeX

@article{Kim_2024,
  title={BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretraining},
  volume={38},
  ISSN={2159-5399},
  url={http://dx.doi.org/10.1609/aaai.v38i16.29798},
  DOI={10.1609/aaai.v38i16.29798},
  number={16},
  journal={Proceedings of the AAAI Conference on Artificial Intelligence},
  publisher={Association for the Advancement of Artificial Intelligence (AAAI)},
  author={Kim, MinJun and Song, SeungWoo and Lee, YouHan and Jang, Haneol and Lim, KyungTae},
  year={2024},
  month=Mar,
  pages={18381–18389}
}
More projects from KAIST MLP Lab →