Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean

Large language models (LLMs) use pretraining to predict the subsequent word; however, their expansion requires significant computing resources. Numerous big tech companies and research institutes have developed multilingual LLMs (MLLMs) to meet current demands, overlooking less-resourced languages (LRLs). This study proposed three strategies to enhance the performance of LRLs based on the publicly available MLLMs. First, the MLLM vocabularies of LRLs were expanded to enhance expressiveness. Second, bilingual data were used for pretraining to align the high- and less-resourced languages. Third, a high-quality small-scale instruction dataset was constructed and instruction-tuning was performed to augment the LRL. The experiments employed the Llama2 model and Korean was used as the LRL, which was quantitatively evaluated against other developed LLMs across eight tasks. Furthermore, a qualitative assessment was performed based on human evaluation and GPT4. Experimental results showed that our proposed Bllossom model exhibited superior performance in qualitative analyses compared to previously proposed Korean monolingual models.

Llama2 saw only 0.06% Korean data during pretraining, and its SentencePiece tokenizer falls back to UTF-8 bytes for most Korean syllables: the phrase 햄버거를 먹는 공룡 (“a dinosaur eating a hamburger”) becomes 21 tokens, mostly byte tokens such as <0xED>, whereas the expanded tokenizer produces 8. Bllossom augments Korean in three steps:

1. Vocabulary expansion. The Llama2 vocabulary (32,000 entries) is merged with the morpheme-aware KoBERT vocabulary (8,002 entries), giving 39,478 entries; the 7,478 new embeddings are randomly initialized.
2. Bilingual pretraining. The expanded model is further pretrained with causal language modeling on 33.09 GB of Korean and English text (22.41 GB public Korean news/web data, 0.76 GB Korean Wikipedia, 9.92 GB English Wikipedia) for one epoch, to link Korean with the model's mostly English knowledge.
3. Korean LIMA instruction tuning. The 1,030 English LIMA examples are machine-translated, reviewed by native speakers, and adapted to Korean culture (named entities replaced, topics changed); the model is tuned on Korean (and English) LIMA data.

Training uses LoRA on the attention query/key/value projections plus the expanded embeddings (5.977% of the parameters), which fits on a single A6000 GPU. Models are compared on eight Korean benchmarks (KLUE NLI, STS and YNAT; Ko-SBI; KoBEST BoolQ, HellaSwag, SentiNeg and COPA) and by human and GPT-4 preference on 300 LIMA test prompts translated into Korean.