ELO: Efficient Layer-Specific Optimization for Continual Pretraining of Multilingual LLMs

Continual pretraining of only the detached first and last layers to add a target language to a multilingual LLM, up to 6.46× faster than full fine-tuning

EACL 2026 Industry

Hangyeol Yoo1,*, ChangSu Choi1,2,*,†, Minjun Kim3, Seohyun Song1, SeungWoo Song1, Inho Won3, Jongyoul Park1, Cheoneum Park4, KyungTae Lim3,‡

1Seoul National University of Science and Technology 2LG CNS 3Korea Advanced Institute of Science and Technology (KAIST) 4Hanbat National University

*Equal contribution   †Work done during an internship at LG CNS   ‡Corresponding author

Paper (ACL Anthology) PDF arXiv
Three-part diagram. Left: the original model with token embedding, a stack of decoder layers and LM head. Middle: the small ELO model built from the frozen token embedding and LM head plus two detached layers (the first and last), which are trained. Right: the trained layers replace the first and last layers of the original model, after which the whole model is aligned.
Figure 1. The ELO training process. The first and last decoder layers are detached from the original model together with the token embedding and LM head (kept frozen) to form a small ELO model; only the two layers are pretrained on target-language data and then replace the corresponding layers of the original model, followed by a brief full fine-tuning (layer alignment) on 1 GB of data.

Abstract

We propose an efficient layer-specific optimization (ELO) method designed to enhance continual pretraining (CP) for specific languages in multilingual large language models (MLLMs). This approach addresses the common challenges of high computational cost and degradation of source language performance associated with traditional CP. The ELO method consists of two main stages: (1) ELO Pretraining, where a small subset of specific layers, identified in our experiments as the critically important first and last layers, are detached from the original MLLM and trained with the target language. This significantly reduces not only the number of trainable parameters but also the total parameters computed during the forward pass, minimizing GPU memory consumption and accelerating the training process. (2) Layer Alignment, where the newly trained layers are reintegrated into the original model, followed by a brief full fine-tuning step on a small dataset to align the parameters. Experimental results demonstrate that the ELO method achieves a training speedup of up to 6.46 times compared to existing methods, while improving target language performance by up to 6.2% on qualitative benchmarks and effectively preserving source language (English) capabilities.

Keywords Continual pretraining (CP) Language adaptation Multilingual LLMs Korean LLM Japanese LLM Layer-specific training Training efficiency LoRA comparison Chat vector

Method

Continual pretraining (CP) is the usual way to strengthen a multilingual LLM in a specific language, but it is expensive and tends to hurt English. LoRA and selective layer tuning reduce the number of trainable parameters, yet the forward pass still runs through the whole model, so the speedup over full fine-tuning (FFT) is small. ELO instead trains a much smaller, separate model:

  1. ELO pretraining. Detach the first and last decoder layers (λ = {ℓ1, ℓn}) together with the token embedding and LM head into a small ELO model. Only the two layers are trained, on 9 GB of data with an English : target-language ratio of 1 : 9, so both the trainable and the computed parameters shrink.
  2. Layer replacement and alignment. Put the trained layers back into the original model and run a brief full fine-tuning on an additional 1 GB of data to align them with the untouched layers, which keep their English knowledge.
  3. Bilingual instruction tuning. Add a chat vector (instruction-tuned minus pretrained weights) and fine-tune on 31K instruction samples in a 1 : 1 target-language / English ratio.

Korean and Japanese are the target languages, with Llama 3.1-8B, Mistral-7B-v0.3 and Qwen2-7B as base models. Evaluation uses MMLU and MT-Bench for English, KoBEST and LogicKor for Korean, and MARC-ja and MT-Bench(ja) for Japanese (MT-Bench and LogicKor judged by GPT-4 on a 10-point scale).

Main Results

  • 5.26–6.46× faster than FFT. With 10 GB of pretraining data, ELO cuts total training time by 5.88× on average (e.g. Llama 3.1: 3.5 h vs. 19.8 h for Korean).
  • Better target-language generation. ELO models outperform FFT on LogicKor (Llama 3.1: 7.76 vs. 7.31, +6.2%; Qwen2: 7.22 vs. 6.95), and Llama3.1-ELO improves LogicKor by 22.2% over Llama3.1-8B-Instruct.
  • English preserved. The average MMLU gap between the instruct models and ELO is 2.42%, while KoBEST improves over the instruct models by 8.55% to 23.57%.
  • Faster than LoRA at scale. On Llama3.1-8B, ELO is 5.29× faster than LoRA and 6.04× faster than FFT with 50 GB of data, and 10.72× / 12.23× faster with 200 GB.
  • First and last layers matter. Training layers (1, 32) gives LogicKor 7.76, while (8, 24) gives 5.00 and (16, 17) 5.47.
Table 1. Korean as the target language: total training time and scores of the official instruct models, full fine-tuning (FFT) and ELO. MMLU and KoBEST are quantitative; MT-Bench and LogicKor are GPT-4-judged scores out of 10. Bold as in the paper (best per model group). (Subset of Table 1 in the paper: Korean block only; the Japanese block and data sizes are in the paper.)
ModelTotal timeMMLU (en)KoBEST (ko)MT-Bench (en)LogicKor (ko)
Llama3.1-8B-Instruct-68.0756.026.966.03
Llama3.1-FFT19.8 h67.5166.086.707.31
Llama3.1-ELO (ours)3.5 h66.6960.816.797.76
Mistral-7B-Instruct-v0.3-59.6949.016.724.49
Mistral-FFT31.0 h57.4761.686.616.50
Mistral-ELO (ours)4.8 h58.9060.566.976.59
Qwen2-7B-Instruct-69.8960.177.676.90
Qwen2-FFT20.5 h70.2372.377.116.95
Qwen2-ELO (ours)3.9 h70.1171.577.257.22

BibTeX

@inproceedings{yoo-etal-2026-elo,
    title = "{ELO}: Efficient Layer-Specific Optimization for Continual Pretraining of Multilingual {LLM}s",
    author = "Yoo, Hangyeol  and
      Choi, ChangSu  and
      Kim, Minjun  and
      Song, Seohyun  and
      Song, SeungWoo  and
      Won, Inho  and
      Park, Jongyoul  and
      Park, Cheoneum  and
      Lim, KyungTae",
    editor = {Matusevych, Yevgen  and
      Eryi{\u{g}}it, G{\"u}l{\c{s}}en  and
      Aletras, Nikolaos},
    booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 5: Industry Track)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.eacl-industry.55/",
    doi = "10.18653/v1/2026.eacl-industry.55",
    pages = "752--763",
    ISBN = "979-8-89176-384-5"
}
More projects from KAIST MLP Lab →