KORMo: Korean Open Reasoning Model for Everyone

A fully open 10.8B Korean–English bilingual LLM trained from scratch, mostly on synthetic Korean data

arXiv 2025

Minjun Kim1,4,*, Hyeonseok Lim1,4,*, Hangyeol Yoo1,4,*, Inho Won1,*, Seungwoo Song1,4, Minkyung Cho1, Junghun Yuk1, Changsu Choi1,4, Dongjae Shin1,4, Huije Lee2, Hoyun Song1, Alice Oh3, KyungTae Lim1,†

1KAIST MLP Lab 2KAIST NLPCL Lab 3KAIST U&I Lab 4SeoulTech

*Core contributors   †Corresponding author

Paper (arXiv) PDF Model Datasets Code (tutorial)
Two line charts of KORMo-10B benchmark averages against trained tokens from 0 to 2.8 trillion. (a) English average rises from about 0.36 to about 0.63 by the end of Stage 1 at roughly 1.0T tokens and to about 0.66 by the end of Stage 2. (b) Korean average rises from about 0.25 to about 0.51 in Stage 1, jumps at the start of Stage 2, and reaches about 0.56 to 0.57, without collapse.
Figure 7. Average English (a) and Korean (b) evaluation scores of KORMo-10B during pretraining. Stage 1 (about 1T tokens, mostly web data) and Stage 2 (1.8T tokens of high-quality web text, synthetic data, and short reasoning data) both improve steadily, showing that large amounts of synthetic data sustain long-horizon pretraining without model collapse. The released base model scores 64.20 (English) and 58.15 (Korean) on average.

Abstract

This work presents the first large-scale investigation into constructing a fully open bilingual large language model (LLM) for a non-English language, specifically Korean, trained predominantly on synthetic data. We introduce KORMo-10B, a 10.8B-parameter model trained from scratch on a Korean–English corpus in which 68.74% of the Korean portion is synthetic. Through systematic experimentation, we demonstrate that synthetic data, when carefully curated with balanced linguistic coverage and diverse instruction styles, does not cause instability or degradation during large-scale pretraining. Furthermore, the model achieves performance comparable to that of contemporary open-weight multilingual baselines across a wide range of reasoning, knowledge, and instruction-following benchmarks. Our experiments reveal two key findings: (1) synthetic data can reliably sustain long-horizon pretraining without model collapse, and (2) bilingual instruction tuning enables near-native reasoning and discourse coherence in Korean. By fully releasing all components including data, code, training recipes, and logs, this work establishes a transparent framework for developing synthetic data–driven fully open models (FOMs) in low-resource settings and sets a reproducible precedent for future multilingual LLM research. All model checkpoints, datasets, and source codes are publicly available at huggingface.co/kormo-lm.

Keywords Korean LLM Fully open model (FOM) Open-source LLM Bilingual Korean–English Synthetic pretraining data Pretraining from scratch Tokenizer design Model collapse Instruction tuning Low-resource languages

How KORMo Was Built

KORMo is a fully open model: besides the weights, the authors release data sources, filtering rules, scripts, hyperparameters, logs, and checkpoints. The pipeline:

  1. Proxy experiments. In a 1B-parameter, 60B-token proxy setting, a 100% synthetic condition is compared with a 100% non-synthetic condition across normalization (Pre-LN vs. MixLN), attention masking, next- vs. multi-token prediction, and learning-rate, batch-size, and sequence-length curricula, to check that synthetic data does not destabilize training.
  2. Tokenizer from scratch. A byte-level BPE tokenizer is trained on a mixture of crawled, synthetic, and code data, with compression compared against commercial tokenizers and its vocabulary screened for biased or harmful Korean tokens. KORMo-10B uses a 125,184-token vocabulary.
  3. Two-stage pretraining (2.9T tokens). Stage 1: about 1T tokens of mostly web data (DCLM 960B, Korean Web 36.3B, Korean-CC 6.2B). Stage 2: 1.8T tokens (1.7T English, 0.1T Korean) of high-quality web text, synthetic data, and short reasoning traces. Korean data is kept above 5% of the mix; synthetic Korean data is generated with several model families (e.g. Qwen and GPT-OSS) for diversity, since a single-source synthetic set caused a failed run.
  4. Architecture. Pre-LN transformer decoder with 10.75B parameters, 40 layers, hidden size 4096, 32 attention heads with 8 key/value heads, SwiGLU, RMSNorm, intra-document attention masking, and next-token prediction.
  5. Mid-training and post-training. Long-context extension and reasoning-trace training, followed by Base SFT and Instruction-Following SFT on bilingual data, plus preference-learning datasets (APO, ORPO) for math and reasoning.

Main Results

  • Base model on par with larger fully open models. KORMo-10B (2.9T tokens) averages 64.20 on English benchmarks, close to OLMo2-13B (65.32, 5.5T tokens), and 58.15 on Korean benchmarks, far above OLMo2-13B (39.34) and SmolLM3-3B (47.37).
  • Competitive with open-weight multilingual models. It scores 1.84 points higher than Kanana1.5-8B in English but 2.79 lower in Korean, with a Korean share of about 5.6% versus a reported 10% for Kanana; its English–Korean gap is about 6 points.
  • Strong instruction following after SFT only. On GPT-4o-judged benchmarks, KORMo-10B reaches MT-Bench 8.32, KO-MT-Bench 8.54, and LogicKor 8.96, for the highest average (8.61) among the compared models, ahead of Gemma3-12B (8.56) and Qwen3-8B (8.50).
  • Weak spots. Specialist knowledge (MMLU-Pro 40.18, KMMLU 46.48) and fine-grained lexical-semantic discrimination (KoBALT 22.86) remain below the largest multilingual models.
Table 21 (subset). Base-model accuracy (%) of fully open and open-weight multilingual models on English and Korean benchmarks; training tokens in parentheses. Averages are over all 14 English and 9 Korean benchmarks in the paper; a representative subset of benchmarks and models is shown (Gemma3-4B omitted; full table in the paper).
BenchmarkKORMo-10B
(2.9T)		SmolLM3-3B
(10.7T)		OLMo2-7B
(4T)		OLMo2-13B
(5.5T)		Kanana1.5-8B
(3.2T)		Qwen3-8B
(36T)		Llama3.1-8B
(15T)		Gemma3-12B
(12T)
ARC-Challenge58.9655.5559.1361.0156.4863.8254.6163.82
HellaSwag60.2556.7861.5265.0259.9359.5460.9663.67
MMLU67.9661.3762.8166.8563.7376.9565.0373.58
MMLU-Pro40.1834.9427.2932.5034.9356.5836.2337.07
English Avg.64.2060.8062.4865.3262.3668.5562.6567.72
HAE-RAE68.2955.8231.6237.5880.7567.1953.2574.34
K2-Eval84.8975.2349.5463.4384.7284.7276.6285.42
KMMLU46.4838.5233.0531.2448.8656.9340.7551.60
KR-Clinical-QA77.3253.9748.3346.2265.8480.0063.5477.22
Korean Avg.58.1547.3735.8239.3460.9463.3549.6060.37

KORMo-10B, SmolLM3, and OLMo2 are fully open models; Kanana1.5, Qwen3, Llama3.1, and Gemma3 are open-weight models.

BibTeX

@misc{kim2025kormokoreanopenreasoning,
      title={KORMo: Korean Open Reasoning Model for Everyone},
      author={Minjun Kim and Hyeonseok Lim and Hangyeol Yoo and Inho Won and Seungwoo Song and Minkyung Cho and Junhun Yuk and Changsu Choi and Dongjae Shin and Huige Lee and Hoyun Song and Alice Oh and Kyungtae Lim},
      year={2025},
      eprint={2510.09426},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.09426},
}
More projects from KAIST MLP Lab →