TReX: Tokenizer Regression for Optimal Data Mixture

Predicting the best multilingual data mixture for tokenizer training with a regression model fitted on small proxy tokenizers

EACL 2026

Inho Won1,*, Hangyeol Yoo2,*, Minkyung Cho1, Jungyeul Park1,3, Hoyun Song4,†, KyungTae Lim1,4,†

1KAIST CT 2Seoul National University of Science and Technology 3Upstage AI 4KAIST InnoCORE PRISM-AI Center

*Equal contribution   †Corresponding authors

Paper (ACL Anthology) PDF arXiv Code
Four-panel diagram. (1) Data mixtures over languages such as French, Chinese, Greek and Japanese are sampled from a Dirichlet distribution. (2) A proxy tokenizer is trained on each mixture and its compression is measured (e.g. 0.888, 0.970, 0.883). (3) A tree-based regression model is fitted from mixtures to compressions. (4) The model predicts compression for many possible mixtures, and the minimum gives the optimal data mixture w*.
Figure 1. TReX overview: (1) N data mixtures are sampled, (2) a small proxy tokenizer is trained for each mixture and its compression C is measured, (3) a regression model is fitted with the mixture w as input and C as the target, and (4) the model predicts C for candidate mixtures to identify the optimum.

Abstract

Building effective tokenizers for multilingual Large Language Models (LLMs) requires careful control over language-specific data mixtures. While a tokenizer’s compression performance critically affects the efficiency of LLM training and inference, existing approaches rely on heuristics or costly large-scale searches to determine optimal language ratios. We introduce Tokenizer Regression for Optimal Data MiXture (TReX), a regression-based framework that efficiently predicts the optimal data mixture for tokenizer training. TReX trains small-scale proxy tokenizers on random mixtures, gathers their compression statistics, and learns to predict compression performance from data mixtures. This learned model enables scalable mixture search before large-scale tokenizer training, mitigating the accuracy-cost trade-off in multilingual tokenizer design. Tokenizers trained with TReX’s predicted mixtures outperform mixtures based on LLaMA3 and uniform distributions by up to 12% in both in- and out-of-distribution compression efficiency, demonstrating strong scalability, robustness, and practical effectiveness.

Keywords Multilingual tokenizer Data mixture optimization Language ratios Tokenizer compression Normalized sequence length (NSL) Proxy tokenizers Regression (LightGBM) Multilingual LLMs

Method

A tokenizer's compression is measured as the Normalized Sequence Length (NSL): the number of tokens it produces on a test corpus divided by the number produced by a reference tokenizer (GPT-4o); lower is better. For a fixed corpus size S and vocabulary size V, the goal is the language mixture w (weights over k languages summing to 1) that minimizes this score. Instead of hand-tuned ratios or repeated large-scale training, TReX learns the mapping from mixture to compression:

  1. Sample mixtures. Draw N = 512 language mixtures from a Dirichlet distribution based on each language's data size (19 languages of FineWeb2-HQ).
  2. Train proxy tokenizers. Train one small tokenizer per mixture (S = 1 GB, V = 64K) and measure its compression.
  3. Fit a regression model. Fit LightGBM on 480 (mixture, compression) pairs, keeping 32 for evaluation.
  4. Search. Predict compression for a very large number of candidate mixtures (over 50M within a few seconds) and train the final large tokenizer (S = 30 GB, V = 200K) on the best one.

This relies on rank invariance: the ranking of mixtures by compression stays stable when the corpus and vocabulary grow. Across 1GB–64K, 5GB–64K, 10GB–100K and 30GB–200K settings, all pairwise Spearman correlations are at least 0.96 (0.974 between the smallest and largest settings).

Main Results

  • Accurate prediction. On unseen mixtures the regression model reaches a MAPE of 1.989 (less than 2% average error) and a Spearman rank correlation of 0.979.
  • Best compression. The TReX mixture gives the lowest compression score both in-distribution (0.871 on FineWeb2-HQ vs. 0.883 for the GPT-4o-vocabulary mixture and 0.970 for language buckets, a 10.21% improvement over the latter) and out-of-distribution (0.877 on FLORES-200 vs. 0.904 for the uniform mixture and 0.997 for language buckets).
  • Non-Latin scripts. On the five non-Latin languages the average score is 0.814, vs. 0.848 (uniform) and 0.863 (LLaMA3 mixture), although TReX assigns only 28.2% of the data to them (LLaMA3 mixture: 36.8%).
  • Training cost. For a 13B model on 3T tokens, the TReX tokenizer gives the shortest estimated training time (8,009.6 hours), more than 1,000 hours less than the language-bucket mixture; finding the mixture costs 52.2% less time than AdaptMix.
  • Domain-specific use. Optimizing for medical text from the Pile, the regression model reaches a MAPE of 0.921.
Table 2. Compression score C (NSL relative to the GPT-4o tokenizer; lower is better) of large-scale tokenizers (S = 30 GB, V = 200K) trained with different data mixtures, averaged over all 19 languages (subset: the “All Languages” row; per-language scores are in the paper). Uniform = equal weights; Language buckets = mixture of Abagyan et al.; GPT-4o / LLaMA3 = language proportions in those tokenizers' vocabularies. Bold marks the best score. (Table 2 in the paper.)
Data mixtureIn-distribution (FineWeb2-HQ) ↓Out-of-distribution (FLORES-200) ↓
Uniform (wuni)0.8880.904
Language buckets (wLB)0.9700.997
GPT-4o vocabulary (wgpt)0.8830.907
LLaMA3 vocabulary (wllama)0.9070.906
TReX (ours)0.8710.877

BibTeX

@inproceedings{won-etal-2026-trex,
    title = "{TR}e{X}: Tokenizer Regression for Optimal Data Mixture",
    author = "Won, Inho  and
      Yoo, Hangyeol  and
      Cho, Minkyung  and
      Park, Jungyeul  and
      Song, Hoyun  and
      Lim, KyungTae",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.eacl-long.298/",
    doi = "10.18653/v1/2026.eacl-long.298",
    pages = "6353--6370",
    ISBN = "979-8-89176-380-7"
}
More projects from KAIST MLP Lab →