TELLME: Test-Enhanced Learning for Language Model Enrichment

Kim, Minjun; Won, Inho; Lim, HyeonSeok; Kim, MinKyu; Yuk, Junghun; Go, Wooyoung; Park, Jongyoul; Park, Jungyeul; Lim, KyungTae

doi:10.18653/v1/2026.findings-eacl.84

Continual pre-training with descriptive quiz-style QA, inspired by the testing effect, for domain knowledge acquisition and long-term retention in LLMs

Findings of EACL 2026

Minjun Kim^1,*, Inho Won^1,*, HyeonSeok Lim¹, MinKyu Kim², Junghun Yuk¹, Wooyoung Go³, Jongyoul Park², Jungyeul Park¹, KyungTae Lim^1,†

¹Korea Advanced Institute of Science and Technology (KAIST) ²Seoul National University of Science and Technology ³National Security Research Institute

^*Equal contribution ^†Corresponding author

Abstract

Continual pre-training (CPT) has been widely adopted as a method for domain expansion in large language models. However, CPT has consistently been accompanied by challenges, such as the difficulty of acquiring large-scale domain-specific datasets and high computational costs. In this study, we propose a novel method called Test-Enhanced Learning for Language Model Enrichment (TELLME) to alleviate these issues. TELLME leverages the Test-Enhanced Learning (TEL) principle, whereby the model’s learning efficiency is improved using quizzes during training. It integrates this principle with CPT, thereby promoting efficient domain-specific knowledge acquisition and long-term memory retention. Experimental results demonstrate that TELLME outperforms existing methods by up to 23.6% in the financial domain and achieves a 9.8% improvement in long-term memory retention.

Keywords Test-enhanced learning (TEL) Testing effect Continual pre-training (CPT) Domain adaptation Synthetic QA data Knowledge retention Catastrophic forgetting Financial LLM Medical LLM

Method

Schematic of seven training paradigms showing which segments receive loss. Continual pre-training: plain text, all tokens trained. Instruction tuning: question masked, answer trained. PIT: plain text and QA as separate samples, question masked. InstPT: plain text, question and answer all trained. TELLME: plain text followed by question and answer in one sequence, question masked. Inv-TEL: question and answer before plain text. TEL-Q/L: like TELLME but the question is also trained.

Figure 4. Data composition and token-level loss masking across training paradigms (green check: loss computed; red cross: masked). TELLME places plain text and its QA pairs in a single training sequence and computes the loss on plain-text and answer tokens only; Inv-TEL reverses the order and TEL-Q/L also trains on question tokens. (Appendix figure in the paper.)

In educational psychology, test-enhanced learning (TEL) describes the testing effect: taking quizzes, especially open-ended explanatory ones, improves learning and long-term retention more than re-reading. TELLME transfers this idea to continual pre-training (CPT) of LLMs:

Quiz generation. For each domain document, GPT-4o-mini generates M = 3 descriptive question–answer pairs that are related to the text but cannot be answered by copying from it (concept mapping, e.g. “How do high-frequency trading strategies impact market volatility?”). 100K samples were built from PubMed abstracts and Bloomberg financial news for about $12; an LLM-as-a-judge rated them 4.03 out of 5 on average.
Joint training sample. Each training sequence is X = (t, q₁, a₁, …, q_M, a_M): the plain text followed by its quiz.
Loss masking. The causal language modeling loss is computed on plain-text and answer tokens, while question tokens are masked, so CPT and quiz learning happen in a single forward–backward pass.

TELLME is compared with CPT, CPT followed by instruction tuning (CPT + IT, same tokens as TELLME), and InstPT on Llama-3.2-1B, Llama-3.2-3B, Llama-3.1-8B and SmolLM2-1.7B, using FOMC, NIFTY and MMLU-Finance for finance and HeadQA, MedMCQA and MMLU-Clinical for medicine.

Main Results

Best overall average. TELLME achieves the highest average score across finance and medicine for every model (e.g. 40.73 on Llama-3.1-8B vs. 39.94 for CPT), outperforming CPT + IT by 10.0% and InstPT by 6.3% overall.
Finance gains. In the finance domain TELLME improves on the base models by about 9.8% on average and outperforms InstPT by 5.1% on average, although InstPT trains on nearly twice as much QA data.
Long-term retention. When a finance-trained SmolLM2-1.7B is further trained on medical data, the CPT model loses 5.72% of its finance performance but the TELLME model only 0.94%, ending 9.8% (3.15 points) higher.
Efficiency and robustness. TELLME reaches the same perplexity 1.4× faster than CPT; with Mistral-generated or self-generated quizzes it still beats InstPT by 2.27 and 2.61 points (Llama-3.2-3B, finance), and Korean TELLME data raise OLMo2-1B's KoBEST average by 8.4 points.

**Table 2.** Average accuracy in the finance (FOMC, NIFTY, MMLU-F) and medicine (HeadQA, MedMCQA, MMLU-C) domains under different training methods (subset: domain averages only; per-benchmark scores are in the paper). Bold marks the best value per model, as in the paper. (Table 2 in the paper.)
Model	Method	Finance avg.	Medicine avg.	Average
Llama-3.2-1B	Base	30.73	32.16	31.45
	+ CPT	29.05	32.26	30.66
	+ CPT + IT	30.36	29.76	30.06
	+ InstPT	31.15	30.08	30.61
	+ TELLME (ours)	33.46	32.62	33.04
Llama-3.2-3B	Base	32.97	37.63	35.30
	+ CPT	29.98	37.22	33.60
	+ CPT + IT	30.87	35.64	33.26
	+ InstPT	31.03	35.16	33.10
	+ TELLME (ours)	33.87	38.22	36.05
Llama-3.1-8B	Base	33.07	43.99	38.53
	+ CPT	37.44	42.44	39.94
	+ CPT + IT	31.41	36.62	34.02
	+ InstPT	36.99	38.66	37.83
	+ TELLME (ours)	38.82	42.63	40.73
SmolLM2-1.7B	Base	32.53	35.49	34.01
	+ CPT	34.28	35.28	34.78
	+ CPT + IT	34.62	35.71	35.17
	+ InstPT	35.95	35.39	35.67
	+ TELLME (ours)	35.82	36.14	35.98

@inproceedings{kim-etal-2026-tellme, title = "{TELLME}: Test-Enhanced Learning for Language Model Enrichment", author = "Kim, Minjun and Won, Inho and Lim, HyeonSeok and Kim, MinKyu and Yuk, Junghun and Go, Wooyoung and Park, Jongyoul and Park, Jungyeul and Lim, KyungTae", editor = "Demberg, Vera and Inui, Kentaro and Marquez, Llu{\'i}s", booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EACL} 2026", month = mar, year = "2026", address = "Rabat, Morocco", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2026.findings-eacl.84/", doi = "10.18653/v1/2026.findings-eacl.84", pages = "1655--1677", ISBN = "979-8-89176-386-9" }

TELLME: Test-Enhanced Learning for Language Model Enrichment

Abstract

Method

Main Results

BibTeX