Beyond Accuracy: Alignment and Error Detection across Languages in the Bi-GSM8K Math-Teaching Benchmark

A bilingual English–Korean GSM8K benchmark with teacher solutions, simulated student solutions and first-error annotations for evaluating LLMs as math tutors

Findings of EACL 2026

Jieun Park†,*, KyungTae Lim‡,*, Joon-ho Lim†

†Tutorus Labs (AI Lab) ‡Korea Advanced Institute of Science and Technology (KAIST)

*Equally contributed authors

Paper (ACL Anthology) PDF
Step alignment between a student solution (left) and the answer solution (right) for a cleaning-time word problem. The student's first step matches the teacher's first step; the student's second step 'Thus, 3D = 38' is marked as the first error; teacher steps 'set up the equation 3D + 5 = 38' and the final upstairs-time step are labeled missing.
Figure 2. When a student's solution is submitted, it is first aligned with the answer solution using language-model similarity functions and the Needleman–Wunsch (NW) algorithm, revealing matching, omitted and extraneous steps; a separate LLM-based error detection module then identifies the first erroneous step (here, “Thus, 3D = 38”, where the student drops the +5).

Abstract

Recent advancements in LLMs have significantly improved mathematical problem-solving, with models like GPT-4 achieving human-level performance. However, proficiently solving mathematical problems differs fundamentally from effectively teaching mathematics. To bridge this gap, we introduce the Bi-GSM8K benchmark, a bilingual English-Korean dataset enriched with teacher solutions, student solutions, and annotations marking students’ initial errors. This dataset is designed to evaluate two core capabilities of LLMs: (1) measuring similarity between student and teacher solutions, and (2) identifying the initial error point in student solutions. Our method achieves high agreement with human judgments, with Pearson 0.89 and Spearman 0.88 on English, and Pearson 0.89 and Spearman 0.87 on Korean. It also offers significantly lower latency and resource usage than commercial APIs, demonstrating strong computational efficiency. In the error detection task, open-source models achieved approximately 86% accuracy, with performance within 10% points of commercial LLMs API, suggesting strong practical potential. Our key contributions include the open-source release of Bi-GSM8K, novel evaluation metrics, and comparative analyses of LLM performance across languages.

Keywords Math education LLM tutoring Student error detection Solution step alignment Needleman-Wunsch GSM8K Korean–English bilingual Benchmark

Benchmark and Method

Solving math problems and teaching math are different skills: a tutor must follow a student's reasoning and find where it goes wrong. GSM8K only provides problems and solved answers, so the paper builds Bi-GSM8K, 7,985 JSON entries with:

  • Curriculum-aligned bilingual problems. GSM8K was translated into Korean and refined to the Korean national curriculum (names, units, vocabulary) by mathematics education experts, then translated back into English with GPT-4o.
  • Teacher solutions (correct_solution) written step by step by experts and reviewed for accuracy and pedagogy.
  • Simulated student solutions (error_solution) written by education annotators to reflect realistic misconceptions (not real student data).
  • Initial error annotations marking the first step with a numerical or logical inconsistency, cross-validated by experts.

Two evaluation tasks are defined on top of the data:

  1. Ground Truth Alignment (GTA). Compute semantic similarity between teacher and student steps (cosine, Pearson or SemScore of LLM embeddings, or BERTScore), align the two step sequences with a Needleman–Wunsch algorithm whose gap penalty shrinks for semantically similar steps, and score the solution as the fraction of matched steps (0 to 1).
  2. Solution Error Detection (SED). A 3-shot prompt gives the problem, the teacher solution and the student solution and asks the LLM for the step number of the first error (0 if correct).

Both tasks are evaluated on 500 randomly selected problems in English and Korean, with GTA compared against five human annotators.

Main Results

  • Human-level alignment scores. With BERTScore (bert-large-uncased), GTA reaches Pearson 0.8942 (EN) and 0.8929 (KO) and Spearman 0.8752 (EN) against human similarity judgments; open-source configurations are within about 0.03 points of commercial LLM APIs on average.
  • Much cheaper alignment. bart-large and bert-large-uncased take 0.070 s and 0.113 s per problem with about 700–1,010 MB of memory, vs. 3.614 s for GPT-4o-mini through the API.
  • First-error detection. GPT-4o finds the student's first error with 94.4% (EN) and 95.8% (KO) accuracy; the best open-source models reach 86.4% (DeepSeek-R1-Distill-Qwen-7B-Multilingual, EN) and 86.0% (Qwen2.5-7B-Instruct, KO).
  • Language effects. Multilingual training does not help both languages equally: DeepSeek-R1-Distill-Qwen-7B-Multilingual is stronger in English than Qwen2.5-7B-Instruct (86.4 vs. 79.4) but weaker in Korean (83.4 vs. 86.0).
Table 2. Solution Error Detection on 500 Bi-GSM8K problems: accuracy (%) of identifying the student's first erroneous step in English (EN) and Korean (KO), and latency (sec). Commercial APIs above the line, open-source models below; bold marks the best value per group, as in the paper. (Subset of Table 2 in the paper: throughput and peak memory omitted.)
ModelAccuracy ENAccuracy KOLatency (sec)
GPT-4o94.495.88.39
Claude-Sonnet-487.890.25.31
Gemini-2.5-Flash90.491.44.91
Llama-3.1-8B-Instruct80.475.2711.60
DeepSeek-R1-Distill-Llama-8B55.457.8720.06
DeepSeek-llama3.1-Bllossom-8B63.262.0716.86
Qwen2.5-7B-Instruct79.486.0783.91
DeepSeek-R1-Distill-Qwen-7B82.880.4801.63
DeepSeek-R1-Distill-Qwen-7B-Multilingual86.483.4847.68
Mistral-7B-Instruct-v0.367.067.01168.91
Phi-4-mini-instruct76.678.6355.01

BibTeX

@inproceedings{park-etal-2026-beyond,
    title = "Beyond Accuracy: Alignment and Error Detection across Languages in the {B}i-{GSM}8{K} Math-Teaching Benchmark",
    author = "Park, Jieun  and
      Lim, KyungTae  and
      Lim, Joon-ho",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EACL} 2026",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.findings-eacl.85/",
    doi = "10.18653/v1/2026.findings-eacl.85",
    pages = "1678--1704",
    ISBN = "979-8-89176-386-9"
}
More projects from KAIST MLP Lab →