ScholarBench: A Bilingual Benchmark for Abstraction, Comprehension, and Reasoning Evaluation in Academic Contexts

Noh, Dongwon; Koh, Donghyeok; Yuk, Junghun; Kim, Gyuwan; Lee, Jae Yong; Lim, KyungTae; Park, Cheoneum

doi:10.18653/v1/2025.findings-emnlp.465

An English–Korean benchmark of academic-paper questions across eight research domains

Findings of EMNLP 2025

Dongwon Noh^1,*, Donghyeok Koh^1,*, Junghun Yuk^2,*, Gyuwan Kim^3,*, Jae Yong Lee⁴, KyungTae Lim^2,†, Cheoneum Park^1,†

¹HBNU ²KAIST ³UCSB ⁴KISTI

^*Equal contribution ^†Corresponding authors

Abstract

Prior benchmarks for evaluating the domain-specific knowledge of large language models (LLMs) lack the scalability to handle complex academic tasks. To address this, we introduce ScholarBench, a benchmark centered on deep expert knowledge and complex academic problem-solving, which evaluates the academic reasoning ability of LLMs and is constructed through a three-step process. ScholarBench targets more specialized and logically complex contexts derived from academic literature, encompassing five distinct problem types. Unlike prior benchmarks, ScholarBench evaluates the abstraction, comprehension, and reasoning capabilities of LLMs across eight distinct research domains. To ensure high-quality evaluation data, we define category-specific example attributes and design questions that are aligned with the characteristic research methodologies and discourse structures of each domain. Additionally, this benchmark operates as an English-Korean bilingual dataset, facilitating simultaneous evaluation for linguistic capabilities of LLMs in both languages. The benchmark comprises 5,031 examples in Korean and 5,309 in English, with even state-of-the-art models like o3-mini achieving an average evaluation score of only 0.543, demonstrating the challenging nature of this benchmark.

Keywords Academic benchmark Scholarly QA LLM evaluation Domain-specific knowledge Bilingual (English–Korean) Korean LLM benchmark Academic reasoning Summarization Reading comprehension

Benchmark Construction

Three-step data construction pipeline: Step 1 task design, crawling and filtering of open-access papers by citations and recency; Step 2 GPT-4o generates summary, short-answer, multiple-choice, multiple-select and true/false questions from a paragraph, shown with English and Korean examples; Step 3 GPT-4o regenerates questions with two-shot samples, followed by human review and revision into ScholarBench (En and Ko).

Figure 3. Data construction pipeline. English example questions are generated from Attention Is All You Need; Korean examples from a Korean paper on CNN-LSTM handwritten digit recognition.

ScholarBench tests three abilities: abstraction (summarizing an academic paragraph), reasoning (answering from parametric knowledge in a closed-book setting) and comprehension (answering when the source paragraph is given, open-book). Every paragraph yields five question types: summarization, short answer, multiple choice, multiple selection, and true/false.

Task design and collection. Eight academic categories with 63 English and 65 Korean attributes are defined. English sources are 200 articles from 80 open-access journals ranked in the top 10 by Google Scholar H5-index; Korean sources are 1,916 articles from 138 journals selected by KCI citation index.
Question generation. GPT-4o generates the five question types from each source paragraph.
Refinement and review. Two sampled outputs are used as few-shot examples to refine generation, then 15 reviewers (11 English, 4 Korean) check the paragraph, verify each answer, and rate quality. Inter-annotator Kappa is 0.621 for English and 0.742 for Korean.

Summarization is scored with ROUGE-1/2/L, short answers with BERTScore, and multiple-choice, multiple-selection (2, 3 or 4 correct options: A-2, A-3, A-4) and true/false questions with accuracy.

Main Results

Size. 5,309 English and 5,031 Korean examples across eight categories and five question types; 18.7% of the questions are parallel English–Korean pairs.
Hard for current LLMs. In the closed-book setting the best average score is 0.543 (o3-mini), followed by GPT-4o (0.534), Llama-70b (0.530) and Qwen-72b (0.529).
No single best model. GPT-4o leads ROUGE-1 (0.417), Mistral-24b leads ROUGE-2/L (0.159 / 0.359), Qwen-72b leads MCQ (0.755) and true/false (0.811), and Qwen-32b-reasoning leads three-answer multiple selection (0.640).
Context helps. With the source paragraph provided (open-book), average scores rise, e.g. GPT-4o from 0.534 to 0.629 and o3-mini from 0.543 to 0.609; Qwen-72b reaches 0.900 MCQ accuracy.
Bilingual gap. On the parallel subset, most models score higher in Korean than in English, while some global models show the opposite pattern.

**Table 2.** Overall closed-book results on ScholarBench. R-1/R-2/R-L are ROUGE scores for summarization (the only task given the paragraph), BERTScore is for short answers, and A-2/A-3/A-4 are multiple-selection accuracy with 2, 3 or 4 correct options. Avg is the average over all listed metrics. Bold marks the best score per column.
Model	Summarization			Short Answer	Multiple Selection			MCQ	Boolean	Avg
Model	R-1	R-2	R-L	BERTScore	A-2	A-3	A-4	MCQ	Boolean	Avg
o3-mini	0.392	0.130	0.320	0.860	0.666	0.518	0.482	0.728	0.786	0.543
o1-mini	0.409	0.151	0.348	0.857	0.567	0.469	0.436	0.702	0.771	0.523
GPT-4o	0.417	0.151	0.356	0.857	0.586	0.497	0.465	0.736	0.743	0.534
Qwen-72b	0.396	0.151	0.345	0.859	0.522	0.468	0.452	0.755	0.811	0.529
Llama-70b	0.397	0.146	0.341	0.858	0.578	0.485	0.455	0.746	0.769	0.530
Bllossom-70b	0.349	0.129	0.299	0.848	0.650	0.463	0.440	0.683	0.724	0.509
Qwen-32b-reasoning	0.350	0.116	0.303	0.853	0.539	0.640	0.423	0.721	0.793	0.527
Exaone-32b	0.321	0.094	0.267	0.852	0.590	0.466	0.431	0.713	0.751	0.498
Exaone-32b-reasoning	0.316	0.092	0.267	0.847	0.492	0.397	0.359	0.686	0.667	0.458
Gemma2-27b	0.329	0.117	0.283	0.856	0.577	0.479	0.453	0.707	0.796	0.511
Mistral-24b	0.414	0.159	0.359	0.853	0.584	0.488	0.458	0.694	0.696	0.523
Gemma2-9b	0.294	0.096	0.248	0.851	0.556	0.520	0.444	0.684	0.774	0.496
Exaone-8b	0.317	0.092	0.265	0.846	0.577	0.417	0.386	0.692	0.756	0.483
Mistral-8b	0.402	0.151	0.350	0.842	0.504	0.374	0.355	0.656	0.582	0.468
Llama-8b	0.381	0.136	0.327	0.845	0.501	0.419	0.395	0.658	0.556	0.469
Bllossom-8b	0.346	0.129	0.301	0.844	0.537	0.419	0.383	0.633	0.581	0.464
Qwen-7b	0.388	0.144	0.338	0.847	0.559	0.452	0.423	0.699	0.756	0.512

@inproceedings{noh-etal-2025-scholarbench, title = "{S}cholar{B}ench: A Bilingual Benchmark for Abstraction, Comprehension, and Reasoning Evaluation in Academic Contexts", author = "Noh, Dongwon and Koh, Donghyeok and Yuk, Junghun and Kim, Gyuwan and Lee, Jae Yong and Lim, KyungTae and Park, Cheoneum", editor = "Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet", booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EMNLP} 2025", month = nov, year = "2025", address = "Suzhou, China", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2025.findings-emnlp.465/", doi = "10.18653/v1/2025.findings-emnlp.465", pages = "8750--8782", ISBN = "979-8-89176-335-7" }

ScholarBench: A Bilingual Benchmark for Abstraction, Comprehension, and Reasoning Evaluation in Academic Contexts

Abstract

Benchmark Construction

Main Results

BibTeX