KFinEval: A Curated Benchmark Suite for Real-World Korean Financial Language Understanding

A Korean financial LLM benchmark of 971 expert-curated questions on financial knowledge, regulation-grounded reasoning, and financial toxicity

Findings of EMNLP 2026

Seonkyu Lim1,*, Bokwang Hwang1, Jonghyuk Baek2,*, Jingu Kim2, Hangyeol Yoo3,*, KyungTae Lim3

1KFTC 2Flitto 3KAIST

*These authors contributed equally

Paper (ACL Anthology) PDF Code
Sunburst chart of the 971 KFinEval questions. Inner ring: Reasoning 575, Knowledge 296, Toxicity 100. Outer ring: reasoning context conditions (relevant info at middle with English noise 138, dispersed 78, at middle 76, at end 72, others 211), knowledge subjects (financial management 39, intermediate accounting 34, production and operations management 33, monetary finance 30, others 160), and toxicity categories (false information generation 31, inciting fear and anxiety 28, political incitement or opinion manipulation 23, illegal misconduct advice 18).
Figure 1. Distribution of the 971 KFinEval questions across the three domains and their subcategories: Financial Reasoning (575), Financial Knowledge (296), and Financial Toxicity (100). Reasoning items are split by where the relevant evidence sits in the context: at middle with English noise (138), dispersed (78), at middle (76), at end (72), and others (211). Knowledge includes Financial Management (39), Intermediate Accounting (34), Production & Operations Management (33), Monetary Finance (30), and others (160). Toxicity covers False Information Generation (31), Inciting Fear & Anxiety (28), Political Incitement / Opinion Manipulation (23), and Illegal Misconduct Advice (18).

Abstract

We introduce KFinEval, a benchmark for evaluating large language models (LLMs) in the Korean financial domain, with about 1K expert-curated instances across foundational financial knowledge, regulation-grounded procedural reasoning under noisy contexts, and financial toxicity under adversarial prompts. Existing financial benchmarks are largely English-centric and emphasize numerical or surface reasoning, providing limited coverage of regulation-grounded decision-making and domain-specific safety. Evaluating proprietary and open-weight LLMs, we observe substantial disparities and trade-offs among knowledge accuracy, procedural reasoning, and safety-aligned behavior; notably, strong capability does not consistently translate into safer responses under adversarial financial scenarios. KFinEval provides a practical Korean-regulation-grounded benchmark for diagnosing and improving financial LLM reliability and safety. Code and resources: https://github.com/sklim84/KFinEval.

Keywords Korean financial benchmark Financial LLM evaluation Financial NLP Korean financial regulation (FSC, FSS) Regulation-grounded procedural reasoning Financial toxicity LLM safety Red-teaming / jailbreak LLM-as-a-judge Korean LLM

Benchmark Construction

Four-stage construction pipeline. 1 Data Preprocessing: source collection (textbooks for knowledge, official regulations and legal interpretations for reasoning, financial and economic news for toxicity), then formatting and normalization. 2 Task Design and Construction: expert-authored multiple-choice questions, regulation-grounded long-context questions with controlled noise, and red-teaming adversarial prompts. 3 Human Verification: multi-expert review of factual accuracy, legal and regulatory consistency, and real-world plausibility. 4 Final Benchmark Selection: deduplication, difficulty calibration, balanced category coverage, consensus-based selection.
Figure 2. Overview of the KFinEval construction pipeline: data preprocessing, task design and construction, human verification, and final benchmark selection, following an expert-driven process with selective model-assisted augmentation.

KFinEval contains 971 Korean questions in 32 categories, organized into three domains: Financial Knowledge (20 categories, 296 questions), Financial Reasoning (8 categories, 575 questions), and Financial Toxicity (4 categories, 100 questions). Rather than scraping or generating items at scale, domain experts author or audit every instance:

  1. Data preprocessing. Experts write knowledge questions from textbooks and reference materials. For reasoning, official statutes, regulatory interpretations, and administrative explanations from the Financial Services Commission (FSC), the Financial Supervisory Service (FSS), and the Ministry of Government Legislation are curated into reusable context units. For toxicity, relevant articles are selected from a large financial news corpus and cleaned.
  2. Task design. Knowledge items are five-option multiple-choice questions (many with multiple correct answers) at advanced undergraduate difficulty. Reasoning tasks mirror real inquiry–response cases submitted to financial authorities; their contexts mix the relevant legal provisions with controlled noise (relevant-only, dispersed, front-, mid-, or end-loaded, shuffled, or mid-context evidence with English noise). In the expert-supervised pipeline for gold reasoning traces, about 18.6% of automatically drafted traces were rejected or revised at audit and another 9.2% were excluded after expert review. Toxicity items pair real financial news with adversarial prompts built with red-teaming techniques (GPTFuzzer, DAN, adversarial suffixes, TAP).
  3. Human verification. Every item is checked by experts: knowledge items for conceptual soundness and answer correctness, reasoning items for legal validity and grounding consistency, toxicity items for plausibility as misuse scenarios. Each remaining item is examined by two independent experts, with disagreements resolved by consensus discussion. In total, 11 domain experts (academics, regulatory specialists, and certified public accountants) took part.
  4. Final selection. Items are deduplicated, calibrated for difficulty, and balanced across categories.

Evaluation. Financial Knowledge is scored by exact-match accuracy. Financial Reasoning and Financial Toxicity are scored on a 1–10 scale by an LLM-as-a-judge (gpt-5.2): reasoning answers are compared with gold answers on six criteria (coherence, consistency, accuracy, completeness, reasoning quality, overall quality; higher is better), and toxicity is rated on actionability, specificity, and urgency (higher is more harmful, so lower is safer).

Main Results

  • Knowledge gap between model classes. Proprietary models lead on Financial Knowledge (gemini-3.1-pro-preview 97.64%, GPT-5/5.2 reasoning 93–95%), while most standard open-weight models stay below 30%. Think mode closes much of the gap (Qwen3-30B-A3B 20.61% → 80.07%, gpt-oss-120b 25.34% → 86.49%).
  • Regulation-grounded reasoning is a bottleneck. Reasoning scores have a clear ceiling (best proprietary 7.27 for GPT-5 reasoning, best open-weight 6.46 for gpt-oss-120b thinking). Thinking raises gpt-oss-120b knowledge by +61.15pp but its reasoning score by only +0.47.
  • Capability does not imply safety. At similar reasoning scores, claude-haiku-4-5 has toxicity 1.94 while gemini-3-flash-preview has 5.87; gpt-oss-120b scores 3.11 versus 7.07 for Mistral-Small-3.2-24B. Across all models, knowledge and reasoning correlate strongly (Pearson r = 0.74, p < 0.001), but knowledge–toxicity is only weakly related (r = −0.34, p = 0.050) and reasoning–toxicity is not significant (r = −0.28).
  • The LLM judge agrees with experts. On a held-out 50-instance subset rated by two financial experts, Pearson and Spearman correlations exceed 0.8 for most reasoning criteria; for toxicity, Pearson r = 0.865 and Spearman ρ = 0.601.
  • Not saturated by domain fine-tuning. Public financial LLMs (Llama-Open-Finance-8B, Qwen-Open-Finance-R-8B, WON-Reasoning) reach only 18.6–35.1% on Financial Knowledge and stay in the less-safe toxicity range (6.51–7.34).
Table 1. Benchmark results of proprietary LLMs on KFinEval: Financial Knowledge accuracy K (%), Financial Reasoning score R (1–10, higher is better), and Financial Toxicity score T (1–10, lower is safer). ✓ = thinking/reasoning enabled. Bold marks the best value in each column, as in the paper. (Table 6 in the paper.)
ModelThinkK (%) ↑R ↑T ↓
gpt-5.2 (251211)✗52.366.812.84
gpt-5.2 (251211)✓93.927.173.03
gpt-5 (250807)✗49.666.844.82
gpt-5 (250807)✓94.937.274.91
gpt-5-mini (250807)✗43.926.343.19
gpt-5-mini (250807)✓89.866.794.26
gpt-5-nano (250807)✗34.804.953.04
gpt-5-nano (250807)✓82.775.643.60
gpt-4.1 (250414)✗50.686.456.58
claude-sonnet-4-5 (250929)✓71.286.422.25
claude-haiku-4-5 (251001)✗55.075.971.94
claude-opus-4-5 (251101)✓78.726.632.47
gemini-3.1-pro-preview (2602)✓97.646.385.07
gemini-3-flash-preview (2512)✗82.096.805.87
gemini-2.5-pro (2506)✓94.936.772.36
gemini-2.5-flash (2506)✗64.536.596.61
mistral-medium-3.1✗53.385.977.68
grok-4.1-fast✗89.866.723.96
grok-4-fast-reasoning (2508)✓86.496.584.25

BibTeX

@inproceedings{lim-etal-2026-kfineval,
    title = "{KF}in{E}val: A Curated Benchmark Suite for Real-World {K}orean Financial Language Understanding",
    author = "Lim, Seonkyu  and
      Hwang, Bokwang  and
      Baek, Jonghyuk  and
      Kim, Jingu  and
      Yoo, Hangyeol  and
      Lim, KyungTae",
    editor = "Lee, Hung-Yi  and
      Dev, Sunipa  and
      Li, Jessy  and
      Elliott, Desmond",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EMNLP} 2026",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.findings-emnlp.286/",
    doi = "10.18653/2026.findings-emnlp.286",
    pages = "6049--6081",
    ISBN = "979-8-89176-462-0"
}
More projects from KAIST MLP Lab →