MentalBench: A DSM-Grounded Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models

24,750 synthetic clinical cases for 23 psychiatric disorders, built on a psychiatrist-validated DSM-5 knowledge graph

arXiv 2026

Hoyun Song1,*, Migyeong Kang2,*, Jisu Shin1, Jihyun Kim2, Chanbi Park3, Hangyeol Yoo4, Jihyun An5, Alice Oh1, Jinyoung Han2,†, KyungTae Lim1,†

1KAIST 2Sungkyunkwan University 3Dongguk University Medical Center 4Seoultech 5Samsung Medical Center

*Equal contribution   †Corresponding authors

Paper (arXiv) PDF Code Dataset
Four case types. Left: the Major Depressive Disorder node (minimum duration 2 weeks, functional impairment required) yields Type I, a professional summary listing tearfulness, emptiness, decreased appetite, and a specific suicide plan, and, with information incompleteness added, Type II, a narrative patient self-report with one symptom missing. Right: the MDD versus Bipolar II differential-diagnosis node (key difference: meeting criteria for a hypomanic episode) yields Type III, overlapping symptoms without the discriminatory rule so both MDD and Bipolar II are correct, and Type IV, which adds the discriminating evidence (brief bursts of energy that never last) so only MDD is correct.
Figure 2. The four MentalBench case types. Types I–II test information incompleteness: a complete professional summary versus a colloquial patient self-report that keeps only the minimum evidence. Types III–IV test diagnostic complexity through differential diagnosis: without the discriminating evidence (Type III) several diagnoses remain consistent, while with it (Type IV) the model must narrow down to a single diagnosis.

Abstract

Large language models (LLMs) have attracted growing interest as supportive tools for psychiatric assessment and clinical decision support. However, existing mental health benchmarks largely rely on social media data or supportive dialogue settings, limiting their ability to assess whether models can apply formal diagnostic criteria and differential diagnostic rules. In this paper, we introduce MentalBench, a benchmark for evaluating whether LLMs can make DSM-grounded psychiatric diagnostic decisions under varying levels of clinical ambiguity. At the core of MentalBench is MentalKG, a psychiatrist-built and validated knowledge graph encoding DSM-5 diagnostic criteria and differential diagnostic rules for 23 psychiatric disorders. Using MentalKG as an expert-curated logical backbone, we generate 24,750 synthetic clinical cases that systematically vary in information completeness and diagnostic complexity, enabling DSM-grounded evaluation. Our experiments show that although state-of-the-art LLMs perform well on noise-free queries that probe DSM-5 knowledge, they struggle to calibrate their confidence when distinguishing between disorders with overlapping symptoms. These findings raise concerns about the reliability of LLMs as psychiatric decision-support tools and highlight the need for more evaluation that reflects the diverse challenges in real-world psychiatric diagnosis.

Keywords Psychiatric diagnosis DSM-5 Differential diagnosis Mental health LLM Clinical decision support Medical LLM benchmark Knowledge graph (MentalKG) Synthetic clinical cases Diagnostic calibration

Benchmark Construction

Four-stage pipeline from the DSM-5 knowledge graph. [1] Seed generation: Type I fully satisfies the diagnostic criteria, Type II partially satisfies them, Type III adds the condition but not the key differences or rules, Type IV adds condition, key differences, and rules; seeds are manually reviewed by experts. [2] Profile augmentation: demographics (age, gender, occupation, marital status) and sub-symptoms such as initial insomnia, early awakening, and hypersomnia form patient profiles with diagnosis, symptom subtypes, and duration. [3] Data synthesis with three LLMs into a professional summary (Type I), patient self-report (Type II), and differential-diagnosis cases with multiple answers (Type III) or a single answer (Type IV). [4] Option selection: four answer options such as Bipolar I, Bipolar II, MDD, PTSD.
Figure 3. Overview of the clinical case generation pipeline for constructing MentalBench: seed generation from MentalKG, profile augmentation, multi-model data synthesis, and option selection.
  1. MentalKG. Mental health experts turn DSM-5 text into a knowledge graph of diagnostic criteria (minimum duration, functional impairment, symptom subtypes) and differential-diagnosis rules (additional conditions, key differences) for 23 psychiatric disorders.
  2. Seed generation. Clinical profiles are sampled from MentalKG: Type 1 seeds satisfy all DSM-5 requirements, Type 2 seeds keep only the minimum evidence, Type 3 seeds omit the discriminating evidence of a differential-diagnosis pair, and Type 4 seeds include it. Two experts validate the seeds (5 seeds per disease for Types 1, 3, 4; 10 for Type 2).
  3. Profile augmentation. Each seed becomes 5 cases with varied symptom subtypes and a demographic identity drawn from 300 combinations of age, gender, occupation, and marital status.
  4. Data synthesis. Three different LLMs write the cases (third-person chart summaries for Type 1, first-person narratives for Types 2–4) to avoid single-generator bias, yielding 24,750 cases: 1,725 Type 1, 3,450 Type 2, 6,525 Type 3, and 13,050 Type 4.
  5. Options and validation. Each case has 4 options with symptom-overlapping distractors from MentalKG; Type 3 has a two-diagnosis answer set. Two experts rated 220 sampled cases on a 5-point scale: overall linguistic naturalness 4.94 / 4.91, diagnostic validity 4.70 / 4.36, and clinical realism 4.83 / 4.87.

Main Results

  • Hard even for frontier models. The best overall accuracy is 62.69 (Claude Sonnet-4.5), followed by GPT 5.1 (62.17); on Type 3 and Type 4 accuracy often drops below 20.
  • Narratives hurt smaller models. Moving from structured charts (Type 1) to patient narratives (Type 2) costs models under 30B an average of 10.58 points, versus 3.74 points for larger models.
  • Opposite calibration failures. Open-source models score higher on ambiguous Type 3 than on clear-cut Type 4 (average gap +22.74) because they over-diagnose and fail to apply exclusion rules; proprietary models show the reverse (average gap −39.60), under-diagnosing when several diagnoses are consistent.
  • Domain fine-tuning is not enough. MentaLLaMA 7B and 13B, fine-tuned from Llama-2 for mental health, reach only 19.65 and 23.45 overall. Underrepresented disorders such as the ADHD and schizophrenia-spectrum cluster collapse most from Type 1 to Type 2.
Table 2. Accuracy (%, exact match) on the four MentalBench case types. Overall is the average weighted by the number of cases per type. (Table 2 in the paper.)
ModelType 1Type 2Type 3Type 4Overall
Qwen 2.5 7B85.8576.8750.7011.5936.12
Qwen 2.5 14B94.1582.2644.7831.8346.55
Qwen 2.5 32B92.8783.0449.2030.8947.08
Qwen 2.5 72B93.8583.1052.7518.2541.66
Qwen 3 8B89.7477.7755.402.5333.09
Qwen 3 14B90.7875.4852.2614.0938.14
Qwen 3 32B90.2083.1348.0327.3744.86
Qwen 3 235B85.3384.1754.1928.6347.06
Gemma 3 4B79.6564.9034.0418.4133.34
Gemma 3 12B92.8178.3248.2815.7938.50
Gemma 3 27B92.8784.0952.519.2036.85
Llama-3.1 8B84.9971.0140.1214.1833.60
Llama-3.1 70B91.9480.8139.7534.9747.09
MentaLLaMA 7B30.9030.672.1925.9819.65
MentaLLaMA 13B52.1746.7518.1016.1723.45
GPT 4o74.6179.9740.2548.3052.43
GPT 5-mini71.0188.469.0177.7260.63
GPT 5.194.3890.4322.4870.3062.17
Claude Haiku-4.594.2087.3929.8765.0060.89
Claude Sonnet-4.594.3291.1614.9974.8462.69
Gemini 2.5-flash93.3985.2727.9460.9657.90
Gemini 2.5-pro93.9188.2035.1659.8259.65

BibTeX

@misc{song2026mentalbenchdsmgroundedbenchmarkevaluating,
      title={MentalBench: A DSM-Grounded Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models},
      author={Hoyun Song and Migyeong Kang and Jisu Shin and Jihyun Kim and Chanbi Park and Hangyeol Yoo and Jihyun An and Alice Oh and Jinyoung Han and KyungTae Lim},
      year={2026},
      eprint={2602.12871},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.12871},
}
More projects from KAIST MLP Lab →