Unveiling the Performance Gap Between Deep Text Mining and Human Coders in Environmental Information Extraction

Comparing a BiLSTM text-mining classifier with trained human coders on Korean environmental news

International Journal of Human–Computer Interaction 2026

JungJin Kim1, Kyungtae Lim2, Jan Adamowski3, Hanseok Jeong1,4

1Institute of Environmental Technology, Seoul National University of Science and Technology 2Graduate School of Culture Technology, KAIST 3Department of Bioresource Engineering, McGill University 4Department of Environmental Engineering, Seoul National University of Science and Technology

Paper (Int. J. Human–Computer Interaction)

Abstract

Automated text mining is increasingly used in government monitoring and policy-oriented workflows, but its value depends on how closely automated labels align with human interpretation. This study compares a BiLSTM classifier with two trained human coders in classifying Korean environmental news across five categories: Climate, Air, Water, Waste/Garbage, and Others. Using a time-separated evaluation (2006–2017 training; 2018–2022 testing), the classifier achieved high accuracy (> 0.98), but lower agreement with human coders (F1 = 0.76) than coder-to-coder agreement (F1 = 0.89). Discrepancies were most evident in low-text or image-dominant articles, where human integrative reasoning captured nuanced environmental information that the classifier missed. Cofusion was highest where semantic overlap was common (e.g., Climate vs. Air), whereas Water exhibited stable agreement. These findings highlight the practical importance of human-in-the-loop validation, encompassing uncertainty-based routing and periodic agreement audits, for accountable and transparent environmental information systems. Although limited to a text-based corpus, the framework provides a foundation for future multimodal and multilingual extensions.

Keywords Text mining Human coders Human–machine agreement Inter-coder agreement BiLSTM News classification Korean environmental news Environmental information extraction Content analysis Human-in-the-loop

Main Findings

  • Setup. A BiLSTM classifier and two trained human coders label Korean environmental news into five categories (Climate, Air, Water, Waste/Garbage, Others), with a time-separated split: training on 2006–2017 articles, testing on 2018–2022 articles.
  • High accuracy, lower human agreement. The classifier achieves accuracy above 0.98, but its agreement with the human coders (F1 = 0.76) is lower than the agreement between the two coders (F1 = 0.89).
  • Where they disagree. Discrepancies concentrate in low-text or image-dominant articles, and confusion is highest between semantically overlapping categories such as Climate vs. Air, while Water shows stable agreement.
  • Implication. The authors recommend human-in-the-loop validation, including uncertainty-based routing and periodic agreement audits, for environmental information systems.

This summary is based on the publisher’s abstract; see the paper for full details.

BibTeX

@article{Kim_2026,
  title={Unveiling the Performance Gap Between Deep Text Mining and Human Coders in Environmental Information Extraction},
  ISSN={1532-7590},
  url={http://dx.doi.org/10.1080/10447318.2026.2669061},
  DOI={10.1080/10447318.2026.2669061},
  journal={International Journal of Human–Computer Interaction},
  publisher={Informa UK Limited},
  author={Kim, JungJin and Lim, Kyungtae and Adamowski, Jan and Jeong, Hanseok},
  year={2026},
  month=May,
  pages={1–18}
}
More projects from KAIST MLP Lab →