Natural Language Response Formats for Assessing Depression and Worry with Large Language Models: A Sequential Evaluation with Model Pre-Registration

Details

• Journal Article:
  Assessment
• Personal Author:
  Gu, Zhuojun ; Kjell, Katarina ; Schwartz, H. Andrew ; Kjell, Oscar
• Description:
  Large language models can transform individuals' mental health descriptions into scores that correlate with rating scales approaching theoretical upper limits. However, such analyses have combined word- and text responses with little known about their differences. We develop response formats ranging from closed-ended to open-ended: (a) select words from lists, write (b) descriptive words, (c) phrases, or (d) texts. Participants answered questions about their depression/worry using the response formats and related rating scales. Language responses were transformed into word embeddings and trained to rating scales. We compare the validity (concurrent, incremental, face, discriminant, and external validity) and reliability (prospective sample and test-retest reliability) of the response formats. Using the Sequential Evaluation with Model Pre-Registration design, machine-learning models were trained on a development dataset (N = 963), and then pre-registered before tested on a prospective sample (N = 145). The pre-registered models demonstrate strong validity and reliability, yielding high accuracy in the prospective sample (r = .60-.79). Additionally, the models demonstrated external validity to self-reported sick-leave/healthcare visits, where the text-format yielded the strongest correlations (being higher/equal to rating scales for 9 of 12 cases). The overall high validity and reliability across formats suggest the possibility of choosing formats according to clinical needs. Description provided by NIOSH
• Subjects:
  • Artificial Intelligence
  • Depression
  • Mental Health
  • Occupational Health
  • Safety
• Keywords:
  • Artificial intelligence
  • Machine learning
  • Psychological factors
  • Mental health
  • Automation
  • Models
  • Psychological testing
  • Author Keywords: artificial intelligence
  • large language models
  • natural language
  More +
• ISSN:
  1073-1911
• Document Type:
  Text
• Funding:
  Cooperative Agreement
• Genre:
  Journal Article
• Place as Subject:
  New York ; New York City Region OSHA
• CIO:
  National Institute for Occupational Safety and Health (NIOSH)
• Topic:
  Workplace Safety & Health
• Location:
  North America
• Pages in Document:
  27 pdf pages
• NIOSHTIC Number:
  nn:20071731
• Citation:
  Assessment 2025 Sep; :Epub ahead of print
• Contact Point Address:
  Zhuojun Gu, Department of Psychology, Lund University, Allhelgona kyrkogata 16a, Lund, Skåne 223 62, Sweden
• Email:
  zhuojun.gu.se@gmail.com
• Federal Fiscal Year:
  2025
• Performing Organization:
  State University of New York, Stony Brook
• Peer Reviewed:
  True
• Start Date:
  2022/07/01
• End Date:
  2026/06/30
• Download URL:
  https://stacks.cdc.gov/view/cdc/259956/cdc_259956_DS1.pdf
• File Type:
  [PDF - 2.45 MB]
• Collection(s):
  National Institute for Occupational Safety and Health
• Main Document Checksum:
  urn:sha-512:bac919aded76f98339e9259f9ac4dd05a97bcf0e60240aad3ed5cdc49a4a96fdfab686b0c4130c09d20191ea7ca36bf565639d68708c36e53d108325368ee082