Systematic Evaluation of Auto-Encoding and Large Language Model Representations for Capturing Author States and Traits

Details

• Personal Author:
  Singh, Khushboo ; Varadarajan, Vasudha ; Ganesan, Adithya V. ; Nilsson, August H. ; Soni, Nikita ; Mahwish, Syeda ; Chitale, Pranav ; Boyd, Ryan L. ; Ungar, Lyle ; Rosenthal, Richard N. ; Schwartz, H. Andrew
  More +
• Description:
  Large Language Models (LLMs) are increasingly used in human-centered applications, yet their ability to model diverse psychological constructs is not well understood. In this study, we systematically evaluate a range of Transformer-LMs to predict psychological variables across five major dimensions: affect, substance use, mental health, sociodemographics, and personality. Analyses span three temporal levels-short daily text responses about current affect, text aggregated over two-weeks, and user-level text collected over two years-allowing us to examine how each model's strengths align with the underlying stability of different constructs. The findings show that mental health signals emerge as the most accurately predicted dimensions (r=0.6) across all temporal scales. At the daily scale, smaller models like DeBERTa and HaRT often performed better, whereas, at longer scales or with greater context, larger model like Llama3-8B performed the best. Also, aggregating text over the entire study period yielded stronger correlations for outcomes, such as age and income. Overall, these results suggest the importance of selecting appropriate model architectures and temporal aggregation techniques based on the stability and nature of the target variable. Description provided by NIOSH
• Subjects:
  • Artificial Intelligence
  • Mental Health
  • Occupational Health
  • Safety
• Keywords:
  • Artificial intelligence; Psychological responses; Psychological factors; Mental health; Personality traits; Sociodemographic factors; Automation
• Source:
  Findings of the Association for Computational Linguistics: ACL 2025, July 27 - August 1, 2025, Vienna, Austria. ; Kerrville, TX: Association for Computational Linguistics (ACL), 2025 Jul; :18955-18973
• ISBN:
  9798891762565
• Document Type:
  Text
• Funding:
  Cooperative Agreement
• Genre:
  Proceedings
• Place as Subject:
  Dallas Region OSHA ; New York ; New York City Region OSHA ; Pennsylvania ; Philadelphia Region OSHA ; Texas
• CIO:
  National Institute for Occupational Safety and Health (NIOSH)
• Topic:
  Workplace Safety & Health
• Location:
  North America
• Pages in Document:
  21 pdf pages
• NIOSHTIC Number:
  nn:20071674
• Email:
  khusingh@cs.stonybrook.edu
• Federal Fiscal Year:
  2025
• Performing Organization:
  State University of New York, Stony Brook
• Peer Reviewed:
  False
• Start Date:
  20220701
• End Date:
  20260630
• Download URL:
  https://stacks.cdc.gov/view/cdc/259446/cdc_259446_DS1.pdf
• File Type:
  [PDF - 4.59 MB]
• Collection(s):
  National Institute for Occupational Safety and Health
• Main Document Checksum:
  urn:sha-512:17c10bfd81dcac3bda1184a25b847f88ac1d464b931bc8b9ea9480e4df0154cc4b3efed9fcd589956305eeddb8b1eec6eb9a0610c9c8d3d96d92927ff76d665d