Clinical evaluation requires the integration of multiple data sources, including a patient’s history, physical examination findings, imaging studies, and laboratory results.^1,2 Physicians must synthesize these diverse data to arrive at accurate diagnoses and guide treatment decisions. To streamline and standardize this process, artificial intelligence (AI) has been increasingly applied to support clinical decision-making. However, most current AI tools are designed for single tasks and rely on isolated data types.^3,4 This narrow focus limits their applicability in the multifaceted context of real-world clinical care.

Multimodal AI refers to machine learning (ML) models capable of processing and integrating information from multiple modalities or types of data.⁵ These modalities can include text, images, audio, video, and other forms of sensory input. By combining complementary information from different sources, these models can compensate for noise, variability, or missing data in any single modality, thereby enhancing both robustness and predictive accuracy. In various medical specialties, multimodal AI has demonstrated promising results in both initial diagnosis and longer-term patient management.^6–8

In otolaryngology, many prior studies have developed AI models based on convolutional neural networks (CNNs) in the analysis of laryngoscopic and endoscopic images and videos, including the classification of various VF pathologies.^9–11 This application of deep learning to endoscopic data, referred to as “videomics” as defined by Paderno et al., offers a promising opportunity to expand access to diagnostic evaluations, especially in low-resource settings.¹² Voice-based algorithms have also been explored, though with generally less robust performance.^13,14 However, few prior studies have evaluated the integration of multiple modalities or compared the performance of multimodal models to unimodal classifiers.^15,16

This study aims to develop a prediction model for VF pathologies by integrating 3 distinct data modalities: stroboscopic video, audio recordings of voice samples extracted from these videos, and clinicodemographic data from patient electronic health records. To our knowledge, this is a novel application of multimodal classification using 3 modalities to classify 3 types of VF pathology. By leveraging the complementary strengths of each data type, our multimodal approach seeks to enhance the classification of 3 categories of VF states, healthy vocal folds (HVF), unilateral vocal fold paralysis (UVFP), and VF lesions, including both benign and malignant pathologies, as compared to single modality models. This novel integrated approach more closely approximates the multifaceted evaluation performed by clinicians, potentially offering improved diagnostic accuracy with potential for clinical utility in laryngological practice. The objective was to explore the utility and benefit of multimodal classification performance compared to audio-only or video-only models in video-stroboscopy.

This work demonstrated that integrating stroboscopic video, audio, and clinicodemographic data in a multimodal DL framework improves the diagnosis of VF pathologies compared to relying on unimodal approaches. The multimodal classifier achieved an absolute accuracy gain of 15.38% over the video-only model (76.92% vs. 61.54%) and 11.54% over the audio-only model (76.92% vs. 65.38%). The observed trend highlights the advantage of combining heterogeneous data sources, as each modality captures different aspects of VF pathology that, when fused, yield more robust diagnostic predictions leveraging cross-modality patterns. This multimodal approach is more representative of the clinical decision-making process by human experts and therefore holds significant promise in providing more reliable results in low validity settings.⁵

Few studies to date have implemented multimodal approaches in otolaryngology, and this is the first study to combine stroboscopic video and audio data with EHR text data for VF pathology classification. Prior work has typically focused on unimodal classifiers, reporting mean accuracies of 89.0% for acoustic studies and 91.3% for imaging studies.¹⁹ Similar to our approach, multimodal application by Kwon et al. and Fang et al. achieved improved classification accuracy with image and voice data, and voice and medical record data, respectively.^15,16 The comparatively lower accuracy of our unimodal models may be attributed to differences in data preprocessing pipelines, modeling architectures for video and audio inputs, and potential institutional variability in data quality, including differences in recording protocols and patient populations. The model’s ability to perform reasonably well despite equipment and mouth-to-microphone distance variability suggests better real-world applicability to diverse clinical settings, as strict standardization, while ideal for model training, can be impractical. Importantly, despite the lower baseline performance of our individual modalities, our multimodal framework achieved a better performance over the best-performing unimodal model than was observed in prior multimodal studies.¹⁵

Several key methodological decisions guided the development of the multimodal model. First, the gated attention mechanism employed an uncertainty-based weighting approach similar to that described by Kendall et. al, allowing the model to dynamically adjust the contribution of each modality based on the estimated uncertainty of its predictions for each sample.²⁰ When analyzing the confusion matrices, this adaptive weighting approach proved particularly valuable for VF lesion cases, where the audio modality often showed higher uncertainty. In these instances, the fusion mechanism appropriately increased the influence of the video modality, which demonstrated stronger discrimination capabilities for lesions. Conversely, when classifying healthy cases, where the video modality showed lower confidence, the model adjusted to leverage the audio features more heavily. This dynamic balancing contributed to the improved performance observed in the multimodal approach. In contrast, clinical decision-making by human experts often relies on fixed or subjective weighting of disparate inputs and may not fully account for complex, non-linear relationships between modalities.⁵ This highlights the potential for multimodal AI to enhance diagnostic accuracy by systematically integrating and weighting diverse sources of information in ways that may be difficult for human cognition to replicate.

Second, raw VLS video recordings, without manual frame selection or extensive preprocessing, were utilized. This approach contrasts with current research that employs careful manual selection of video frames or time-intensive labeling of individual frames to train automated informative frame classifiers.²¹ While our approach yielded lower performance for the video-only classifier relative to other approaches with a mean accuracy of 91%, minimal pre-processing was chosen to enhance clinical applicability by streamlining the data preparation process and reduce reliance on labor-intensive curation.^22,23 To address the limitations of minimal video preprocessing, we employed a transformer-based architecture (pretrained DINOv2), which is effective in modeling long-range dependencies and spatiotemporal patterns. DINOv2’s self-supervised learning capabilities allowed it to learn rich, high-level feature representations from the video data without the need for extensive manual labeling.²⁴ This capability leverages the inherent structure in the data to improve model performance. As delineated by prior uses of DINOv2, transformers can streamline the process of generating high-quality pseudo-labels, which can then be used to train models more effectively.^12,25

Across all classification tasks, the multimodal approach demonstrated the best overall performance. However, in line with existing literature, both the unimodal and multimodal models exhibited reduced performance when evaluated on an external dataset.^14,26,27 Compared to unimodal classifiers, multimodal models are particularly vulnerable to single-modality variations, as discrepancies in any one input can disrupt the integration of cross-modal information. This sensitivity degrades overall performance and limits generalizability across clinical settings. Nonetheless, through the integration of varying data streams, multimodal models can support more accurate, comprehensive decision-making, particularly for non-expert providers who may lack subspecialty expertise. With ongoing efforts to improve external validation and model robustness, multimodal VLS automation holds significant potential to expand access to high-quality diagnostic support in resource-limited settings, where lack of access to specialty referral often leads to delays in the diagnosis and management of laryngeal disorders.⁹

While this proof-of-concept study provides a novel multimodal approach for the integration of video, audio, and clinicodemographic data, it is not without limitations. First, the small sample size reduces statistical power, contributing to wide confidence intervals and limiting the reliability of performance estimates. In addition, the observed accuracy drop in the external validation set highlights the challenges of generalizability across diverse clinical settings. Expanding the size and diversity of validation cohorts, including through prospective data collection and potentially synthetic data generation, will be critical to improving robustness and external validity.

Further, training a clinicodemographic-only model was avoided because the limited variables would likely cause underfitting, preventing the capture of meaningful patterns and rendering it an uninformative benchmark. This contrasts Fang et al.’s successful analysis of a demographics-only model due to a richer dataset containing over 34 detailed features. However, future work should include sensitivity studies to assess the relative contribution of each modality, including clinico-demographic data, to overall model performance, further clarifying the added value of multimodal integration. Additionally, the audio samples were trimmed; however, truncating voice tasks can misrepresent the phonation time and quality, which are important clinical features in UVFP and VF lesions. While our preprocessing pipeline is less labor-intensive than those used in prior studies, it remains time-consuming and should be further optimized in future work. Lastly, although transformer-based video models provide strong feature representations, they lack inherent interpretability; future studies applying model explainability techniques and uncertainty quantification can help clinicians understand predictions and gauge confidence in clinical decision support.

This study successfully developed a multimodal dataset and classifier for diagnosing VF pathologies, demonstrating the potential benefits of integrating stroboscopic video, audio, and clinicodemographic data for automated laryngeal disease classification. The multimodal classifier achieved higher accuracy than the video-only and audio-only models on both the test and external datasets. Addressing variability in external datasets and increasing sample size remain crucial for advancing the clinical applicability of these models. Nevertheless, our findings demonstrate the potential superiority of multimodal approaches over unimodal AI models for voice pathology classification.