Videolaryngostroboscopy (VLS) is the mainstay of objective laryngeal evaluation in patients with dysphonia and is currently almost exclusively performed in otolaryngology – head and neck surgery practices. Unlike traditional office laryngoscopy with a constant light source, VLS employs a stroboscopic light that flashes faster than the patient’s fundamental frequency. This allows physicians to capture detailed images of vocal folds (VF) at various phases of the glottal cycle, enabling slow-motion visualization of VF oscillations¹. Additionally, stroboscopic videos provide a unique opportunity to integrate high-quality dynamic imaging with synchronized voice recordings.

Compared to state-of-the-art assessment by otolaryngologists with access to VLS, voice evaluation in the primary care setting remains rudimentary, largely relying on patient-reported symptoms and auditory-perceptual assessment by untrained clinicians. This unfortunately often leads to delays in diagnosis and management of laryngeal disorders². Even when patients are referred to specialists, otolaryngologists often face difficulties triaging referrals based on primary care clinician assessment³. Furthermore, access to VLS varies significantly by region⁴, with relative scarcity in the contiguous Mountain, West South Central and East South Central divisions of the United States, where referral for dysphonia evaluation is particularly challenging.

Applications of deep learning (DL), in upper aerodigestive tract endoscopy, dubbed “videomics” by Paderno et al.⁵, have gained interest in the past decade⁶, partly motivated by the desire to improve access to diagnostic exams in surgical deserts. DL techniques using convolutional neural networks (CNN) have been applied for analysis of laryngoscopic images, such as the classification of various VF pathologies, with performance comparable to that of human experts^2,7,8. The overall vision for implementing these algorithms in low-resource settings is to provide clinical decision support (CDS) during endoscopy, potentially to non-experts who could screen for pathologies requiring urgent evaluation by specialists⁹.

DL applications in healthcare have expanded into voice, speech, and respiratory sounds, with rapid expansion during the COVID-19 pandemic. This growing field of data science focused on identification of acoustic biomarkers of health and disease has been recently named “audiomics.”¹⁰ Combined with video data, VLS audio recordings facilitate the integration of videomics and audiomics through multimodal machine learning. Currently, research on DL capabilities for voice-based diagnosis of VF pathologies is inconclusive, primarily due to a lack of large, clinically validated voice datasets that include sociodemographic information. Further, voice prints are considered biometric identifiers under the Health Insurance Portability and Accountability Act (HIPAA), complicating data sharing¹¹.

While CNN-based voice classifiers differentiate between binary states (i.e. healthy and pathological voices) with sensitivity^3,12–14,15, their performance declines when classifying multiple vocal states¹⁶ – suggesting their potential as screening tools and not as diagnostic tools. Their performance in pathology discrimination based on voice is akin to that of auditory-perceptual judgement by experts. In a Taiwanese study, expert laryngologists with more than 10 years of experience demonstrated limited accuracy in distinguishing different VF pathologies based on voice samples of sustained vowel sound /a:/ along with the recording of a Mandarin passage¹⁶, with a maximum overall accuracy of 60%, though superior to that of general otolaryngologists, with a maximum overall accuracy of 40%.

Here, we aimed to investigate the screening and diagnostic capabilities of CNNs trained on voice recordings from VLS. Acoustic classifiers were developed to 1) differentiate voices produced by patients with healthy vocal folds (HVF) and pathological VFs; and 2) discriminate voice produced by patients with HVF, unilateral vocal fold paralysis (UVFP) and vocal folds with lesion (VF lesion). These classifiers were also tested on an external dataset to evaluate generalizability (Figure 1). We hypothesized that these classifiers would 1) distinguish normal from pathological voice with good accuracy, but not perform as well in multi-class discrimination and 2) perform with reduced accuracy on the external dataset.

We chose to focus on HVF, UVFP and VF lesions, as they are clinically associated with significantly different voice outputs. UVFP results in the complete immobility of a vocal fold, imparting a breathy quality to the voice due to inadequate closure of the VFs during phonation¹⁷. VF lesions often limit proper vocal fold closure, causing breathiness, and lead to phase asymmetry, resulting in a grainier voice¹⁸. To our knowledge, this is the first study evaluating the screening and diagnostic capabilities of voice-based CNN models based on acoustic samples extracted from VLS recordings.

The mean age of patients included in the Weill Cornell dataset was 57±19 years, with 122 (58%) females. The mean age of patients included in the Columbia dataset was 51±21 years, with 19 (48%) females. There were no significant differences in age (p=0.0511) or sex (p=0.2255) between the two cohorts. There were no significant differences in the duration of audio files between the HVF, VF lesion, and UVFP cohorts in the Weill Cornell dataset (p= 0.4120).

Here, we investigated the screening and diagnostic capabilities of DL-based classifiers in differentiating between voices produced with HVF and pathological VFs using acoustic data collected from VLS recordings. The binary classifier trained on single-vowel phonation distinguished normal from pathological voice with 80% accuracy. However, the accuracy dropped to 40% when distinguishing voices produced with HVF and voices produced with VF lesions or UVFP. In other words, while there is moderate success in binary classification of healthy versus pathological voices, the performance significantly diminishes in the multi-class classification of HVF, VF lesions, and UVFP. This aligns with the results described by Hu et al., where the model accuracy was 95% for the binary distinction of healthy from pathological voices, but dropped to 67% when discriminating between five classes, including VF lesions and UVFP¹⁶. The performance of our DL classifiers appears similar to human expert aptitude in discriminating different etiologies of dysphonia¹⁶. Consistent with prior findings^16,32,33, the present study suggests that classifiers trained on voice data alone have limited diagnostic utility but may serve as screening tool for general dysphonia.

Though voice alone may have limited diagnostic value, it may bolster the precision of DL-based CDS tools if combined with other modalities, such as imaging or clinicodemographic information. Wang et al. demonstrated that a multimodal learning system combining acoustic signals and medical records outperformed unimodal systems in pathological voice classification³³. Kwon et al. showed that the classification accuracy of glottic cancer improved when image and voice classifiers were combined with ensemble learning³². Such multimodal approaches are more representative of the clinical decision-making process by human experts, and are expected to provide more reliable results in low validity settings³⁴. With DL models applied to VLS, it will be important to explore whether combining audio with video data enhances performance compared to using video input alone.

The voice dataset used for this study has several advantages. First, unlike previous studies that recorded voice data in controlled environments^3,12–16, the present study used voice data collected during office laryngoscopy, i.e. a real-world clinical setting. Hasebe et al. evaluated the impact of Gaussian noise on the performance of voice-based CNN classifiers, finding a substantial decrease in performance with increasing intensities of added noise³⁵. This underscores the challenges of using noisy acoustic data and highlights the value of such data for real-world implementation of AI. Thus, training classifiers on a noisy acoustic dataset representative of the realistic clinical environment is a key strength of our study. Second, each patient only contributed one recording to the dataset. Although this restricted the dataset size, it helped maintain dataset heterogeneity and avoid overrepresenting any individual patient. Finally, only voice samples collected during initial visits or pre-intervention visits were included to accurately represent the pathological conditions.

Analysis of feature importance demonstrated that, across three ML models of increasing complexity, features associated with first to third formants and MFCCs were key contributors to model prediction. In a healthy voice, the fundamental frequency energy is localized at the first two formants of vowel /AH/; this is shown to be affected in dysphonic voices with polyp and adductor spasmodic dysphonia³⁶. Glottal insufficiency is correlated with changes in the intensity region of the first and third formants (F1, F3)³⁷, and previous studies have shown an increase in this frequency region in patients with UVFP^38,39. MFCC-related features, particularly MFCC 0–4, have been shown to be key contributors in ML-based discrimination in voices with vocal fold polyps and UVFP⁴⁰ and distinguishing healthy from pathological voice⁴¹. These coefficients are thought to assess the vocal tract independently of the vocal folds.

Using a large standard acoustic parameter set of 88 features, we demonstrate that machine learning algorithms of increasing complexity can differentiate between pathological from healthy voices with high accuracy, but underperform when discriminating between three classes. Further, both the binary and multi-class classifiers performed poorly on the external validation set. Similarly, the DL binary and multiclass classifiers achieved low performance on the external dataset, suggesting limited generalizability across varied datasets. This also shows that voice DL classifiers, sensitive to external variables such as microphone type, recording specifications, and background noise, may not perform reliably in diverse, real-world clinical settings. The classifiers also showed poor generatability to different voice tasks. The binary model trained on Dataset 1 (longest sustained /i/ from each patient) outperformed the binary models trained on Dataset 2 (sustained /i/, alternating /i/ sniffs, and pitch glides) and Dataset 3 (all voice tasks) on all evaluation metrics. The observed trends may be due to overfitting, a phenomenon in which the model learns the training data too closely and cannot generalize to novel datasets. This effect is exacerbated when the size of the training dataset is small, as in our study. Using a multi-institutional training dataset could have diversified background noise and collection methods, potentially enhancing our classifiers’ generalizability.

Recognizing that a small training dataset and class imbalance can severely impair model performance, we applied augmentation techniques to artificially increase dataset size and introduce heterogenity⁴². However, given the short duration of our audio samples, methods like masking or time shifting did not significantly enhance model performance; pitch augmentation was used instead. Weighted Cross Entropy loss function was employed to address class imbalance in ResNet models, and stratified shuffle split was used for more robust evaluation of ML models. Our models continued to perform poorly on validation despite these techniques, highlighting that dataset size significantly impacts machine learning training and that regularization cannot fully overcome this limitation. Unlike the binary classifiers, which achieved highest accuracy when trained on the smallest dataset (Dataset 1, n=209), the multi-class classifiers performed better with larger training datasets (Dataset 2, n=423; Dataset 3, n=681). This suggests that while the current dataset sufficed for simple binary classification, larger datasets are necessary for more complex multi-class tasks. It is important to note, however, that the multi-class classifier trained on the largest dataset (Dataset 3) still underperformed compared to the binary classifier trained on the smallest dataset (Dataset 1). This underscores the importance of prioritizing dataset quality and diversity over sheer size. Balancing the quality, quantity, and diversity of the training datasets will be the key for improving model performance in audiomics moving forward.

In addition to the small dataset size and lack of heterogeneity, our study had several limitations. The audio samples were derived from a retrospective dataset containing diverse, and sometimes limited, voice tasks during VLS. The examiner’s voice occasionally overlapped with the patient’s voice, which required the audio samples to be trimmed. Truncating voice tasks can misrepresent the phonation time and quality, which are important clinical features in UVFP and VF lesions. To address this, the longest trimmed sample was selected for each patient in Dataset 1 and 2. We also demonstrated that there were no significant differences in the duration of the audio samples between the three classes. Additionally, we excluded patients with multiple VF pathologies. This may not reflect real-world scenarios, where UVFP is often accompanied by lesions and other pathologies.

Further, all audio samples were isolated by a trained non-expert researcher, though 10% of the dataset was verified by two expert laryngologists to ensure data quality. There is also a lack of explainability in the model’s predictions. Although not explicitly shown in the results, there were a few cases that were misclassified with high confidence by the DL algorithms. This emphasizes a need for understanding the model’s decision-making process to shed light on the cause of misclassification. Finally, the poor algorithmic performance on the external dataset may be the result of different microphone set up and background noise. Such real-world limitations are important to incorporate, however, to evaluate the implementation potential of AI algorithms in different clinical settings. Our findings are reinformed by a recent scoping review, demonstrating that 88% of 82 studies using voice data for detection of vocal fold pathology since 2000 were aimed at this binary task⁴³. None of the studies included in the scoping review conducted external validation of their AI models. This lack of external validation is a significant gap in the current literature, and we believe that our study addresses this issue by validating our models on an external dataset. Our study prioritizes bias mitigation, interpretability, and robustness over pure performance optimization. Explainability of models is crucial, as otolaryngologists are more likely to trust ML-CDST output when provided with a specific explanation of its logic⁴⁴. Model agnostic feature importance analysis was used to assess the key acoustic features contributing to model prediction and whether these align with known pathophysiological changes. Further, only the default hyperparameters were used and optimizing for performance gains was avoided to focus on robustness, interpretability, and trustworthiness of the models. Future analyses using feature selection for CNN training and visualization techniques, such as t-distributed stochastic neighbor embedding⁴⁵ and gradient-weighted class activation mapping⁴⁶, could further enhance our efforts by elucidating features in the Mel-spectrogram contributing to model predictions in “black box” neural networks⁴⁷, enhancing interpretability and trust in AI-based diagnostic tools.

DL-based binary and multiclass classifiers were developed to discriminate normal voices from voices with VF lesions and UVFP. Although the algorithm demonstrated high discrimination between normal and pathological voice, its ability to discriminate between HVF, VF lesion, and UVFP was mediocre. Both models generalized poorly to the external validation dataset, suggesting their limited utility in heterogeneous healthcare settings. Acoustic features associated with prediction were elucidated to enhance the interpretability and reliability of the models. Our findings highlight that the voice data collected during office stroboscopy can serve as an accurate screening tool but lacks the reliability for definitive diagnosis, and that training data from diverse real-world setting is critical to ensure generalizability of voice-based DL classifiers. Future work should explore multimodal approaches combining voice and VLS imaging, and employ methods to enhance model explainability.