Congenital heart defects (CHD), characterized by structural heart and thoracic vessel abnormalities, are the most common birth defects in infants (Mai et al., 2019), with a prevalence of 8 per 1,000 live births in the U.S. (Hoffman & Kaplan, 2002; Reller et al., 2008), representing a leading cause of infant mortality (Almli et al., 2017). CHD represents a spectrum of structural heart defects affecting valves, chambers, and blood vessels, leading to a heterogenous spectrum of cardiac and non-cardiac complications (Lui et al., 2019). Some present with CHD in infancy (Oster et al., 2013), and others present in adulthood (van der Bom et al., 2011). Approximately 1.4 million adults are living with CHD in the U.S., which is more than estimated pediatric CHD cases (Gilboa et al., 2016). CHD patients may develop late complications including atrial fibrillation, hypertension, heart failure, and other cardiac-related conditions, which significantly increase morbidity and mortality (Billett et al., 2008; Kuehl et al., 1999; Lewis et al., 2023; Mahle et al., 2009; Menachem et al., 2020; Yang et al., 2002). While advancements in medical screenings and treatments have improved survival of individuals with CHD, these individuals face substantial late morbidity and premature mortality (Gilboa et al., 2016; Liberman et al., 2023; Massin & Dessy, 2006; Müller et al., 2022). Numerous factors affect late outcomes, including defect anatomy, type of repair, number and type of interventions, age, access to healthcare, and the treatment plan (Bhatt et al., 2015; Brida & Gatzoulis, 2019; Stout et al., 2019).

Population-based CHD surveillance often relies upon International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) and Tenth Revision, Clinical Modification (ICD-10-CM) codes, within extensive administrative and clinical datasets to estimate CHD prevalence, healthcare utilization, and various health outcomes (Ivey et al., 2024; Glidewell et al., 2021; Lui et al., 2022). The accuracy of ICD codes to identify CHD in administrative data can differ depending on age, medical practices, healthcare systems, and geographic regions; consequently, CHD ICD codes can have low positive predictive values (PPV), which describe the percentage of people who actually have CHD among those who have documented CHD codes. Some CHD ICD codes lack specificity and correspond to more than one CHD diagnosis, and some have high false positive (FP) rates ineffectively distinguishing individuals with a heart defect from those without a CHD (Agarwal et al., 2016; Broberg et al., 2015; Khan et al., 2018; Rodriguez et al., 2018, 2022). For example, the ICD-9-CM code 745.5, used to identify both secundum atrial septal defects (ASD) and patent foramen ovale (PFO), exhibits a high FP rate for CHD where only 3 in 10 adults with that code have a true CHD (Rodriguez et al., 2018). Similarly, high FP rates have been observed for subaortic stenosis and pulmonary valve stenosis (Khan et al., 2018), and the FP rate for ICD-9-CM codes related to shunt lesions has been reported to be around 50% (Broberg et al., 2015). From a healthcare system perspective, FPs may result from use of “rule out” codes when ordering tests, applying CHD ICD codes to other heritable conditions or from data entry errors. Limiting datasets to patients with severe CHD codes improves the PPV of ICD codes, but also restricts the understanding of outcomes in patients with non-severe defects, such as bicuspid aortic valves, which can still lead to considerable morbidity (Udholm et al., 2019; Wallace et al., 2022).

Our previous research has found that the ability to identify people with CHD in administrative and clinical data can be significantly improved by performing machine learning (ML) methods without restricting to the most severe cases (Shi et al., 2023). ML involves the creation of computer algorithms that enhance task performance by learning and adapting from data. ML has found numerous applications in healthcare delivery, including computer-assisted diagnostics, risk prediction and stratification, clinical decision support, deep phenotyping, and precision medicine (Jone et al., 2022). The recent exponential growth of ML applications in healthcare can be attributed to the advancements in computational power, which now enable the processing of vast amounts of complex reference data at high speeds.

Using data from combined U.S.-based clinical and administrative sources, we aimed to develop and test generalized ML models that improve the accuracy of identifying CHD using various training approaches. Furthermore, this study aimed to identify the most crucial features that contribute to accurate identification of CHD.

Key contributions of this work include (1) the development and validation of a generalized ML pipeline for CHD prediction, which has shown robust performance across multiple clinical sites and holds significant promise for application on external datasets. By incorporating a transfer learning approach, the model dynamically adapts as more data becomes available, improving generalization and mitigating the risk of overfitting. This adaptability enhances the model’s robustness and scalability, making it more effective in diverse clinical settings and with expanding datasets. Furthermore, the pipeline (2) addresses key limitations of traditional ICD-based CHD classifications, which often have low PPV and high FP rates.

This study utilized training and validation data from four datasets, collected under a Cooperative Agreement with the Centers for Disease Control and Prevention (CDC-RFA-DD19–1202B) for CHD surveillance during 2010–2019. The study received Institutional Review Board approval on August 26, 2020 (STUDY00001030) and was granted a complete waiver of Health Insurance Portability and Accountability Act (HIPAA) authorization and informed consent. Participating sites were responsible for their respective Institutional Review Board approval. The cohort for case validation consisted of individuals born between 1/1/1955 and 12/31/2019, who were identified with a healthcare encounter between 1/1/2010–12/31/2019. The ICD codes used to define the CHD cohort are in Supplemental Table 1. Throughout the manuscript, the term “cases” will refer to identified individuals with CHD-related healthcare encounters between 1/1/2010–12/31/2019, including those who may not actually have a CHD.

The findings from this study are clinically relevant for CHD surveillance as FP cases in a dataset may lead to inaccurate conclusions. For example, if a CHD code for secundum atrial septal defect/patent foramen ovale is used to order an echocardiogram for someone with a stroke who turns out not to have a CHD, one could draw erroneous conclusions about the prevalence of stroke in the CHD population. FN cases may lead to an incomplete picture of the population. The PPV of CHD ICD codes alone (baseline) varied from 53.2% (KPGA) to 84.0% (CHFT) across site datasets and a PPV of 71.6% for a dataset combining all sites. By developing and employing an XGBoost machine learning model for the present data, we were able to increase the PPV from 72% to 95% overall, while maintaining the NPV below 54%, and for each site, PPV improved by at least 10 percentage points. Therefore, ML revealed to be an effective tool for improving the accuracy and reducing FP, of identifying individuals with CHD in clinical and administrative data using only features based on ICD-9 or ICD-10 codes. A comprehensive set of clinical and procedural features (2010–2019 encounter-level data) was developed (Supplemental Table 2; Figures 2 and 3) to capture patients’ medical histories and healthcare utilization, facilitating accurate CHD identification. The highest baseline PPV occurred for cases from a tertiary care referral system with legacy medical records with cardiologist-entered diagnoses, which perhaps lead to a more accurate diagnosis by specialists or a more complex CHD population within a tertiary referral dataset

In a previous paper, XGBoost outperformed logistic regression, Gaussian Naive Bayes, and random forest models when using clinical and administrative data to identify CHD cases (Shi et al., 2023). Using an XGBoost model and the transfer learning strategy for the current data, each site’s PPV was improved by at least 10 percentage points while maintaining F1-scores between 70% and 93%, suggesting that the model can adapt to data from new sites if a fraction of validated samples are available for training. Future studies implementing this strategy to introduce new site data should ensure an appropriate balance between data added to the training set and data reserved for testing. The cutoffs for splitting the data may need to be adjusted accordingly.

An operational threshold can be chosen to emphasize improvement in PPV or the reduction of FNs. In this scenario, the PPV was set at 95% on the curve. Beyond this operational point, elevating PPV results in an exponential rise in FN rates. However, future studies using this model can choose other operation points along the PPV-FNR curve in Figure 5 to set as the thresholds depending on the trade-off between PPV and FNR that are deemed acceptable for surveillance or research purposes.

While prior research has applied ML to assess CHD risk factors and predict the risk of CHD in infants during pregnancy (Luo et al., 2017; Rani & Masood, 2020), this study follows previous work on development of a generalized ML pipeline for the detection of true CHD cases within administrative records using both ICD-9-CM and ICD-10-CM across 4 distinct validated datasets (Shi et al., 2023). In both studies, the primary aim was to create a standardized ML pipeline to distinguish individuals with true CHD from those with CHD codes but no CHD, with the ultimate goal of enhancing CHD surveillance, understanding healthcare utilizations, and improving patient outcomes. In this study, a transfer learning strategy was added to demonstrate how models can potentially adapt to data from new sites.

One other recent publication sought to improve the identification of CHD cases in administrative data through ML(Marelli et al., 2024). Marelli et al. developed a ML model using administrative data collected continuously from 1983 to 2000 for 19,187 Canadian patients with at least 1 CHD-related ICD-9 diagnosis and/or a surgical procedure documented by any clinician other than a cardiovascular specialist. Models were trained on a dataset with labels (true CHD or no CHD) that had been generated by a time-consuming, but highly accurate algorithm previously validated in the same data and developed by clinicians. By improving timeliness of CHD classification and achieving a 99% F1-score in evaluation, this study similarly concluded that ML is a useful tool to identify CHD cases in administrative data. However, generalizability of the model to U.S.-based data may present a challenge given the differences in patient access to healthcare and in the continuity of collected healthcare data when comparing a universal healthcare system (Canada) to the U.S. healthcare system. Additionally, some U.S. administrative databases may not reliably capture cardiac specialty provider type.

By leveraging diverse clinical and administrative data sources, the model improves CHD identification accuracy. Unlike previous methods, which were limited to severe cases, this approach broadens the scope of CHD detection, capturing both severe and milder forms of the condition. This broad applicability not only enhances the accuracy of CHD detection, but also lays the groundwork for extending the pipeline to other conditions that require multi-site data integration. In addition to improving prediction accuracy, this study identifies key predictive features that contribute to accurate CHD classification, enhancing the interpretability and clinical relevance of the model. The generalized nature of the pipeline also provides a flexible framework for adaptation to other healthcare datasets, underscoring its potential for widespread utility in CHD surveillance and the detection of similar complex conditions. This work establishes a scalable, high-performance solution for CHD identification that can drive more effective CHD surveillance, facilitate early intervention, and support improved long-term care strategies for affected populations.

Traditional machine learning (ML) models were chosen for this study, despite the current popularity of advanced techniques like deep learning (DL) and artificial intelligence (AI), due to practical demands of clinical applications. First, traditional ML models offer greater interpretability, which is especially important in healthcare contexts where the transparency of diagnostic criteria is crucial. Models like XGBoost or Random Forest allow clinicians to understand the specific features influencing the classification of CHD cases, fostering trust and aiding in clinical decision-making. Additionally, the dataset used in this study, although substantial, does not reach the size or complexity that typically warrants the use of deep learning, which thrives on vast and often unstructured data. Traditional ML methods are more suitable for medium-sized, structured datasets, and algorithms like XGBoost are particularly adept at feature selection, efficiently narrowing down a large set of variables—such as reducing 2,105 Clinical Classification Software (CCS) features to 137 critical ones. This dimensionality reduction is essential for maintaining high performance without sacrificing interpretability. Finally, the objective of maximizing PPV for CHD identification aligns well with the strengths of traditional ML methods. By tuning these models to prioritize PPV while balancing the FN rate, the study achieved a PPV of 95% across data from 4 different sites, demonstrating the effectiveness of traditional ML in meeting the high-accuracy demands of clinical applications.

ML considers features in the context of other features and whether those features have a positive or negative impact on the classification, as quantified by the SHAP value. These features act as either positive or negative predictors for accurate CHD classification, with some related to more frequent classification as a TP and some as a FP. Cases were more likely to be classified as TP when the number of encounters with codes related to CCS categories for cardiac anomalies, other congenital anomalies of the heart, congenital insufficiency of aortic valve, ventricular septal defect, and congenital anomalies and CCSR category codes for circulatory congenital anomalies increased. Conversely, cases were more likely to be classified as FP as the number of encounters with CCS category codes for atrial septal defect, hypertension, factors influencing healthcare, and cardiac dysrhythmias increased. This feature reduction process prevents the model from being a ‘black box,’ where the variables influencing the algorithm’s outcomes are uncertain. By utilizing overlapping features identified as important across all datasets, the model’s generalizability is reserved and reliance on a model that performs well in one dataset, but cannot replicate results in another dataset is avoided.

Principal component analysis (PCA) was attempted on all features, but more than 60 principal components would be required to account for 90% of the total variance contribution, rendering PCA an ineffective method for feature reduction; the dataset proved to be too complex to be adequately represented by a few principal components. Therefore, an alternative approach was taken to reduce the model to the essential features specific to each site.

The limited sample size for both SC and KGPA may be a factor contributing to the potential for further improvement in performance. Limitations in the availability and diversity of medical records for abstraction may have resulted in misclassification of some TP as FP, especially within the KPGA data since its system does not incorporate data from external records. Unlike CHFT and Emory/CHOA, the KPGA data do not draw medical record information from other data sources that the medical record abstractors use when confirming that case has CHD; thus, PPV could be underestimated for this dataset. Additionally, the KPGA dataset had the highest proportion of older cases and cases without severe CHD than the other datasets, which are both associated with lower PPV (Rodriguez et al., 2022). The SC site, with a similarly low baseline PPV of 58.2%, included data from several diverse community-based data sources. Therefore, less accurate coding by non-cardiac specialists in this sample might explain the low PPV for this dataset. The algorithm classified patients as having any CHD or not having any CHD, but did not determine PPV of individual CHD codes. Nevertheless, the PPV for all sites’ cases were substantially improved from baseline. The constraint on the feature space ranking, corresponding to the number of patients, may technically limit the application of statistical feature selection algorithms like those based on eigen-analysis.

Additionally, certain features selected, such as the number of emergency department visits and a diagnosis in the respiratory system group, may exhibit variations during the coronavirus disease 2019 pandemic that could impact performance for future datasets with subsequent years of surveillance data.

The accuracy of administrative datasets in detecting CHD can be enhanced through the application of machine learning techniques, with potential for further improvement through the utilization of larger, validated datasets. Employing a generalized machine learning pipeline for CHD case classification within administrative data has the capacity to enhance public health surveillance endeavors, understand healthcare utilizations, and improve patient outcomes. To validate this discovery, additional machine learning studies involving diverse datasets from various locations, time periods, and patient demographics would be informative.