The adult immunization schedule in the United States has become increasingly complex as new vaccines are added for the general population or specific risk groups [1-3]. While new vaccines represent new opportunities for disease prevention, adult vaccination programs have historically received less attention than childhood programs, public health resources are more limited to promote vaccination in adults, and vaccination coverage for recommended vaccines in adults remains low [4-7]. Vaccination coverage estimates can help guide public health resources to areas of greatest need [7]. Historically, coverage in US adults has been assessed through national surveys, such as the National Health Interview Survey (NHIS) [7], the Behavioral Risk Factor Surveillance Survey (BRFSS) [8], the National Immunization Survey Adult COVID Module [9,10], and Internet panel surveys [11]. Although survey-based assessments have many strengths and remain the foundation for adult coverage surveillance at the Centers for Disease Control and Prevention (CDC), important limitations include: low survey response rates; nonresponse bias; reliance on self-reported vaccination status; small sample size for subgroup analyses; survey costs; and timeliness of data for many survey platforms [7,12].

Over the past decade, non-survey-based data sources have increasingly been used to assess adult vaccination coverage [13-16]. Many of these data sources rely on electronic data created through documentation of care provided (i.e. electronic health records [EHR]) or payment for care received (i.e. administrative claims). Using EHR and claims for this purpose is appealing as these data sources rely on already-existing data, are potentially more timely and less expensive than surveys, and usually cover large populations across multiple states [13-16]. Jurisdiction-specific immunization information systems (IIS) can also be used to assess population-level coverage: IIS are confidential, population-based, computerized records of vaccine doses administered in specific geographic regions in the United States [17]. IIS were initially focused on pediatric rather than adult populations, and not all vaccination providers report to a jurisdictional IIS [17]. Additionally, laws and policies regarding the submission of vaccination data to an IIS vary by jurisdiction, which may impact the completeness of available vaccine data. While non-survey-based data sources have the potential to make important contributions to coverage assessments, the strengths and limitations of these data sources have not been fully described [18].

Additionally, the national landscape is rapidly changing, with continued adoption of EHR in diverse settings [19], expansion of insurer-linked consolidated electronic databases utilized for research purposes [20], the pandemic-related requirement that COVID-19 vaccination sites electronically report vaccine administration data to the federal government, ideally through jurisdictional IIS [18], and new vaccine recommendations for adults [21,22]. In this context it is important to evaluate the utility of non-survey-based methods for estimating adult vaccination coverage. No other studies have comprehensively evaluated this question. The objectives of this narrative review were to 1) identify published studies that used non-survey-based data sources to estimate adult vaccination coverage for one or more vaccines; and 2) assess the strengths and limitations of these data sources for coverage estimation.

In this narrative review of non-survey data sources for assessing adult vaccination coverage for routinely recommended vaccines in the United States, we assessed 9 data source types contributing more than 30 peer-reviewed publications from 2007 through 2022. Studies reported coverage for a wide range of the vaccines currently recommended for adults, with multiple studies examining coverage in populations defined by their health status (e.g. pregnancy or presence of a chronic health condition). The number of data sources and publications, and the diversity of vaccines and populations examined, suggest that non-survey data sources are viable resources for estimating adult vaccination coverage at the national or sub-national level. However, it is important to highlight that these data sources have distinct strengths and limitations. Additionally, when reviewing manuscripts for this study, detailed information on the limitations of various data sources was often missing. Finally, the merits of one data source compared to others varied depending upon the research questions being posed within individual studies.

While some studies made inferences about data quality, none of the reviewed studies presented any formal assessment of vaccination data validity. In the context of claims- and EHR-based vaccine administration data, false negatives (i.e. an individual who appeared unvaccinated but had received the vaccine of interest) may occur for vaccines administered outside the health system, and are arguably more likely to occur than false positives (i.e. an individual who appeared vaccinated who truly was not), and false negatives would underestimate vaccination coverage. The likelihood of missing vaccination information probably differs by vaccine and data source, however. For example, influenza vaccine, relatively inexpensive and often available in workplaces, may be more likely misclassified than other vaccines, because vaccination may not be billed to insurance [61,62]. In some settings with EHRs, vaccination data in EHRs are supplemented by vaccination data from jurisdictional IIS [63], making these systems less prone to missing vaccination information than claims-based systems. In the reviewed studies, the accuracy of vaccination coverage estimates is often inferred by comparing study estimates to those based on national surveys; while this approach is understandable, survey-derived estimates are also susceptible to error, such as error created by biases inherent in sampling, response, and recall of vaccination. Validating the vaccination data in non-survey databases would entail triangulating vaccination data across data sources (such as comparisons between claims, EHR, IIS, and self-report), manually reviewing individual EHR records to look for coding errors, or surveying a sample of individuals from these data sources. While these validation studies would be challenging to conduct, they would greatly enhance understanding of the validity of non-survey-based data sources.

In the reviewed studies, it was also difficult to assess the validity of the denominator used to calculate coverage, particularly for studies in general populations (i.e. those not identified according to health status). For example, when using the IQVIA database to estimate pneumococcal conjugate vaccine uptake among adults aged ≥65 years, a ‘proprietary algorithm’ was used to identify a ‘stable panel,’ from which coverage estimates were extrapolated to the US population [14]. Li and colleagues used continuous insurance enrollment to define their population denominator in an Optum-based study of influenza vaccination coverage; a table was provided comparing the study population demographics to the US population, further aiding study interpretability [15]. When estimating live attenuated zoster vaccination coverage for adults aged ≥50 years using MarketScan, Zhang and colleagues also used continuous health insurance enrollment to define their population denominator; while this may represent an accurate and interpretable denominator, few other details were provided [13]. While limiting vaccination coverage assessments to adults with continuous health insurance may reduce data missingness (e.g. for vaccines received or for health status), doing so limits generalizability to groups with stable insurance over multiple years; the estimates generated would not reflect vaccination coverage among uninsured or incarcerated adults, for example.

Persistent racial and ethnic disparities in adult vaccination coverage have been documented [7,64], and reducing these disparities is an important national public health goal. Surveys such as the NHIS are critical to understanding these disparities, because self-reported vaccination status is collected alongside individual-level data on race, ethnicity, and socioeconomic factors such as household income and educational attainment [7,64]. For several of the data sources reviewed here, the methods for obtaining race and ethnicity data were complex and opaque, and potentially subject to bias. In a study assessing racial and ethnic disparities in influenza vaccination coverage, Li and colleagues used Optum claims data linked to ‘additional sources’ in ‘proprietary analytic and demographic models’ to obtain race and ethnicity data; individuals excluded from analyses due to missing demographic information were more likely from racial and ethnic minority groups [15]. In a study examining racial and ethnic disparities in pneumococcal conjugate vaccination coverage among adults aged ≥65 years, McLaughlin and colleagues used IQVIA claims data linked to the Experian database to obtain individual-level sociodemographic information; Experian data ‘come primarily from consumer participation in the credit market’ [14]. The authors acknowledged that older adults living in poverty or with less formal education may be underrepresented in the credit market, leading to differential exclusion of these populations in the IQVIA-Experian linked dataset used for their analyses [14]. These examples illustrate the challenges and potential direction of bias when using non-survey-based data sources to monitor disparities: if racial and ethnic minority groups are underrepresented in analytic datasets, the extent of disparities in vaccine coverage may be underestimated. Finally, while Medicare and EHR-based systems collect data on race and ethnicity, the completeness and accuracy of the race and ethnicity data may create challenges [65], and these data sources have limitations to generalizability (Medicare beneficiaries are predominantly ≥65 years of age, VSD members are predominantly privately insured and reside in a limited number of US states) [16,40,66].

Because adults with chronic health conditions are particularly vulnerable to certain vaccine-preventable diseases, monitoring coverage in these populations is important to help guide interventions and other public health activities. The reviewed studies demonstrated substantial capacity for assessing vaccination coverage in these special populations, including among adults with asplenia [45], complement deficiencies [46], immunocompromise [47], sickle cell disease [33], and end-stage renal disease [38]. The conditions of interest were most often identified using International Classification of Diseases, Ninth or Tenth Revision (ICD-9, ICD-10) diagnosis codes or procedure codes; however, the accuracy of using these codes to identify the study cohort was not validated. Additionally, the nature of these studies increases the likelihood of missing historical vaccination: the conditions of focus may have been present for many years and prompted vaccination years earlier. While these studies often required one year or more of continuous health insurance enrollment to be included in the study cohort, vaccination prior to insurance enrollment may have been missing. Non-survey data sources have also been used extensively to examine vaccination coverage among pregnant women [25,27,30-32,36,37,50-54]. While issues of cohort identification and missing historical vaccination may be less problematic in studies of pregnant women, issues of generalizability arise. For example, while the VSD has developed and validated an algorithm to identify pregnant women [55,56], and supplements EHR-based data with vaccine data from jurisdictional IIS [63], vaccination coverage in the VSD population likely does not reflect coverage in other populations (such as pregnant women with public health insurance or intermittent private insurance) [66].

The findings in this narrative review should be interpreted in the context of several limitations. First, we limited our review to manuscripts published in peer-reviewed journals; we did not directly interrogate the data sources themselves. While the CDC has developed publicly facing ‘dashboards’ for preliminary influenza vaccination coverage estimates [67], and state immunization programs have used jurisdictional IIS for adult vaccination coverage reports, these types of assessments were not included in our review. Second, we did not review coverage studies based on place of employment (e.g. coverage among health care personnel) or residence (e.g. coverage among residents of nursing homes). Third, although our search strategy was broad and iterative, it is possible that we did not identify all published manuscripts and all relevant data sources for inclusion in our review. Related, we have not included any manuscripts published after our literature search was completed in May 2022. Fourth, because we were not able to identify an already-existing schema for comparing and rating the reviewed data sources, we developed and defined a set of key study attributes relevant to adult vaccination coverage. While the schema that we developed was useful in guiding our review, the process and the resulting ratings were subjective and qualitative in nature. It is certainly possible that other researchers would have prioritized other key attributes or rated the reviewed data sources differently. However, we used a process with multiple reviewers and adjudication of discrepancies in judgment between reviewers. Fifth, when evaluating different data sources, we were reliant upon the study methods described in the manuscripts, and at times relevant details were not included. In such circumstances, our assessment of key attributes was inferred from knowledge about the limitations of certain types of data; for example, vaccination data from claims-based sources was generally viewed as less complete than the vaccination data from EHR-based sources such as the VSD, recognizing that exceptions may exist. Sixth, only a single publication was identified for each of several data source types (e.g. all-payer claims databases [25], IQVIA [14], single-state Medicaid [37], and PRISM [48]), which limited the assessment of the attributes of these data sources. Finally, this review focused on published studies regarding adult vaccination coverage, and the findings and limitations do not apply to using these data sources (e.g. MarketScan or Optum) for other purposes.

In conclusion, while multiple non-survey data sources have been used to assess adult vaccination coverage in the United States, across a range of populations and vaccines, important limitations exist related to generalizability, data quality, and the resulting risk of bias. In the context of four use cases of critical public health importance, no single data source had high utility with minimal limitations. Therefore, it is important to carefully consider the strengths and limitations of these data sources, which were not developed for research or surveillance purposes, when answering a particular assessment or research question. To improve the validity of coverage estimates, it may be necessary to synthesize findings across more than one data source.

National surveys remain the cornerstone for adult vaccination coverage surveillance in the United States, and have many advantages compared to non-survey-based data sources, which are not collected for research or surveillance purposes. Critically, well-resourced national surveys can employ complex sampling and weighting strategies to strengthen the representativeness and generalizability of survey findings. Additionally, surveys can collect respondent-reported data on socioeconomic factors such as race, ethnicity, education, and income, which are important for identifying disparities in coverage. Data on current health status and respondent attitudes and knowledge about vaccines can also be collected through surveys; these data are important to design interventions to increase vaccine acceptance and coverage, including among individuals with risk-based indications for vaccination.

As described in this narrative review, more than 30 peer-reviewed publications have used non-survey-based data sources to estimate adult coverage in a US population. The sheer quantity of publications does not necessarily provide insight into the quality or utility of these data sources, however. While these data sources may mitigate some of the limitations of surveys, such as nonresponse bias, recall bias, and challenges with timeliness, non-survey-based data sources introduce other significant limitations. To illustrate this point, we provide three examples related to the numerator (i.e. the vaccines received), the denominator, and the representativeness of the study population. In studies reliant on health insurance claims to estimate vaccination coverage, not all vaccines administered will generate a claim, resulting in an underestimation of the true coverage. In studies using statelevel IIS to estimate coverage, defining an adult population denominator is challenging due to people moving out of the jurisdiction without being removed from the denominator; this presumably leads to an inflated denominator which would bias results toward an underestimation of true coverage. With respect to representativeness, studies relying on EHR data are typically restricted to a stably insured population; as such, coverage estimates may be higher than the true population-wide coverage. The purpose of this review has been to not only detail the extent of coverage-related publications using non-survey-based data sources but interrogate the limitations of these data sources.

How could the contribution of non-survey-based data sources to estimating adult vaccination coverage be strengthened? First, there is a need to fully understand how these data sources are constructed, how coverage estimates are calculated, and to what extent these estimates reflect true coverage in a population. Our narrative review found a lack of clarity in describing the construction and representativeness of the populations included. Methods sections were frequently brief and opaque regarding how the cohort was defined, the denominator calculated, and what geographic areas contributed to the data sources. Regarding the completeness and accuracy of the underlying data, systematic data quality assessments are needed. For example, a manual review of records could be conducted to verify vaccination status for a sample of records, and a sample of individuals could be contacted to verify their vaccination status. Researchers using claims or EHR-based systems could also link records with their jurisdictional IIS (while maintaining privacy and confidentiality), again with the goal of conducting a more formal analysis of the accuracy of vaccine administration data.

Reinforcement of these expectations could be addressed through the review process at scientific journals. At the time manuscripts are submitted to journals, peer reviewers and editors could request a detailed description of how the cohort was developed and the denominator determined, and a quantitative assessment of the representativeness of the study population. Regarding data quality issues, reviewers and editors could hold studies using non-survey-based sources to similar standards for data validity that they apply to studies using surveys. At a minimum, studies reporting coverage estimates based on non-survey-based data sources should transparently describe the potential biases inherent in the data sources and analyses in relation to the research question being addressed. While non-survey-based data sources have the potential to provide rapid coverage assessments in large populations, the promise of these data sources may not be fully realized until more foundational work is completed around data quality and population representativeness.