Because of the substantial morbidity and mortality caused by influenza viruses every year in the United States [1], annual influenza vaccination is recommended for all individuals aged 6 months and older [2]. Achieving high influenza vaccination coverage is a national health goal [3], and vaccination coverage is closely monitored [4]. Coverage data can be used to measure adherence with recommendations, identify populations with low coverage, prompt targeted vaccination campaigns, and guide public health activities [5].

National influenza vaccination coverage in adults is routinely assessed through surveys, such as the National Health Interview Survey (NHIS) [4], the Behavioral Risk Factor Surveillance Survey (BRFSS) [6], and Internet panel surveys [7]. While survey-based assessments have many strengths, and remain the foundation of national coverage surveillance, limitations exist. Influenza vaccination is determined by self-report, and these platforms are not designed to verify self-report against provider records [4,6,7]. Respondents may report influenza vaccination despite not being vaccinated (due to social desirability), may mistake influenza for another vaccine, or may not recall whether they received the vaccine this season versus a prior season [8,9]. Survey response rates generally are declining, and survey-based methods are subject to selection and response biases [10]. Electronic health record (EHR) [11] and claims [12] data have also been used to assess influenza vaccination coverage; however, these data sources may not completely capture vaccines received (e.g., vaccines administered in workplaces or pharmacies).

The primary objective of this study was to assess by survey the validity of EHR-based influenza vaccination data among adults in a multistate research network. A secondary objective was to determine whether survey respondents differed from non-respondents by demographic and clinical characteristics.

For the 2018–2019 influenza season, 777 of 1740 participants (44.7 %) responded to the survey; for the 2019–2020 influenza season, 505 of 1246 participants (40.5 %) responded; the response rate was significantly higher for 2018–2019 than for 2019–2020 (p < 0.02). Combining both survey years, 44.6 % responded by internet and 55.4 % responded by mail. Survey response rates were lowest among non-Hispanic Black individuals (31.7 % in 2018–2019, 35.0 % in 2019–2020) and highest among non-Hispanic White individuals (52.4 % in 2018–2019, 51.0 % in 2019–2020). As shown in Table 1, survey respondents differed from non-respondents by age group, race and ethnicity, presence of a chronic health condition, and EHR-based influenza vaccination status for the current and prior influenza seasons. In a multivariable model examining factors associated with survey non-response (Table 2), non-Hispanic Black race and ethnicity was significantly associated with non-response (both survey years combined, adjusted odds ratio [aOR] 1.91, 95 % CI 1.10, 3.32) after adjusting for other factors.

As shown in Table 3, the percent having received influenza vaccine per EHR data was higher among survey respondents than among the entire sample, suggesting selection bias. Calculated as an absolute percentage point difference, estimated selection bias was 8.8 % overall. Stratified by race and ethnicity, estimated selection bias was 10.5 % for non-Hispanic Black individuals and 8.7 % for non-Hispanic White individuals.

Combining results from both survey years, 55.7 % of respondents were not vaccinated for influenza according to EHR data; 26.1 % of these self-reported that they were vaccinated. Additionally, 44.3 % of respondents were vaccinated for influenza according to EHR data; 1.5 % of these self-reported that they were unvaccinated. The sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of EHR-based influenza vaccination data are presented in Table 4. Of all validity measures calculated, NPV was the lowest value; NPV for both years combined was 73.9 % (95 % CI 68.0, 79.3). The validity measures differed by several respondent characteristics (Table 4). For example, the sensitivity of EHR vaccination data was highest for individuals aged ≥65 years at 90.4 % (95 % CI 83.0, 95.3) and lowest for individuals aged 18–24 years at 40.7 % (95 % CI 8.6, 80.4). The level of agreement between EHR-based data and self-report of influenza vaccination status was moderate (kappa coefficient both years combined 0.70, 95 % CI 0.63, 0.77).

All respondents who self-reported vaccination were asked the location of their vaccination. For those who were unvaccinated according to EHR data but reported vaccination (EHR false negatives), 24.5 % were vaccinated in a workplace, 18.1 % in a hospital, and 43.5 % in a physician’s office, clinic, or health center. In contrast, for those vaccinated according to EHR data who also self-reported vaccination (EHR true positives), 2.1 % were vaccinated in a workplace, 26.1 % in a hospital, and 65.3 % in a physician’s office, clinic, or health center. These data are also shown in Supplemental Table 1.

Approximately 49.4 % of adults in the United States received an influenza vaccine during the 2021–2022 season [6], far below the Healthy People 2030 goal of 70 % coverage [3]. Efforts are needed to improve influenza vaccination coverage, and accurate coverage estimates are necessary to help guide these efforts. In a survey of 1282 adults in a multistate research network, we found moderate agreement between EHR-based influenza vaccination data and self-report, with high specificity (98.4 %) but lower sensitivity (75.0 %) of EHR-based data compared to self-report. Stratified by age, the sensitivity was highest among individuals aged ≥65 years. The NPV of EHR-based influenza vaccination data was 73.9 %, indicating that for respondents who appeared unvaccinated in EHR data, 26.1 % reported having received influenza vaccine. Vaccination outside the medical home (e.g., in the workplace or at a pharmacy) may contribute to disagreement between EHR-based data and self-report.

It is informative to compare these findings to prior studies, particularly studies which treated self-reported influenza vaccination as the criterion standard. Greene and colleagues compared EHR-based data to self-report among adults aged 50–70 years following the 2007–2008 influenza season; the NPV of EHR-based influenza vaccination data was 66.6 % [18]. In a study by Sy and colleagues among adults aged 50–79 years, also following the 2007–2008 season, the NPV of EHR-based influenza vaccination data was 79.5 % [19]. Our study included adults aged ≥18 years, and age may affect NPV: although confidence intervals were wide and overlapping, the NPV point estimate was lowest (60.5 %) for adults aged 18–24 years and highest (84.6 %) for adults aged 50–64 years. It is plausible that younger adults are more likely to receive vaccines outside their medical home.

Our findings highlight the risk of response bias in survey-based assessments of influenza vaccination coverage [14]. In weighted analyses, the percent vaccinated per EHR data was higher among respondents than among the full sample, overall and for non-Hispanic White, non-Hispanic Black, and non-Hispanic other groups, although estimates were imprecise. This suggests that the weighting of national survey estimates may not fully mitigate non-response bias, and national surveys possibly overestimate true influenza vaccination coverage. However, it is also important to recognize that our survey was introduced as related to “flu vaccination” whereas other surveys such as NHIS [4] and BRFSS [10] concern health more broadly, and response bias could differ across different survey platforms.

In addition to response bias, this study is subject to other limitations. First, self-reported vaccination was treated as the criterion standard, but respondents could have been mistaken in their self-report. Second, EHR-derived race and ethnicity data could have been misclassified. Third, these results largely reflect a pre-pandemic landscape, and the COVID-19 pandemic may have fundamentally altered what immunization information is shared across entities. Fourth, the VSD comprises large healthcare organizations caring for predominantly insured populations [13,20]; results may not be generalizable to other settings.

In conclusion, in surveys conducted among adults in a multistate research network across two recent influenza seasons, there was moderate agreement between EHR-based vaccination data and self-report, with 26.1 % of respondents who appeared unvaccinated in EHR data reporting having received influenza vaccine. A possible selection bias was detected. More accurate assessments of national coverage may be possible by integrating multiple sources of influenza vaccination data.