Test-negative design (TND) studies of influenza vaccine effectiveness (VE), in which symptomatic patients are enrolled as test-positive cases or test-negative controls at the point of care, were originally developed to reduce biases in case-control studies introduced by potential case status misclassification and differential health care-seeking between cases and non-cases.^1,2 However, bias, while reduced overall, can still persist in TND studies due to incomplete capture of confounders such as exposure history, effects of vaccine policy, or propensity to seek vaccination among patients.^1,2 Until recently, studies of influenza vaccine effectiveness (VE) in adults have not required extensive consideration of biases introduced by other vaccine-preventable respiratory pathogens. The emergence of SARS-CoV-2 and the development of COVID-19 vaccines introduced a scenario of co-circulating pathogens in which patients frequently receive vaccines for both influenza and SARS-CoV-2. In addition, an RSV vaccine was approved for adults aged ≥60 years in 2023. When the uptake of vaccines against two pathogens is correlated, VE may be underestimated if cases of a potentially confounding pathogen are included among the controls for estimates of VE against the pathogen of interest.^3,4 As cases of one pathogen may be more likely to be unvaccinated for both pathogens or vaccinated for both pathogens, their inclusion among controls for the estimation of VE against the other pathogen may increase the proportion of unvaccinated controls, resulting in a lower VE estimate.

Correlation in vaccine uptake may occur, for example, due to concerns raised over one vaccine (e.g., safety, mistrust) that inhibit the uptake of the other,³ or patient recommendations from healthcare providers (i.e., to get vaccinated for multiple respiratory pathogens) that promote uptake of the other.⁵ Some recent influenza VE studies^6,7 have observed moderate downward bias when COVID-19 cases were included among controls and have excluded COVID-19 cases from the control group to mitigate this bias. Further explorations of both influenza and COVID-19 VE have found that downward bias may increase with higher proportions of confounding pathogen cases in the control group, higher effectiveness of the vaccine of interest and of the vaccine against the confounding pathogen, or higher vaccination coverage.^4,8

However, recent studies of both influenza and COVID-19 VE have suggested that other causes of residual confounding beyond correlated vaccination behavior could still bias VE estimates upwards or downwards. Although recent influenza and COVID-19 VE analyses have typically adjusted for covariates such as age, sex, race, ethnicity, location, or comorbidities,^6,7,9 these adjustments do not always account for complex biases. For example, downward bias due to confounding by indication can occur when persons who are more likely to become cases are also more likely to be vaccinated due to health-seeking behavior or healthcare provider intervention (e.g., those who are frail or have underlying comorbidities). During previous influenza seasons, null or negative VE against influenza–associated hospitalization has often been attributed to selection biases in which influenza cases admitted to the hospital were more likely to be vaccinated against influenza.^10,11 Similarly, recent assessments of COVID-19 VE bias have suggested differences in levels of exposure to the virus (e.g., greater risk-taking following vaccination) and propensity to seek care between vaccine recipients and non-recipients could push VE estimates downward.^12,13 In contrast, upward bias theoretically could occur if other health-seeking or health-promoting behaviors (e.g., adoption of non-pharmaceutical interventions to prevent infection) are associated with vaccination for a respiratory pathogen.

To understand the nature and extent of bias introduced by co-circulating, vaccine-preventable, diseases existing in real-world observational data used for VE estimation, we used a sample of prospectively enrolled, hospitalized patients from a network of medical centers to examine the sources and magnitude of bias in VE estimates against influenza–associated and COVID-19–associated hospitalization, including correlation of vaccination status across pathogens and other potential causes. After summarizing the results, we provide recommendations for interpreting and mitigating bias in VE estimates for influenza, COVID-19, and now RSV as a vaccine-preventable respiratory infection.

Twenty-four hospitals from 19 US states within the Investigating Respiratory Viruses in the Acutely Ill (IVY) Network prospectively enrolled patients into this multi-pathogen acute respiratory illness (ARI) surveillance program from September 1, 2022–March 31, 2023. The IVY Network is funded by the Centers for Disease Control and Prevention (CDC) and coordinated from Vanderbilt University Medical Center. This program was given a waiver of informed consent by CDC and each participating site and was conducted in a manner consistent with applicable federal law and CDC policy.^a Personnel at all sites were trained to follow a standard protocol outlining eligibility criteria, data collection, and specimen collection procedures. Patient eligibility criteria included: (1) age ≥18 years; (2) hospital admission; (3) clinical presentation consistent with ARI, defined as having ≥1 of the following: fever, cough, shortness of breath, new hypoxemia, or new pulmonary findings on chest imaging consistent with pneumonia; and (4) a clinically obtained molecular or antigen test for influenza virus, SARS-CoV-2, or respiratory syncytial virus (RSV) within 10 days of symptom onset and within 3 days of hospital admission. Relevant exclusions were applied to both influenza and COVID-19 VE models (Supplementary Table 1). Methods of clinical and laboratory testing and vaccination ascertainment have been described elsewhere.^6,7

The analysis comprised five different models to evaluate bias in VE based on seasonal influenza vaccination (compared with patients who did not receive the current season’s dose) or bivalent COVID-19 vaccination (compared with patients who have never received any COVID-19 vaccine [monovalent-only vaccine recipients excluded]) (see Table 1). During August 2022–March 2023, CDC recommended a bivalent COVID-19 mRNA vaccine dose for all persons aged ≥5 years who had received a monovalent primary vaccination series ≥2 months prior, and in December 2022–April 2023 recommended a bivalent vaccine dose for most persons aged ≥6 years regardless of previous monovalent vaccination, plus optional additional bivalent booster doses for moderately or severely immunocompromised persons aged ≥6 months and adults aged ≥65 years.¹⁴ During the same period, influenza vaccination was recommended for everyone aged ≥6 months (excepting those with allergies to influenza vaccines), with a preferential recommendation for the use of higher dose and adjuvanted flu vaccines in people aged ≥65 years, over standard dose, unadjuvanted flu vaccines.¹⁵

In Analysis 1, correlation was calculated as relative risk (RR) of influenza vaccination given bivalent COVID-19 vaccination, or vice versa, and defined as low (RR: 0 to <2), moderate (RR: 2 to <4), and high (RR: ≥4). Analyses 2–5 used logistic regression to compare the odds of vaccination among hospitalized case patients versus controls. All VE models, except where specified explicitly as “unadjusted,” were adjusted for categorical age in years (≤5960, 60–64, 65–74, ≥75); sex (male/female), race and ethnicity (Non-Hispanic White, Non-Hispanic Black, Hispanic, other, unknown), Census region (Northeast, Midwest, South, or West, to capture regional variation in transmission), and biweekly period of hospital admission. VE against influenza–associated hospitalization was then calculated as: (1 − adjusted odds ratio of vaccination) × 100%. Bias was estimated as the percentage point difference (i.e., absolute change) in VE from the referent outcome and categorized as low (0 to <5 points), moderate (5 to <10 points) and high (≥10 points). The five analyses were conducted as follows:

Correlation of vaccination status: The overall RR of vaccination for the confounding pathogen was calculated for influenza and COVID-19 vaccine recipients.

A total of 759 influenza-positive patients were enrolled as cases, and 3262 influenza- and COVID-19-negative controls were enrolled as influenza VE controls; the number of controls increased to 5400 when 2138 COVID-19 cases, accounting for 40% of controls, were included (see Table 2). A total of 1126 COVID-19-positive patients were enrolled as cases and 1823 COVID-19- and influenza-negative controls were enrolled as bivalent COVID-19 VE controls; the number of controls increased to 2210 when 387 influenza cases, accounting for 18% of controls, were included. Finally, a total of 399 RSV patients were enrolled as cases and 4269 RSV-negative, influenza-negative, and COVID-19-negative (i.e., triple-negative) patients were used as controls for the RSV negative control outcome analysis. Most patients were aged ≥60 years and seasonal influenza vaccination was more common than bivalent COVID-19 vaccination.

This analysis of patients hospitalized with acute respiratory illness during the 2022–2023 season reveals a significant correlation between seasonal influenza and bivalent COVID-19 vaccination, suggesting a strong association in vaccine acceptance for influenza and COVID-19. Although the cause of higher vaccination correlation observed in younger and healthier patients is unknown, explanations could include more self-selection for all available respiratory pathogen vaccinations among younger patients or vaccine mandates for working-age adults in some occupations. Notably, the correlation between vaccines was not evenly balanced. People who received COVID-19 vaccine were very likely to also receive the influenza vaccine, but this effect was not as strong for the likelihood of receiving the bivalent COVID-19 vaccine if one had received influenza vaccine. Therefore, the presence and impact of bias needs to be evaluated specifically for each virus when estimating VE, to account for variation in co-vaccination and year-to-year differences in seasonality.

Estimates of VE against hospitalization associated with each pathogen decreased when cases of the opposite pathogen were included in the control group, albeit within a threshold (i.e., <10 percentage points) that has been categorized as low in previous studies.⁴ The downward bias was greater for influenza VE compared to bivalent COVID-19 VE in the adjusted models, which may reflect the relatively large number of COVID-19 cases compared with influenza, and the consequently higher proportion of COVID-19 cases among influenza controls. Adjusting for vaccination status for the confounding pathogen in models of VE against hospitalization associated with the pathogen of interest did not correct the downward bias observed when cases of the opposite pathogen were included as controls. This lack of mitigation may be because the bias introduced by correlated vaccination status is a proxy for the bias introduced by other factors (e.g., similarities between patients of certain age groups) that are addressed by other VE model covariates.

Real-world observational data from 2022–2023 showed that including influenza cases among controls for estimation of bivalent COVID-19 VE against COVID-19-associated hospitalizations and emergency department/urgent care (ED/UC) encounters shifted VE estimates downward by <5 percentage points,⁸ consistent with the bias observed here. Similarly, a study from the 2021–2022 influenza season estimated that the inclusion of COVID-19 cases among controls when estimating VE against influenza–associated hospitalization shifted estimates downward by 7–11 percentage points depending on the age group assessed,⁶ which was slightly greater decrease than observed in the current study. Our simulation analysis aligns with previous studies suggesting that bias will increase with higher proportions of cases of the confounding pathogen among controls, as well as higher VE for both the vaccine of interest and the confounding vaccine.^4,8

Negative control exposure (NCE) analyses showed significant influenza VE against COVID-19–associated hospitalization and significant but lower bivalent COVID-19 VE against influenza–associated hospitalization. Most estimates were slightly lower when confounding pathogen cases were included among controls (compared with excluded), suggesting that vaccination for influenza and COVID-19 are sufficiently correlated to exert some downward pressure on VE even when the vaccine exposure is biologically unrelated to the pathogen of interest. Adding vaccination status for the confounding pathogen as a covariate in the NCE models brought both VE estimates close to the null, suggesting that correlation of vaccination status accounted for much of the observed VE in the NCE analyses.

In the negative control outcome (NCO) analysis, null or negative bivalent COVID-19 VE and influenza VE were observed against RSV–associated hospitalization. While null VE against RSV would be interpreted as an absence of confounding bias,¹⁷ negative VE estimates were observed in some of the adjusted analyses. may suggest that patients in this sample who get vaccinated for COVID-19 or influenza have more risk factors (e.g., underlying conditions) that increase the likelihood of RSV–associated hospitalization. These factors could exert some “hidden” residual downward bias on VE estimates in test-negative VE studies that are not necessarily related to correlated vaccination status.¹² In hospital-based networks, VE against influenza–associated hospitalization may be lower than anticipated due to high rates of vaccination among influenza cases in the inpatient population and comparatively poor health associated with reduced immune protection from the vaccine or more severe influenza–associated outcomes.^10,11 Including patients with cases of other respiratory diseases in the control group for inpatient influenza VE has previously appeared to drive VE downward, even during the pre-COVID era when vaccine-preventable pathogens accounted for a much lower proportion of other circulating respiratory pathogens. A study of VE against influenza–associated hospitalization across the 2014–2015 and 2015–2016 influenza seasons found that, when compared with a control group presumed negative for influenza and all other respiratory pathogens, VE decreased by 2–5% when including patients testing positive for other respiratory pathogens and by 6–17% using only patients testing positive for other respiratory pathogens as controls.¹⁸

Correlated vaccination status thus likely accounts for some but not all of the low to moderate downward bias observed in our influenza and bivalent COVID-19 VE estimates. Where VE bias can be expected to be at least moderate, as in the case of influenza VE during a time of continued high COVID-19 incidence, excluding confounding pathogen cases from controls remains a method to reduce downward bias. However, this mitigation technique may not be sustainable during periods of multiple co-circulating, vaccine-preventable respiratory pathogens, which now include RSV.^19–21 In this context, excluding cases of confounding pathogens from control groups may unacceptably reduce the size or comparability of control groups, and the precision of VE estimates. Researchers may therefore need to explain potential sources of bias more fully and identify more directly the factors (e.g., underlying health status, risk factors) that could influence both vaccine uptake and risk of infection across pathogens to support additional multivariable strategies.

This study has several limitations. First, the sample size limited VE estimation or precision for some subgroups, and the degree of bias could vary for subgroups that were not considered here. Second, our VE analyses focuses on hospitalization associated with influenza, COVID-19, or RSV as outcomes of interest and observed bias may differ for other clinical endpoints. Third, although the analysis controlled for several relevant potential confounders, residual confounding is possible. Fourth, while the results suggest that correlated vaccination status biases VE toward the null, it is not possible to estimate its precise contribution to the overall downward bias detected or to ascertain the other sources of residual confounding that exert downward bias.

Several strategies could be employed to properly communicate or further investigate the potential impact of biases on VE estimates. For example, future studies could report multiple VE estimates, with and without cases of other pathogens excluded from controls, and potentially with additional sensitivity analyses for VE against outcomes such as ICU admission and mortality. However, such strategies could lead to challenges in communication, reduce the generalizability of the findings to the overall population, and create uncertainty regarding which estimate should be considered the primary result. Researchers could also consider specifying a bias cutoff (e.g., >10 percentage points) to determine when to exclude alternate viruses from controls, or to report the general level of downward bias in a pre-specified set of sub-analyses. If excluding cases of a relevant pathogen from a control group renders VE estimates too imprecise, the potential for downward bias should be acknowledged when reporting the estimate from the primary analysis inclusive of all enrolled controls.

In addition, future studies might collect additional data to disentangle the relationships between health status, health-seeking, vaccination, and VE when computing VE. For example, a joint estimation of vaccination for the pathogen of interest and effectiveness of the vaccine of interest, or more detailed adjustment for propensity for vaccination, seeking medical care, or being hospitalized for an acute respiratory illness, may help reduce these biases. Depending on the viral pathogen of interest, adjustment for propensity of different vaccination combinations could be explored. Finally, the Bayesian adaptive design framework could be modified to extend into examining within-season VE with priors (e.g., vaccination coverage and correlation between vaccinations) informed as data collection advances throughout a season.

Excluding cases of other vaccine-preventable respiratory pathogens from the controls in a VE analysis is an effective method to reduce downward bias. This approach may be particularly important during periods of high co-circulation resulting in large proportions of cases of these other pathogens among controls.