We evaluated routinely available U.S. data on influenza cases, influenza vaccination coverage, and vaccine effectiveness from 2010–11 to 2016–17 by age group (6 months-4 years, 5–17 years, 18–49 years, 50–64 years, and ≥65 years). The number of influenza cases (including both medically-attended and non-attended) occurring each month was estimated from the Influenza Hospitalization Surveillance Network (FluSurv-NET) [5,9,10]. In brief, influenza hospitalization rates from 9% of U.S. hospitals are adjusted for testing frequency, converted to counts and multiplied by a previously-estimated ratio of cases to hospitalizations to derive the total number of influenza cases. For each age group, the number of cases is a constant multiple of the number of hospitalizations, and so in this manuscript we will refer only to cases. We obtained the prevalent proportion of the total population vaccinated at the end of each month from survey data [3], and used this figure to determine the incident number vaccinated during each month. We obtained vaccine effectiveness estimates from the U.S. Influenza Vaccine Effectiveness Network [11] and age-group specific population data from U.S. census estimates [12].

We built a reference model with seven compartments defined by combinations of persons that were ill or well, vaccinated or non-vaccinated, and immune or susceptible (Fig. 1 and Supplemental Table 1). We found that accounting for persons with pre-existing immunity at the start of the season (Fig. 1, G) had a minimal effect when the probability of vaccination did not differ by prior immunity status (data not shown); therefore, all calculations presented here assume no pre-existing immunity. Accordingly, all population members are susceptible, non-vaccinated, non-cases at the beginning of each season (Fig. 1, A). When vaccinated, persons move to compartment B for the immune lag period (generally 14 days), during which they are susceptible to infection and after which they either remain susceptible (C) or become immune (D). We assumed no indirect protection (i.e., herd immunity) and an all-or-none vaccination effect: vaccinated persons either developed complete immunity or remained fully susceptible. We assumed that vaccine would be given with equal frequency to both previously uninfected persons and those who had been infected earlier in the season. Susceptible persons (Fig. 1, A–C) could become infected and move to compartment E or F, after which they would be immune to further infection.

If the daily number of cases in the absence of vaccination is specified, calculations can be made for infection risk (=cases/number at risk without vaccination) and cases with vaccination (=infection risk · number at risk with vaccination; Supplemental Table 1). In contrast to some infectious disease models, in our model infection risk did not depend on the number of cases on prior days. Conversely, the number of cases with vaccination can be specified and cases without vaccination calculated (Supplemental Table 2, Method 7). The number of averted cases is the difference between the numbers of cases without vs. with vaccination. The number of averted cases estimated by the reference model was the “gold standard” to which we compared each test method.

Using information from a descriptive analysis on available monthly-aggregated influenza cases and vaccinations observed during the 2010–11 to 2016–17 seasons stratified by age group, we created a synthetic dataset. The dataset had a population of 1,0, 000 and rates of vaccination and illness typical of the real (i.e., collected by surveillance/observational studies) data, including: 55% vaccination coverage, 48% vaccine effectiveness, and 12% infection rate in the absence of vaccination (Supplemental Table 3). Daily counts of vaccinations and influenza cases were simulated using the normal distribution probability density function (Fig. 2A). We applied the reference model (Fig. 1, Supplemental Table 1) to this synthetic daily data to determine the true numbers of cases that would have occurred with vaccination and the cases averted by vaccination. We then aggregated the synthetic daily counts by month to simulate the format of real data (Fig. 2B), analyzed this aggregated data by seven test methods (see below), and compared the numbers of calculated averted cases between the reference model and the test methods. We considered test methods with the smallest differences in averted case estimates relative to the reference model to be the most accurate.

Next, we performed more detailed evaluations of the two most accurate test methods across a variety of potential influenza seasons. We created several additional synthetic datasets with normal-distribution daily case and vaccination counts but with variation of the following characteristics: proportion of vaccine given before cases (or “vaccination timing”; 5–98%), vaccine effectiveness (10–70%), vaccine coverage (10–80%) and infection risk (2%–50%). We also evaluated the test methods in skewed distributions and datasets with distributions similar to real data (Supplemental Table 4); because results were similar, we do not show these additional results.

We wanted to determine how the methods performed with less separation between months of vaccine administration and case occurrence. To measure this, we defined “vaccination timing” as the proportion of vaccinations occurring before case accrual. Vaccination timing could vary from 0% (no vaccine given before cases occurred) to 100% (all vaccine given before cases) and was defined as the sum over all months of vc_m (1-case_cum_m), where vc_m is the proportion of total vaccine given in month m and case_cum_m is the proportion of total cases that occurred by the end of month m.

We evaluated the following seven test methods (Supplemental Table 2):

We analyzed real month-level data from 2010–11 to 2016–17 by age-group and season to compare the numbers of averted cases estimated by method 1 (the current method) and the two best-performing test methods. We created all-age estimates by umming age-group-specific estimates. Finally, we performed a sensitivity analysis to examine the effect on estimates if true vaccine coverage were 10–20% lower than estimated by CDC using self-reported vaccination status [3].

Data for this analysis came from publicly available sources, preexisting research projects with human subjects approval, and public health surveillance systems that have been determined to not require human subjects review. Calculations were done using SAS version 9.4 (Cary, N.C.) and R [13].

In the initial synthetic dataset, there were 120,000 cases in the absence of vaccination and 55% of the population was vaccinated. The reference model estimated 91,232 cases with vaccination and 28,768 cases averted by vaccination (Table 2). Compared with the reference model, the test methods varied from a 9% underestimate (method 1) to a 14% overestimate (method 6) of averted cases. Estimates from methods 2 and 3 were most accurate (<1% error).

We next compared the current method (method 1) and the two best-performing test methods (methods 2 and 3) in simulated datasets with widely varying characteristics (Table 3). As vaccination timing (the proportion vaccinated before cases) increased from 5% to 98%, true averted cases increased from 1630 to 31,159, and method 1 errors varied from an 90% overestimate to an 12% underestimate. In contrast, relative errors for methods 2 and 3 were much lower; of note, the maximum errors of 6.0–6.2% for methods 2 and 3 corresponded to absolute differences of <100 averted cases. When we varied vaccine effectiveness, vaccine coverage, and infection risk one at a time, we observed that method 1 errors varied but methods 2 and 3 repeatedly performed well, with maximum relative errors of 3.1% and 2.6%, respectively. However, when multiple factors were varied to produce high numbers of averted cases (vaccine effectiveness 70%, vaccine coverage 80%, and infection rate 50%), methods 2 and 3 produced overestimates of up to 26% and 24%, respectively.

We also evaluated the effect of varying the immune lag (i.e., days from vaccination to protective immunity) from 1 to 28 days in the simulated data (Table 3). For a lag of 14 days, methods 2 and 3 performed well with errors of 0.2% or less. However these methods produced underestimates of up to 4% when lag was <14 days and overestimates of up to 7% when lag was >14 days.

We analyzed the real data from 2010–11 to 2016–17 seasons, comparing averted cases among methods 1–3 (Table 4). For all ages combined, methods 2 and 3 produced estimates that were a median 8% higher than method 1; only in 2014–15 did methods 2 and 3 produce estimates that were lower than method 1 (median 3% lower). Among the age groups, median differences were highest for those 6 months-4 years (methods 2 and 3 were both 17% higher than method 1) and lowest for those 18–49 years (methods 2 and 3 were both 2% higher than method 1).

Using method 3 (method 2 gave nearly identical results), we determined the effect on our estimates if true vaccine coverage was lower than reported by CDC based on self-reported vaccination status. Among the 35 data subsets defined by season and age group, if true coverage were 10% lower on a relative scale (e.g., a change from 50% to 45%), averted cases would be a median 12% (range 10–15%) lower (also on a relative scale) than originally estimated; if true coverage was 20% lower, averted cases would be a median 24% (range 20–29%) lower than originally estimated.