The test-negative study design has been applied for estimating influenza vaccine effectiveness in many countries since 2005 and monitoring changes in vaccine effectiveness from year to year.^4-11 In this variant of the case control study, patients presenting for ambulatory care with acute respiratory infection are tested for influenza virus. Cases are defined as patients testing positive for influenza virus, while those testing negative form the control group. Similar to a traditional case control study, vaccine effectiveness in a test-negative design is estimated as one minus the odds ratio of vaccination coverage among test-positive patients versus test-negative patients, adjusted for potential confounding factors, such as age, to permit a causal interpretation of the adjusted association as a measure of vaccine effectiveness.^12-14

Influenza viruses co-circulate with other respiratory viruses during the winters in temperate locations. Virus interference describes an ecologic phenomenon that is occasionally reported, where an epidemic of one virus appears to ‘interfere’ with the occurrence of another epidemic, either by delaying its start, or by advancing its end if it has already started.^15,16 Studies have reported apparent interference between epidemics of influenza and epidemics of other respiratory viruses, most notably rhinoviruses, which are the most frequently detected viruses in many studies of patients with acute respiratory infection.^16-20 Some studies have suggested that virus interference may occur in some seasons but not others, or may occur after epidemics of some influenza types/subtypes but not others.^16,21 This phenomenon is still controversial and is not consistently observed. The biological mechanism causing the ecological phenomenon of virus interference could be temporary non-specific immunity following an episode of acute respiratory infection, due to stimulation of the innate immune response during and for a short time after infection.^22,23 Consistent with that mechanism, one small randomized trial of influenza vaccination in Hong Kong reported a significantly increased risk of other respiratory viruses among recipients of influenza vaccination.²⁴

To permit valid estimation of vaccine effectiveness, one assumption of the test-negative design is that the probability of infection with another respiratory virus is not affected by receipt of influenza vaccination.^26,27 Whether virus interference could affect vaccine effectiveness estimation in the test-negative design remains unclear, although a simulation study showed that only a small bias would be introduced during typical influenza epidemics because the cumulative incidence of influenza virus infection is thought to be relatively low in a typical epidemic and therefore any consequent interference effect would be limited.^27-29 A study examining the potential for virus interference to affect vaccine effectiveness by comparing estimates from test-negative design-type analyses verses cohort-type analyses of data from randomized controlled trials of live attenuated influenza vaccines also found little difference between estimates generated by the two types of study.³⁰ However, when the incidence of influenza infection is relatively high compared to those of other respiratory viruses, or when non-specific immunity following infection could last for a longer period, vaccine effectiveness estimates obtained from the test-negative design could be biased.^28,31

To examine the potential for virus interference in a test-negative study in the United States, we used data from the Influenza Incidence Surveillance Project (IISP), which was established by the US Centers for Disease Control and Prevention (CDC).^32-35 Data on patients' self-reported vaccination history and influenza and other respiratory virus testing results were collected for the 2010-11, 2011-12, and 2012-13 influenza seasons. Vaccine strains were generally well matched to the prevalent strains each season.^36-38 The objectives of the present study were to evaluate the potential for virus interference to affect vaccine effectiveness estimates during the entire influenza season or different phases in each influenza season by (1) determining the sensitivity of vaccine effectiveness estimates to the choice of control group and; (2) examining the associations between influenza vaccination and detection of other respiratory viruses in patients who tested negative for influenza. If virus interference were to occur, people who received influenza vaccination could have a lower risk of influenza virus infection and a higher risk of infection with other respiratory viruses. Patients who tested positive for other respiratory viruses might be more likely to be vaccinated than the general population.²⁴ Thus estimates by different control groups could be different, and association between influenza vaccination and detection of other respiratory viruses could be away from null.

From August 2, 2010 to December 31, 2013, 17652 patient specimens were collected. We excluded patients who were not tested for influenza virus (n=572), who were less than 6 months of age (n=397), or presented outside of influenza seasons (n=6033). At least 10 test-positive patients per week were identified for 20, 17, and 23 consecutive weeks in the 2010-11, 2011-12, and 2012-13 seasons respectively, and the weekly proportions of patients testing positive for influenza peaked in late February 2011, in mid-March 2012, and in early January 2013, respectively (Figure 1). The influenza season in 2010-11 was a mixed season with co-circulation of three influenza types/subtypes, while the season in 2011-12 and 2012-13 was dominated by A(H3N2). A total of 4208, 2164, and 4278 patients were included in each of these three influenza seasons respectively, and a total of 10650/17652 (60%) patients were included in the main analyses. Among 10650 patients tested for influenza, 10614 (99.7%) patients were further tested for rhinoviruses, RSV, PIV 1-3, MPV and adenovirus.

In both influenza-positive cases and the pan-negative control group (those who tested negative for both influenza and other respiratory viruses), the majority of patients were school-age children 6 to 17 years of age and adults 18 to 49 years of age (Table 1). In the other respiratory virus positive control group 1330/2835 (47%) patients were young children aged 6 months to 5 years, while in the pan-negative control group there were only 703/4036 (17%) young children aged 6 months to 5 years. Slightly more than half of the patients were female among influenza-positive cases, other virus positive controls and pan-negative controls while a larger proportion of females was observed in the pan-negative control group (58%). Influenza positive patients less frequently reported receipt of influenza vaccination compared to other control groups, and approximately 20% of all groups were missing vaccination status (Table 1). The most commonly detected other respiratory virus was rhinovirus (n=1101), followed by RSV (n=783) (eTable 1). Children aged 6 months to 5 years comprised a large proportion of the cases infected with rhinovirus (n=442) and RSV (n=496) (eTable 1).

We compared influenza vaccination coverage between influenza negative controls, other respiratory virus positive controls and pan-negative controls, stratified by influenza season and age group after multiple imputation (Table 2). There was no difference in vaccine coverage within each age group and influenza season in complete data analysis verses multiple imputation (results not shown). Vaccination coverage was higher among younger children aged 6 months to 5 years and adults aged >50 years, and lowest among young adults aged 18 to 49 years across three influenza seasons. We found no differences in vaccination coverage for each age group by influenza season except among children aged 6 months to 5 years among which vaccination coverage was higher in pan-negative controls (48%) during the 2010-11 season and in other virus positive controls (58%) during 2011-12. We also examined influenza vaccination coverage by other respiratory virus tested and found no association between influenza vaccination and detection of RSV, rhinoviruses, PIV 1-3, MPV, and adenovirus for each age group compared with pan-negative controls (eTable 1).

For our first analysis, we estimated vaccine effectiveness using conditional logistic regression matching by calendar week. The overall vaccine effectiveness for those testing influenza negative was 47% (95% confidence interval, CI: 42%, 52%) (Table 3). Consistent between each of the control groups, vaccine effectiveness tended to decrease with age. Using each of the three control groups, vaccine effectiveness estimates against influenza A(H3N2) across three seasons fell in the range 50%-60% for children aged 6 months to 5 years while point estimates were generally lower for the other three age groups (Figure 2). The vaccine effectiveness estimates by influenza type/subtype indicated a higher effectiveness against influenza A(H1N1) compared to A(H3N2) and B. Although estimation for adults older than 50 years was not possible, vaccine effectiveness estimates against influenza (H1N1) were all larger than 45% for other age groups. Estimates of vaccine effectiveness using controls testing positive for another respiratory virus were slightly higher than estimates using controls that were influenza negative or pan-negative, but estimates by all control groups had widely overlapping confidence intervals, and differences between estimates were mostly less than 10 percentage points when stratified by season or influenza virus type/subtype (Figure 2). The estimates by three control groups were consistent and no differences by control group were found when stratified by age groups, influenza season, and/or type/subtype (Figure 2, eFigure 1, Table 3).

In our second analysis, we further examined the association of influenza vaccine effectiveness by age group and with detection of other respiratory viruses in the early, middle and late phases of each season as defined above. We found no major differences between the three vaccine effectiveness estimates in each phase of each influenza season (eFigure 2). In our third analysis, we found no clear evidence of an association between other respiratory virus detection and receipt of influenza vaccination (Figure 3). Likewise, we found no association between influenza vaccination and other respiratory virus detection when further stratified by age group (Figure 4) or individual respiratory virus detection (eFigure 3). When analyses were restricted to patients meeting the influenza-like illness case definition only, 6792 patients in three influenza seasons were included in statistical analysis, and the findings were very similar and with wider confidence intervals on point estimates (data not shown).

Based on 10650 patients 6 months to 99 years of age, we found no difference in estimates of vaccine effectiveness against any influenza virus when using influenza test negative controls, other respiratory virus positive controls, or pan-negative controls (those who tested negative for both influenza and other respiratory viruses) over a period of three seasons in the United States. Previously, a randomized controlled trial for children aged 6 to 17 years in Hong Kong reported that vaccine effectiveness estimates may differ depending on choice of control groups, and thus needed further research.²⁴ Using surveillance data from the United States in the present study, we addressed the impact of control groups and observed no substantial differences across the three control groups in overall vaccine effectiveness estimates and estimates stratified by age, type/subtype, or season (Table 3, Figure 2), which is consistent with previous theoretical considerations of the study design.^26,31 Although comparatively larger differences were observed between vaccine effectiveness estimates based on the three control groups during the 2011-12 season (Figure 2, eFigure 1, eFigure 2), this could be explained by the smaller influenza peak and thus smaller sample size during that season (Figure 1, Table 3). Moreover, no difference in effectiveness estimates based on the three control groups were found in this season overall or when stratified by age group, influenza type/subtype, and early/middle/late phase.

The results of the present study are consistent with other studies that compared alternative control groups when estimating influenza vaccine effectiveness in different countries and settings.^7,29,41-44 Four studies examined virus interference from pooled estimates over more than one influenza season: one of these studies recruited children in hospitals in Australia,⁴² one recruited children from hospitals in Hong Kong,⁷ one recruited hospitalized pneumonia patients of all ages in the United States⁴³ and one included children and older adults seeking ambulatory care or admitted to hospitals in the US.⁴¹ Consistently, no effect of virus interference was detected according to pooled estimates across several influenza seasons in these studies. Moreover, two other studies from Australia and New Zealand examined vaccine effectiveness estimates for a single influenza season by alternative control groups, and very similar estimates were derived using influenza negative controls and other respiratory viruses positive controls.^29,44 Three other studies that recruited relatively smaller numbers of patients in single influenza seasons found larger differences in vaccine effectiveness point estimates by alternative control groups with overlapping confidence intervals.^21,31,45 Although virus interference was given as a possible explanation for differences in vaccine effectiveness estimates by control group, those studies may have been underpowered to identify real effects. Previous studies have suggested that choosing a control group testing positive for another respiratory virus might be preferred to one based only on testing negative for influenza because detection of other viruses confirms adequate swab quality, especially for older children or adults, who can have lower levels of virus shedding compared to young children.^42,45 In our study, we did find that other respiratory viruses were more likely to be detected among young children compared to other age groups, but there were no major differences between vaccine effectiveness estimates using the control group testing positive for other respiratory viruses and the control group testing negative for influenza_- (Table 3, Figure 2).

We found consistent detection of other respiratory viruses among vaccinated and unvaccinated patients during three phases of each influenza season (Figure 3), providing further evidence that virus interference did not have any substantial effect on vaccine effectiveness estimates in this study. Based on the hypothesis that unvaccinated persons might have a higher risk of influenza virus infection, but then a higher proportion of unvaccinated persons would have temporary protection around and after the peak in influenza activity due to short-term stimulation of innate non-specific immunity^22,46 or reduced exposure during convalescence,²⁵ we examined the association between influenza vaccination and detection of other respiratory virus during the early, middle, and late phases of each influenza epidemic among influenza-negative patients. However, no differential association was found among any age group in each season (Figure 4). This evidence, across three seasons, indicates that virus interference did not appear to have any substantial effect on estimates of vaccine effectiveness using the test-negative design.

While vaccine effectiveness estimates varied by age group, vaccination coverage was quite similar among control groups, with the exception of children aged 6 months to 5 years in the 2010-11 and 2011-12 influenza seasons. We found no association between influenza vaccination and detection of any other respiratory virus by age group, and consistently, our finding suggested no age-group specific association between influenza vaccination and detection with each of the other respiratory viruses tested including RSV, rhinoviruses, PIV 1-3, MPV and adenovirus (eTable 1). Furthermore, there was no association between influenza vaccination and detection of other respiratory viruses after adjusting for age group and sex and conditioning on calendar week (eFigure 3). We found that the risk of other respiratory virus detection was not associated with influenza vaccination for each age group or overall, with point estimates around unity and confidence intervals all across unity (Figure 4). Similarly, the 2013 study by Sundaram et al. examined the association between influenza vaccination and detection of RSV, adenovirus, MPV, PIV 1-4, rhinoviruses, or coronaviruses among children aged 6 months to 5 years and adults elder than 50 years using univariate analysis.⁴¹ No association was identified except for PIV 1-4, and the association was reversed between children and adults. Furthermore, a hospital-based study conducted in Spain also examined vaccine effectiveness against RSV-related hospitalization and reported no association between receipt of influenza vaccination and detection of RSV.⁴⁷

A strength of this study is that the IISP platform incorporates routine molecular testing for influenza and other respiratory viruses, allowing comparison of vaccine effectiveness based on different control groups and estimation of potential of virus interference. A second advantage of this study is the large sample size, which permitted precise estimation of season-specific vaccine effectiveness and vaccine effectiveness by early/middle and late phases of each influenza season in most age groups, which has not been reported in previous studies. We were also able to examine the impact of choices of alternative test-negative groups on a seasonal basis for most age groups (eFigure 1. eFigure 2), and thus provide robust evidence on the potential for virus interference to affect VE estimates. Nevertheless, the difference in point estimates was 7 percentage points between the vaccine effectiveness estimate using the control group testing positive for other respiratory viruses and the estimate using the pan-negative control group (Table 3). While the confidence interval was across 0, and despite our large sample size, we could not rule out small effects of virus interference on vaccine effectiveness. Our study is limited by self-reported data on vaccination status, which precludes evaluation by type of vaccine received and might introduce recall bias. If the bias was non-differential, vaccine effectiveness could be underestimated across three alternative control groups. In addition, we could not adjust for underlying medical conditions, which is a potential confounding factor. Another limitation is that around 20% of patients had missing vaccination records. We conducted multiple imputations by generalized additive model to minimize potential bias and make the most of all available data. The IISP is a large surveillance program; however, within year, age group and specific influenza virus types, we were insufficiently powered to calculate vaccine effectiveness for some subgroups of patients. Finally, patients testing pan-negative may have had false negative results due to low levels of virus shedding or imperfect swab collection, and that would lead to misclassification of outcome status and reduce the power of our study to identify differences between the various control groups.

In conclusion, we did not find any evidence that virus interference affected vaccine effectiveness estimation in a test-negative study using all influenza negative, other virus positive, and all virus negative control groups over three consecutive seasons in the United States. The dynamic interactions of respiratory viruses and their circulation among humans are not well understood, and virus interference remains a controversial phenomenon. Simulation studies have indicated that unless there is very high incidence of influenza virus infections in the community, and prolonged non-specific immunity following one infection, interference is unlikely to cause any major bias in vaccine effectiveness estimates from the test-negative design.²⁸ Our empirical findings also support the use of the test-negative control study design for the assessment of vaccine effectiveness in protecting individuals of all ages against influenza virus infection.