78028714294Int J EpidemiolInt J EpidemiolInternational journal of epidemiology0300-57711464-368526979985502533610.1093/ije/dyw022HHSPA774445ArticleThe case test-negative design for studies of the effectiveness of influenza vaccine in inpatient settingsFoppaIvo M.12*FerdinandsJill M.12ChavesSandra S.1HaberMichael J.3ReynoldsSue B.12FlanneryBrendan1FryAlicia M.1Influenza Division, Centers for Disease Control and Prevention, Atlanta, GA, USABattelle Memorial Institute, Atlanta, GA, USARollins School of Public Health, Emory University, Atlanta, GA, USACorresponding author: Centers for Disease Control and Prevention, 1600 Clifton Road NE, MS A-20, Atlanta, GA 30333, USA. vor1@cdc.gov54201601122016245201745620522059This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.comBackground

The test-negative design (TND) to evaluate influenza vaccine effectiveness is based on patients seeking care for acute respiratory infection, with those who test positive for influenza as cases and the test-negatives serving as controls. This design has not been validated for the inpatient setting where selection bias might be different from an outpatient setting.

Methods

We derived mathematical expressions for vaccine effectiveness (VE) against laboratory-confirmed influenza hospitalizations and used numerical simulations to verify theoretical results exploring expected biases under various scenarios. We explored meaningful interpretations of VE estimates from inpatient TND studies.

Results

VE estimates from inpatient TND studies capture the vaccine-mediated protection of the source population against laboratory-confirmed influenza hospitalizations. If vaccination does not modify disease severity, these estimates are equivalent to VE against influenza virus infection. If chronic cardiopulmonary individuals are enrolled because of non-infectious exacerbation, biased VE estimates (too high) will result. If chronic cardiopulmonary disease status is adjusted for accurately, the VE estimates will be unbiased. If chronic cardiopulmonary illness cannot be adequately be characterized, excluding these individuals may provide unbiased VE estimates.

Conclusions

The inpatient TND offers logistic advantages and can provide valid estimates of influenza VE. If highly vaccinated patients with respiratory exacerbation of chronic cardiopulmonary conditions are eligible for study inclusion, biased VE estimates will result unless this group is well characterized and the analysis can adequately adjust for it. Otherwise, such groups of subjects should be excluded from the analysis.

Influenza vaccine effectivenesstest-negative designinfluenza-associated hospitalizationselection biascomputer simulationIntroduction

The antigenic variability of seasonal influenza viruses requires frequent reassessments of the effectiveness of vaccines designed to prevent influenza infection and morbidity. However, randomized controlled trials are no longer ethical in settings where seasonal influenza vaccination is widely recommended, such as in the USA.1 The socalled ‘test-negative’ design (TND) has become a popular choice for post-licensure observational studies of the effectiveness of vaccines for influenza.2–10 Both cases and controls are identified in a clinical setting among patients meeting certain clinical criteria, e.g. for acute respiratory infection (ARI) or ‘influenza-like illness’ (ILI) and who consent to participate in the study. Those testing positive for influenza with a sensitive and specific assay, usually by reverse-transcription polymerase chain reaction (RT-PCR), are designated cases and those who test negative for influenza are used as controls. Vaccine effectiveness (VE) is calculated as VE = (1- ORV) * 100%,11 where ORV is the ratio of the odds of being vaccinated in cases vs the odds being vaccinated in controls or, equivalently, the odds of being a case in vaccinated vs the odds of being a case in unvaccinated study subjects. The popularity of this design in ambulatory settings can be explained not only by its ease of implementation, but also by the implicit conditioning on healthcare-seeking practices, which eliminates an important source of selection bias. The general validity of VE estimates obtained from TND studies conducted in ambulatory settings (ambulatory TND) has recently been confirmed for a wide range of conditions.11–13

Recent investigations have used a TND approach to examine the effectiveness of influenza vaccination against influenza-related hospitalizations.14–25 VE estimates from both inpatient and outpatient TND studies are used as measures of VE for specific influenza seasons and inform public health responses. The validity of the TND in the inpatient setting (inpatient TND), however, has yet to be examined. In fact, some VE estimates from inpatient TND studies appear to exceed VEs usually encountered in ambulatory settings. For example, Belongia et al. estimated influenza vaccine effectiveness for all ages against influenza-associated hospitalizations for the seasons 2006–7 in the USA to be 88% (95% confidence interval (CI) 13%, 100%), whereas North American ambulatory TND studies reported adjusted VE estimates of 52% (CI 22%, 70%)4 and 46% (CI 17%, 65%)26 for the same season. Similarly, Gefenaite et al.22 reported for the season 2012–13 an adjusted adult influenza VE of 86% (CI 19%, 97%) from an inpatient TND study in Lithuania. A European ambulatory TND study from the same year reported a VE of 49% (95% CI 32%, 62%).27 This discrepancy could reflect an effect of vaccination on disease severity, i.e. vaccinated individuals might tend to develop less severe influenza disease if infected and thus be less likely to require hospitalization, or it might simply reflect the lack of precision in the inpatient TND VE estimates. However, it could also be the result of selection bias if influenza-negative controls are substantially different from the source population in their uptake of influenza vaccination. For instance, subjects suffering from chronic conditions such as congestive heart failure or chronic obstructive pulmonary disease (COPD) (CP individuals) are more likely to be hospitalized with non-infectious respiratory disease (e.g., decompensation of their heart condition or COPD exacerbation). If they are also more likely to be vaccinated than the source population from which the cases are drawn, selection bias would result.28

Here, we first examine the interpretation of VE estimates obtained from inpatient TND studies: does VE represent the level of protection against influenza hospitalizations? We then examine the effect on VE estimates of enrolling CP individuals for respiratory non-ARI exacerbation. We also investigate the effect of accurate and inaccurate adjustment for CP status on VE estimates. We theoretically derive the mathematical quantities of interest and use a simulator to verify the theoretical results.

MethodsAssumptions

We make use of notation previously described.11 Parameters with their symbols and baseline values are shown in Table 1. We assume that the incidence of influenza ARI and ARI of other aetiologies (‘non-influenza’) is driven by incidence rates λI(t) and λNI(t), respectively, where t represents time in days. We assume that influenza viruses represent one antigenic entity such that infection with the virus results in full immunity to influenza viruses for the remainder of the study. ARI of non-influenza aetiology does not change the future risk of acquiring ARI of any aetiology (influenza, non-influenza). Study inclusion criteria are broad enough to allow for the enrolment of subjects admitted to inpatient care for respiratory exacerbation of underlying chronic medical conditions, such as COPD, asthma or congestive heart failure, hence referred to as CP conditions. To differentiate such events from ARI-related events we will use the term non-ARI events. Individuals suffering from CP conditions are also assumed to have a higher vaccination uptake than the remainder of the population. All subjects are susceptible to influenza infection before influenza vaccination or natural infection with influenza virus. Influenza vaccination is completed before the beginning of the study period. VE is the same for CP and non-CP subjects and an ‘all-or-none’ model of the vaccination effect is assumed according to which a proportion φ (=VE) of those susceptible to influenza who were vaccinated become fully immune to influenza infection (vaccination-mediated); accordingly, influenza vaccination fails to protect a proportion 1 − φ against influenza virus infection (vaccine failure). Despite vaccine failure, an individual’s probability of becoming hospitalized with influenza may be reduced by the factor (1 − ι) (Greek letter iota). This represents a mitigating effect of influenza vaccination on influenza disease severity. Influenza vaccination does not directly modify the probability of non-influenza outcomes. CP subjects are at higher risk for non-ARI events than non-CP subjects (λCP1(t) and λCP0(t), respectively). Given an ARI or non-ARI event, subjects are hospitalized with a given probability which depends on the type of event and on their CP status: For example, γNI1 is the probability of a CP subject being hospitalized with non-influenza ARI; see Table 1. All subjects hospitalized with an ARI or non-ARI event are tested for influenza and included in the study with probability σ. The laboratory test used to assess influenza infection has perfect accuracy. Similarly, both vaccination and CP status (0 or 1) are assessed accurately. We assume that CP status is adjusted for in the analysis by the use of a binary covariate in the logistic regression analysis. For the sake of simplicity we ignored confounders of practical importance such as age and calendar time.

Simulation study

We simulated the daily incidence of the events of interest (influenza and non-influenza ARI, non-ARI events), the occurrence of relevant downstream events (hospitalization, influenza testing, study participation etc.) which resulted in simulated data sets that were analysed using logistic regression analysis. This allowed us to investigate the effect of certain parameters on resulting VE estimates (Supplement 2, available as Supplementary data at IJE online). We did not model transmission, but based the daily incidence on given incidence rate functions for all events (see below). Briefly, the population is subdivided into non-CP (‘normal’), and CP subjects, who can be either vaccinated or unvaccinated. Infections with influenza virus, with non-influenza ARI agents and non-ARI events were generated for each day of the simulation by applying the respective incidence rates to the respective population groups. Driven by values of the parameters that represent conditional probabilities, such as the probability of non-CP subjects being hospitalized given influenza infection (e.g. γI0), data generation from an inpatient TND study is simulated. To assess the ‘meaning’ of VE estimates from inpatient TND studies, hypothetical cohort studies, representing the whole population, were simulated in which inpatient TND studies were nested. We then varied ι to produce a specific VE against hospitalization:

φ* = 1 − (1 − ι) (1 − φ) (see Supplement 1 S9 available as Supplementary data at IJE online) and estimated φ* as φ^*=1−RR where RR is the relative risk of influenza hospitalization in those vaccinated compared with those not vaccinated, estimated by CP-adjusted binomial regression and exponentiating the coefficient estimate associated with vaccination. For each simulated TND study we estimated VE using logistic regression analysis, adjusting for CP status. We then compared the inpatient TND VE estimates with the cohort values in 10 000 simulations for each φ*.

To investigate the effect of CP status on VE estimates from inpatient TND studies, we simulated 10 000 studies, for each calculating the crude VE, VE by CP status (separate analysis for CP and non-CP subjects) and adjusted VE using logistic regression analysis (Supplement 1). We also simulated the situation where CP subjects were not homogeneous with respect to their vaccination uptake and risk of non-ARI events, such that their marginal vaccination uptake and non-ARI risk remained the same (80% and 4 per 1000 per day, respectively), but two-thirds of them suffered 5-fold higher rates of non-ARI events compared with non-CP subjects and one-third suffered 20-fold higher rates of non-ARI events than non-CP subjects. Of the former group 75% were vaccinated, whereas 90% were vaccinated of the latter group.

The simulation model was implemented using R 3.1.129 and can be downloaded (Supplement 2). All simulations were based on fixed sets of parameters (Table 1) unless explicitly stated. The bias was calculated as the median difference between the estimated VE and φ, along with the 2.5th and 97.5th percentiles of that difference as empirical 95% confidence intervals (CI).

ResultsThe interpretation of VE estimates from inpatient TND studies

Theoretically, if the controls are representative of the source population with regard to vaccine receipt, if both outcome and vaccination status are accurately measured and if the vaccine provides ‘all-or-none’ protection, VE estimates from inpatient TND studies should represent unbiased estimates of the true VE against laboratory-confirmed influenza hospitalization for the general population (Supplement 1, equation S9). This was confirmed by simulation where the VE estimates based in inpatient studies were highly consistent with the actual protection from influenza hospitalization in the population cohort (Table 2, columns 2–4) even though they are derived from hospitalized subjects only. If, in addition, vaccination does not modify influenza disease severity (ι = 0), VE against hospitalization equals VE against influenza virus infection (Supplement 1, S10, Table 2, first row). Given VE against infection (φ), VE against laboratory-confirmed influenza hospitalization, φ*, increases linearly with the ‘attenuation factor’ ι because φ* = 1 − (1 − ι) (1 − φ). This was confirmed by simulation (Table 2). The difference between VE against infection (φ) and the VE against laboratory-confirmed influenza hospitalization (φ*), given a certain ‘attenuation factor’ ι, is larger for smaller values of φ than it is for larger values (Supplement 1, S9). As only outcomes that test positive for influenza, usually by molecular methods such as RT-PCR, define case status, hospitalizations due to late complications of influenza will not be captured and the VE estimated in inpatient TND studies relates only to hospitalization with laboratory-confirmed influenza and not necessarily to all influenza-associated hospitalizations.

Bias in VE estimate from an inpatient TND

If CP status is associated both with high vaccine uptake and with the risk of non-ARI events, and if such events are eligible for study inclusion, VE estimates that are not adjusted for the CP status (‘crude’) will be biased. This bias can be classified as selection bias because highly vaccinated controls are selectively included in the study, which leads to a misrepresentation of the source population in terms of vaccination prevalence, biasing VE estimates towards falsely high values (Table 3 and Supplement 1). The magnitude of the bias depends on the assumptions regarding the incidence of non-ARI events in both non-CP and CP individuals, as well as the vaccination coverage in both groups. If vaccination coverage does not depend on CP status, then unbiased estimates result without CP-adjustment (Supplement 1; simulation results not shown). CP-adjusted VE estimates were sensitive to parameters that drive the accuracy of the influenza test, as well as assessment of vaccination and CP status (Supplement 1). If CP status is either adjusted for accurately or VE is estimated separately for CP and non-CP individuals, unbiased estimates will result (Table 3 and Supplement 1).

Bias in VE estimates from inpatient TND with heterogeneity in CP status

If CP status is not a binary characteristic, but CP individuals rather fall into two or more categories defined by vaccination uptake and study inclusion probabilities (controls), and if that heterogeneity is not fully characterized, adjustment for CP status will not result in full removal of the bias (Table 3). This scenario mimics a situation in which CP status cannot be fully characterized. Similarly, stratification on the imperfect binary CP indicator will result in a biased VE estimated among CP individuals (restricted analysis) (Table 3). However, as long as non-CP subjects (CP = 0) are homogeneous with respect to vaccination uptake and probability of being included in the study as controls, the analysis restricted to non CP subjects will give rise to unbiased VE estimates (Table 3).

Discussion

We demonstrated that, under certain general assumptions, VE estimates from inpatient TND studies can be interpreted as the level of protection bestowed by influenza vaccination against hospitalization with laboratory-confirmed influenza for the source population, despite the fact that the study is conducted only among inpatients. If vaccination does not mitigate disease severity in break-through influenza infections, then VE estimates from inpatient TND studies also quantify protection from influenza virus infection. Because proof of active infection with influenza virus is required for the case definition, late complications of influenza leading to hospitalization are not captured by inpatient TND study-derived VE estimates. Such test-negative VE estimates may thus underestimate the level of protection from complications of influenza infection.

The TND offers logistical advantages over other study designs since all enrolled patients are utilized either as cases or controls. Even though the methodological issues faced by inpatient TND studies do not fundamentally differ from the issues encountered by TND studies conducted in ambulatory patients, there are important practical differences: In studies of ambulatory patients seeking care for non-life-threatening ARI, the choice by individuals to visit a healthcare provider is a sine qua non for study inclusion. The TND implicitly corrects for the selection bias associated with healthcare-seeking behaviour11–13 which is, besides its practical advantages, the main benefit of the ambulatory TND. In the inpatient setting, access to care may introduce selection bias that could be avoided by conditioning on hospital admission.

We did, however, identify a potential source of selection bias that could affect VE estimates from inpatient TND studies. Certain chronic conditions, here referred to as CP conditions, that may be associated with high vaccination coverage may also cause inpatient admission of subjects for non-ARI events, e.g. respiratory exacerbation of their underlying condition that are not associated with influenza infection. If these events meet the study inclusion criteria, these subjects would be enrolled as controls. There, in fact, is indirect evidence for the over-representation of CP subjects among controls in some inpatient TND. For example, cases and controls may differ in indicators of chronic illness that are predictive both of vaccination status and of the probability to be hospitalized with respiratory non-ARI events. In several inpatient VE studies,18,23,24,30,31 controls were older than cases and much more likely to suffer from heart disease and pulmonary disease, as well as other conditions which, conceivably, might have increased their vaccination coverage and their probability of being admitted for inpatient care for non-ARI events and of thus of being enrolled in the study as controls.

We have shown, both by theoretical considerations and by simulation that, if these conditions (CP conditions) are adequately adjusted for or if the analysis is restricted to non-CP conditions and if both influenza infection and vaccination status are assessed with a high level of accuracy, unbiased VE estimates can be obtained from inpatient TND studies even if subjects are enrolled in the study because of non-ARI events. In reality, however, adjustment for CP status may not be straightforward. If, for example, “CP subjects” are heterogeneous with respect to their risk for non-ARI events that may lead to study inclusion and with respect to their vaccination uptake, inaccurate adjustment by CP status will result in biased VE estimates. This problem of inaccurate adjustment of selection bias is well known, for example, in educational research.32 A sensitivity analysis comparing full VE estimates with VE estimates obtained from the data restricted to not chronically ill patients might indicate a problem with selection bias if the two estimates are substantially different. In that case, the restricted estimate should be reported. It is important to note that we have focused our analysis on a scenario which is more representative of inpatient TND studies in adults rather paediatric populations which may offer quite different challenges.33

Our analysis of the inpatient TND has some limitations. First, we assumed perfect accuracy in the assessment of both influenza and vaccination status. Jackson et al.34 recently showed that, although misclassification of influenza status tended to result in a slightly greater bias of VE estimates in TND studies compared with other designs, the magnitude of the bias was trivial under realistic assumptions regarding VE, the accuracy of RT-PCR and influenza attack rates. The difficulty in detecting late complications of influenza infection can also be construed as a problem of sensitivity. Even though the impact of inaccurate assessment of vaccination status is less understood, the potential effect of inaccurate characterization of vaccination status on VE estimates is concerning, although not specific for inpatient TND studies. It has been shown that vaccination self-reports may be unreliable, often leading to underreporting of influenza vaccination,35–37 but occasionally to over-reporting.38 On the other hand, neither medical records nor vaccine registries are likely perfect sources for vaccination status.39 Misclassification of vaccination status could have unpredictable consequences for the resulting VE estimates. Our sensitivity analysis (Supplement 1) confirms that misclassification of case-control, vaccination or CP status is a source for concern. Second, we assumed an ‘all-or-none’ vaccination effect model, according to which vaccination either results in full immunity or full susceptibility to infection. As we have shown previously,11 odds ratio-derived VE estimates are biased toward 0 if vaccination reduces the instantaneous risk by a given fraction (VE) instead, a mechanism referred to as ‘leaky vaccine’ model.40 Third, our quantitative evaluation of biases in VE estimates is based on assumptions about the incidence of influenza infection, non-influenza ARI and non-ARI events, as well as assumptions about hospitalization probabilities and, importantly, the prevalence of underlying medical conditions that are associated with both vaccination coverage and the likelihood of study inclusion as controls (CP status). These parameters are highly context-dependent and can be chosen to produce both trivial and massive biases of VE estimates not adjusted for CP. Finally, we assumed influenza virus to represent a single antigenic entity. This assumption clearly does not capture the antigenic and immunological complexity of questions regarding vaccine effectiveness that arise from the interaction between sequential natural exposure and vaccination responses.41,42. More refined models of influenza circulation and of immunological mechanisms involved in the effect of influenza vaccination may reveal different sources of biases.

Supplementary Material

Supplementary Data

Supplementary data are available at IJE online.

Conflict of interest: We are not aware of any potential conflict of interest.

We would like to thank Drs David Shay and Jerry Tokars, both CDC, for their helpful comments on the manuscript.

Funding

MJH’s research was supported by the National Institute of Allergies and Infectious Diseases of the National Institutes of Health (NIH) under Award R01AI110474, and by Intergovernmental Personnel Act (IPA) 1110376-05 from the Centers for Disease Controls and Prevention (CDC). The content of the manuscript is solely the responsibility of the authors and does not necessarily represent the official views of the NIH or the CDC.

ReferencesCenters for Disease Control and PreventionPrevention and control of influenza with vaccines: recommendations of the Advisory Committee on Immunization Practices (ACIP) - United States, 2012–13 influenza seasonMMWR Morb Mortal Wkly Rep20126161361822895385BarretASDonnellJOO’HoraAEffectiveness of 2010/2011 seasonal influenza vaccine in IrelandIr Med J2012105394222455236BelongiaEAKiekeBADonahueJGInfluenza vaccine effectiveness in Wisconsin during the 2007–08 season: Comparison of interim and final resultsVaccine2011296558656321767593BelongiaEAKiekeBADonahueJGEffectiveness of inactivated influenza vaccines varied substantially with antigenic match from the 2004–2005 season to the 2006–2007 seasonJ Infect Dis200919915916719086915CastillaJMartinez-ArtolaVSalcedoEVaccine effectiveness in preventing influenza hospitalizations in Navarre, Spain, 2010–2011: cohort and case-control studyVaccine20113019520022100636FieldingJEGrantKATranTKellyHAModerate influenza vaccine effectiveness in Victoria, Australia, 2011Euro Surveill201217pii: 20115Jimenez-JorgeSSavulescuCPozoFEffectiveness of the 2010–11 seasonal trivalent influenza vaccine in Spain: cycEVA studyVaccine2012303595360222472792KisslingEValencianoMEarly estimates of seasonal influenza vaccine effectiveness in Europe among target groups for vaccination: results from the I-MOVE multicentre case-control study, 2011/12Euro Surveill201217pii:20146KisslingEValencianoMCohenJMI-MOVE multi-centre case control study 2010–11: overall and stratified estimates of influenza vaccine effectiveness in EuropePLoS One20116e2762222110695SkowronskiDMMasaroCKwindtTLEstimating vaccine effectiveness against laboratory-confirmed influenza using a sentinel physician network: results from the 2005–2006 season of dual A and B vaccine mismatch in CanadaVaccine2007252842285117081662FoppaIMHaberMFerdinandsJMShayDKThe case test-negative design for studies of the effectiveness of influenza vaccineVaccine2013313104310923624093JacksonMLNelsonJCThe test-negative design for estimating influenza vaccine effectivenessVaccine2013312165216823499601HaberMAnQFoppaIMShayDKFerdinandsJMOrensteinWAA probability model for evaluating the bias and precision of influenza vaccine effectiveness estimates from case-control studiesEpidemiol Infect20151431417142625147970TreanorJJTalbotHKOhmitSEEffectiveness of seasonal influenza vaccines in the United States during a season with circulation of all three vaccine strainsClin Infect Dis20125595195922843783SuzukiMYoshimineHHaradaYEstimating the influenza vaccine effectiveness against medically attended influenza in clinical settings: a hospital-based case-control study with a rapid diagnostic test in JapanPLoS One20138e5210323326324SuzukiMMinh leNYoshimineHVaccine effectiveness against medically attended laboratory-confirmed influenza in Japan, 2011–2012 seasonPLoS One20149e8881324551167SkowronskiDMDe SerresGCrowcroftNSAssociation between the 2008–09 seasonal influenza vaccine and pandemic H1N1 illness during Spring-Summer 2009: four observational studies from CanadaPLoS Med20107e100025820386731Puig-BarberaJArnedo-PenaAPardo-SerranoFEffectiveness of seasonal 2008–2009, 2009–2010 and pandemic vaccines, to prevent influenza hospitalizations during the autumn 2009 influenza pandemic wave in Castellon, Spain. A test-negative, hospital-based, case-control studyVaccine2010287460746720875486Martinez-BazIMartinez-ArtolaVReinaGEffectiveness of the trivalent influenza vaccine in Navarre, Spain, 2010–2011: a population-based test-negative case-control studyBMC Public Health20131319123496887KwongJCCampitelliMAGubbayJBVaccine effectiveness against laboratory-confirmed influenza hospitalizations among elderly adults during the 2010–2011 seasonClin Infect Dis20135782082723788243HellenbrandWJorgensenPSchweigerBProspective hospital- based case-control study to assess the effectiveness of pandemic influenza A(H1N1)pdm09 vaccination and risk factors for hospitalization in 2009–2010 using matched hospital and test-negative controlsBMC Infect Dis20121212722650369GefenaiteGRahamat-LangendoenJAmbrozaitisASeasonal influenza vaccine effectiveness against influenza in 2012–2013: a hospital-based case-control study in LithuaniaVaccine20143285786324370711ChengACKotsimbosTKellyHAEffectiveness of H1N1/09 monovalent and trivalent influenza vaccines against hospitalization with laboratory-confirmed H1N1/09 influenza in Australia: a test-negative case control studyVaccine2011297320732521810450ChengACHolmesMIrvingLBInfluenza vaccine effectiveness against hospitalisation with confirmed influenza in the 2010–11 seasons: a test-negative observational studyPLoS One20138e6876023874754CastillaJMartinez-ArtolaVSalcedoEVaccine effectiveness in preventing influenza hospitalizations in Navarre, Spain, 2010–2011: cohort and case-control studyVaccine20123019520022100636SkowronskiDMDe SerresGDickinsonJComponent-specific effectiveness of trivalent influenza vaccine as monitored through a sentinel surveillance network in Canada, 2006–2007J Infect Dis200919916817919086914KisslingEValencianoMBuchholzUInfluenza vaccine effectiveness estimates in Europe in a season with three influenza type/subtypes circulating: the I-MOVE multicentre case-control study, influenza season 2012/13Euro Surveill201419pii:20115WacholderSMcLaughlinJKSilvermanDTMandelJSSelection of controls in case-control studies. I. PrinciplesAm J Epidemiol199213510191595688R Core TeamR: A Language and Environment for Statistical Computing2014ViennaR Foundation for Statistical ComputingTalbotHKGriffinMRChenQZhuYWilliamsJVEdwardsKMEffectiveness of seasonal vaccine in preventing confirmed influenza-associated hospitalizations in community dwelling older adultsJ Infect Dis201120350050821220776TalbotHKZhuYChenQWilliamsJVThompsonMGGriffinMREffectiveness of influenza vaccine for preventing laboratory-confirmed influenza hospitalizations in adults, 2011–2012 influenza seasonClin Infect Dis2013561774177723449269SteinerPMCookTDShadishWROn the importance of reliable covariate measurement in selection bias adjustments using propensity scoresJ Educ Behav Stat201136213236FerdinandsJMOlshoLEAganAAEffectiveness of influenza vaccine against life-threatening RT-PCR-confirmed influenza illness in US children, 2010–2012J Infect Dis201421067468324676207JacksonMLRothmanKJEffects of imperfect test sensitivity and specificity on observational studies of influenza vaccine effectivenessVaccine2015331313131625659280Jimenez-GarciaRHernandez-BarreraVRodriguez-RieiroCComparison of self-report influenza vaccination coverage with data from a population based computerized vaccination registry and factors associated with discordanceVaccine2014324386439224968159RolnickSJParkerEDNordinJDSelf-report compared to electronic medical record across eight adult vaccines: do results vary by demographic factors?Vaccine2013313928393523806243BrownCClayton-BoswellHChavesSSValidity of parental report of influenza vaccination in young children seeking medical careVaccine2011299488949222015394Grimaldi-BensoudaLAubrunELeightonPAgreement between patients’ self-report and medical records for vaccination: the PGRx databasePharmacoepidemiol Drug Saf20132227828523319286IrvingSADonahueJGShayDKEllis-CoyleTLBelongiaEAEvaluation of self-reported and registry-based influenza vaccination status in a Wisconsin cohortVaccine2009276546654919729083HalloranMEHaberMLonginiIMJrStruchinerCJDirect and indirect effects in vaccine efficacy and effectivenessAm J Epidemiol19911333233311899778BoltonKJMcCawJMBrownLJacksonDKedzierskaKMcVernonJPrior population immunity reduces the expected impact of CTL-inducing vaccines for pandemic influenza controlPLoS One201510e012013825811654FonvilleJMWilksSHJamesSLAntibody landscapes after influenza virus infection or vaccinationScience2014346996100025414313

Parameter descriptions and symbols with default values. The index 0/1 denotes parameter values for individuals without chronic cardiopulmonary (CP) conditions and CP individuals, respectively.

ParameterSymbolBaseline value
Total population sizeNpop1E + 06
Target no.of cases-500
Target no.of controls-1000
Duration of study (days)τ150
Prevalence of CP statusη0.2
Vaccination uptake (non-CP)ν00.4
Vaccination uptake (CP)ν10.8
Vaccine efficacy against infectionφ0.6
Proportion of influenza hospitalizations prevented by vaccine, given infectionι0
Incidence constant (maximum daily influenza incidence rate per 1000 per day)λ04.0
Incidence rate of influenza infection at time tλI(t)λosint/τπ
Incidence rate of non-influenza infection at time tλNI(t)λocost/τπ + abs(min(costv/eτrπb, a rt; ∈; (;0;,;τ;))
Incidence rate of non-acute respiratory infection (ARI) events (non-CP)λCP0Mean(λNI(t))/10
Incidence rate of non-ARI events (CP)λCP1Mean(λNI(t))
Testing probability of inpatientsσ0.5
Test sensitivityα1.0
Test sensitivity reduction by vaccinationξ0.0
Influenza test specificityε1.0
Vaccine status assessment sensitivityψ1.0
Vaccine status assessment specificityω1.0
CP status assessment sensitivityζ1.0
CP status assessment specificityθ1.0
Probability of hosp.resulting from influenza (non-CP)γI00.01
Probability of hospitalization (hosp) resulting from influenza (CP)γI10.05
Probability of hosp.with non-influenza ARI (non-CP)γNI00.01
Probability of hosp.with non-influenza ARI (CP)γNI10.05
Probability of hosp.with non-ARI events (non-CP)γCP00.02
Probability of hosp.with non-ARI events (CP)γCP10.1

The comparison of vaccine effectiveness (VE) estimates from simulated inpatient test-negative design (TND) studies with the actual vaccine protection from influenza hospitalization (hosp.) in simulated cohort studies (see text) for different values of the proportion of influenza hospitalization that is prevented by vaccination (l)in ‘vaccine failures’. VE against infections is 60% for all scenarios. For each value l, 1 000 simulations were performed

ιaVE against hosp.bCohort VE against hosp. (%)Inpatient TND VE (%)VE difference (% points)
0.06060 (54.4, 65)60.1 (48.5, 69)d−0.1 (−9.1, 10.8)
0.16464 (58.9, 68.6)64.1 (53.8, 72.1)−0.1 (−8.1, 9.7)
0.26868 (63.1, 72.2)   68 (59, 75.1)−0.1 (−7.2, 8.5)
0.37272 (67.7, 75.8)72.2 (64.3, 78.4)−0.2 (−6.4, 7.3)
0.47676 (72.2, 79.5)76.1 (69, 81.5)−0.1 (−5.6, 6.5)
0.58080 (76.6, 83)80.1 (74.3, 84.8)−0.1 (−4.7, 5.4)
0.68484 (81, 86.6)84.1 (79.2, 87.9)−0.1 (−4, 4.4)
0.78888 (85.5, 90.1)   88 (84.1, 91.2)     0 (−3.1, 3.5)
0.89292 (90.1, 93.6)92.1 (89.2, 94.3)−0.1 (−2.2, 2.5)
0.99696 (94.8, 97)   96 (94.2, 97.5)     0 (−1.3, 1.5)

Type equation here

Influenza hosp. prevented by vaccination, given infection.

Calculated as φ* = 1 − (1 − ι) (1 − φ).

Median (2.5th, 97.5th percentile).

Adjusted for chronic cardiopulmonary (CP) status.

Empirical bias distribution of crude vaccine effectiveness (VE) estimates and VE estimates adjusted for chronic cardiopulmonary (CP) status. These estimates were obtained from simulated inpatient test-negative design (TND) studies, using default parameter values in 10 000 simulations

RemarksAnalysisBias (% points)
No heterogeneitywithin CPcategoriesCrude10.66 (3.24, 16.68)a
Restricted to subjectswithout CP  0.02 (−23.45, 15)
Restricted to subjectswith CP−0.06 (−13.27, 10.05)
CP-adjusted−0.08 (−11.09, 8.81)
Heterogeneitywithin CP = 1(see text)Crude13.23 (6.28, 18.89)
Restricted to subjectswithout CP  0.17 (−32.77, 18.48)
Restricted to subjectswith CP       8 (−1.87, 15.93)
CP-adjusted,imperfectly  6.78 (−2.55, 14.32)

Median (2.5th, 97.5th percentile).

Key Messages

Test-negative design (TND) studies of influenza vaccine effectiveness (VE) are increasingly used with inpatients

In the absence of bias, VE estimates from inpatient TND studies represent the protection of the source population from hospitalization with influenza

Late complications of influenza are not captured by this study design

Selection bias due to the inclusion of subjects for exacerbation of chronic pulmonary conditions that are not caused by acute respiratory infection is a potential problem of inpatient TND studies which requires careful consideration.