Clin Infect DisClin. Infect. DiscidcidClinical Infectious Diseases: An Official Publication of the Infectious Diseases Society of America1058-48381537-6591Oxford University Press25414260430436310.1093/cid/ciu749ciu749Articles and CommentariesTriggering Interventions for Influenza: The ALERT AlgorithmReichNicholas G.1CummingsDerek A. T.2LauerStephen A.1ZornMartha1RobinsonChristine3NyquistAnn-Christine35PriceConnie S.45SimberkoffMichael6RadonovichLewis J.7PerlTrish M.8University of Massachusetts, AmherstJohns Hopkins Bloomberg School of Public Health, Baltimore, MarylandChildren's Hospital Colorado, AuroraDivision of Infectious Diseases, Denver Health Medical CenterUniversity of Colorado School of Medicine, Aurora, ColoradoVA New York Harbor Healthcare System, New YorkVeterans Health Administration, Gainesville, FloridaJohns Hopkins University School of Medicine, Baltimore, MarylandCorrespondence: Nicholas G. Reich, BA, PhD, Division of Biostatistics and Epidemiology, University of Massachusetts School of Public Health, 715 North Pleasant St, Amherst, MA 01003 (nick@schoolph.umass.edu).1522015191120141911201460449950416520141592014© The Author 2014. Published by Oxford University Press on behalf of the Infectious Diseases Society of America.2014This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs licence (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial reproduction and distribution of the work, in any medium, provided the original work is not altered or transformed in any way, and that the work is properly cited. For commercial re-use, please contact journals.permissions@oup.com.

Our new method provides a simple, robust, and accurate metric for determining the start of influenza season at the community level.

Background. Early, accurate predictions of the onset of influenza season enable targeted implementation of control efforts. Our objective was to develop a tool to assist public health practitioners, researchers, and clinicians in defining the community-level onset of seasonal influenza epidemics.

Methods. Using recent surveillance data on virologically confirmed infections of influenza, we developed the Above Local Elevated Respiratory Illness Threshold (ALERT) algorithm, a method to identify the period of highest seasonal influenza activity. We used data from 2 large hospitals that serve Baltimore, Maryland and Denver, Colorado, and the surrounding geographic areas. The data used by ALERT are routinely collected surveillance data: weekly case counts of laboratory-confirmed influenza A virus. The main outcome is the percentage of prospective seasonal influenza cases identified by the ALERT algorithm.

Results. When ALERT thresholds designed to capture 90% of all cases were applied prospectively to the 2011–2012 and 2012–2013 influenza seasons in both hospitals, 71%–91% of all reported cases fell within the ALERT period.

Conclusions. The ALERT algorithm provides a simple, robust, and accurate metric for determining the onset of elevated influenza activity at the community level. This new algorithm provides valuable information that can impact infection prevention recommendations, public health practice, and healthcare delivery.

influenzaoutbreak detectionsurveillancehospital epidemiologyinfection control

Influenza, a leading cause of death in the United States [1], causes approximately 40 000 deaths each year, more than motor vehicle accidents [2]. Despite the infections and hospitalizations triggered by influenza annually [35], the ability to predict the timing of these outbreaks remains elusive. Public health practitioners and clinicians lack a reliable, simple method for determining the onset of a period of elevated influenza incidence in a community. Early, accurate predictions that influenza transmission is rising would enable a proactive and fast response to increased transmission and outbreaks.

As one example of how these predictions may be used, many hospitals establish a time period of enhanced precautions for healthcare workers, family members, and visitors during the peak winter influenza season [6]. Karanfil et al implemented a similar strategy to identify an influx of infectious children with respiratory illness and decreased nosocomial transmission by almost 50% [7]. The Centers for Disease Control and Prevention (CDC) and the American Academy of Pediatrics recommend a number of strategies to be used to control influenza during periods of increased risk, including minimizing elective visits of individuals with suspected influenza and setting up triage stations [8, 9]. Identification of periods of increased influenza incidence could also help target screening for influenza antiviral use. However, little guidance is available for facilities to identify periods of increased influenza incidence.

In recent years, researchers have attempted to predict the course of seasonal influenza epidemics [1015]. Some introduce important methodological advances in disease forecasting. Yet, facilities face multiple challenges in implementing these methods to define a period of enhanced precautions. Some methods have shown promise for influenza prediction but require extensive methodological training to implement [15]. Another prominent example, Google Flu Trends, a publicly available and easily accessible tool, can provide timely insight into overall trends of cases in a given region [10, 16]. However, Google Flu Trends has shown poor performance in estimating the burden of influenza [17], correlates better with influenza-like illness than with laboratory-confirmed influenza [18], cannot accommodate data from healthcare settings, and is considered a supplementary rather than authoritative source for influenza surveillance [1921].

There has been a need, therefore, for a translational system that uses surveillance data from a particular setting to inform a simple rule to identify periods of high influenza incidence and guide the initiation of enhanced precautions. By identifying outbreaks early enough to implement public health measures, many potential cases could be prevented. Ideally, such a tool is both sensitive and specific in identifying increasing activity. The costs of activating too late are clear: preventable morbidity and mortality. On the other hand, there are real costs to intervening too early, leading to waste of resources and, potentially, increased community member fatigue.

In the context of a large-scale multisite clinical trial, the Respiratory Protection Effectiveness Clinical Trial (ResPECT), we developed the Above Local Elevated Respiratory Illness Threshold (ALERT) algorithm, a method to trigger the study intervention period so that it overlapped with the period of highest seasonal influenza activity [1, 6, 22]. This simple, easy-to-implement, and data-driven algorithm can help public health practitioners, researchers, and clinicians define the onset of seasonal influenza epidemics in a community, such as a city or a hospital, that systematically collects surveillance data on influenza.

METHODSSettings

The Johns Hopkins Hospital (JHH) is a 900-bed tertiary care center in Baltimore, Maryland, with a 200-bed children's hospital. During the period for which we have data, surveillance for respiratory viruses was a mix of active surveillance (for a limited time in pediatric units) and passive surveillance, as previously described [7, 23]. Children's Hospital of Colorado (CHCO) is a 414-bed hospital serving Denver, Colorado, and surrounding areas. A passive surveillance system is in place at CHCO, where children with respiratory symptoms are tested for common respiratory viruses [24]. In both centers, testing included culture, antigen testing, and polymerase chain reaction.

Data Sources

We obtained weekly case counts of laboratory-confirmed influenza A virus from surveillance systems at both institutions between 2001 through 2013. We excluded data from the summer of 2009, due to anomalous H1N1 disease transmission associated with the pandemic.

The ALERT Algorithm

The ALERT algorithm works by defining an “ALERT period”—a window of time in a given influenza season when elevated incidence is expected. In brief, historical data from the same surveillance system are used to establish a number of cases or threshold (eg, 5 cases). When the observed number of cases in a given week is greater than or equal to the chosen threshold, the ALERT period begins. In a given season, real-time case data determine the start of the ALERT period.

However, choosing the right threshold poses a challenge. To guide the user to an evidence-based decision, the ALERT algorithm summarizes data from previous years as if each of several thresholds had been applied. For each threshold considered, the ALERT algorithm calculates and reports a set of metrics across all years of historical data. A complete list of these metrics is provided in Table 1.

ALERT Algorithm Metrics

The median ALERT period duration.
The percentage of all influenza cases in an entire season contained within the ALERT period (median, minimum, maximum).
The fraction of seasons in which the ALERT period contained the peak week.
The fraction of seasons in which the ALERT period contained the peak week ± k weeks (where k is specified by the user and defaults to 2 weeks).
The mean number of weeks included in the ALERT period with counts less than the threshold.
The mean difference between, for each season, the duration of the ALERT period and the duration of the shortest period needed to capture P percent of cases for that season.

Abbreviation: ALERT, Above Local Elevated Respiratory Illness Threshold.

Given these summary metrics for different thresholds, the user may choose the threshold that fits their needs. This threshold may then be applied to future surveillance data in real time. In the applications shown below, we chose the largest thresholds that had captured at least 90% of cases in half of the previous influenza seasons.

The ALERT algorithm calculations may be implemented by using a Web applet [25], the ALERT package for the R statistical programming language [26, 27], or an Excel spreadsheet provided as a supplement. The spreadsheet provides sufficient calculations to use the ALERT algorithm, but computes a subset of the metrics that the web applet and the R package provide. Detailed instructions on how to use the ALERT algorithm are found in the Supplementary Data.

Data analysis for this project was approved by the institutional review boards at the University of Massachusetts, Johns Hopkins University, University of Colorado, and the CDC.

Validation

To estimate the performance of the ALERT algorithm in prospective use, we performed validation by leaving 1 year out of the training data set, fitting the model, and assessing the fit to the held-out data. We performed this “leave-one-season-out” cross-validation for each year, in turn (Supplementary Data).

RESULTS

Using reported influenza A cases from both hospitals between August 2001 and August 2011 (Figure 1), we prospectively applied the ALERT algorithm to compute historical performance metrics for a range of possible thresholds. Table 2 presents reformatted output from the ALERT R package that shows metrics summarizing the performance for multiple possible thresholds.

Summary of ALERT Performance Across Different Thresholds for Johns Hopkins Hospital and Children's Hospital of Colorado

SiteThresholdMedian DurationCases Captured, %
Peaks Captured
Mean Weeks Below ThresholdMean Duration Difference
MedianMinimumMaximum%% ±2
CHCO11998.972.699.4100902.37.1
215.597.369.998.3100901.23.5
31393.968.997.8100901.21.3
412.591.968.996.6100701.30.6
51289.968.996.1100701.10
611.589.968.996.1100701.3−0.5
79.584.160.796.1100601.4−1.6
88.581.960.796.1100501.6−2.6
98.580.760.796.1100502−2.8
JHH11896.9099.870703.73.7
21897.461.399.490902.95.1
315.595.65898.790901.81.8
414.59457.398.7908020.8
51391.857.396.390802.2−0.3
612.590.357.396.290802.6−0.9
71286.447.794.980802.7−1.4
81182.647.794.980803−2.1
910.582.6094.980602.2−2.44
10976.8094.970602.2−3.11

See Table 1 for a complete description of all the metrics. The “Peaks Captured” columns provide the percentage of seasons in which the ALERT period contained the peak of the flu season (the “%” column) and the peak of the flu season ±2 weeks (the “% ±2” column). The “Mean Duration Difference” column displays the average difference in duration between the ALERT period and the shortest number of consecutive weeks needed to capture 90% of cases across all seasons.

Abbreviations: ALERT, Above Local Elevated Respiratory Illness Threshold; CHCO, Children's Hospital of Colorado; JHH, Johns Hopkins Hospital.

Historical influenza surveillance data from Johns Hopkins Hospital and Children's Hospital Colorado.

For each location, we chose the highest threshold that had captured at least 90% of cases ≥5 times in the past 10 years. For CHCO, this resulted in a threshold of 4 cases per week. For JHH, this led to a threshold of 6 cases.

The ALERT thresholds were applied to the 2011–2012 and 2012–2013 influenza seasons in both hospitals. Figure 2 depicts the timeline of reported cases in the 2012–2013 season. At JHH, 71% and 91% of all reported cases fell in the ALERT period in the 2 seasons, respectively. At CHCO, the ALERT period captured 77% and 89% of all influenza cases during the 2 seasons.

Examples of real-time implementation of the Above Local Elevated Respiratory Illness Threshold (ALERT) algorithm, using the 2012–2013 data from Johns Hopkins Hospital and Children's Hospital Colorado. The shaded region indicates the ALERT period.

To determine the robustness of the ALERT algorithm, we cross-validated our metrics for our decision rule (“highest threshold capturing >90% at least half the time”) and others, by calculating the “prospective” performance by leaving 1 season of data out at a time. The historical performance metrics were similar to the prospective performance metrics (Supplementary Table 2), demonstrating that the simple historical calculations may be sufficient for estimating prospective performance.

DISCUSSION

We have developed a simple strategy to prospectively determine the start and end to a period of elevated influenza incidence in a community. We demonstrated the utility, flexibility, and robustness of the ALERT algorithm, which can serve as a valuable tool for communities, schools, hospitals, and other institutions looking for a method to objectively define a period when, for example, enhanced patient contact precautions, empiric therapy, or other prevention measures should be implemented. Accurate and robust thresholds could aid in preventing morbidity and mortality while also reducing costs associated with excessive precautions.

The ALERT algorithm can be easily operationalized. It is not meant to provide detailed predictions of week-to-week incident counts. In recent years, some models have made large strides predicting influenza [10, 11, 13, 14]. However, we are not aware of any open-source, publicly available tool for defining the influenza season that (1) is currently in use by public health practitioners or (2) can be utilized by someone who does not have substantial graduate-level training in a quantitative, statistical, or computational science. The ALERT algorithm may prove to be especially useful for entities that need to define a programmatic “elevated respiratory illness season” (eg, a health system, hospital, or a clinical trial). While the ALERT algorithm could also be used in nonclinical settings (such as schools or other community milieus), we have not provided a validation of its performance in settings without laboratory-confirmed cases. Furthermore, in these situations it would be especially important to understand the correspondence between unconfirmed case counts and laboratory-confirmed cases.

Our results suggest that the ALERT algorithm performs well at predicting the beginning and end of a seasonal period of increased influenza incidence. However, we expect variation in the results when applying this algorithm. Indeed, we see with the results from JHH that in 1 year, 91% of cases were captured, whereas in the other year 71% of cases were captured. In both years the same threshold was used. The lower-than-expected number resulted from an influenza season that was unlike others in the dataset (only 1 other season in our data had a lower seasonal total). However, in future years of implementation, these data could be utilized as historical data and therefore incorporated into the threshold calculations.

A strength of the ALERT algorithm is that its results are tailored to a specific location. Without involvement of a third party, laboratory-confirmed cases from a facility can be used in the algorithm. In comparison, a system such as Google Flu Trends may also provide useful information for hospital administrators, but ultimately it only provides noisy proxy measures of influenza incidence in a particular setting [1921]. There is nothing preventing the ALERT algorithm from being used with data from a larger community (eg, city- or state-level surveillance data) or with other pathogens that follow seasonal trends (eg, respiratory syncytial virus, norovirus), although we have not evaluated its performance in these settings. It could also provide guidance to municipal public health authorities about when to implement behavioral messaging campaigns to a population.

Picking an ALERT threshold is a task that will depend on the goals of the user. For example, one institution might put a high cost on having ALERT periods that last for >12 weeks. Another might want to capture 95% (or only 75%) of the cases. Each of these specifications might lead to a different way to evaluate the ALERT algorithm output. However, because the ALERT algorithm presents a series of options, it makes it easy for the users to apply their particular priorities. As seen in Table 2, lower thresholds have longer durations and a larger percentages of cases captured. If long durations are not cost-prohibitive, then having the threshold be more sensitive (ie, the threshold is a lower number) may be desired, although this should be weighed against the increased likelihood of an early false alarm.

We have used only influenza A case data in the work presented here because the explicit goal in the ResPECT study (our motivating example) was to maximize capture of influenza A. However, influenza A and B case data could be combined to compute a combined ALERT threshold.

Ultimately, the power of the ALERT algorithm lies in its simplicity, flexibility, and generalizability. However, some technological extensions to the algorithm as it stands now could enhance its utility and empower public health and medical practitioners. As those of us in the public health community, both in and out of healthcare, struggle with planning and managing annual viral epidemics, we must develop tools that will allow us to respond nimbly yet reliably. The ALERT algorithm is an additional tool that does not require sophisticated modeling and is accessible to the public health community. As with all these efforts, additional research and refinement will enhance our ability to be ready for such epidemics.

Supplementary Data

Supplementary materials are available at Clinical Infectious Diseases online (http://cid.oxfordjournals.org). Supplementary materials consist of data provided by the author that are published to benefit the reader. The posted materials are not copyedited. The contents of all supplementary data are the sole responsibility of the authors. Questions or messages regarding errors should be addressed to the author.

Supplementary Data
Notes

Acknowledgments. We acknowledge the efforts of the entire Respiratory Protection Effectiveness Clinical Trial (ResPECT) study team for providing feedback on early drafts of the algorithm. In particular, we wish to thank the site principal investigators from the ResPECT study: Drs Mary Bessessen, Charlotte Gaydos, Cynthia Gibert, and Maria Rodriguez-Barradas. We thank Dr Alexandra Valsamakis and the virology laboratory personnel of the Johns Hopkins Hospital for making data available.

Disclaimer. The findings and conclusions in this manuscript are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs, the Department of Health and Human Services, or the Centers for Disease Control and Prevention (CDC). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Financial support. This work was supported by the US Department of Veterans Affairs and the CDC. Additionally, Derek AT Cummings was supported by grants from the National Institute of General Medical Sciences (MIDAS U54GM088491) and the CDC (U01CK000337).

Potential conflicts of interest. All authors: No potential conflicts of interest.

All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.

ReferencesHoyertDLXuJQDeaths: preliminary data for 20112012Hyattsville, MDNational Vital Statistics Reports, National Center for Health StatisticsUS Census BureauTable 1103: Motor vehicle accidents—number and deaths: 1990 to 20092012693Available at: http://www.census.gov/compendia/statab/2012/tables/12s1103.pdf. Accessed 13 August 2013DushoffJMortality due to influenza in the United States—an annualized regression approach using multiple-cause mortality dataAm J Epidemiol2005163181716319291ThompsonWWShayDKWeintraubEMortality associated with influenza and respiratory syncytial virus in the United StatesJAMA20032891798612517228ThompsonMGShayDKZhouHEstimates of deaths associated with seasonal influenza—United States, 1976–2007MMWR Morb Mortal Wkly Rep20105910576220798667SalgadoCDFarrBMHallKKHaydenFGInfluenza in the acute hospital settingLancet Infect Dis200221455511944184KaranfilLVConlonMLykensKReducing the rate of nosocomially transmitted respiratory syncytial virusAm J Infect Control19992791610196485Centers for Disease Control and PreventionPrevention strategies for seasonal influenza in healthcare settingsAvailable at: http://www.cdc.gov/flu/professionals/infectioncontrol/healthcaresettings.htm Accessed 18 March 2014PickeringLKimberlinDWBakerCJLongSSRed book201229th edElk Grove Village, IL: American Academy of PediatricsGinsbergJMohebbiMHPatelRSBrammerLSmolinskiMSBrilliantLDetecting influenza epidemics using search engine query dataNature20094571012U419020500ShamanJPitzerVViboudCLipsitchMGrenfellBAbsolute humidity and the seasonal onset of influenza in the continental USPLoS Curr20091RRN1138RhodesCJHollingsworthTDVariational data assimilation with epidemic modelsJ Theor Biol200925859160219268475GoldsteinECobeySTakahashiSMillerJCLipsitchMPredicting the epidemic sizes of influenza A/H1N1, A/H3N2, and B: a statistical methodPLoS Med20118e100105121750666ShamanJKarspeckAForecasting seasonal outbreaks of influenzaProc Natl Acad Sci U S A2012109204253023184969ShamanJKarspeckAYangWTameriusJLipsitchMReal-time influenza forecasts during the 2012–2013 seasonNat Comm20134CookSConradCFowlkesALMohebbiMHAssessing Google Flu Trends performance in the United States during the 2009 influenza virus A (H1N1) pandemicPLoS One20116e2361021886802LazerDKennedyRKingGVespignaniAThe parable of Google Flu: traps in big data analysisScience20143431203524626916OrtizJRZhouHShayDKNeuzilKMFowlkesALGossCHMonitoring influenza activity in the United States: a comparison of traditional surveillance systems with Google Flu TrendsPLoS One20116e1868721556151WilsonNMasonKTobiasMPeaceyMHuangQSBakerMInterpreting Google Flu Trends data for pandemic H1N1 influenza: the New Zealand experienceEuro Surveill200914ii19386OlsonDRKontyKJPaladiniMViboudCSimonsenLReassessing Google Flu Trends data for detection of seasonal and pandemic influenza: a comparative epidemiological study at three geographic scalesPLoS Comput Biol20139e100325624146603OlsonDRBaerAColettaMASearching for better flu surveillance? A brief communication arising from Ginsberg et alNature200945710124Nature Precedings. 2009; 713. doi:10.1038/npre.2009.3493.119020500ClinicalTrials.gov.The Respiratory Protection Effectiveness Clinical Trial (ResPECT)Available at: http://clinicaltrials.gov/show/NCT01249625 Accessed 27 June 2013MilstoneAMPerlTMValsamakisAEpidemiology of respiratory viruses in children admitted to an infant/toddler unitAm J Infect Control201240462421907457WestKRobinsonCContagious comments: respiratory season 2013–14Available at: http://www.childrenscolorado.org/File%20Library/Health-Pros/Publications/Contagious%20Comments/Contagious-Comments-November-December-2013-14-Respiratory-Season.pdf Accessed 7 May 2014LauerSAReichNGThe ALERT algorithmAvailable at: http://iddynamics.jhsph.edu/apps/shiny/ALERT/ Accessed 2 April 2014IhakaRGentlemanRR: A language for data analysis and graphicsJ Comput Graph Stat19965299314ReichNGLauerSAALERT—The Above Local Elevated Respiratory Illness Threshold (ALERT) algorithm (v0.1)Available at: https://github.com/nickreich/ALERT. Accessed 12 September 2014