Since the early 2010s, short-term forecasting for infectious diseases has become an increasingly common activity, often through collaborations between governmental and industry or academic partners. Starting during the 2013/2014 influenza season, the FluSight collaborative forecasting exercises organized by the US Centers for Disease Control and Prevention (CDC) have brought together teams of academic, industry and governmental researchers to produce probabilistic forecasts of influenza activity in the United States (Biggerstaff et al., 2016). A primary motivation for the FluSight challenges is to carefully evaluate forecasts against real data and to use them to improve situational awareness and yield more effective public health actions in realms such as the distribution of limited medical resources or the implementation of non-pharmaceutical interventions (Wallinga et al., 2010; Lipsitch et al., 2011; Lutz et al., 2019).

Starting with FluSight seasons just following the COVID pandemic, new high-resolution and low-latency data streams that came online during the pandemic have been used as the ‘‘ground truth’’ target for forecasting exercises. Specifically, from 2022 through 2024, the forecasting targets for FluSight were based on hospital admissions reported in the CDC’s National Healthcare Safety Network (NHSN) surveillance system (Centers for Disease Control and Prevention, 2023b). Because reporting of influenza hospital admissions through NHSN began during the COVID pandemic, the NHSN data have only a limited amount of historical information about influenza hospitalizations. While the NHSN data surpassed previous surveillance systems in terms of providing a specific measure of influenza activity at fine spatial and temporal scales, the lack of an extensive history of data for model training introduced challenges to learning about seasonal patterns in influenza burden in the US. To address this challenge, for the 2023/2024 FluSight season we developed a new model, called Flusion, which pulled in data from external data sources with a longer history of observations. This model was the top-performing model submitted to FluSight in the 2023/2024 season, and it even outperformed the ensemble forecast that the FluSight organizers produced by combining predictions from all contributing models; these ensemble forecasts have typically had some of the most accurate performance overall in past influenza seasons (McGowan et al., 2019; Reich et al., 2019b). The purpose of this paper is to describe the Flusion model and provide insights into what aspects of its design were associated with its strong performance.

Stimulated in large part by outbreak forecasting challenges organized by governmental agencies (e.g., Viboud et al., 2018; McGowan et al., 2019; Johansson et al., 2019), there has been substantial methodological development in the area of infectious disease forecasting over the last 10 years, both for stand-alone and ensemble modeling strategies (Yamana et al., 2016; Reich et al., 2019b; Reis et al., 2019; Ray et al., 2023). Most successful stand-alone approaches can be characterized broadly as using statistical or machine learning methods in conjunction with conceptual models that are informed by infectious disease transmission dynamics, either explicitly through a compartmental transmission model or implicitly through inclusion of seasonality or observations from recent time-points (Reich et al., 2019a; Lopez et al., 2024). There has simultaneously been a large increase in worldwide participation in online data science competition sites like Kaggle, including many competitions with time series forecasting and even a series of five weeks of real time forecasting of COVID-19 in 2020 (Addison Howard, 2020; Hamner, 2020; Walter Reade, 2020a,b,c). The top performers in these competitions make heavy use of advanced feature engineering in conjunction with gradient boosted tree methods (Friedman, 2001), using the popular packages XGBoost, Light-GBM, and CatBoost. For example, the majority of the top-performing competitors in the M5 forecast accuracy competition used gradient boosting (Makridakis et al., 2022).

The Flusion model fits into the machine learning model category, although it uses features that emphasize both seasonality as well as recent observations and recent trends as a way of encoding underlying transmission dynamics of influenza. In terms of underlying methodology, our primary focus in this work is on forecasting using gradient boosting. In particular, our work closely follows the approach of Lainder and Wolfinger (2022), which was the leading approach in several time series forecasting competitions including strong performance for forecasting COVID-19 (Lopez et al., 2024). Our methods adapt that work to forecasting seasonal influenza by including features intended to help the model identify seasonal trends of disease activity and by incorporating multiple data sources measuring influenza activity.

A key area where the Flusion model provides a methodological innovation is in how it leverages long historical time-series data that are closely related to the target data source that has little history. There is existing work that focuses on situations where auxiliary data signals related to a primary signal of interest are used to improve short-term predictions. For example, in settings where a primary signal has delayed or unreliable recent reporting, research has shown that other signals that are available contemporaneously or with a shorter lag can help inform estimation and prediction of trends (Yang et al., 2015; Farrow, 2016; Osthus et al., 2019; Leuba et al., 2020; McDonald et al., 2021; Jahja et al., 2022). Some of these modeling efforts, among others, also highlight how sharing contemporaneous information across different locations can be helpful (Farrow, 2016; McDonald et al., 2021; Osthus and Moran, 2021).

In contrast, the Flusion model, in addition to borrowing information across locations, augments a primary data signal with a very short history with other closely related signals that measure the same general system but for a longer history. This approach bears a close resemblance to multi-task learning and transfer learning, where models are trained on related tasks with a large number of available observations either simultaneously with training on more limited data for the primary task, or in a first stage that is followed by fine-tuning on data for the target task (Pan and Yang, 2009). In a forecasting context, these methods have been used in domains such as energy use and climate control (Ribeiro et al., 2018; Grubinger et al., 2017). However, to our knowledge the literature contains only a few applications of this general approach to disease outbreak forecasting. We are aware of two studies that investigated forecasting a pathogen of interest using data from other pathogens to inform model estimation: Roster et al. (2022) considered using historical data on dengue to inform Zika predictions as well as the use of past influenza data to inform COVID-19 forecasts, while Coelho et al. (2020) used models trained on data measuring dengue incidence to generate predictions for Chikungunya and Zika. In another line of work, Zou et al. used multitask learning to forecast influenza-like illness by training models jointly on data for multiple geographic locations, and demonstrated that this was helpful in settings where some locations had only limited data but others had a longer history of observations (Zou et al., 2018). Unlike this existing literature, our work augments a signal measuring seasonal influenza in the US that has a limited history with other signals that measure the same disease system over a longer time span.

We note that contemporaneously with our development of Flusion, two other groups also explored methods for forecasting influenza hospitalizations as reported in NHSN using other data streams to extend the amount of available training data (Meyer et al., 2024; Benefield et al., 2024). Their approaches differed from ours in that they used these other data streams to impute additional historical observations of hospitalizations, rather than training a model directly on data from multiple surveillance signals. Although those methods did not outperform Flusion or the FluSight ensemble, they were ranked 2nd and 13th among the 29 individual models included in our evaluation of FluSight forecasts for the 2023/24 season (see Section 6.2). The success of the Flusion model and these other methods that incorporated multiple data sources in the 2023/2024 FluSight season suggests that this approach may be a promising direction, especially in the context of public health data modernization initiatives that may replace long-standing surveillance data systems with new data streams.

The remainder of this article is structured as follows. We begin with some context for our work, including a discussion of the data sources we use in Section 2 and an overview of the FluSight forecasting exercise in 3. We establish notation in Section 4 and describe our modeling approaches in Section 5, and then we discuss the performance of our real-time forecasts relative to other contributors to FluSight in the 2023/24 season in Section 6. Section 7 details a series of ablation studies that provide insight about which aspects of our modeling approach were important to its success. We conclude in Section 8.

Our model used three measures of influenza activity (Fig. 1). The first of these was weekly influenza hospital admissions reported to the National Healthcare Safety Network (NHSN, (Centers for Disease Control and Prevention, 2023b)), which was the target signal for FluSight (see Section 3 for more detail). NHSN reports counts of hospital admissions where the patient tested positive for influenza at the time of admission or within the previous two weeks. Requirements for facilities reporting to NHSN have changed over time, but for the time period included in our analyses reporting was generally mandatory for all hospitals registered with the Centers for Medicare and Medicaid Services. We worked with reported data that had been aggregated to the state level for each of the 50 US states, the District of Columbia, Puerto Rico and the national level. Reporting of influenza hospitalizations through NHSN began in 2020, and the first two years of reporting for this signal showed very low influenza activity during the COVID-19 pandemic. As a result, at the start of the 2023/24 season this signal had only one season’s worth of data showing patterns typical of seasonal influenza.

To address this limitation of the target surveillance signal, our model used two other measures of weekly influenza activity that have a longer reporting history. The first of these was a measure of hospital admissions where the patient has a positive influenza test as reported by FluSurv-NET, expressed as a rate per 100,000 population in the catchment areas of selected hospital facilities (Centers for Disease Control and Prevention, 2023a). Our models used data from 14 sites in 13 states (including two sites in New York, which we aggregated to the state level), as well the ‘‘Entire Network’’, which we use as an estimate of influenza activity at the national level. The other signal was ILI+, which is an approximate measure of the proportion of outpatient doctor visits where the patient has influenza that is derived by combining data from ILINet, the World Health Organization (WHO), and the National Respiratory Enteric Virus Surveillance System (NREVSS) (Goldstein et al., 2011; Centers for Disease Control and Prevention, 2023b). Our models use ILI+ data at the state level and national level, as well as at the regional level for the ten Health and Human Services regions, which are groups of between 4 and 10 states and/or territories of the US. As will be described further in Section 5, in this work our goal in using these data was to provide models with additional historical data to use for learning about trends in seasonal influenza activity. Because contemporaneous use of these signals was not our primary goal, and in order to avoid complications with integrating multiple data sources with different reporting lags and backfill behaviors, in this work we only used historical data prior to the 2023/24 season for these alternative data signals.

For both of these additional signals, we included adjustments designed to correct for known challenges with interpreting the data collected by FluSurv-NET and ILINet as consistent measures of influenza activity over time. The FluSurv-NET data report on patients with a positive influenza test, and as such are subject to varying levels of underreporting depending on changing testing rates and test sensitivity from season to season. The CDC produces annual estimates of hospital burden due to influenza that adjust for these factors (Centers for Disease Control and Prevention, 2024b). We used these total burden estimates to estimate season-specific inflation factors that scale up reported rates from FluSurv-NET, with the intent of producing a more consistent measure of influenza activity over time; see supplemental section 2 for more details. These inflation factors were generally larger in earlier seasons than in later seasons, indicating that FluSurv-NET undercounted influenza activity more in early seasons than it did in later seasons.

ILINet reports a measure of influenza-like illness (ILI), as the percent of outpatient doctor visits where the patient has a fever with a temperature of at least 100 °F (37.8 °C) and a cough and/or a sore throat (Centers for Disease Control and Prevention, 2023b). Because ILI is defined symptomatically, this signal generally includes some patients who have respiratory diseases other than influenza such as RSV and COVID. To address this, we computed ILI+ as the product of this ILI signal and influenza test positivity rates from laboratory testing sites reporting to WHO and NREVSS systems. After converting to a proportion scale, ILI+ can be interpreted as an estimate of the proportion of outpatient doctor visits where the patient has influenza, and has been used in previous forecasting work as a more specific measure of influenza activity than ILI (e.g. Goldstein et al., 2011; Shaman et al., 2013).

Our models were not trained on the 2008/09 and 2009/10 seasons, which were impacted by pandemic H1N1 influenza, and the 2020/21 and 2021/22 seasons, which were impacted by low influenza activity during the COVID pandemic. Additionally, some of our models were not trained on the influenza off-season (see Section 5.2 for more detail on our models). Designating US Epidemic week 31 as season week 1 (generally falling in early August, a time of consistently low influenza activity), we define the on-season as season weeks 10 through 40 and the off-season to be the remaining weeks, which fall during the summer months (Centers for Disease Control and Prevention, 2024a). Omission of the off-season from the training set forms a natural implementation of purging/embargoing, a method that is popular in econometric time series forecasting in which the time series is split into blocks for cross-validation with a buffer between the time blocks to prevent leakage of information across the cross-validation folds (De Prado, 2018; Lainder and Wolfinger, 2022).

For the 2023/24 season, the primary target for forecasts collected in the FluSight forecasting exercise organized by CDC was weekly hospital admissions with confirmed influenza as reported in the NHSN data set for each of the 50 US states, the District of Columbia, Puerto Rico, and in total at the national level (Fig. 1, top panel). Predictions were submitted to FluSight on Wednesday each week. The week of submission is anchored relative to the Saturday after the submission date, which corresponds to the final day of that US epidemic week and is denoted as the reference date for the predictions. Predictions were made for hospital admissions from Sunday to Saturday in the current week and each of the three following weeks, corresponding to forecast horizons of 0, 1, 2, and 3 weeks ahead relative to the week of submission. A new data release from NHSN was made public at approximately noon on Wednesday each week including reported admissions up through the previous Saturday, and predictions were based on models fit to that data release. This data release was delayed by one day on the week ending on April 13, 2024, and for that week the forecast submission due date was extended so that the latest data could be used.

For the hospital admissions target, probabilistic predictions were represented with a set of predictive quantiles at K=23 quantile levels αk, corresponding to a predictive median and the endpoints of 11 central prediction intervals at the nominal 10%, ..., 90%, 95%, and 98% levels. For example, the predictive median corresponds to α12=0.5 and the 98% interval corresponds to predictive quantiles at the levels α1=0.01 and α23=0.99.

We use zs,l,t to denote the observed value of surveillance signal s (with s=1 for NHSN data, s=2 for FluSurv data, and s=3 for ILI+ data) in location l in week t. We denote the predictive quantile at level αk for the value of the signal s in location l, generated on reference date d at forecast horizon h by qs,l,d,h,k. Note that the observation zs,l,t at time t is the prediction target across the four combinations of reference date d and forecast horizon h with d+h=t. For brevity, we will often use the index i to refer to a forecast task consisting of a combination of values of s=s(i), l=l(i), d=d(i), and h=h(i), with qi,k denoting the prediction at quantile level αk for that task and zi denoting the corresponding observed value.

In this manuscript, we evaluate forecasts using three metrics: the mean absolute error (MAE) of the predictive median, the mean weighted interval score (MWIS), and the coverage rates of central 50% and 95% prediction intervals. In some figures, we will also examine one-sided quantile coverage rates. Our evaluations of forecast skill cover only predictions of NHSN admissions s=1. All of these metrics have been used by FluSight for forecast evaluation. The absolute error and weighted interval score are proper scoring rules for the predictive median and the full collection of predictive quantiles collected by FluSight, respectively (Gneiting, 2011; Bracher et al., 2021). This means that in expectation under a forecaster’s predictive distribution, the scores are optimized by submitting the predictive median or the collection of elicited predictive quantiles of that distribution.

MAE measures the average distance between the predictive median and the eventual observation, with smaller values indicating better forecast accuracy. Recalling that the predictive median corresponds to the quantile level α12=0.5 (i.e., k=12), the absolute error of the prediction for task i is qi,12−zi. In our evaluations, the mean absolute error averages the absolute error across predictions for different locations, dates, and horizons.

WIS can be viewed as a generalization of the absolute error to a set of quantile predictions, and is equivalent to an average of quantile scores (sometimes referred to as pinball losses) computed for each quantile prediction (Bracher et al., 2021): WISqi,k:k=1,…,K,zi=1K∑k2⋅QSαkqi,k,ziQSαkqi,k,zi=αkmaxzi−qi,k,0+1−αkmaxqi,k−zi,0 The quantile score for a single quantile prediction assigns an asymmetric penalty to the distance between the prediction and the observation. The magnitudes of the penalties for underprediction and overprediction are set so that the expected value of the quantile score with respect to the forecast distribution is minimized by the quantile of the predictive distribution at the specified quantile level (Gneiting, 2011). The quantile score for the quantile level 0.5 is equal to one half of the absolute error of the median. Again, lower values of WIS indicate a better alignment of forecasts with the observed data. MWIS is the average WIS across multiple locations, dates, and horizons.

We also compute relative versions of MAE and MWIS, denoted rMAE and rMWIS, using the ‘‘pairwise tournament’’ approach outlined in Cramer et al. (2022). In a comparison of forecast accuracy among M models, computation of rMAE (rMWIS) has two steps. First, we obtain a summary of the average performance for model m relative to each other model m′, denoted θm. This summary is computed as the geometric mean of the ratio of the MAE (MWIS) for model m to the MAE (MWIS) for each other model m′, where for each model pair the MAE averages across the set Im,m′ of locations, dates, and forecast horizons for which both models submitted predictions. The rMAE (or rMWIS) then normalizes this geometric mean relative to the value of θm for a baseline model: rMAEm=θmθbaseline,whereθm=∏m′≠mMAEIm,m′mMAEIm,m′m′1/(M−1) Here, MAEIm denotes the MAE for model m across all predictions for tasks i in the index set I.

The primary purpose of this procedure is to correct for the varying level of difficulty of the predictions submitted by different forecasters in settings where some forecasters did not provide predictions for all locations or time points. In this situation, comparing MAE or MWIS could be misleading because a forecaster could achieve good MAE or MWIS by omitting forecasts for difficult locations or time points, such as near a peak in disease incidence. The relative metrics account for this by evaluating each pair of forecasters on the prediction tasks they shared in common, enabling a direct comparison of their scores. However, ultimately the rMAE and rMWIS metrics may also be susceptible to gaming by a forecaster who selectively omits forecasts that may have poor performance. Of note, considerations about missingness are relevant to the evaluation of submissions to FluSight in Section 6, but in the experimental results in Section 7 all forecasts were provided. A secondary goal of the relative scores is to standardize scores relative to a baseline model with known behavior, in our case the flat baseline described in Section 6. Smaller values of rMAE or rMWIS indicate better performance relative to the other models in the comparison pool, and in particular values less than 1 indicate performance that is better than the baseline.

Finally, we examine probabilistic calibration through the empirical coverage rates of 50% and 95% prediction intervals and individual quantiles. Let l, u∈{1,…,K} denote the indices for the lower and upper quantile levels defining the central prediction interval at a specified nominal level. For example, with our notation 95% intervals correspond to l=2 and u=22, with αl=0.025 and αu=0.975. The empirical prediction interval coverage rate across prediction tasks in the index set I is defined as 1|I|∑i∈I𝟙qi,l≤zi≤qi,u, where |I| is the size of the index set I and 𝟙qi,l≤zi≤qi,u is 1 if qi,l≤zi≤qi,u and 0 otherwise.

One-sided coverage rates at individual quantile levels can provide more detail than interval coverage rates. In some figures below, we show the difference between the empirical one-sided coverage rate at each quantile level and the corresponding nominal coverage rate: δk=1|I|∑i∈I𝟙zi≤qi,k−αk. A well-calibrated forecaster will have δk≈0, while a conservative forecaster with wide prediction intervals will have δk>0 for αk>0.5 and δk<0 for αk<0.5.

Flusion was constructed as an ensemble of statistical and machine learning time series models. All models were fitted to data that were preprocessed to standardize across different signals, locations, and time points, and we describe these preprocessing steps in Section 5.1. We describe the component models in Sections 5.2 and 5.3 and the ensemble methods in Section 5.4. The precise formulation of the models included in the ensemble and the ensembling methods varied slightly over the course of the season, and we describe these aspects of our setup for generating real-time predictions in Section 5.5.

In this section, we summarize model performance results for real-time submissions to FluSight in the 2023/24 season.

In this section, we investigate the degree to which the following aspects of Flusion contributed to its strong performance: 1. the formulation of Flusion as an ensemble of three individual models; 2. joint training on multiple data sets and multiple locations; 3. data preprocessing, including corrections for reporting inconsistencies in the ILINet and FluSurv-NET data, the use of a fourth root data transform, and the importance of the features that were used by the GBQR model.

The Flusion model documented in this manuscript was the top-ranked model in the FluSight forecasting exercise for the 2023/24 season as measured by MAE and MWIS, and its probabilistic calibration was comparable to or better than that of other participating models. The experimental results presented here indicate that this strong performance was primarily driven by the use of a gradient boosting model that was trained jointly on data from multiple surveillance signals and locations. In contrast, other modeling decisions we made had a more minor impact on forecast accuracy. For example, forming predictions as an ensemble of the GBQR model and a more classical ARX model yielded only a small gain in performance, and our attempts to compensate for irregularities in reporting for FluSurv-NET and ILI data were counterproductive.

Not all modelers that contributed to FluSight provided a description of their methods, but we reviewed the available model meta-data (Centers for Disease Control and Prevention) and identified at least 3 other models that in some way used data for multiple surveillance signals and trained jointly on data for multiple locations, along with an additional 13 that used multiple surveillance signals and another 8 that were trained jointly on data for multiple locations. As with previous forecast evaluation efforts that have not found consistent associations between individual model design decisions and forecast skill (e.g., Mathis et al., 2024), these methods had a range of performance rankings. This indicates the challenge of identifying important modeling decisions by comparing scores from models that differ in many aspects of their model formulations, and highlights a strength of our study: the use of ablation studies to investigate the value-added from specific aspects of our model setup while controlling for other parts of the model design. That said, the second-best individual FluSight model overall (denoted as ‘‘Other Model #1’’ in Table 2) also used gradient boosting and multiple surveillance signals (Meyer et al., 2024), which lends additional support to our finding that these design choices were helpful.

A limitation of the results presented in this manuscript is that they report on performance only for a single season in the United States. Ongoing evaluation of the methods we have outlined will be necessary to ensure that the strong performance we documented here generalizes across multiple influenza seasons. Additionally, it would be valuable to understand how our methods would perform in the face of an influenza pandemic. The strong performance of a similar model using gradient boosting to forecast COVID offers some reassurance on this front (Lopez et al., 2024), but our reliance on a long history of training data representing seasonal influenza activity may impact model performance in other settings.

There are numerous avenues for improving on our methods, and we are pursuing some of these in future work. Our intuition is that it would be valuable to use contemporaneous observations of multiple signals, such as NHSN admissions, FluSurv-NET, and ILI together, to inform predictions of trends in disease incidence. Care will need to be taken with this since the relative magnitudes of these signals can vary across geographies and over time. Along these lines, it may also be possible to use other signals, such as insurance claims or internet activity, to improve forecast accuracy (e.g., Osthus et al., 2019; McDonald et al., 2021). A challenge with any approach integrating multiple data streams in real time is that they may have different reporting lags and different revision behaviors.

Although our models were trained jointly on data from multiple spatial units, the forecasts they produced were not directly informed by the spatial structure. We anticipate that improvements in accuracy could be achieved by including features that measure trends in flu activity in neighboring locations (Thivierge et al., 2024). Alternatively, it may be possible to improve forecast accuracy by reconciling predictions made at multiple hierarchical levels, for example generating probabilistic forecasts at the state, regional, and national levels and subsequently adjusting the predictions so that the forecasts at those scales are consistent with each other (Athanasopoulos et al., 2024). These methods would likely be particularly useful for locations with small populations, which typically have a relatively low signal-to-noise ratio in reported surveillance data.

Forecasting the turning point of an epidemic wave is a challenging problem (Castro et al., 2020), and as we saw in Section 6.2 both Flusion and the FluSight ensemble often failed to accurately predict the turnaround after a local peak in disease activity or the increase leading into a second peak within the same season. We conjecture that it may be possible to improve forecasting performance in these instances through better handling of holiday effects or by extending our model to take into account epidemiological understanding of disease transmission. Improved handling of holiday effects, e.g. through features designed to measure trends accounting for the typical magnitude of holiday effects, could help models to predict a decline in activity immediately after the holidays and might allow the model to better see a second wave coming.

Features derived from mechanistic insights, such as measures of vaccine uptake and efficacy or the circulation of multiple strains of the influenza virus at different times over the course of the season may also be helpful (Kandula et al., 2017). For instance, a second wave of influenza later in the season is often at least partially driven by a different influenza type or subtype than was circulating early in the season. A model that saw data on disease incidence broken down by flu strain might be better able to predict a second wave in these situations. Information about vaccine uptake and efficacy might help a model predict season severity more accurately. A challenge with these approaches is that timely data may not be available for use in a real-time forecasting exercise.

We expect that the main insights presented in this work regarding the value of using data from multiple surveillance signals and locations could be useful in multiple modeling frameworks and for forecasting infectious diseases other than seasonal influenza. These results are of particular importance to the infectious disease forecasting community since they indicate a path forward in settings where new public health surveillance systems may come online and shut down in a span of a few years. In the absence of a long history of data for a signal of interest, borrowing information from similar data sets can provide important context for models to learn about patterns of disease transmission.

Appendix A. Supplementary data

Supplementary material related to this article can be found online at https://doi.org/10.1016/j.epidem.2024.100810.