Beyond Forecast Leaderboards: Measuring Individual Model Importance Based on Contribution to Ensemble Accuracy.

Details

• Alternative Title:
  International Journal of Forecasting, 2026, v. 42, no. 3: Beyond Forecast Leaderboards: Measuring Individual Model Importance Based on Contribution to Ensemble Accuracy.
• Journal Article:
  International Journal of Forecasting
• Personal Author:
  Kim, Minsu ; Ray, Evan L. ; Reich, Nicholas G.
• Description:
  Ensemble forecasts often outperform forecasts from individual standalone models, and have been used to support decision-making and policy planning in various fields. As collaborative forecasting efforts to create effective ensembles grow, so does interest in understanding individual models' relative importance in the ensemble. To this end, we propose two practical methods that measure the difference between ensemble performance when a given model is or is not included in the ensemble: a leave-one-model-out algorithm and a leave-all-subsets-of-models-out algorithm, which is based on the Shapley value. We explore the relationship between these metrics, forecast accuracy, and the similarity of errors, both analytically and through simulations. We illustrate this measure of the value a component model adds to an ensemble in the presence of other models using US COVID-19 death probabilistic forecasts. This study offers valuable insight into individual models' unique features within an ensemble, which standard accuracy metrics alone cannot reveal.
• Content Notes:
  Author manuscript
• Subjects:
  • COVID-19
  • Algorithms
  • Policy
• Keywords:
  • Probabilistic forecasts
  • Ensemble
  • Model importance
  • Shapley value
  • COVID-19 Forecasting
  • Forecast
• Source:
  International Journal of Forecasting, 2026, v. 42, no. 3
• DOI:
  https://doi.org/10.1016/j.ijforecast.2025.12.006
• ISSN:
  0169-2070
• Pubmed ID:
  42182609
• Pubmed Central ID:
  PMC13197096
• Document Type:
  Text
• Funding:
  R35 GM119582/GM/NIGMS NIH HHSUnited States/ ; U01 IP001122/IP/NCIRD CDC HHSUnited States/
• Genre:
  Journal Article
• Pages in Document:
  924-936 (26 pdf pages)
• Volume:
  42
• Issue:
  3
• Download URL:
  https://stacks.cdc.gov/view/cdc/260668/cdc_260668_DS1.pdf
• File Type:
  [PDF - 1.04 MB]
• Collection(s):
  CDC Public Access
• Main Document Checksum:
  urn:sha-512:8b8527c49de7a5ed6c6adcf7e7252ef3ef47c11b3523516849f21bebd0846c340c178e8f3e045daf6c1b98700003fe2eabf55e7a55397dcf295808f54e432f82