Evaluating Infectious Disease Forecasts With Allocation Scoring Rules

Details

• Alternative Title:
  Journal of the Royal Statistical Society. Series A, (Statistics in Society), December 2024: Evaluating Infectious Disease Forecasts With Allocation Scoring Rules
• Journal Article:
  Journal of the Royal Statistical Society. Series A, (Statistics in Society)
• Personal Author:
  Gerding, Aaron ; Reich, Nicholas G. ; Rogers, Benjamin ; Ray, Evan L.
• Description:
  Recent years have seen increasing efforts to forecast infectious disease burdens, with a primary goal being to help public health workers make informed policy decisions. However, there has been only limited discussion of how predominant forecast evaluation metrics might indicate the success of policies based in part on those forecasts. We explore one possible tether between forecasts and policy: the allocation of limited medical resources so as to minimize unmet need. We use probabilistic forecasts of disease burden in each of several regions to determine optimal resource allocations, and then we score forecasts according to how much unmet need their associated allocations would have allowed. We illustrate with forecasts of COVID-19 hospitalizations in the U.S., and we find that the forecast skill ranking given by this allocation scoring rule can vary substantially from the ranking given by the weighted interval score. We see this as evidence that the allocation scoring rule detects forecast value that is missed by traditional accuracy measures and that the general strategy of designing scoring rules that are directly linked to policy performance is a promising direction for epidemic forecast evaluation.
• Content Notes:
  Author manuscript
• Subjects:
  • HumansMeSH Scope Note for Humans: Members of the species Homo sapiens.
  • COVID-19
  • Goals
  • Public Health
  • Resource Allocation
  • Communicable Diseases
  • Hospitalization
  • Cost of Illness
  • Policy
• Keywords:
  • epidemiology
  • forecast evaluation
  • proper scoring rules
  • public health
  • resource allocation
• Source:
  Journal of the Royal Statistical Society. Series A, (Statistics in Society), December 2024
• DOI:
  https://doi.org/10.1093/jrsssa/qnae136
• ISSN:
  0964-1998 ; 1467-985X
• Pubmed ID:
  40881183
• Pubmed Central ID:
  PMC12371526
• Document Type:
  Text
• Funding:
  R35 GM119582/GM/NIGMS NIH HHSUnited States/ ; U01 IP001122/IP/NCIRD CDC HHSUnited States/
• Genre:
  Journal Article
• Pages in Document:
  38 pdf pages
• Download URL:
  https://stacks.cdc.gov/view/cdc/261131/cdc_261131_DS1.pdf
• File Type:
  [PDF - 1.72 MB]
• Collection(s):
  CDC Public Access
• Main Document Checksum:
  urn:sha-512:5056659e1472d41f51a881750e097eff505480d6bd85cdfdb36e4146133c1f573e4c33a81c16c93665cec5c84a85ab325e2f30f6979d96016d76f2d855e256e0