During 2018–2022, the estimated number of human immunodeficiency virus (HIV) infections in the United States decreased each year from 36 200 to 31 800.¹ HIV testing is essential for persons to know their HIV status^2,3; however, 13% of the estimated number of persons who are infected with HIV in the United States are not aware of their infection.¹ Of 1 018 430 cases of persons with HIV who were reported to CDC and were at least 13 years old in 2022, 248 855 (24%) did not receive HIV medical care in that year.⁴

To address HIV in the United States, the Centers for Disease Control and Prevention (CDC) annually funds health departments to conduct comprehensive HIV prevention programs. In 2018, CDC funded 60 health departments (50 states, the District of Columbia, Puerto Rico, the US Virgin Islands, and 7 Metropolitan Statistical Areas) to provide such programs as part of Funding Opportunity Announcement PS18–1802.⁵ Of these 60 health departments, 20^* successfully competed for additional PS18–1802 funds to conduct non-research demonstration projects. Each health department decided for itself what their project would focus on; examples of HIV prevention activities included HIV testing, HIV pre-exposure prophylaxis, partner services, outbreak planning, and social marketing. These projects emphasized collaboration between programmatic and scientific staff and are referred to as program-led program science.⁶

Traditionally, measurement of the impact of science has been academic, only quantitative, limited in scope, and slow.^7–9 Although publication metrics (eg, number of times articles are cited) are commonly used to define impact, this approach does not account for results that are more directly relevant to public health practice and outcomes. As a result, publication metrics likely substantially underestimate the full impact of scientific work. Alternative metrics (altmetrics) are based on online information sources (eg, news and social media, blogs, public policy documents) and increasingly used to complement publication metrics and be more timely. Use of altmetrics and a new tool that assesses impact for health-related policy and guidelines (https://impactanalytics.bmj.com/about/) enhance documentation of impact but are still limited. Because of these limitations, CDC developed the Science Impact Framework to assess more broadly the impact of public health science.^8,10,11 This framework consists of five domains of impact that allow for quantitative and qualitative assessment: Disseminating Science, Creating Awareness, Catalyzing Action, Effecting Change, and Shaping the Future.⁸ These domains are interrelated, do not necessarily occur incrementally or chronologically, and can be assessed retrospectively or prospectively.

Each domain has specific indicators.⁸ Disseminating Science includes publication of findings in peer reviewed journals and other reports, conference presentations, and media channels. Creating Awareness includes awards received, new discussion based on sharing science, acceptance of a concept or findings by policy makers or the scientific community, and increased awareness. Catalyzing Action includes actions taken as a result of scientific work, partnerships, collaborations, and new funding. Effecting Change includes change as a result of scientific work or actions taken; building capacity; and policy, legal, or behavioral change. Shaping the Future includes new hypotheses or strategies, better understanding science or program implementation, and improving the quality of public health.

Recipients of annual CDC funding for comprehensive HIV prevention programs are typically required to report to CDC progress, changes, successes, challenges, lessons learned, and technical assistance needs. For the 20 non-research demonstration projects, health departments for the first time had to submit to CDC a final report that documents local public health impact by reporting how their health department benefited from the project. Our objective was to document the collective public health impact of these demonstration projects.

The 20 health departments implemented and conducted their non-research demonstration projects during July 2018-December 2022. On September 20, 2022, CDC provided written guidance for final reports that required health departments to provide examples of indicators of public health impact that were based on the CDC Science Impact Framework.¹² CDC required health departments during the preparation of their final reports to complete a table that had for each domain examples of indicators to check and an open text field if “Other (Describe briefly)” was checked (Table, Supplementary Digital Content, Appendix, available at http://links.lww.com/JPHMP/B445). Health departments were credited for the “Other (Describe briefly)” indicator, regardless of whether narrative was provided. Indicators that were not checked were considered to have not been done, thus a value of zero and not missing. CDC determined the work to be a public health program activity, so institutional review board approval was not needed.

One author independently abstracted information from the final reports and entered it into an Excel spreadsheet and conferred with another author for questions. This second author also doublechecked all information entered into the spreadsheet to support accuracy of the abstraction and data entry process. Microsoft Excel was used to calculate frequency distributions, means, medians, and ranges. Although health departments documented various types of detailed impact that were specific to their chosen HIV prevention activities, the primary intent of this analysis was to document the collective impact of all 20 projects, so the analysis is based on Table, which had the same general options for all 20 health departments.

The 20 health departments reported 282 examples of public health impact (mean 14.1, median 14.5, range 1–26): 80 for Disseminating Information (mean 4.0, median 4.0, range 0–9), which included 3 “other” responses, 60 for Catalyzing Action (mean 3.0, median 3.0, range 0–5), which included 3 “other” responses, 57 for Shaping the Future (mean 2.9, median 3.0, range 0–6), which included 2 “other” responses, 43 for Creating Awareness (mean 2.2, median 2.5, range 0–5), which included 3 “other” responses, and 42 for Effecting Change (mean 2.1, median 2.0, range 0–5), which included 0 “other” responses (Table). The most common public health impacts were new partnerships (n = 17, Catalyzing Action), capacity building (n = 16, Effecting Change), new project or initiative begun (n = 15, Shaping the Future), new collaborations (n = 14, Catalyzing Action), improved program (n = 13, Shaping the Future), new guidelines or practice (n = 13, Effecting Change), and informed persons affected by work (n = 13, Creating Awareness).

Our results show substantial public health impact documented by health departments that conducted non-research HIV demonstration projects. The most frequently reported domains of impact were Disseminating Information, Catalyzing Action, and Shaping the Future. The most frequently reported indicators were new partnerships and capacity building, which are critical to the success of public health HIV prevention programs. We believe that public heath assessments based on broader definitions of impact such as the CDC Science Impact Framework provide a more comprehensive, timely, and accurate representation of impact than using publication metrics and altmetrics. Additionally, impact results can be shared with others. For example, demonstrating a more complete understanding of the impact of health department activities is important for community-based organizations, funders, partners, and other groups (eg, journalists, general public) with vested interests in accountability and outcomes of public health activities and helps them better understand the value of public health work.

We designed our approach to be simple, timely, and comprehensive in documenting the collective impact of program-science demonstration projects, in part to benefit programmatic and scientific staff who work at the federal and local levels. Advantages of our methodology include its simplicity and timeliness (eg, results are available within months of completed projects compared with more rigorous approaches that might take years). However, it is possible to design a more rigorous impact analysis that is prospective and based on a more timely and standardized data collection process that occurs as a project evolves. For example, the Cancer Prevention and Control Research Network uses an online monitoring and evaluation reporting tool to collect data biannually in progress reports with questions that are based on the CDC Science Impact Framework.¹³ Participating centers report narrative responses that are classified into key indicators by two coders who independently code the information and resolve discrepancies.

Our approach has several limitations, which collectively are likely to have overestimated the impact of the projects. Because more impact may have accrued since health departments submitted their final reports or may still accrue, this limitation may have underestimated the true impact of these projects. However, other limitations are likely to have overestimated impact. Because the methods were based on self-report by CDC-funded health departments, at least some degree of social desirability bias likely occurred, eg, motivation to report favorable results. Other limitations may have further increased the overreporting of impact. For example, detailed instructions were not provided to health departments, so health department staff may have interpreted the domains and indicators of public health impact differently, and some categories were overlapping, thus possibly causing misclassification bias. Additionally, of the 11 reported “other” responses, one such response did not include what the impact was, which could affect the estimated impact; however, this was unlikely to have had a substantial effect, because only 0.3% (1/282) of the total responses were “other” without stating the impact.

Thus, future use and benefit of our approach would be enhanced with more standardized and detailed instructions and methods, and more detailed analyses (eg, verification and descriptions of specific examples of indicators, interval-level categorization of responses, stratification by health department or type of project activity, having a health equity focus). We hope that scientists develop more rigorous methods to assess public health impact for different situations and needs, which could help determine how best to balance timeliness, accuracy, usefulness, and public health relevance. Ideally, options that range from the simple to the complex could best help the likely numerous needs and capacities of programmatic and scientific staff who wish to conduct public health impact analyses. Additionally, the application of implementation science that is relevant to the practice of public health could help increase public health impact and its measurement.¹⁴

This CDC Science Impact Framework provides a comprehensive assessment that accounts for quantitative and qualitative data and is adaptable for different situations at a national, state, or local level. In our case, we used it to determine the public health impact of non-research HIV demonstration projects in support of the national strategic plan to end the HIV epidemic in the United States.¹⁵ This plan encourages staff to focus their efforts and resources to achieve the greatest impact on the HIV epidemic. We hope that programmatic and scientific staff collaborate to increase planning for maximal public health impact and develop additional refinements and new methodologies to further advance the practice, policy, and impact of public health activities.