Background

pmc

7801727

3636

Eval Program Plann

Evaluation and program planning

0149-71891873-7870

40435551

12439252

10.1016/j.evalprogplan.2025.102610

NIHMS2085686

Article

Advancing Methods for Program Evaluation in Substance Use by Strengthening the Application of Epidemiology, Economic Evaluation, and Implementation Science: Reflections on the Opportunities for Synergy

Blackburn

Natalie A.

12*Philbrick

Sarah

1Patel

Sheila V.

1Cance

Jessica D.

1Elek

Elvira

3Gibbons

Brent

3Khan

Sazid

1Lambdin

Barrot H.

4Graham

Phillip W.

1RTI International, Research Triangle Park, NC, United States2NOVA National School of Public Health, Comprehensive Health Research Centre, NOVA University Lisbon, Lisbon, Portugal3RTI International, 701 13th Street NW, Suite 750, Washington, DC, United States4RTI International, 2150 Shattuck Avenue, 8th Floor, Berkeley, CA, United States

Author Statement

Natalie A. Blackburn: conceptualization, methodology, writing – original draft. Sarah Philbrick: methodology, writing – original draft. Sheila V. Patel: conceptualization, methodology, writing – reviewing and editing. Jessica D. Cance: conceptualization, methodology, writing – reviewing and editing. Elvira Elek: methodology, writing – reviewing and editing. Brent Gibbons: writing – reviewing and editing. Sazid Khan: writing – reviewing and editing. Barrot Lambdin: conceptualization, funding acquisition, writing – reviewing and editing. Phillip Graham: conceptualization, funding acquisition, writing – reviewing and editing.

Authors’ Contributions

NAB facilitated the development, drafting, editing, and corrections of the manuscript. SP contributed with a critical review; helped draft the sections; and contributed with additions, literature citations, and edits in different subsections of the manuscript. SVP contributed with a critical review, literature citations, and edits in different subsections of the manuscript. JDC contributed with edits in different subsections of the manuscript. EE contributed with edits in different subsections of the manuscript. BG contributed with literature citations and edits in different subsections of the manuscript. SK contributed with literature citations and edits in different subsections of the manuscript. BHL provided early support for the main conceptual ideas and contributed with input for different subsections of the manuscript. PG conceived the main conceptual ideas and contributed with edits in different subsections of the manuscript. All authors contributed to the article and approved the submitted version.

*Correspondence: Natalie A. Blackburn, Campo Mártires da Pátria 130, Lisbon, 1169-056, Portugal, natblackburn@gmail.com

3052025

82025

1952025

0182026

111102610102610

Program evaluation is necessary to assess the success of evidence-based programs in real-world, non-trial settings. In its most basic form, it involves understanding processes that impact the health and well-being of individuals and communities. Increasingly, robust program evaluations are integrating approaches from implementation science and economic evaluation to strengthen assessment, achievement, and sustainment of the intended impacts of evidence-based programs, while traditionally employing epidemiological methods to assess outcomes. Epidemiology, economic evaluation, and implementation science operate in overlapping yet siloed arenas, which can make it challenging for program evaluators to effectively engage all three disciplines. Furthermore, given their overlap, practitioners in these disciplines must understand differences in their purpose and lexicon to foster synergies. We use the Centers for Disease Control and Prevention Evaluation Framework and an example from substance use prevention to reflect on specific concepts and terminology within each of these disciplines and how distinctions impact the ability of multidisciplinary teams to collaborate on evaluations. The implications apply to those designing, testing, implementing, and evaluating evidence-based programs—all of whom have the opportunity to consider and strengthen these elements when developing processes and measuring impact. The substance use prevention field needs to better understand how these disciplines can complement one another to enrich process, outcome, and impact evaluations, given the complexities of substance use behaviors and evidence-based programs to address them. With this synergy, clearer communication of findings from the different components can be achieved.

economic evaluationepidemiologyimplementation scienceprogram evaluationsubstance use prevention

1Background

Once programs are determined to be evidence based through rigorous research, they need to be disseminated and implemented in practice thoughtfully to ensure anticipated outcomes are achieved in the “real world.” Program evaluation is an approach that allows for assessing programs in the real world, enabling the monitoring of delivery to identify when a program fails to achieve anticipated outcomes, and what is not working for whom and how, to prompt changes to facilitate outcome achievement (1). Although some may understand the purpose and language of individual disciplines (e.g., epidemiology, economic evaluation, implementation science) for program evaluation, a gap remains in understanding how these individual disciplines can complement one another in the service of program evaluation. Part of this gap stems from discipline-specific lexicons that make communicating commonalities challenging.

In its most basic form, program evaluation provides a systematic method for collecting, analyzing, and communicating answers to questions, particularly about effectiveness, efficiencies in delivery, and resource use (2). Rooted in a rich history of education programs and social sciences, program evaluation is interdisciplinary in its nature and approach. To varying degrees across different areas, program evaluations traditionally incorporate the strengths of multiple disciplines including psychology, anthropology, and sociology in order to assess the worth and impact of a program. Public health program evaluation was further strengthened with the development of the CDC Framework for Program Evaluation in Public Health (1). Many evaluation approaches exist to guide program evaluations; indeed, like application of the CDC Evaluation Framework, these approaches are used to document the strengths and weaknesses of programs and, when appropriate, make recommendations for program improvements.

The substance use prevention landscape has placed increasing importance on the value (3) and sustainability of programs while still remaining steadfast about program impact. Applied epidemiology, as a key discipline informing program evaluation, is pivotal for assessing impact and has provided a set of approaches for defining outcomes of interest and measuring effectiveness as part of the development, testing, and evaluation of evidence-based programs to prevent substance misuse. A program evaluator may use epidemiological measures, such as incidence and prevalence of prescription opioid misuse among high school seniors, to assess the impact of a substance misuse prevention education program. Alongside impact, the focus on value and sustainability requires stronger incorporation of economic evaluation and implementation science, along with greater clarity in the application of these fields within program evaluation. For example, implementation research defines and tests strategies to overcome challenges and leverage strengths to successfully integrate, deliver, scale, and sustain a program. Economic evaluations provide information on the comparative value of programs, information needed by policymakers and potential adopters when making decisions about which programs are the best use of limited budget resources. Additionally, economic evaluations can help program adopters and deliverers to understand what costs to expect when starting up and delivering a program. Therefore, planning for and collecting cost data often plays a critical role in program evaluations. Although collecting implementation-related and cost data is not new in program evaluation, collection often gets integrated late in the evaluation process. Integrating more refined implementation and economic evaluation methods at the outset of substance use prevention program evaluations will produce more precise information to better inform decision makers and potential adopters.

Implementation scientists and health economists have each sought to clarify and define key terminology in their own fields (4–6). As research integrating these two fields has grown, with some even conducting economic evaluation of implementation strategies, efforts to ensure common language across disciplines have fallen short (7–9). In addition to a discordant lexicon, timelines in data collection and when the fields are relevant in the evaluation process differ such that opportunities to communicate and engage with one another when conducting program evaluations are missed. Program evaluation, with its interdisciplinary nature, requires communicating across different discipline lexicons. The lack of opportunities to engage with one another makes efforts to define common language even more challenging. Thus, there is a need to engage these disciplines early in the evaluation process, crosswalk key concepts, and harmonize language across these disciplines. Doing so avoids duplication of efforts and supports real-world delivery and evaluation of evidence-based programs.

2Aims and Objectives

This paper seeks to bring together different methodological perspectives in designing and conducting program evaluation in substance use prevention in three specific disciplines: epidemiology, economic evaluation, and implementation science. These disciplines can strengthen the design and delivery of program evaluations and inform policy and practice. We focus on three key areas: language within each discipline, application of each discipline in evaluation, and the opportunities for synergy across disciplines. Specifically, we aim to provide tangible approaches to integrating these disciplines within existing program evaluation frameworks. We first describe each discipline’s lexicon and how similar terms are applied, then use the Centers for Disease Control and Prevention (CDC) Evaluation Framework (10–12) as an example for incorporating disciplines into program evaluation, showcasing opportunities for application, and strengthening existing evaluation approaches.

3Discussion3.1Discipline Summaries

Summaries of the disciplines, including definitions, key elements, and example publication reporting standards, are presented in Table 1 and described as follows.

3.1.1Epidemiology

Briefly, epidemiology developed as a discipline to consider disease across populations rather than individuals (13). The definition of epidemiology has evolved and continues to differ among epidemiologists (14). Furthermore, current definitions vary across institutions; CDC defines epidemiology as “the study of the distribution and determinants of health-related states or events in specified populations, and the application of this study to the control of health problems.” Program evaluators can find that epidemiological practices, methods, and techniques can be of use in many areas. For example, epidemiological data provide the foundation for understanding the health status, needs, and risk factors of a population of focus, which is crucial for the framework of an evaluation, such as for the initial needs assessment and potential re-visitation of the assessment in a long-term evaluation (15). Furthermore, epidemiological methods to quantitatively measure health outcomes enable evaluators to assess the impact of the program on the health status of the population of focus, including changes in incidence, prevalence, and mortality. Increasingly, as epidemiological and data science methodologies intersect to tackle population-level questions, epidemiological data provide a reference for model parameterization (16). This role of epidemiological data has become important with the utilization of machine learning and statistical models (1) to assess “big data” and (2) for compartmental modeling (e.g., using administrative data to inform decision making about the results of a program evaluation and subsequent policy decisions) (17). Epidemiological principles shape the language of program evaluation by guiding the selection of proximal, intermediate, and distal outcomes and by focusing more broadly on the impact of a program.

3.1.2Economic Evaluation

Economic evaluation developed to inform decision making in the context of limited resources (18). Economic evaluation has varied definitions and forms but is generally characterized as “the comparative analysis of alternative courses of action of both their costs and consequences” (19). From CDC, economic evaluation is used “to identify, measure, value, and compare the costs and consequences of different public health [programs]” (20). Incorporating economic evaluation into program evaluation provides a systematic approach to assessing the costs associated with program implementation, including direct and indirect costs, which are vital for understanding the program’s financial feasibility. Economic evaluation techniques like cost-effectiveness analysis, cost-utility analysis, and cost-benefit analysis enable evaluators to determine the program’s value in terms of health outcomes relative to costs, facilitating evidence-based resource allocation decisions. As we think more critically about which programs to adopt, economic evaluation can also help model the long-term sustainability and efficiency of programs, ensuring that the benefits justify the investment and that resources are used as efficiently as possible. Economic evaluation shapes the language of program evaluation in part by focusing on costs and asking—in a world with limited funding for programs aiming to prevent substance misuse—how we can make decisions on the best programs to fund and what budget we need to implement them at scale.

3.1.3Implementation Science

Implementation science evolved from a demonstrated need to accelerate the slow pace with which evidence-based programs are put into practice (21). Implementation science provides insights into the processes by which programs are adopted, implemented, and sustained in real-world settings, which is essential for evaluating the program’s adoption and reach, the fidelity with which it is implemented, and the extent to which it is sustained. Through a focus on barriers and facilitators to implementation, implementation science helps identify factors that influence a program’s success (i.e., implementation determinants) and further develops and tests implementation strategies to enhance uptake and real-world effectiveness. Given the broader focus in program evaluation on sustainability, implementation science more recently has encouraged greater focus on the scalability of successful programs, ensuring that they work in pilot or controlled settings and can be effectively expanded and sustained in broader contexts.

The frameworks that are pivotal to implementation science guide the way the discipline organizes and reflects on program delivery (22). Commonly used theories and frameworks in implementation science include diffusion of innovations, the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) Evaluation Framework (23), and the Consolidated Framework for Implementation Research (24). In addition, the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework allows for better understanding of how implementation determinants (and thus strategies and outcomes) may vary across phases of implementation (i.e., exploration, preparation, implementation, sustainment) (25). Implementation science shapes the language of program evaluation through application of concepts such as tailoring to context, strategizing to overcome anticipated barriers, and scaling services with fidelity (26). The work of implementation science is also often mixed methods, using both qualitative and quantitative methods, with data triangulation to understand these concepts in a deeper and more comprehensive way (27).

3.2Language Considerations

Pivotal to the application of these disciplines is the lexicon within which they operate and communicate. Those designing and conducting program evaluations also use their own lexicon. Table 2 aligns key terms used by each discipline to key terms in program evaluation.

Table 2 highlights how program evaluation already incorporates some important concepts from the highlighted disciplines and that, in many areas, the language is aligned across disciplines. For example, “fidelity” is a term used commonly in process evaluations and is a well-described implementation science outcome. However, some words and phrases used in the different disciplines have disparate meanings; thus, there is an opportunity to describe and provide greater clarity. For example, “strategy” can be a confusing term, given multiple meanings across disciplines. Although program evaluators may view strategies and interventions as interchangeable terms (i.e., evidence-based interventions vs. strategies), “strategy” has a very specific meaning within implementation science. An implementation strategy, defined by Proctor et al., is the “methods or techniques used to enhance the adoption, implementation, and sustainability of a clinical program or practice” (32). For implementation scientists, uncovering effective implementation strategies leads to strengthened delivery of a program and the potential to share those strategies as best practices with other settings considering adoption.

There are also examples of similar concepts described using different words across the disciplines. For example, the parts of an intervention, meaning the different pieces or parts that make up the intervention itself, can sometimes be referred to as “components.” In economic evaluation, costs are often measured by the “activities” involved in starting (e.g., training) or delivering (e.g., clinical supervision) the intervention. As another example, economic evaluation may use the term “start-up” costs to describe the costs related to starting a new program or putting a program “in place” before delivery of services. Implementation scientists may call these same costs preparation and implementation costs. Similarly, “implementation” in economic evaluation generally refers to what implementation science would call “intervention delivery.”

3.3Opportunities for Synergy

Finding synergy across disciplines provides opportunities to strengthen the design and application of process, outcome, and impact evaluations. Figure 1 presents a visualization of opportunities for incorporating each discipline using the CDC Evaluation Framework (33) as an example. This framework outlines three cross-cutting actions (engage collaboratively, learn from and use insights, and advance equity) and six steps to design and tailor an evaluation of a particular program: (1) assess context, (2) describe the program, (3) focus the evaluation questions and design, (4) gather credible evidence, (5) generate and support conclusions, and (6) act on findings. The following sections describe these steps further, with the cross-cutting principles incorporated, and demonstrates how each discipline can be leveraged. To illuminate each step in a more applied way, we use the example of evaluating a substance use screening and brief intervention (SBI) program delivered in a school-based setting. Examples from the steps are also summarized in Table 3.

3.3.1Assess Context

The first phase of the CDC Evaluation Framework is assess context. This phase is key to program evaluation, as it ensures the subsequent phases are useful, relevant, and impactful. This is when a needs assessment might occur. Here also is when identification of relevant interested parties who will facilitate program adaptation, delivery, and evaluation is key and this may come from carrying out the needs assessment which includes collecting preliminary or previous research or engaging with key informants. In the language of implementation science, a needs assessment is an example of an exploration phase strategy that can describe the problem in need of addressing. Those conducting and describing the needs assessments to inform program selection can include, for example, members of the evaluation team, academic partners, and community representatives.

In this phase, academic partnerships with epidemiologists can help to better understand the scope of the problem using historical data for context about the disease and estimates for the potential distribution of the disease. For example, epidemiologists may provide prevalence data on adolescent risky behaviors by county, indicating potential areas of high substance use rates that require intervening. Epidemiologists can also use these data to assist in the prioritization of programs that have the most potential to address the problem at hand. Health economists can further help understand the economic burden of the problem on different sectors, investigate what causal research says about potential solutions where programs may be focused, and assess the cost literature and economic evaluation of relevant programs. Engaging epidemiologists and economists in this way can be helpful in setting the foundation for the work and aid in data-informed decision making.

Implementation science can strengthen identification and engagement of other interested or involved parties. A structured approach or model to engage interested parties or develop partnerships—like the Center for Implementation’s Relational Pathway—can help to ensure an effective program evaluation (34). In the Relational Pathway, engagement starts with building buy-in and establishing formal and informal partnerships. For example, in the context of a school-based SBI, engagement could include holding meetings with school leaders, staff, parents, and students to discuss the need for preventing substance misuse; selecting and adapting an appropriate program; creating partnership agreements with school leadership to determine roles and responsibilities or co-creating a governance structure that could include advisory boards or work groups; establishing means of communicating within the team; and conducting informational sessions with community partners who can provide services outside the school setting to students requiring higher-intensity intervention (35). These partnerships may lead to more rich evaluation data conducted via qualitative interviews or surveys.

3.3.2Describe the Program

An important step before gathering evidence about a program is to describe it thoroughly. This goes beyond describing the components of the program to include the expected strategies needed to implement and maintain it. This also includes reviewing descriptives on the expected effects. Here, a logic model is helpful to guide the goals and objectives of the evaluation. Well known to program evaluators as a tool for structuring a program evaluation, logic models include the definition of critical components and where evaluation may be most pivotal for determining processes and health impact (36). The structure of logic models varies, but they are often made up of two sides: process and outcomes. The process side can include inputs, activities, and outputs; the outcomes include those that are short term, intermediate, and long term (Figure 2).

3.3.2.1Process

“Inputs” are defined as “resources that go into a program or intervention—what we invest” (37). In describing the inputs of the logic model, economists in this phase can use previous research to help determine what resources may be necessary and to align with current standards of describing costs of SBI delivery. During this “describe the program” phase, epidemiology provides the foundation for the evaluation in identifying past trends and historical information, as well as any recent information such as population risk factors and the current health status of the population of focus (37). For example, the inputs for SBI would be consideration of the age-specific substance use risk factors and identifying resources required for a school-based SBI, including resources needed to conduct such a program in the school, such as funding, staffing to deliver SBI, and external providers to refer to.

“Activities” are “events undertaken by the program or partners to produce desired outcomes—what we do” (37). Logic model activities can vary in terms of effort, timing, and importance. In addition to activities directly related to delivering SBI, implementation science can help to define and describe strategies to integrate SBI into the school. For example, an implementation strategy of training school-based health center staff to identify and intervene on risky behaviors may be listed as an activity in the logic model (38). Should economists choose to take an activity-based costing approach in the evaluation, activities in the logic model inform which activities are selected to cost.

“Outputs” are “the direct, tangible results of activities—what we get” (37). Economists and implementation scientists can work with program evaluators to better understand and define the outputs of the SBI program by defining what the intermediate requirements are in order to achieve the primary health-related outcomes desired.

3.3.2.2Outcomes

Outcomes are defined as “the desired result of the program—what we achieve” (37). Outcomes are generally distinguished by order of operation (e.g., proximal and distal) or length of time (e.g., short term, intermediate, and long term). Short-term and intermediate outcomes can be an important piece of evidence to share with interested parties, including funders and community partners, to engage collaboratively and maintain support for a program in addition to ensuring that a program is on a good track towards long-term outcomes while linking together all three.

“Short-term outcomes” are the “immediate effects of the program or intervention activities” and “often focus on the knowledge and attitudes of the intended audience” (e.g., increase in staff knowledge of substance use among youth), whereas “intermediate outcomes” are “behavior, normative, and policy changes” (e.g., increase in number of students screened) (37). Here is where it is important to identify implementation outcomes in describing the program. The Proctor et al. Implementation Outcomes Framework (IOF) offers short-term outcomes that are often qualitative and perception based and more aligned with exploration, like appropriateness, acceptability, and feasibility, and intermediate outcomes that are more quantitative and behavior based and aligned with implementation, such as adoption, fidelity, and reach (39, 40).

“Long term outcomes” “refer to the desired results of the program and can take years to accomplish (37)” (e.g., reduced risk behavior among youth, reduced time to treatment referral for students with substance use disorder). These are often the primary focus of funders who are seeking observable health impacts. From epidemiology, long-term outcomes are the expected effects of the program that draw from identified risk factors and causes of disease that a program aims to address. A well-defined outcome of interest includes consideration of whether it is important to measure prevalence versus incidence and define the population of focus and the unit of analysis. Epidemiologists may provide guidance on measuring impacts (e.g., the main impact of the SBI program may be a reduced rate of substance misuse among youth). This description would be the clear specification of the outcome of interest, along with identification of the “problem or condition.” From the IOF, sustainability of the program (implementation outcome) and equity (service outcome) may be long-term outcomes of interest (39).

Economists, in discussion with the evaluation team, consider the best health outcomes to use as part of the economic evaluation. For example, if evaluating an SBI delivered in a school, it may be important to consider a primary health outcome of the number of days of drug use youth self-report in the past month (41) or perhaps a measure of health utility. The types of economic analyses planned, such as cost-effectiveness or budget impact analysis, will require using the inputs, activities, and outcomes of the logic model; these analyses occur in the next phase of focusing the evaluation design.

3.3.3Focus the Evaluation Questions and Design

The focus of the evaluation questions and design builds on the phase in which the program is described. It is important to define the evaluation’s purpose, including who will use the results and what methods will be used to carry out the evaluation. This includes developing relevant evaluation questions based on the missions and objectives of the evaluation and linking them with the methods used to provide responses to those questions. The RE-AIM framework can be used to structure and guide this process. Since its formal development and publication more than 25 years ago (42), program evaluators have used RE-AIM to categorize and describe the impact of their interventions. For example, an evaluation may be interested in answering what the “reach” of a program was (e.g., the number of students who receive a substance use screening at a school-based health center) and how “effective” it was (e.g., the extent to which substance misuse was reduced among students who were identified as at-risk). In this phase, implementation science methods can be used to describe an implementation plan, such as identifying via interviews or case audit who will be delivering the SBI intervention in the school setting and what are the needed implementation strategies.

Epidemiological methodology can also help inform the development of effective questions that can provide specificity and clarity to a program evaluation. Design considerations may include whether to have a comparison group (e.g., another school not implementing SBI) or a one-group design. This also ties back to having a well-defined outcome and the importance of determining the unit of analysis along with determining the outcome data sources (i.e., secondary sources when available or primary data collection).

Integrated implementation and economic evaluation plans can also occur. Economists, with input from other team members, can develop an economic evaluation plan within the larger evaluation design, including decisions about approach (e.g., taking a health care sector vs. societal analytic perspective) and plans for collecting and analyzing data as well as whether to cost the implementation strategies (“start-up costs”) in addition to intervention delivery. Some of these decisions will depend on the funding available for an economic evaluation component. For example, are resources available for a more comprehensive micro-costing approach like activity-based costing, and if so, what level of data collection burden are program deliverers willing to undertake to support such an approach? This phase includes development of data collection instruments and discussions with programmatic team members to ensure buy-in and to determine where different team members may need to collect data. For example, collection of direct and indirect cost data for an SBI in a school-based setting may be done by school administrative staff rather than the program evaluation team. Consequently, collecting these economic costs requires understanding and awareness on the part of those administrative staff responsible for providing that information (43). Furthermore, the methods and data for valuing resources should be specified, along with the methods and data for analyses that will be performed.

3.3.4Gather Credible Evidence

In this phase of the evaluation, evidence about the program is collected and analyzed. This extends beyond the typical focus on gathering evidence on outcomes and identifying intervention impacts; it is key to gather implementation and intervention delivery evidence using qualitative and quantitative methods. For example, collect start-up and intervention cost data during this phase. Based on the focus of the evaluation, health utility data may also be collected in this phase. The economic evaluation team needs to provide support, and in some cases, conduct qualitative interviews, to ensure data collected are accurate and consistent. Secondary data or reviews of relevant literature may be conducted during this phase as necessary for analyses. For example, if a societal perspective is used, costs from other sectors, such as the youth legal system or other social service settings, may be needed. If the analysis uses a modeling approach, the model and associated input parameters can be built in this phase.

There is also a great opportunity for synergy between implementation science and economic evaluation when activity-based costing is undertaken. Implementation science methods can facilitate the detailing and description of each intervention component as part of this phase of describing the program, and in turn, this information is helpful for providing context on the activities measured for cost purposes. Furthermore, tracking program adaptations can help to understand why certain outcomes were or were not achieved. The Framework for Reporting Adaptations and Modifications-Enhanced (FRAME) is one tool from implementation science that can help guide this effort (44). It encourages thinking around when in the process the modification was made, whether a modification was planned/proactive or unplanned/reactive, who determined the modification was needed, what was modified and at what level (e.g., content of the substance use screener was modified at the individual student level vs. for all students at the school level), the nature of the modification (e.g., added screening elements), whether the modification is fidelity-consistent, and the reasons for the modification. Tools like this can help organize important information collected qualitatively about program delivery in dynamic real-world settings.

3.3.5Generate and Support Conclusions

After gathering credible evidence, interpretation of the evidence is required. Results should provide answers to the evaluation questions along with the level of statistical confidence. If collected, supporting qualitative data (e.g., details of and perspectives on program activities and impacts) can provide a richer understanding of results and, for example, the acceptability of the program among practitioners in addition to those receiving it or unplanned adaptations during program delivery that change core components. This can support interpretation of outcomes achieved in light of implementation outcomes. The findings based on decisions made in the earlier phases of describing the program and focusing the evaluation will be pivotal for the justification of conclusions. Strong application of epidemiologic and economic methods is needed to appropriately analyze the data, following the specified methodological plan. Conclusions should be based on the results, including results from any sensitivity analyses. These conclusions will inform any resulting program recommendations.

3.3.6Act on Findings

Ultimately, program evaluation requires a focus on effectively disseminating results and lessons learned. Throughout the phases of the program evaluation process, engagement with the primary users (i.e., the recipients of the program and the evaluation results) is needed to ensure its utility. Sharing evaluation results and lessons involves considering the format in which information will be communicated and how it will be tailored for different audiences. Here, there is an opportunity to have cross-disciplinary dissemination that pairs epidemiological, economic evaluation, and implementation results. Epidemiologic methods can identify causes of disease, along with associations and potential solutions that can help inform policy and public health decisions. Within implementation science, methods of effective dissemination can inform this phase; these methods include the cross-cutting action of “engage collaboratively” with an analysis of those parties, their roles, and the preferred communication methods. For example, the results of the evaluation of the school-based SBI program may require engagement with policymakers around financing strategies and opportunities for becoming part of a payment schedule to be covered by health insurance (45).

4Conclusions

Program evaluation is a key element to ensuring that implementation of evidence-based programs is effectively delivered and achieves intended impacts in the real world. Leveraging epidemiology, economic evaluation, and implementation science establishes a stronger multidimensional approach to address the “what works,” “how,” “for whom,” and “at what cost” aspects of program evaluation.

Epidemiology clarifies what programs need to address and whether a program works (is effective) by providing data on health problems, risks, and outcomes to guide the focus of the program evaluation. Economic evaluation assesses at what cost the program achieves its outcomes, offering a critical perspective on the program’s value, which is essential for comprehensive outcome evaluation. Implementation science delves into how practitioners implement the program, identifying strategies to enhance adoption, fidelity, reach, and sustainability, thereby informing the process evaluation component. Economic evaluation and implementation science are critical for dissemination by providing start-up and intervention information beyond effectiveness, including costs, that can inform when a program is scaled up.

The relationship between these concepts and program evaluation can be synergistic, with each contributing unique insights that collectively provide a thorough approach to assessing and improving efforts to address priority problems such as substance misuse. In this article, we aim to demonstrate ways in which program evaluators can engage these disciplines; communication and collaboration across disciplines can help identify ways to optimize limited resources, strengthen methods, and improve synthesis. For example, evaluation teams can strengthen the integration of implementation science and economic evaluation by agreeing on the start-up and other implementation activities versus program activities (7).

Funding for program evaluation is often limited, further challenging the inclusion of these different disciplines. Without additional funding for program evaluation, consideration is needed for how to include these disciplines without overwhelming project budgets. From an epidemiologic perspective, secondary data can also be a useful solution when the population is over-surveyed or experiencing survey fatigue. For example, the economic evaluation will require engaging an economist on the team, but some approaches to costing, such as using secondary data, can be less resource intensive. In other cases where an economist cannot be supported, efforts to collect cost data are still valuable; these data would allow for follow-up economic evaluation studies. For implementation-related questions, program evaluators can use existing literature and resources on basic implementation science concepts that are most helpful in designing the process components of an evaluation. Specifically, skills in how to identify and measure implementation outcomes (46) can provide a great deal of support in the early steps of program evaluation.

This article was supported by the Helping to End Addiction Long-term (HEAL) Initiative^® launched by the National Institutes of Health in 2018. Through this initiative, the National Institute on Drug Abuse administered the HEAL Preventing Opioid Use Disorder (OUD) research program to focus specifically on identifying effective upstream prevention interventions. Within this program, the HEAL Prevention Cooperative is composed of 10 research projects testing interventions to prevent opioid misuse or escalation to OUD among youth and young adults in real-world settings, along with the HEAL Prevention Coordinating Center (HPCC). The HPCC learned many lessons regarding the need to integrate and align epidemiological outcomes, implementation measures, and cost data collection regardless of delivery setting. Understanding the processes for evaluating a program can lead to better program descriptions, result in faster development of an evidence base for a program, and accelerate the pace with which programs become part of routine practice (47). An integrated approach ensures that program evaluations are not only comprehensive in scope but also practical in informing future practice to improve health outcomes at a population level. Integrating the principles of epidemiology, economic evaluation, and implementation science into program evaluation not only enriches the assessment process but also establishes a comprehensive framework that addresses the multifaceted and complex nature of programs to prevent substance misuse.

Acknowledgments

We would like to thank those who participated in a workshop on this topic at the Society for Prevention Research 32nd Annual Meeting.

Funding

This work is supported by the National Institutes of Health (NIH) through the NIH Helping to End Addiction Long-term^® Initiative (HEAL Initiative^®) as part of the HEAL Preventing Opioid Use Disorder Research Program U24DA050182. The content is solely the responsibility of the authors and does not necessarily represent the official views of NIH.

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethics Approval and Consent to Participate

Given the nature of the article, neither ethical approval from the authors’ institutions nor informed consent from participants was necessary.

Availability of Data and Materials

Data sharing is not applicable to this article, as no datasets were generated or analyzed during the current study.

References1.

Centers for Disease Control and Prevention. CDC Approach to Program Evaluation 2024 [updated August 18. Available from: https://www.cdc.gov/evaluation/php/about/index.html.

Spiegelman

. Evaluating public health interventions: 1. Examples, definitions, and a personal note. Am J Public Health. 2016;106(1):70–3.26562122

Smith

, Ory

. Measuring success: evaluation article types for the public health education and promotion section of frontiers in public health. Front Public Health. 2014;2:111.25165688

Powell

, Waltz

, Chinman

, Damschroder

, Smith

, Matthieu

, A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10(1):21.25889199

Drummond

, Sculpher

, Torrance

, O’Brien

, Stoddart

. Methods for the Economic Evaluation of Health Care Programmes: Oxford University Press; 2005 31 Oct 2023.

Sanders

, Neumann

, Basu

, Brock

, Feeny

, Krahn

, Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: Second panel on cost-effectiveness in health and medicine. JAMA. 2016;316(10):1093–103.27623463

Malhotra

, Thompson

, Kagoya

, Masiye

, Mbewe

, Mosepele

, Economic evaluation of implementation science outcomes in low- and middle-income countries: a scoping review. Implement Sci. 2022;17(1):76.36384807

Roberts

SLE

, Healey

, Sevdalis

. Use of health economic evaluation in the implementation and improvement science fields-a systematic literature review. Implement Sci. 2019;14(1):72.31307489

Krebs

, Nosyk

. Cost-effectiveness analysis in implementation science: A research agenda and call for wider application. Curr HIV/AIDS Rep. 2021;18(3):176–85.33743138

10.

Milstein

, Wetterhall

. Framework for program evaluation in public health. . MMWR Recomm Rep. 1999;48(RR-11):1–40.

11.

Kidder

, Fierro

, Luna

, Salvaggio

, McWhorter

, Bowen

S-A

, CDC program evaluation framework, 2024. MMWR Recomm Rep. 2024;73(6):1–37.

12.

Centers for Disease Control and Prevention. CDC program evaluation framework 2024 [updated August 20. Available from: https://www.cdc.gov/evaluation/php/evaluation-framework/index.html.

13.

Foundations Of Epidemiology. Lilienfeld

, Stolley

, Lilienfeld

, editors: Oxford University Press; 1994 31 Oct 2023.

14.

Frerot

, Lefebvre

, Aho

, Callier

, Astruc

, Aho Glele

. What is epidemiology? Changing definitions of epidemiology 1978–2017. PLoS One. 2018;13(12):e0208442.30532230

15.

Antoniadis

, Lubker

. Epidemiology as an essential tool for establishing prevention programs and evaluating their impact and outcome. J Commun Disord. 1997;30(4):269–83; quiz 83–4.9208363

16.

Fairchild

, Tasseff

, Khalsa

, Generous

, Daughton

, Velappan

, Epidemiological data challenges: Planning for a more robust future through data standards. Front Public Health. 2018;6:336.30533407

17.

Zanti

, Berkowitz

, Katz

, Nelson

, Burnett

, Culhane

, Leveraging integrated data for program evaluation: Recommendations from the field. Eval Program Plann. 2022;95:102093.36027757

18.

Goodacre

, McCabe

. An introduction to economic evaluation. Emerg Med J. 2002;19(3):198–201.11971826

19.

Drummond

, Sculpher

, Claxton

, Stoddart

, Torrance

. Methods for the Economic Evaluation of Health Care Programmes. 4th ed: Oxford University Press; 2015.

20.

Centers for Disease Control and Prevention. Economic evaluation overview Atlanta, GA: Centers for Disease Control and Prevention; 2021 [updated March 3. Available from: https://www.cdc.gov/policy/polaris/economics/index.html.

21.

Morris

, Wooding

, Grant

. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med. 2011;104(12):510–20.22179294

22.

Nilsen

. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53.25895742

23.

Glasgow

, Harden

, Gaglio

, Rabin

, Smith

, Porter

, RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review. Front Public Health. 2019;7:64.30984733

24.

Damschroder

, Reardon

, Widerquist

MAO

, Lowery

. The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci. 2022;17(1):75.36309746

25.

Aarons

, Hurlburt

, Horwitz

. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38(1):4–23.21197565

26.

Rabin

, Brownson

, Haire-Joshu

, Kreuter

, Weaver

. A glossary for dissemination and implementation research in health. J Public Health Manag Pract. 2008;14(2):117–23.18287916

27.

Palinkas

, Cooper

. Mixed methods evaluation in dissemination and implementation science. In: Brownson

, Colditz

, Proctor

, editors. Dissemination and implementation research in health: Translating science to practice. 2nd ed. New York, NY: Oxford University Press; 2017. p. 335–54.

28.

Eccles

, Mittman

. Welcome to Implementation Science. Implement Sci. 2006;1(1):1.

29.

von Elm

, Altman

, Egger

, Pocock

, Gotzsche

, Vandenbroucke

, The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. Int J Surg. 2014;12(12):1495–9.25046131

30.

Husereau

, Drummond

, Augustovski

, de Bekker-Grob

, Briggs

, Carswell

, Consolidated Health Economic Evaluation Reporting Standards 2022 (CHEERS 2022) statement: updated reporting guidance for health economic evaluations. Value Health. 2022;25(1):3–9.35031096

31.

Hoffmann

, Glasziou

, Boutron

, Milne

, Perera

, Moher

, Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687.24609605

32.

Proctor

, Powell

, McMillen

. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8(1):139.24289295

33.

Centers for Disease Control and Prevention. A framework for program evaluation Atlanta, GA: Centers for Disease Control and Prevention; 2017 [updated May 15. Available from: https://www.cdc.gov/evaluation/framework/index.htm.

34.

Moore

, Khan

. Toronto, Ontario: The Center for Implementation. 2022. [cited 2022]. Available from: https://www.linkedin.com/pulse/considering-role-relationships-implement…; .

35.

Maslowsky

, Whelan Capell

, Moberg

, Brown

. Universal school-based implementation of Screening Brief Intervention and Referral to Treatment to reduce and prevent alcohol, marijuana, tobacco, and other drug use: process and feasibility. Subst Abuse. 2017;11:1178221817746668.

36.

Frechtling

. Logic modeling methods in program evaluation. Hoboken, NJ, US: Jossey-Bass/Wiley; 2007. xiii, 146–xiii, p.

37.

State Heart Disease and Stroke Prevention Program. Evaluation guide: developing and using a logic model. Atlanta, GA: CDC Division for Heart Disease and Stroke Prevention.

38.

Ramos

, Sebastian

, Murphy

, Oreskovich

, Condon

. Adolescent substance use: assessing the knowledge, attitudes, and practices of a school-based health center workforce. Substance Abuse. 2017;38(2):230–6.28328312

39.

Proctor

, Silmere

, Raghavan

, Hovmand

, Aarons

, Bunger

, Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76.20957426

40.

Reilly

, Kennedy

, Porter

, Estabrooks

. Comparing, Contrasting, and Integrating Dissemination and Implementation Outcomes Included in the RE-AIM and Implementation Outcomes Frameworks. Front Public Health. 2020;8:430.32984239

41.

Mitchell

, Gryczynski

, Gonzales

, Moseley

, Peterson

, O’Grady

, Screening, Brief Intervention, and Referral to Treatment (SBIRT) for substance use in a school-based program: services and outcomes. Am J Addict. 2012;21 Suppl 1(0 1):S5–13.23786511

42.

Glasgow

, Vogt

, Boles

. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7.10474547

43.

Bray

, Mallonee

, Dowd

, Aldridge

, Cowell

, Vendetti

, editors. Program- and service-level costs of seven screening, brief intervention, and referral to treatment programs2014.

44.

Wiltsey Stirman

, Baumann

, Miller

. The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implement Sci. 2019;14(1).

45.

Reif

, Brolin

, Beyene

, D’Agostino

, Stewart

, Horgan

. Payment and financing for substance use screening and brief intervention for adolescents and adults in health, school, and community settings. J Adolesc Health. 2022;71(4s):S73–s82.36122974

46.

Proctor

, Bunger

, Lengnick-Hall

, Gerke

, Martin

, Phillips

, Ten years of implementation outcomes research: a scoping review. Implement Sci. 2023;18(1):31.37491242

47.

Department of Health and Human Services. The Helping to End Addiction Long-term Initiative: supporting science-based solutions to the opioid crisis 2023 [Available from: https://heal.nih.gov/.

Figure 1.

Incorporating epidemiology, implementation science, and economic evaluation into the steps of program evaluation

Figure 2.

CDC layout of a general logic model

Sources: CDC Division for Heart Disease and Stroke Prevention. n.d. Evaluation Guide: Developing and Using a Logic Model. https://www.cdc.gov/cardiovascular-resources/media/pdfs/logic_model.pdf; CDC Office of Policy, Performance, and Evaluation. 2018. Program Evaluation Framework Checklist for Step 2. https://www.cdc.gov/evaluation/steps/step2/index.htm

Table 1.

Summary of disciplines

	Program Evaluation in Public Health	Epidemiology	Economic Evaluation	Implementation Science
Definitions	“The systematic investigation of the merit, worth, or significance of an object. ” (1)	“Epidemiology is the study of the distribution and determinants of health-related states or events in specified populations, and the application of this study to the control of health problems” (14).	“The comparative analysis of alternative courses of action of both their costs and consequences” (5).	“The scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice, and, hence, to improve the quality and effectiveness of health services” (28).
Origins	Psychology, Education, Curricula Assessments	Medicine, Germ Theory, John Snow, Mathematics	Health economics, Political Science, Economic theory	Rogers Diffusion of Innovation, Kingdon’s Policy Streams, Knowledge Translation
Key elements or features within program evaluation	-	Measurement of morbidity and mortality of a population; measurement of the determinants of morbidity and mortality.	Analysis of cost and outcomes.Comparative assessment of program value (i.e., requires a comparator).	Measurement of adoption, reach, and sustainment of programs.Development and assessment of strategies for implementing programs.
Example publication	American Evaluation Association	Strengthening Reporting in Observational Studies	Consolidated Health Economic Evaluation	Template for Intervention Description and Replication
reporting standards	Guiding Principles for Evaluators	in Epidemiology (STROBE) (29)	Reporting Standards (CHEERS) (30)	(TIDieR), Standards for Reporting Implementation Studies (StaRI) (31)

Table 2.

Select program evaluation terms—crosswalk of overlapping and diverging language applied across disciplines

Program Evaluation Term	Epidemiology	Economic Evaluation	Implementation Science
Program Strategy Intervention Evidence-based or -informed intervention, strategy, or practice Program activities	Strategies Exposure	Intervention Program	InnovationEvidence-based Intervention or Practice7 P’s: pills, programs, practices, principles, products, policies, proceduresCore vs. peripheral components
-	-	Start-up	Pre-implementation strategiesImplementation strategiesSustainment strategies
Setting	Setting, descriptives	Setting	Implementation contextInner/internal setting (characteristics of the delivery setting)Outer/external setting (characteristics of the broader environment)
Program adaptation, modification	-	-	AdaptationTailoringPlanned/unplanned modification
Costs Resources		Start-up costsAdaptation costsIntervention costsOpportunity costMarginal costIndirect and direct costsFixed and variable costs	Implementation costStaffing needsInfrastructure needsCost impact
Implementation	-	Start-up periodImplementation period	Preparation phaseAdoptionImplementation phaseInitial implementationUptakeSpread/scale
Fidelity Quality of program delivery	Adherence	Fidelity	FidelityQuality
Reach	Distribution	Reach	ReachPenetration
Sustainability	-	Affordability	Sustainment phaseMaintenanceScale-up
Outcome	Proximal outcomes (e.g., opioid use behavior),Distal outcomes (e.g., opioid overdose mortality)	Primary outcomeHealth utility (e.g., quality-adjusted life year [QALY])Effectiveness	Implementation outcomes (e.g., appropriateness, feasibility)Service outcomes (e.g., effectiveness, equity)Client outcomes (e.g., satisfaction)Health outcome (e.g., substance use)
Challenges Barriers	Confounders	Confounders	Implementation determinantsBarriersFacilitators

Table 3.

Application of Epidemiology, Economic Evaluation, and Implementation Science within the CDC Evaluation Framework

CDC Evaluation Framework Phase (12)	Disciplinary Examples and Opportunities for Synergy
CDC Evaluation Framework Phase (12)	Epidemiology	Economic Evaluation	Implementation Science
Assess context	Assist in the prioritization of programsObtain data on substance use outcomes among adolescents in a school district and review available evidence on potential programs to identify which have the most potential to address the problem at hand. Often quantitative data that may come from health service records as well as youth surveys and school-based records.	Assess the cost literature on existing programsIdentify and assess what cost literature exists on existing SBI programs as this may allow for determining what costs do not need to be collected because some are already available and determine what costs may need to be collected during the Gather Credible Evidence phase.	Develop partnershipsPartnerships developed during the assessing context phase can help build buy-in among formal and informal partners (e.g., school boards, school leaders and staff, parent groups) to facilitate successful evaluation of the SBI through, for example, participation in qualitative focus groups, interviews, and surveys.
Assess context	Opportunities for Synergy: Engaging partners in the acquisition of epidemiologic data as well as developing partnerships to better understand what the existing programs are and where to identify cost-related literature for those SBI programs
Describe the program	Identify risk factors and causes of diseaseFor a school-based SBI program, this might include identifying past trends and historical information on youth substance use behaviors.	Identify activities to cost using the program logic modelIn order to carry out activity-based costing, the activities in the SBI program logic model inform which activities are selected to cost. If the SBI program included substance use prevention counseling with students, identifying activities to cost might include consideration of the counselorrelated activities including preparing for the session with students, carrying out the session, and following up with a phone call or emailing students after the session is complete to schedule a follow-up.	Identify implementation outcomesIdentification of implementation outcomes may help to define and describe implementation strategies to integrate the SBI program into the school. If the SBI program included a counseling component, it may be important to assess implementation outcomes like feasibility and acceptability among the school counselors via qualitative interviews or surveys.
	Opportunities for Synergy: Epidemiologists, economists, and implementation scientists work with program evaluators to better understand and define the outputs of the SBI program by defining what the intermediate requirements are to achieve the primary health-related outcomes desired. The types of economic analyses planned, such as cost-effectiveness or budget impact analysis, will require using the inputs, activities, and outcomes of the logic model.
Focus the evaluation questions and design	Determine outcome data sourcesIn design considerations for evaluation of an SBI program, this might include whether to have another school not implementing the SBI as a comparison group.	Develop an economic evaluation planEconomic evaluation plan within the larger evaluation design for the SBI program, including decisions about approach (e.g., taking a health care sector or school-based health center vs. societal analytic perspective) and plans for collecting and analyzing the cost-related data	Describe an implementation planThrough qualitative methods, identifying who will be delivering the SBI program in the school setting and what are the needed implementation strategies.
Focus the evaluation questions and design		Opportunities for Synergy: Costing of the implementation strategies in addition to intervention delivery.
Gather credible evidence	Identify intervention impacts Estimate individual risk of disease.This includes assessing the impacts of an SBI on the population of focus, such as what changes in substance use behaviors may have occurred among students because of the SBI.	Collect start up and intervention cost data.In some instances, qualitative data collection such as conducting interviews among persons delivering the SBI, may help to ensure data collected are accurate and consistent.	Describe and track program adaptations.Using qualitative methods, detail and describe each of the SBI intervention components and track any program adaptations that may occur, for example due to the intervention context in order to understand why certain substance use outcomes were or were not achieved.
Gather credible evidence		Opportunities for Synergy The intervention component descriptions can be helpful for providing background and detail on the activities being measured for cost purposes.
Generate and support conclusions	Interpret sensitivity analyses in light of primary outcomesConclusions about the impact of the SBI on substance use behavior outcomes should be based on the results, including results from any sensitivity analyses such as looking at impacts specific to certain student population or school.	Assess value of intervention and potential budget impacts in wider dissemination.This includes carrying out the cost-effectiveness analysis and budget impact analysis for the school-based setting of the SBI program.	Interpret outcomes achieved in light of implementation outcomes.The findings based on decisions made in the earlier phases of describing the SBI program and focusing the evaluation will be pivotal for the justification of conclusions.
	Opportunities for Synergy Strong integration of the epidemiologic and economic methods is needed to appropriately analyze the data, following the specified methodological plan.
	Opportunities for Synergy Qualitative implementation outcomes can help triangulate quantitative epidemiological and cost outcomes.
Act on findings	Share findings linking disease cause, associations, and potential solutionsEpidemiologic methods can identify causes of substance use misuse, along with associations and potential solutions that can help inform policy and public health decisions on substance use prevention programming	Share cost findings with policymakersEngagement with influential policymakers in the school and government policy space around financing strategies and opportunities for embedding an effective SBI into school-based setting through consideration of cost-related findings.	Disseminate best practicesThis is an opportunity to leverage the cross-cutting action of “engage collaboratively” – given the number of interested parties for a school-based intervention, an analysis of those parties, their roles, and the preferred communication methods facilitates dissemination.

Highlights

The disciplines of epidemiology, economic evaluation, and implementation science have confusing and conflicting terminologies when applied to program evaluation

Clearer understanding of how these disciplines overlap and where they differ are needed in program evaluation

Program evaluations and evaluators benefit for clarity on how to integrate epidemiology, economic evaluation, and implementation science methods into evaluations

Examples of how epidemiology, economic evaluation, and implementation science fit within program evaluation are described in the context of a program evaluation framework