The DELTA FOCUS program is intentionally oriented toward learning and applying data-informed lessons learned in real time throughout the project rather than just gathering insights at the end of the project. When introducing a developmental evaluation approach, Patton et al^14(pV) wrote, “developmental evaluation provides evaluative information and feedback to social innovators, and their funders and supporters, to inform adaptive development of change initiatives in complex dynamic environments.” The DELTA FOCUS evaluation model addresses the challenges of evaluating a multisite program that supports state domestic violence coalitions, embedded in different complex social environments, which are directly implementing and supporting the implementation of multiple, and often evolving, approaches to prevent IPV in communities.¹¹ To learn which approaches might contribute to the knowledge base of promising community-level IPV prevention strategies, CDC adopted a developmental evaluation approach and the Data-to-Action Framework¹⁵ to support adaptive learning and enhance evaluation practice.

As grantees’ strategies evolved over time (eg, changes in which strategies were implemented and how strategies were implemented), CDC’s approach to program oversight, administration, and support to grantees required flexibility. As a result, CDC staff elected to implement a data-to-action process guided by the Data-to-Action Framework to support project-wide management and continuous quality improvement. The Data-to-Action Framework was designed specifically for early-stage programs that can benefit from rapid feedback for the purposes of program development, refinement, improvement, and identification of barriers to implementation.¹⁵ The 4 key elements of the Data-to-Action Framework are a team-based approach, ongoing adaptations for early-stage developmental programs, use of rapid feedback cycles, and generation of actionable data.¹⁵ The rapid feedback cycles involved 5 steps:

The DELTA FOCUS program adopted the 5-step Data-to-Action rapid feedback cycle as a framework to assist in answering 2 evaluation questions: (1) What degree do the prevention strategies implemented by grantees at state and local levels contribute to what we know about IPV prevention? and (2) How well is the national DELTA FOCUS program being implemented? In the first year, CDC staff and the evaluation contractor (further referred to as consultants) convened meetings to clarify the intent and topics (step 1 of cycle) for each data collection (step 2 of cycle) or analysis of secondary data, analyzed and produced reports (step 3 of cycle), and held debriefing meetings to facilitate learning and decision making based on the reports (steps 4 and 5 of cycle). Findings from the reports and data collections were synthesized in a final Recommendations Report, together with recommendations for a data-to-action process for the next year. See the Table for detailed descriptions of the topics, findings, and use of findings.

To execute a data-to-action process at the project level, grantees generated the materials used as a data source; the consultants, contracted by CDC, analyzed the data and generated reports; the consultants then led CDC staff, as a team, through debrief meetings to reflect on the findings; and CDC staff used the findings for programmatic decision making and improvements with grantees (see Figure 1). The reports (described in detail later) follow the 5-step rapid feedback cycle and form the structure of the data-to-action process and assist in answering the program evaluation questions (see Figure 2). At a minimum, each report is completed annually, except for Actionable Reports, which are developed multiple times throughout each year, to ensure the production and use of actionable data. Zakocs and colleagues¹⁵(p473) define actionable data as “information that helps intended users make programmatic decisions.”

The Actionable Report Protocol is the planning document that guides the report and data collection topics for each year (see the Table for audiences and uses) and is generated between steps 1 and 2 of the rapid feedback cycle. The Actionable Report Protocol helps clarify the intent for each Actionable Report by determining the viability of the topic using a series of templates and guiding questions to generate and prioritize report and data collection topics based on the project team’s information needs. Similarly, a Supplemental Data Collection Plan is developed between steps 1 and 2 for the same purpose. Supplemental Data Collections (SDCs) are used to collect primary data from a subset of grantees through interviews and surveys in order to seek additional information or to clarify information beyond what was provided in the data management information system.

When the topics for the reports and data collections are settled, and the data are collected and analyzed (step 2), the Actionable Reports are generated (step 3). For Actionable Reports, data grantees submitted (Action Plans, Progress Reports, Evaluation Plans, and supplemental materials shared with CDC through a data management information system) are analyzed and summarized on the basis of the topics identified in the Actionable Report Protocol. The Recommendations Report, illustrated in the Table as occurring between steps 3 and 4, is used to document findings from SDCs, synthesize findings across Actionable Reports, to summarize what was learned through the data-to-action process throughout the year (including findings from Synthesis Reports), and to make recommendations for the following year’s process.

Synthesis Reports are the evolution of Actionable Reports in the final 2 years of the project. They underwent the same process and relied on the same data source as Actionable Reports but were designed to assist the project team in summarizing and describing program impacts and outcomes instead of identifying opportunities for program improvement.^* Specifically, in the first 2 years of the program, the project team needed to know what grantees were implementing in order to identify training needs, inform technical assistance to grantees on project calls and at grantee meetings, and assist grantees in forming a community of practice by identifying peers with whom they could communicate or collaborate. In the final years of the project, the team’s information needs shifted to needing to know what grantees were accomplishing and the types of impact they were making at the local, state, and national levels to inform reporting and accountability to leadership. The change in information needs resulted in the shift from producing Actionable Reports to Synthesis Reports.

Once all of the reports were generated, CDC consultants guided CDC staff through debrief sessions to discuss the findings of the reports and identify opportunities for program improvements (step 4). The CDC staff would then make decisions about how to execute the program improvements (step 5). For example, in the first Actionable Report, CDC sought to understand the scope of the IPV prevention strategies and targeted outcomes of the grantees’ efforts since they were given significant flexibility in deciding what to implement. In the process of developing the report, CDC staff and consultants discovered that there was substantial variability in the grantees’ strategies, which presented challenges to fully understanding and articulating the types of strategies grantees adopted. Thus, there was a need to categorize their work, which led to the development of prevention strategy categories and outcome types.

To accomplish this categorization, CDC consultants reviewed, coded, and aggregated information from the annual work plans (ie, Action Plans) of the program grantees. Key findings from this Actionable Report included the following: a majority of strategies in Action Plans were prevention focused; identification of a typology of prevention strategies was challenging because there was wide variation in how program grantees described their strategies; prevention strategies varied in desired change (ie, outcome) and target audience; and program grantees’ understanding of types of evidence differed from CDC’s definitions. As a result, CDC created a guidance document to clarify key elements of a strategy, developed a typology of prevention strategies, and created a document defining the different evidence types. This guidance was developed and shared in enough time for grantees to include the information in their next progress reporting.

Because of these Actionable Report findings, definitions for prevention strategy categories and outcome types were developed over time and, with grantee input, discussed and described on a technical assistance call, and then used by grantees when entering data into the data management information system. A year later, when this process was complete, another Actionable Report was developed that better described the scope of the prevention strategies implemented and the strategies’ measured outcomes. A similar process was used to clarify definitions of the different types of evidence. The CDC staff created a guidance document that defined the different types of evidence and provided examples, discussed this guidance on an evaluation technical assistance call with grantees, and requested that grantees update these designations on the basis of their new shared understanding. With each Actionable Report produced, CDC staff were able to describe program grantees’ efforts across sites, detect any issues that could hinder CDC’s evaluation or support to grantees, and provide proactive technical assistance to support grantees in the implementation and evaluation of their efforts (see the Table).

Data-to-action report findings have also provided useful material and information that helps ensure that CDC staff are being accountable to both internal agency leadership and the grantees themselves. Charts, tables, and other graphics that are regularly included in the data-to-action reports have been useful when CDC staff are requested to provide updates to management and leadership at different levels of the organization. Often visuals from Actionable Reports are used when CDC staff and consultants share report findings with grantees to facilitate their collaborations with each other, improve their reporting, and/or build their capacity for IPV prevention.

In addition, the SDCs are helpful in supporting CDC staff’s provision of technical assistance and building a learning community for DELTA FOCUS program grantees. The purpose of the first SDC (in project year 1) was to conduct interviews that would generate insights about how program grantees were experiencing aspects of DELTA FOCUS—communication, technical assistance, and a community of learning—so that CDC staff could take action to improve implementation. To address this purpose, CDC consultants conducted unstructured and thematic telephone interviews with a subset of grantee representatives identified by CDC staff. The SDC findings were discussed in the annual Recommendations Report. Themes emerged from the first SDC, including the following: monthly meetings with CDC staff and the in-person grantee meeting were noted as the most valuable support; all program grantees experienced positive communication with their CDC project officers; the community of practice platform and written feedback were experienced as the least valuable supports because of lack of use by peers and perceived inconsistency between CDC role as a partner and funder, respectively, and although program grantees desired to learn from and support each other, they all experienced barriers to creating the desired learning community and requested more face time with each other.

After debriefing the report findings CDC staff used the information to engage in a series of brainstorming sessions that resulted in the creation of a plan for programmatic improvement that was shared with program grantees, which covered topics that were identified as important to acknowledge and address. For example, on the topic of face time, CDC staff responded by hosting an additional grantee meeting to offer opportunities for more grantee interaction. Specifically, the seating for some sessions was arranged to facilitate program grantees doing similar work to sit together, and concurrent sessions were organized topically so that program grantees doing similar work would have the opportunity to collaborate. This program improvement was effective at achieving its goals, as a majority of attendees indicated that the meeting helped build relationships with fellow grantees and increase face time as requested. As for the community of practice platform, CDC staff acknowledged the challenges and launched a listserv as an easier way to share resources and announcements with each other and partners in their DELTA FOCUS programs.

The rapid feedback cycles urge staff involved in planning, program implementation, and evaluation to come together frequently and identify emerging challenges and ways to overcome these challenges. A data-to-action process also creates space to consider whether the evaluation is adequately capturing the necessary data and whether there are any opportunities to enhance implementation or evaluation in previously unanticipated ways. As a result, the project level evaluation of the DELTA FOCUS program is both systematic and flexible and maximizes the opportunity to engage in ongoing program improvement in using timely data and feedback to make any necessary modifications in real time. While the findings from the data-to-action process are project-specific, the process itself is generalizable across settings and at different program levels.

For example, data-to-action informs DELTA FOCUS grantees’ evaluation of their prevention strategies, in part, because CDC required grantees to implement their own data-to-action process and also because CDC intentionally designed its data-to-action process to include grantees as a secondary, and at times primary, audience (see Figure 2). During project years 1 and 2, one of the Actionable Reports was focused on a topic of interest to grantees. The CDC staff solicited topics from grantees over project calls, through an electronic community of practice platform, and in-person during grantee meetings. The CDC staff and consultants presented and debriefed the findings with grantees once the reports were completed. Over time, it was clear that grantees and CDC staff were interested in the same topics (eg, what are grantees implementing and how are they evaluating their strategies). During project year 3, the CDC staff and consultants updated or repeated topics with new information. When all of the Actionable Reports were completed, the findings across the reports were shared with grantees. In project year 4, the findings of Synthesis Reports were also shared with grantees.

In terms of the requirement for DELTA FOCUS program grantees to have their own data-to-action process, grantees benefit from the very nature of having a plan in place to assess implementation progress and success along the way and then use those findings for program improvement. One benefit is the ability to make midcourse corrections in how a program is implemented, which several grantees have reported doing by either changing program content or limiting the number of participants in an activity. Grantees have also used data-to-action findings to adjust data collection methods by switching from repeated survey administration (which was causing program participant fatigue and impacting the quality of the data collected) to optimizing existing strategy activities to collect real-time data in creative ways (eg, using interactive theatrical performances to gather information from the audience). While it is not within the scope of this article to go into full detail on grantees’ data-to-action processes and applications, it is important to acknowledge that the DELTA FOCUS program requirement for grantees to have their own data-to-action process resulted in intentional, systematic processes that are beneficial and increasing the likelihood that these programs will be successful.

Throughout the course of DELTA FOCUS, and representative of using a developmental evaluation approach, the data-to-action process yielded timely data that led to actionable information for CDC staff and consultants’ support of the program grantees and of the IPV prevention program implementation and evaluation. In addition to programmatic improvements, the data-to-action process itself improved over time. Some of those improvements include the following:

As suggested by the list of improvements and examples previously, and in response to the complexity of evaluating multisite programs with a variety of interventions, a structured yet flexible process is desirable because priorities and needs can change over a period of 5 years. Particularly for new or developing IPV prevention programs, the Data-to-Action Framework provides a useful evaluation approach that allows for and encourages ongoing opportunities to reflect on how the program is progressing. It may be especially helpful for multistate projects that have substantial complexity in how program goals are operationalized (ie, assists with standardization). This intentional effort to engage in a data-to-action process at the project level not only increases the quality of support and technical assistance but can also lead to outcomes that are more successful.