Programs selected for review were divided into two priority groups (i.e., “first priority” or “second priority”) based on the degree to which program goals and characteristics aligned with project focus and the potential to withstand rigorous evaluation. Further details related to priority groups and inclusion/exclusion criteria are outlined below.

Programs classified as first priority for inclusion in the review met the following criteria:

Programs classified as second priority for inclusion in the review met the following criteria:

Programs excluded from review met the following criteria:

To identify HBYVP programs for inclusion in the review, we conducted a search of the 2017 American Hospital Association (AHA) Annual Survey of Hospitals to identify potential HBYVP programs. The AHA Annual Survey of Hospitals database represents information that is directly provided by hospitals and health care systems and covers an array of data points, including demographics, operations, service lines, staffing, expenses, physician organization structures, beds, utilization, and population health (American Hospital Association [AHA], 2017). A research analyst from the project team extracted and organized program details for all programs. Additional information for each program was gathered via environmental scan and other EA activities (i.e., interviews with key program staff).

There were 6,282 hospital organizations (hospitals or hospital systems) in the 2017 AHA database. Of the 6,282 hospital organizations included in the AHA database, 1,200 hospitals included a Level I, II, or III trauma center. Nine hundred of the 1,200 hospital organizations with a trauma center reported having an affiliated violence prevention program operating in or around the hospital. Thus, the universe of programs under consideration was comprised of 900 hundred violence prevention programs.

Of the 900 violence prevention programs reported to be affiliated with a Level I-III trauma center, there were 465 violence prevention programs that reported specifically operating within a Level I-III trauma center. Of the 465 programs operating within a Level I-III trauma center, 82 were duplicates and excluded from review. The remaining 383 programs were reviewed and classified into priority groups as outlined by the SSA methodology.

From this review of the 383 non-duplicated programs operating within a Level I-III trauma center, 105 programs were selected for inclusion in the review. Of the 105 programs included in the review, 57 programs were classified as first priority for inclusion in review as they focused on youth and child violence and matched the inclusion criteria (i.e., implementation for at least 1 year, monitoring and measurement of youth violence prevention outcomes, and inclusion of community-based partnerships or wraparound services). Forty-eight additional programs were classified as second priority for inclusion in the review as they did not have a primary focus on violence prevention, reported little to no violence prevention-related data collection, and focused on a coordinated nurse response rather than prevention. Additional information was not available for the remaining 278 programs that were excluded.

The team was able to reach 31 of the 105 first and second-priority programs identified via email or phone to verify and collect additional information. Programs that were not reachable tended to have outdated contact information listed and/or were no longer in operation. After initial outreach activities, 15 programs were confirmed to have met the inclusion criteria and were recommended to be included in the panel review. The other 16 programs were not recommended as they were either community-based programs with services that did not originate in the hospital, were implemented for less than 12 months, were currently involved in an outcome evaluation, and/or are no longer in existence. Figure 2 outlines the screening process used to identify HBYVP programs included in the review.

In preparation for the review panel, the team populated a program description form using information collected during the environmental scan. The program description form included details about the program’s location, delivery setting, number of youths served, priority areas, implementation duration, violence prevention data, target population, a detailed description of the program, type of data collected, and data collection method. Additional information included on the program description forms was gathered from discussions conducted with hospital staff at each of the 15 programs. Discussions with hospital staff were guided by formative questions used to help confirm that the programs were considered active and were providing services to the target population at the time of the review. Additional questions were used to help identify additional and knowledgeable point(s) of contact for each program, guiding theories/theory of change used by the program, and other important background information related to program development and implementation. The program description forms served as the primary reference materials for the reviewers during the initial review panel.

To reduce the likelihood of bias during the review process, team analysts ensured that all programs were de-identified, assigned a color pseudonym, and all potential identifiable information was removed from the forms shared with reviewers (e.g., program city, name of affiliated programs, and partnerships). This study received an exemption from the Office of Management and Budget responsible for reviewing federal regulations and handling proposals and communications under the Paperwork Reduction Act of 1980.

Two review panels were conducted with 12 violence prevention experts experienced in various aspects of HBYVP (e.g., program administrators, designers, facilitators, and researchers) to identify characteristics of programs suitable for an EA. Reviewers assessed and provided feedback for each program described in the program description forms using an online platform known as SurveyMonkey Apply (SurveyMonkey Inc., San Mateo, California). Team analysts also recorded meeting minutes during the review panels. The scoring guide used to rank the programs included six categories (i.e., program design, potential impact, evaluation capacity, sustainability, replicability, and transferability). Reviewers were provided with the scoring criteria/ description for each category and asked to rank the program using a point system. Cumulative scores were used to rank the programs. Thematic analysis of the panel discussion notes was used to identify common themes that were shared during the panel session. These themes were reviewed by the group of panelists as a part of the member-checking process. The panelist suggested the following summary of characteristics as those to consider when selecting programs for an evaluability assessment (EA).

EA is a pre-evaluation activity designed to maximize the chances that any subsequent evaluation of programs, practices, or policies will result in useful information (Leviton et al., 2010). Nine programs were contacted to conduct an EA site visit. One program did not respond to numerous outreach efforts, resulting in eight programs included in the EA process. While EAs are traditionally conducted in-person, travel restrictions and safety concerns related to the COVID-19 pandemic resulted in a minor limitation, causing the project team to adjust the original approach and develop a virtual alternative. As a result, the EA site visits for this project were conducted virtually using the Zoom web conferencing platform (Gray et al., 2020).

Prior to each virtual visit, program materials were reviewed by the site visit team, followed by virtual site visit interviews, and virtual site visit debrief. During each virtual visit, at least two members of the project team assessed program implementation, data collection, and outcomes. All virtual EA site visit sessions lasted approximately 1 hour. Semi-structured guides were developed by the project team during study development and the guides focused on program design, potential impact, evaluation potential, sustainability, replicability, and transferability. Questions for the key informant interview guides were grouped into the following categories and sub-categories: (a) History (Program Description, Implementation Feasibility, Replicability/Transferability); (b) HBYVP Program (Program Design, Potential Impact, Implementation Feasibility, Sustainability, Replicability/Transferability); (c) Community Support and Partnerships (Reach to Target Population, Implementation Feasibility, Sustainability, Replicability/Transferability, Staff/Organizational Capacity); (d) Evaluation (Potential Impact, Evaluation Capacity, Options for Further Evaluation, Staff/Organizational Capacity); (e) Funding (Implementation Feasibility, Sustainability, Replicability/Transferability). EA project team members interviewed key program staff, including lead program administrators, partners, program staff, evaluators, and other stakeholders.

During each virtual EA site visit, two EA project team members took detailed notes of the program’s responses. Thematic analysis was used to group the notes into major categories and themes. Team members compared the content of their notes and themes identified. The notes were synthesized into a report identifying any areas of ambiguity. This summary report was shared with the programs after each virtual site visit as a part of the collaborative member-checking process. A preliminary logic model (based on the formative data from environmental scan activities) was drafted prior to each virtual EA site visit to describe the program’s resources, activities, and outputs. This logic model was reviewed and further refined with the program during the virtual EA site visit and debrief. EA team members then drafted a summary report, describing the program’s approach, major elements as planned and implemented, observations and findings from the team members, and the program’s evaluation potential, impact, and recommendations. The report also included the detailed updated logic model of the program and an appendix of reviewed documents. Programs were instructed to review the notes and logic model and provide clarifying details to the EA team within 2 weeks following the virtual interview.

After the summary report was verified by the program staff, EA project team members were responsible for de-identifying all program documents including any available annual reports and classifying information for each program using a color-coding system to classify programs before they were shared with the panel reviewers. This de-identified information, along with the initial program information, was made available to the review panel for a second review during Step 5.

Characteristics of violence prevention programs ready for a rigorous evaluation included a program champion, a strong connection to the community, warm referrals/handoffs to community resources, robust or tailored case management, and some experience with violence prevention, either through training or past lived experience. If programs were built on a violence prevention theory, it was either the Cure Violence model (Butts et al., 2015), SNUG (guns spelled backward and an adaptation from Cure Violence), or some other strengths-based prevention model designed to highlight and build on existing resources and strengths of the community and individuals served. All programs had some qualitative or quantitative data that could be used or optimized for future evaluation efforts. Overall, each of the programs’ features was tailored to their respective city or county depending on the community or hospital needs.

Nine of the original panelists returned to review the EA findings for the eight programs that participated in the EA and provided feedback using SurveyMonkey Apply during a second virtual panel review meeting. In preparation for the virtual panel review, panelists reviewed program materials and resources (i.e., program brochures and overview documents, intake forms, data collection forms, available annual reports, and other program documents as available). After reviewing the information, panelists were asked to score the programs using the scoring guide and criteria used in the initial review (i.e., program design, potential impact, evaluation capacity, sustainability, replicability, and transferability). Like the initial review panel meeting outlined in Step 3, panelists highlighted shared characteristics of the programs that should be considered for further review and potential selection as a program with potential to be recommended for rigorous evaluation. Panelists’ scores and feedback from the initial and second panel reviews were extracted, compiled for thematic analysis, and shared with panelists as a part of the member-checking process during the panel session.

Across the eight programs, major themes and concepts discussed included the target population (the total number of clients served within the target population [youth 10–24 years old]); program goals (the program’s alignment with priority areas and outcomes); data infrastructure (the program’s capacity to collect and measure youth prevention); sustainability (the funding mechanisms for longevity of the program); and transferability/replicability (the strategies that could or could not be duplicated in different and similar settings).

Major areas for discussion included the program’s primary goals, data accessibility, service availability, and program duration. Some of the programs’ priorities were not within the scope of this project’s specific aims. For example, two programs did not offer a holistic approach to violence prevention (e.g., referral and access to community services, complementary and alternative therapies, and outpatient clinical services). Instead, only one aspect of a promising violence prevention program, such as mental health, was highlighted. Strengths of the programs included access to multiple data sources, such as the hospital’s trauma registry and/or electronic health records, and the ability to share data among these systems. Although three programs had access to these data, the ability to merge data across systems did not appear feasible without major modifications or resource investments. Seven programs implemented a client-centered approach and provided wraparound services to their clients. However, the client’s duration from enrollment to completion of the program was not clear or indefinite for all programs.

Findings from the SSA methodology, including the EA site visits and two virtual panel reviews, identified three hospital-based programs ready for a more rigorous outcome evaluation. The remaining four programs revealed there were factors that indicated a strong readiness for evaluation, even if the program was not yet prepared. The remaining hospital-based program did not include factors that made them ready for an evaluation. These factors included:

Factors associated with limited readiness during the virtual panel review included:

The capacity for future rigorous evaluation should be a significant consideration during program planning and design. Findings of this review indicate that there is a need to educate hospital-based program designers on the importance of including multi-level evaluation (i.e., formative, process, and outcome) at every stage of program design, from program planning to program implementation and dissemination of findings. The CDC’s Framework for Program Evaluation in Public Health states that program evaluation should be “practical, ongoing. . .and involve all program stakeholders, not just evaluation experts” (Milstein & Wetterhall, 1999). Programs that do not incorporate these elements, including the creation of evaluation plans at the beginning of program development, risk implementation of programs that do not have the infrastructure necessary to identify areas for improvement, or ability to demonstrate a return on investment.