As identified in Figure 1, step 1 of identifying community-defined indicators of programmatic success of a youth violence prevention program involved focus groups and community engagement activities. During planning for the PARTNERS project, qualitative, participatory methods were employed to help the program developers “hear” and accommodate how the community casts and prioritizes the problem of youth violence.¹² With recruitment efforts fostered by PARCC, four focus groups were held with community residents (n = 22), youth-serving agency representatives (n = 11), parents and caregivers (n = 3), and community leaders (n = 10). Results from the four focus groups and other community engagement activities revealed a total of 43 community-defined indicators reflecting community perceptions of violence prevention program success.¹³ Examples of the community-defined indicators included traditional outcomes such as reduced violence and neighborhood trash, but also included newer constructs such as “more adults intervening for youth,” which was defined as the expectation that adults would reach out for youth in positive ways. The results are described in detail in Hausman and colleagues.¹³

Reported more fully in Hausman and colleagues,¹³ step 2 in Figure 1 involved matching the community-defined indicators with existing and publically available data. This step involved an iterative process of review of data availability and accessibility, and feedback from members of our community advisory board. Matching efforts focused on large publicly available data sets, such as crime data, the Litter Index,¹⁴ and other regionally specific population surveys: 23 databases with a total of 47 datasets were reviewed. Publically available data were found for 19 of the 43 community-defined indicators. For example, “Cleaner Streets, Cleaner Neighborhoods” was considered measureable by the Litter Index, a nationally standardized measure collected locally in Philadelphia.¹⁴ Review and feedback from our community advisory board indicated that only two of the identified sources of data were considered unreliable to community leaders and therefore were not further considered. For example, one dataset that we accessed was not suitable for our work because the process by which the community products it counted reflected larger political forces and not local, community-driven efforts.

Step 3 of matching community-defined indicators to available data focused on data being collected for the preliminary evaluation of the PARTNERS intervention project.¹⁰ Step 3 in Figure 1 is the key focus in this manuscript and emphasized the use of data already collected or planned for collection. This strategy provided several key advantages. First, we did not add to the participant’s research burden, an underlying value consistently expressed by the members of PCVPC and the larger community. Respect for this concern focused the measurement building process on data that was or would become available through the surveys already being implemented in the community for the PARTNERS evaluation. Second, the PARTNERS evaluation used a variety of established psychometric scales with known reliability and validity to evaluate specific constructs in youth development. The standardized instruments used by the PARTNERS team to evaluate the effectiveness of the intervention included measures of aggression, oppositional defiant disorder, social information processing, anger management, attitudes towards violence, youth assets, self-esteem and leadership.^15–26 These instruments provided a pool of individual items whose essential quality could be relied upon. Both academic and community partners from PARCC saw strength in using these data from the PARTNERS project, with the anticipated possibility that we could eventually see how the “new” community constructs would compare with those measured by the established scales. The matching process was conducted as “proof of concept” that elements of existing standardized psychometric tools could be used to measure community defined constructs that reflect but not replicate more traditional intervention outcomes. For purposes of describing the process, we focus herein on the results of the matching process involved with the youth instruments.

An eight-member team composed of five academic and three community researchers from the PCVPC was formed to conduct a systematic item-by-item review of the evaluative standardized instruments administered in the youth violence prevention intervention. For clarity here, we will call these team members “raters.” Academic partners included four faculty members and one doctoral student training with the PCVPC. The three community members were members of PARCC and PCVPC, and lived and/or worked in West/Southwest Philadelphia. They had backgrounds in business, grassroots community organizations, and community and economic development. The community members were nominated by PARCC to participate in this research because of their ability to represent the intervention community and they demonstrated a clear interest in promoting the health of the communities in West/Southwest Philadelphia. These community members had been involved with the development and implementation of PCVPC’s research endeavors from the outset.

The process of matching the indicators to the evaluation tools started with having one rater (an academic partner) review all of the items in the youth surveys used in the evaluation (n = 208). The first rater assigned each item to an indicator where it appeared to be relevant; if no match was found, the item was discarded from the matching process. The academic and community partners chose this as a first step to help expedite the process of review. Although several members of the research team had participated in the original analysis of the focus group data and the entire research team had discussed the community-defined indicators, the research team did not formulate standardized definitions of the community-defined indicators to use during the matching process. This opportunity for further interpretation had strengths and limitations.

During the initial step with the first rater, 98 youth survey items were matched to 11 (of the 43) community-defined indicators. In keeping with a process that aimed to be inclusive of different interpretations for matching, individual survey items could be matched to more than one community-defined indicator. The 11 community-defined indicators initially matched by the first rater were academic performance, future orientation, helping others, increased civility, decreased truancy, more participation in community organizations, less cursing, more parental involvement, showing kids love, more adults intervening for youth, and kids helping around the house.

Once items were grouped under their matched community-defined indicators by the initial rater, the seven other raters reviewed the initial matching and scored their agreement (yes/no) with the match. The matching by the initial and subsequent raters was recorded and examined for patterns of agreement. Results of the matching process for each item were discussed among the team and this provided opportunity for any needed clarification or questions answered. We then reviewed the patterns of agreement across the team for each set of items matched for each indicator. After discussion and review of the empirical data from all raters, the research team decided that five of the remaining seven raters needed to agree on a match in order for an item to be retained for future analyses. This allowed for a clear majority of the group to agree on a matching. Additionally, this solidified that no item would be retained that had the three community team partners disagreeing with a match. At the end of the process, 92 youth survey items were identified and agreed upon as potential matches to 11 community-defined indicators. For example, 14 items from the Alabama Parenting Questionnaire,¹⁷ 1 item from the HARE-Area Specific Self-Esteem Scale,¹⁶ and 3 items from the Youth Asset Survey¹⁵ matched to the indicator “More Parental Involvement.”

In our matching results, it is important to note that disagreement with matching of items to an indicator did not fall along academic/community lines. There was only one match of an item to an indicator that was retained where two out of the three community raters disagreed with the academic remainder raters. For the rest of the items retained in the matching process, two or more community partners agreed with the academic partners. No items had all three community members disagreeing with the rest of the raters. We saw this as strength in the process for communication and common views. We had one example of where an agreed upon definition of an indicator construct might have likely yielded different results was observed in the matching process for “increased civility.” For this indicator, two academic team members consistently disagreed with the rest of the raters on 43 items. The decision rule of five out of seven agreement maintained that the 43 items could not be rejected, but two important points emerged. First, no other indicator had 51 items to be reviewed for matching. Second, through discussion, we assessed that the two dissenters were clearly defining the construct in a different way than the rest of the team. Keeping true to the established process required keeping the results as is, but it became clear that this was one construct where further work was needed.

We recognize the limitations to our process. First, the reliance on existing instruments used by the PARTNERS evaluation limited the number of available pool of items. Second, all eight raters did not review each item from all of the standardized instruments. Although the first rater purposefully erred on the side of inclusion rather than exclusion, there may be items that the first reviewer did not include for further review by the group that others may have included. Additionally, because it was an academic team member who did the first pass, it might introduce an academic bias to the process. There might also have been even more uniformity in agreement with proposed matches if standardized definitions of the community-defined indicators were provided at the beginning, rather than leaving interpretation to the raters. However, there were patterns of high agreement among raters and results indicating acceptable internal consistency, even without standard definitions.

We proposed important next steps in this process (steps 4 and 5 in Figure 1). We need to conduct statistical analysis to assess the validity and reliability of the new indices derived from the matched items from the different established scales. Statistical evaluation (validity and reliability) of the new indices will likely refine the number of items that match to indicators. We also need feedback from the larger community to strengthen the process and validity of the work.

Because we used tools that are being used to collect data in the context of the PARTNERS evaluation, we will have access to data with which to conduct these analyses. We are careful not to interpret the results of our matching as a reflection on the valid, standardized instruments and the constructs they were originally designed to measure. The instruments were originally developed to measure specific constructs in youth development and are in fact of interest to the PARTNERS intervention. We will have the opportunity to analyze the new item configurations and evaluate consistencies and differences in the data between the original scales in PARTNERS and the items that comprise the new indices of the community-identified indicators.

Results of the entire matching process (to public data [step 2] and to PARTNERS evaluation data [step 3]) yielded six community-defined indicators that were matched to both public and PARTNERS data (increased civility, future orientation, academic performance, helping others, decreased truancy, and more residents participating in community organizations). This presents an additional area of evaluation in future phases, where we can compare and contrast the community-defined indicators based on data from these different sources.

Although the community-defined indicators of violence prevention programmatic success may not yet be considered universal and the availability of data will certainly vary by context, the process of reviewing and comparing community-defined indicators to academically defined outcomes and measures is informative and provides an opportunity to engage community members in the evaluation research process. The lessons learned from our experience encourage replication of the process in other communities and intervention program contexts and demonstrate how community voice can be woven into evaluation science in meaningful and important ways. Table 1 highlights recommendations based on our success and lessons learned.