Step One. Semi-structured teleconference interviews were conducted between June and August 2004 with each Pfizer Grantee to gain a clearer understanding of proposed program implementation, evaluation plans, and program contexts. Central to these discussions were grant proposals submitted and funded by the Pfizer Initiative.²⁸ Teleconferences were developed 1) to gain insight into how programs may have evolved in scope, goals and objectives since the initial proposal submission; 2) to identify how program success would be defined; 3) to assess intervention data needs, current data collection methods and procedures; 4) to identify how collected data would be used to assess program success; and 5) to determine technical assistance needs. This process was critical in order to determine whether each CBO was prepared to engage in evaluation capacity-building and cross-site assessment. A by-product was identification of each organization’s perceptions of the links between program components and outcomes, as well as the organizational and community factors that moderate program function.

Step Two. The MSM PRC assisted each Pfizer Grantee in the development of a logic model specific to their program needs and foci. The logic model conceptualization provides a blueprint for developing a clear connection between program components, facilitates timely monitoring of planned activities, and illustrates the order of events necessary for successful achievement of objectives.^29–34 A logic model is a conceptual and visual depiction of a program’s goals, inputs, strategies, outputs, outcomes, objectives, and the assumed causal relationship between them. A major strength of logic models is their utility in illustrating the linkages between existing conditions, activities, outcomes and effects. Each element should be guided and informed by community stakeholders, among others, to ensure that both community needs and assets are understood and that activities are realistic and appropriate for the community context.³⁵ Logic model development is important to evaluability assessment because it supports determination of whether programs have intentionally thought through the connection between their effort and anticipated outcomes. Preliminary logic models were drafted by the MSM PRC based on program descriptions submitted in initial funding applications and semi-structured teleconference interviews.

Step Three. Site visits were conducted by MSM PRC trained evaluators July to September 2004 to gain a first-hand perspective into the contexts in which each Pfizer Grantee existed. Meetings with key administrative staff, program staff and target audiences provided a two-way learning opportunity for the MSM PRC and CBOs 1) to establish trust among key stakeholders; 2) to engage in dialogue and to review documents critical to assessing evaluation capacities, including drafted logic models; and 3) to discuss technical assistance related to Pfizer Initiative evaluation requirements. Logic models drafted by the MSM PRC were also revised and approved by each Pfizer Grantee during site-visits to ensure that they accurately reflected program components and success indicators. Table 1 includes results of the EA process that advised evaluation capacity-building activities.

Each step in the evaluability assessment was systematically staged to build upon each other in preparation for evaluation capacity-building through understanding of each CBO’s program and evaluation context. Employing a one-size-does-not-fit-all approach, the semi-structured interviews engaged each CBO to better understand evaluation TA needs, as well as to contribute to identifying content for cross-site evaluation capacity-building implementation. The information gathered through each interview was used to develop the drafted logic model representing each program’s activities and to link program success and its measurement. Site visits were conducted to not only confirm the accuracy of the logic model drafted by the MSM PRC with CBO program staff, but to establish trust, positive rapport, and to identify local program stakeholders that would be engaged in the Pfizer Initiative over time. As detailed by Painter et al., the exercise of EA lies in its ability to explore (compare and contrast) perspective of the program among program stakeholders, defining the programs logic, and defining program achievements towards the development of more useful and culturally relevant evaluation.⁶ Figure 1 includes each step in the EA process, how each preceding step informed the next towards the development of evaluation capacity-building implementation (Step 4).

Step Four. Morehouse School of Medicine PRC used EA results to develop evaluation capacity-building content and subsequent implementation. The MSM PRC rationale for development of targeted approaches, advised by the EA, that are tailored to community-stakeholders is supported in the literature, which indicates that the design of activities must account for, among other elements, ECB participant characteristics, assessing their organizational resources.^36–37 Pfizer Grantee executive directors and program staff members were actively engaged in formal didactic presentations, in-person training workshops, teleconferences, and web conferences (all with feedback and discussion sessions) designed to enhance capabilities to plan, implement, and evaluate their community-level interventions to prevent HIV/AIDS. Capacity-building used a participatory evaluation framework with all sessions offering interactive opportunities to apply learning to funded interventions and discussions to maximize inputs through opportunities for questions and answers, reflection on and application to program implementation challenges and experiences.^22,34,38 Table 1 contains the schedule and content description of evaluation capacity-building provided.

As part of its overall evaluation, MSM PRC developed and conducted an initial cross-site program assessment survey (C-PAS) of the CBOs to determine their knowledge, skills, and abilities for planning and conducting community interventions and their technical needs.²⁶ The C-PAS was adapted from a previous instrument used successfully in substance abuse intervention training, pretested among MSM PRC community coalition board members, and reviewed and modified prior to administration.³⁸

The C-PAS questionnaire was designed to be completed in 15–20 minutes and to capture the key personnel’s self-reported knowledge and skills relating to key steps in developing, implementing, and evaluating a community-based intervention as well as the organization’s specific abilities to perform essential functions. We used knowledge to refer to the individual understanding of queried activities and skills to represent individual proficiency to perform certain activities. Knowledge and skills were measured on a five-point Likert scale using the six key variable constructs of community program planning and implementation: problem identification, needs assessment, developing goals and objectives, gathering program input and feedback, prevention implementation planning and evaluation of community intervention. Technical assistance needs were assessed (as a yes/no variable) for the following: logic model development, data collection tool development, data management, protocol development, qualitative and quantitative methods, and evaluation. Table 2 provides examples of questions including the C-PAS questionnaire.

The C-PAS was initially administered February 2005 and at two subsequent time points to determine changes in knowledge, skills, abilities, and technical needs prior to each capacity-building effort during the Initiative. Self-administered questionnaires were sent to each organizational designee by U.S. and electronic mail to gain varying perspectives on program processes. Pfizer Grantees received two U.S. mail reminders and one telephone reminder to return surveys. The survey response rate was 84.8% for the first C-PAS, 78.3% for the second C-PAS and 93.2% for the third C-PAS. Differences in responses among executive directors and primary program staff people were combined due to non-significant differences between groups.

Following completion of all evaluation-capacity-building activities, MSM PRC conducted one-on-one semi-structured teleconferences from September to December 2006 with each Pfizer Grantee to gather feedback on the evaluation capacity-building partnership throughout the Initiative and continued challenges and technical assistance to be overcome as they worked to implement new competencies after the Initiative’s conclusion. Teleconferences were conducted with each Pfizer Grantee organization between October and December 2006. Twenty Pfizer Grantees participated in the teleconference, resulting in a 91% participation rate. Each teleconference was taped with prior permission of participants and transcribed. Qualitative data analysis was conducted by two reviewers, who independently identified trends and themes in responses for teleconferences.

Evaluability assessment helped to identify areas for targeted evaluation capacity-building among Pfizer Grantees. Challenges identified during this first step were the evidence-base for the development of evaluation capacity-building activities for the duration of Pfizer Initiative. Each area described below represents identified challenges. Table 3 details each identified challenge that was addressed.

Comprehensive C-PAS results have been detailed elsewhere and are briefly highlighted here to indicate key successes in implementation and to provide context associated with the implementation of community-engaged model and its associated results.³⁴ They indicate a steady positive progression in CBO’s abilities to plan, implement and evaluate programs. Further, technical assistance needs significantly decreased from 48.7% for logic model development from the initial C-PAS to 27.8% for the second CPAS and 12.2% for the third C-PAS (p=.002). The technical assistance needs for utilizing qualitative and quantitative methods also decreased, from 64.1% in the first C-PAS to 50.0% and 36.3% in the second and third C-PAS, respectively (p=.048). Evaluation development technical assistance needs decreased from 84.6% at the initial C-PAS to 63.9% in the second C-PAS and 39.0% in the third C-PAS (p<.001). Table 4 includes a sample of questions included in the C-PAS questionnaire.

Table 4 details composite scores for knowledge, skills, and abilities measured across each administration of the C-PAS. Individual knowledge scores, among those in the composite, associated with evaluation indicated significant increases for goal and objective development at the second and third C-PAS (p=.045). Among individual ability scores in the composite, developing data collection tools (p=.002) and analyzing collected data (p=.011) indicated significant increases across C-PAS administrations.

Pfizer Grantee organizations were asked to describe the aspect(s) of evaluation capacity-building activities provided by the MSM PRC as a means through which the MSM PRC could identify the strengths and challenges of its ECB model. The training format preferred by most Pfizer Grantees was face-to-face conferences (cited 14 times). Hands-on application and networking were valued characteristics of the face-to-face format. The quotations below represent the perspectives of Grantees on the process.

Okay, definitely the face-to-face are much, much, much more effective …. They just are. I don’t think you can really make up for being together in the same room sharing. It’s just so much more education that way. I know it’s much more expensive to do things that way but it really is the thing that works the best.

…I’d say the hands-on application …because we had opportunities to participate in several group-related activities to apply some of the information that we’ve learned just with each other but also knowing how to bring back once we got back to our individual sites.

…Cause I think you get more and plus you’re focused then. On teleconferences no matter how much you’re trying to not have other stuff around you it’s hard to keep that away.

Organizations identified data entry training, logic model training, and qualitative training, respectively, as the most valuable preferred content areas offered by MSM PRC in evaluation capacity-building activities.

Epi Info …that was that thing that we’re going to be using more than anything else because what we needed was to have a way of collecting data that was simple and efficient and that anybody could utilize and anyone can use.

The other thing is the logic model of the subjects that we learned, that was the one that is most effective and that we needed to know more than anything else because that’s something that we can use for other grants and other programs and it helps us set up our programs in a logical fashion that shows effective change and then the other thing is that the last conference we were talking about data collection and things like that at the conference.

Well, the logic models was the best thing that happened and the fact that it was insisted and stressed so much at every meeting and every encounter that we had with [MSM PRC] I think it was great. Plus it’s easy for us [not] to develop a logic model for other programs that we had or programs that we wanted to implement.

Pfizer Grantees were asked if there were areas specific to their organizations that continue to be program evaluation challenges. Most discussed program evaluation challenges related to data collection, entry, and analysis.

All that [data] is handwritten now and how we could capture that on a form and then it in a system that we can sort of look at is our challenge.

Yeah I would say the data input. We’re just very small. We don’t have anyone. It is partly our fault too but it’s an area that we constantly face [as] a challenge.

…Looking at having a consistent model to track our staff so they have better understanding—our entire staff-so they have a better understanding of data collection and analysis and then have regular like quarterly opportunities to review data, to analyze it and to reflect on it as a staff.

Organizational and staff resources were all cited challenges faced by Pfizer Grantees when considering the priority of evaluation. Many expressed program evaluation challenges surrounding staff and resources and staff involvement and buy-in.

Well there is a cap, there is a 7% cap on the amount of money, federal money, that you can use in your agency for administration. So if you can imagine you have a $100,000 grant but you can only use 7% of it for any of your infrastructure, any of your data collection and all of that doesn’t count, it makes it very, very difficult because they don’t want you spending their money on [evaluation] …many of our agencies run with this Catch-22.

…A couple of us [are] no longer here and I have myself but there’s only a couple of people with the organization that understand what evaluation is, what its value is, why we do it and the fact of the matters is if the staff doesn’t understand and appreciate evaluation …. So I think that talking about evaluation with our whole staff is important because they’re the ones that are going to input the data. They’re the ones that are going to be collecting this.

I think our biggest challenge is time or being understaffed here and it’s hard to really allocate the time to do this because it is important and it’s hard to take the time and do it when you have so much else to do ….

Assessment should be on-going in order to increase the likelihood of local evaluation buy-in and the diffusion of skills, knowledge and abilities within organizations. In addition to limited resources (human, fiscal and time), many organizations experience high staff turnover at the administrative and support levels. Participants in capacity-building activities during the first year of a project may no longer be involved in projects by the funding cycle’s conclusion. This may affect the degree of improvement in evaluation capacity. On-going assessment will also help in the design of training activities that can reasonably accommodate the format and content preferences of participants.

Pfizer Grantees preferred methods that would be most relevant to their current programmatic needs, rather than those that were abstract or perceived as “academic.” Sensitivity to the balance between theoretical relevance of reliable and valid evaluation measures must be coupled with a respect for the resources and value-added from the perspective of the target audiences. Some organizations reported the value of electronic tools and templates that could also be conveniently used at their local organizations such that they would not, as they said “have to reinvent the wheel.”

Knowledge, ability and skill acquisition are often challenged by the contexts within which programs are implemented. While quantitative (e.g., survey) data provide a perspective that is frequently said to be more objective, qualitative data collected from the target audiences, through teleconferences, focus groups and open-ended questions help to shed light on what the data means. Both methods can be used together to develop the most effective strategies which are tailored and informed by the unique needs of recipients.

While Pfizer Grantees have made important strides in evaluation capacity, it has been incremental and operationalized within the contexts of staff turnover, the competing demands of stakeholders, and among other priorities. Organizations have stressed the importance of offering program-specific support in areas including data entry, management and analysis. Programs may alternatively be offered resources through which they can access free or low cost trainings or on-line modules that can be shared with other program staff in order to institutionalize evaluation skills within their organizations.

Evaluability assessment, capacity-building activities and qualitative assessment show that evaluation capacity-building is an iterative process, requiring partnerships between funders, evaluators and grantees over time to fine-tune strategies in an audience-targeted manner because one size does not fit all. Current trends in community-based participatory research and evaluation encourage the empowerment of community organizations to understand, plan, and conduct local evaluation efforts. The process toward these practices necessitates the acquisition of practical skills that are valued within each organization and become part of all programmatic activities. The benefits of evaluation skill development and buy-in include the development of measurable indicators, replicable implementation and evaluation plans, and dissemination of systematic, data-driven results, greatly influencing a program’s likelihood of sustainability. While processes, outcomes and recommendations described in this article represent a community-campus partnership to strengthen small to mid-sized community-based organizations (organizations with annual revenues of less than $1 million) conducting HIV/AIDS prevention in the South, this model may be adapted to similar partnerships with organizations that are addressing other chronic health disparities. Stated differently, strategic community-based participatory approaches, including evaluability assessment and evidence-based capacity-building initiatives are relevant approaches to consider to increase the likelihood of ownership and buying in evaluation partnerships with CBOs.

The MSM PRC seeks to identify the degree to which programs are achieving developed goals and objectives and to enhance the skills critical to planning, implementing, and evaluating programs for increased program effectiveness and sustainability. This report represents evaluation measures, interactions and processes that were integral to increased evaluation capacity, as measured by results of the Cross-site Program Assessment Survey (see Table 4). It is anticipated that the dissemination of cross-site program assessment and capacity-building activities will help to inform and improve the quality of measurable, replicable evidence-based HIV/AIDS prevention programs.