The second phase [2009–2013] of the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) transitioned from scaling up HIV prevention and treatment to promoting country ownership and capacity building for sustaining program’s monitoring performance and evaluating key program indicators for strategic decision-making (Office of the Global AIDS Coordinator, 2004). In close collaboration with its global and national partners, PEPFAR became well-positioned to transform the successes and challenges associated with early rapid scale-up of anti-retroviral treatment (ART) into future opportunities for sustainable, cost-effective, country-owned programs and systems (Basheka & Byamugisha, 2015; Collins et al., 2010; Porter et al., 2012; Palen et al., 2012; UNAIDS, 2004). As part of increased program strengthening and capacity building, many African, Asian, and Latin American countries began establishing national HIV/AIDS monitoring and evaluation (M&E) units, implementing new programs focused on strengthening health systems, and increasing health workforce capacity (Porter et al., 2012).

Despite a variety of guidelines and harmonizing efforts by numerous stakeholders to integrate M&E as a fundamental aspect of global HIV program planning, implementation, and improvement, efforts were hampered by lack of human resources, inadequate physical infrastructure, low organizational capacity, and insufficient funding (Collins et al., 2010; Porter et al., 2012; Program/Project Monitoring & Evaluation for Accountability and Learning (PMEAL) - Self-paced Certificate Course ∣ ReliefWeb). While there has been an increase in the number of M&E training programs in the past several years, most programs are online, cost-prohibitive, or targeted at higher levels of the healthcare providers rather than site-level healthcare workers (Basheka & Byamugisha, 2015; Program/Project Monitoring & Evaluation for Accountability and Learning (PMEAL) - Self-paced Certificate Course ∣ ReliefWeb; UNAIDS, 2008). Many of the PEPFAR-supported sites have limited or no internet access (Basheka & Byamugisha, 2015). Furthermore, the real-life experiential sharing with peers and trainers have demonstrated improved competencies (Kolb, 1984). Many countries do not offer national-level certification or training for M&E for site-level healthcare workers and most people learn on the job, through peer-to-peer training and guidance documents (Armstrong & Kendall, 2010; Kolb, 1984; Program/Project Monitoring; UNAIDS, 2008). Usually the M&E curriculums are standardized with little flexibility to cater to a country’s specific M&E needs (Basheka & Byamugisha, 2015; Murray & Wibulpolprasert, 2004; Program/Project Monitoring; UNAIDS, 2008). To address these gaps, as part of CDC’s larger continuous effort for M&E capacity building, CDC staff in collaboration with ICF Macro, developed an M&E curriculum in a workshop format with five modules to serve as a capacity-building resource for PEPFAR/CDC-supported countries. The curriculum was intended to have enough flexibility to suit both site-level healthcare workers and program staff who may or may not have received training in M&E previously. We describe implementation of the M&E curriculum in three PEPFAR-supported countries, and whether knowledge gained during the training, translated into practice. We plan to use the recommendations from these findings to revise and improve future M&E capacity-building trainings.

As baseline knowledge of M&E varied by audience in the different countries, headquarter CDC staff closely coordinated with the CDC country office staff, Ministry of Health (MOH) officials, and implementing partners to determine training scope, delivery language, translation if multiple languages were needed, and logistics. We offered customizable trainings to PEPFAR supported countries through in-person discussions and M&E distribution lists. Three PEPFAR-supported countries (Ethiopia, Guatemala, and Cameroon) volunteered and accepted this capacity building activity customized to their country context. The curriculum content was translated into Spanish and French to facilitate the delivery of the trainings in Guatemala and Cameroon, respectively.

Relevant modules were selected by each of the countries based on the country’s HIV epidemic control priorities and the M&E needs of the participants (Table 1).

We collected evaluation data through three activities. Before and after each training, we assessed the perception of confidence and knowledge gained. We also assessed if or how the knowledge gained from the M&E curriculum was translated into practice, through an online survey administered in February 2020. This survey was administered approximately one year after the Ethiopia training, nine months after the Guatemala training, and five months after the Cameroon training. All data collection tools were translated into the relevant language of the participants, including Spanish for participants from Guatemala and French for participants in Cameroon, and then back translated into English by CDC headquarters and country staff to ensure reliability and completeness of the answers and interpretation.

Since evaluation of these trainings were capacity building activities supported through CDC country offices, CDC determined this as a public health program evaluation activity, and not research, as defined in 45 CFR 46.102.

The M&E curriculum promoted building capacity through practical examples and exercises. Since the curriculum was customized to fit specific M&E needs, knowledge was translated into practice through awareness, acceptance, and adoption by implementing partners, and health facility staff in 3 PEPFAR-supported countries. There was improvement in confidence, satisfaction and knowledge gained by all participants.

It is important to note that this was the first M&E training for a majority of the participants in Cameroon, despite most of them being employed as M&E officers and data clerks in health facilities. While knowledge gained remained low in Cameroon (the average score of 58 % on the post-workshop knowledge test), one-third of the participants who responded reported using the information learned while being on the job. From the training in Cameroon, it became clear that despite some of the participants learning basic concepts of M&E for the first time, they were able to translate knowledge gained through this training into practice with the potential to improve M&E capacity at health facilities.

The empowerment of participants, with their perception of knowledge and -confidence gain as well as their reflection of where they applied knowledge into practice was a recurring theme. Usually, evaluations of M&E capacity building activities reveal recent knowledge gain at the end of trainings without assessing whether knowledge got translated into practice. This is usually seen as a gap in previously published work on M&E capacity building activities (Amde et al., 2014; Nalubega & Uwizeyimana, 2019; Zachariah et al., 2009). While the quantitative results showed improved confidence for participants who attended the M&E trainings and responded to CAS for select modules within countries, they also showed statistically significant improvement in confidence levels both within and between the three countries. Since the number of participants who responded to CAS for each of the learning objectives varied, mean differences between pre-post confidence surveys should not be over-interpreted. For example, a smaller number of participants attended and responded to fewer modules, especially for learning objectives relating to “data use for program planning and decision making” or “develop a plan for data analysis and use”, that may have impacted statistical significance of the results.

Qualitative results coded into primary themes and then secondary themes based on content analysis demonstrated which components of the training may need to be revised. While feedback was predominantly positive, specific learning objectives for specific modules such as Data Quality and Program Monitoring may require revisions, including, but not limited to, more practical examples, and details on how to do run charts, or designing more interactive sessions on indicator development.

To customize the M&E training modules for country needs, many factors may have influenced assuring the level of translation and delivery of the training materials in the appropriate language. Considerations on time needed to customize, prepare, and implement not only the training materials, but also translate all the materials including evaluation tools need to be factored in for delivery planning and resource management (Omar et al., 2009). Also, note that language translation may have impacted understanding and adoption of knowledge as well as feedback gathering through evaluation (Omar et al., 2009). Therefore, participants’ satisfaction with module content, their confidence and knowledge gained, as well as their desire to apply these concepts to real-world settings varied.

To our knowledge, other M&E curriculum delivery models have not examined long-term impacts in the months and years following training implementation, – which leaves a gap in understanding if participants are using knowledge gained to benefit their practice. We attempted to address this gap by administering TKP surveys where we discovered participants were widely implementing the lessons learned from the M&E curriculum in their routine practice. While participants from the Cameroon and Ethiopia responded to share specifically how they were using the knowledge acquired during this M&E training 6–12 months later, it is clear from their descriptions of the concrete examples that many adequately understood the concepts taught in the workshop and were able to apply them to their work. By only measuring knowledge gain or satisfaction, we would have missed out on a valuable insight; scores on a test do not translate well to the ability to conduct work in the real world.

We developed and implemented the curriculum customizing the materials to best fit participant needs. However, all 3 countries did not complete all of the 3 evaluation activities to analyze impact of the training i.e., CAS, PPKT, and TKP (e.g., Due to logistical issues, Guatemala could not implement PPKT or TKP). This limited the ability to assess in-depth differences in the outcomes between the 3 countries. Moreover, since the surveys assessed self-reported confidence, we cannot rule out a Hawthorn effect (when study subjects change their behavior in response to being observed). Furthermore, though modules were selected by the countries, customization for maximum benefits may have compromised comparability of data across countries for evaluation purposes. This led to fewer participants responding to evaluation of certain modules. Thus, survey responses for some learning objectives may have varied. Moreover, the customized module selection made it challenging to compare within and between survey results. TKP surveys had a low response rate and may not reflect all participants’ perspectives. The TKP surveys were implemented during the first quarter of the COVID-19 pandemic, which may have reduced the response rate. In addition, while the majority of TKP survey respondents demonstrated meaningful changes to their M&E practices, this may not be generalizable to all participants as those who made changes may be more motivated to respond to a survey.

Previous reviews of M&E trainings indicate a focus on the technical and contextual aspects of M&E but very few do a thorough job of accommodating the institutional factors such as national M&E plans, policies and frameworks or stakeholders’ organizational planning structures (Amde et al., 2014; Mapitsa & Khumalo, 2018). Our M&E training addresses some of these institutional factors by incorporating a more flexible curriculum adaptable to country-specific contexts, and examples from different levels of the government such as local, regional and national level stakeholders. Even after a year, a sub-group of participants not only felt they had mastery of M&E concepts but were also able to apply knowledge into practice. Incorporating feedback from participants on additional training requests, especially for training on new indicators, and advanced data analysis and visualizations will be considered as the M&E curriculum is revised.

To align with PEPFAR’s strategic priority of building capacity as the cornerstone of transitioning HIV treatment programs to country ownership sustainably, scale-up of training activities could include creating virtual capacity building activities to accommodate the rapidly changing training landscape due to COVID-19 (Chan et al., 2010; El-Sadr et al., 2012). A potential next step would be to share best practices and deliver a revised curriculum through virtual communities of practice (Struminger et al., 2017). Conducting trainings on virtual data quality assessments or data analysis, and re-formatting and recording videos of modules for wider dissemination and incorporation into an online learning platform [U.S. CDC DGHT’s Monitoring and Data Analysis Branch (MEDAB)] may be explored as a natural next step (Bian et al., 2020). Continuing to conduct a post-training assessment after 6–12 months would ensure data capture and documentation on whether knowledge is getting translated into action. This will be critical to institutionalize and sustain a culture of M&E in program planning and decision making. This process may be iterative and build upon participant feedback to improve the training experience while still retaining the flexibility and adaptability of the curriculum. This will help develop local as well as global M&E capacity building environments to support achieving the UNAIDS 95–95-95 targets by 2030 (UNAIDS, 2015).